11institutetext: Department of Computer Science, University of Manchester,
Manchester, UK
11email<EMAIL_ADDRESS>
{caroline.jay<EMAIL_ADDRESS>22institutetext: Idiap Research
Institute
Rue Marconi 19, Martigny, 1920, Switzerland

# Scholarly AI system diagrams as an access point to mental models

Guy Clarke Marshall 11 Caroline Jay 11 and André Freitas 1122

###### Abstract

Complex systems, such as Artificial Intelligence (AI) systems, are comprised
of many interrelated components. In order to represent these systems,
demonstrating the relations between components is essential. Perhaps because
of this, diagrams, as “icons of relation”, are a prevalent medium for
signifying complex systems. Diagrams used to communicate AI system
architectures are currently extremely varied. The diversity in diagrammatic
conceptual modelling choices provides an opportunity to gain insight into the
aspects which are being prioritised for communication. In this philosophical
exploration of AI systems diagrams, we integrate theories of conceptual
models, communication theory, and semiotics. We discuss consequences of
standardised diagrammatic languages for AI systems, concluding that while we
expect engineers implementing systems to benefit from standards, researchers
would have a larger benefit from guidelines.

###### Keywords:

Neural Networks Systems Diagrams Conceptual Models

##  1 Introduction

Diagrams are a signifier, cognitive aid, and communication channel. In
describing software systems, diagrams often provide a level of abstraction
that facilitates an understanding of the overall structure, and the relation
between the computational artifacts of the system. Software system diagrams
have a dual role bridging between cognition and communication of humans, and
representation of mechanisms entailed by machines. In the words of Horn [10]:
“Visual language has the potential for increasing human bandwidth, the
capacity to take in, comprehend, and more efficiently synthesize large amounts
of new information”.

In the field of AI, the pace of development is high, and as such conferences
are the most prestigious academic venues. In their scholarly proceedings, we
find that the majority of papers include a system architecture diagram by way
of structural explanation. This is common across Computer Science and other
system-centric domains. What is less common is the variety of these
representations, even when compared with other research and engineering
domains. Despite being based on similar and mathematically-well-formalised
computational artifacts, such as neural networks, the diagrams have very low
consistency. In this paper, we utilise the opportunity provided by the lack of
convention to gain insight into the way the creators of AI systems are
choosing to communicate their systems.

This study is motivated by a number of questions. Why are diagrams being used
to describe NN systems? Why are the diagrams so heterogeneous? What would be
the impact if this heterogeneity was reduced through standards or guidelines?
In order to make progress towards these broad questions, we investigate the
following hypotheses:

  * •

H1: There is a relation between content included in NN system diagrams and
their role in the scholarly community as conceptual models.

  * •

H2: There are visual encoding prioritisations which align with subcategories
within mental models theories.

  * •

H3: At present, guidelines for diagram creation would be beneficial for
communication.

##  2 Background

AI systems are software, written in a programming language, and often based on
neural networks. A neural network takes an input (usually text, numbers,
images or video), and then processes this through a series of _layers_ , to
create an output (usually classification or prediction). Each layer contains a
set of _nodes_ which hold information and transmit signals to nodes in other
layers. Specific mathematical functions or operations are also used in these
systems, such as sigmoid, concatenate, softmax, max pooling, and loss.
Different architectures are used for different types of activities:
Convolutional Neural Networks (CNN), inspired by the human visual system, are
commonly used for processing images, and Long Short Term Memory networks
(LSTM), a type of Recurrent Neural Network (RNN) which are designed for
processing sequences, are often used for text. These neural networks “learn” a
function, but have to be trained to do so. Training consists of providing
inputs and expected outputs, so the system can learn how an input should be
processed. The system is then tested with unseen inputs, to see if it is able
to process these correctly. The data perspective (focused on vectors and
matrices) and the functional perspective are fairly distinct ways of thinking
about what is happening, which is part of the problem of communicating in this
area. The architectural perspective on the system can be to a greater or
lesser extent encompassing the data manipulations, which leads to a broad
spectrum of possible representations.

In terms of communicating about AI systems, it is not just _from researchers
to the public_ , but also _between researchers_ that attention is needed. The
act of communicating between researchers is not only about making code
reproducible [11], though we agree that is important, but making cognitively
accessible the systems and the computational artifacts of which they are
comprised. As a result of this, diagrams have a role to play in the ethics of
AI, and are a key way of establishing transparency and managing risks around
what the system is doing.

##  3 H1: There is a relation between content included in NN system diagrams
and their role in the scholarly community as conceptual models

To discuss this hypothesis, we first describe the heterogeneity in
representation, and then proceed to classify the observed phenomena, linking
the observed diagrams to conceptual and mental models.

###  3.1 Heterogeneity in representation

In principle, the primary purpose of diagrams in scholarly publications is
communicative. The authors are attempting to communicate through a
diagrammatic medium some kind of relational structure. Diagrams are ideally
suited to this task, and are used for this purpose in many domains [17].
However, there are other social aspects. Without passing too critical an eye
over the scientific endeavour, having a “good looking” diagram (both
aesthetically and technically) may improve perception of a paper, thereby
making it more likely to pass through peer review. While the visual encoding
methods are quite unconstrained and heterogeneous, it is conventional to
include an architecture diagram if the paper presents a new system. The
subsequent discussion attempts to make steps towards understanding the reasons
for, and consequences of, this heterogeneity.

Perhaps due to the complexity of these systems, there are a number of
diagrammatic representational choices that are made by different authors
attempting to express different things. Given the social nature of research,
it is curious that there is not more prevalent “copying” or adoption of
informal diagrammatic encoding conventions, or even convenience-based
similarity caused by the use of popular diagramming tools. Given that it would
be practically easier for authors to directly copy existing styles, it is
unlikely to be chance, but rather we argue that there must be a compelling
reason for authors to be creating such different diagrammatic representations.

A partial explanation for heterogeneity could be a lack of appropriate
diagramming tools. In a recent interview study involving technical domain
experts, Ma’ayan et al. [17] found that “To illustrate concepts effectively,
experts find appropriate visual representations and translate concepts into
concrete shapes. This translation step is not supported explicitly by current
diagramming tools”. This does not explain why AI software system diagrams are
so heterogeneous compared to other domains, nor does it explain the lack of
informal conventions.

Nefdt [19] argues that the state of understanding of AI systems is such that
“meaningful components” and compositional structures have not been established
for deep neural networks. He does not extend his argument to the entire
system, focusing instead on the mathematical “black box” that is a deep neural
network. If we adopt this stance over an increased scope, this would seem to
support the claim that there is an underlying cognitive reason for the
differences in representations. As such, we argue that insight into the
current state of understanding can be gained partially as a result of the
current “epistemic opacity”, and therefore allows an insight into the
representational priorities of the author.

We hypothesise that a particular representational aspect is prioritised by the
author either because it shows what they think is important, or because it is
what they would want to see in a diagram authored by their peers. In either
case the priority is effective communication. Differences in prioritisation
may be causing the creation of bespoke diagrammatic encodings. When
representing an AI system, the diagram author can prioritise different aspects
of what is represented:

  * •

Function: Operations which occur, representation transformations, and the
purpose of parts of the system

  * •

Data: The data model, type, dimensionality and how it is manipulated

  * •

Example: Showing the steps of an example input through that system

  * •

Contribution: Focusing on the scientific novelty of the approach, giving much
more detail in that area

  * •

Code: Important class names and the order in which they are called

  * •

Mathematics: Including specific mathematical functions

  * •

Index to text: Using a label structure to allow for easier referencing

These representational priorities result in different content being displayed
through different visual encoding mechanisms. It is usual to have aspects of
several of these priorities, as it is not the case that the prioritisation of
one aspect necessarily inhibits another. In terms of how the diagrams are
presented within a paper, some papers include multiple diagrams, either by
multiple figures or sub-figures. Often, sub-figures or boxes are used to give
both schematic and detailed views within the same diagram. Dependencies are
often indicated by arrows. Diagrams almost always represent important content
in natural language, such as labels or descriptions.

###  3.2 The content of diagrams

The inclusion of an “example” makes the diagram be “of” a particular run of
that system. However, it is understood that this is signifying the system
itself. There are some diagram users who take an example instantiation and use
this to generalise to the operation of the overall system [18]. Indeed, in
this way the diagram is supporting logical induction, rather than deduction
from general rules (which would be more classically descriptive of Function).
Similarly the omission of inputs and outputs (e.g. “text” or “probability”)
from the diagram makes the diagram be “of” a set of operators rather than of a
functioning system, which is equally understood as signifying the system
(indeed, including data).

Rarely are AI systems’ Knowledge Representations (links with semantics)
represented visually in this scholarly community. Instead, labels are favoured
for this important system component. In some diagrams, they include the
dimensions of the embedding space, a drawing of an arbitrary graph, or a
simplified example of part of the knowledge representation. The high
dimensionality and the sparsity involved makes visual representation
challenging.

###  3.3 Specific graphical objects

Arrows have been linked with the concept of functional processes [8]. All
surveyed diagrams used arrows, perhaps suggesting that the systems are
understood as being compositional and sequential (rather than as objects which
enact a function).

Labels for “layers”, sets of neural nodes often performing a particular neural
network function, are common. These neural network functions often have
metaphorical or descriptive names, such as “embedding” or “attention”, which
form an informal lexicon of computational artefacts. The labels are often used
alongside more complex sets of graphical objects signifying the system,
performing the role of (linguistically) simplifying and linearising the system
without taking a sentential form. To take a specific example, the layer labels
are often of the form “embedding layer”, i.e. mentioning the function. This in
turn might be labelling a set of blocks labelled “BERT”. In this sense, this
type of diagram contains cognitive support for two parallel mental models,
that of function in the original diagram, and that of state in the layer
labels. This particular aggregation-and-switching-mental-model usage of labels
is very common, but not exclusive. Some diagrams may instead have mathematical
functions (e.g. “tanh”) labelling layers. Principally the association between
these representations is grouping by alignment, but may additionally make use
of “linking” graphical objects such as arrows, brackets, or blocks, or
grouping by colour.

Many graphical objects, and almost all diagrams, utilise labels (in English).
This suggests a low level of semantic content within the graphical objects.
Perhaps this is because of the non-physical nature of the systems, which makes
it challenging to signify visually without the aid of an established formal
language.

##  4 H2: There are visual encoding prioritisations which align with
subcategories within mental models theories

This subsection aims to make progress towards the classification of AI system
diagrams. The link between diagramming priorities and mental models is
speculative, but could serve to explain some of the manifest heterogeneity.

###  4.1 Mental models and conceptual models

We can further refine this schema to correspond more directly to a theory of
mental models. Definitions of mental models are various, but can perhaps be
summarised by: “A mental model is a simplified representation of reality that
allows people to interact with the world.” [14]

In Computer Science, Guarino et al. [7] recently developed a model for the
relationship between mental models and conceptual models. For Guarino et al.,
mental models are “personal, partial accounts of the external reality,
filtered through the lens of a conceptualization, that people use to interact
with the world around them.” A mental model (or perhaps less ambiguously a
conceptual mental representation), is described by a conceptual model, in our
case the system diagram with a communicative purpose. “Note that, being an
information object, a conceptual model is always the result of an intentional
act. In other words, conceptual models are artifacts produced with the
deliberate intention of describing a conceptualized reality.” With further
specific reference to the Computer Science domain, Guarino et al. state that
“we may say that a computer program is a conceptual model of the computer’s
internal behavior, but only as long as its programming language’s primitives
denote concepts concerning computer behavior. If they rather denote data, we
conclude that such a computer program is not a conceptual model.” Guarino et
al.’s work highlights the challenges of philosophical precision in computer
science when discussing mental models. Applying these definitions to AI
systems diagrams, it is variable whether the diagram is a conceptual model. As
such, in this work we use mental models as a metaphor, rather than in the
strict sense. Usage of the term in this manner is common practice in human-
computer interaction.

To summarise our working definitions: The conceptual model is the artefact
itself, with communicative purpose, to articulate concepts and the
relationships between concepts. Diagrams are often used for this purpose, and
have been found to aid learning and reasoning [17]. In our usage, a mental
model is the way an author is thinking of the system when creating that
conceptual diagram for use in scholarly publication. We draw on psychological
mental model research, itself utilising a more rigorous definition, in order
to assist in the classification of conceptual diagrams, because the fit
appears natural. We explore potential consequences if this is aligned.
However, we do not suggest a deep claim about mental operation or cognition,
which would require an empirical study.

###  4.2 Classification using mental models theory

Figure 1: Representational choices in diagrams of AI systems

In this work, we are focusing on the diagrammatic metaphor being used for
_what is represented_ rather than the visual encoding. Bringing types of AI
software system diagrams closer to mental models, we can draw parallels with
mental model categories [22] and types of diagram observed. Diagrams may have
aspects of multiple categories.

  * •

Function: Explaining how the system operates by emphasising functional
aspects, such as mapping, input and output. Operations used as a verb. For
example, “word embedding” (a general term) rather than “BERT” (a specific
architecture for embedding [6]). This type often includes example input and
output.

  * •

Contribution (Purpose): Omits the majority of information other than that
required to understand the sub-section of the system that contains the novelty
of the system or approach.

  * •

Schematic (Form): Describes the system at a high level, uses probably a block-
style without iconic graphical objects. Does not include mathematical or data
details. The schematic may relate to classes or packages used. In order to be
distinct from Function, it commonly uses e.g. “BERT” rather than “word
embedding”. Also often aggregates graphical objects into modules. (In order to
be “state” (i.e. what it is doing) the block diagram should be verbs).

  * •

Data manipulation (State): Includes data dimensions, and usually a visual
representation of the data itself. It describes what the system does to the
data, so this also includes where operations are primarily labelled arrows
(rather than inside blocks).

  * •

Example: How example data transforms. Includes example input, output and
intermediate steps. Usually better relates to Function (how it operates)
rather than Form (what it looks like), but this depends on the graphical
objects used. It is useful to disambiguate this, particularly for Image
Processing, where often the diagram is a visual representation of the data
manipulation. In one sense this is the Form of the example data manipulation,
and in another sense the Function of the system on a single example. It is the
latter that we are concerned with, in our assertion that the diagrammatic
representation is signifying the system rather than the example. Note also
that the inclusion of intermediate step as using the same example is
important. Many diagrams include example inputs and outputs, but in the
processing of the system they do not utilise the example and the diagram can
be “schematic”).

Sometimes the diagrammatic representations found in conference proceedings
contain errors. In addition to typographic errors, these can be visualisation
errors, in the sense that the diagrams may cause confusion or inaccurately
reflect the reality of the underlying system. For example, the circles
representing vectors can represent a precise number of objects, or not. Fig. 2
shows an example where the pairs of circles represent two LSTM output vectors
(a common representational choice), while the three circles of $\hat{P}$ do
not represent three vectors but rather $j$ vectors, where $j$ is the number of
words in the sentence. The omission of the ellipsis following the embedding
layer appears to have led to this visual encoding choice. This unmeaningful
3-vector is repeated perhaps more dangerously in the final concatenation
before “multi-feedforward”. This can be understood by careful reading of the
words and formulae in the text, but could be misleading, as found in an
interview study by Marshall et al. [18].

Figure 2: An example diagram, used by Li et al. [16], using labels for Form,
graphical objects variously for Form and State, input and output for Purpose

We claim these potential different forms of diagrammatic representamen are
signifying the same referent, a software system, through different lenses. We
are not claiming that precisely the same sense is made with different
diagrams. Even strictly isomorphic representations have different semiotic
properties, such as the perceptual and cognitive properties of mathematics as
performed by equation, Euler diagram or language, or indeed without the
ability to perform gestures.

The separation of Form from Function is neither self-evident nor entirely
natural. In some software paradigms, such as functional programming, the
essence of Form and Function are entangled. By being faithful to the
perspective of the system’s Form and Function, rather than to the human
cognitive process which it is automating, we are able to be crisper with this
distinction.

The types of representamen employed are of particular interest. Our hypothesis
is that the diagrams, and the variety we see exhibiting the above
principalities, are a result of the range of cognitive functions being
employed by different users. At present, there is no common language they are
using to communicate. No suitable representation providing as cognitive
support such as symbolic equations gave to the mathematicians of ancient
Babylon, or the letter $x$ gave to Descartes [5], has been sought nor found.
In Physics, Coecke and Kissinger [4] use diagrams to reason about quantum
processes and to diagrammatically perform the calculations required to
understand them.

Another important aspect is reproducibility. From diagrams, the building of
great works of industrial-revolution engineering such as the SS Great Britain
was possible [24]. Indeed, the technique to perform diagrammatic projection
from 3D to 2D is seen as one of Marc Isambard Brunel’s main contributions
[20]. Circuit diagrams and other standard diagrammatic representations, often
implemented or overseen by professional bodies, have also enabled this
standard form of communication and reproducibility across many domains,
including Computer Science. With engineering diagramming technology in mind,
we proceed to consider specific graphical objects used in AI system diagrams.

###  4.3 Mental operations

Ryan Tweney’s lifetime of work examining Faraday’s cognitive processes through
images provides insight into a single author’s cognitive process (see Ippolito
and Tweney [12]). Without the breadth and depth of work of a single author, it
is not practical to carry out a similar analysis in AI systems. Nevertheless
this work served to inspire the approach taken here, and necessitates brief
discussion of “mental models” in the cognitive science sense.

All three primary mental operations of apprehension, judgement, and inference
[9] are at play in using AI system diagrams for research. Apprehension, being
the forming of a picture in one’s mind, is important for depicting and
understanding of the system. Any example of Judgement could be “this is
relevant”, and Inference “this is a contribution”. We focus on the creation of
mental representation, i.e. Apprehension. Note that “Mental models can be
constructed from perception, imagination, or the comprehension of discourse”
[13], and conference proceeding AI software system diagrams are in some sense
all three of these: Perception of the diagram, imagination of the system
operation, and comprehending scientific discourse.

We can categorise mental models as being about Purpose, Function, State and
Form [21], which has been directly applied to complex systems in a team
context [22]. In this sense,

  * •

Purpose is why a system exists;

  * •

Function is how the system operates;

  * •

State is what the system is doing;

  * •

and Form is what the system looks like.

These categories of mental models can be related to the types of diagrams
found in AI systems literature, as shown in Fig. 1. As highlighted by Rouse
and Morris [22], these facilitate describing, explaining and predicting:
“Mental models are the mechanisms whereby humans are able to generate
descriptions of system purpose and form, explanations of system functioning
and observed system states, and predictions (or expectations) of future system
states.”

At first glance, it may seem that the correspondence between mental models is
not strong for the “Function” model. As noted previously, in what is observed
some diagrams prioritise an instantiated example. This can be interpreted as a
different way of reasoning (inductive rather than deductive), and in both
cases appears to be focused around articulating Function. The detail-centric
vs function-centric comparison partially describes how abstract or schematic
the diagram is. A difference might be the inclusion of a block labelled as
“CNN”, only including hyperparameters in the former. The inclusion of
hyperparameters, which are non-structural configuration details, demonstrates
that the diagram supports a different level of reasoning.

Aspects related to mental models:

  * •

Purpose is why a system exists, specifically the activity it performs. There
are canonical tasks often linked to human cognitive operations (such as Named
Entity Recognition or Image Classification). The discipline is evolvable and
systems are comparable because there are these canonical tasks. Manifestation:
Often evidenced by inputs and outputs. Title, caption, other language or
images describing the task.

  * •

Function is how the system operates. Manifestation: The inclusion of system
operations as category terms. Graphical objects representing nodes.
Mathematical equations.

  * •

State is what the system is doing, with temporal elements. In AI systems, this
is about data states at a particular functional position. Manifestation: Often
shows how the dimensions of the data are changed, and usually a visual
representation of the data itself. It describes what the system does to the
data, so this also includes where operations are primarily labelled arrows
(rather than inside blocks).

  * •

Form is what the system looks like, and how the system is arranged. It is
succinct, expressive and general, comprised of the representational and
functional choices, how they are structured and composed, with emphasis on the
dependencies between components. The physical structure analogy is perhaps
closest to the classes and modules of code, and the shape of the data. This is
usually succinct. Manifestation: The inclusion of system components as nouns.
Often aggregating into labelled modules.

Note that a State Diagram in Computer Science is used to describe a system
with a finite number of states and their transitions, is available in a
standardised form in UML [2]. Whilst theoretically this could be used to
describe many AI systems, empirically this is a very uncommon scholarly choice
of representation.

###  4.4 Abstraction levels

Figure 3: Abstraction levels used in interacting with AI systems

Fig. 3 shows part of the context of abstractions and formalisms surrounding
system architecture diagrams, particularly in the absence of a formalism to
describe functional modules, or specified visual encoding frameworks. It is
quite possible that the system creator does not have a detailed knowledge of
how their code will execute (or even where, in the case of cloud computing).
Of course, mathematical formalisms and related work may also be expected to
form part of the author’s model of the system. With the complexity of AI
software systems, it is unlikely to be practical to perceive both the granular
underlying mathematics and the overall system structure in one instant.
Indeed, the structural explanation given by a diagram is necessarily
schematic, omitting information in order to be more efficient. In Fig. 3, each
of the arrows has some loss. Not all aspects of the author’s model are
included in the system architecture diagram, for example.

In a single diagram, there can be different levels of granularity. Graphical
representations of vectors are employed alongside higher level routines such
as biGRU or LSTM, themselves often comprised of multiple vector operations.
The different abstractions and granularity levels available to diagram authors
facilitates the diversity of representations that are observed, both within a
single diagram and across the diagrammatic corpus.

###  4.5 Diagram formats representing the same system

Figure 4: Example representations of the same system, obfuscating detail and
graphical encoding differences

Fig. 4 shows examples of the overall structure of diagrams commonly found in
AI systems. These schematics omit many details and complications of “real”
diagrams. See Fig. 2 for a typical example from ACL 2018, noting the lack of
linearity and variety of visual encodings used even within a single diagram.
Whilst the two versions of Function and State represent equivalent content (in
that there exists an isomorphism to convert between them), they are not
identical. Each structure places emphasis on different aspects of the
representation.

We can apply this framework to real examples. Fig. 2 shows a diagram taken
from proceedings of a top conference, describing the system architecture that
is their contribution. The diagram has labelled layers describing Form,
limited Purpose (example input and output, with no intermediate steps, but the
labels of “plot” and “ending” facilitate abduction), and the majority of
diagrammatic real estate given over to graphical elements describing vectors
(the circles). Independently of the text, abbreviations such as $s_{i}$ may be
understood by their context and abduction. Concatenation is a (nominalised)
verb on arrows, while Feature Extractor, element-wise product, and the
majority of other labels are nouns. Combined with the emphasis on the
data/vectors themselves, although they do not have dimensions indicated either
numerically or pictorially, it would seem this diagram is primarily State-
based, with Form labels. We hypothesise the author is thinking about the
system as manipulating data, as they prioritise communicating State but
without much information about the data itself (just grouping by colour, and
repetition for scale). This suggests that the author feels the “important”
part of the system to communicate is not necessarily the way the data changes
but the overall system. We hypothesise this “perceived importance” is in
itself a window into the mental representation of the authors: In the absence
of any independent advice into effective diagrams, authors using “examples” to
instantiate their cognition about a system will see it as important for others
to include “examples” in their own diagrams.

##  5 H3: At present, guidelines for diagram creation would be beneficial for
communication

Hypothesis H3 arose from the prior analysis, combined with reasoning
concerning the consequences of a formal language or standards for AI system
diagrams, balancing this against benefits associated with some level of
convention or familiarity, such as the reduction of “Semiotic-Semantic failure
in encoding” [15].

There are two aspects relating to the utility of standards or conventions. The
first is uptake, the second is effectiveness. In the domain of Human Computer
Interaction (HCI), where standards are commonplace, this would be seen as
usability and utility. In HCI, the “standards versus guidelines” distinction
is known, with the flexibility of guidelines being important for software
[23]. A more granular and up-to-date comparison between design guidelines,
style guides, principles and patterns is summarised by Al-Sa’di [1], with a
breadth of citations.

If standards were enforced, for example by publishers or conference organisers
(such as ACL, the Association for Computational Linguistics) then the concern
would be whether those standards genuinely improved the quality of the
communication. Standards would facilitate comparison between systems,
facilitating inter-diagram reasoning. If thoughtfully composed, standards
could be expected to streamline the diagram creation and interpretation
processes. However, by using standards, we would risk losing the author-
specific window into what is specifically important about the diagram in
question. For particularly novel architectures that make use of some hitherto
un-thought-of function, there are additional challenges, as the existing
standards may not facilitate effective representation of this novel element.
Further, we risk authors not being able to communicate their mental model.
However, it may be that readers have different ways of thinking about the
systems, and that a different diagram prioritising communication of different
aspects would be more pragmatically effective.

For well established architectures, standards would restrict the ability of
talented diagram authors to innovate. AI systems are rapidly developing [3].
The ability to create novel representations in a rapidly changing domain such
as AI systems is of value, as the representational requirements are also
changing. The useful information about an architecture can even change.
Initially, it may be that the key thing is to communicate novelty and utility,
primarily by getting the surrounding paper accepted into a prestigious
academic venue. In later stages of the system’s communicative “life”, the key
thing to communicate may be how to use, build, or adapt the system, in order
to obtain a higher number of citations and higher impact.

For the reader, standards or conventions would help avoid the barrier of
interpreting each individual author’s way of representing things. They would
reduce ambiguity, and reduce the potential for misunderstanding. In this way,
we would expect standards to make the diagram interpretation significantly
more efficient. Similarly, diagram creation would be more straightforward for
authors, as the content, graphical objects and method of arranging them
becomes more prescribed. The challenge is whether the content prescribed as
being included in the diagram represents _what is useful to be communicated
about the system_ in context of the wider communicative system.

In order to maximise the usefulness of any standards or conventions, it would
be necessary to understand deeply the requirements of the community, for
authors and readers, and to have an adaptive mechanism for updating the
standards as the requirements of the community change. We might expect this to
happen naturally, not being consciously or scientifically based on the needs
of users, but rather being optimised for authors and the peer review process.

For those involved in engineering of systems, the arguments in favour of
standard diagrammatic languages are strong. The allowance for comparison,
scale, precision, unambiguity, completeness and replicability, such as is
available to Mechanical Engineers, for example. It would be necessary that the
standards would allow communication about novel systems while complying with
those standards.

For researchers primarily concerned with a higher level of abstraction (rather
than system-building), the arguments are more nuanced. In their heterogeneity,
there is a window into the communicative goal of the author, allowing the
readers to gain some insight into their conception of what their system is,
how it works, and the contribution it makes. This insight is crucial for
scholars in order to deeply understand the publication. It would be
theoretically possible for representation to be facilitated by a domain-
specific language, as is done in mathematics or in quantum mechanics, as
described in Section 4. For AI systems, due to the prevalence of relations
between components involved, it would be intuitive to have this language be
diagrammatic. In the absence of such a mature language, there is still the
important issue of having to understand each individual author’s way of
diagrammatically encoding their system. Comparison between systems, important
for positioning the contribution of the new architecture against related work,
is difficult. Guidelines, a set of principles providing direction for
designing diagrams, would aid in reducing this cognitive barrier, and reduce
misunderstandings, whilst still affording the ability to customise depending
on the nature of the author’s representational requirement. Due to this, we
advocate the creation of unenforced diagram guidelines.

Guideline adoption could be organic, and their creation involving the
community, with inbuilt mechanisms for adaptation and evolution. This would
ensure higher effectiveness and longevity, at the expense of slower uptake.
The slower uptake would reduce effectiveness since, at least in the short
term, the format would be unlikely to be widespread. A gradual movement
towards _consistency where appropriate_ , particularly if we are able to draw
from the community of diagrams researchers a “fitness function” for assessing
these guidelines, would appear to be the most effective solution. This
approach would facilitate evolution, to create better representations that
better support the representational requirements of the community. Indeed, it
may be that future diagrammatic representation of systems will be interactive.
Even in this case, guidelines could support the design of this new medium.
Whether ultimately the solution is guidelines, standards, conventions, or a
new representational technology, remains to be seen. For AI system
engineering, when the field is at an appropriate level of maturity, a
standardised language would likely be very useful. From a research
perspective, the nature of AI system research is such that it is inherently
changing, with new architectures being designed which can be expected to have
novel representational requirements. Unless the field of AI system research
stabilises in terms of architectural representational requirements or changes
focus, thereby becoming something different to it is at present, a
standardised language would be inappropriate. Instead, evidence-based
guidelines should be encouraged.

##  6 Conclusion

We argue that the heterogeneity in diagrammatic representations of AI systems
is due to the inherent complexity of what is being represented and a lack of
obvious good representational choice for the system themselves. In the case of
many other scientific and engineering disciplines, a standard has quickly
emerged. It may be that we are too early in the life of “AI science and
engineering” to see this, and instead are able to gain insight from the
heterogeneity. The heterogeneity seen today is a manifestation of a lack of
conventional system elements and visual encoding principles.

AI systems are a new medium, which at its lowest representational level are
not easily interpretable. Currently there is heterogeneity in diagrammatic
representation. Diagrams are a useful, and efficient, way of understanding AI
systems. Diagrams are a fundamental signification layer, encoding the design
and meta-description of these emerging systems. We have linked this to a
morphic semiotic perspective: “code interpretation” being more computationally
expensive than “diagram interpretation”. We have hypothesised the diagrams
used at present to have relationship to mental models. As the community
evolves, and the representational requirements of these diagrams become
clearer, as with many languages, we expect some consolidation around effective
ways of signifying the underlying system, and providing cognitive support for
communicating and reasoning about these systems.

Finally, we encourage the creation of evidence-based guidelines for AI system
architecture diagrams, that are designed to be usable for creators of novel
architectures, and created mindful of the needs of diagram users.

## References

  * Al-Sa’di [2018] Al-Sa’di, A.: User interface guidelines for Tablet PC Arabic educational applications. Ph.D. thesis, Auckland University of Technology (2018)
  * Booch et al. [1998] Booch, G., Rumbaugh, J., Jacobson, I., Wesley, A.: Unified Modeling Language User Guide, The. Tech. rep., Rational Software Corporation (1998)
  * Chen et al. [2018] Chen, M., Herrera, F., Hwang, K.: Cognitive computing: architecture, technologies and intelligent applications. IEEE Access 6, 19774–19783 (2018)
  * Coecke and Kissinger [2017] Coecke, B., Kissinger, A.: Picturing quantum processes. Cambridge University Press (2017)
  * Descartes [1637] Descartes, R.: Discours de la methode pour bien conduire sa raison, et chercher la verité dans les sciences. Plus la Dioptriqve. Les Meteores. Et la Geometrie. – Qui sont des essais de cete Methode. De l’Imprimerie de Ian Maire (1637)
  * Devlin et al. [2019] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2019)
  * Guarino et al. [2020] Guarino, N., Guizzardi, G., Mylopoulos, J.: On the philosophical foundations of conceptual models. Information Modelling and Knowledge Bases XXXI 321, 1 (2020)
  * Heiser and Tversky [2006] Heiser, J., Tversky, B.: Arrows in comprehending and producing mechanical diagrams. Cognitive science 30(3), 581–592 (2006)
  * Hobhouse [2013] Hobhouse, L.T.: The Theory of Knowledge (Routledge Revivals): A Contribution to Some Problems of Logic and Metaphysics. Routledge (2013)
  * Horn [2002] Horn, R.E.: Visual language and converging technologies in the next 10-15 years (and beyond). Converging Technologies for Improving Human Performance 52(7), 124 (2002)
  * Hutson [2018] Hutson, M.: Artificial intelligence faces reproducibility crisis. Science 359(6377), 725–726 (2018), ISSN 0036-8075, doi:10.1126/science.359.6377.725
  * Ippolito and Tweney [1995] Ippolito, M.F., Tweney, R.D.: The inception of insight. In: The nature of insight, chap. 13, pp. 433–462, The MIT Press (1995)
  * Johnson-Laird [1983] Johnson-Laird, P.N.: Mental models: Towards a cognitive science of language, inference, and consciousness. 6, Harvard University Press (1983)
  * Jones et al. [2011] Jones, N.A., Ross, H., Lynam, T., Perez, P., Leitch, A.: Mental models: an interdisciplinary synthesis of theory and methods. Ecology and Society 16(1) (2011)
  * Lanigan [2013] Lanigan, R.L.: Information theories. Theories and models of communication 1, 59–83 (2013)
  * Li et al. [2018] Li, Q., Li, Z., Wei, J.M., Gu, Y., Jatowt, A., Yang, Z.: A multi-attention based neural network with external knowledge for story ending predicting task. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1754–1762 (2018)
  * Ma’ayan et al. [2020] Ma’ayan, D., Ni, W., Ye, K., Kulkarni, C., Sunshine, J.: How domain experts create conceptual diagrams and implications for tool design. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2020)
  * Marshall et al. [2020] Marshall, G.C., Jay, C., Freitas, A.: Understanding scholarly natural language processing system diagrams through application of the richards-engelhardt framework. arXiv preprint arXiv:2008.11785 (2020)
  * Nefdt [2020] Nefdt, R.M.: A puzzle concerning compositionality in machines. Minds and Machines pp. 1–29 (2020)
  * Rolt [1957] Rolt, L.T.C.: Isambard Kingdom Brunel: A Biography. Longmans, Green (1957)
  * Rouse et al. [1992] Rouse, W.B., Cannon-Bowers, J.A., Salas, E.: The role of mental models in team performance in complex systems. IEEE transactions on systems, man, and cybernetics 22(6), 1296–1308 (1992)
  * Rouse and Morris [1986] Rouse, W.B., Morris, N.M.: On looking into the black box: Prospects and limits in the search for mental models. Psychological bulletin 100(3), 349 (1986)
  * Smith [1986] Smith, S.L.: Standards versus guidelines for designing user interface software. Behaviour & information technology 5(1), 47–61 (1986)
  * Weale [1847] Weale, J.: The Great Britain Atlantic Steam Ship. John Weale (1847)


# Exploring and Evaluating Personalized Models
for Code Generation

Andrei Zlotchevski  McGill UniversityMontrealQuebecCanada
<EMAIL_ADDRESS>,  Dawn Drain  AnthropicSan FranciscoCAUSA
<EMAIL_ADDRESS>,  Alexey Svyatkovskiy  MicrosoftRedmondWAUSA
<EMAIL_ADDRESS>,  Colin Clement  MicrosoftRedmondWAUSA
<EMAIL_ADDRESS>,  Neel Sundaresan  MicrosoftRedmondWAUSA
<EMAIL_ADDRESS>and  Michele Tufano  MicrosoftRedmondWAUSA
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Large Transformer models achieved the state-of-the-art status for Natural
Language Understanding tasks and are increasingly becoming the baseline model
architecture for modeling source code. Transformers are usually pre-trained on
large unsupervised corpora, learning token representations and transformations
relevant to modeling generally available text, and are then fine-tuned on a
particular downstream task of interest. While fine-tuning is a tried-and-true
method for adapting a model to a new domain – for example, question-answering
on a given topic – generalization remains an on-going challenge. In this
paper, we explore and evaluate transformer model fine-tuning for
personalization. In the context of generating unit tests for Java methods, we
evaluate learning to personalize to a specific software project using several
personalization techniques. We consider three key approaches: (i) custom fine-
tuning, which allows all the model parameters to be tuned; (ii) lightweight
fine-tuning, which freezes most of the model’s parameters, allowing tuning of
the token embeddings and softmax layer only or the final layer alone; (iii)
prefix tuning, which keeps model parameters frozen, but optimizes a small
project-specific prefix vector. Each of these techniques offers a trade-off in
total compute cost and predictive performance, which we evaluate by code and
task-specific metrics, training time, and total computational operations. We
compare these fine-tuning strategies for code generation and discuss the
potential generalization and cost benefits of each in various deployment
scenarios.

Personalized Models, Code Generation

††copyright: acmcopyright††price: 15.00††doi:
10.1145/3540250.3558959††journalyear: 2022††submissionid:
fse22ind-p60-p††isbn: 978-1-4503-9413-0/22/11††conference: Proceedings of the
30th ACM Joint European Software Engineering Conference and Symposium on the
Foundations of Software Engineering; November 14–18, 2022; Singapore,
Singapore††booktitle: Proceedings of the 30th ACM Joint European Software
Engineering Conference and Symposium on the Foundations of Software
Engineering (ESEC/FSE ’22), November 14–18, 2022, Singapore, Singapore††ccs:
Software and its engineering Software testing and debugging††ccs: Information
systems Recommender systems

##  1\. Introduction

It is well-known that even the best models can fail to generalize properly to
new domains, and even to new users of said models. For example, a model
trained to answer questions in general may not answer StackOverflow questions
as well as the questions in the training domain, or a software developer in an
Enterprise environment with private code may have libraries and attribute name
which differ from public source code used to train a code synthesis model.

The current dominant paradigm in Natural Language Processing (NLP) modeling is
to pre-train a large transformer model (Vaswani et al., 2017a) on a large
corpus and then fine-tune it on a particular task of interest. For example, a
question-answering (Q&A) model is generally first pre-trained on a large
corpus of textual data for the specific language (e.g., Wikipedia, and news
articles in English), then fine-tuned on a task-specific dataset of paired
questions and corresponding answers. The pre-training process aims at learning
semantic vector representation of the language and words, while the fine-
tuning process specializes the model for a specific domain.

Transformer models are also increasingly the baseline architecture used for
code generation tasks, such as writing methods from natural language
description (Clement et al., 2020; Austin et al., 2021; Chen et al., 2021), or
generating test cases from the focal method under test (Tufano et al., 2021).
Similarly for NLP tasks these models are pre-trained on a large corpus of
natural text and publicly available source code and then fine-tuned on a
specific code-related task. Further, these models also may not generalize to
new domains of interest, and can benefit from task or even user-specific fine-
tuning, here called customization or personalization. Customization is
particularly relevant for code generation models since it provides several
benefits:

  * •

allows fine-tuning on source code data that may not be available when training
a base model (e.g., private repositories or internal codebases), enabling
improved overall performances on codebases with proprietary dependencies and
code styles;

  * •

the opportunity to improve data privacy by considering private or sensitive
data only during the customization process on the client side;

  * •

the opportunity to reduce deployment cost as customized models can offer
better user performance without increasing model size.

Custom models can provide clear benefits to users and model providers. We
envision serving tens or hundreds of thousands of custom models, but doing so
presents several logistical hurdles, including the costs of training, storing,
and loading these models into GPU memory for inference. Worse, memory costs
will only be exacerbated when working with ever larger and more powerful
models.

For these reasons, we investigate several customization approaches, some of
which can dramatically reduce the memory footprint and amortized computational
cost introduced by custom models. Specifically, we consider three fine-tuning
approaches: (i) custom fine-tuning, which allows all the model parameters to
be tuned; (ii) lightweight fine-tuning, which only optimizes the token
embedding representations or the final softmax layer; (iii) prefix tuning,
which keeps language model parameters frozen, but optimizes a small project-
specific vector prefix.

In our extensive empirical evaluation we found that all the customization
strategies lead to significant model improvements on a target project in terms
of both intrinsic and task-specific metrics. While there is no unambiguous
winner among the customization strategies, each approach can provide specific
benefits in particular deployment scenarios. This paper provides insights on
these customization strategies, their benefits and drawbacks, as well as
providing guidelines and suggestions on which one to use based on the training
cost, memory and storage, number of users, and deployment scenarios.

##  2\. Motivation

Software projects are often classified based on their architecture (e.g., web,
server, monolithic), domain (e.g., finance, healthcare), topic or usages
(e.g., games, editors). In this context, several techniques have been proposed
in the literature for the task of software categorization, which aims at
organizing projects into groups that broadly describe the behavior or topic of
the software. MUDABlue (Kawaguchi et al., 2006), relies on Latent Semantic
Indexing (LSI), an Information Retrieval (IR) technique, to automatically
categorize software systems in open source software repositories. For the same
task, LACT (Tian et al., 2009) uses Latent Dirichlet Allocation (LDA), and
recently neural text classification with word embeddings has been used
(LeClair et al., 2018) to categorize similar software projects.

While projects can be broadly categorized, each individual software project,
apart from trivial forks and clones, have peculiar characteristics which make
them unique. Codebases have different user-defined types, API usages, specific
coding styles, and identifiers’ preferences chosen by developers. These
idiosyncrasies represent an additional level of complexity for models that aim
at generating code for a variety of software projects.

This is exacerbated by the fact that transformer models only receive a
limited-size input during inference, often considering only the current source
code file. This confined window of tokens (commonly set at 1024 tokens) cannot
provide a complete view of the project with its peculiarities. An accurate
generation requires information about packages, classes, APIs, and identifiers
that are external to the portion of code provided as input. Thus, we argue for
personalized models that can generate custom code for specific projects.

Figure 1. Heat-map displaying the ratios of shared tokens among software
projects. Most projects share relatively few identifiers with other codebases.

As an exploratory study, we begin by observing projects’ diversity in terms of
tokens used in their source code. Specifically, we’re interested in
understanding the amount of shared tokens among different software projects.
This could serve as an initial, rough, proxy metric to measure project
diversity and potentially motivate the need for personalized models.

We select 930 Java software projects randomly sampled from GitHub, declaring
an open source license, which have been updated within the last five years,
and are not forks. These projects belong to the validation set of the open
dataset Methods2Test (Tufano et al., 2021). For each project, we collect all
the available .java files combining them into a single-project corpus. Next,
we remove licensing information and comments using regex, then tokenize the
corpus using gensim (Řehůřek and Sojka, 2010) tokenizer (with lowercase
setting). From the list of tokens, we compute the set of unique tokens used
within the project, and exclude the Java keywords from this set (similar to
stopwords).

(a) Custom

(b) L-EO

(c) L-LDB

(d) Prefix

Figure 2. Overview of the Customization Approaches - Transformer models during
fine-tuning, where the frozen parts of the model (not trainable) are displayed
in gray: (a) Custom fine-tuning modifies all parameters during training; (b)
L-EO trains only the embedding and output layers; (c) L-LDB allows to train
only the parameters of the last decoder block; (d) Prefix tuning adds a
trainable prefix to the encoder and decoder blocks.

For each pair of projects $p_{i}$ and $p_{j}$, with token sets $T_{i}$ and
$T_{j}$, we compute the shared token set $T_{i,j}=T_{i}\cap T_{j}$. Next, for
both $p_{i}$ and $p_{j}$, we compute their corresponding ratios of shared
tokens as follows: $R_{i,j}=|T_{i,j}|/|T_{i}|$ and
$R_{j,i}=|T_{i,j}|/|T_{j}|$.

Figure 1 shows the ratio of shared tokens between each pair of projects as a
heat-map. Projects are sorted in ascending order of the number of unique
tokens used in their source code. Blue values indicate a low ratio of shared
tokens (the darker, the lower), while red values indicate project pairs with
substantial amount of shared tokens. The heat-map appears mostly blue,
indicating that the majority of projects share relatively few tokens among
each others. The upper-right corner shows pairs with higher ratios (white/red
points), these are ratio computed for very small projects whose tokens are
contained in very large projects, hence the corner position. Overall, the
majority of project share relatively few identifiers with other projects.
Specifically, if we consider all the values in the matrix $R$ except for the
diagonal (i.e., token shared with the project itself), a project on median
shares only 13% of its tokens with another project, and the third quartile of
the distribution is below 23%.

We consider this study only as a preliminary analysis into the diversity of
software projects, which could motivate the need for personalized models. We
acknowledge the limitations of this study, which could be extended considering
different types of tokenizers, preprocessing steps, and metrics. In Sec. 4 we
design an experimental study that analyzes in details the impact of
personalization on the performances of transformer-based code generation
models.

##  3\. Approach

This section describes the proposed customization approach for code generation
models. We begin by formally defining the customization process, then we
provide details for each of the fine-tuning strategies.

###  3.1. Customization Process

We use the term customization to refer to the process of fine-tuning a model
$m$, previously trained on a generic dataset for a task $t$, with the goal of
improving its performance on a specific dataset $p$. The performance of a
machine learning model $m$ on a dataset $p$ is measured by one or more
evaluation functions $f(m,p)$, where $f$ can be either a maximization (e.g.,
BLEU, top-k accuracy) or minimization (e.g., perplexity) function. The
customization process is designed to modify the trainable parameters of the
model $m$, obtaining the model $m^{\prime}$, such that the performance of
$m^{\prime}$ on $p$ is better than what was attained by $m$. Specifically,
$f(m^{\prime},p)>f(m,p)$ for maximization functions, or
$f(m^{\prime},p)<f(m,p)$ for minimization functions.

In this work, $m$ is an encoder-decoder transformer model, $t$ is a code
generation task, and $p$ is a target software project to which we intend to
customize $m$.

###  3.2. Custom Fine-tuning

Custom fine-tuning is the most straightforward customization approach. The
model to be customized is taken as is and trained on a selected project. All
parameters are trainable during this process. Figure 2(a) shows the model
during fine-tuning, where all the parameters from the encoder and decoder
blocks, as well as embeddings and output layers can be modified.

###  3.3. Lightweight Fine-tuning - Embeddings and Output Layer (L-EO)

Fully fine-tuning a model for every project or user may be prohibitive in
terms of storage and memory costs. As a result, we explore ways to mitigate
these costs by reducing the number of parameters that vary from one custom
model to another. In our lightweight fine-tuning experiments, we achieve this
by freezing most parameters in the baseline model, and only keeping a small
subset trainable. Figure 2(b) shows the Lightweight fine-tuning - Embeddings
and Output Layer (L-EO) design, where most of the model parameters are frozen
(displayed in gray), and we allow only the embedding and output layers
parameters to be fine-tuned, following the approach in (Lu et al., 2021).

###  3.4. Lightweight Fine-tuning - Last Decoder Block (L-LDB)

In this lightweight fine-tuning strategy, shown in Figure 2(c) (L-LDB), most
of the model’s parameters are kept frozen, while only the parameters in the
last decoder block are trainable, this includes: self-attention, encoder-
decoder attention, layernorm and feedforward layers. This design decision of
training only the last decoder block is motivated by experimental results
analyzing the model’s parameter changes during custom fine-tuning. Figure 3
reports the average absolute changes, during fine-tuning, in the parameters
belonging to different Encoder and Decoder blocks for a BART model. We observe
that, as we go through the transformer model, the average change in parameter
values tends to increase, with the last decoder block showing the highest
changes in parameter values. As a result, we hypothesize that it could be
sufficient to tune the last decoder block and obtain performance improvements
similar to the fully custom fine-tuned model.

Figure 3. This figure shows the total average parameter change after fine-
tuning to a new project domain, showing that the largest parameter changes
occur in deeper parts of the model. This motivates our choice to try only
fine-tuning the later layers of the model.

###  3.5. Prefix Tuning

Prefix tuning was first introduced by Li and Liang (2021), with the goal of
fine-tuning a general model for different tasks. The technique concatenates a
sequence (prefix) of virtual tokens (trainable parameters) to the front of the
input of every encoder and decoder block. In our context, the intuition behind
this approach is that the prefix embeds the properties of a specific project,
which allows the model to generate customized responses for that repository.
Practically, we set the prefix length to 200 tokens, and thus with an
embedding size of 1024, this gives a total of $1024\times 200\times 24\times
2\approx 10$M trainable parameters. The prefix is initialized to the most
frequent words in the repository for which the model is customized.

###  3.6. Trainable Parameters during Fine-tuning

Table 1 provides an overview of the number of total and trainable parameters
involved in each customization process, in the case of a BART Transformer
model with 406M parameters. Custom fine-tuning allows to train 100% of the
406M available parameters in the model. During L-EO finetuning, instead, only
13% (53M) parameters are trained. The L-LDB finetuning reduces the number of
trainable parameters to 4.2% (17M). Finally, Prefix tuning has the lowest
number of trainable parameters, only 2.4% (10M) of the total, but these are
additional parameters added to the model (total 416M).

##  4\. Experimental Design

The goal of our experimental design is to investigate whether custom models
outperform the baseline model, leading to performance improvements in terms of
intrinsic metrics (RQ1), as well as extrinsic task-specific metrics (RQ2).
Next, we analyze and compare the different customization approaches in terms
of training and compute costs (RQ3) as well as model size and required storage
for deployment.

In our case study, we chose Unit Test Case generation as our code generation
task $t$, and AthenaTest by Tufano et al. (2021) as our baseline model $m$,
which is a BART transformer model pre-trained on source code and English, and
fine-tuned on Java unit test generation. The task is modeled as a translation
task, where the input is a focal method (i.e., method under test), and the
output is a test case which tests the focal method’s correctness. We randomly
sample 20 projects from the test set, each of those representing the dataset
$p$ on which a custom model is fine-tuned. Specifically, for each project $p$,
we start from the baseline model $m$ and fine-tune four different custom
models according to the four proposed fine-tuning strategies. For each project
and fine-tuning strategy (e.g., L-EO), we fine-tune and evaluate the models
using 4-fold cross-validation. The models are trained until the best
validation loss is reached, independently for every fold, every repository,
and every customization approach. In total, we fine-tune and evaluate
$20(projects)\times 4(approaches)\times 4(folds)=320$ models.

Table 1. Comparing the number of trainable parameters in each fine-tuning method. Customization Process | Parameters
---|---
Total | Trained
Custom | 406M | 406M (100%)
L-EO | 406M | 53M (13%)
L-LDB | 406M | 17M (4.2%)
Prefix | 416M | 10M (2.4%)

###  4.1. Dataset

Table 2 reports information about the 20 GitHub repositories sampled from the
test set, which will be used to customize our models. The table shows (i) the
Project ID, which will be used in the paper to reference a specific project;
(ii) the project name; (iii) the project size in terms of disk usage; (iv) the
popularity of the project in terms of number of stars obtained on GitHub; (v)
and the dataset size, which corresponds to the number of data points for the
unit test generation task (i.e., pair of focal method and test case). The list
of projects represent a diverse set of repositories with different size,
domain, and popularity. They span from small personal projects (e.g.,
Tutorials with 6 stars), to open source projects developed by large
organizations such as Apache and Google.

Table 2. Dataset - Projects used for customization Project ID | Name | Project Size (MB) | Stars | Dataset Size
---|---|---|---|---
26644682 | Talend Data Prep | 68.8 | 56 | 651
40735368 | GeoTools | 62.4 | 8 | 653
107330274 | Titus Control Plane | 36.0 | 302 | 660
52972024 | Smart Actors | 57.8 | 22 | 704
9714608 | Arakhnê Foundation Classes | 17.9 | 13 | 753
60701247 | Android Plugin for IntelliJ IDEA | 1026.7 | 716 | 754
14550159 | EverRest | 5.3 | 24 | 761
9278888 | Brave | 18.8 | 2084 | 787
66940520 | DHIS 2 | 118.1 | 211 | 862
33645537 | Tutorials | 34.4 | 6 | 878
62253355 | Mobi | 62.6 | 35 | 986
155883728 | OakPAL | 15.0 | 9 | 1005
4710920 | Apache Dubbo | 36.1 | 36231 | 1058
29603649 | Wilma | 6.7 | 40 | 1074
42949039 | Herd | 227.2 | 127 | 1249
1381673 | Drools | 176.7 | 3908 | 1394
1244027 | ModeShape | 131.1 | 212 | 1550
73948366 | AthenZ | 38.8 | 639 | 1920
660443 | Chemistry Development Kit (CDK) | 214.8 | 305 | 2591
87849739 | Eclipse Ditto™ Project | 52.5 | 311 | 2842

###  4.2. $\mathrm{RQ}_{1}$: Intrinsic Evaluation Metrics

RQ1: Do custom models obtain better performances on intrinsic metrics, such as
BLEU and perplexity, w.r.t. the baseline? To begin, we investigate how the
different model customization approaches described in Sec. 3 score on
intrinsic metrics such as BLEU and perplexity. All approaches entail fine-
tuning the baseline model to the dataset of a specific project, with the
choice of parameters being tuned depending on the approach taken. The four
variants are trained independently until the best validation loss is achieved.
We report the BLEU4 score and the mean perplexity per token on the test fold,
for all the 20 projects. Next, we perform statistical tests to investigate
whether the observed differences between the baseline and custom models are
significant, as well as differences among the customization approaches.
Specifically, we rely on the Kruskal-Wallis test, a non-parametric statistical
test.

###  4.3. RQ2: Task-specific Performances

RQ2: Do custom models improve on performance metrics specific to unit test
generation? We want to investigate how the different customization approaches
compare with respect to the downstream task of generating unit tests. Beyond
BLEU score and perplexity, we would like to see if custom models can produce
the correct target code, how closely their unit tests mimic the repository
style, or even if they can perfectly match the desired output.

  * •

Perfect Matches: We compare the model’s output string with the target
developer-written unit test. If the two strings are identical, this is
considered a perfect match. We do not take spacing and indentation into
account as we are using a Java dataset (where indentation is not required). We
report the proportion of perfect matches among the top 5 model predictions.

  * •

Abstracted Code Matches: We pass the model output and target output through
the src2abs tool (Tufano, 2018), to obtain an abstracted version, masking
variable names, method names, etc. We also do not distinguish between
different objects of the same type.

  * •

Coding Style: For each project’s custom model, we would like to determine how
closely the model learns the developer’s personal programming style and
preferences. To this end, we extract the collection of all identifiers (i.e.,
variables and functions’ names) from the unit tests written by the developer
as well as those generated by the models. We then pass these text outputs
through a tf-idf vectorizer and compute the cosine similarity between them.
This allows us to compare the developer’s and the models’ word usage. We
examine the similarity between the developer’s unit tests and the baseline and
custom models generated tests. This scores the vocabulary similarity of the
unit tests with the model generated code.

(a) Exact (solid) and abstracted (dotted) match rate

(b) Coding style as identifier similarity

Figure 4. Task-specific metrics (a) custom models outperform the baseline in
terms of perfect matches (solid line) and abstract matches (dotted line); (b)
custom models generate code that uses identifiers (i.e., variable and function
names) that are more similar to the project codebase.

###  4.4. RQ3: Training Cost Comparison

RQ3: Given the same amount of compute, which custom models achieve the biggest
performance improvement? Since our four training regimes tune a different
number of parameters, simply comparing the training time or number of
optimization steps to reach the best validation loss may not be appropriate.
For a model with $N$ parameters, we approximate the computation cost of a
forward pass to be $C\approx 2N$ floating point operations per training token,
with an additional correction for embedding layers. The backward pass takes
roughly twice the amount of compute, but it is unnecessary for layers that are
frozen. For additional details, we refer to Table 1 in (Kaplan et al., 2020).
We report the resulting compute in petaFLOPS-seconds.

##  5\. Results

###  5.1. RQ1: Intrinsic Evaluation Metrics

Table 3 presents the results of custom models in terms of the intrinsic
metrics: BLEU and perplexity. Specifically, for each project, we report the
average BLEU and perplexity over the four folds, achieved on the test set by
the different customization strategy, as well as the baseline model. We
observe notable improvements in both metrics for every project w.r.t. the
baseline, with BLEU going from 16.1 achieved by the baseline model to 36-38 by
custom models.

The statistical tests reported in Table 4 confirm that the improvement
observed by the four customization techniques are statistically significant
w.r.t. the baseline ($p<1$e-7). However, we do not observe statistical
significance in the differences among the customization strategies, meaning
that, in terms of intrinsic metrics performances, the differences are within
margin of error.

Table 3. The BLEU score and perplexity for the customization methods evaluated on the 20 projects in our test set. Project | BLEU4 | Perplexity
---|---|---
Base | Cust. | L-EO | L-LDB | Prefix | Base | Cust. | L-EO | L-LDB | Prefix
26644682 | 14.1 | 32.9 | 31.6 | 31.9 | 34.0 | 1.275 | 1.212 | 1.208 | 1.238 | 1.197
40735368 | 18.5 | 30.7 | 29.0 | 29.4 | 29.1 | 1.276 | 1.186 | 1.197 | 1.186 | 1.194
107330274 | 14.8 | 38.0 | 35.0 | 35.9 | 35.7 | 1.273 | 1.160 | 1.168 | 1.164 | 1.175
52972024 | 10.2 | 31.8 | 33.2 | 32.2 | 30.1 | 1.271 | 1.142 | 1.146 | 1.135 | 1.135
9714608 | 14.7 | 41.0 | 38.1 | 40.4 | 40.2 | 1.263 | 1.155 | 1.145 | 1.150 | 1.138
60701247 | 10.8 | 28.9 | 24.4 | 25.9 | 26.6 | 1.267 | 1.187 | 1.190 | 1.172 | 1.176
14550159 | 20.0 | 49.5 | 47.2 | 46.6 | 46.4 | 1.245 | 1.121 | 1.122 | 1.116 | 1.124
9278888 | 17.3 | 46.8 | 44.5 | 47.2 | 47.8 | 1.272 | 1.137 | 1.152 | 1.138 | 1.140
66940520 | 17.4 | 37.9 | 33.9 | 35.5 | 37.7 | 1.264 | 1.154 | 1.163 | 1.154 | 1.150
33645537 | 17.0 | 30.4 | 31.2 | 32.0 | 31.0 | 1.264 | 1.231 | 1.200 | 1.192 | 1.211
62253355 | 14.7 | 48.0 | 45.7 | 47.3 | 48.0 | 1.292 | 1.113 | 1.114 | 1.114 | 1.116
155883728 | 13.7 | 41.3 | 37.5 | 39.3 | 39.5 | 1.238 | 1.132 | 1.148 | 1.146 | 1.140
4710920 | 28.2 | 39.1 | 38.1 | 38.8 | 38.6 | 1.218 | 1.161 | 1.162 | 1.167 | 1.160
29603649 | 19.1 | 58.4 | 54.9 | 56.6 | 56.8 | 1.266 | 1.096 | 1.110 | 1.099 | 1.098
42949039 | 17.0 | 38.2 | 37.7 | 37.5 | 37.3 | 1.238 | 1.154 | 1.152 | 1.154 | 1.148
1381673 | 14.3 | 33.3 | 29.3 | 30.9 | 30.8 | 1.261 | 1.133 | 1.152 | 1.138 | 1.138
1244027 | 19.6 | 30.1 | 29.7 | 30.0 | 30.0 | 1.244 | 1.142 | 1.160 | 1.142 | 1.150
73948366 | 12.0 | 33.1 | 31.8 | 34.0 | 33.6 | 1.267 | 1.161 | 1.157 | 1.159 | 1.164
660443 | 15.0 | 34.0 | 37.2 | 36.5 | 34.3 | 1.281 | 1.180 | 1.170 | 1.169 | 1.177
87849739 | 13.4 | 45.1 | 47.0 | 48.9 | 46.8 | 1.259 | 1.138 | 1.136 | 1.124 | 1.144
Average | 16.1 | 38.4 | 36.9 | 37.8 | 37.7 | 1.262 | 1.153 | 1.158 | 1.153 | 1.154
Table 4. Kruskal-Wallis Test p-values testing the significance of the pairwise hypothesis that one customization method is superior than another. Custom strategies are significantly better than baseline. | BLEU4 | Perplexity
---|---|---
Base | Cust. | L-EO | L-LDB | Prefix | Base | Cust. | L-EO | L-LDB | Prefix
Base | - | 3e-08 | 3e-08 | 3e-08 | 3e-08 | - | 3e-08 | 3e-08 | 3e-08 | 3e-08
Cust. | 3e-08 | - | 0.4 | 0.7 | 0.7 | 3e-08 | - | 0.5 | 0.9 | 0.9
EO | 3e-08 | 0.4 | - | 0.5 | 0.7 | 3e-08 | 0.5 | - | 0.5 | 0.5
LDB | 3e-08 | 0.7 | 0.5 | - | 0.9 | 3e-08 | 0.9 | 0.5 | - | 0.8
Prefix | 3e-08 | 0.7 | 0.7 | 0.9 | - | 3e-08 | 0.9 | 0.5 | 0.8 | -

###  5.2. RQ2: Task-specific Performances

The results in terms of task-specific performances are presented in Figure 4.
The plot 4(a) shows the top-k accuracy for perfect matches (solid line) and
abstracted matches (dotted line), aggregated over the 20 projects. The
baseline model outputs the same code structure (abstracted) in roughly 3% of
all cases, and virtually never produces the exact target output (<1%).
Moreover, its performance does not improve as we consider more predictions.
All customization processes show significant improvement compared to the
baseline. Specifically, these improvements are observed for every single
project (full results will be available on our online appendix). Customized
models produce the correct code structure as their top prediction in
$\sim$13-14% of instances, and a perfect match in $\sim$4-6% of cases. They
also tend to improve as we consider their top 5 predictions. Between the
different customization processes, Custom consistently performs the best,
closely followed by Prefix and L-LDB. When considering abstracted code
matches, these three approaches are nearly identical. L-EO, however, performs
slightly worse than the others.

Plot 4(b) shows the distribution of tf-idf cosine similarity computed between
identifiers used in the developers’ written tests and the models’ generated
outputs. We observe that the distribution for custom models is skewed towards
the higher values of cosine similarity. This result demonstrates that custom
models tend to use variable and function names that are more similar to what
developers used in their own tests.

Figure 5. Validation Loss vs Compute (PF-seconds) - Light lines represent the
validation loss curve for each individual project and fold, while the bold
line represents the average for each custom strategy. Custom is the most
efficient, lightweight approaches require slightly more compute to reach a
comparable validation loss, while prefix is the least efficient, suffering
from poor initialization.

###  5.3. RQ3: Training Cost Comparison

For each customization process, we plot validation loss as a function of
compute, as defined in section 4.4. The results are presented in Figure 5,
where the light lines represent the validation loss curve for each individual
project and fold, while the bold line represents the average for each custom
strategy. First note that Custom achieves very large gains during the first
epoch, as evidenced by the fact that its validation loss starts much lower
than L-EO and L-LDB. Custom also outperforms other customization processes
when given a limited amount of compute. However, we observe that beyond a
certain amount of compute, Custom and L-LDB tend to achieve similar
performances. In contrast, L-EO starts at the same validation loss as L-LDB
but converges much slower to the best loss, requiring 2-3 times as much
compute.

Since the prefix parameters suffer from poor initialization, Prefix is the
most expensive customization process. To overcome this problem, it is possible
to first train the prefix on a large generic dataset. Then, given proper
hyperparameter tuning, it is possible to substantially cut down compute cost
for customizing the prefix.

##  6\. Discussion & Lessons Learned

The four customization strategies considered in this work are effective in
improving a code generation model’s performances on a given software project.
Specifically, all custom models significantly outperform the baseline in terms
of intrinsic metrics (i.e., BLEU and perplexity) as well as task-specific
metrics (i.e., abstract and raw matches). While the differences among the
customization approaches are not significant (no clear winner), each strategy
offers specific advantages in different circumstances and deployment
scenarios.

Custom fine-tuning achieves the overall best performances and the
customization process is relatively fast and efficient. This is somewhat
expected, since this customization strategy allows all the model’s parameters
to be tuned on the specific project. This characteristic also leads to the
major disadvantage of this approach: each custom model is an entire copy of
the original model. Storage and inference costs could become prohibitive when
serving many users with personalized custom models.

Lightweight fine-tuning achieves good results while training fewer parameters.
This allows to serve potentially many users with custom models which can be
stored and loaded efficiently. Specifically, L-LDB trains fewer parameters
than L-EO, however the latter could allow to deploy the embedding and output
layers on the user side, with a privacy-preserving focus.

Prefix fine-tuning trains the lowest number of parameters (only 2.4% for a
BART model), while improving over the baseline. However, it increases the
total number of parameters of the model (prefixes are additional virtual
tokens) and requires more compute time to achieve good performances, mostly
due to the prefix initialization problem. On the bright side, this strategy
allows to batch together requests from different users (with different
prefixes), which can be processed by a single model, generating personalized
outputs.

##  7\. Threats to Validity

The major threats to our study relate to external validity, which concerns the
generalization of our findings. Our study is performed on a specific coding
task (i.e., test case generation) and for a single programming language (i.e.,
Java). The findings could not generalize to all the coding tasks and different
programming languages. However, our extensive empirical analysis investigating
different personalization strategies in terms of several performance metrics,
can provide guidelines for applying these techniques on different coding
tasks, languages, and datasets. It is important to note that, while each
coding task has its peculiarities, test case generation task involves the
generation of complete methods, variable assignments, method calls, and
different types of statements, thus could serve as a good generic coding task.
Java language is among the most popular and similar to other programming
languages such as C# and Ruby. As part of our future works we intend to apply
these personalization techniques to different coding tasks and languages.

##  8\. Related Work

This work is related to two areas of the existing literature: neural source
code generation and model personalization. Neural code generation has
generated an intense recent interest in NLP, using Transformer models (Vaswani
et al., 2017b) in particular for code completion (Svyatkovskiy et al., 2020,
2019; Clement et al., 2020; Raychev et al., 2014; Bruch et al., 2009;
Brockschmidt et al., 2018), code synthesis from examples (Chen et al., 2018),
natural language to code (Clement et al., 2020; Chen et al., 2018; Austin et
al., 2021), code feature summarization (Liu et al., 2021; Moreno et al., 2013;
Scalabrino et al., 2017; Wan et al., 2018; Alon et al., 2018; Moreno et al.,
2014), code search (Husain et al., 2019; Feng et al., 2020), unit test
generation (Tufano et al., 2021) and even bug fixing (Drain et al., 2021) and
detection (Zhai et al., 2020). This paper naturally is an extension and
evaluation of personalized unit test generations as studied by Tufano et al.
(2021), and an important contribution to the understanding optimization in a
deployment scenario.

Much of the previous literature on personalized models focuses on client-side
training to keep data on device (Shor et al., 2019; Popov et al., 2018), and
most work is in the domain of search query completion (Jaech and Ostendorf,
2018), natural language completion (Popov et al., 2018), or even automated
speech recognition (Shor et al., 2019). Naturally this work extends the domain
of evaluation beyond natural language tasks and into the software engineering
domain. This paper does not evaluate methods for client side training with
restricted resources, however, as the most powerful large language models
which enable state of the art code synthesis have 10-100 million parameters.
At the time of writing such large models cannot be executed in a reasonable
amount of time on most consumer laptops. We leave to future work extending
these studies to models which have been pruned, quantized, distilled, and
optimized to be ran in limited resource environments.

##  9\. Conclusion

In this paper we explored different ways to customize a code generation model
for a given codebase, with the goal of improving its performances on a target
project. We described and analyzed four customization strategies and applied
them on 20 different software projects for the task of generating unit test
cases.

Specifically, we considered the following strategies: (i) custom fine-tuning,
which allows all the model parameters to be tuned on the target project; (ii)
L-EO fine-tuning, a lightweight training which freezes most of the model’s
parameters, tuning only embedding and output layers; (iii) L-LDB fine-tuning,
a lightweight training which only tunes the last decoder block; (iv) prefix
tuning, which keeps language model parameters frozen, but optimizes a small
project-specific vector (prefix).

In our extensive empirical evaluation we found that all the customization
strategies lead to significant model’s improvements on a target project, in
terms of both intrinsic and task-specific metrics, with the custom models
adapting to the coding style of the target project. While there is no clear
winner among the customization strategies, each approach can provide specific
benefits in particular deployment scenarios.

## References

  * (1)
  * Alon et al. (2018) Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2018\.  code2seq: Generating sequences from structured representations of code.  _arXiv preprint arXiv:1808.01400_ (2018).
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021\.  Program Synthesis with Large Language Models.  _arXiv preprint arXiv:2108.07732_ (2021).
  * Brockschmidt et al. (2018) Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. 2018.  Generative Code Modeling with Graphs. In _International Conference on Learning Representations_.  https://doi.org/10.48550/arXiv.1805.08490
  * Bruch et al. (2009) Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009\.  Learning from examples to improve code completion systems. In _Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering_. 213–222.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chen et al. (2018) Xinyun Chen, Chang Liu, and Dawn Song. 2018.  Towards Synthesizing Complex Programs From Input-Output Examples. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=Skp1ESxRZ
  * Clement et al. (2020) Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020\.  PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9052–9065.
  * Drain et al. (2021) Dawn Drain, Chen Wu, Alexey Svyatkovskiy, and Neel Sundaresan. 2021.  Generating Bug-Fixes Using Pretrained Transformers.  _arXiv preprint arXiv:2104.07896_ (2021).
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020\.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 1536–1547.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.  _arXiv preprint arXiv:1909.09436_ (2019).
  * Jaech and Ostendorf (2018) Aaron Jaech and Mari Ostendorf. 2018.  Personalized Language Model for Query Auto-Completion. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. 700–705.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling Laws for Neural Language Models.  _CoRR_ abs/2001.08361 (2020).  arXiv:2001.08361 https://arxiv.org/abs/2001.08361
  * Kawaguchi et al. (2006) Shinji Kawaguchi, Pankaj K Garg, Makoto Matsushita, and Katsuro Inoue. 2006.  Mudablue: An automatic categorization system for open source repositories.  _Journal of Systems and Software_ 79, 7 (2006), 939–953.
  * LeClair et al. (2018) Alexander LeClair, Zachary Eberhart, and Collin McMillan. 2018\.  Adapting neural text classification for improved software categorization. In _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE.  https://doi.org/10.1109/ICSME.2018.00056
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, 4582–4597.  https://doi.org/10.18653/v1/2021.acl-long.353
  * Liu et al. (2021) Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2021.  Retrieval-Augmented Generation for Code Summarization via Hybrid GNN. In _International Conference on Learning Representations (ICLR)_.  https://doi.org/10.48550/arXiv.2006.05405
  * Lu et al. (2021) Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. 2021\.  Pretrained Transformers as Universal Computation Engines.  _CoRR_ abs/2103.05247 (2021).  arXiv:2103.05247 https://arxiv.org/abs/2103.05247
  * Moreno et al. (2013) Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K Vijay-Shanker. 2013.  Automatic generation of natural language summaries for java classes. In _2013 21st International Conference on Program Comprehension (ICPC)_. IEEE, 23–32.
  * Moreno et al. (2014) Laura Moreno, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrian Marcus, and Gerardo Canfora. 2014.  Automatic generation of release notes. In _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_. 484–495.
  * Popov et al. (2018) Vadim Popov, Mikhail Kudinov, Irina Piontkovskaya, Petr Vytovtov, and Alex Nevidomsky. 2018\.  Distributed fine-tuning of language models on private data. In _International Conference on Learning Representations_.
  * Raychev et al. (2014) Veselin Raychev, Martin Vechev, and Eran Yahav. 2014\.  Code completion with statistical language models. In _Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation_. 419–428.
  * Řehůřek and Sojka (2010) Radim Řehůřek and Petr Sojka. 2010.  Software Framework for Topic Modelling with Large Corpora. In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_. ELRA, Valletta, Malta, 45–50.  http://is.muni.cz/publication/884893/en.
  * Scalabrino et al. (2017) Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2017\.  Automatically assessing code understandability: How far are we?. In _2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 417–427.
  * Shor et al. (2019) Joel Shor, Dotan Emanuel, Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, et al. 2019\.  Personalizing ASR for dysarthric and accented speech with limited data.  _arXiv preprint arXiv:1907.13511_ (2019).  https://doi.org/10.21437/Interspeech.2019-1427
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  IntelliCode compose: code generation using transformer. In _ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020_ , Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 1433–1443.  https://doi.org/10.1145/3368089.3417058
  * Svyatkovskiy et al. (2019) Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019.  Pythia: AI-assisted Code Completion System. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2727–2735.
  * Tian et al. (2009) Kai Tian, Meghan Revelle, and Denys Poshyvanyk. 2009.  Using latent dirichlet allocation for automatic categorization of software. In _2009 6th IEEE International Working Conference on Mining Software Repositories_. IEEE, 163–166.  https://doi.org/10.1109/MSR.2009.5069496
  * Tufano (2018) Michele Tufano. 2018\.  src2abs.  https://github.com/micheletufano/src2abs.
  * Tufano et al. (2021) Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021\.  Unit Test Case Generation with Transformers and Focal Context.  _arXiv preprint arXiv:2009.05617_ (2021).  arXiv:2009.05617 [cs.SE]
  * Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017a.  Attention Is All You Need.  _CoRR_ abs/1706.03762 (2017).  arXiv:1706.03762 http://arxiv.org/abs/1706.03762
  * Vaswani et al. (2017b) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017b.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wan et al. (2018) Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S Yu. 2018.  Improving automatic source code summarization via deep reinforcement learning. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_. 397–407.
  * Zhai et al. (2020) Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020\.  CPC: Automatically classifying and propagating natural language comments via program analysis. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 1359–1371.