# Individual and Collective Autonomous Development

Marco Lippi, Stefano Mariani,
Matteo Martinelli, and Franco Zambonelli

###### Abstract

The increasing complexity and unpredictability of many ICT scenarios let us
envision that future systems will have to dynamically learn how to act and
adapt to face evolving situations with little or no a priori knowledge, both
at the level of individual components and at the collective level. In other
words, such systems should become able to autonomously develop models of
themselves and of their environment. Autonomous development includes: learning
models of own capabilities; learning how to act purposefully towards the
achievement of specific goals; and learning how to act collectively, i.e.,
accounting for the presence of others. In this paper, we introduce the vision
of autonomous development in ICT systems, by framing its key concepts and by
illustrating suitable application domains. Then, we overview the many research
areas that are contributing or can potentially contribute to the realization
of the vision, and identify some key research challenges.

###### Index Terms:

Autonomous development, sense of agency, learning, self-adaptation, self-
organization.

##  I Introduction

In their early months, human infants start experiencing with their own body,
moving hands, touching objects, and interacting with people around. Such
activities are part of an overall process of autonomous development (a.k.a.
self-development), which lets them gradually develop cognitive and behavioral
capabilities [1]. These skills include the capability to recognize situations
around, the sense of self, the sense of agency (i.e., understanding the effect
of own actions in an environment), the capability to act purposefully towards
a goal, and some primitive social capabilities (i.e., knowing how to act in
the presence of others).

Moving from humans to machines, the possibility of building ICT systems
capable to autonomously develop their own mental and social models and to act
purposefully in an environment, is increasingly recognized as a key challenge
in many areas of artificial intelligence (AI), such as robotics [2],
intelligent IoT [3], autonomous vehicles management [4].

Indeed, for small-scale and static scenarios, and for simple goal-oriented
tasks, it is possible to “hardwire” a model of the environment within a
system, alongside some pre-designed plans of action. However, for larger and
dynamic scenarios, and for complex tasks, individual components of ICT systems
should be able to autonomously (i.e., without human supervision and with
little or no innate knowledge): _(i)_ build environmental models and
continuously update them as situations evolve; _(ii)_ develop the capability
of recognizing and modeling the effect of their own actions on the context
(which variables of the environment can or cannot be directly affected by
which actuators, which variables and actuators relate to each other); _(iii)_
learn to achieve goals on this basis and depending on the current situation;
_(iv)_ learn how to organize and coordinate actions among multiple distributed
components whenever necessary.

The ambitious idea of building systems capable of autonomous development is
not new, and its opportunity has been already advocated since several years
[5], sometime under the notion of “self-aware” computing systems [6]. However,
the topic is now even more timely. Many recent research results in areas such
as machine learning, causal analysis, multi-agent systems, and collective
behaviors, have started shedding light on the various mechanisms that have to
be involved in the process of autonomous development, hinting at the fact that
the vision (at least in specific application areas) is close to become
reality. Furthermore, unfolding the key concepts and mechanisms underlying
autonomous development can also somewhat contribute to understand the many
mental mechanisms behind artificial general intelligence [7].

Against this background, the contribution of this paper is to frame the key
concepts of autonomous development in ICT systems and to identify challenges
and promising research directions. Specifically, Section II introduces a
general conceptual framework for the (continuous and adaptive) process of
autonomous development, both at the individual and at the collective level;
Section III sketches key application scenarios; Section IV analyzes the most
promising approaches in the area of machine learning, causal analysis, multi-
agent systems, and collective adaptive systems that can contribute with
fundamental building blocks towards realizing the vision of autonomous
development, each _per se_ challenging; Section V identifies additional
horizontal challenges to be attacked.

##  II Conceptual Framework

Autonomous development, beside being the process that infants carry out during
the early stages of their life [1], also involves any “agent” whenever it is
incarnated in a new body and immersed in a new environment.

As an example to quickly and intuitively introduce our general framework
(Figure 1) let us consider what we do whenever we start playing a new video-
game. At first, we observe the game environment on the screen and the commands
we have available on the joystick; that is, we get acknowledged with our
_embodiment and perception_ on the video-game. We spend a few seconds trying
the commands to assess their effects in the game environment; that is, we try
to acquire a _sense of agency_. Then, we understand what is the goal of the
game and how we can use the commands to achieve it; that is, we start acting
in a _goal-oriented_ way.

Typically, we recognize in the video-game the presence of other “agents”,
virtual characters that are not under our control; that is, we distinguish
between _self and non-self_. The acquisition of such a skill implies that we
acknowledge that we should tune our actions also in dependence of the actions
of these other agents (_strategic thinking_). All this process is typically
repeated in a cyclic way (i.e., when reaching a new level in the game) to
adapt to new environments, new situations, new tools available to play with,
new goals, and new enemies appearing.

In the case of multiplayer games, beside recognizing the presence of players
different from ourselves, and recognizing the need to act also accounting for
them, we should understand: whether we have _communication_ tools available;
how to use these tools to affect and influence the actions of others, i.e., to
_coordinate_ with them, so that eventually _institutional_ ways to act
together towards a goal can be established. Again, this process may be
cyclically repeated as the game advances.

Truly intelligent and adaptive ICT systems should undergo a similar process
and autonomously develop through similar phases.

Figure 1: The conceptual framework of autonomous development.

###  II-A The Individual Level

Let us firstly consider a single agent $X$ (purely software or physically
embodied) immersed in a (virtual or physical) environment. The agent can
observe a set of environmental variables
$\mathcal{V}=\\{v_{1},v_{2},\ldots,v_{m}\\}$. As the agent is part of the
environment, internal variables of the agent itself (i.e., its current status
and configuration) are included in the set. In addition, the agent has a set
of actions that it can choose $\mathcal{A}=\\{a_{0},\ldots,a_{n-1},null\\}$,
including the $null$ action.

_Embodiment and Perception_. In this very early phase, the agent should
autonomously recognize the existence of $\mathcal{A}$ and $\mathcal{V}$, that
is, it should get acknowledged to its actuation and sensorial capabilities.
Without resorting to complex AI techniques, methods from the reflective and
self-adaptive programming systems can effectively apply in this phase [8] to
let the agent dynamically self-inspect its capabilities and start analyzing
the observed variables. Still in this phase, the agent can also start
acquiring some understanding of the relations between the observed variables
over time, as well as some simple prediction capabilities.

_Sense of Agency_. In this exploratory phase, the agent starts trying to
understand what are the effects of $\mathcal{A}$ on $\mathcal{V}$, by trying
to apply actions (even without any goal in mind) to see their effects. That
is, it will eventually recognize that, given the current state of
$\mathcal{V}_{current}$, the application of an action $a_{i}$ (or of a
sequence of actions) will eventually lead (with some probability) to state
$\mathcal{V}_{next}$. This mechanism enables the construction of the basic
sense of agency [1], and of the sense of causality.

_Goal-orientedness_. In this exploitation phase, the agent starts applying
$\mathcal{A}$ with goals in mind. That is, given the current state
$\mathcal{V}_{current}$ and a desired future state $\mathcal{V}_{g}$ (the
goal, aka the desired “state of the affairs”), the agents resorts to the
acquired sense of agency by applying the action that can possibly lead to
$\mathcal{V}_{g}$. This also involves achieving the capability of planning the
required sequence of actions to achieve a specific goal.

_Self and Non-Self_. As soon as an individual agent starts exploring its own
actions $\mathcal{A}$, and recognizes that such actions have effect on the
environment, it also understands that there are effects that are not under its
own control. That is, there are “non-self” entities acting in the environment,
too. By learning how to apply $\mathcal{A}$, the agent also learns the limits
of such actions because of the non-self entities affecting some variable
$v_{i}$.

_Strategic Thinking_. The agent has built a model of the world, that is, of
how $\mathcal{A}$ affects $\mathcal{V}$, and it starts including the mental
models of others (non-self) [9] while acting, as well as while designing
strategies. That is, it can recognize that there are goals that it can
possibly (or hopefully) attain only by accounting for the actions of others.

As in the video-game example, autonomous development is not to be conceived as
a “once-and-for-all” process. Rather, it is a continuous, never-ending
process: environmental conditions can change, new sensors may become available
to enable more detailed observations, and new actions become feasible (or vice
versa, some sensors and actions may no longer be available). This requires the
agents to re-tune their learned sense of agency, and re-think how to achieve
goals in isolation and in the presence of non-self entities.

###  II-B The Collective Level

In the presence of multiple agent acting in the same environment, an agent
recognizes that there are goals that cannot be achieved in isolation or by
simply applying strategic thinking. Thus as part of their autonomous
development, they should collectively develop some forms of “autonomous social
engagement”.

Formally, this corresponds to considering a set of $K$ agents
$X_{0},\ldots,X_{K-1}$, where _(i)_ each agent can choose the actions to
perform from its own set (possibly disjoint or only partially overlapping from
those of the other agents); _(ii)_ not necessarily all the agents can observe
the whole set of environmental variables, but more likely each agent $X_{j}$
has the capability to perceive and/or control a subset of them. Thus, for
specific goals $\mathcal{V}_{g}$ to be achieved, there is the need of properly
combining and sequencing actions by different agents, e.g. $X_{i}$ executes
$a^{i}_{w}$ whereas $X_{j}$ executes $a^{j}_{z}$, and so on.

_Communication_. To overcome the limitations of strategic thinking, agents
should be provided with a specific set of _communication actions_ , i.e.,
actions that are devoted to influence the actions of other agents. These could
take the form of explicit communication acts, i.e., messages, that the agent
should learn how to receive and send as an additional – social – form of
perception and action. However, they could also take the form of more implicit
actions aimed at affecting the behavior of others, i.e., leaving signs in the
environment (stigmergy) or by adopting peculiar behaviors aimed at be noticed
by others (behavioral implicit communication) [10]. All these scenarios can be
formalized by augmenting the $\mathcal{A}$ set to include communication, and
possibly $\mathcal{V}$, to include observable signs in the environment.

_Coordination_. By exploring its available communication actions, agents start
understanding how such acts can be used to get access and to affect some of
the variables of the environment, and in particular those that are not
observable and controllable by themselves. For instance, they can learn how to
use communication acts to get access to the value of some non-observable
variables $v_{i}$ or to direct other agents in executing the actions that can
affect its values as required for a goal to be achieved. In other words, such
explorations enable learning basic forms of coordination, which can be thought
of as a social form for the sense of agency.

_Institution_. Eventually, after exploring coordination protocols, the agents
can “institutionalise” their patterns of interaction towards collective
actions. That is, they will learn those acceptable social patterns of
coordination, and the set of social norms and social incentives, that enables
them to systematically achieve goals together [11]. Formally, this corresponds
to having agents in the collective recognize and adhere to a set of
constraints $\mathcal{C(A,V)}$ ruling the way (communication) actions
$\mathcal{A}$ can be performed in specific conditions $\mathcal{V}$, as well
as the commitments and expectations each communication action sets on the
agents participating in the protocol.

As for the case of the individual level, the dynamics of the environment or of
the agent population may require the above collective process to assume a
continuous cyclic nature.

We emphasize that communication, coordination, and institutions are not
strictly necessary to promote complex goal-oriented collective actions [12].
Nevertheless, whenever communication mechanisms are available, learning to
exploit them is a natural part of the autonomous development process, and can
facilitate the effectiveness of the social development.

##  III Application Scenarios

There are diverse application scenarios that can potentially take advantage of
systems capable of autonomous development, at the individual and/or collective
level.

###  III-A Robotics

Robotics is the area which first identified the profitability of building
systems capable to autonomously build “by experience” a model of their own
capabilities, and consequently learn to achieve specific goals.

In general, developers can define a model of their robots at design time, and
can easily wire such model directly in embedded software without necessarily
having the robot learn it in autonomy. However, autonomous learning may become
necessary when the robot gets damaged while in operation, thus (partially)
losing its original capabilities (e.g., a four-legged robot with one leg
damaged). In that case, the robot should autonomously learn a new model to
understand what it can do according to its residual operational capabilities
(e.g., how to walk with three legs only), and how it can re-learn to achieve
goals with them [13].

A different situation is that of modular robots [14], where the robot can re-
arrange its shape to serve different tasks. In this case, having the
developers foresee all the possible shapes that the robot can assume to serve
different tasks can be time-consuming. Also, it can prevent emergent (not
previously envisioned) shapes – functional to peculiar tasks – to be
autonomously identified by the robot itself. In this case, having the modular
robot try to assume a variety of forms and understand its action capabilities
for the different shapes could be very useful before deployment (other than
after deployment, to recover from injuries and from the loss of some of its
modules).

At the level of collective robotics (i.e., group of robots living together in
an environment and having to cooperate to achieve collective task) most
current approaches relies on coordination schemes defined at design time.
However, also in this case it has been argued that the autonomous evolution of
communication and coordination capabilities can be of fundamental importance
to acquire the capability of the collective to act in unknown and dynamically
changing scenarios [15].

###  III-B Smart factories

Similarly to a collective robotic system, a complex manufacturing system can
be seen as an aggregated group of components that act together in order to
achieve a production goal. Beside their basic scheme of functioning, defined
at design time, if one component of the manufacturing system breaks or has
some unexpected behavior, the manufacturing system should ideally adapt to the
new situation, so as to overcome the problem without undermining production.

Exploring in advance all possible contingencies and hard-wiring solutions to
them in the system is almost impossible. The system should rather learn
autonomously how to act upon changing conditions (whether of a temporary or
ultimate nature) to maintain its overall functioning. For example, the system
may explore the possibility of deviating the material flow from the broken
component to another one, to learn the effect of such actions, and its overall
impact on the production. In doing so, a local or global production re-
scheduling might be necessary, with the initiative autonomously coming from
the different components of the manufacturing system, without involving the
production planning office. Clearly, such capabilities of autonomous
adaptation can also apply at the level of individual components of the system,
whenever these do not impact other components.

The need for integrating adaptability and flexibility in manufacturing systems
is explicitly recognized as a key challenge in Industry 4.0 initiatives, and
some examples of agent-based production control systems – exhibiting some
limited forms of adaptivity – can be already found, e.g., in the automotive
industry [16]. However, these are still far from the adaptivity level that
could be reached by fully-fledged autonomous development approaches.

###  III-C Smart homes

Buildings and homes are increasingly being enriched with sensors and actuators
(i.e., IoT devices, in general), to facilitate our interactions with the
environment and, by monitoring our activities and habits, to increase our
safety and comfort. However, such systems are typically based on design-time
decisions w.r.t. deployment of devices, their interactions, and the types of
services to be provided. Learning capabilities are typically limited to
monitoring user activities and adapt the parameters of services (e.g., the
levels of light and heating) accordingly [17].

From the perspective of autonomous development, we envision that once IoT
devices are deployed in an environment, they should be activated to
autonomously explore their own individual and collective capabilities (i.e.,
the individual sense of agency and the impact of inter-device interactions) so
as to eventually learn how they can affect the home environment and how, and
apply such capabilities once users will start populating the environment.
Then, the overall smart home/building system will continue to dynamically and
continuously modify its functioning to adapt to the presence of different
users with different profiles, of users whose habits tend to evolve over time,
or simply to react to contingencies (e.g., modifications in the number and
type of available devices, or structural modifications to the environment).

We have conducted some simple preliminary experience to show the potential
feasibility of autonomous development in a simple two-rooms smart home test-
bed [3]. In particular, we have shown that IoT devices in a room, when left
free to explore the effects of their actions, can eventually build a sound
causal model of the room, and can use such model to actuate specific
environmental condition in that room (e.g., closing windows and turning off
lights to achieve darkness). Also, by merging the models built for the two
rooms, devices can learn to cooperatively actuate house-level environmental
conditions (e.g., when a curtained window connects the two adjacent rooms,
they have to agree on keeping the curtain closed so as to have light in one
room and darkness in the other). Exploring larger and more complex environment
and different learning techniques, and in the presence of users, will give us
better clues on the general applicability of the approach and its possible
limitations.

###  III-D Smart cities

Most of the considerations above for smart homes and buildings scenario can
be, in theory, transferred to the larger scenario of smart cities. That is, to
all those ICT and IoT systems that can get deployed to automate and regulate
the activities of modern cities (e.g., mobility, energy management, garbage
collection). Indeed, the need to deploy robust systems capable – with limited
design efforts and limited human intervention – to dynamically adapt their
behavior to continuously changing urban conditions is widely recognized as a
key challenge for harmonic and sustainable urban development [18].

The substantial difference between smart homes/buildings and smart cities is
that cities already exist and are already inhabited. Thus, you cannot think at
let a smart city system free to explore the effect of its actions and
interactions to eventually become capable to act in a goal-oriented way (which
you can instead do before a new building becomes inhabited).

However, one can think at exploiting a simulation-based approach to this
purpose. Given that accurate and reliable simulators exists to study different
urban aspects, one can think at having system components explore and learn in
a simulated environment towards full mental development, before being
eventually deployed in the real world [4]. Indeed, exploiting simulators to
support autonomous development, the same as children are given toys and
playgrounds to explore and develop their capabilities, can be a key for any
application area.

##  IV Research Approaches

Figure 2: The galaxy of autonomous development.

The idea of autonomous development, at both the individual and collective
level, has been widely investigated in areas such as cognitive psychology,
neuroscience, philosophy, and ethics [5]. We hereby focus on the computational
perspective, and in particular on the galaxy of different approaches that can
contribute to unfold the mechanisms involved in autonomous development vision
and to eventually realize the vision (Figure 2). Although most of these
approaches can play or are already playing a fundamental role, they still have
to attack several challenging problems to become practical tools for future
systems engineering.

We do not focus here on the basic levels of individual autonomous development,
i.e., perception and embodiment, in that tools already exist to give agents
sophisticated sensing abilities (e.g., convolutional neural networks to
recognize objects, scenes, and activities) and the capability of controlling
their own actuators purposefully.

###  IV-A Goal-oriented Learning

The broad area of reinforcement learning shares with our vision the objective
of training machines to act in a goal-oriented way in a specific context.
However, despite the amazing recent results in the area, in particular with
deep Q-learning [19], most current approaches do not aim at building systems
with a sense of agency and capable of developing an interpretable world model,
but rather at achieving goals based on explicit, domain-based rewards, that
are named extrinsic. This makes most approaches highly ineffective in scaling
up to learning tasks in complex contexts, or across domains, or despite the
ever-changing dynamics of the environment.

Curriculum-based approaches to machine learning go somewhat in the direction
of gradually developing the capability to act in complex scenarios [20]. The
agent is first trained on simple tasks, and the gained knowledge is
accumulated and exploited in increasingly complex scenarios, where further
skills can thus be effectively learnt. Yet again, most of these approaches do
not focus on the development of a world model and of an explicit sense of
agency.

Reinforcement learning approaches based on _intrinsic_ rewards [21], instead,
more closely exploit the idea of exploring the world to develop a sense of
agency. In fact, while extrinsic rewards are typically designed by a “teacher”
(e.g., the score in a videogame) intrinsic rewards are developed by the agent
itself to satisfy its curiosity (i.e., when it discovers how to achieve
specific tasks). For example, in [22] intrinsic rewards are computed as the
error in forecasting the consequence of the action performed by the agent
given its current state.

Recent approaches based on the theory of affordances [23] propose to have
agents gradually learn the effects of their actions. By having them act in
constrained environments where only a limited set of actions apply, they
eventually develop an explicit sense of agency, i.e., a model of how their
actions affect the environment.

Al these approaches face the key challenge of building general conceptual and
practical tools to: (i) learn to effectively act in an environment by
exploiting the power of model-free sub-symbolic (deep learning) approaches;
and, at the same time, (ii) learn incremental and reusable, possibly causal
models of the world. The latter being increasingly recognized as a key
ingredient for intelligence and autonomous development, whereas they mostly
still lack effort and success is in building _explicit, actionable models_ of
the world, there including the environment and the society of agents.

###  IV-B Learning causality

Understanding and leveraging causality is recognized as a key general
challenge for AI in the coming years [24]. In particular, Judea Pearl [25] has
proposed the idea of a “causal hierarchy” (also named “ladder of causation”)
to define different levels of causality recognition and exploitation by an
intelligent agent.

The first level of the ladder consists in simply detecting relations as
associations, whereas the second one assumes the possibility to intervene in
the environment and observe the (causal) effects of the taken actions.
Finally, the third level enables reasoning and planning on the basis of
counterfactual analysis. Such layers correspond to some of the phases of the
autonomous development loop we defined: the first one is mostly involved in
the perception phase, whereas the second one is associated to the development
of a sense of agency and to recognition of self and non-self. The final layer
clearly enables goal-oriented behavior, strategic thinking, and collective
coordination.

Bayesian and causal networks are among the models that are most widely
exploited in order to build interpretable causal models of the world. A recent
contribution that is in line with the ideas we envision for autonomous
development is the application of curriculum learning to the problem of
learning the structure of Bayesian networks [26]. On a pure sub-symbolic
level, on the other hand, another recent work proposes to learn causal models
in an online setting [27], with the aim to find (and strengthen) causal links
between input and output variables.

We argue that key challenges in this area concern, again, understanding how to
synergistically exploit symbolic and sub-symbolic approaches to learn,
represent, and evolve causal models in autonomous development scenarios, and
how to use them to adaptively achieve goals.

###  IV-C Autocurricula

When multiple agents act in a shared environment, their actions and their
effectiveness in achieving goals are affected by what others do. Game-
theoretic approaches to strategic thinking have deeply investigated this
problem and the decision-making processes behind [28]. In this context, it has
also been shown that agents can effectively learn in autonomy to improve their
performance in dealing with others [29].

However, when moving from theoretical settings (e.g., the prisoner’s dilemma)
to complex and realistic scenarios where agents have complex goals, peculiar
phenomena arise. The more one agent learns, the more it challenges others,
triggering a continuous increase in complexity of behaviour, ultimately
enabling to incrementally learn more sophisticated means to act. This somewhat
resembles the increase of complexity that agents face in curricula approaches
to reinforcement learning. The key difference being that, in the presence of
multiple agents, the increase in complexity and capabilities of agents is
promoted and self-sustained by the system itself, hence the term
_autocurricula_ [12].

Recently, autocurricula-based approaches have produced stunning results in
multiagent environments, both cooperative and competitive. For instance, in a
hide and seek scenario [30]), agents moving in a complex simulated environment
have learned how to effectively compete (hiders against seekers) and cooperate
(coalitions of cooperating seekers/hiders) in very elaborated ways, in a
continuous self-sustained learning process. Indeed, we consider such
approaches fundamental towards the autonomous development of complex agent
societies. Yet, a deep understanding of the process that drives evolution of
individual and collective behaviors is still missing, and is a key challenge
for the next few years. To this end, providing agents an _explicit_ modeling
(possibly in _causal_ terms) of the others’ behavior and of the overall
societal behavior, may be necessary [9]. Also, autocurricula approaches do not
currently account for the possibility of explicitly interacting (e.g., through
speech acts) with other agents, which may indeed fundamental to improve
collective learning.

###  IV-D Learning to communicate and coordinate

Agents may communicate and coordinate by explicit messages, by leaving traces
in the environment, or implicitly [10].

These forms of communication are already exploited in multiagent learning,
mostly to improve the individual learning process by letting agents share
information (e.g., for merging their individual causal models of the world
[31]) ) and coordinate actions. However, these communication approaches are
usually assumed as an _innate_ capability of agents, rather than one to be
learnt. That is, agents have an _a-priori_ sense of agency with respect to
communication actions, whereas in our vision it should be developed by
learning.

For example, with reference to explicit communication acts, [32] proposes a
voting game to let agents learn to share a communication language and to
develop a strategy to communicate. In [33], it is shown that reinforcement
learning can be effectively applied to let agents learn how to communicate in
order to achieve a specific effect. In the case of _implicit_ communication,
instead, forms of implicit behavioral communications have been shown to emerge
in simple system components that purposefully move in an environment [34], as
they learn to affect others with _ad-hoc_ actions. Learning to use stigmergy
to effectively coordinate is under-explored in the literature, which instead
focuses on the opposite – using stigmergy to boost learning.

In any case, the development of general approaches to let agents developed
fully-fledged forms of communication and coordination is still an open
challenge, which may call for agents to develop not only a model of the world,
but an overall model of the society, i.e., a social sense of agency explicitly
modeling how communications and coordination actions affect other agents in
the shared environment.

###  IV-E Emergence of Institutions

Whereas learning to communicate is about understanding how to use
communication to coordinate actions with others, enabling and sustaining
global collective achievement of goals requires “institutionalized” means of
acting at the collective level, i.e., a set of shared beliefs and of shared
social conventions and norms aimed at ruling collective actions [35]. The
mechanisms leading to the spontaneous emergence of institutions in human
society, there included the mechanisms to promote and sustain altruistic and
cooperative behavior (e.g., reputation and shared rewards), have been widely
investigated [36]. However, most approaches to building multiagent systems
assume such mechanisms as _explicitly designed_ [35].

Yet, some promising studies related to the emergence of institutionalized
behaviors in multiagent systems have been undertaken (see [11] for a recent
survey). For instance, [37] proposes a collective learning framework where
agents learn to adopt norms in repeated coordination, i.e., agents eventually
_learn_ that a social norm has emerged, and “institutionalize” their behaviour
in their (social) decision making processes _implicitly_ , by behaving so as
to comply to the norm. Another interesting work [38] integrates rational
thought, reinforcement learning, and social interactions to model norms
emergence in a society: agents incrementally develop a social behaviour (a
social norm) while _internalising_ it within their cognitive model.

However, the development of general models and tools to support the proper
learning and evolution of institutionalized mechanisms of coordination,
through the construction of explicit norms representations and adoption by
agents cognitive models is still missing, and so are the solutions to the many
problems involved in this process. For instance: how to avoid that an agent
learns that free-riding is better than abiding to norms; or how to avoid
inconsistencies and misunderstandings in their interpretation.

##  V Horizontal Challenges

The presented approaches and techniques are still at the research stage, and
many research challenges have been identified for each of them. In addition,
it is possible to identify several additional “horizontal” challenges, i.e.,
of a general nature independently of the specific approach.

The specific nature of such challenges, in our opinion, makes them
specifically suited for being pursued by the most diverse research communities
in the area of machine learning, reinforcement learning, autonomous and multi-
agent systems.

_Engineering_. Many of the presented approaches are grounded in the area of
machine learning and multi-agent systems, disciplines with plenty of years of
research behind, but in which traditional software engineering problems are
sometimes considered mundane. Systems are often developed _ad-hoc_ for a
specific task or problem domain, with little attention to modularity, re-
usability, dependability, thus missing the flexibility to adopt them across
different domains, tasks, data-sets [39]. In addition, given that the diverse
approaches presented can each contribute important pieces to the overall
vision of autonomous development, sound engineering approaches are needed to
try to integrate such a heterogeneous plethora into a coherent whole. These
represent multi-faceted and horizontal research challenges that, in our
opinion, could and should be profitably attacked by promoting synergies and
collaboration with the software engineering research community.

_Controlling evolution_. Autonomous development raises the issue of somewhat
controlling how behaviors evolve, as individual learns new skills and tasks,
and as the collective learns new way of coordinating and acting together. How
can we _steer_ a learning process towards desired outcomes without putting
bias in it? How can we _constrain_ the boundaries within which individual and
collective behaviors should stay (e.g., in terms of safety)? What
_interventions_ can we make to re-direct an agent or a collective that has
taken an unpredictable or unsafe autonomous development path? Experience in
self-adaptive components based on feedback, as well as in the study of
emergent behaviors in self-organizing systems and definitely help in finding
proper technical answers, and – why not – _ethical_ ones [40].

_Humans in the Loop_. The more technologies based on autonomous development
will advance, the more humans will have to actively interact with them. This
interaction will raise technical issues (will we have “handles” to control or
block such systems in some ways and to some extent?) and ethical problems
(will we be rather “handled” by these systems and subjects to their
decisions?). Some of these problems already emerged, like in the moral machine
experiment [41] or in AI-based hiring technology. Technical challenges will be
meat for the HCI and distributed systems communities (there included the self-
organizing systems one). Ethical and moral ones will be meat for politicians
and lawyers, although deep joint work with technical experts will always be
necessary. A key ingredient involves institutions, since they represent humans
as a group: laws and regulations need to be developed to regulate the global
actors into the day-by-day technology usage. Nevertheless, a deeper
interaction between researchers in science and technology and public
institutions is needed to support the regulation design phase.

_Sustainability_. Algorithms for autonomous development will most likely
require extensive computational resources. For example, the mentioned “hide
and seek’ experiment by OpenAI involved a distributed infrastructure of
128,000 pre-emptible CPU cores and 256 GPUs on GCP [30]: the default model
optimized over 1.6 million parameters taking 34 hours to reach the fourth
stage over six of agents skills progression. This example is a sort of best-
in-class projects; anyway, it is clear that if autonomously developing systems
will be based on similar learning approaches, they will require massive
amounts of computational resources. Therefore, a key challenge for the
community will be to devise algorithmic and system-level means to make
autonomous development systems sustainable, and affordable by others other
then the big technology players.

_Explainability_. Being able to inspect and explain the decision making
process of AI systems is already a hot topic, so much that an entire research
field (XAI, from eXplainable AI) has born. We already commented several times
how such problems should be compulsory accounted for also for autonomous
development, possibly with the help of causal models or other symbolic models.
This is indeed a key challenge for the several research research community,
too, where explaining individual and social behaviors, and patterns and global
configurations emerging from local interactions is mostly still considered a
“holy grail”.

##  VI Conclusions

The general vision of autonomous development is still far to be reality.
However, several ideas in the areas of machine learning, causality, multi-
agent systems, are already showing its potential feasibility and
applicability, at least in specific application areas. We argue that
researchers in these areas have plenty of room for further exploring the topic
and contribute to advance the vision, possibly addressing the many open issues
that we have tried to identify.

## References

  * [1] P. Rochat, “Self-perception and action in infancy,” _Experimental brain research_ , vol. 123, no. 1-2, pp. 102–109, 1998.
  * [2] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, “Building machines that learn and think like people,” _Behavioral and brain sciences_ , vol. 40, 2017.
  * [3] M. Lippi, S. Mariani, and F. Zambonelli, “Developing a sense of agency in IoT systems: Preliminary experiments in a smart home scenario,” in _17th CoMoRea workshop at PerCom_. IEEE, 2021.
  * [4] Y. Yang, M. Taylor, J. Luo, Y. Wen, O. Slumbers, D. Graves, H. Bou Ammar, and J. Wang, “Diverse auto-curriculum is critical for successful real-world multiagent learning systems,” in _20th International Conference on Autonomous Agents and Multiagent Systems_. IFAAMAS, 2021.
  * [5] J. Weng, J. McClelland, A. Pentland, O. Sporns, I. Stockman, M. Sur, and E. Thelen, “Autonomous mental development by robots and animals,” _Science_ , vol. 291, no. 5504, pp. 599–600, 2001.
  * [6] S. Kounev, P. Lewis, K. L. Bellman, N. Bencomo, J. Camara, A. Diaconescu, L. Esterle, K. Geihs, H. Giese, S. Götz, _et al._ , “The notion of self-aware computing,” in _Self-Aware Computing Systems_. Springer, 2017, pp. 3–16.
  * [7] G. Marcus and E. Davis, “Insights for ai from the human mind,” _Commun. ACM_ , vol. 64, no. 1, p. 38–41, Dec. 2021.
  * [8] M. Salehie and L. Tahvildari, “Self-adaptive software: Landscape and research challenges,” _ACM transactions on autonomous and adaptive systems_ , vol. 4, no. 2, pp. 1–42, 2009.
  * [9] B. Subagdja and A.-H. Tan, “Beyond autonomy: The self and life of social agents,” in _Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems_ , 2019, pp. 1654–1658.
  * [10] S. Mariani and A. Omicini, “Anticipatory coordination in socio-technical knowledge-intensive environments: Behavioural implicit communication in MoK,” in _AI*IA 2015_. Springer, 2015, pp. 102–115.
  * [11] A. Morris-Martin, M. De Vos, and J. Padget, “Norm emergence in multiagent systems: a viewpoint paper,” _Autonomous Agents and Multi-Agent Systems_ , vol. 33, no. 6, pp. 706–749, 2019.
  * [12] J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel, “Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research,” _arXiv preprint arXiv:1903.00742_ , 2019.
  * [13] J. Bongard, V. Zykov, and H. Lipson, “Resilient machines through continuous self-modeling,” _Science_ , vol. 314, no. 5802, pp. 1118–1121, 2006.
  * [14] M. Yim, Y. Zhang, and D. Duff, “Modular robots,” _IEEE Spectrum_ , vol. 39, no. 2, pp. 30–34, 2002.
  * [15] N. Cambier, R. Miletitch, V. Frémont, M. Dorigo, E. Ferrante, and V. Trianni, “Language evolution in swarm robotics: A perspective,” _Frontiers in Robotics and AI_ , vol. 7, p. 12, 2020.
  * [16] S. Bussmann and K. Schild, “Self-organizing manufacturing control: an industrial application of agent technology,” in _Proceedings Fourth International Conference on MultiAgent Systems_ , 2000, pp. 87–94.
  * [17] J. Ye, S. Dobson, and F. Zambonelli, “Lifelong learning in sensor-based human activity recognition,” _IEEE Pervasive Comput._ , vol. 18, no. 3, pp. 49–58, 2019.
  * [18] Z. Ullah, F. Al-Turjman, L. Mostarda, and R. Gagliardi, “Applications of artificial intelligence and machine learning in smart cities,” _Computer Communications_ , vol. 154, pp. 313–323, 2020.
  * [19] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [20] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th International Conference on Machine Learning_ , 2009, pp. 41–48.
  * [21] J. Schmidhuber, “Formal theory of creativity, fun, and intrinsic motivation (1990–2010),” _IEEE Transactions on Autonomous Mental Development_ , vol. 2, no. 3, pp. 230–247, 2010.
  * [22] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,” in _International Conference on Learning Representations_ , 2018.
  * [23] K. Khetarpal, Z. Ahmed, G. Comanici, D. Abel, and D. Precup, “What can i do here? a theory of affordances in reinforcement learning,” in _International Conference on Machine Learning_ , 2020, pp. 5243–5253.
  * [24] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, “Toward causal representation learning,” _Proceedings of the IEEE_ , 2021.
  * [25] J. Pearl, “The seven tools of causal inference, with reflections on machine learning,” _Commun. ACM_ , vol. 62, no. 3, pp. 54–60, Feb. 2019.
  * [26] Y. Zhao, Y. Chen, K. Tu, and J. Tian, “Learning bayesian network structures under incremental construction curricula,” _Neurocomputing_ , vol. 258, pp. 30–40, 2017.
  * [27] K. Javed, M. White, and Y. Bengio, “Learning causal models online,” _arXiv preprint arXiv:2006.07461_ , 2020.
  * [28] R. B. Myerson, _Game theory_. Harvard university press, 2013.
  * [29] A. Nowé, P. Vrancx, and Y.-M. De Hauwere, “Game theory and multi-agent reinforcement learning,” in _Reinforcement Learning_. Springer, 2012, pp. 441–470.
  * [30] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch, “Emergent tool use from multi-agent autocurricula,” _arXiv preprint arXiv:1909.07528_ , 2020.
  * [31] S. Meganck, S. Maes, B. Manderick, and P. Leray, “Distributed learning of multi-agent causal models,” in _IEEE/WIC/ACM International Conference on Intelligent Agent Technology_. IEEE, 2005, pp. 285–288.
  * [32] S. Gupta and A. Dukkipati, “Winning an election: On emergent strategic communication in multi-agent networks,” in _International Conference on Autonomous Agents and Multiagent Systems_ , 2020, pp. 1861–1863.
  * [33] J. N. Foerster, Y. M. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” _arXiv preprint arXiv:1605.06676_ , 2016.
  * [34] N. A. Grupen, D. D. Lee, and B. Selman, “Low-bandwidth communication emerges naturally in multi-agent learning systems,” _arXiv preprint arXiv:2011.14890_ , 2020.
  * [35] M. Esteva, J.-A. Rodriguez-Aguilar, C. Sierra, P. Garcia, and J. L. Arcos, “On the formal specification of electronic institutions,” in _Agent mediated electronic commerce_. Springer, 2001, pp. 126–147.
  * [36] M. A. Nowak, “Five rules for the evolution of cooperation,” _Science_ , vol. 314, no. 5805, pp. 1560–1563, 2006.
  * [37] C. Yu, M. Zhang, and F. Ren, “Collective learning for the emergence of social norms in networked multiagent systems,” _IEEE Trans. Cybern._ , vol. 44, no. 12, pp. 2342–2355, 2014.
  * [38] R. Beheshti, “Normative agents for real-world scenarios,” in _International Conference on Autonomous Agents and Multi-Agent Systems_ , 2014, pp. 1749–1750.
  * [39] B. Porter and R. Rodrigues Filho, “Distributed emergent software: Assembling, perceiving and learning systems at scale,” in _IEEE International Conference on Self-Adaptive and Self-Organizing Systems_ , 2019, pp. 127–136.
  * [40] A. Yapo and J. Weiss, “Ethical implications of bias in machine learning,” in _51st Hawaii International Conference on System Sciences_ , 2018.
  * [41] E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan, “The moral machine experiment,” _Nature_ , vol. 563, no. 7729, pp. 59–64, 2018.


# CoTexT: Multi-task Learning with Code-Text Transformer

Long Phan  Case Western Reserve University, Ohio, USA  Hieu Tran  University
of Science, VNU-HCM, Vietnam  Daniel Le  Case Western Reserve University,
Ohio, USA  Hieu Nguyen  Case Western Reserve University, Ohio, USA  James
Anibal  Case Western Reserve University, Ohio, USA  Alec Peltekian  Case
Western Reserve University, Ohio, USA  Yanfang Ye  Case Western Reserve
University, Ohio, USA

###### Abstract

We present CoTexT, a pre-trained, transformer-based encoder-decoder model that
learns the representative context between natural language (NL) and
programming language (PL). Using self-supervision, CoTexT is pre-trained on
large programming language corpora to learn a general understanding of
language and code. CoTexT supports downstream NL-PL tasks such as code
summarizing/documentation, code generation, defect detection, and code
debugging. We train CoTexT on different combinations of available PL corpus
including both ”bimodal” and ”unimodal” data. Here, bimodal data is the
combination of text and corresponding code snippets, whereas unimodal data is
merely code snippets. We first evaluate CoTexT with multi-task learning: we
perform Code Summarization on 6 different programming languages and Code
Refinement on both small and medium size featured in the CodeXGLUE dataset. We
further conduct extensive experiments to investigate CoTexT on other tasks
within the CodeXGlue dataset, including Code Generation and Defect Detection.
We consistently achieve SOTA results in these tasks, demonstrating the
versatility of our models.

##  1 Introduction

In recent years, pre-trained language models (LM) have played a crucial role
in the development of many natural language processing (NLP) systems. Before
the emergence of large LMs, traditional word embedding gives each word/token a
global representation. Large pre-trained models such as ELMo Peters et al.
(2018), GPT Brown et al. (2020), BERT Devlin et al. (2018), and XLNet Yang et
al. (2020) can derive contextualized word vector representations from large
corpora. These methods can learn generalized representations of language and
have significantly improved a broad range of downstream NLP tasks. These LMs
make use of learning objectives such as Masked Language Modeling (MLM) Devlin
et al. (2018) where random tokens in a sequence are masked and the model
predicts the original tokens to learn the context. The success of pre-trained
models in NLP has created a path for domain-specific pre-trained LMs, such as
BioBERT Lee et al. (2019a) on biomedical text, or TaBERT Yin et al. (2020) on
NL text and tabular data.

We introduce CoTexT (Code and Text Transfer Transformer), a pre-trained model
for both natural language (NL) and programming language (PL) such as Java,
Python, Javascript, PHP, etc. CoTexT follows the encoder-decoder architecture
proposed by Vaswani et al. (2017) with attention mechanisms. We then adapt the
model to match T5 framework proposed by Raffel et al. (2019). We test CoTexT
by performing exhaustive experiments on multi-task learning of multiple
programming languages and other related tasks.

We train CoTexT using large programming language corpora containing multiple
programming languages (including Java, Python, JavaScript, Ruby, etc.). Here,
we test different combinations of unimodal and bimodal data to produce the
best result for each downstream task. We then fine-tune CoTexT on four
CodeXGLUE tasks Lu et al. (2021) including CodeSummarization, CodeGeneration,
Defect Detection and Code Refinement (small and medium dataset). Results show
that we achieve state-of-the-art values for each of the four tasks. We found
that CoTexT outperforms current SOTA models such as CodeBERT Feng et al.
(2020) and PLBART Ahmad et al. (2021a).

In this paper we offer the following contribution:

  * •

Three different versions of CoTexT that achieve state-of-the-art on the
CodeXGLUE’s CodeSummarization, CodeGeneration, Defect Detection and Code
Refinement (small and medium dataset) tasks. We publicize our CoTexT pre-
trained checkpoints and related source code available for future studies and
improvements.

##  2 Related Work

Recent work on domain adaptation of BERT show improvements compared to the
general BERT model. BioBERT Lee et al. (2019b) is further trained from
BERTBASE on biomedical articles such as PubMed abstracts and PMC articles.
Similarly, SciBERT Beltagy et al. (2019) is trained on the full text of
biomedical and computer science papers. The experimental results of these
models on domain-specific datasets show the enhanced performance compared to
BERTBASE.

Relating specfically to our work, CodeBERT is Feng et al. (2020) trained on
bimodal data of NL-PL pairs. This strategy allows CodeBERT to learn general-
purpose representations of both natural language and programming language.
GraphCodeBERT Guo et al. (2021) is an extension of CodeBERT that moves beyond
syntactic-level structure and uses data flow in the pre-training stage to
capture the semantic-level structure of code. More recently, PLBART Ahmad et
al. (2021b) is a pre-trained sequence-to-sequence model for NL and PL. Through
denoising autoencoding, this model can perform well on NL-PL understanding and
generation tasks.

##  3 CoTexT

###  3.1 Vocabulary

Following the example of T5 Raffel et al. (2019), we use the Sentence Piece
Unsupervised Text Tokenizer proposed by Kudo and Richardson (2018). The
Sentence Piece model extracts the sub-words that contain the semantic context
of a sequence. We employ Sentence Piece as a vocabulary model for all of our
contributed CoTexT models. However, the special tokens used in code (such as
”[”, ”{”, ”$”, etc) are out-of-vocab for the SentencePiece model
111https://github.com/google/sentencepiece. These tokens have a crucial
representative context in programming languages. Therefore, to enhance the
robustness of the model, we encode all of these missing tokens into a natural
language representation during both self-supervised and supervised training.

Figure 1: An illustration about Fill-in-the-blank objective

###  3.2 Pre-training CoTexT

We train CoTexT on both bimodal and unimodal data. Bimodal data contains both
code snippets and the corresponding natural text in each sequence, while
unimodal data contains only the sequence of code. We use two main datasets
during self-supervised training: CodeSearchNet Corpus Collection Husain et al.
(2020) and GitHub
Repositories222https://console.cloud.google.com/marketplace/details/github/github-
repos data. The combinations of corpus used to train CoTexT are listed in
Table 1. To save both time and computing resources, we initialized the
checkpoints from the original T5 that was trained on the C4 corpus. Raffel et
al. (2019).

####  3.2.1 CodeSearchNet Corpus Collection

CodeSearchNet Corpus Husain et al. (2020) contains coded functions from open-
source non-forked Github repositories. This dataset spans 6 coding languages
(Python, Java, Javascript, PHP, Ruby, Go), which facilitates multi-task
learning. CodeSearchNet also contains a natural language description for each
function. For bimodal data, we simply concatenate the natural language snippet
with the corresponding code snippet to create one input sequence. These data
are then processed as described in 3.1.

Table 1: Pre-training CoTexT on different combinations of natural language and programming language copora Model | N-modal | Corpus combination
---|---|---
T5 | NL | C4
CoTexT (1-CC) | PL |  C4 + CodeSearchNet
CoTexT (2-CC) | NL-PL |  C4 + CodeSearchNet
CoTexT (1-CCG) | PL |  C4 + CodeSearchNet + Github Repos

####  3.2.2 GitHub repositories

We download a large collection of Java and Python functions from the GitHub
repositories dataset available on Google BigQuery. These Java and Python
functions are then extracted and the natural language descriptions are
obtained using the pre-processing pipeline from Lachaux et al. (2020). These
datapoints also run through a pipeline to replace special tokens (as described
in 3.1).

###  3.3 Input/Output Representations

CoTexT converts all NLP problems into a text-to-text format. This means that
during both self-supervised pre-training and supervised training, we use an
input sequence and a target sequence. For the bimodal model, we concatenate a
sequence of natural language text and the corresponding sequence of
programming language text as an input. For the unimodal model, we simply use
each coded function as an input sequence. During self-supervised training,
spans of the input sequence are randomly masked and the target sequence Raffel
et al. (2019) is formed as the concatenation of the same sentinel tokens and
the real masked spans/tokens.

###  3.4 Model Architecture

CoTexT follows the sequence-to-sequence encoder-decoder architecture proposed
by Vaswani et al. (2017). We initialize the Base T5 model released by Raffel
et al. (2019) which has 220 million parameters. We train the model with a
0.001 learning rate and an input/target length of 1024. With the provided TPU
v2-8 on Google Colab, we train with the recommended setting of model
parallelism 2 and batch size 128.

###  3.5 Multi-task Learning

The model is trained with maximum likelihood objective (that is using ”teacher
forcing” Williams and Zipser (1989)) regardless of the text-code or code-text
tasks. Therefore, for CoTexT, we leverage the potential for Multi-Task
learning Raffel et al. (2019) to complete both text-code and code-text
generation on CodeSummarization and Code Refinement tasks. To specify the task
our model should perform, we simply add a task-specific prefix to the input
sequence. For example, when fine-tuning of the CodeSummarization task for each
programming language, we simply prepend a prefix for each PL name (i.e., Java)
to the input sequence.

Figure 2: An illustration about Multi-task learning

##  4 Experiments

In this section, we will first describe the benchmark dataset for code
intelligence CodeXGLUE, then we will explain the experimental setup on the
tasks we perform and discuss the results of each task. The evaluation datasets
are summarized in Table 3.

###  4.1 CodeXGLUE

General Language Understanding Evaluation benchmark for CODE (CodeXGLUE) Lu et
al. (2021) is a benchmark dataset to facilitate machine learning studies on
code understanding and code generation problems. This dataset includes a
collection of code intelligence tasks (both classification and generation), a
platform for model evaluation, and a leaderboard for comparison. CodeXGLUE has
10 code intelligence tasks including code-text, text-code, code-code, and
text-text scenarios. For CoTexT, we focus on Code Summarization, Code
Generation, Code Refinement, and Defect Detection tasks.

Table 2: The input and target sequence length settings for each self-supervised learning, code summarization, code generation, code refinement, and defect detection task Task | Dataset | Task Type | Input Length | Target Length
---|---|---|---|---
Self-supervised Learning | CodSearchNet Corpus |  | 1024 | 1024
GitHub Repositories |  | 1024 | 1024
Code Summarization | CodeSearchNet | Multi-Task | 512 | 512
Code Generation | CONCODE | Single-Task | 256 | 256
Code Refinement | Bugs2Fixsmall | Multi-Task | 512 | 512
| Bugs2Fixmedium
Defect Detection | Devign | Single-Task | 1024 | 5

###  4.2 Evaluation Tasks

We evaluate our programming language and natural language generation tasks on
TPU v2-8 with the settings from the original T5 model Raffel et al. (2019).
The input length and target length for each task are described in Table 2.

####  4.2.1 Code Summarization

For Code Summarization, the objective is to generate a natural language
description for a given code snippet. The task includes a CodeSearchNet
dataset Husain et al. (2019) with 6 different programming languages: Python,
Java, Javascript, PHP, Ruby, Go. The data comes from public open-source non-
fork GitHub repositories and the annotations are extracted from function
documentation as described in Husain et al. (2019).

####  4.2.2 Code Generation

Text-to-Code Generation aims to generate a coded function given a natural
language description. This task is completed using the CONCODE dataset Iyer et
al. (2018), a well-known dataset for Java language generation. Within the
dataset, there are tuples which contain a natural language description, code
environments, and code snippets. The goal is to generate the correct Java
function from the natural language description in the form of Javadoc-style
method comments.

####  4.2.3 Code Refinement

Code Refinement, or Code Repair, aims to automatically correct bugs in Java
code. We used the Bug2Fix corpus released by CodeXGLUE Lu et al. (2021), which
divides the task into 2 subsets: SMALL and MEDIUM The small dataset includes
only Java code functions with fewer than 50 tokens. The medium dataset
includes functions with 50-100 tokens.

####  4.2.4 Defect Detection

For Defect Detection tasks, we attempt to classify whether a PL snippet
contains vulnerabilities that could lead to damaging outcomes such as resource
leaks or DoS attacks. The task uses the Devign dataset Zhou et al. (2019),
which contains C programming language from open-source projects. This dataset
is labeled based on security-related commits. For details on the annotation
process, refer to Zhou et al. (2019).

Table 3: Data statistics about Code Intelligence datasets Category |  Task |  Dataset | Size |  Language
---|---|---|---|---
|  |  Train |  Val |  Test
Code-Text |  Code Summarization Lu et al. (2021) |  CodeSearchNet |  164K |  5.1K |  10.9K |  Java
|  |  58K |  3.8K |  3.2K |  Javascript
|  |  251K |  13.9K |  14.9K |  Python
|  |  241K |  12.9K |  14K |  PHP
|  |  167K |  7.3K |  8.1K |  Go
|  |  24K |  1.4K |  1.2K |  Ruby
Code-Code |  Defect Detection |  Devign |  21K |  2.7K |  2.7K |  C
Zhou et al. (2019) |  |  |  |
Code Refinement Lu et al. (2021) |  Bugs2Fixsmall |  46K |  5.8K |  5.8K |  Java
|  Bugs2Fixmedium |  52K |  6.5K |  6.5K
Text-Code |  Code Generation |  CONCODE |  100K |  2K |  2K |  Java
Iyer et al. (2018) |  |  |  |

###  4.3 Experimental Setup

####  4.3.1 Baselines

We compare our model with some well-known pre-trained models:

  * •

CodeGPT, CodeGPT-adapted are based on the architecture and training objective
of GPT-2 Budzianowski and Vulic (2019). CodeGPT is pre-trained from scratch on
CodeSearchNet dataset Lu et al. (2021) while CodeGPT-adapted learns this
dataset starting from the GPT-2 checkpoint.

  * •

CodeBERT Feng et al. (2020) employs the same architecture as RoBERTa Liu et
al. (2020) but aims to minimize the combined loss from masked language
modeling and replaced token detection.

  * •

PLBART Ahmad et al. (2021b) is a Transformer-based model. BART Lewis et al.
(2019) is trained on PL corpora using three learning strategies: token
masking, token deletion, and token infilling.

####  4.3.2 Performance Metrics

  * •

BLEU Papineni et al. (2002) is an algorithm which performs automatic
evaluation of machine-translated text. This method calculates the n-gram
similarity of a candidate translation compared to a set of reference texts.
Similar to Feng et al. (2020) and Ahmad et al. (2021b), we use smooth BLEU-4
score lin-och-2004-orange for Code Summarization and corpus-level BLEU score
for all remaining tasks.

  * •

CodeBLEU Ren et al. (2020) is designed to consider syntactic and semantic
features of codes based on the abstract syntax tree and the data flow
structure.

  * •

Accuracy is the ratio of the number of generated sequences that harmonise the
reference to the total number of observations.

Table 4: Test result on Code Generation task

Model | Text2Code Generation |
---|---|---
EM | BLEU | CodeBLEU |
PLBART | 18.75 |  36.69 | 38.52 |
CodeGPT-adapted | 20.10 | 32.79 | 35.98 |
CodeGPT | 18.25 | 28.69 | 32.71 |
T5 | 18.65 | 32.74 | 35.95 |
CoText (1-CCG) | 19.45 | 35.40 | 38.47 |
CoText (2-CC) |  20.10 | 36.51 |  39.49 |
CoText (1-CC) | 20.10 | 37.40 | 40.14 |

  * •

Notes: The best scores are in bold and second best scores are underlined. The
baseline scores were obtained from the CodeXGLUE’s Leaderboard
(https://microsoft.github.io/CodeXGLUE/)

Table 5: Test result on Code Summarization task

Model |  All |  Ruby |  Javascript |  Go |  Python |  Java |  PHP
---|---|---|---|---|---|---|---
RoBERTa |  16.57 |  11.17 |  11.90 |  17.72 |  18.14 |  16.47 |  24.02
CodeBERT |  17.83 |  12.16 |  14.90 |  18.07 |  19.06 |  17.65 |  25.16
PLBART |  18.32 |  14.11 |  15.56 |  18.91 |  19.3 |  18.45 |  23.58
T5 |  18.35 |  14.18 |  14.57 |  19.17 |  19.26 |  18.35 |  24.59
CoTexT (1-CCG) |  18.00 |  13.23 |  14.75 |  18.95 |  19.35 |  18.75 |  22.97
CoTexT (2-CC) |  18.38 |  13.07 |  14.77 |  19.37 |  19.52 |  19.1 |  24.47
CoTexT (1-CC) |  18.55 |  14.02 |  14.96 |  18.86 |  19.73 |  19.06 |  24.58

  * •

Notes: The best scores are in bold and second best scores are underlined. The
baseline scores were obtained from the CodeXGLUE’s Leaderboard
(https://microsoft.github.io/CodeXGLUE/)

Table 6: Test result on Code Refinement task

| Small test set | Medium test set
---|---|---
Model |  BLEU |  Acc(%) |  CodeBLEU |  BLEU |  Acc(%) |  CodeBLEU
Transformer |  77.21 |  14.70 |  73.31 |  89.25 |  3.70 |  81.72
CodeBERT |  77.42 |  16.40 |  75.58 |  91.07 |  5.16 |  87.52
PLBART |  77.02 |  19.21 |  / |  88.5 |  8.98 |  /
T5 |  74.94 |  15.3 |  75.85 |  88.28 |  4.11 |  85.61
CoTexT (1-CCG) |  76.87 |  20.39 |  77.34 |  88.58 |  12.88 |  86.05
CoTexT (2-CC) |  77.28 |  21.58 |  77.38 |  88.68 |  13.03 |  84.41
CoTexT (1-CC) |  77.79 |  21.03 |  76.15 |  88.4 |  13.11 |  85.83

  * •

Notes: The best scores are in bold and second best scores are underlined. The
baseline scores were obtained from the CodeXGLUE’s Leaderboard
(https://microsoft.github.io/CodeXGLUE/)

Table 7: Test result on Defect Detection task

Model |  Accuracy
---|---
RoBERTa |  61.05
CodeBERT |  62.08
PLBART |  63.18
T5 |  61.93
CoTexT (1-CCG) |  66.62
CoTexT (2-CC) |  64.49
CoTexT (1-CC) |  65.99

  * •

Notes: The best scores are in bold and second best scores are underlined. The
baseline scores were obtained from the CodeXGLUE’s Leaderboard
(https://microsoft.github.io/CodeXGLUE/)

##  5 Results

###  5.1 Multi-Task Learning

We first report the result of CoTexT in Multi-Task Learning tasks including
Code Summarization and Code Refinement.

####  5.1.1 Code Summarization

For the Code Summarization task, we perform Multi-Task Learning by using the
T5 framework Raffel et al. (2019) to finetune CoTexT on 6 diferent programming
language (Ruby, Javascript, Go, Python, Java, and PHP). The results of the
Code Summarization task are shown in Table 5.

First, we observe that the base T5, which is pre-trained only on the general
domain corpus (C4), is effective on this task. In fact, base T5 achieves
higher overall results on the BLEU-4 metric compared to all other related
models on the CodeXGLUE leaderboard. This shows the importance of domain-
specific T5 models, which we expect to achieve superior results compared to
base T5.

We further observe that CoTexT achieves state-of-the-art (SOTA) on the overall
score, the Python-specific score, the Java-specific score, and the Go-specific
score. While CoTexT does not significantly outperform other pre-trained
models, we observe that CoTexT achieves SOTA on two very common programming
languages (Python and Java) while still obtaining competitive results on other
programming languages. We attribute this result to the large amount of
training data for Python and Java compared to the other languages (training
size described in Table 3). Based on this result, CoTeXT has the potential to
further surpass competitor models as more training data becomes availible.

####  5.1.2 Code Refinement

We also tested CoTexT by performing multi-task learning for Code Refinement.
In this case, both the small and medium test sets have a task registry with
respective prefix prepending to the input sequence.

The Code Refinement results of each model are shown in Table 6. For this task,
the base T5, which is pre-trained only on natural language text, does not
perform well compared to other transformer-based models. Yet, after the
training on a large programming language corpus, the result from CoTexT
improves significantly on all metrics for both small and medium test sets.
CoTexT achieves SOTA for all metrics on the small test set and on the accuracy
metric for the medium test set.

###  5.2 Single-Task Learning

In addition to multi-task learning, we also evaluate CoTexT performance
single-task learning with a Code Generation Task and a classification task
relating to Defect Detection.

####  5.2.1 Code Generation

In Table 4, we reported our results for the Code Generation task wherein
natural language is translated into Java code. The result shows that our
proposed model achieves SOTA results based on 3 metrics: Exact Match (EM),
BLEU, and CodeBLEU. For each individual metric, CoTexT has only slightly
outperformed other models (e.g both CoTexT and CodeGPT-adapted achieve 20.10
for EM). However, our model is consistently superior across the 3 metrics.
Prior to CoTexT, CodeGPT-adapted was SOTA for the EM metric and PLBART was
SOTA for the BLUE/CodeBLUE metrics. From this result, we infer that CoTexT has
the best overall performance on this task and has great potential in the area
of code generation.

####  5.2.2 Defect Detection

The Defect Detection results are shown in Table 7. Specifically, CoText
outperforms the previous SOTA model (PLBART) by 3.44%. For this task, extra
training on a large programming corpus allows CoTexT to outperform all other
models and achieve SOTA results. The Defect Detection dataset consists of code
written in the C programming language, which is not contained in our training
data. Our model has a strong understanding of similar languages, and is thus
able to perform Defect Detection in C with improved results compared to
competitor models.

##  6 Conclusion

In this manuscript, we introduced CoTexT, a pre-trained language
representation for both programming language and natural language. CoTexT
focused on text-code and code-text understanding and generating. Leveraging
the T5 framework Raffel et al. (2019), we showed that pre-training on a large
programming language corpus is effective for a diverse array of tasks within
the natural language and programming language domain. CoTexT achieves state-
of-the-art results on 4 CodeXGLUE code intelligence tasks: Code Summarization,
Code Generation, Code Refinement, and Code Detection. For future work, we plan
to test CoTexT on a broader range of programming language and natural language
generation tasks, such as autocompletion or code translation.

## References

  * Ahmad et al. (2021a) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021a.  Unified pre-training for program understanding and generation.
  * Ahmad et al. (2021b) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021b.  Unified pre-training for program understanding and generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Beltagy et al. (2019) Iz Beltagy, Arman Cohan, and Kyle Lo. 2019.  Scibert: Pretrained contextualized embeddings for scientific text.  _CoRR_ , abs/1903.10676.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Budzianowski and Vulic (2019) Pawel Budzianowski and Ivan Vulic. 2019.  Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems.  _CoRR_ , abs/1907.05774.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  Codebert: A pre-trained model for programming and natural languages.  _CoRR_ , abs/2002.08155.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021.  Graphcode{bert}: Pre-training code representations with data flow.  In _International Conference on Learning Representations_.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _CoRR_ , abs/1909.09436.
  * Husain et al. (2020) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020.  Codesearchnet challenge: Evaluating the state of semantic code search.
  * Iyer et al. (2018) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018.  Mapping language to code in programmatic context.  _CoRR_ , abs/1808.09588.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _CoRR_ , abs/1808.06226.
  * Lachaux et al. (2020) Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. 2020\.  Unsupervised translation of programming languages.
  * Lee et al. (2019a) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019a.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746.
  * Lee et al. (2019b) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019b.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021.  Codexglue: A machine learning benchmark dataset for code understanding and generation.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _CoRR_ , abs/1802.05365.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020.  Codebleu: a method for automatic evaluation of code synthesis.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Williams and Zipser (1989) Ronald J. Williams and David Zipser. 1989.  A learning algorithm for continually running fully recurrent neural networks.  _Neural Comput._ , 1(2):270–280.
  * Yang et al. (2020) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020.  Xlnet: Generalized autoregressive pretraining for language understanding.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020.  Tabert: Pretraining for joint understanding of textual and tabular data.
  * Zhou et al. (2019) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.  Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.