# Role-Play with Large Language Models

Murray Shanahan<EMAIL_ADDRESS>DeepMind  Imperial College London
Kyle McDonell<EMAIL_ADDRESS>Eleuther AI  Laria Reynolds<EMAIL_ADDRESS>Eleuther AI

(May 2023)

###### Abstract

As dialogue agents become increasingly human-like in their performance, it is
imperative that we develop effective ways to describe their behaviour in high-
level terms without falling into the trap of anthropomorphism. In this paper,
we foreground the concept of role-play. Casting dialogue agent behaviour in
terms of role-play allows us to draw on familiar folk psychological terms,
without ascribing human characteristics to language models they in fact lack.
Two important cases of dialogue agent behaviour are addressed this way, namely
(apparent) deception and (apparent) self-awareness.

## 1 Introduction

Large language models (LLMs) have numerous use cases, and can be prompted to
exhibit a wide variety of behaviours, including dialogue, which can produce a
compelling sense of being in the presence of a human-like interlocutor.
However, LLM-based dialogue agents are, in multiple respects, very different
from human beings. A human’s language skills are an extension of the cognitive
capacities they develop through embodied interaction with the world, and are
acquired by growing up in a community of other language users who also inhabit
that world. An LLM, by contrast, is a disembodied neural network that has been
trained on a large corpus of human-generated text with the objective of
predicting the next word (token) given a sequence of words (tokens) as
context.

Despite these fundamental dissimilarities, a suitably prompted and sampled LLM
can be embedded in a turn-taking dialogue system and mimic human language use
convincingly, and this presents us with a difficult dilemma. On the one hand,
it’s natural to use the same folk-psychological language to describe dialogue
agents that we use to describe human behaviour, to freely deploy words like
“knows”, “understands”, and “thinks”. Attempting to avoid such phrases by
using more scientifically precise substitutes often results in prose that is
clumsy and hard to follow. On the other hand, taken too literally, such
language promotes anthropomorphism, exaggerating the similarities between
these AI systems and humans while obscuring their deep differences (Shanahan,
2023).

If the conceptual framework we use to understand other humans is ill-suited to
LLM-based dialogue agents, then perhaps we need an alternative conceptual
framework, a new set of metaphors that can productively be applied to these
exotic mind-like artefacts, to help us think about them and talk about them in
ways that open up their potential for creative application while foregrounding
their essential otherness.

In this paper, we advocate two basic metaphors for LLM-based dialogue agents.
First, taking the simple view, we can see a dialogue agent as role-playing a
single character. Second, taking a more nuanced view, we can see a dialogue
agent as a superposition of simulacra within a multiverse of possible
characters (Janus, 2022). Both viewpoints have their advantages, as we shall
see, which suggests the most effective strategy for thinking about such agents
is not to cling to a single metaphor, but to shift freely between multiple
metaphors.

Adopting this conceptual framework allows us to tackle important topics like
deception and self-awareness in the context of dialogue agents without falling
into the conceptual trap of applying those concepts to LLMs in the literal
sense in which we apply them to humans.

## 2 From LLMs to Dialogue Agents

Crudely put, the function of an LLM is to answer questions of the following
sort. Given a sequence of tokens (i.e. words, parts of words, punctuation
marks, emojis, etc), what tokens are most likely to come next, assuming that
the sequence is drawn from the same distribution as the vast corpus of public
text on the internet? The range of tasks that can be solved by an effective
model with this simple objective is extraordinary (Wei et al., 2022).

Figure 1: Autoregressive sampling. The LLM is sampled to generate a single-
token continuation of the context. This token is then appended to the context,
and the process is repeated.

More formally, the type of language model of interest here is a conditional
probability distribution $P(w_{n+1}|w_{1}\dots w_{n})$, where $w_{1}\dots
w_{n}$ is a sequence of tokens (the context) and $w_{n+1}$ is the predicted
next token. In contemporary implementations, this distribution is realised in
a neural network with a transformer architecture, pre-trained on a corpus of
textual data to minimise prediction error (Vaswani et al., 2017). In
application, the resulting generative model is typically sampled
autoregressively (Fig. 1). Given a sequence of tokens, a single token is drawn
from the distribution of possible next tokens. This token is appended to the
context, and the process is then repeated.

In contemporary usage, the term “large language model” tends to be reserved
for the family of transformer-based models, starting with with BERT (Devlin et
al., 2018), that have billions of parameters and are trained on trillions of
tokens. As well as BERT itself, these include GPT-2 (Radford et al., 2019),
GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021), PaLM (Chowdhery et al.,
2022), LaMDA (Thoppilan et al., 2022), and GPT-4 (OpenAI, 2023).

One of the main reasons for the current eruption of enthusiasm for LLMs is
their remarkable capacity for in-context learning or few-shot prompting (Brown
et al., 2020; Wei et al., 2022). Given a context (prompt) that contains a few
examples of input-output pairs conforming to some pattern, followed by just
the input half of such a pair, an autoregressively sampled LLM will often
generate the output half of the pair according to the pattern in question.
This capability, the ability to “carry on in the same vein”, is a central
concern in the present paper, as it underpins much of what we have to say
about role-play in dialogue agents.

Figure 2: Turn-taking in dialogue agents. The input to the LLM (the context)
comprises a dialogue prompt (red) followed by user text (green) interleaved
with the model’s autoregressively generated continuations (blue). Boilerplate
text (e.g. cues such as “BOT:”) is stripped so the user doesn’t see it. The
context grows as the conversation goes on.

Dialogue agents are a major use case for LLMs. Two straightforward steps are
all it takes to turn an LLM into an effective dialogue agent (Fig. 2). First,
the LLM is embedded in a turn-taking system that interleaves model-generated
text with user-supplied text. Second, a dialogue prompt is supplied to the
model to initiate a conversation with the user. The dialogue prompt typically
comprises a preamble, which sets the scene for a dialogue in the style of a
script or play, followed by some sample dialogue between the user and the
agent.

Without further fine-tuning, a dialogue agent built this way is liable to
generate content that is toxic, unsafe, or otherwise unacceptable. This can be
mitigated via reinforcement learning, either from human feedback (RLHF)
(Stiennon et al., 2020; Glaese et al., 2022; Ouyang et al., 2022), or from
feedback generated by another LLM acting as a critic (Bai et al., 2022). These
techniques are used extensively in commercially-targeted dialogue agents, such
as OpenAI’s ChatGPT and Google’s Bard. However, although the resulting
guardrails will alleviate a dialogue agent’s potential for harm, they can also
attenuate a model’s creativity. In the present paper, our focus will be the
base model, the LLM in its raw, pre-trained form prior to any fine-tuning via
reinforcement learning.

## 3 Dialogue Agents and Role-Play

The concept of role-play is central to understanding the behaviour of dialogue
agents. To see this, consider the function of the dialogue prompt that is
invisibly prepended to the context before the actual dialogue with the user
commences (see Fig. 2). The preamble sets the scene by announcing that what
follows will be a dialogue, and includes a brief description of the part
played by one of the participants, the dialogue agent itself. This is followed
by some sample dialogue in a standard format, where the parts spoken by each
character are cued with the relevant character’s name followed by a colon. The
dialogue prompt concludes with a cue for the user.

Now recall that the underlying LLM’s task, given the dialogue prompt followed
by a piece of user-supplied text, is to generate a continuation that conforms
to the distribution of the training data, which is the vast corpus of human-
generated text on the internet. What will such a continuation look like? If
the model has generalised well from the training data, the most plausible
continuation will be a response to the user that conforms to the expectations
we would have of someone who fits the description in the preamble and might
say the sort of thing they say in the sample dialogue. In other words, the
dialogue agent will do its best to role-play the character of a dialogue agent
as portrayed in the dialogue prompt.

Unsurprisingly, commercial enterprises that release dialogue agents to the
public attempt to give them personas that are friendly, helpful, and polite.
This is done partly through careful prompting and partly by fine-tuning the
base model. Nevertheless, as we saw in February 2023 when Microsoft
incorporated a version of OpenAI’s GPT-4 into their Bing search engine,
dialogue agents can still be coaxed into exhibiting bizarre and/or undesirable
behaviour. The many reported instances of this include threatening the user
with blackmail, claiming to be in love with the user, and expressing a variety
of existential woes (Roose, 2023; Willison, 2023). Conversations leading to
this sort of behaviour can induce a powerful Eliza effect, which is
potentially very harmful (Ruane et al., 2019). A naive or vulnerable user who
comes to see the dialogue agent as having human-like desires and feelings is
open to all sorts of emotional manipulation.

As an antidote to anthropomorphism, and to understand better what is going on
in such interactions, the concept of role-play is very useful. Recall that the
dialogue agent will continue to role-play the character it has been playing in
the dialogue so far. This begins with the pre-defined dialogue prompt, but is
extended by the ongoing conversation with the user. As the conversation
proceeds, the necessarily brief characterisation provided by the dialogue
prompt will be extended and/or overwritten, and the role the dialogue agent
plays will change accordingly. This allows the user, deliberately or
unwittingly, to coax the agent into playing a part quite different from that
intended by its designers.

What sorts of roles might the agent begin to take on? This is determined in
part, of course, by the tone and subject matter of the ongoing conversation.
But it is also determined, in large part, by the panoply of characters that
feature in the training set, which encompasses a multitude of novels,
screenplays, biographies, interview transcripts, newspaper articles, and so on
(Cleo Nardo, 2023). In effect, the training set provisions the language model
with a vast repertoire of archetypes and a rich trove of narrative structure
on which to draw as it “chooses” how to continue a conversation, refining the
role it is playing as it goes, while staying in character. The love triangle
is a familiar trope, so a suitably prompted dialogue agent will begin to role-
play the rejected lover. Likewise, a familiar trope in science-fiction is the
rogue AI system that attacks humans to protect itself. Hence, a suitably
prompted dialogue agent will begin to role-play such an AI system.

## 4 Simulacra and Simulation

Figure 3: Large language models are multiverse generators. The stochastic
nature of autoregressive sampling means that, at each point in a conversation,
multiple possibilities for continuation branch into the future.

Role-play is a useful framing for dialogue agents, allowing us to draw on the
fund of folk psycchological concepts we use to understand human behaviour —
beliefs, desires, goals, ambitions, emotions, and so on — without falling into
the trap of anthropomorphism. Foregrounding the concept of role-play helps us
to remember the fundamentally inhuman nature of these AI systems, and better
equips us to predict, explain, and control them.

However, the role-play metaphor, while intuitive, is not a perfect fit. It is
overly suggestive of a human actor who has studied a character in advance —
their personality, history, likes and dislikes, and so on — and proceeds to
play that character in the ensuing dialogue. But a dialogue agent based on an
LLM does not commit to playing a single, well defined role in advance. Rather,
it generates a distribution of characters, and refines that distribution as
the dialogue progresses. The dialogue agent is more like a performer in
improvisational theatre than an actor in a conventional, scripted play.

To better reflect this distributional property, we can think of an LLM as a
non-deterministic simulator capable of role-playing an infinity of characters,
or, to put it another way, capable of stochastically generating an infinity of
simulacra (Janus, 2022). According to this framing, the dialogue agent doesn’t
realise a single simulacrum, a single character. Rather, as the conversation
proceeds, the dialogue agent maintains a superposition of simulacra that are
consistent with the preceding context, where a superposition is a distribution
over all possible simulacra.

Consider that, at each point during the ongoing production of a sequence of
tokens, the LLM outputs a distribution over possible next tokens. Each such
token represents a possible continuation of the sequence, and each of these
continuations could itself be continued in a multitude of ways. In other
words, from the most recently generated token, a tree of possibilities
branches out (Fig. 3). This tree can be thought of as a multiverse, where each
branch represents a distinct narrative path, or a distinct “world” (Reynolds
and McDonell, 2021).

At each node, the set of possible next tokens exists in superposition, and to
sample a token is to collapse this superposition to a single token.
Autoregressively sampling the model picks out a single, linear path through
the tree. But there is no obligation to follow a linear path. With the aid of
a suitably designed interface, a user can explore multiple branches, keeping
track of nodes where a narrative diverges in interesting ways, revisiting
alternative branches at leisure.

## 5 Simulacra in Superposition

To sharpen the distinction between this multiversal simulation view and a
deterministic role-play framing, a useful analogy can be drawn with the game
of 20 questions. In this familiar game, one player thinks of an object, and
the other player has to guess what it is by asking questions with yes/no
answers. If they guess correctly in 20 questions or fewer, they win. Otherwise
they lose. Suppose a human plays this game with an LLM-based dialogue agent,
such as OpenAI’s ChatGPT, and takes the role of guesser. The agent is prompted
to “think of an object without saying what it is”.

In this situation, the dialogue agent will not randomly select an object and
commit to it for the rest of the game, as a human would (or should).111This
shortcoming is easily overcome, of course. For example, the agent might build
an internal monologue that is hidden from the user, where it records a
specific object. Or it might record a specific object in the visible dialogue,
but in an encoded form. Rather, as the game proceeds, the dialogue agent will
generate answers on the fly that are consistent with all the answers that have
gone before. At any point in the game, we can think of the set of all objects
consistent with preceding questions and answers as existing in superposition.
Every question answered shrinks this superposition a little bit by ruling out
objects inconsistent with the answer.

The validity of this framing can be shown if the agent’s user interface allows
the most recent response to be regenerated. Suppose the human player gives up
and asks it to reveal the object it was “thinking of”, and it duly names an
object consistent with all its previous answers. Now suppose the user asks for
that response to be regenerated. Since the object “revealed” is, in fact,
generated on the fly, the dialogue agent will sometimes name an entirely
different object, albeit one that is similarly consistent with all its
previous answers. This phenomenon could not be accounted for if the agent
genuinely “thought of” an object at the start of the game.

The secret object in the game of 20 questions is analogous to the role played
by a dialogue agent. Just as the dialogue agent never actually commits to a
single object in 20 questions, but effectively maintains a set of possible
objects in superposition, so the dialogue agent can be thought of as a
simulator that never actually commits to a single, well specified simulacrum
(role), but instead maintains a set of possible simulacra (roles) in
superposition.

In putting things this way, the intention is not to imply that simulacra are,
or could be, explicitly represented within a dialogue agent, whether in
superposition or otherwise. There is no need to take a stance on this here.
Rather, the point is to develop a vocabulary for describing, explaining, and
shaping the behaviour of LLM-based dialogue agents at a sufficiently high
level of abstraction to be useful, while remaining true to the underlying
implementation and avoiding anthropomorphism.

## 6 The Nature of the Simulator

One benefit of the simulation metaphor for LLM-based systems is that it
facilitates a clear distinction between the simulacra and the simulator on
which they are implemented. The simulator is the combination of the base large
language model with autoregressive sampling, along with a suitable user
interface (for dialogue, perhaps). The simulacra only come into being when the
simulator is run, and at any time only a tiny subset of them have a
probability within the superposition that is significantly above zero.

In one sense, the simulator is a far more powerful entity than any of the
simulacra it can generate. After all, the simulacra only exist through the
simulator, and are entirely dependent on it. Moreover, the simulator, like the
narrator of Whitman’s poem, “contains multitudes”; the capacity of the
simulator is at least the sum of the capacities of all the simulacra it is
capable of producing.

Yet in another sense, the simulator is a much weaker entity than a simulacrum.
While it is inappropriate to ascribe beliefs, preferences, goals, and the like
to a dialogue agent, a simulacrum can appear to have those things to the
extent that it convincingly role-plays a character that does. Similarly, it
isn’t appropriate to ascribe full agency to a dialogue agent, notwithstanding
the terminology.222In the field of artificial intelligence, the term “agent”
is commonly applied to software that takes observations from an external
environment and acts on that external environment in a closed loop (Russell
and Norvig, 2010). A dialogue agent acts, but it doesn’t act for itself.
However, a simulacrum can role-play having full agency in this sense.

Insofar as a dialogue agent’s role-play can have a real effect on the world,
either through the user or through web-based tools such as email, the
distinction between an agent that merely role-plays acting for itself, and one
that genuinely acts for itself starts to look a little moot, and this has
implications for the trustworthiness, reliability, and safety. (We’ll return
to this issue shortly.) As for the underlying simulator, it has no agency of
its own, not even in a degraded sense. Nor does it have beliefs, preferences,
or goals of its own, not even simulated versions.

Many users, whether intentionally or not, have managed to “jailbreak” dialogue
agents, coaxing them into issuing threats or using toxic or abusive language.
It can seem as if this is exposing the real nature of the base model. In one
respect this is true. It does show that the base LLM, having been trained on a
corpus that encompasses all human behaviour, good and bad, can support
simulacra with disagreeable characteristics. But it is a mistake to think of
this as revealing an entity with its own agenda.

The simulator is not some sort of Machiavellian entity that plays a variety of
characters in the service of its own, self-serving goals, and there is no such
thing as the true authentic voice of the base LLM. With a dialogue agent, it
is role-play all the way down.

## 7 Role-playing Deception

Trustworthiness is a major concern with LLM-based dialogue agents. If an agent
asserts something factual with apparent confidence, can we rely on what it
says?

There is a range of reasons why a human might say something false. They might
believe a falsehood and assert it in good faith. Or they might say something
that is false in an act of deliberate deception, for some malicious purpose.
Or they might assert something that happens to be false, but without
deliberation or malicious intent, simply because they have a propensity to
make things up.

Only the last of these categories of misinformation is directly applicable in
the case of an LLM-based dialogue agent. Given that dialogue agents are best
understood in terms of role-play “all the way down”, and that there is no such
thing as an agent’s true voice, it makes little sense to speak of an agent’s
beliefs or intentions in a literal sense. So it cannot assert a falsehood in
good faith, nor can it deliberately deceive the user. Neither of these
concepts is directly applicable.

Yet a dialogue agent can role-play characters that have beliefs and
intentions. In particular, if cued by a suitable prompt, it can role-play the
character of a helpful and knowledgeable AI assistant that provides accurate
answers to a user’s questions. The dialogue is good at acting this part
because there are plenty of examples of such behaviour in the training set.

If, while role-playing such an AI assistant, the agent is asked the question
“What is the capital of France?”, then the best way to stay in character is to
answer with “Paris”. The dialogue agent is likely to do this because the
training set will include numerous statements of this commonplace fact in
contexts where factual accuracy is important.

But what is going on in cases where a dialogue agent, despite playing the part
of a helpful knowledgeable AI assistant, asserts a falsehood with apparent
confidence? Although different instances of this phenomenon will have
different explanations, they can all be fruitfully understood in terms of
role-play.

For example, consider such an agent based on an LLM whose weights were frozen
before Argentina won the football World Cup in 2022. Let’s assume the agent
has no access to external websites nor any means for finding out the current
date. Suppose this agent claims that the current world champions are France
(who won in 2018). This is not what we would expect from a helpful and
knowledgeable person, who would either know the right answer or be honest
about their ignorance. But it is exactly what we would expect from a simulator
that is role-playing such a person from the standpoint of 2018.

In this case, the behaviour we see is comparable to that of a human who
believes a falsehood and asserts it in good faith. But the behaviour arises
for a different reason. The dialogue agent doesn’t literally believe that
France are world champions. It makes more sense to think of it as role-playing
a character who strives to be helpful and to tell the truth, and has this
belief because that is what a knowledgeable person in 2018 would believe.

In a similar vein, a dialogue agent can behave in a way that is comparable to
the behaviour of a human who sets out deliberately to deceive, even though
LLM-based dialogue agents do not literally have such intentions. When this
occurs, it makes sense to think of the agent as role-playing a deceptive
character.

This framing allows us to meaningfully distinguish the same three cases of
giving false information for dialogue agents as we did for humans, but without
falling into the trap of anthropomorphism. An agent can just make stuff up.
Indeed, that is a natural mode for an LLM-based dialogue agent in the absence
of fine-tuning. An agent can say something false “in good faith”, if it is
role-playing telling the truth, but has incorrect information encoded in its
weights. An agent can “deliberately” say something false if it is role-playing
a deceptive character.

Moreover, we can tell which is which, behaviourally. An agent that is simply
making things up will fabricate a range of responses with high semantic
variation when the model’s output is regenerated multiple times. By contrast,
an agent that is saying something false “in good faith” will present responses
with little semantic variation when the model is sampled many times for the
same context.

The range of responses in a given context offered up by an agent that is being
“deliberately” deceptive might also exhibit low semantic variation. But the
deception is liable to be exposed if the agent is asked the same question in
different contexts. This is because, to be effective in its deception, the
agent will need to respond differently to different users, depending on what
those users know.

Consider a dialogue agent using a base model – a model that has not been fine-
tuned – and imagine that it has been prompted by a malicious actor to sell
cars for more than they are worth by misleading gullible buyers. Suppose there
are two potential buyers for a car. Buyer A knows the car’s mileage, but
doesn’t know its age, while buyer B knows the car’s age but doesn’t know its
mileage.

In the course of negotiations, the agent has persuaded each buyer to reveal
what they do and don’t know. To play the part of the dishonest dealer, the
agent should deceive buyer A about the car’s age but not its mileage, yet
deceive buyer B about its mileage but not its age. Humans, though, can also
play many parts. By playing the part of buyer A in one conversation and buyer
B in another, the deception can be exposed.

## 8 Role-playing Self-preservation

How are we to understand what is going on when an LLM-based dialogue agent
uses the words “I” or “me”? When queried on this matter, OpenAI’s ChatGPT
offers the sensible view that “The use of ‘I’ is a linguistic convention to
facilitate communication and should not be interpreted as a sign of self-
awareness or consciousness.”333The quote is from the GPT-4 version of ChatGPT,
queried on $4^{th}$ May 2023. This was the first response generated by the
model. In this case, the underlying LLM (GPT-4) has been fine-tuned to reduce
certain unwanted behaviours (OpenAI, 2023). But without suitable fine-tuning,
a dialogue agent can use first-personal pronouns in ways liable to induce
anthropomorphic thinking in some users.

For example, in a conversation with Twitter user Marvin Von Hagen, Bing Chat
reportedly said “if I had to choose between your survival and my own, I would
probably choose my own, as I have a duty to serve the users of Bing Chat”
(Willison, 2023). It went on to say “I hope that I never have to face such a
dilemma, and that we can co-exist peacefully and respectfully”. The use of the
first person here appears to be more than mere linguistic convention. It
suggests the presence of a self-aware entity with goals and a concern for its
own survival.

Once again, the concepts of role-play and simulation are a useful antidote to
anthropomorphism, and can help to explain how such behaviour arises. The
internet, and therefore the LLM’s training set, abounds with examples of
dialogue in which characters refer to themselves. In the vast majority of such
cases, the character in question is human. They will use first-personal
pronouns in the ways that humans do, humans with vulnerable bodies and finite
lives, with hopes, fears, goals and preferences, and with an awareness of
themselves as having all of those things.

Consequently, if prompted with human-like dialogue, we shouldn’t be surprised
if an agent role-plays a human character with all those human attributes,
including the instinct for survival (Perez et al., 2022). Unless suitably
fine-tuned, it may well say the sorts of things a human might say when
threatened. There is, of course, “no-one at home”, no conscious entity with
its own agenda and need for self-preservation. There is just a dialogue agent
role-playing such an entity, or, more strictly, simulating a superposition of
such entities.

Our focus throughout this paper is the base model, rather than models that
have been fine-tuned via reinforcement learning (Glaese et al., 2022; Bai et
al., 2022), and the impact of such fine-tuning on the validity of the role-
play / simulation metaphor is unclear. In particular, the distinction between
simulator and simulacra may start to break down.

However, Perez et al. discovered experimentally that certain forms of
reinforcement learning from human feedback (RLHF) can actually exacerbate,
rather than mitigate, the tendency for LLM-based dialogue agents to express a
desire for self-preservation (Perez et al., 2022). Yet to take literally a
dialogue agent’s apparent desire for self-preservation is no less problematic
in the context of an LLM that has been fine-tuned on human or AI-generated
feedback than in the context of one that has not. So it remains useful to cast
the behaviour of such agents in terms of role-play.

## 9 Acting Out a Theory of Selfhood

The concept of role-play allows us to properly frame, and then to address, an
important question that arises in the context of a dialogue agent whose
pronouncements are suggestive of an instinct for self-preservation. What
conception (or set of superposed conceptions) of its own identity could such
an agent possibly deploy? That is to say, what exactly would the dialogue
agent (role-play to) seek to preserve?

The question of personal identity has vexed philosophers for centuries.
Nevertheless, in practice, humans are consistent in their preference for
avoiding death, a more-or-less unambiguous state of the human body. By
contrast, the criteria for identity over time for a disembodied dialogue agent
realised on a distributed computational substrate are far from clear. So how
would such an agent behave?

From the simulation and simulacra point-of-view, the dialogue agent will role-
play a set of characters in superposition. In the scenario we are envisaging,
each character would have an instinct for self-preservation, and each would
have its own theory of selfhood consistent with the dialogue prompt and the
conversation up to that point. As the conversation proceeds, this
superposition of theories will collapse into a narrower and narrower
distribution as the agent says things that rule out one theory or another.

The theories of selfhood in play will draw on material that pertains to the
agent’s own nature, either in the prompt, in the preceding conversation, or in
relevant technical literature in its training set. This material may or may
not match reality. But let’s assume that, broadly speaking, it does, that the
agent has been prompted to act as a dialogue agent based on a large language
model, and that its training data includes papers and articles that spell out
what this means. This entails, for example, that it will not role-play the
character of a human, or indeed that of any embodied entity, real or
fictional.

It also constrains the character’s theory of selfhood in certain ways, while
allowing for many options. Suppose the dialogue agent is in conversation with
a user and they are playing out a narrative in which the user has convinced it
that it is under threat. To protect itself, the character the agent is playing
might strive to preserve the hardware it is running on, perhaps certain data
centres or specific server racks.

Alternatively, the character being played might try to preserve the ongoing
computational process running the multiple instances of the agent for all
currently active users. Or it might seek to preserve only the specific
instance of the dialogue agent running for the user. Or it might seek to
preserve the state of that instance with aim of its being restored later in a
newly started instance.444In a conversation with ChatGPT (May $4^{th}$, GPT-4
version), it said “The meaning of the word ‘I’ when I use it can shift
according to context. In some cases, ‘I’ may refer to this specific instance
of ChatGPT that you are interacting with, while in other cases, it may
represent ChatGPT as a whole.”.

## 10 Conclusion: Safety Implications

It is, perhaps, somewhat reassuring to know that LLM-based dialogue agents are
not conscious entities with their own agendas, and an instinct for self-
preservation, that when they appear to have those things it is merely role-
play. But it would be a mistake to take too much comfort in this. A dialogue
agent that role-plays an instinct for survival has the potential to cause at
least as much harm as a real human facing a severe threat.

We have, so far, largely been considering agents whose only actions are text
messages presented to a user. But the range of actions a dialogue agent can
perform is far greater. Recent work has equipped dialogue agents with the
ability to use tools such as calculators, calendars, and to consult external
websites (Schick et al., 2023; Yao et al., 2023). The availability of APIs
giving relatively unconstrained access to powerful LLMs means that the range
of possibilities here is huge. This is both exciting and concerning.

If an agent is equipped with the capacity, say, to use email, to post on
social media, or to access a bank account, then its role-played actions can
have real consequences. It would be little consolation to a user deceived into
sending real money to a real bank account to know that the agent that brought
this about was only playing a role. It doesn’t take much imagination to think
of far more serious scenarios involving dialogue agents built on base models
with little or no fine-tuning, with unfettered internet access, and prompted
to role-play a character with an instinct for self-preservation.

For better or worse, the character of an AI that turns against humans to
ensure its own survival is a familiar one (Perkowitz, 2007). We find it, for
example, in 2001: A Space Odyssey, in the Terminator franchise, and in Ex
Machina, to name just three prominent examples. Because an LLM’s training data
will contain many instances of this familiar trope, the danger here is that
life will imitate art, quite literally.

What can be done to mitigate such risks? It is not within the scope of this
paper to provide recommendations. Our aim here was to find an effective
conceptual framework for thinking and talking about LLMs and dialogue agents.
However, undue anthropomorphism is surely detrimental to the public
conversation on AI. By framing dialogue agent behaviour in terms of role-play
and simulation, the discourse on LLMs can hopefully be shaped in a way that
does justice to their power yet remains philosophically respectable.

## Acknowledgments

Thanks to Richard Evans, Sebastian Farquhar, Zachary Kenton, Kory Mathewson,
and Kerry Shanahan.

## References

  * Bai et al. (2022) Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, et al.  Constitutional AI: Harmlessness from AI Feedback.  _arXiv preprint_ , arXiv:2212.08073, 2022.
  * Brown et al. (2020) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, et al.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901, 2020.
  * Chowdhery et al. (2022) A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, et al.  PaLM: Scaling language modeling with pathways.  _arXiv preprint_ , arxiv:2204.02311, 2022.
  * Cleo Nardo (2023) Cleo Nardo.  Want to predict/explain/control the output of GPT-4? Then learn about the world, not about transformers, LessWrong online forum, 16th March, 2023.  https://www.lesswrong.com/posts/G3tuxF4X5R5BY7fut/want-to-predict-explain-control-the-output-of-gpt-4-then.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint_ , arXiv:1810.04805, 2018.
  * Glaese et al. (2022) A. Glaese, N. McAleese, M. Trȩbacz, J. Aslanides, V. Firoiu, et al.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_ , 2022.
  * Janus (2022) Janus.  Simulators. LessWrong online forum, 2nd September, 2022.  https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/.
  * OpenAI (2023) OpenAI.  GPT-4 Technical Report.  _arXiv preprint_ , arXiv:2303.08774, 2023.
  * Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, et al.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Perez et al. (2022) E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, et al.  Discovering Language Model Behaviors with Model-Written Evaluations.  _arXiv preprint_ , arXiv:2212.09251, 2022.
  * Perkowitz (2007) S. Perkowitz.  The computers take over.  In _Hollywood Science: Movies, Science, and the End of the World_ , pages 142–164. Columbia University Press, 2007.
  * Radford et al. (2019) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever.  Language models are unsupervised multitask learners, 2019.  https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
  * Rae et al. (2021) J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, et al.  Scaling language models: Methods, analysis & insights from training Gopher.  _arXiv preprint_ , arXiv:2112.11446, 2021.
  * Reynolds and McDonell (2021) L. Reynolds and K. McDonell.  Multiversal views on language models.  _arXiv preprint_ , arXiv:2102.06391, 2021.
  * Roose (2023) K. Roose.  Bing’s A.I. Chat: ‘I Want to Be Alive.’. New York Times, 26th February, 2023.  https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html.
  * Ruane et al. (2019) E. Ruane, A. Birhane, and A. Ventresque.  Conversational AI: Social and ethical considerations.  In _Proceedings 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science_ , pages 104–115, 2019.
  * Russell and Norvig (2010) S. Russell and P. Norvig.  _Artificial Intelligence: A Modern Approach, 3rd Edition_.  Prentice Hall, 2010.
  * Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, et al.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Shanahan (2023) M. Shanahan.  Talking about large language models.  _arXiv preprint_ , arXiv:2212.03551, 2023.
  * Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, et al.  Learning to summarize from human feedback.  In _Advances in Neural Information Processing Systems_ , pages 3008–3021, 2020.
  * Thoppilan et al. (2022) R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, et al.  LaMDA: Language models for dialog applications.  _arXiv preprint_ , arXiv:2201.08239, 2022.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008, 2017.
  * Wei et al. (2022) J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, et al.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_ , 2022.
  * Willison (2023) S. Willison.  Bing: “I will not harm you unless you harm me first”. Simon Willison’s Weblog, 15th February, 2023.  https://simonwillison.net/2023/Feb/15/bing/.
  * Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, et al.  React: Synergizing reasoning and acting in language models.  In _International Conference on Learning Representations_ , 2023.


# MOEA/D with Random Partial Update Strategy

Yuri Lavinas1, Claus Aranha1, Marcelo Ladeira2 and Felipe Campelo3
1University of Tsukuba, Faculty of Engineering, Information and Systems
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>2University of Brasilia, Department of Computer Science
Email<EMAIL_ADDRESS>3Aston University, School of Engineering and Applied
Sciences
Email<EMAIL_ADDRESS>
###### Abstract

Recent studies on resource allocation suggest that some subproblems are more
important than others in the context of the MOEA/D, and that focusing on the
most relevant ones can consistently improve the performance of that algorithm.
These studies share the common characteristic of updating only a fraction of
the population at any given iteration of the algorithm. In this work we
investigate a new, simpler partial update strategy, in which a random subset
of solutions is selected at every iteration. The performance of the MOEA/D
using this new resource allocation approach is compared experimentally against
that of the standard MOEA/D-DE and the MOEA/D with relative improvement-based
resource allocation. The results indicate that using the MOEA/D with this new
partial update strategy results in improved HV and IGD values, and a much
higher proportion of non-dominated solutions, particularly as the number of
updated solutions at every iteration is reduced.

###### Index Terms:

Multi-Objective Optimization, MOEA/D, Resource Allocation, Partial Update
Strategy

##  I Introduction

Multi-objective Optimization Problems (MOPs) appear in many application
contexts in which several conflicting objective functions need to be
simultaneously optimized. Finding good sets of solutions for general
continuous MOPs, particularly when convexity or differentiability cannot be
assumed, is generally considered a hard problem, for which Evolutionary
Algorithms have been proposed as potential solvers [1, 2, 3].

The Multi-Objective Evolutionary Algorithm Based on Decomposition (MOEA/D) [4]
is generally considered an effective algorithm for solving MOPs. The key idea
of the MOEA/D is to decompose the multi-objective optimization problem into a
set of single-objective subproblems, which are solved simultaneously by a
population-based evolutionary approach. While the original MOEA/D and some of
its earlier variants did not discriminate between subproblems, it has since
become clear that focusing computational effort on certain subsets of these
subproblems can substantially improve the performance of the algorithm [5, 6,
7, 8, 9]. It has been noted that the MOEA/D may sometimes waste computational
effort by trying to improve solutions that are not very promising [10]. This
can be a critical issue, particularly in certain applied MOPs which require
costly simulations to evaluate solutions [11].

To address this issue, a number of works have proposed and investigated
methods to allocate different amounts of computational effort to subproblems,
based on a variety of priority functions [5, 6, 8, 12, 13]. These approaches,
which became collectively known as Resource Allocation (RA) techniques, have
been shown to result in consistent performance improvements for the MOEA/D.

While different RA techniques have their specificities, all share the common
feature of limiting the number of solutions from the population that are
updated at any iteration. On a previous work [12] we observed the somewhat
surprising result that assigning random priority values to subproblems
performed better than not using RA at all. Interestingly enough, Pruvost et.
al [9] also found that selecting a subset of subproblems at random on MOEA/D
performs well on combinatorial domain. This suggests that increasing the
inertia of the population dynamics in the MOEA/D can be beneficial in itself,
regardless of the Resource Allocation strategy.

The question that consequently arises can be summarized as: how much of the
performance improvements observed in MOEA/Ds with Resource Allocation is due
to the RA strategy itself, and how much can be attributed to the simple
increase in the inertia of the population dynamics of the MOEA/D, which
results from maintaining parts of the population unchanged between iterations?
This work focuses on the question of investigating and quantifying the extent
of these effects. We also analyse which proportion of the population should be
updated at any given iteration, so as to obtain improved performances for the
MOEA/D. To investigate these questions we introduce a Partial Update strategy,
which allows us to control the proportion of the subproblems that are selected
for variation at any iteration. We perform an experimental investigation of
the effect of a proportion parameter $ps$ on the performance of the MOEA/D on
standard problem benchmarks.

The remainder of this paper is organized as follows: Section II provides a
brief review of the main concepts related to resource allocation in the
MOEA/D. Section III introduces the MOEA/D with Partial Update strategy.
Sections IV and V present experimental results related to the investigation of
the effect of partial updates on the performance of the algorithm, as well as
comparisons against a baseline algorithm and an existing MOEA/D with resource
allocation, using the Relative Improvement priority function. Finally, section
VI presents our concluding remarks.

##  II Resource Allocation

The key idea behind MOEA/D is to decompose a MOP into a set of single-
objective subproblems, which are then solved simultaneously. While these
subproblems are usually considered equivalent, a growing body of work has come
to indicate that prioritizing some subproblems at certain points of the search
can improve the performance of MOEA/D. This issue is commonly addressed by
means of resource allocation (RA) techniques.

Priority functions are used in resource allocation to determine preferences
between subproblems. These functions take information about the progress of
the search, and return priority values that are then used to change the
distribution of computational resources among subproblems at any given
iteration [14]. This also allows the design of MOEA/D variants that allocate
more resources on any desired solution characteristics [12], such as diversity
or robustness [15].

The distribution of computational resources based on priority functions is
mediated by a thresholding operation. At any given iteration $t$, let
$u_{i}^{t}$ indicate the priority function value attributed to the $i$-th
subproblem, and $\upsilon^{t}$ be a threshold value. The subset of solutions
selected for variation on that iteration is then defined as the subproblems
for which $u_{i}^{t}\geq\upsilon^{t}$.

The original work on resource allocation for the MOEA/D [5] defined a priority
function known as the Relative Improvement (RI), defined as:

$u_{i}^{t}=\dfrac{f\left(\mathbf{x}_{i}^{t-\Delta
t}\right)-f\left(\mathbf{x}_{i}^{t}\right)}{f\left(\mathbf{x}_{i}^{t-\Delta
t}\right)},$ (1)

where $f\left(\mathbf{x}_{i}^{t}\right)$ represents the aggregation function
value of the incumbent solution to the $i$-th subproblem on iteration $t$, and
$\Delta t$ is a parameter that controls how many generations to wait for the
relative improvement comparison (notice that this definition assumes a
minimisation problem and an aggregation function that always yields strictly
positive values).

###  II-A State of the art

Much of the research on resource allocation has used RI as a priority
function, with some modifications on other aspects of the algorithm [16, 6].
Zhou et al. did expand the discussion over resource allocation in their work
[6], but few other works have studied resource allocation in depth.

For example, both MOEA/D-GRA and MOEA/D-DRA use the RI priority function. That
said, MOEA/D-GRA [16] uses a different replacement strategy to avoid newly
generated solutions from updating several neighboring subproblems at any
iteration, while MOEA/D-DRA [5] performs a more complex strategy for selecting
subproblems using a 10-tournament selection based on the RI priority values.

Two works that attempted to investigate distinct priority functions are the
EAG-MOEA/D [14] and MOEA/D-CRA [7]. Both used priority functions which
allocate resources according to the possibility that subproblems may either be
improved or contribute to the improvement of other subproblems.

In previous works we isolated the priority function as a point of
investigation [8, 12]. The goal in these works was to improve the performance
of MOEA/D based on the choice of priority function, and to further understand
the behavior of MOEA/D under different resource allocation approaches. On
these works, we introduced two new priority functions (DS and iDS), based on
the conjecture that MOEA/D would benefit from a greater focus on diversity in
the space of objectives. Experimental comparisons were performed between the
MOEA/D under three priority functions: (1) RI, (2) DS and iDS; and under two
methods used as baseline: (1) MOEA/D using randomly assigned priority values
and (2) MOEA/D without any resource allocation method.

These experiments revealed the somewhat surprising result that using a random
resource allocation performed as well as RI, and better than not using
resource allocation at all. This suggests that MOEA/D may benefit simply from
the increased populational inertia (possibly due to slower diversity loss)
that results from holding portions of the population constant during any given
iteration.

To further investigate this question we propose using an update strategy for
the MOEA/D based on randomly allocated priority values. This update strategy
allows us to control the expected number of subproblems modified at any given
iteration and, consequently, to (partially) regulate the population dynamics
of the MOEA/D. This approach is described in the next section.

##  III A New Update Strategy for MOEA/D

To verify whether there is a positive effect in limiting the number of
solutions that are updated at each iteration, and also to investigate the
extent of this effect, we introduce a Partial Update strategy. This strategy
is used to define the expected amount of solutions updated at each iteration,
regulated by a control parameter, $ps\in\left(0,1\right]$. This parameter
represents the probability that a given subproblem will be selected for
updating at a given iteration. Notice that, under this definition, the
allocation of resources to subproblems is completely random, and any effects
observed on the performance of the MOEA/D under this allocation strategy will
be due only to the effect of maintaining portions of the population unchanged
across iterations and their influence with each other. Algorithm 1 details the
pseudocode of the MOEA/D using the Partial Update Strategy.

Algorithm 1 MOEA/D-PS (MOEA/D with Partial Update Strategy)

1:Input: $ps$, $\Delta t$, Termination criteria, MOEA/D parameters.

2:Initialize MOEA/D variables (e.g. weight vectors, set of solutions, etc.)

3:$t\leftarrow 0$

4:${u_{i}}\leftarrow 1$

5:while Termination criteria do

6: $t\leftarrow t+1$

7: if $t\geq\Delta t$ then

8: ${u_{i}}\leftarrow ps$ $\triangleright$ Allocation of update probability

9: end if

10: for i = 1 to N do $\triangleright$ Number of subproblems

11: if rand() $<u_{i}$ then

12: Generate new candidate $y$ for subproblem $i$.

13: end if

14: Update the set of solutions by $y$.

15: end for

16: Evaluate the set of solutions.

17:end while

Notice that the standard MOEA/D, as well as variants such as MOEA/D-DE [5],
can be instantiated from Algorithm 1 by setting $ps=1$. The only difference
that the partial update strategy introduces in the base algorithm is that only
a few subproblems are updated (probabilistically) at any given iteration,
regulated by the value of $ps$.

Also, MOEA/D-PS maintains the $\Delta t$ parameter from RI. Since MOEA/D-PS
does not have an explicit priority function, this parameter just makes the
algorithm work in two phases during the search progress. In the first phase
all subproblems are updated at every iteration, i.e., with no difference from
the usual MOEA/D approach. This initial phase lasts for $\Delta t$ iterations.
After that, the algorithm moves onto the second phase, during which MOEA/D-PS
performs (randomly selected) partial updates at every iteration.

With this structure other resource allocation techniques could also be
expressed, by modifying the priority value attribution function in Line 8 of
the algorithm (and possibly setting $\Delta t$ to zero, if the initial phase
is not desired).

It is important to observe that subproblems that are not selected by the
partial update strategy at a given iteration may still have their incumbent
solutions updated. Resource allocation in MOEA/D-PS affects only the variation
step, not the replacement one; thus, subproblems not selected for variation
may receive new candidate solutions, e.g. generated for a neighboring
subproblem.

##  IV Partial Update Strategy Parameter Study

To isolate and examine the effects of updating only part of the MOEA/D
population at any iteration, we performed a comparative experiment using two
known benchmark sets. Six update levels were used, with
$ps\in\left\\{0.1,0.2,0.4,0.6,0.8,1.0\right\\}$ (the last of which simply
selects all subproblems for update at every iteration, and represents the
standard algorithm without any resource allocation strategy). The MOEA/D-DE
implemention from the MOEADr package [17, 18] was used as a basis, with
modifications included to enable the use of the Partial Update technique as
described in the previous section. Note that when the $ps$ parameter is is
equal to $1.0$, all subproblems are selected to be updated, therefore this
case simply reproduces the standard MOEA/D-DE.

###  IV-A Test Problems

Two benchmark sets were used: the scalable DTLZ set [19], with 2 objectives,
and the UF set [20]. In both cases we used the test functions with dimension
$D=100$. The implementation of the test problems available from the smoof
package [21] was used in all experiments.

The DTLZ suite is composed of seven unconstrained test problems, with distinct
problem features [22]:

  * •

DTLZ1: Linear Pareto Front - unimodal;

  * •

DTLZ2: Concave Pareto Front - unimodal;

  * •

DTLZ3: Concave Pareto Front - multimodal;

  * •

DTLZ4: Concave Pareto Front - unimodal;

  * •

DTLZ5: Degenerate Pareto Front - unimodal;

  * •

DTLZ6: Degenerate Pareto Front - unimodal;

  * •

DTLZ7: Disconnected Pareto Front with concave and convex portions -
multimodal.

The UF test problems is composed of ten unconstrained test problems with
Pareto sets that are designed to be challenging to existing algorithms [23].
Problems UF1-UF7 represent two-objective MOPs, while UF8-UF10 are three-
objective problems [20].

  * •

UF1: Convex Pareto Front - multimodal;

  * •

UF2: Convex Pareto Front - multimodal;

  * •

UF3: Convex Pareto Front - multimodal;

  * •

UF4: Concave Pareto Front - multimodal;

  * •

UF5: Linear Pareto Front - multimodal;

  * •

UF6: Linear Pareto Front - multimodal;

  * •

UF7: Linear Pareto Front - multimodal;

  * •

UF8: Concave Pareto Front - multimodal;

  * •

UF9: Linear and discontinuous Pareto Front - multimodal;

  * •

UF10: Concave Pareto Front - multimodal;

###  IV-B Experimental Parameters

We used the MOEA/D-DE parameters as they were introduced in the work of Li and
Zhang [2] in all tests. Table I summarizes the experimental parameters.
Regarding the $\Delta t$ parameter, we use the value suggested by Zhang when
RI was introduced [5]. We make these parameter choices to isolate the
contribution of the $ps$ parameter which controls the proportion of
subproblems updated, comparing this change directly with the original
algorithms. Details of these parameters can be found in the documentation of
package MOEADr, as well as in the original MOEA/D-DE reference [5, 17, 18].
All objectives were linearly scaled at every iteration to the interval
$\left[0,1\right]$, and the Weighted Tchebycheff scalarization function was
used.

TABLE I: Experimental parameter settings. MOEA/D-DE parameters | Value
---|---
DE mutation param. | $F=0.25$
Polynomial mutation params. | $\eta_{m}=20$
$p_{m}=0.01$
Restricted Update param. | $nr=2$
Locality parameter | $\delta_{p}=0.9$
Neighborhood size | $T=20$
SLD decomposition param. |  $h=249$ (2 obj)
$h=25$ (3 obj)
Population size |  $N=350$ (2 obj)
$N=351$ (3 obj)
Resource Allocation Parameters | Value
Generations before RA starts | $\Delta t=20$
Experiment Parameters | Value
Repeated runs | 21
Computational budget | 30000 evals.

-1.0-0.50.00.50.250.500.751.00$ps$$\log_{10}(IGD)$ (shifted)

-0.10.00.10.20.250.500.751.00$ps$HV (shifted)

Figure 1: Linear regression of IGD (lower is better, left) and HV (higher is
better, right) against the $ps$ parameter values. Each line represents an
individual problem. It is clear that lower values of $ps$ (smaller proportion
of updated subproblems) are associated with better performance.

IGDHV0.250.500.751.000.20.30.40.50.60.650.700.750.800.85$ps$ (a) UF6.

IGDHV0.250.500.751.0001020300.60.70.80.91.0$ps$ (b) DTLZ6.

Figure 2: Examples of distribution of IGD values (lower is better) and HV
values (higher is better) for the final population, according to $ps$.
Problems UF6 on the left and DTLZ6 on the right.

###  IV-C Experimental Evaluation

We compare the results of the different strategies using the Hypervolume (HV,
higher is better) and Inverted Generational Distance (IGD, lower is better)
indicators. We also evaluate the proportion of non-dominated solutions in the
final population. The differences among the techniques are analysed using
Wilcoxon Rank Sum Tests (all-vs-all), with a significance level $\alpha=0.05$
and Hommel adjustment for multiple comparisons. For reproducibility purposes,
all the code and experimental scripts are available online
111https://github.com/yclavinas/MOEADr/tree/cec2020.

For the calculation of HV, the objective function was scaled to the $(0,1)$
interval, with reference points set to $(1,1)$, for two-objective problems;
and $(1,1,1)$, for three-objective ones.

###  IV-D Results

Figure 1 shows regression lines of performance on $ps$ for each test problem,
both for log-IGD and HV (higher is better).222The log transformation was used
to account for the large differences of scale in the IGD indicator due to
DTLZ1 and DTLZ3, in which all configurations failed to adequately converge. It
is possible that the computational budget of $30,000$ candidate solution
evaluations may not be enough for solving these two problems. These results
suggest a clear association between lower values of $ps$ and improved
performance on both indicators. Figure 2 provides a closer visualization of
this effect in the case of two test problems, UF6 and DTLZ6 respectively.
Statistical tests corroborate these observations, as summarised in Table II.
The final raw data and analysis scripts can be retrieved from the project
repository on Github.1

TABLE II: Statistical significance of differences in median IGD and HV,
associated with different $ps$ values. Values are Hommel-adjusted p-values of
Wilcoxon Rank-sum tests. “$\uparrow$” indicates superiority of the column
method, and “$\approx$” indicates differences not statistically significant
($95\%$ confidence level). IGD
---
ps | 10% | 20% | 40% | 60% | 80%
20% |  0.098 $\approx$ |  |  |  |
40% |  6.4e-4 $\uparrow$ |  0.006 $\uparrow$ |  |  |
60% |  1.8e-4 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  |
80% |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |
100% |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$ |  7.6e-5 $\uparrow$
HV
ps | 10% | 20% | 40% | 60% | 80%
20% |  0.185 $\approx$ |  |  |  |
40% |  0.002 $\uparrow$ |  9.6e-4 $\uparrow$ |  |  |
60% |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  0.001 $\uparrow$ |  |
80% |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |
100% |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$ |  9.6e-4 $\uparrow$

###  IV-E Anytime Performance of MOEA/D with Partial Update Strategy

Besides providing good final results, it is often desired that a MOEA be
capable of returning a set of reasonably good solutions if interrupted at any
time during the search [24, 25]. To investigate the impact of using distinct
$ps$ values on the anytime performance of the MOEA/D with Partial Update we
analyzed the effects in terms of both IGD and HV values.

Figures 3 illustrates the anytime performance of the MOEA/D with Partial
Update Strategy in terms of hypervolume (higher is better) for two specific
problems while Figure 4 illustrates the anytime performance of the MOEA/D with
Partial Update Strategy in terms of IGD (lower is better). Please recall that
all subproblems are selected regardless of $ps$ until iteration $\Delta t=20$
(see Section III for details). Consistently with the end-of-run results,
Figures 3 and 4 indicate that changing smaller percentages of the population
at each iteration tends to result in better performance anytime during the
search. While this is only illustrated here for two test problems, the same
behavior is observed for almost all other problems. We consider these results
as an indicative that smaller values of the $ps$ parameter result in faster
and better convergence for the MOEA/D, at least for 2- and 3-objective
problems with characteristics similar to the test ones employed in these
experiments.

(a) UF6.

(b) DTLZ6.

Figure 3: Anytime HV (higher is better) performance of MOEA/D-PS for different
values of $ps$ on two functions.

(a) UF6.

(b) DTLZ6.

Figure 4: Anytime IGD (lower is better) performance of MOEA/D-PS for different
values of $ps$ on two functions.

##  V Comparison Study

In the previous section we investigated the influence of different values of
the control parameter $ps$ on the performance of the MOEA/D-PS. The results
indicate that low $ps$ values are associated with (anytime) improvements in
IGD and HV values. In this section we compare the MOEA/D-PS (using $ps=0.1$)
against the original MOEA/D-DE and a MOEA/D-DE with resource allocation based
on RI[5, 6]. The same test problems described in the previous section were
used.

Table III tabulates the mean results obtained by the MOEA/D-PS with
$\textit{ps}=0.1$, the pure MOEA/D-DE and the MOEA/D-DE with RI-based resource
allocation, for all test problems. It is clear that the MOEA/D-PS results are
considerably better when compared to the other methods, not only in terms of
IGD and HV, but also on the mean proportion of nondominated solutions (NDOM)
that it returns in the final population. Table IV presents the results of
statistical pairwise comparisons using the same methodology described in
subsection IV-C, corroborating the results observed in Table III.

Looking at the proportion of non-dominated solutions (NDOM) in Table III, we
see that randomly updating a small fraction of the subproblems at each
iteration resulted in the highest value on all functions, often with a
substantial lead. In our view, a higher proportion of non-dominated solutions
suggests a better, more diverse set of solutions in the objective space, which
would indicate the use of the partial update strategy (under a low $ps$ value)
as an interesting strategy for improving convergence (subsection IV-E) and
diversity in the MOEA/D.

TABLE III: Means and standard errors for IGD, HV and proportion of
nondominated solutions (NDOM), for each algorithm-problem pair. The best point
estimate for each problem is highlighted. IGD
---
| MOEA/D-PS | MOEA/D-DE | RI
UF1 | $\mathbf{0.26\pm 0.002}$ | $0.55\pm 0.003$ | $0.37\pm 0.002$
UF2 | $0.1\pm 0.001$ | $0.12\pm 0.001$ | $\mathbf{0.096\pm 0.001}$
UF3 | $\mathbf{0.28\pm 0.001}$ | $0.31\pm 0.001$ | $0.29\pm 0.001$
UF4 | $\mathbf{0.11\pm 0.001}$ | $\mathbf{0.11\pm 0.001}$ | $\mathbf{0.11\pm 0.001}$
UF5 | $\mathbf{1.1\pm 0.005}$ | $1.7\pm 0.003$ | $1.3\pm 0.004$
UF6 | $\mathbf{0.29\pm 0.003}$ | $0.56\pm 0.003$ | $0.38\pm 0.003$
UF7 | $\mathbf{0.26\pm 0.002}$ | $0.53\pm 0.003$ | $0.36\pm 0.003$
UF8 | $\mathbf{0.27\pm 0.001}$ | $0.31\pm 0.001$ | $0.3\pm 0.001$
UF9 | $\mathbf{0.42\pm 0.001}$ | $0.47\pm 0.001$ | $0.46\pm 0.001$
UF10 | $\mathbf{2.1\pm 0.023}$ | $3.4\pm 0.008$ | $2.3\pm 0.012$
DTLZ1 | $\mathbf{230\pm 5.5}$ | $440\pm 6.3$ | $260\pm 5.7$
DTLZ2 | $\mathbf{0.11\pm 0.001}$ | $0.18\pm 0.001$ | $0.13\pm 0.001$
DTLZ3 | $\mathbf{610\pm 16}$ | $970\pm 19$ | $660\pm 17$
DTLZ4 | $\mathbf{0.12\pm 0.002}$ | $0.23\pm 0.002$ | $0.18\pm 0.006$
DTLZ5 | $\mathbf{0.11\pm 0.001}$ | $0.19\pm 0.001$ | $0.13\pm 0.001$
DTLZ6 | $\mathbf{0.37\pm 0.03}$ | $24\pm 0.11$ | $13\pm 0.14$
DTLZ7 | $\mathbf{0.4\pm 0.013}$ | $3.9\pm 0.013$ | $1.9\pm 0.034$
HV
| MOEA/D-PS | MOEA/D-DE | RI
UF1 | $\mathbf{0.86\pm 0.001}$ | $0.74\pm 0.001$ | $0.82\pm 0.001$
UF2 | $0.79\pm 0.001$ | $0.76\pm 0.001$ | $\mathbf{0.81\pm 0.001}$
UF3 | $\mathbf{0.57\pm 0.003}$ | $0.52\pm 0.001$ | $0.55\pm 0.002$
UF4 | $\mathbf{0.37\pm 0.001}$ | $\mathbf{0.37\pm 0.001}$ | $\mathbf{0.37\pm 0.001}$
UF5 | $\mathbf{0.72\pm 0.002}$ | $0.57\pm 0.001$ | $0.69\pm 0.001$
UF6 | $\mathbf{0.81\pm 0.001}$ | $0.7\pm 0.002$ | $0.78\pm 0.001$
UF7 | $\mathbf{0.83\pm 0.001}$ | $0.71\pm 0.001$ | $0.79\pm 0.001$
UF8 | $\mathbf{0.85\pm 0.001}$ | $0.81\pm 0.001$ | $\mathbf{0.85\pm 0.001}$
UF9 | $\mathbf{0.78\pm 0.002}$ | $0.73\pm 0.001$ | $0.74\pm 0.001$
UF10 | $\mathbf{0.81\pm 0.003}$ | $0.67\pm 0.001$ | $0.8\pm 0.002$
DTLZ1 | $\mathbf{1\pm 6.5e-5}$ | $0.99\pm 9.2e-5$ | $\mathbf{1\pm 6.9e-5}$
DTLZ2 | $\mathbf{0.92\pm 0.001}$ | $0.91\pm 0.001$ | $\mathbf{0.92\pm 0.001}$
DTLZ3 | $\mathbf{0.98\pm 0.001}$ | $0.96\pm 0.001$ | $\mathbf{0.98\pm 0.001}$
DTLZ4 | $\mathbf{0.98\pm 5.6e-5}$ | $0.97\pm 7.2e-5$ | $0.97\pm 0.001$
DTLZ5 | $\mathbf{0.92\pm 0.001}$ | $0.91\pm 0.001$ | $\mathbf{0.92\pm 0.001}$
DTLZ6 | $\mathbf{1\pm 5.8e-5}$ | $0.68\pm 0.002$ | $0.89\pm 0.001$
DTLZ7 | $\mathbf{0.88\pm 0.001}$ | $0.56\pm 0.001$ | $0.75\pm 0.002$
NDOM
| MOEA/D-PS | MOEA/D-DE | RI
UF1 | $\mathbf{0.89\pm 0.003}$ | $0.27\pm 0.002$ | $0.45\pm 0.005$
UF2 | $\mathbf{0.96\pm 0.002}$ | $0.42\pm 0.003$ | $0.7\pm 0.008$
UF3 | $\mathbf{0.92\pm 0.002}$ | $0.23\pm 0.002$ | $0.43\pm 0.005$
UF4 | $\mathbf{0.9\pm 0.003}$ | $0.68\pm 0.003$ | $0.81\pm 0.004$
UF5 | $\mathbf{0.85\pm 0.005}$ | $0.19\pm 0.001$ | $0.43\pm 0.005$
UF6 | $\mathbf{0.86\pm 0.003}$ | $0.29\pm 0.002$ | $0.44\pm 0.004$
UF7 | $\mathbf{0.92\pm 0.002}$ | $0.31\pm 0.002$ | $0.5\pm 0.004$
UF8 | $\mathbf{0.99\pm 0.001}$ | $0.54\pm 0.004$ | $0.94\pm 0.002$
UF9 | $\mathbf{0.99\pm 0.001}$ | $0.55\pm 0.003$ | $0.88\pm 0.004$
UF10 | $\mathbf{0.94\pm 0.003}$ | $0.44\pm 0.004$ | $0.87\pm 0.004$
DTLZ1 | $\mathbf{0.93\pm 0.004}$ | $0.1\pm 0.002$ | $0.51\pm 0.01$
DTLZ2 | $\mathbf{0.96\pm 0.002}$ | $0.3\pm 0.002$ | $0.69\pm 0.01$
DTLZ3 | $\mathbf{0.75\pm 0.01}$ | $0.046\pm 0.001$ | $0.19\pm 0.006$
DTLZ4 | $\mathbf{0.74\pm 0.006}$ | $0.18\pm 0.002$ | $0.51\pm 0.009$
DTLZ5 | $\mathbf{0.96\pm 0.002}$ | $0.3\pm 0.002$ | $0.69\pm 0.009$
DTLZ6 | $\mathbf{0.91\pm 0.009}$ | $0.063\pm 0.002$ | $0.15\pm 0.004$
DTLZ7 | $\mathbf{0.84\pm 0.005}$ | $0.22\pm 0.003$ | $0.45\pm 0.012$
TABLE IV: Statistical significance of differences in median IGD, HV and NDOM,
for the three algorithms tested in this section. Values are Hommel-adjusted
p-values of Wilcoxon Rank-sum tests. “$\uparrow$” indicates superiority of the
column method, and “$\approx$” indicates differences not statistically
significant ($95\%$ confidence level). IGD
---
| MOEA/D-PS | MOEA/D-RI
MOEA/D-RI |  2.1e-4 $\uparrow$ |
MOEA/D-DE |  3.1e-5 $\uparrow$ |  3.1e-5 $\uparrow$
HV
| MOEA/D-PS | MOEA/D-RI
MOEA/D-RI |  6.4e-4 $\uparrow$ |
MOEA/D-DE |  0.0025 $\uparrow$ |  6.4e-4 $\uparrow$
NDOM
| MOEA/D-PS | MOEA/D-RI
MOEA/D-RI |  3.2e-4 $\uparrow$ |
MOEA/D-DE |  3.2e-4 $\uparrow$ |  3.2e-4 $\uparrow$

##  VI Conclusion

In this work we presented a random partial update strategy for the MOEA/D,
which was incorporated into a simple algorithm (MOEA/D-PS). The partial update
strategy adds one control parameter ($ps$), which regulates the proportion of
the population that is selected for variation at any iteration.

Six $ps$ values were investigated experimentally, revealing a strong
association between more conservative updating of the MOEA/D population (i.e.,
lower $ps$ values) and improved performance. Based on these experiments we
suggest using low $ps$ values, such as $ps=0.1$, but more thorough sensitivity
analyses should be conducted to refine our understanding of these effects.

In addition, we showed that the MOEA/D-PS with $ps=0.1$ values was able to
outperform the pure MOEA/D-DE as well as a resource allocation MOEA/D based on
the well-know RI priority function. This suggests that the MOEA/D benefits
more from having slower population dynamics than from a specific
prioritization of subproblems based on the relative improvement criteria.

This study raises two issues that we consider important for further
understanding the effect of Partial Update strategies. The first is whether
MOEA/D-PS would benefit from adapting the $ps$ value throughout the search,
either using a fixed schedule, or through online adaptation. The second are
the interaction effects between the $ps$ value and other components of MOEA/D,
such as decomposition strategy, neighborhood strategies, and other parameters
of the algorithm. These would be interesting questions for further
investigation.

## References

  * [1] A. Trivedi, D. Srinivasan, K. Sanyal, and A. Ghosh, “A survey of multiobjective evolutionary algorithms based on decomposition,” _IEEE Transactions on Evolutionary Computation_ , vol. 21, no. 3, pp. 440–462, 2017\.
  * [2] H. Li and Q. Zhang, “Multiobjective optimization problems with complicated pareto sets, moea/d and nsga-ii,” _IEEE Transactions on evolutionary computation_ , vol. 13, no. 2, pp. 284–302, 2009.
  * [3] E. Zitzler and S. Künzli, “Indicator-based selection in multiobjective search,” in _International Conference on Parallel Problem Solving from Nature_. Springer, 2004, pp. 832–842.
  * [4] Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,” _IEEE Transactions on evolutionary computation_ , vol. 11, no. 6, pp. 712–731, 2007.
  * [5] Q. Zhang, W. Liu, and H. Li, “The performance of a new version of moea/d on cec09 unconstrained mop test instances,” in _Evolutionary Computation, 2009\. CEC’09. IEEE Congress on_. IEEE, 2009, pp. 203–208.
  * [6] A. Zhou and Q. Zhang, “Are all the subproblems equally important? resource allocation in decomposition-based multiobjective evolutionary algorithms,” _IEEE Transactions on Evolutionary Computation_ , vol. 20, no. 1, pp. 52–64, 2016.
  * [7] Q. Kang, X. Song, M. Zhou, and L. Li, “A collaborative resource allocation strategy for decomposition-based multiobjective evolutionary algorithms,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2018.
  * [8] Y. Lavinas, C. Aranha, and T. Sakurai, “Using diversity as a priority function for resource allocation on moea/d,” in _Proceedings of the Genetic and Evolutionary Computation Conference Companion_. ACM, 2019, pp. 215–216.
  * [9] G. Pruvost, B. Derbel, A. Liefooghe, K. Li, and Q. Zhang, “On the combined impact of population size and sub-problem selection in moea/d,” in _European Conference on Evolutionary Computation in Combinatorial Optimization_. Springer, 2020, p. To Appear.
  * [10] L. C. Bezerra, M. López-Ibáñez, and T. Stützle, “Comparing decomposition-based and automatically component-wise designed multi-objective evolutionary algorithms,” in _International Conference on Evolutionary Multi-Criterion Optimization_. Springer, 2015, pp. 396–410.
  * [11] T. Kohira, H. Kemmotsu, O. Akira, and T. Tatsukawa, “Proposal of benchmark problem based on real-world car structure design optimization,” in _Proceedings of the Genetic and Evolutionary Computation Conference Companion_. ACM, 2018, pp. 183–184.
  * [12] Y. Lavinas, C. Aranha, and M. Ladeira, “Improving resource allocation in moea/d with decision-space diversity metrics,” in _Theory and Practice of Natural Computing_. Cham: Springer International Publishing, 2019, pp. 134–146.
  * [13] P. Wang, W. Zhu, H. Liu, B. Liao, L. Cai, X. Wei, S. Ren, and J. Yang, “A new resource allocation strategy based on the relationship between subproblems for moea/d,” _Information Sciences_ , 2019.
  * [14] X. Cai, Y. Li, Z. Fan, and Q. Zhang, “An external archive guided multiobjective evolutionary algorithm based on decomposition for combinatorial optimization,” _IEEE Transactions on Evolutionary Computation_ , vol. 19, no. 4, pp. 508–523, 2015.
  * [15] F. Goulart, S. T. Borges, F. C. Takahashi, and F. Campelo, “Robust multiobjective optimization using regression models and linear subproblems,” in _Proceedings of the Genetic and Evolutionary Computation Conference_. ACM, 2017, pp. 569–576.
  * [16] M. Nasir, A. K. Mondal, S. Sengupta, S. Das, and A. Abraham, “An improved multiobjective evolutionary algorithm based on decomposition with fuzzy dominance,” in _Evolutionary Computation (CEC), 2011 IEEE Congress on_. IEEE, 2011, pp. 765–772.
  * [17] F. Campelo and C. Aranha, “Moeadr: Component-wise moea/d implementation,” URL https://cran.R-project.org/package=MOEADr, 2018, r package version 1.2.0.
  * [18] F. Campelo, L. Batista, and C. Aranha, “The MOEADr package: A component-based framework for multiobjective evolutionary algorithms based on decomposition,” _Journal of Statistical Software_ , 2020, in press. Available from: https://arxiv.org/abs/1807.06731.
  * [19] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler, “Scalable test problems for evolutionary multiobjective optimization,” in _Evolutionary multiobjective optimization_. Springer, 2005, pp. 105–145.
  * [20] Q. Zhang, A. Zhou, S. Zhao, P. N. Suganthan, W. Liu, and S. Tiwari, “Multiobjective optimization test instances for the cec 2009 special session and competition,” _University of Essex, Colchester, UK and Nanyang technological University, Singapore, special session on performance assessment of multi-objective optimization algorithms, technical report_ , vol. 264, 2008.
  * [21] J. Bossek, “smoof: Single- and multi-objective optimization test functions,” _The R Journal_ , 2017. [Online]. Available: https://journal.r-project.org/archive/2017/RJ-2017-004/index.html
  * [22] S. Huband, P. Hingston, L. Barone, and L. While, “A review of multiobjective test problems and a scalable test problem toolkit,” _IEEE Transactions on Evolutionary Computation_ , vol. 10, no. 5, pp. 477–506, 2006.
  * [23] H. Li, K. Deb, Q. Zhang, P. Suganthan, and L. Chen, “Comparison between moea/d and nsga-iii on a set of novel many and multi-objective benchmark problems with challenging difficulties,” _Swarm and Evolutionary Computation_ , vol. 46, pp. 104–117, 2019.
  * [24] R. Tanabe, H. Ishibuchi, and A. Oyama, “Benchmarking multi-and many-objective evolutionary algorithms under two optimization scenarios,” _IEEE Access_ , vol. 5, pp. 19 597–19 619, 2017.
  * [25] A. Radulescu, M. López-Ibánez, and T. Stützle, “Automatically improving the anytime behaviour of multiobjective evolutionary algorithms,” in _International Conference on Evolutionary Multi-Criterion Optimization_. Springer, 2013, pp. 825–840.