2021

1]Multimodal Digital Media Analysis Lab, IIITD

11footnotetext: These authors contributed equally to this work.

# Advancements in Scientific Controllable Text Generation Methods

Arnav Goel*<EMAIL_ADDRESS>Medha Hira*<EMAIL_ADDRESS>Avinash
Anand*<EMAIL_ADDRESS>Siddhesh Bangar*<EMAIL_ADDRESS>Dr Rajiv
Ratn Shah<EMAIL_ADDRESS>[

###### Abstract

The previous work on controllable text generation is organized using a new
schema we provide in this study. Seven components make up the schema, and each
one is crucial to the creation process. To accomplish controlled generation
for scientific literature, we describe the various modulation strategies
utilised to modulate each of the seven components. We also offer a theoretical
study and qualitative examination of these methods. This insight makes
possible new architectures based on combinations of these components. Future
research will compare these methods empirically to learn more about their
strengths and utility.

###### keywords:

Controllable text generation, Neural text generation, Natural Language
Processing, Sequence to Sequence models, Transformers

##  1 Introduction

The amount of free text on the Internet is enormous many orders of magnitude
greater than the number of labelled benchmark data sets. Modern language
models (LM) are trained on a huge scale using unsupervised Web data. When
generating samples from a language model, we have minimal influence over the
resulting text’s desired topic, style, sentiment, and other properties.
Controlled text creation means producing coherent sentences while maintaining
control over various properties. These properties encompass elements such as
style, sentiment, formality and intent; demographic aspects like gender or age
and the organization of events or information, such as the arrangement of plot
summaries. By manipulating multiple text generation properties, diverse
objectives can be accomplished. Areas of focus in the field of dialogue
response generation include the control of persona, where efforts have been
made to manipulate and shape the character or identity expressed in the
generated dialogues (zhang2018personalizing, ; li2016persona, ), controlling
various response characteristics such as politeness (niu2018polite, ),
formality, intent (gu2022controllable, ; jung2022intent, ), etc., response
grounding in fixed information (zhou2018dataset, ; dinan2018wizard, ;
ghazvininejad2018knowledge, ), and controlling topic sequence (tang2019target,
; prabhumoye2020love, ).

Figure 1: Modules Schema for the scientific controllable text generation
process.

Despite the large volume of past research on programmable text production,
there is no overarching topic or subject that embraces it all. Each study
focuses on specific tasks within particular contexts. Consequently, the
challenge lies in determining how to guide a potent unconditioned language
model in accordance with personal preferences and desired outcomes. In this
study, we provide a novel paradigm that links earlier research and sheds light
on many facets of controlled text synthesis. The schema comprises 7 modules
that span the pipeline, explain how each part affects the generating process,
and explain each technique and method that might provide better insights. We
also offer an analysis of the parallels between earlier work that concentrated
on particular schema elements we present here.

The proposed schema in this study, illustrated in Figure 1, comprises seven
modules designed to facilitate the production process. The initial module,
labeled as (1) External Inputs, serves as the starting point for the
generation process. At each generational time step $t$, the input is sourced
from the (2) Sequential Inputs module. Furthermore, this input can be
simultaneously shared and trained with the (3) Discriminator module to acquire
valuable feedback on the utilized inputs. However, it is important to note
that the discriminator should not be employed indefinitely during the
development of new architectures.

The (4) Encoding Operations module is responsible for performing consistent
operations or computations on each input at every time step. These
computations are then propagated forward, generating relevant outputs through
the (5) Decoding Strategies module. Subsequently, the (6) Output is projected
onto the vocabulary space in order to predict the next token. Finally, the (7)
Training Objectives module handles the necessary loss functions for training
the generator.

In summary, the proposed schema consists of interconnected modules, each
contributing to different aspects of the producing process. The flow begins
with external inputs, progresses through sequential inputs, encoding
operations, decoding strategies, and output projection, ultimately guided by
training objectives.

The contributions of the various modules as well as the techniques and methods
for controlling text production, are explained by this schema. This work
focuses on using this schema to explain controlled text production,
specifically emphasising the application of auto-regressive and uncontrollable
language models. Through this effort, new model architectures based on the
schema may be designed. This may be accomplished by selecting promising
approaches and techniques for each module before fusing them together. This
research also provides the linked work in text creation, spotlighting the
earlier work while introducing the techniques.

##  2 Related Work

The initial study on related work generation Hoang et al. hoang2010towards
predates the widespread use of neural networks. Their work focused on
summarization, specifically the development of ReWoS (Related Work
Summarization). ReWoS employed two distinct strategies for generating
summaries: General Content Summarization (GCSumm) and Specific Content
Summarization (SCSumm). As a heuristic-based system, ReWoS effectively mapped
these strategies to the topic tree structure, which served as the input for
the summarization process.

In a subsequent study, Wang et al. (wang2019controllable, ) explored the
explicit extraction of Cited Text Spans (CTS) from cited publications. These
CTSs were specific text fragments within the referenced work that were closely
related to a particular citation. In addition to utilizing a topic model, the
researchers employed a two-layered ensemble model for classifying and
extracting the CTS. They employed a greedy algorithm to select candidate
phrases, creating connected parts of the text and forming coherent sections.
This approach aimed to enhance the controllability and accuracy of the
collected and retrieved data.

Addressing the challenge of automatically generating citation texts within
academic works, Xing et al. (xing2020automatic, ) proposed a novel approach.
Due to a scarcity of training data, the researchers annotated a dataset and
trained an implicit citation extraction algorithm. They suggested the use of a
multi-source pointer-generation network with cross-attention mechanisms to
effectively tackle this issue. This method allowed for more precise and
efficient automatic generation of citation texts.

In contrast to the work by Ge et al. ge2021baco  and Chen et al.
chen2021capturing , Luu et al. luu2020explaining  aims to utilize citation
sentences as a form of partial supervision for elucidating the connections
between two scientific articles. They explore two approaches in their study.
Firstly, they fine-tune the GPT2 model on scientific texts using a specific
dataset and employ a neural network to generate concise summaries. Secondly,
they establish a direct relationship between the target papers and the
articles that both cite them and are cited by them by extracting the citation
sentences.

When it comes to constructing the related work section, Chen et al.
chen2021capturing , building upon the configuration proposed by Hu et al.
hu2014automatic , utilize the title’s keywords to identify relevant themes.
They leverage a discriminative feature graph to select pertinent sentences. To
overcome the computational complexity of the set cover problem, they introduce
a greedy approximation technique that prioritizes phrases containing the most
uncovered information at each step. This technique facilitates the efficient
construction of the related work section while ensuring the inclusion of
critical details from the available sources.

Ge et al. ge2021baco  extended the approach proposed by Xing et al.
xing2020automatic  by incorporating a graph attention network (GAT) to encode
the citation network information. They also utilized a hierarchical
bidirectional LSTM to store the citation context and the abstracts of the
cited papers.

Beltagy et al. beltagy2020longformer  introduced SCI BERT, a pre-trained
language model specifically designed for scientific text based on BERT. This
model was trained on various scientific tasks and datasets from diverse
fields. In evaluations, SCI BERT outperformed BERT-Base on several tasks,
particularly in computer science, achieving new state-of-the-art (SOTA)
results with improvements of +3.55 and +0.49 F1 scores through fine-tuning.
The performance of SCI BERT even surpassed the results of previously published
BIOBERT lee2020biobert  on biomedical tasks.

Abura et al. (abura2020automatic, ) employs a straightforward sequence-to-
sequence strategy in their research utilising traditional designs to generate
citation sentences from the title and abstract of the referenced work. For
this objective, they use Transformer, OpenNMT-py, and the Pointer-Generator
Network (PGN). The PGN incorporates copy-attention and coverage methods.

Jadika et al. (jadika2011literature, ) looked at 20 reviews of the literature
obtained from journal articles published in JASIST to establish the
fundamental knowledge and comprehension of the referenced papers, source
papers, and literature review papers. To grasp and emulate the intricacies of
human writing in literature reviews, the researchers meticulously analysed
linguistic and content attributes present in reviews extracted from esteemed
journals within the field of information science. The study aimed to gain a
deeper understanding of the stylistic and substantive aspects employed in the
scholarly discourse by subjecting these reviews to a thorough evaluation. This
endeavour sought to pave the way for the development of automated systems that
can mimic the quality and essence of human-authored literature reviews.

Galactica (taylor2022galactica, ) is an LLM capable of retaining, integrating,
and applying scientific data through its reasoning abilities. This proficiency
was developed through extensive training utilizing scientific publications,
books, databases, and other resources. Galactica performs better on various
scientific problems than the current models. Galactica’s corpus is excellent
and carefully managed, unlike other language models, which rely on a crawl-
based paradigm that is not properly vetted. Without overfitting, they could
train on it for several epochs. Upstream and downstream performance increases
with the usage of repeated tokens.

Wu et al. (wu2021towards, ), Jung et al. (jung2022intent, ) and Gu et al.
(gu2022controllable, ) improve upon the limitations in existing citation
scientific text generation systems such as those proposed by Xing et al.
(xing2020automatic, ) and Luu et al. (luu2020explaining, ). The limitations
include not accommodating the ability of authors to summarise multiple studies
into one citation sentence and the ability of the authors to control the
intent of the citation.

According to Cohan et al. (cohan-etal-2019-structural, ), the classification
of citation texts involves categorizing them into three intent classes:
”Background,” ”Method,” and ”Result.” In this context, when paper $B$ cites
paper $A$, ”Background” indicates that $B$ is utilizing $A$’s idea as a
foundational element for its own work. ”Method” refers to $B$ adopting $A$’s
methodology, such as experimentation or dataset preparation. On the other
hand, ”Result” signifies $B$’s comparison of its findings with those obtained
by $A$. These classifications are incorporated in the Sci-Cite dataset, and a
structural scaffold is employed to accomplish the categorization.

Gu et al. (gu2022controllable, ) provides the SciCCG dataset where each
citation text is between 5 and 200 words. An intent was randomly chosen from
among the three to prepare the baselines. The keywords were chosen using
KeyBERT (grootendorst2020keybert, ) and the relevant sentences using
SentenceBERT (reimers2019sentence, ). They are extraction techniques based on
BERT token embeddings to extract keywords and sentences from a text. This
baseline is then compared with the attribute suggestor’s keyword and sentence
extractor. The latter performs much better due to triple-loss while fine-
tuning it. The overall citation generator system gave very high accuracy
compared to ground truth citations when all three attributes, along with
contextual input, were fed in as input. On human evaluation, the model gave
much higher preference to other existing systems and showed appropriate usage
of the suggested keywords and sentences.

##  3 External Inputs

In this part, we go through the many methods that may be used to modify the
initialization of the generator to regulate the encoding and decoding process.
External Inputs are equivalent to Sequential Inputs in the conventional
generating method.

###  3.1 Decompose

It is possible to divide the encoder representation into many subspaces, each
representing a property that may be manipulated. The encoder representation is
divided into two parts by Liu et al. (liu2018learning, ), one of which
reflects the document’s structure and the other of which contains its semantic
content. Balchandran et al. (balachandran2020structsum, ) utilized this
approach to manage structure in abstractive summarization. With regard to the
encoder representation’s dimensions, this work splits. The technique makes the
first n dimensions of the encoder representation for the structure and the
last n dimensions to represent meaning. Additionally, Balchandra et al.
(balachandran2020structsum, ) provide quantitative and qualitative analyses of
the various document structures that may be learned using this method.

A document is represented by its abstract and title for scientific citation
text generation. The encoder representation is divided into subspaces for
different permutations with the citing and cited paper. Xing et al.
(xing2020automatic, ) decomposes the encoder representation into two different
encoders. It uses only the referencing paper’s context and the referenced
paper’s abstract to generate relevant target citations. Both are seen as
separate sequences of words encoded by the two different encoders to two
separate vectors denoting hidden states. Jung et al. (jung2022intent, )
encodes the two different documents into the same input text. It varies the
presence of tokens representing the abstract and title of the citing paper and
thus encodes for the two separately. Gu et al. (gu2022controllable, )
decomposes the encoder representation into many subspaces, adding subspace to
store the local context of the citing paper. Local context refers to the
tokens of five sentences before our target citation. Thus, encoder
representations can store local context to improve text generation at specific
target places.

###  3.2 Arithmetic and Linear Transform

Concatenating a control vector to the encoder’s output is one of the simplest
ways to regulate the generation. [Output of the encoder; control vector],
where $[a\mathchar 24635\relax\;b]$ implies concatenation, will be the
external input of the decoder. Here, the control vector would deliver a potent
signal to the generator to direct the generating process.

In Fu et al. (fu2018style, ), the encoder creates style-free representations
that are solely kept content. The encoder representation is then joined to the
style control vector to initialize the decoder. The approach described here is
commonly employed to merge information obtained from external sources with the
context of a discussion to generate coherent dialogue responses
(ghazvininejad2018knowledge, ; zhou2018dataset, ; dinan2018wizard, ). This
method is also used to add controls to consider user preferences. Gu et al.
(gu2022controllable, ) prepends the attribute tokens such as intent, keywords
and relevant sentences to encoder input before the local and global context to
guide the citation generation process. Wu et al. (wu2021towards, ) and Jung et
al. (jung2022intent, ) similarly prepend intent tokens and corresponding
control codes to ensure intent-controlled citation generation for research
papers.

###  3.3 Stochastic Changes

Variational auto-encoder, introduced by Kingma et al. (kingma2013auto, ),
allows you to extract a continuous latent variable from a Gaussian
distribution stochastically. This latent variable serves as the foundation for
the initialization of the generator. This idea is used by Bowan et al.
(bowman2015generating, ) to produce sentences from this continuous latent
representation. Only the Kullback-Leibler (KL) Divergence training goal may be
employed with this method of modifying the encoder state.

##  4 Sequential Inputs

In this part, we go through various methods for influencing the sequential
input to the decoder at each time step:

###  4.1 Arithmetic and Linear Transform

Sequential Inputs can undergo the same procedure as Arithmetic and Linear
Transforms for External Inputs. By incorporating a few additional control
vectors $s$, we can modify the input to the decoder similar to how we alter
the initialization. To achieve this, the data at each time step is
concatenated with the control vectors. During training, the generator commonly
employs the teacher forcing technique williams1989learning . At each time step
$t$, the predictor determines the word to be generated, denoted as $y$, based
on the anticipated word embedding $x$ at $t-1$. Notably, $x$ corresponds to
$y-1$. To facilitate this process, the input $x$ is concatenated with the
context variable $s$ at each time step $t$, resulting in $\hat{x}=[x\mathchar
24635\relax\;s]$.

##  5 Discriminator

This part discusses the discriminator’s function in creating controlled text.
Although the discriminator is not always required for training the
architecture, its inclusion can offer helpful feedback and enhance the
resulting text’s content.

###  5.1 External Feedback

Controlling the external input to the generator is frequently done with a
regularizer. A common external feedback approach is employing an adversarial
loss to alter the latent space. This effectively controls the encoder’s latent
space, which is then given to the generator as initialization. A multi-layer
perceptron is employed in Fu et al. (fu2018style, ) to predict the input style
labels. Similarly, Wang et al. (wang2019controllable, ) also uses adversarial
loss to regulate the latent representation for style characteristics. To
ensure that the meaning representation remains devoid of style indications,
the loss function is employed in the study conducted by Romanow et al.
(romanov2018adversarial, ). It trains a discriminator, which analyses a
representation as input and assesses if it produces the specified loss. This
technique, like the adversarial loss, employs a motivating loss to verify that
the style representation truly includes the stylistic information. The cross-
entropy loss is used by Gu et al. (gu2022controllable, ) to fine-tune SCIBERT
(beltagy2020longformer, ) to steer the process of text production depending on
purpose. Based on user-suggested keywords and phrases, triplet loss is
utilised to fine-tune and direct the text production process. This use of loss
to offer external input ensures that the model generates a citation text that
has these features and is more human-friendly than other produced texts.

##  6 Encoding Options

One step in the generator process is the choice of encoding. We will briefly
review some of the encoding choices most likely applied to developing
controlled text generation.

###  6.1 Gradient Based Search

#### Auto Prompt (shin2020autoprompt, ):

A novel technique is employed to automate the generation of ideas for various
tasks through the use of gradient-based search. This method involves creating
a prompt that incorporates task-specific inputs and trigger tokens within a
predefined template. These trigger tokens, which are shared among all inputs,
possess universal applicability.

To identify the universal trigger tokens, a gradient-guided search approach
similar to the one utilized in Wallace et al. wallace2019universal  is
adopted. By optimizing the trigger tokens with respect to the desired output
using a universal configuration, all inputs from a given dataset can
contribute to the process. Initially, each trigger token’s embedding is set to
a default value, which is subsequently adjusted to minimize the first-order
Taylor expansion of the task-specific loss at the current token embedding.

#### Prefix Tuning:

Intelligent prompt design creates effective context, which may result in the
desired completion. Li et al. (li2021prefixtuning, ) proposed the notion of
Prefix-Tuning in response to the aforementioned phenomenon. This method
includes initialising a limited set of trainable parameters at the start of an
input sequence, known as the ”prefix.” Prefix-Tuning is used to guide a
Language Model, allowing it to provide more controlled and focused outputs.

#### P-Tuning:

P-Tuning liu2021gpt  is a technique that involves training continuous prompt
embeddings, employing specific alternatives for trainable parameters and
architecture. Unlike Prefix-Tuning, P-Tuning only requires the input to
function correctly, while facing optimization challenges related to
discreteness and association.

#### Prompt Tuning:

Prompt tuning lester2021power  simplifies the concept of prefix tuning by
limiting the number of changeable tokens prepended to the input text for each
downstream operation to a maximum of $k$. This approach achieves results
similar to model fine-tuning, even for large models with billions of
parameters. This finding is noteworthy since optimizing and executing large
models during inference can be computationally expensive. Prompt tuning proves
beneficial for transfer learning when adapting to new domains with learned
task-specific parameters, outperforming fine-tuning in addressing domain shift
concerns. Additionally, the study demonstrates that prompt ensembling, which
involves combining multiple prompts for the same task, further improves
performance.

###  6.2 Recurrent Neural Networks

Recurrent neural networks (RNNs) operate sequentially, processing elements one
by one while considering previous computations. This iterative nature enables
them to maintain a form of memory, facilitating the propagation of contextual
information across the network. Although RNNs can theoretically handle
arbitrarily long sequences of data, in practice, they often struggle with
dependencies beyond a few time steps. To address this limitation, Long Short-
Term Memory (LSTM) (hochreiter1997long, ) units were introduced as a variant
of RNNs, incorporating specialized ”memory cells” in addition to the standard
units. These memory cells enable the retention of information over extended
periods, and a set of gates control the input, output, and forgetting of
information within the memory. This architectural design allows LSTMs to
capture longer-term dependencies effectively. The vanishing gradient problem
commonly encountered in RNNs is mitigated by these advancements.

Another variation of RNNs, known as gated recurrent units (GRUs), also employs
a similar foundational structure as LSTMs to capture correlations on
dynamically adapting time scales. Additionally, GRUs employ a gating mechanism
to regulate the flow of information. In contrast to traditional models that
utilize multiple memory cells and gates, GRUs achieve comparable functionality
with a reduced number of separate memory cells and gates.

Researchers such as Wen et al. (wen2015semantically, ) have made modifications
to LSTMs, introducing methods to control the generation process by
incorporating conversation act information. In various text generation tasks,
RNNs, LSTMs, and GRUs are commonly employed, as evidenced by the works of
Prabhumoye et al. (prabhumoye2020exploring, )., Rao et al. rao2018dear , See
et al. see2017get , Zhou et al. zhou2018dataset , and Fu et al. fu2018style .
Despite the utility of these models, their ability to handle long sequences
remains a challenge, often necessitating the integration of attention
mechanisms to enhance performance on the original sequence.

###  6.3 Transformer

The utilization of the Transformer model, as described by Vaswani et al.
vaswani2017attention , enables the establishment of global connections between
input and output by leveraging attention mechanisms. Within the Transformer
architecture, both the encoder and decoder components consist of multiple
layers of self-attention and fully connected layers. Specifically, the encoder
comprises N identical layers, each containing two sublayers: a self-attention
mechanism with multiple heads in the first sublayer, and a position-wise fully
connected feed-forward network in the second sublayer. Layer normalization and
residual connections are applied to each of these sublayers. Additionally, the
decoder includes an additional third sublayer that performs multi-head
attention over the output of the encoder stack.

By employing an attention mechanism as its core component, the decoder in the
Transformer model can selectively focus on any position within the input
sequence. Consequently, computations across the sequence can be parallelized,
resulting in improved efficiency. However, it is important to note that the
Transformer architecture has not been extensively explored for studying
modifications to recurrent neural network (RNN) computing units that
incorporate specialized parameters for controlling features such as style,
dialogue act, and so on.

###  6.4 Pre-trained Language Model

Newly pre-trained conditional language models, including XLNet yang2019xlnet ,
GPT radford2018improving , GPT-2 radford2019language , and GPT-3
brown2020language , have been widely utilized in text generation tasks.
Researchers have made efforts to enhance these pre-trained models for specific
controlled text generation tasks in various studies sudhakar2019transforming ;
dinan2018wizard ; urbanek2019learning . However, a recent model called
Galactica taylor2022galactica  has outperformed the more recent GPT-3 model in
technical knowledge-related evaluations, achieving a higher score of 68.2%
compared to 49.0%.

While pre-trained models often generate fluent and grammatically correct
content, adapting them for sequence-to-sequence applications like machine
translation and abstractive summarization can be challenging. One model that
excels in text generation when modified is the denoising autoencoder BART
lewis2019bart , which utilizes a sequence-to-sequence architecture. On the
other hand, T5 raffel2020exploring  treats every natural language processing
(NLP) problem as a ”text-to-text” problem, where it takes text as input and
produces new text as output, making it well-suited for controlled text
generation tasks.

Another approach for controlled language generation is the Plug and Play
Language Model (PPLM) introduced by Dathathri et al. dathathri2019plug . PPLM
combines a pre-trained language model with one or more attribute classifiers,
eliminating the need for training from scratch and allowing it to drive text
generation based on specific attributes.

Large Language Models (LLMs) are trained on a large corpus of general text.
These models have been performing extremely well on text-generation tasks. It
gives a promising performance on downstream tasks such as complex reasoning,
problem-solving, question-answering, etc. PaLM (chowdhery2022palm, ) is a 540
billion parameter, dense decoder-only transformer model. This, combined with
fine-tuning techniques like Chain-of-Thought Prompting (wei2022chain, ), shows
remarkable performance on reasoning and understanding-based text generation
tasks. LLAMA (touvron2023llama, ) is a collection of LLMs with parameters
ranging from 7 billion to 65 billion. This has been fine-tuned by techniques
to create Alpaca (taori2023alpaca, ), which is fine-tuned on 52K instruction-
following demonstrations generated in the technique called self-instruct
(wang2023selfinstruct, ). This technique uses a smaller policy Language Model
to create similar demonstrations as those in the existing dataset. This
creates a larger corpus of data for the bigger language model to be fine-tuned
with.

These large language models perform well on text generation tasks as they are
trained on a large corpus of texts using attention-based transformers, which
helps ensure fluency and diction in the output text. Additionally, these can
be fine-tuned using techniques such as prompting to show remarkable
improvements on domain-specific downstream tasks. However, these language
models often suffer problems such as bias, toxicity and hallucinations which
can severely deteriorate accuracy in certain cases.

##  7 Decoding Strategies

During training, these tactics are not employed as a loss aim. Numerous of
these goals rely on post-hoc decoding techniques. In this section, we will
discuss decoding methods such as Top k-sampling, nucleus sampling, or versions
of beam search.

#### Greedy Search:

Using a greedy search, which chooses the most likely word at each stage of the
sequence of output, is a straightforward approximation. Then the decoder’s
output is projected onto the entire vocabulary space, and we compute the
softmax probabilities for each predicted token $y_{t}$ at time step $t$. We
choose the token with the highest softmax probability, which gives us the word
with the maximum likelihood or probability at each step $t$ (jung2022intent,
). Another technique, as given by (gu2022controllable, ), tries to find
keywords by ranking candidates’ keywords’ embeddings based on their cosine
similarity with the contextual text embeddings. It uses this ranking to then
fine-tune SCI BERT on triplet loss (zhong2020extractive, ) and chooses the
most highly ranked keyword as the output. These are all greedy methods of
decoding and choosing outputs, also known as greedy decoding. These methods
have the advantage of being very quick, but the final output sequences’
quality might not be at its best.

#### Beam Search:

Greedy search solution might not output the best quality sentence because of
only giving locally optimal solutions. Hence, the beam search technique is
used. With a constrained bandwidth, it effectively performs a breadth-first
search, one token for each tree level. Beam search extends all descendants of
the best candidates at each level of the search tree and maintains a record of
the best candidates (referred to as ”beam width”) at each level. If a beam
search encounters the EOS (end-of-sentence) token, it may cease extending a
node. However, high-quality generation is not a given with maximization-based
decoding.

#### Top-K Sampling:

In their work, Fan et al. fan2018hierarchical  presented a straightforward yet
highly effective sampling technique known as Top-K sampling. This approach
distributes the probability mass exclusively among the top $K$ words predicted
to have the highest likelihood. GPT2 incorporated this sampling strategy,
which played a significant role in enhancing its performance in narrative
generation.

Figure 2: Top-K Sampling Pass 1

Figure 3: Top-K Sampling Pass 2

In Figure 3, we limit our selection pool to 4 words in both sampling phases
with $K=4$. While the first phase defines two-thirds of the entire probability
mass, the second step encompasses practically all probability mass. Figure 3
depicts the removal of unnecessary terms such as
${}^{\prime}little^{\prime},^{\prime}large^{\prime},and^{\prime}not^{\prime}$.

#### Top-P (Nucleus) Sampling:

Holtzman et al. holtzman2019curious  introduced the concept of Top-p sampling,
which differs from traditional methods by selecting words based on their
cumulative probability exceeding a threshold value, denoted as $p$, instead of
solely considering the highest probability $K$. This approach involves
applying a revised probability distribution to the selected set of words.
Consequently, the size of the word set, or the number of words within it, can
vary dynamically depending on the probability distribution associated with the
subsequent word.

Figure 4: Top-P Sampling Pass 1

Figure 5: Top-P Sampling Pass 2

Top-p sampling selects the fewest words that surpass $p=91\%$ when $p=0.91$ is
specified. The words are chosen as top-p candidates when the probability mass
surpasses the p level. In figure 5, the approach includes the five most likely
terms, but in figure 5, it only has to select the top three words to surpass
91%. It can be seen that it maintains a large range of words where the
following word is arguably less foreseeable, such as
$input=^{\prime}The^{\prime}$, and just a few words where the next word is
more predictable, such as $input=^{\prime}The^{\prime},^{\prime}car^{\prime}$.

#### Penalized Sampling:

A unique sampling technique was introduced by the CTRL study keskar2019ctrl
to address the issue of repetitions in generated text. This approach penalizes
the scores of previously generated tokens, effectively discouraging the
generation of duplicate substrings and mitigating the common failure scenario
associated with such repetitions.

#### Guided Decoding:

Traditional decoding methods rely on sampling tokens solely based on their
likelihood without considering any additional information. However,
customizing the candidate ranking score can influence the generated samples
based on specific preferences related to subject matter or attitude. By
incorporating a selected set of feature discriminators, the ranking score for
token selection at each decoding step can be personalized. These
discriminators can be designed to evaluate human preferences through
heuristics ghazvininejad2017hafez , supervised learning holtzman2018learning ,
or real-world testing using reinforcement learning li2017learning .

#### Trainable Decoding:

Gu et al. gu2017trainable  proposed a trainable greedy decoding strategy that
enables the sampling of sequences from a trained language model in order to
optimize a given objective. This method, known as Noisy Parallel Decoding
(NPAD), is based on the concept of approximation. To address potential
performance degradation, NPAD introduces unstructured noise into the hidden
states of the model and performs multiple parallel noisy decodings. Taking
this concept further, trainable greedy decoding replaces the unstructured
noise with a learnable random variable. A reinforcement learning (RL) agent is
employed to predict this random variable using context, previously decoded
tokens, and prior hidden states as inputs. In essence, this decoding approach
trains an RL actor to manipulate the model’s hidden states to achieve desired
outcomes.

In a related work, Grover et al. grover2019bias  trained a binary classifier
to distinguish between samples generated by the data distribution and the
generative model. By employing a likelihood-free importance weighting (LFIW)
technique, this classifier determines the significance weights required for
generating a new unnormalized distribution.

##  8 Output

The output of the generator module is projected into the vocabulary space to
forecast the next token during the normal generating process. There are
numerous techniques of modifying the sequential output before it is projected
into the vocabulary space at each time step $t$.

###  8.1 Attention

Attention is a widely employed mechanism that guides the generation process by
directing the focus towards the source sequence chorowski2015attention . In
the context of the generator, the attention module takes the current hidden
state and aims to identify relevant source-side information, encapsulated in a
context vector, to aid in token prediction. In the case of global attention,
the encoder’s hidden states are taken into account when computing the context
vector. However, it is important to note that this approach incurs a high
computational cost, particularly for longer source sequences such as
documents.

###  8.2 External Feedback

External feedback can be leveraged to manipulate the latent space of the
generator’s output. Adversarial loss is one such technique that affects both
the output latent space, denoted as $s$, and the external input $x$.
Logeswaran et al. logeswaran2018content  employ an adversarial loss to
encourage the generation of words that are both plausible and compatible with
desired attributes. The objective of the adversarial loss is to estimate the
distribution of sentence and attribute vector pairs $(x,s)$, where the
sentence can be either real or intentionally generated.

##  9 Training Objectives

This section explores various methods for regulating the generation process
using objective functions. At each generation step, a linear transformation is
applied to project the output into the vocabulary space. By applying a softmax
function to the transformed output and selecting the token with the highest
probability, a token from the vocabulary can be predicted. The predicted token
is then compared to the reference token using a loss function. By manipulating
the loss function, the generated text can exhibit desired control properties.

###  9.1 General Loss

Cross Entropy Loss: Every text generation mechanism uses this fundamental loss
to compare the created tokens to the reference tokens. The generator must
anticipate a token from the vocabulary at each time step. Therefore, it might
be viewed as a classification issue whereby the number of classes equals the
vocabulary size.

For intent-controlled scientific text generation, the following is used:

$L=-\frac{exp(x_{intent}(i_{true}))}{\sum_{i\in intents}{exp(x_{intent}(i))}}$

Here $i$ represents the different intents:

  * •

$i=1$ refers to ”background” when one paper summarizes the related work and
concepts of the other paper.

  * •

$i=2$ refers to ”method” when one paper uses a certain method or dataset of
the other paper.

  * •

$i=3$ refers to ”result” when one paper compares its results with those of the
other paper.

$x_{\text{intent}}(i)$ is the output obtained when the last hidden state of a
prepend token, used to connect the local and global context, is input into the
intent prediction header.

###  9.2 Prompt Tuning Loss

Prompting is a new technique for fine-tuning large language models (LLMs).
explore into the topic of graph pre-training frameworks, with the goal of
smoothly integrating pre-training with downstream tasks for graph neural
networks. Formulating a loss function based on a standardised sub-graph
similarity template is a major aspect of their technique. This allows for the
optimisation of adaptive prompts through the use of task-specific sub-graph
representations aided by prompts. The loss function is defined as follows:

$\mathcal{L}_{\text{prompt}}(p_{t})=-\sum_{(x_{i},y_{i})\in\tau_{t}}\ln(\frac{\exp{(\text{sim}(s_{(t,x_{i})},\tilde{s}_{(t,y_{i})})/\tau)}}{\sum_{c\in
Y}\exp{(\text{sim}(s_{(t,x_{i})},\tilde{s}_{(t,c)})/\tau)}})$

  * •

$\tau_{t}$ : Labelled Training Set for task $t$. Represented as a set of
$(x_{i},y_{i})$

  * •

$x_{i}$: An Instance or a Node in a Graph

  * •

$y_{i}$: a subset of Y, is the class label for corresponding $x_{i}$

  * •

$\tilde{s}_{(t,c)}$: Representation of a class prototypical subgraph for class
c.

###  9.3 Triplet Loss

The triplet loss function is used in Gu et al. (gu2022controllable, ) to fine-
tune SciBERT, a language model, with the goal of increasing keyword extraction
from text. They hoped to improve scientific text production by recommending
appropriate features by utilising the triplet loss. The anchor sample
($x_{i}$) is matched with a comparable positive sample ($x_{p}$) and a
dissimilar negative sample ($x_{n}$) in the triplet loss formulation. The goal
is to reduce the distance between the anchor and the positive samples while
increasing the distance between the anchor and the negative samples:

$\mathcal{L}_{\text{triplet}}=\sum_{i=1}^{N}\max\left(d(x_{i},x_{p})-d(x_{i},x_{n})+\alpha,0\right)$

$d(a,b)$ represents a distance metric, typically Euclidean distance or cosine
similarity.

$\alpha$ is a margin that defines the desired difference between positive and
negative samples.

The phrase within the $max$ function guarantees that the loss is only incurred
if the distance between the anchor and positive samples is higher by at least
the margin $alpha$ than the distance between the anchor and negative samples.
If this criterion is not met, the loss is zero. There are other varieties of
triplet loss, such as semi-hard or hard triplet loss, which try to pick
negative samples that are more difficult to discriminate against the anchor
sample, resulting in greater convergence and discriminative feature
embeddings.

The use of triplet loss has found widespread applications when the primary
goal is to get a succinct feature space that can successfully discriminate
between various entities, such as face recognition and person re-
identification.

###  9.4 Validation Loss

Validation loss assesses the model during the validation phase. It measures
the difference between expected and actual outputs on a separate validation
dataset. Validation loss provides insight into the model’s performance beyond
the training data by examining how well the model generalises to new, unknown
data. It aids in detecting over-fitting and under-fitting by guiding changes
in model architecture, hyperparameters, and regularisation strategies.
Monitoring validation loss aids in developing models that perform effectively
in real-world circumstances.

###  9.5 Training Loss

#### Unlikelihood Loss

Including frequent tokens, recurring tokens (or n-grams), and the continuous
updating of this set with each token generation helps maintain a pool of
negative possibilities (welleck2019neural, ). The objective is to minimize
repetition across generations, accomplished at both the individual token and
sequence levels. This approach can be applied to any task during training
alongside the primary objective of maximizing the probability of the target.

#### KL Divergence Loss

The Kullback-Leibler (KL) Divergence is a measure of dissimilarity between two
probability distributions. It quantifies the difference between distributions
$Q$ and $P$, denoted as $KL(P\|Q)$, where the notation $\|$ represents the
divergence of $P$ from $Q$. It is important to note that KL Divergence is not
symmetric, i.e., $KL(P\|Q)\neq KL(Q\|P)$.

#### Classifier Loss

The purpose of this loss is to ensure that the generated tokens possess the
desired control attributes. It is important to differentiate this loss, which
operates at the token level, from the external feedback loss that operates on
latent or hidden representations. The classifier loss employed here is
distinct from the external feedback loss utilized in the external input and
output modules.

###  9.6 Task Specific Loss

#### Strategy Loss:

A conversation strategy-based objective is used by Zhou et al.
(zhou2019augmenting, ) to create replies for negotiation assignments. Ground
truth techniques used in this assignment result in more effective
negotiations. Given the dialogue history, this loss represents the likelihood
that a certain method will be used in the subsequent speech. It directs the
generator to match certain techniques with specific answers.

#### Coverage Loss:

Text generation systems often face the challenge of producing repeated words
or phrases, particularly in tasks involving multi-sentence text generation
like abstractive document summarization. To address this issue, See et al.
see2017get  introduced a coverage loss that discourages the model from
repeatedly attending to the same locations in the source document.

#### Structure Loss:

In the context of abstractive document summarization, Li et al.
li2018improving  proposed two additional loss targets based on sentence-level
attention. These objectives, namely structural compression and structural
coverage, are specifically designed to improve summarization performance.
Structural compression aims to generate a summary sentence by consolidating
several distinct source sentences into a concise form. On the other hand,
structural coverage focuses on capturing important features from the original
text. These loss objectives leverage the structural properties of the document
summarization task, evaluating how effectively the generative model can
produce shorter and more precise summaries.

##  10 Fine Tuning Models

#### Conditional Training:

A conditional language model was trained for 2-step story generation by Fan et
al. (fan2018hierarchical, ). A narrative writing model then develops a tale
based on the sketch the model first generated. A fusion model architecture
implements the sketch conditioning method. The story writing model may
concentrate on figuring out what the first sketch generation model lacks since
the fusion model imposes a type of residual learning. (peng2018towards, )
experimented with a narrative generator LM that ended valence-conditioned.

#### RL Fine Tuning:

It has already been established that fine-tuning sequential models using RL is
successful for any arbitrary, non-differentiable reward function
(ranzato2015sequence, ). There are various drawbacks to the instructor-forcing
strategy in sequence generation that may be overcome by RL fine-tuning. During
training in instructor forcing, the model minimises the maximum-likelihood
loss at each decoding step. During testing, however, the model is expected to
create the entire sequence from the beginning. Exposure bias and cumulative
mistake can result from the mismatch between training and testing. RL fine-
tuning helps overcome these challenges by allowing the model to refine its
predictions through reinforcement learning, mitigating the discrepancies
between the training and testing phases. For example, BLEU for translation
(ranzato2015sequence, ; wu2016google, ; nguyen2017reinforcement, ); ROUGE for
summarization (ranzato2015sequence, ; paulus2017deep, ; wu2018learning, ); and
a custom measure for story generation may all be directly optimized by RL
fine-tuning (tambwekar2018controllable, ).

Li et al. (li2023guiding, ) tries to use a policy language model (LM) to fine-
tune a black-box LLM on downstream tasks using RL. ROUGE scores are used for
making the reward function. This generates better keywords and improves
performance on summarization tasks. Peng et al. (peng2023check, ) propose an
LLM-Augmenter to enhance language model (LM) performance. Acting as a plug-
and-play (PnP) module, it accesses evidence from external databases to improve
response generation. Given the impracticality of adjusting the multitude of
parameters in large LMs, PnP modules serve as a means to provide automatic
feedback and external knowledge, enhancing performance while minimizing
computational costs. The suggested approach employs RL to develop a policy
function that determines whether to query for evidence, generate a candidate
response from the LLM, or deliver the response to the user. This policy
augments model-generated responses by comparing them with evidence from the
external database. Consequently, the performance of LLMs in addressing queries
requiring external databases, such as specific historical questions, is
significantly improved.

#### RL Fine Tuning with Human Preferences:

In order to ascertain human preferences, the utilization of reward learning is
crucial. While quantitative measurement tools such as BLEU or ROUGE are
commonly used to compute the overlap of words and n-gram phrases across
sequences, they do not necessarily align with the judgments of human
evaluators regarding quality. To address this limitation, the approach of
reward learning from human input (christiano2017deep, ) provides a superior
method of aligning evaluation metrics with actual priorities. This approach
has been successfully applied in various applications, including narrative
production (yi2019towards, ) and summarization (bohm2019better, ;
ziegler2019fine, ; stiennon2020learning, ), where human input is used to train
a reward function.

Yi et al. (yi2019towards, ) collected binary human feedback in four categories
for a given dialogue pair (user utterance, system answer), evaluating whether
the system response is (1) comprehensive, (2) on topic, (3) interesting, and
(4) conducive to continuing the discussion. An evaluator is then trained to
anticipate human input and re-rank the beam search samples, thereby improving
the model or performing both tasks simultaneously. It is important to note
that in their work, supervised fine-tuning rather than reinforcement learning
(RL) fine-tuning was employed, and a discriminator loss derived from the
evaluator was utilized.

#### Prompt-Based Fine-Tuning:

Prompting is a technique that enhances model performance and accuracy on
specific downstream tasks without adjusting model weights (weng2023prompt, ).
Zero-shot prompting involves directly supplying a query to the model and
observing its response. In few-shot prompting, the model is provided with
$n-1$ query-response examples as a prompt, and in the $n^{th}$ example, only
the query is given. This makes the model follow the given query responses as a
template and generate the response similarly. Chain-of-Thought (CoT) prompting
has shown advancements in LLM performance for complex reasoning tasks
(wei2022chain, ). CoT prompting aims to generate concise sequences of logical
statements that progressively lead to problem-solving. However, preparing
human-based annotations for CoT prompting is laborious and challenging.

To address this, Shum et al. (shum2023automatic, ) propose an Augment-Prune-
Select strategy to automate the generation of CoT prompts. This approach
generates multiple chains of thoughts and prunes them based on their ability
to lead to the correct answer. Zhang et al. (zhang2022automatic, ) tries to
find patterns in mistakes made by the model and clusters them. It thus
prevents the frequency of the same type of error demonstration by the model.
Chen et al. (chen2023mixture, ) proposes Mixture of Soft Prompts (MSP). In-
context learning is used to prompt a large black-box language model. Thus,
they use the model as a data augmentation tool rather than directly predicting
the answer. The exemplars generated from the LLM train a smaller policy LM to
generate the final answers. Taori et al. & Li et al. (taori2023alpaca, ;
li2023guiding, ) use a policy LM to fine-tune a larger language model by
prompting techniques. Using LMs or LLMs to fine-tune LLMs is a well-known
technique which has improved their accuracy on various downstream tasks and
helped automate the process of creating large datasets.

#### Unlikelihood Training:

During language model training, the maximization of log-likelihood loss can
lead to an imbalanced token distribution, which cannot be adequately rectified
through decoding techniques alone. Specifically, when deterministic decoding
is employed, these models tend to generate high-frequency words while
neglecting low-frequency terms excessively. In essence, they exhibit
overconfidence in their predictions. The training objective is explicitly
modified by incorporating preferences for less desirable content through
probability training to address this issue. This approach aims to mitigate the
bias towards high-frequency words and promote more balanced and accurate
generation (welleck2019non, ).

##  11 Evaluation

An essential phase in the automated summarizing process is evaluating the
resulting text. In this part, we examine how the suggested systems operate and
how they are assessed in relation to their starting points.

#### ROUGE:

ROUGE is a widely adopted evaluation method used for assessing the quality of
automated summarization. It quantifies the level of overlap between candidate
summaries and reference summaries authored by humans. The ROUGE measures
encompass various statistics, including ROUGE-L, ROUGE-W, ROUGE-N (e.g.,
ROUGE-1, ROUGE-2), and ROUGES (lin2004rouge).

#### BLEU:

BLEU is an algorithm commonly employed for evaluating the accuracy of machine-
translated content across different natural languages. Typically, individual
translated segments or sentences are scored by comparing them against a set of
accurate reference translations. The resulting scores are then averaged to
provide an approximation of the overall translation quality. It is important
to note that BLEU evaluation does not consider grammar or higher-level
semantic aspects. The output of BLEU is a value between 0 and 1, where higher
scores indicate greater similarity between the candidate text and the
reference texts (papineni2002bleu, ).

#### F1 Scores

The F1 score is a widely used evaluation metric in machine learning for
quantifying the accuracy of a model’s predictions on a given dataset. It
combines precision and recall scores to provide a comprehensive assessment of
the model’s performance.

In the context of evaluating generated responses in the Wiki-QA dataset, the
model’s outputs are evaluated using token-level precision, recall, and F1
scores, which are calculated by comparing them to annotated answers.

#### BLEURT Scores

The BLEURT Score is another evaluation metric employed in assessing Natural
Language Generation tasks. It takes a pair of sentences as input and produces
a score that indicates the fluency of the generated text with respect to a
reference text. It also assesses whether the generated text can effectively
convey the same meaning as the reference text. The BLEURT Score has
demonstrated state-of-the-art agreement with human judgments on machine
translation benchmarks.

#### SciBERT Score

The SciBERT Score is designed specifically for evaluating the quality of
generated text in the scientific domain, considering its relevance and
coherence within that domain. This evaluation metric leverages the domain-
specific knowledge captured by the SciBERT language model
beltagy2020longformer  by comparing the generated text to a reference text or
a set of annotated reference texts. The SciBERT Score serves as a statistical
measure for evaluating the effectiveness of scientific and citation text
generation methods.

By providing an informative assessment of the generated content’s quality and
its alignment with the scientific domain, the SciBERT Score enables
researchers and developers to compare different models and drive advancements
in text generation systems tailored specifically for scientific applications.

#### METEOR

The evaluation of translation quality using the Metric for Evaluation of
Translation with Explicit Ordering (METEOR) involves the calculation of
similarity between a machine-generated translation and a reference
translation. This similarity assessment is based on comparing n-grams, which
are consecutive sequences of n words or tokens within a text. N-grams have
proven valuable in various applications, including text generation, text
analysis, and sentiment analysis.

When evaluating machine-generated answers, the METEOR metric functions by
computing a score that considers the matching of words between the generated
output and a provided reference. In cases where multiple references are
available, the generated output is independently scored against each
reference. Consequently, the pair with the highest score is selected as the
most suitable match.

#### Additional Metrics

  * •

Shum et al. (shum2023automatic, ) uses Exact Match Accuracy. This is done
after removing special characters and symbols and checking if the generated
citation text has the exact tokens as the ground truth.

  * •

Other measures often used to measure the model performance include $sentence-
BLEU$, $BERTscore$ and $COMET$.

##  12 Future Work and Recommendations

Although transformer-based language models have significantly improved
scientific text generation and other downstream tasks, they continue
encountering issues like hallucinations, bias, and inaccuracy. Furthermore,
most language models trained on broad web-based corpora (LLMs) struggle with
domain-specific tasks and cannot incorporate substantial user input for
controllability. Here, we comment on future advancements in controllable
scientific text generation and present our recommendations based on the
findings of our survey:

#### Controllable Text Generation:

Gu et al. (gu2022controllable, ) presents a comprehensive approach to multi-
system controllable text generation. They propose identifying specific
attributes that users can control to guide the text generation process
according to their requirements. These attributes include intent, keywords,
and relevant sentences. To enhance the diversity of choices, we suggest fine-
grained divisions of intent. For example, the ”method” intent can be
subdivided into using the dataset and the model described in the cited paper.
Moreover, instead of relying on relevant sentences for context, we propose
using the specific passage from the referenced paper associated with the
intent to generate target citations that provide contextual information. Fine-
tuning techniques like Reinforcement Learning from Human Feedback (RLHF) will
play an important role here. Currently, proposed systems cannot generate a
citation sentence citing multiple documents (wu2021towards, ). This can be
added as a feature that the author can control and influence through a prompt-
based system.

#### Language Models:

Large Language Models (LLMs) can perform with improved accuracy over pre-
existing systems on domain-specific tasks such as scientific text generation
by fine-tuning with those curated corpora of data sets. With the advent of
various open-source models with varying numbers of parameters, future work
will involve their quantisation to improve access and democratisation. Newer
models such as GPT-4, Llama, Alpaca, PaLM (openai2023gpt4, ; touvron2023llama,
; taori2023alpaca, ; chowdhery2022palm, ) and models specifically fine-tuned
on scientific texts should be given higher preference for designing systems
upon. Model architectures can be re-looked by opting for different encoding
and decoding strategies. Newer strategies not solely based on greedy methods
promise to make these processes more efficient and accurate.

#### Prompting:

As mentioned earlier, prompting is a valuable technique for fine-tuning large
language models to enhance performance without requiring weight updates. One
such technique, Chain of Thought Prompting (wei2022chain, ) along with other
discussed methods, helps address hallucinations and improves the model’s
reasoning abilities. Recent studies by Long et al. & Zhou et al.
(long2023large, ; zhou2022least, ) introduce novel prompting techniques,
namely Least-to-Most Prompting and Tree of Thoughts Prompting, respectively.
These techniques build upon the limitations of Chain of Thought Prompting and
further improve model performance. By treating text generation as a reasoning
task, these techniques hold promise in enhancing the controllability of
language models. In a chain-of-thoughts-like manner, keywords and relevant
sentences can be incorporated to guide the model towards generating citation
text that closely resembles the ground truth.

#### Retrieval:

Lazaridou et al. (lazaridou2022internetaugmented, ) looks at document
retrieval to augment LLMs. It proposes using web search for the same. This has
been applied in closed-book question-answering previously. Training LLMs on a
specific and fixed corpus of data can be challenging. Alternatively, employing
retrieval-augmentation methods, as examined by Soong et al.
(soong2023improving, ), within a specific domain can simplify the task. These
methods retrieve relevant context from domain-specific corpora based on user
queries. Subsequently, this extracted information is used as context to seed
the LLM, constraining the model’s responses to the retrieved text. This
innovative technique shows promise in improving model performance and reducing
hallucinations by limiting the model’s domain space without incurring the time
and resource expenses associated with training LLMs.

##  13 Conclusion

The previous work on controllable text generation is organised using a new
schema we provide in this study. Seven components make up the schema, and each
one is crucial to the creation process. To accomplish controlled generation
for scientific literature, we describe the various modulation strategies
utilised to modulate each of the seven components. We also offer a theoretical
study and qualitative examination of these methods. This insight makes new
architectures based on combinations of these components possible. Future
research will compare these methods empirically to learn more about their
strengths and utility.

## References

  *   * (1) Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243 (2018)
  * (2) Li, J., Galley, M., Brockett, C., Spithourakis, G.P., Gao, J., Dolan, B.: A persona-based neural conversation model. arXiv preprint arXiv:1603.06155 (2016)
  * (3) Niu, T., Bansal, M.: Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics 6, 373–389 (2018)
  * (4) Gu, N., Hahnloser, R.H.: Controllable citation text generation. arXiv preprint arXiv:2211.07066 (2022)
  * (5) Jung, S.-Y., Lin, T.-H., Liao, C.-H., Yuan, S.-M., Sun, C.-T.: Intent-controllable citation text generation. Mathematics 10(10), 1763 (2022)
  * (6) Zhou, K., Prabhumoye, S., Black, A.W.: A dataset for document grounded conversations. arXiv preprint arXiv:1809.07358 (2018)
  * (7) Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241 (2018)
  * (8) Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W.-t., Galley, M.: A knowledge-grounded neural conversation model. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
  * (9) Tang, J., Zhao, T., Xiong, C., Liang, X., Xing, E.P., Hu, Z.: Target-guided open-domain conversation. arXiv preprint arXiv:1905.11553 (2019)
  * (10) Prabhumoye, S., Li, M., Urbanek, J., Dinan, E., Kiela, D., Weston, J., Szlam, A.: I love your chain mail! making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents. arXiv preprint arXiv:2002.02878 (2020)
  * (11) Hoang, C.D.V., Kan, M.-Y.: Towards automated related work summarization. In: Coling 2010: Posters, pp. 427–435 (2010)
  * (12) Wang, K., Hua, H., Wan, X.: Controllable unsupervised text attribute transfer via editing entangled latent representation. Advances in Neural Information Processing Systems 32 (2019)
  * (13) Xing, X., Fan, X., Wan, X.: Automatic generation of citation texts in scholarly papers: A pilot study. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6181–6190 (2020)
  * (14) Ge, Y., Dinh, L., Liu, X., Su, J., Lu, Z., Wang, A., Diesner, J.: Baco: A background knowledge-and content-based framework for citing sentence generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1466–1478 (2021)
  * (15) Chen, X., Alamro, H., Li, M., Gao, S., Zhang, X., Zhao, D., Yan, R.: Capturing relations between scientific papers: An abstractive model for related work section generation. (2021). Association for Computational Linguistics
  * (16) Luu, K., Wu, X., Koncel-Kedziorski, R., Lo, K., Cachola, I., Smith, N.A.: Explaining relationships between scientific documents. arXiv preprint arXiv:2002.00317 (2020)
  * (17) Hu, Y., Wan, X.: Automatic generation of related work sections in scientific papers: an optimization approach. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1624–1633 (2014)
  * (18) Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)
  * (19) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
  * (20) AbuRa’ed, A., Saggion, H., Shvets, A., Bravo, À.: Automatic related work section generation: experiments in scientific document abstracting. Scientometrics 125(3), 3159–3185 (2020)
  * (21) Jadika, K., Khoo, C., Na, J.: Literature review writing: A study of information selection from cited papers. In: Asia Pacific Conference Library & Information Education & Practice, pp. 467–477 (2011)
  * (22) Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., Stojnic, R.: Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022)
  * (23) Wu, J.-Y., Shieh, A.T.-W., Hsu, S.-J., Chen, Y.-N.: Towards generating citation sentences for multiple references with intent control. arXiv preprint arXiv:2112.01332 (2021)
  * (24) Cohan, A., Ammar, W., van Zuylen, M., Cady, F.: Structural scaffolds for citation intent classification in scientific publications. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3586–3596. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1361. https://aclanthology.org/N19-1361
  * (25) Grootendorst, M.: Keybert: Minimal keyword extraction with bert. Zenodo (2020)
  * (26) Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)
  * (27) Liu, Y., Lapata, M.: Learning structured text representations. Transactions of the Association for Computational Linguistics 6, 63–75 (2018)
  * (28) Balachandran, V., Pagnoni, A., Lee, J.Y., Rajagopal, D., Carbonell, J., Tsvetkov, Y.: Structsum: Summarization via structured representations. arXiv preprint arXiv:2003.00576 (2020)
  * (29) Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: Style transfer in text: Exploration and evaluation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
  * (30) Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * (31) Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.: Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 (2015)
  * (32) Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2), 270–280 (1989)
  * (33) Romanov, A., Rumshisky, A., Rogers, A., Donahue, D.: Adversarial decomposition of text representation. arXiv preprint arXiv:1808.09042 (2018)
  * (34) Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 (2020)
  * (35) Wallace, E., Feng, S., Kandpal, N., Gardner, M., Singh, S.: Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125 (2019)
  * (36) Li, J., Lu, Y., Yang, Y., Li, D., Chen, L., Carin, L.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
  * (37) Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., Tang, J.: Gpt understands, too. arXiv preprint arXiv:2103.10385 (2021)
  * (38) Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
  * (39) Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * (40) Wen, T.-H., Gasic, M., Mrksic, N., Su, P.-H., Vandyke, D., Young, S.: Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745 (2015)
  * (41) Prabhumoye, S., Black, A.W., Salakhutdinov, R.: Exploring controllable text generation techniques. arXiv preprint arXiv:2005.01822 (2020)
  * (42) Rao, S., Tetreault, J.: Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535 (2018)
  * (43) See, A., Liu, P.J., Manning, C.D.: Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 (2017)
  * (44) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * (45) Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019)
  * (46) Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
  * (47) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * (48) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * (49) Sudhakar, A., Upadhyay, B., Maheswaran, A.: Transforming delete, retrieve, generate approach for controlled text style transfer. arXiv preprint arXiv:1908.09368 (2019)
  * (50) Urbanek, J., Fan, A., Karamcheti, S., Jain, S., Humeau, S., Dinan, E., Rocktäschel, T., Kiela, D., Szlam, A., Weston, J.: Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094 (2019)
  * (51) Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
  * (52) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)
  * (53) Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., Liu, R.: Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164 (2019)
  * (54) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A.M., Pillai, T.S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., Fiedel, N.: PaLM: Scaling Language Modeling with Pathways (2022)
  * (55) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
  * (56) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., Lample, G.: LLaMA: Open and Efficient Foundation Language Models (2023)
  * (57) Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html 3(6), 7 (2023)
  * (58) Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., Hajishirzi, H.: Self-Instruct: Aligning Language Models with Self-Generated Instructions (2023)
  * (59) Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., Huang, X.: Extractive Summarization as Text Matching (2020)
  * (60) Fan, A., Lewis, M., Dauphin, Y.: Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 (2018)
  * (61) Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)
  * (62) Keskar, N.S., McCann, B., Varshney, L.R., Xiong, C., Socher, R.: Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858 (2019)
  * (63) Ghazvininejad, M., Shi, X., Priyadarshi, J., Knight, K.: Hafez: an interactive poetry generation system. In: Proceedings of ACL 2017, System Demonstrations, pp. 43–48 (2017)
  * (64) Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., Choi, Y.: Learning to write with cooperative discriminators. arXiv preprint arXiv:1805.06087 (2018)
  * (65) Li, J., Monroe, W., Jurafsky, D.: Learning to decode for future success. arXiv preprint arXiv:1701.06549 (2017)
  * (66) Gu, J., Cho, K., Li, V.O.: Trainable greedy decoding for neural machine translation. arXiv preprint arXiv:1702.02429 (2017)
  * (67) Grover, A., Song, J., Kapoor, A., Tran, K., Agarwal, A., Horvitz, E.J., Ermon, S.: Bias correction of learned generative models using likelihood-free importance weighting. Advances in neural information processing systems 32 (2019)
  * (68) Chorowski, J.K., Bahdanau, D., Serdyuk, D., Cho, K., Bengio, Y.: Attention-based models for speech recognition. Advances in neural information processing systems 28 (2015)
  * (69) Logeswaran, L., Lee, H., Bengio, S.: Content preserving text generation with attribute controls. Advances in Neural Information Processing Systems 31 (2018)
  * (70) Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., Weston, J.: Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319 (2019)
  * (71) Zhou, Y., Tsvetkov, Y., Black, A.W., Yu, Z.: Augmenting non-collaborative dialog systems with explicit semantic and strategic dialog history. arXiv preprint arXiv:1909.13425 (2019)
  * (72) Li, W., Xiao, X., Lyu, Y., Wang, Y.: Improving neural abstractive document summarization with explicit information selection modeling. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1787–1796 (2018)
  * (73) Peng, N., Ghazvininejad, M., May, J., Knight, K.: Towards controllable story generation. In: Proceedings of the First Workshop on Storytelling, pp. 43–49 (2018)
  * (74) Ranzato, M., Chopra, S., Auli, M., Zaremba, W.: Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732 (2015)
  * (75) Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
  * (76) Nguyen, K., Daumé III, H., Boyd-Graber, J.: Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv:1707.07402 (2017)
  * (77) Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 (2017)
  * (78) Wu, Y., Hu, B.: Learning to extract coherent summary via deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
  * (79) Tambwekar, P., Dhuliawala, M., Martin, L.J., Mehta, A., Harrison, B., Riedl, M.O.: Controllable neural story plot generation via reinforcement learning. arXiv preprint arXiv:1809.10736 (2018)
  * (80) Li, Z., Peng, B., He, P., Galley, M., Gao, J., Yan, X.: Guiding large language models via directional stimulus prompting. arXiv preprint arXiv:2302.11520 (2023)
  * (81) Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al.: Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813 (2023)
  * (82) Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., Amodei, D.: Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017)
  * (83) Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., Hakkani-Tur, D.: Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015 (2019)
  * (84) Böhm, F., Gao, Y., Meyer, C.M., Shapira, O., Dagan, I., Gurevych, I.: Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214 (2019)
  * (85) Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., Irving, G.: Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019)
  * (86) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P.F.: Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33, 3008–3021 (2020)
  * (87) Weng, L.: Prompt engineering. lilianweng.github.io (2023)
  * (88) Shum, K., Diao, S., Zhang, T.: Automatic prompt augmentation and selection with chain-of-thought from labeled data. arXiv preprint arXiv:2302.12822 (2023)
  * (89) Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic Chain of Thought Prompting in Large Language Models (2022)
  * (90) Chen, D., Lee, C., Lu, Y., Rosati, D., Yu, Z.: Mixture of Soft Prompts for Controllable Data Generation (2023)
  * (91) Welleck, S., Brantley, K., Iii, H.D., Cho, K.: Non-monotonic sequential text generation. In: International Conference on Machine Learning, pp. 6716–6726 (2019). PMLR
  * (92) Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311–318 (2002)
  * (93) OpenAI: GPT-4 Technical Report (2023)
  * (94) Long, J.: Large Language Model Guided Tree-of-Thought (2023)
  * (95) Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., Chi, E.: Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 (2022)
  * (96) Lazaridou, A., Gribovskaya, E., Stokowiec, W., Grigorev, N.: Internet-augmented language models through few-shot prompting for open-domain question answering (2022)
  * (97) Soong, D., Sridhar, S., Si, H., Wagner, J.-S., Sá, A.C.C., Yu, C.Y., Karagoz, K., Guan, M., Hamadeh, H., Higgs, B.W.: Improving accuracy of gpt-3/4 results on biomedical data using a retrieval-augmented language model. arXiv preprint arXiv:2305.17116 (2023)


ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024
ea0117 Centrum Wiskunde & Informatica Amsterdam The Netherlands Utrecht
University Utrecht The Netherlands Utrecht University Utrecht The Netherlands
Centrum Wiskunde & Informatica Amsterdam The Netherlands Utrecht University
Utrecht The Netherlands

# A Negotiator’s Backup Plan: Optimal Concessions with a Reservation Value

Extended Abstract

Tamara C.P. Florijn<EMAIL_ADDRESS>,  Pınar Yolum<EMAIL_ADDRESS>and
Tim Baarslag<EMAIL_ADDRESS>
###### Abstract.

Automated negotiation is a well-known mechanism for autonomous agents to reach
agreements. To realize beneficial agreements quickly, it is key to employ a
good bidding strategy. When a negotiating agent has a good back-up plan, i.e.,
a high reservation value, failing to reach an agreement is not necessarily
disadvantageous. Thus, the agent can adopt a risk-seeking strategy, aiming for
outcomes with a higher utilities.

Accordingly, this paper develops an optimal bidding strategy called MIA-
RVelous for bilateral negotiations with private reservation values. The
proposed greedy algorithm finds the optimal bid sequence given the agent’s
beliefs about the opponent in $O(n^{2}D)$ time, with $D$ the maximum number of
rounds and $n$ the number of outcomes. The results obtained here can pave the
way to realizing effective concurrent negotiations, given that concurrent
negotiations can serve as a (probabilistic) backup plan.

###### Key words and phrases:

Reservation value; Bidding strategy; Concessions; Negotiation; One-to-one
negotiation; Optimal offers; Automated negotiation

##  1\. Introduction

Negotiation is often described as making concessions toward a mutually
agreeable outcome Peyton Young (1991). Deciding on the right concessions is a
challenge: being too stubborn can lead to non-agreements, while conceding too
fast can result in bad agreements. An important factor in choosing the right
negotiation strategy is that of a backup plan, which serves as a possible
outcome that a negotiator can safely fall back on in case the negotiation ends
in disagreement. Accordingly, this paper studies how the negotiator’s backup
plan can be used to design an optimal bidding strategy and how it influences a
negotiator’s concession behavior.

We consider the utility of this backup plan conceptually as a reservation
value Mansour and Kowalczyk (2015); Susskind (2011); Malhotra and Bazerman
(2007); Aydoğan et al. (2020, 2021); Curiel-Cabral et al. (2024), interpreted
as the value an agent receives if the negotiation fails Williams et al.
(2014); Baarslag et al. (2016a); Mohammad (2021). Various works consider how a
reservation value can be used in a negotiation. A reservation value often
functions as a lower bound of a bidder’s utility target, signifying what can
be obtained in other concurrent negotiation threads Faratin et al. (1998);
Raiffa (1982) or as a minimum acceptance threshold Mirzayi et al. (2022).
Alternatively, reservation values are used to define heuristics in a multi-
threaded negotiation context Cuihong Li et al. (2005); Williams et al. (2012);
Yavuz et al. (2020). However, these works focus mostly on heuristic strategies
and do not provide a theoretical proof of optimality.

In this work, we are interested in finding the optimal bidding curve given a
private reservation value. The most closely related work is the Greedy
Concession Algorithm (GCA) Baarslag et al. (2015), which finds the
theoretically optimal solution for time-sensitive domains with a static
acceptance model, but without a reservation value. Mohammad Mohammad (2023)
has proven GCA can handle repeating offers. They have also introduced a faster
GCA (QGCA) and extend to negotiations with a nonzero reservation value, but
without direct proof for this setting.

Our main contribution is a bidding strategy called a Marginal Improvement
Algorithm for Reservation Values (MIA-RVelous), which maximizes the marginal
improvement of expected utility in a greedy way to find the optimal bid
sequence given the reservation value. The algorithm finds an optimal sequence
for static acceptance models with a time complexity of $O(n^{2}D)$.

##  2\. Problem Setting

We focus on the bidding strategy (not the acceptance strategy or opponent
model Baarslag et al. (2014, 2016b); Razeghi et al. (2020)) of a general agent
for bilateral negotiation, where agents $A$ and $B$ take turns making bids, in
accordance with the alternating offers protocol Rubinstein (1982). If no
agreement is reached before the maximum number of rounds, i.e. a deadline $D$,
then the agent receives the reservation value rv, which is considered private
information and hence remains unknown to the opponent.

All the possible agreements together are called the negotiation domain
$\Omega=\\{\omega_{1},\omega_{2},...,\omega_{n}\\}$. Both agents have
preferences over the domain $\Omega$; we adopt the perspective of agent $B$
and examine their bidding strategy. Agent $B$ associates with
$\omega_{i}\in\Omega$ a utility between $0$ and $1$, which we denote by
$u(\omega_{i})$. $B$ has access to their own utility function, but $A$’s
utility function is unknown to $B$. Therefore, $B$ uses an opponent model,
with $a(\omega_{i})$ the acceptance probability of bid $\omega_{i}$.

######  Example 1.

Imagine Bo ($B$) would like to go for dinner together with their friend Ace
($A$), and they both have to agree on a restaurant before dinner time, To come
to the best dinner plan, Bo has to decide what bids to make and in what order.
Suppose there are three options to consider: Italian, Sushi, and Fast food. In
Table 1, Bo’s utility for each bid is shown, along with the corresponding
(estimated) probability that Ace accepts the bid. Suppose Bo only has time for
one dinner proposal out of the three options. Using Bo’s belief regarding
Ace’s acceptance probabilities, Bo’s first (and only) bid should maximize the
expected utility, so the best choice is proposing the Fast food place, which
yields an expected utility of $0.27$.

Table 1. Possible bids and their expected utilities for one bid. Bid |  $B$’s utility |  $A$’s acceptance probability |  Expected utility
---|---|---|---
Italian |  0.5 |  0.4 |  0.20
Sushi |  0.9 |  0.2 |  0.18
Fast food |  0.3 |  0.9 |  $0.27$

The example above illustrates one bid only; if we consider multiple bids
strategically, the agent (Bo) can plan multiple bids ahead and decide on the
whole bid sequence $\pi=(x_{1},x_{2},...,x_{k})$, with
$x_{1},...,x_{k}\in\Omega$ before the negotiation start given a static
acceptance model. Note that early termination is irrational without bidding
costs. $B$’s expected utility of making bids, $EU_{\textbf{rv}}$ of $\pi$, is
the expected utility of the first bid, or, if it is rejected, the expected
utility of the second bid, and so on, ending with the reservation value if no
agreement is reached:

$EU_{\textbf{rv}}(\pi)=\sum_{i=1}^{k}u_{i}\cdot
a_{i}\prod_{j=1}^{i-1}(1-a_{j})+\textbf{rv}\cdot\prod_{j=1}^{k}(1-a_{j}).$

The overall aim is to select the optimal sequence of $D$ bids
$\pi^{*}\in\Omega^{D}$ such that the expected utility $EU_{\textbf{rv}}$ is
maximized:

$\pi^{*}=\operatorname*{argmax}\limits_{\pi\in\Omega^{D}}EU_{\textbf{rv}}(\pi).$
(1)

A naive strategy to find this optimal sequence $\pi^{*}$ would be to evaluate
all sequences from $\Omega^{D}$ and pick the sequence with the highest
expected utility. As the agreement space grows larger, this quickly becomes
infeasible. Therefore, we develop a novel strategy called MIA-RVelous that
finds $\pi^{*}$ in $O(n^{2}D)$ time.

##  3\. Concession Behavior

The MIA-RVelous algorithm, a variation of Baarslag et al. (2015), is presented
in Algorithm 1. Intuitively, MIA-RVelous works by first adding the reservation
value as a special bid, and then greedily adding the bid with the best
marginal improvement to utility in expectation. MIA-RVelous selects the
optimal sequence of bids
$\pi^{*}=\operatorname*{argmax}\limits_{\pi\in\Omega^{D}}EU_{\textbf{rv}}(\pi)$
given deadline $D$. We can prove the optimality as formulated in Theorem 3. To
illustrate further, we extend our dinner example from Section 2.

Algorithm 1 Marginal Improvement Algorithm with rv (MIA-RVelous)

1:for $k\in\\{1,\dots,D\\}$ do

2:
$x_{k}\leftarrow\operatorname*{argmax}\limits_{\omega\in\Omega\backslash\\{x_{1},\dots,x_{k-1}\\}}EU_{\textbf{rv}}(\textsf{sort}(\\{x_{1},\dots,x_{k-1},\omega\\}))$

3:$\pi^{*}\leftarrow\textsf{sort}(\\{x_{1},\dots,x_{k}\\})$ //sorted by
decreasing utility

######  Example 2.

Bo and Ace negotiate about where to go for dinner. But this time, if the
negotiation fails, Bo would eat a meal from the freezer, which yields a
reservation value of $0.2$. Given the three options, what bids should Bo
optimally propose? As shown before, the Fast food place should be chosen first
without a backup plan. However, with a reservation value, it may not be
immediately clear. If a bid’s utility is lower than the reservation value, we
can safely ignore the bid. Otherwise, we can use MIA-RVelous to determine what
bid should be chosen with one available bid and a reservation value of 0.2,
and apply Algorithm 1, as shown below.

Table 2. Calculations of the expected utility of Example 1. First bid | If rejected: | Expected utility
---|---|---
Italian | rv | $0.5\cdot 0.4+(1-0.4)\cdot 0.2=0.32.$
Sushi | rv | $0.9\cdot 0.2+(1-0.2)\cdot 0.2=\framebox{0.34}.$
Fast food | rv | $0.3\cdot 0.9+(1-0.1)\cdot 0.2=0.29.$

Since the Sushi bid yields the highest expected utility, the first bid should
be Sushi, which is more risky than Fast food meaning that it has a higher
utility and a lower acceptance probability. This is an example of how the
presence of a reservation value stimulates the agent to seek the riskier
option.

##  4\. Discussion

We can show that our proposed bidding strategy, MIA-RVelous, finds the optimal
bid sequence for bilateral negotiations with a reservation value with a time
complexity of $O(n^{2}D)$.

Our work leads to further research directions. In this paper, we consider a
given reservation value, which we can extend to a scenario where the agent
receives the reservation value with a certain probability. Probabilistic
reservation values can serve their function in concurrent negotiations, as
previously proposed Cuihong Li et al. (2005): an agent can fall back on
ongoing concurrent negotiations after a negotiation thread failed, which
results in probabilistic backup plans. Therefore, an important future
direction is to extend our work on the optimal concession curve with a
reservation value to one-to-many and many-to-many negotiations. The results
obtained here can pave the way to realizing effective concurrent negotiations.

This publication is part of the Vidi project COMBINE (VI.Vidi.203.044) which
is (partly) financed by the Dutch Research Council (NWO).

## References

  * (1)
  * Aydoğan et al. (2020) Reyhan Aydoğan, Katsuhide Fujita, Tim Baarslag, Catholijn M. Jonker, and Takayuki Ito. 2020.  ANAC 2018: Repeated Multilateral Negotiation League. In _Advances in Artificial Intelligence_ _(Advances in Intelligent Systems and Computing, Vol. 1128)_ , Yukio Ohsawa, Katsutoshi Yada, Takayuki Ito, Yasufumi Takama, Eri Sato-Shimokawara, Akinori Abe, Junichiro Mori, and Naohiro Matsumura (Eds.). Springer, Cham, 77–89.  https://doi.org/10.1007/978-3-030-39878-1_8
  * Aydoğan et al. (2021) Reyhan Aydoğan, Katsuhide Fujita, Tim Baarslag, Catholijn M. Jonker, and Takayuki Ito. 2021.  ANAC 2017: Repeated Multilateral Negotiation League. In _Advances in Automated Negotiations_ _(Studies in Computational Intelligence, Vol. 905)_ , Takayuki Ito, Minjie Zhang, and Reyhan Aydoğan (Eds.). Springer, Singapore, 101–115.  https://doi.org/10.1007/978-981-15-5869-6_7
  * Baarslag et al. (2015) Tim Baarslag, Enrico H. Gerding, Reyhan Aydoğan, and M. C. Schraefel. 2015.  Optimal Negotiation Decision Functions in Time-Sensitive Domains. In _2015 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)_ , Vol. 2. IEEE, Singapore, 190–197.  https://doi.org/10.1109/WI-IAT.2015.161
  * Baarslag et al. (2016a) Tim Baarslag, Rafik Hadfi, Koen Hindriks, Takayuki Ito, and Catholijn M. Jonker. 2016a.  Optimal Non-adaptive Concession Strategies with Incomplete Information.  In _Recent Advances in Agent-based Complex Automated Negotiation_ , Naoki Fukuta, Takayuki Ito, Minjie Zhang, Katsuhide Fujita, and Valentin Robu (Eds.). Studies in Computational Intelligence, Vol. 638. Springer, Cham, 39–54.  https://doi.org/10.1007/978-3-319-30307-9_3 Series Title: Studies in Computational Intelligence.
  * Baarslag et al. (2016b) Tim Baarslag, Mark J. C. Hendrikx, Koen V. Hindriks, and Catholijn M. Jonker. 2016b.  Learning about the opponent in automated bilateral negotiation: a comprehensive survey of opponent modeling techniques.  _Autonomous Agents and Multi-Agent Systems_ 30, 5 (Sept. 2016), 849–898.  https://doi.org/10.1007/s10458-015-9309-1
  * Baarslag et al. (2014) Tim Baarslag, Koen Hindriks, Mark Hendrikx, Alexander Dirkzwager, and Catholijn M. Jonker. 2014.  Decoupling Negotiating Agents to Explore the Space of Negotiation Strategies.  In _Novel Insights in Agent-based Complex Automated Negotiation_ , Ivan Marsa-Maestre, Miguel A. Lopez-Carmona, Takayuki Ito, Minjie Zhang, Quan Bai, and Katsuhide Fujita (Eds.). Vol. 535. Springer Japan, Tokyo, 61–83.  https://doi.org/10.1007/978-4-431-54758-7_4 Series Title: Studies in Computational Intelligence.
  * Cuihong Li et al. (2005) Cuihong Li, K. Sycara, and J. Giampapa. 2005.  Dynamic Outside Options in Alternating-Offers Negotiations. In _Proceedings of the 38th Annual Hawaii International Conference on System Sciences_. IEEE, Big Island, HI, USA, 35b–35b.  https://doi.org/10.1109/HICSS.2005.212
  * Curiel-Cabral et al. (2024) Itza Tlaloc Quetzalcoatl Curiel-Cabral, Sonia Di Giannatale, and Giselle Labrador-Badía. 2024.  Risk Aversion, Reservation Utility and Bargaining Power: An Evolutionary Algorithm Approximation of Incentive Contracts.  _Computational Economics_ 63, 2 (Feb. 2024), 477–511.  https://doi.org/10.1007/s10614-022-10349-0
  * Faratin et al. (1998) Peyman Faratin, Carles Sierra, and Nick R. Jennings. 1998.  Negotiation decision functions for autonomous agents.  _Robotics and Autonomous Systems_ 24, 3-4 (Sept. 1998), 159–182.  https://doi.org/10.1016/S0921-8890(98)00029-3
  * Malhotra and Bazerman (2007) Deepak Malhotra and Max Bazerman. 2007.  _Negotiation Genius: How to Overcome Obstacles and Achieve Brilliant Results at the Bargaining Table and Beyond_.  Bantam Books, New York.
  * Mansour and Kowalczyk (2015) Khalid Mansour and Ryszard Kowalczyk. 2015.  An Approach to One-to-Many Concurrent Negotiation.  _Group Decision and Negotiation_ 24, 1 (Jan. 2015), 45–66.  https://doi.org/10.1007/s10726-014-9379-5
  * Mirzayi et al. (2022) Sahar Mirzayi, Fattaneh Taghiyareh, and Faria Nassiri-Mofakham. 2022.  An opponent-adaptive strategy to increase utility and fairness in agents’ negotiation.  _Applied Intelligence_ 52, 4 (2022), 3587–3603.  https://doi.org/10.1007/s10489-021-02638-2
  * Mohammad (2021) Yasser Mohammad. 2021.  Concurrent local negotiations with a global utility function: a greedy approach.  _Autonomous Agents and Multi-Agent Systems_ 35, 2 (June 2021), 28.  https://doi.org/10.1007/s10458-021-09512-y
  * Mohammad (2023) Yasser Mohammad. 2023.  Optimal time-based strategy for automated negotiation.  _Applied Intelligence_ 53, 6 (March 2023), 6710–6735.  https://doi.org/10.1007/s10489-022-03662-6
  * Peyton Young (1991) Hobart Peyton Young (Ed.). 1991.  _Negotiation analysis_.  University of Michigan Press.
  * Raiffa (1982) Howard Raiffa. 1982.  _The art and science of negotiation_.  Harvard University Press, Cambrigde, MA.
  * Razeghi et al. (2020) Yousef Razeghi, Ozan Yavuz, and Reyhan Aydoğan. 2020.  Deep reinforcement learning for acceptance strategy in bilateral negotiations.  _Turkish Journal of Electrical Engineering & Computer Sciences_ 28, 4 (2020), 1824–1840.  https://doi.org/10.3906/elk-1907-215
  * Rubinstein (1982) Ariel Rubinstein. 1982.  Perfect equilibrium in a bargaining model.  _Econometrica: Journal of the Econometric Society_ 50, 1 (1982), 97–109.  https://doi.org/10.2307/1912531 Publisher: JSTOR.
  * Susskind (2011) Noah G. Susskind. 2011.  Wiggle room: Rethinking reservation values in negotiation.  _Ohio State Journal on Dispute Resolution_ 26 (2011), 79–118.
  * Williams et al. (2012) Colin R. Williams, Valentin Robu, Enrico H. Gerding, and Nicholas R. Jennings. 2012.  Negotiating Concurrently with Unknown Opponents in omplex, Real-Time Domains.  In _ECAI 2012_. Frontiers in Artificial Intelligence and Applications, Vol. 242. IOS Press, Amsterdam, 834–839.  https://doi.org/10.3233/978-1-61499-098-7-834
  * Williams et al. (2014) Colin R. Williams, Valentin Robu, Enrico H. Gerding, and Nicholas R. Jennings. 2014.  An Overview of the Results and Insights from the Third Automated Negotiating Agents Competition (ANAC2012).  In _Novel Insights in Agent-based Complex Automated Negotiation_ , Ivan Marsa-Maestre, Miguel A. Lopez-Carmona, Takayuki Ito, Minjie Zhang, Quan Bai, and Katsuhide Fujita (Eds.). Studies in Computational Intelligence, Vol. 535. Springer, Tokyo, 151–162.  https://doi.org/10.1007/978-4-431-54758-7_9
  * Yavuz et al. (2020) Celal O. B. Yavuz, Çağıl Süslü, and Reyhan Aydoğan. 2020.  Taking Inventory Changes into Account While Negotiating in Supply Chain Management. In _Proceedings of the 12th International Conference on Agents and Artificial Intelligence_ , Vol. 1. SciTePress - Science and Technology Publications, Malta, 94–103.  https://doi.org/10.5220/0008976900940103