# Tailoring Self-Rationalizers with Multi-Reward Distillation

Sahana Ramnath${}^{\varheartsuit}$, Brihi Joshi${}^{\varheartsuit}$, Skyler
Hallinan♣, Ximing Lu♣, Liunian Harold Li♠
Aaron Chan${}^{\varheartsuit}$, Jack Hessel♢, Yejin Choi♣♢, Xiang
Ren${}^{\varheartsuit\diamondsuit}$
${}^{\varheartsuit}$University of Southern California ♣University of
Washington
♠University of California Los Angeles ♢Allen Institute for Artificial
Intelligence
<EMAIL_ADDRESS>

###### Abstract

Large language models (LMs) are capable of generating free-text rationales to
aid question answering. However, prior work 1) suggests that useful self-
rationalization is emergent only at significant scales (e.g., 175B parameter
GPT-3); and 2) focuses largely on downstream performance, ignoring the
semantics of the rationales themselves, e.g., are they faithful, true, and
helpful for humans? In this work, we enable small-scale LMs ($\sim$200x
smaller than GPT-3) to generate rationales that not only improve downstream
task performance, but are also more plausible, consistent, and diverse,
assessed both by automatic and human evaluation. Our method, MaRio (Multi-
rewArd RatIOnalization), is a multi-reward conditioned self-rationalization
algorithm that optimizes multiple distinct properties like plausibility,
diversity and consistency. Results on five difficult question-answering
datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC show that not
only does MaRio improve task accuracy, but it also improves the self-
rationalization quality of small LMs across the aforementioned axes better
than a supervised fine-tuning (SFT) baseline. Extensive human evaluations
confirm that MaRio rationales are preferred vs. SFT rationales, as well as
qualitative improvements in plausibility and
consistency111https://inklab.usc.edu/MaRio/.

##  1 Introduction

Figure 1: Our proposed approach, MaRio. While existing self-rationalizing
pipelines require exorbitantly large LMs that are used to primarily improve
task performance, MaRio is a small LM that is initially distilled from
rationales generated by GPT-3, following by multi-reward training that
improves its rationale quality w.r.t three properties: plausibility, diversity
and consistency. In the figure above, MaRio rationale improves over the Sft
rationale by improving over these properties, which is highlighted in red.

In recent years, there has been a surge of interest in using LMs to generate
fluent, human-like, free-text rationales that can explain their decision,
called self-rationalizing LMs (Wiegreffe et al., 2021). Early approaches in
self-rationalizing involved collecting human-written gold rationales and using
them as supervision for training LMs (Wiegreffe et al., 2021; Narang et al.,
2020). Recently, with the advent of large LMs, chain-of-thought prompting (Wei
et al., 2022; Marasovic et al., 2022; Kojima et al., 2022) has revolutionized
the landscape of self-rationalization; now, with just a few well-designed
prompts and demonstrations, LMs can generate explanations for their
predictions. The presence of rationales can make LMs both more interpretable
(Lertvittayakumjorn & Toni, 2021) and more usable (Joshi et al., 2023) from
the perspective of users.

However, prior work in self-rationalization has largely overlooked the quality
of generated rationales themselves; instead, their utility is justified by
measuring downstream task performance (Wei et al., 2022; Zelikman et al.,
2022). This is particularly problematic, as downstream models and users may
use these rationales as justifications for the predicted answer, which can
further propagate these negative quality outcomes (Atanasova et al., 2023;
Joshi et al., 2023; Hovy & Prabhumoye, 2021). Furthermore, it is observed that
rationalization comparable to human quality is only observed with at a
significant LM parameter scales ($\sim$100B or more) (Wei et al., 2022).
Despite some recent interest in using smaller LMs for rationalization (Chen et
al., 2023b), it is still unclear if smaller LMs can be used to generate
similarly high-quality rationales.

In this work, we propose MaRio, a method that focuses on tailoring small-sized
LMs ($<$ 1B parameters) to be strong rationalizers both in terms of improved
downstream performance, and in terms of desirable properties of the rationales
themselves. Instead of relying on human rationale labelling (Wiegreffe et al.,
2021), MaRio considers a setting where a small LM only has access to rewards
that measures factors underlying rationale quality, e.g. a trained LM that
judges the plausibility of a rationale and provides a numerical score. MaRio
first starts with training a small LM to self-rationalize, with the help of
GPT-3 (Brown et al., 2020) (text-davinci-003) generated rationales as initial
supervision, which are shown to be of higher quality Sun et al. (2022). It
then casts the problem into a multi-reward conditioned rationale generation
problem, where the LM is optimized to maximize quality rewards. In order to
achieve this, MaRio extends Quark proposed by Lu et al. (2022) to a multi-
reward setup, where generations from an LM are binned according reward values;
the LM learns distributions conditioned on ‘control-tokens’ corresponding to
every reward and high-quality generations can be obtained via conditioning on
the highest-reward token.

We determine that high-quality rationales should have three necessary
properties: plausibility (makes logical sense), diversity (is not repetitive)
and consistency (supports the correct answer for the instance). Generated
rationales’ rewards are assessed through automated metrics for each of the
three quality properties. We then evaluate MaRio on five question-answering
datasets, and observe that small LMs like T5-large can be effectively trained
to generate rationales that satisfy all of the quality requirements, while
_also_ leading to improvements in task performance over supervised fine-tuned
self-rationalizers (Sft). Via human evaluation, we also observe that
rationales generated by MaRio are more preferred over those generated by Sft,
across all datasets.

We note that tailoring small LMs with multiple quality rewards is a
challenging task. Some of these issues include finding high-quality, stable
rewards that can be effectively incorporated in a self-rationalizing pipeline.
We also observed that a lot of additional desirable properties in rationales
(like factuality and completeness) do not have reliable automated rewards.
Furthermore, improving task accuracy (which is the primary goal while
generating rationales for a lot of these tasks) is challenging in a multi-
reward setup, and show that adding task accuracy as an additional reward term
leads to the best configuration of MaRio. By using small LMs to generate high-
quality rationales that are also supported by human evaluations, we believe
our findings can help guide future work in efficient, real-world situated
methods in rationale generation and evaluation.

##  2 Self-Rationalization

Table 1: Sample Inputs and Outputs for Self-Rationalizing LMs. We use an I-RO
setting for all our experiments. This table shows one example each from the
training set of StrategyQA, OpenBookQA, QuaRel, NumerSense and QASC. The
rationales shown here are the ones sampled from GPT-3.

StrategyQA |  Input (I): Could someone in Tokyo take a taxi to the The Metropolitan Museum of Art?
---|---
Output (RO): The Metropolitan Museum of Art is in New York City, USA. Tokyo,
Japan is over 6,000 miles away. So the answer is no.
OpenBookQA |  Input (I): Our only star provides us with energy that is (a) temporary (b) inferior (c) expensive (d) reusable
Output (RO): The energy from the sun is renewable and reusable. So the answer
is (d).
QuaRel |  Input (I): Cutting bread with a bread knife creates a smooth cut, while cutting cake with a bread knife creates a rough cut. This means that the $\\_\\_\\_\\_\\_$ has less resistance (A) bread (B) cake
Output (RO): When something has less resistance, it is easier to cut through.
Thus, the bread has less resistance, making it easier to cut with the bread
knife. So the answer is (A).
NumerSense |  Input (I): Fungi reproduce in $<$mask$>$ ways. (A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine (L) ten
Output (RO): Fungi reproduce by sexual or asexual reproduction. So the answer
is (D).
QASC |  Input (I): Bees are necessary to (A) haploid plants (B) genetic diversity (C) spread flower seeds (D) prevent evolution (E) Reproduction (F) eat weeds (G) chew flowers (H) important habitats
Output (RO): Bees are necessary to spread flower seeds. Bees pollinate
flowers, which helps the flowers reproduce and spread their seeds. So the
answer is (C).

Throughout this work, we refer to _self-rationalizers_ as LMs that are trained
or prompted to specifically generate free-text rationales, along with their
predictions. These free-text rationales are treated as explanations for their
predictions. For the purpose of our experiments, we explore self-
rationalization on the question answering (QA) task. Specifically, given a
question, the LM must first generate a free-text rationale that explains the
LM’s reasoning process, followed by an answer to the given question. Table 1
shows examples of inputs and outputs by these self-rationalizing LMs for five
QA datasets: StrategyQA (Geva et al., 2021), QuaRel (Tafjord et al., 2019),
OpenBookQA (Mihaylov et al., 2018), NumerSense Lin et al. (2020) and QASC Khot
et al. (2020). These datasets were chosen over others which have existing
human written rationales because all of them require certain level of implicit
or logical reasoning in order to arrive at the answer. As we depict in the
examples, we follow the I-RO format (Wiegreffe et al., 2021), wherein the
input to the LM is the question, and the output is the joint generation of the
rationale and the predicted answer.

In order to determine whether these generated rationales are of good quality,
we focus on three properties that are necessary for any rationale to have,
agnostic of the task it is meant for. First, we note that a rationale should
be plausible. We define plausibility as the rationale making sense on its own
– whether it be common, logical or factual sense depending on the dataset at
hand. For example, if a rationale states ‘Cows can fly’, it is not plausible.
Next, we identify that a rationale should be diverse, where the rationale is
clean and not repetitive. Lastly, we note that a rationale should be
consistent with the gold label for the input. Consistency is important to
ensure that a rationale does not spew irrelevant information, and that it
supports the gold answer. Furthermore, we focus on consistency with respect to
the gold label, as misleading rationales are unhelpful as both LM
justifications, and for human utility (Joshi et al., 2023). We formalise these
properties as rewards in §4 and while these are necessary properties for any
rationale, we also discuss other good-to-have properties in §5.

All of these properties are agnostic of the actual prediction made by the LM.
Since our self-rationalization setup generates a rationale first, followed by
its prediction, we aim to generate rationales with good quality, which should
ideally improve the answer generated by the LM. Therefore, we focus on
improving self-rationalization along these three properties, as well as on
task accuracy. Along with the above rationale properties, we also consider
task correctness as a necessary property of rationales, that they should try
to improve over as a byproduct.

##  3 MaRio: Optimizing for Multiple Rewards

To improve an LMs’ rationalization across multiple properties, we leverage
Quark(Lu et al., 2022), a reinforcement learning-like framework effective on
tasks such as unlearning toxicity in generations. We propose Multi-rewArd
RatIOnalization (MaRio), an extension of Quark to multiple rewards
concurrently. We further propose two variants of MaRio: Canonical and
Additive, that explore different ways of using Quark for in a multi-reward
setup. Figure 2 shows a running example of MaRio. Appendix B shows a second,
more technical illustration of the same.

Figure 2: MaRio pipeline. MaRio uses rationales generated by a larger LM like
GPT-3 as initial supervision, and uses rewards corresponding to three
rationale properties: Plausibility, Diversity and Consistency, to improve
self-rationalization of smaller LMs like T5-large.

###  3.1 Quark

Quark(Lu et al., 2022) is a reinforcement learning-like algorithm that trains
LMs using unique control tokens prepended to the generated text. The Quark
algorithm works iteratively: (1) sampling the a pre-trained LM to generate
more training data, (2) scoring the generated data using a chosen reward
metric, and (3) using instance-level scores to sort and bin the data into a
fixed number of bins, each of which correspond to a unique control token.
During training, the LM learns to associate each control token with the
quality (as determined by the reward metric) of the data it is assigned to.
During inference, in order to obtain the best quality generations, Quark
samples the LM using the control token corresponding to the highest reward
measure. We provide a more detailed description of how Quark works in Appendix
B.

###  3.2 Canonical MaRio

Since Quark is designed only for single reward optimizations, what if we are
interested in improving the rationales along multiple rewards? In order to do
this, we first propose a direct extension to the Quark algorithm: instead of
assigning each instance just one unique control token (which corresponds to
one specific property), we now assign each instance multiple unique control
tokens at once; now, each control token corresponds to a different property we
want to train the LM on. We call this the Canonical MaRio. The order in which
properties are represented in this method is a design decision that we can
choose, and we expand on this further in Appendix C.

###  3.3 Additive MaRio

We now propose a step-by-step multi-reward Quark: instead of training the LM
on all the properties at the same time, we introduce them into training
pipeline additively, in a predefined order of properties. For example, say
that we have properties $P_{1},P_{2},P_{3}$, and that we want the LM to focus
on $P_{3}$ first, before moving on to $P_{1}$ and then later $P_{2}$. In this
method, we use multiple sets of control tokens as in Canonical MaRio, but, we
introduce each set of tokens into the training pipeline successively. For
example, we train the LM with only $P_{3}$ for the first $t$ steps, then we
train the LM on $P_{3}$ and $P_{1}$ from the $t$-th step to the $2t$-th step,
and then from the $2t$-th step onwards, we train the LM on $P_{3},P_{1}$ and
$P_{2}$. We call this method the Additive MaRio. Again, the order in which we
add the control tokens of different rewards to the training pipeline, and
whether each new control token is added to the left or right of the existing
control tokens are decision choices, and we expand on it further in Appendix
C.

##  4 Experiments

###  4.1 Datasets

As we mention in Section 2, we conduct experiments on five datasets:
StrategyQA (Geva et al., 2021), QuaRel (Tafjord et al., 2019), OpenBookQA
(Mihaylov et al., 2018), NumerSense Lin et al. (2020) and QASC Khot et al.
(2020). All datasets have multi-choice questions (yes/no for StrategyQA, a/b
for QuaRel, a/b/c/d for OpenBookQA, a/b/-/l for NumerSense, a/b/-/h for QASC),
and the task is to generate a rationale followed by the predicted answer.

We report details of train, val and test splits in Appendix D. For all the
datasets, we do not require human-written gold rationales, and thus, we sample
GPT-3 (text-davinci-003) Brown et al. (2020) for silver rationales for
supervision. We use chain-of-thought prompts from prior works on these
datasets (refer Appendix H for the full prompts) and sample $5$ rationale
generations for each training set instance with a temperature of 0.9222for
NumerSense we sampled just $1$ rationale, since the generated rationales were
deterministic; however, for supervision we use only the instances where the
answer predicted by GPT-3 is correct.

###  4.2 Rationale Property Rewards and Task Correctness

As described in §2, we determine three rationale quality properties that are
used in MaRio. Here, we formalize these properties with their implementations
that we use as rewards within MaRio. We also show demonstrative examples for
each property in Table 4.2.
Let $Q$ be the input question, $\hat{R}$ represent the LM’s generated
rationale, and $O$ and $\hat{O}$ represent the gold answer and the LM’s
predicted answer respectively. Below are the formulations of each reward:

  * •

Plausibility via Vera (Liu et al., 2023): Vera is a trained commonsense
statement verification T5-11B model, that provides a numerical score between
$0$ and $1$, indicating the plausibility of declarative sentences.

$\textsc{Plausibility}(\hat{R})\ =\textsc{Vera}(\hat{R})$ (1)

We use the Vera release from HuggingFace
333https://huggingface.co/liujch1998/vera.

  * •

Consistency via (Wiegreffe et al., 2021): Wiegreffe et al. (2021) provide a
framework to evaluate the association between a rationale and a label with the
help of two reference LMs that are trained to predict the answer with
($\mathbb{M_{QR}}$) and without ($\mathbb{M_{Q}}$) the rationale in the input.
More formally,

$\textsc{Consistency}(\hat{R})=P_{\mathbb{M_{QR}}}(O|Q,\hat{R})-P_{\mathbb{M_{Q}}}(O|Q)$
(2)

Consistency is a numerical score between $-1$ and $1$ that indicates the
faithfulness of the rationale towards the gold answer, like the implementation
by Wiegreffe et al. (2021). Hyperparameters and training guidelines for the
LMs involved in generating the Consistency score is in Appendix E.

  * •

Diversity via n-gram uniqueness in Li et al. (2022): Li et al. (2022)
calculate diversity of generated text by determining the fraction of unique
n-grams generated. Diversity is a numerical score between $0$ and $1$ that
indicates the lexical diversity of the rationale; this metric also serves the
purpose of ensuring that the rationale does not contain repeated phrases or
sentences.

$\textsc{Diversity}(\hat{R})=\prod_{n=2}^{4}\frac{\text{unique
n-grams}(\hat{R})}{\text{total n-grams}(\hat{R})}$ (3)

  * •

Task-Correctness: As our last reward, we add task correctness of the answer
that is generated following the rationale. This is a binary $0/1$ score,
referring to the wrong and right predicted answer respectively.

$\textsc{Task-Correctness}(\hat{R})=\mathds{1}[\hat{O}=O]$ (4)

We evaluate and report the above four metrics for all our baselines and
experiments. To simplify comparisons, we also report an average normalized
relative gain (NRG) (Chan et al., 2022) (Appendix E has a more detailed
definition).

Table 2: Demonstrative examples for the rationale properties of Plausibility,
Consistency and Diversity.

Question |  Malini is cutting bread with a bread knife which creates a smooth cut, while cutting cake with a bread knife creates a rough cut. This means that the ___ has less resistance (A) bread (B) cake
---|---
Plausibility |  Less resistance implies that the item would be difficult to cut through. Therefore, cake has less resistance.
Less resistance implies ease in cutting through. So the bread has a smooth cut
as it is less resistant.
Consistency |  Less resistance implies ease in cutting through. So the cake has a smooth cut as it is less resistant.
Less resistance implies ease in cutting through. So the bread has a smooth cut
as it is less resistant.
Diversity |  Less resistance implies ease in ease in ease in cutting through. Ease in cutting through. Answer is bread.
Less resistance implies ease in cutting through. So the bread has a smooth cut
as it is less resistant.

###  4.3 Human Preference Evaluation

We first present human preference studies comparing rationales generated by
MaRio and the supervised fine-tuned baseline Sft for all five datasets.

For each instance, we ask three distinct annotators from a pool of qualified
annotators to compare the two rationales across three settings, for a given
question and correct answer pair: Plausibility and Consistency, which are
defined in the same manner as the rewards, and an overall Preference rating.
Preference is meant to indicate that the annotators pick the rationale that
they would find acceptable (Wiegreffe et al., 2022) for the given question. In
Figure 3, we plot the % of instances where majority of annotators prefer only
MaRio’s rationales, only Sft’s rationales, both or none. We note human
annotators prefer MaRio’s only rationales for 83.15%, 75.3%, 71.49%, 67.44%
and 66.6% of instances respectively for StrategyQA, QuaRel OpenBookQA,
NumerSense and QASC. Human annotators also find MaRio’s rationales to be
considerably more plausible and consistent than Sft444We do not perform human
studies for Diversity and Task Accuracy since they are
automatic/straightforward metrics. We use Amazon
MTurk555https://www.mturk.com/ for all our human studies, and Appendix I
provides further details on the same.

Figure 3: Results of human studies comparing MaRio with Sft. Here, we plot the
% of instances in the test set wherein annotators prefer MaRio, Sft, both or
none, with respect to Preference, Plausibility and Consistency. We find that
human annotators vastly prefer MaRio’s rationales, and also find them to be
much more plausible and consistent.

###  4.4 Baselines vs. MaRio

We now empirically compare MaRio with Sft and three additional strong baseline
models in terms of Plausibility, Consistency, Diversity and Task Accuracy (and
their average NRG) . All our baselines and MaRio are built on top of T5-large
LMs ($0.7$B).

  1. 1.

Supervised Fine-tuned Self-Rationalizer (Sft): A fine-tuned T5-large, which
serves as the supervised learning baseline (we use the training data as
described in §4.1), trained to generate rationales and answers.

  2. 2.

Product of Rewards (Product): A multi-reward baseline where we consolidate the
rewards into a single representative metric by taking their product and apply
Quark. Aggregating several rewards into one is common in prior work and is
often done through via product (Lu et al., 2023) or weighted average (Wu et
al., 2023).

  3. 3.

Filtering rationales that lead to correct answers (Filt-Acc): This is a
variant of STaR (Zelikman et al., 2022). We iteratively train and sample new
training data from a T5-large, similar to Quark, but instead of using any
control tokens, we filter out the instances which have the wrong predicted
label. We train this model with only cross-entropy loss.

  4. 4.

Multi-reward variant of Filt-Acc (Filt-All): Again, we iteratively train and
sample new training data from a T5-large, and instead of using control tokens,
we filter out the instances which have the wrong predicted label and instances
that fall under a specified threshold value for Plausibility, Diversity and
Consistency. The threshold value is tuned as a hyperparameter. We train this
model with only cross-entropy loss.

Table 3: Baselines vs. MaRio Results. For each dataset, the best averaged NRG
(across Task Accuracy, Plausibility, Diversity and Consistency) is highlighted
in bold, and each best individual metric is underlined. Cells marked with a *
shows significant improvement for the corresponding MaRio configuration over
Sft ($p<0.05$).

Method $\rightarrow$ |  | Baselines | MaRio
---|---|---|---
Dataset $\downarrow$ | Metric | Sft | Product | Filt-Acc | Filt-All | Canonical | Additive
StrategyQA | Acc. | 57.64 | 62.01 | 61.57 | 61.35 | 60.26 | 65.07
Plau. | 0.33 | 0.35 | 0.34 | 0.36 | 0.38 |  0.39∗
Div. | 0.95 | 0.92 | 0.92 | 0.94 | 0.95 |  0.97∗
Cons. | -0.02 | 0.00 | 0.00 | 0.00 | 0.01 |  0.04∗
Avg. NRG | 58.66 | 59.75 | 59.39 | 60.34 | 60.94 | 63.27
QuaRel | Acc. | 76.99 | 79.53 | 79.53 | 76.45 | 79.89 | 78.99
Plau. | 0.71 | 0.72 | 0.71 | 0.73 |  0.77∗ | 0.75
Div. | 0.95 | 0.95 | 0.95 | 0.95 |  0.97∗ | 0.97
Cons. | 0.18 | 0.21 | 0.20 | 0.17 | 0.19 | 0.20
Avg. NRG | 75.50 | 76.71 | 76.38 | 75.74 | 78.35 | 77.75
OpenBookQA | Acc. | 63.65 | 61.65 | 65.86 | 56.63 | 66.06 | 65.55
Plau. | 0.53 | 0.52 | 0.55 | 0.47 | 0.55 | 0.55
Div. | 0.98 | 0.99 | 0.99 | 0.99 |  0.99∗ | 0.98
Cons. | 0.05 | 0.07 | 0.08 | 0.01 |  0.09∗ | 0.09
Avg. NRG | 66.79 | 66.54 | 68.47 | 63.28 | 68.64 | 68.29
NumerSense | Acc. | 46.23 | 50.75 | 51.76 | 46.73 | 55.28 | 54.27
Plau. | 0.60 | 0.60 | 0.61 | 0.58 |  0.63∗ | 0.63
Div. | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99
Cons. | 0.17 | 0.20 | 0.21 | 0.16 |  0.23∗ | 0.23
Avg. NRG | 66.18 | 67.69 | 68.32 | 65.68 | 69.95 | 69.44
QASC | Acc. | 58.64 | 57.88 | 57.78 | 57.02 | 60.15 | 59.61
Plau. | 0.44 | 0.43 | 0.39 | 0.42 |  0.47∗ | 0.47
Div. | 0.96 | 0.95 | 0.96 | 0.96 |  0.99∗ | 0.99
Cons. | 0.19 | 0.17 | 0.17 | 0.17 | 0.19 | 0.19
Avg. NRG | 64.54 | 63.60 | 62.82 | 63.38 | 66.41 | 66.28

Table 3 shows the comparisons between MaRio and the baselines. For all five
datasets, we note that MaRio is the overall best setup as noted by both the
individual metrics and the averaged NRG metric. Additive MaRio is found to be
the best performing method for StrategyQA, and Canonical MaRio is found to be
the best method for the other 4 datasets. We detail the hyperparameter
configurations for all these models in Appendix E. It is important to note
that not only does the rationale get better (as seen via the rationale
metrics), but the task accuracy also shows a marked improvement over the
baselines. We show some representative examples of rationales generated by
training with MaRio, in comparison with those generated by Sft in Table 4. We
also release the rationales generated by Sft and
MaRio.666https://drive.google.com/drive/folders/1bWBxdiwce8US5y_G6d9-Eki7ObllpR80?usp=sharing

Table 4: Examples of Rationales: Here, we show rationales generated by MaRio
and Sft for some examples for the three datasets. We observe that sometimes,
Sft generates longer rationales which are not consistent, or logically
correct, which is fixed by MaRio.

Dataset | Examples
---|---
StrategyQA |  Question: While on a liquid diet, are there some types of soup you cannot eat? Yes or No? MaRio Rationale: While on a liquid diet, there are some types of soup you cannot eat, such as cream of mushroom soup, which is a type of savory soup. Sft Rationale: While on a liquid diet, there are certain types of soups that are not suitable for a liquid diet such as a liquid diet. Question: Is a beard is moss that grows on a human? Yes or No? MaRio Rationale: A beard is a type of hair that grows on a human. Sft Rationale: A beard is a type of facial hair that grows from a root and is typically wavy to dark in color. It is not moss, but a plant that provides shelter to the moss and helps it grow.
QuaRel |  Question: A bus and a bicycle are both going the same direction. The bus is going 40mph faster than the bicycle. Will the bicycle in one hour cover (A) more distance (B) less distance MaRio Rationale: The bus is going faster than the bicycle. Thus, the bicycle will cover less distance in one hour. Sft Rationale: When something is going faster, it covers more distance. Thus, the bicycle will cover more distance in one hour because it is going faster than the bus. Question: Jim and Joe are riding bikes. Jim stops after fifteen minutes and takes a break. Joe keeps on riding. Who went farther? (A) Jim (B) Joe MaRio Rationale: When someone is riding for a longer period of time, they will go farther. Jim stopped after fifteen minutes and took a break, while Joe kept on riding. Thus, Joe went farther. Sft Rationale: When someone is moving, they will go farther than someone who is not. Thus, Joe went farther because he kept on riding.
OpenBookQA |  Question: In order for crops to grow food safely, pesticides are used on them. When it floods, this causes t he what to be poisonous? (a) air (b) Corn (c) Runoff (d) farmers MaRio Rationale: When it floods, runoff from the crops is sprayed with pesticides, making them poisonous. Sft Rationale: When it floods, pesticides are sprayed onto crops, which can make them poisonous. Question: Plant growth may cause (a) an uptick in the number of leaves (b) a surge in leaf disease (c) a gradual decrease in leaves (d) a rapid decline of the leaves MaRio Rationale: Plant growth may cause an uptick in the number of leaves. Sft Rationale: Plant growth is a process of adding new leaves to the plant. This process can cause a gradual decrease in the number of leaves.
NumerSense |  Question: People have no more than $<$mask$>$ legs. (A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine (L) ten MaRio Rationale: People have two legs. Sft Rationale: People have four legs. Question: One in $<$mask$>$ also means a quarter. (A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven (J) eight (K) nine (L) ten MaRio Rationale: One in four is a quarter. Sft Rationale: One in a quarter.
QASC |  Question: What measures a meter? (A) meter stick (B) football field (C) tetraceratops (D) visibility (E) weather (F) U.S. customary units (G) horses (H) lamphreys MaRio Rationale:The metric system is based on the idea that a meter is equal to 100 centimeters. Sft Rationale: A meter is a unit of length. The most common unit of measurement used in the United States is the meter. Question: What can be used to navigate? (A) music (B) backbones (C) astrology (D) cilia (E) magic (F) sponges (G) astronomy (H) lamphreys MaRio Rationale: Cilia are tiny hairs on the surface of the skin that can be used to navigate. Sft Rationale: Navigating requires using a variety of tools and techniques. Some of these tools include compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, compass, and com

###  4.5 Reference Large LMs vs. MaRio

We now consider three strong reference Large LMs that are used in practice for
self-rationalization (Wei et al., 2022): GPT-3 (Brown et al., 2020) (text-
davinci-003, 175B), Flan-T5 (Chung et al., 2022) (in three sizes, L, XL, XXL)
and LLaMa (Touvron et al., 2023) (in two sizes 7B, 65B); we compare MaRio with
them in terms of both average NRG, with respect to the size of the underlying
LMs (Figure 4) and individual metric scores (Table 10 in the appendix). All
these reference LMs apart from Flan-T5-L are orders of magnitude larger than
our T5-large LM trained with MaRio; we include Flan-T5-L in our comparison
even though it’s of the same size as MaRio because Flan-T5-L is instruction-
finetuned, and few-shot prompted to generate rationales, with the same set of
demonstrations used by other large LMs (shown in Appendix H).

Ideally, we want a small-sized LM (for efficiency purposes) that achieves high
performance, which corresponds to the top-left portion of the graph in Figure
4. As such, to compare two LMs’ performance, the LM which is relatively to the
left and to the top is practically a better choice. We note that for QuaRel,
MaRio results in an LM that is of a very small size (0.7B) but has a very high
performance, almost equivalent to that of GPT-3. For NumerSense, MaRio beats
all models except for Flan-T5-XXL and GPT-3, and for QASC, MaRio beats all
models except for Flan-T5-XXL, LLaMa-65B and GPT-3. For OpenBookQA , we see
that MaRio beats LMs such as Flan-T5-L, Flan-T5-XL and LLaMa-7B. For
StrategyQA we see that our LM beats Flan-T5-L, while performing only a little
worse than Flan-T5-XL.

Figure 4: Reference Large LMs vs. MaRio Results: Here, we show the comparison
of Avg. NRG values w.r.t the LM size (in the order of billion parameters) for
all the datasets.

##  5 Discussion

###  5.1 Properties and Metrics

While the properties we have optimized for in this work are necessary
properties, the question of what constitutes a complete set of properties for
a rationale to be deemed as high-quality remains an open problem (Joshi et
al., 2023; Wiegreffe et al., 2022; Golovneva et al., 2022). Some contemporary
works which work on other necessary rationale properties are REV Chen et al.
(2023a) (novelty of information in rationale and faithfulness of rationale
towards the predicted label), ROSCOE Golovneva et al. (2022) / ReCEval Prasad
et al. (2023) (metrics that score rationales with respect to their step-by-
step reasoning), LAS Hase et al. (2020) (faithfulness of rationale towards
predicted labels) etc. Further, there are also properties which do not have
widespread implementations (to the best of our knowledge) such as factual-
correctness and completeness of rationales (existing completeness metrics
depend upon the presence of gold rationales, which are not easily available
and which mean that they cannot score any alternate reasoning to the answer).
As future work, we hope to collect an extended set of properties and
corresponding metrics, and apply MaRio to improve them.

###  5.2 Multi-reward hacking

As additional experimentation with an alternate set of properties relevant to
our chosen QA datasets, we worked on a set of experiments focusing on factual-
correctness and lexical diversity; specifically for StrategyQA which requires
historical or factual correctness of the rationale (this is different from
common-sense or logical correctness measured by Plausibility, as explained in
Vera(Liu et al., 2023), and used in our work). We started off with an existing
metric for fact verification Loren (Chen et al., 2022) \- while effective, we
couldn’t use this metric in practice since each score prediction required a
Web API call, which is inefficient in our setting with training data
generation and scoring, as done in MaRio. We tried a weaker metric, wherein we
queried the rationale with a larger LM, Flan-T5-XXL and asked if the given
rationale was factually correct or not (we took the probability of ‘yes’ under
yes/no). We noticed that applying Quark/MaRio with this metric led to some
interesting reward hacking, as we show in the first two rows of Table 5.
Trying to improve on just factuality caused the LM to start generating
repetitive text, as an easy way of generating factual statements. When we
tried to counter this by training MaRio on factuality and lexical diversity
together, the LM started generating incomplete rationales.

Table 5: Reward Hacking observed in training using Quark: We show examples of
how if the rewards used are very weak or incompatible with each other, they
can lead to strange reward hacking where the score numerically increase, but
the rationales become worse qualitatively.

Question and Rationale | Errors by other LMs
---|---
Question: Can a snow leopard swim? Rationale: The snow leopard is a cat. The snow leopard is a cat. |  T5-large Repetition
Question: Would someone on Venus be unlikely to experience hypothermia? Rationale Hypothermia is a condition in which the body is unable to produce enough heat. |  T5-large Incomplete
Question: Did Cleopatra have ethnicity closer to Egyptians than Greeks? Rationale: Cleopatra was a member of the Roman dynasty of Egypt. The Roman dynasty of Egypt was a branch of the Roman Empire. The Roman Empire was a branch of the Roman Empire. |  Flan-T5-XL Repetition
Question: Is Fiat Chrysler gaining a new overall corporate identity? Rationale: Fiat Chrysler is a brand of automobiles. Fiat Chrysler is a brand of cars. |  Flan-T5-XL Repetition

We further noted that this kind of repetitive generation is observed even in
larger LMs which on surface seem much better (as seen in rows 3 and 4 in Table
5). Therefore, we note that selecting strong rewards, as well as careful
qualitative investigation is extremely important to prevent this kind of
reward hacking – where an increase in individual numerical reward scores do
not guarantee overall qualitative improvements.

###  5.3 Is accuracy enough of an indicator for rationale quality?

As we discuss in §1, many contemporary works on self-rationalization ignore
the independent quality of the generated rationale, and focus entirely on how
the rationale can contribute to the task performance. In this discussion, we
analyze the reverse: if an LM is trained only with respect to task
performance, what does this mean for the rationale? We refer back to our main
results, Table 3; we specifically look at the rows Sft, Filt-Acc and MaRio. We
first see that both Filt-Acc and Sft both improve upon the Task Accuracy on
all five datasets, as intended. We then see that for StrategyQA, QuaRel,
NumerSense and QASC, the average quality of the rationales generated by MaRio
is decidedly better than the rationales generated by Filt-Acc, as seen by the
values of the individual rationale quality metrics. For OpenBookQA, the
analysis from just the metrics is inconclusive; hence, we perform human
studies comparing Filt-Acc and MaRio, in the same manner as in §4.3.

We find that human annotators prefer MaRio’s rationales highly over that of
Filt-Acc: for $69.65\%$ of the questions, majority of the annotators prefer
MaRio’s rationales (as opposed to $22.88\%$ of preference for Filt-Acc’s
rationales, and $7.46\%$ preference for both). We further performed human
studies for Plausibility and Consistency, and again, MaRio’s rationales were
found to be distinctly better (Plausibility: $49.5\%$ preference for MaRio,
$32.58\%$ for Sft, $13.43\%$ both, $0.99\%$ neither; Consistency: $48\%$
preference for MaRio, $37.31\%$ for Sft, $9.45\%$ both, $2.48\%$ neither). In
conclusion, we find that optimizing for task performance does not naturally
improve rationale performance, which further motivates the introduction of
MaRio.

##  6 Conclusion and Future Work

Existing self-rationalization LMs use rationales as a means for improving
downstream task accuracy, with the help of large-scale LMs. In this work, we
propose MaRio, an algorithm that performs multi-reward optimization of small
self-rationalizing LMs to jointly improve the quality of their rationales as
well as their task accuracy. We present strong experimental results on a small
LM, T5-large, over competitive baselines, on datasets StrategyQA, QuaRel
OpenBookQA, NumerSense and QASC. In addition to a strong improvement in task
accuracy, we see that rationales produced by training an LM with our method
are strongly preferred by human annotators. Lastly, we discuss intricacies of
reward-conditioned rationale generation for small LMs, issues faced with
selecting appropriate rewards, as well as shortcuts taken by Quark to improve
reward scores that do not translate well to qualitative improvement. As future
work, we hope to extend our algorithm to improving rationales along more
dimensions like completeness, factuality as well as human utility.

## Ethical Considerations

Like any natural language generation system/algorithm, MaRio can
unintentionally lead to toxic and harmful text; it is up to the user of the
algorithm to use it responsibly, with non-harmful reward metrics, to prevent
the generation of biased and malicious outputs. As noted in McGuffie &
Newhouse (2020), this is a deliberate misuse of text generation models, and we
strongly denounce such practices.

##### Data.

All the datasets that we use in our work are released publicly for usage and
have been duly attributed to their original authors.

##### Crowdsourcing.

All our crowdworkers are from countries where English is the primary language.
For all our human studies, the task is setup in a manner that ensure that the
annotators receive compensation that is above minimum wage ($20/hour). Since
we conduct extensive qualification tasks before annotations, crowdworkers that
participate in the qualification are compensated more than the task, given the
time taken to read and understand task instructions and examples. Furthermore,
we ensure that we correspond with crowdworkers over email to address their
queries. Crowdworkers have also been given bonuses for flagging errors in the
task, or consistently providing good-quality annotations.

## Reproducibility

For all our experimental results and models, we report (1) the complete
hyperparameter setting and any bounds explored (Appendix E) as well as the
sizes and versions/pretrained-model links of all models used, (2) the time
taken per experiment, and infrastructure used, (3) the mathematical equations
(§4.2, Appendix B) for all algorithms and metrics used, (4) descriptions of
datasets, and demonstrations used to sample rationales from GPT-3. All our
codes and datasets are publicly released at https://github.com/INK-
USC/RationaleMultiRewardDistillation.

## References

  * Atanasova et al. (2023) Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein.  Faithfulness tests for natural language explanations.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 283–294, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.25.  URL https://aclanthology.org/2023.acl-short.25.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv_ , abs/2204.05862, 2022.  URL https://api.semanticscholar.org/CorpusID:248118878.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom.  e-snli: Natural language inference with natural language explanations.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper_files/paper/2018/file/4c7a167bb329bd92580a99ce422d6fa6-Paper.pdf.
  * Chan et al. (2022) Aaron Chan, Maziar Sanjabi, Lambert Mathias, Liang Tan, Shaoliang Nie, Xiaochang Peng, Xiang Ren, and Hamed Firooz.  Unirex: A unified learning framework for language model rationale extraction.  In _International Conference on Machine Learning_ , pp. 2867–2889. PMLR, 2022.
  * Chan et al. (2023) Aaron Chan, Zhiyuan Zeng, Wyatt Lake, Brihi Joshi, Hanjie Chen, and Xiang Ren.  Knife: Distilling reasoning knowledge from free-text rationales, 2023.
  * Chen et al. (2023a) Hanjie Chen, Faeze Brahman, Xiang Ren, Yangfeng Ji, Yejin Choi, and Swabha Swayamdipta.  REV: Information-theoretic evaluation of free-text rationales.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2007–2030, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.112.  URL https://aclanthology.org/2023.acl-long.112.
  * Chen et al. (2022) Jiangjie Chen, Qiaoben Bao, Changzhi Sun, Xinbo Zhang, Jiaze Chen, Hao Zhou, Yanghua Xiao, and Lei Li.  Loren: Logic-regularized reasoning for interpretable fact verification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 10482–10491, 2022.
  * Chen et al. (2023b) Wei-Lin Chen, An-Zi Yen, Hen-Hsen Huang, Cheng-Kuang Wu, and Hsin-Hsi Chen.  Zara: Improving few-shot self-rationalization for small language models, 2023b.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and T. Zhang.  Raft: Reward ranked finetuning for generative foundation model alignment.  _ArXiv_ , abs/2304.06767, 2023.  URL https://api.semanticscholar.org/CorpusID:258170300.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _ArXiv_ , abs/2305.14387, 2023.  URL https://api.semanticscholar.org/CorpusID:258865545.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361, 2021.
  * Golovneva et al. (2022) Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz.  Roscoe: A suite of metrics for scoring step-by-step reasoning.  _arXiv preprint arXiv:2212.07919_ , 2022.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, A. Doucet, Orhan Firat, and Nando de Freitas.  Reinforced self-training (rest) for language modeling.  _ArXiv_ , abs/2308.08998, 2023.  URL https://api.semanticscholar.org/CorpusID:261031028.
  * Hase et al. (2020) Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal.  Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language?  _arXiv preprint arXiv:2010.04119_ , 2020.
  * Hovy & Prabhumoye (2021) Dirk Hovy and Shrimai Prabhumoye.  Five sources of bias in natural language processing.  _Language and Linguistics Compass_ , 15(8):e12432, 2021.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 8003–8017, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.507.  URL https://aclanthology.org/2023.findings-acl.507.
  * Joshi et al. (2023) Brihi Joshi, Ziyi Liu, Sahana Ramnath, Aaron Chan, Zhewei Tong, Shaoliang Nie, Qifan Wang, Yejin Choi, and Xiang Ren.  Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 7103–7128, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.392.  URL https://aclanthology.org/2023.acl-long.392.
  * Jung et al. (2022) Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi.  Maieutic prompting: Logically consistent reasoning with recursive explanations.  _arXiv preprint arXiv:2205.11822_ , 2022.
  * Khot et al. (2020) Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal.  Qasc: A dataset for question answering via sentence composition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 8082–8090, 2020.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213, 2022.
  * Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners, 2023.
  * Lertvittayakumjorn & Toni (2021) Piyawat Lertvittayakumjorn and Francesca Toni.  Explanation-based human debugging of NLP models: A survey.  _Transactions of the Association for Computational Linguistics_ , 9:1508–1528, 2021.  doi: 10.1162/tacl˙a˙00440.  URL https://aclanthology.org/2021.tacl-1.90.
  * Li et al. (2023a) Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi.  Symbolic chain-of-thought distillation: Small models can also “think” step-by-step.  _ArXiv_ , abs/2306.14050, 2023a.  URL https://api.semanticscholar.org/CorpusID:259251773.
  * Li et al. (2023b) Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi.  Symbolic chain-of-thought distillation: Small models can also “think” step-by-step.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 2665–2679, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.150.  URL https://aclanthology.org/2023.acl-long.150.
  * Li et al. (2022) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis.  Contrastive decoding: Open-ended text generation as optimization.  _arXiv preprint arXiv:2210.15097_ , 2022.
  * Li et al. (2023c) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.  Making language models better reasoners with step-aware verifier.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 5315–5333, 2023c.
  * Lin et al. (2020) Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren.  Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models.  _arXiv preprint arXiv:2005.00683_ , 2020.
  * Liu et al. (2022) Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi.  Rainier: Reinforced knowledge introspector for commonsense question answering.  _arXiv preprint arXiv:2210.03078_ , 2022.
  * Liu et al. (2023) Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah A Smith, Yejin Choi, and Hannaneh Hajishirzi.  Vera: A general-purpose plausibility estimation model for commonsense statements.  _arXiv preprint arXiv:2305.03695_ , 2023.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35:27591–27609, 2022.
  * Lu et al. (2023) Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, et al.  Inference-time policy adapters (ipa): Tailoring extreme-scale lms without fine-tuning.  _arXiv preprint arXiv:2305.15065_ , 2023.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  _ArXiv_ , abs/2303.17651, 2023.  URL https://api.semanticscholar.org/CorpusID:257900871.
  * Marasovic et al. (2022) Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters.  Few-shot self-rationalization with natural language prompts.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pp. 410–424, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-naacl.31.  URL https://aclanthology.org/2022.findings-naacl.31.
  * McGuffie & Newhouse (2020) Kris McGuffie and Alex Newhouse.  The radicalization risks of gpt-3 and advanced neural language models.  _arXiv preprint arXiv:2009.06807_ , 2020.
  * Meister et al. (2020) Clara Meister, Elizabeth Salesky, and Ryan Cotterell.  Generalized entropy regularization or: There’s nothing special about label smoothing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6870–6886, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.615.  URL https://aclanthology.org/2020.acl-main.615.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_ , 2018.
  * Narang et al. (2020) Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan.  Wt5?! training text-to-text models to explain their predictions.  _arXiv preprint arXiv:2004.14546_ , 2020.
  * Prasad et al. (2023) Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal.  Receval: Evaluating reasoning chains via correctness and informativeness.  _arXiv preprint arXiv:2304.10703_ , 2023.
  * Pruthi et al. (2022) Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen.  Evaluating explanations: How much do explanations from the teacher aid students?  _Transactions of the Association for Computational Linguistics_ , 10:359–375, 2022.  doi: 10.1162/tacl˙a˙00465.  URL https://aclanthology.org/2022.tacl-1.21.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher.  Explain yourself! leveraging language models for commonsense reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4932–4942, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1487.  URL https://aclanthology.org/P19-1487.
  * Saha et al. (2023) Swarnadeep Saha, Peter Hase, and Mohit Bansal.  Can language models teach weaker agents? teacher explanations improve students via theory of mind, 2023.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _ArXiv_ , abs/1707.06347, 2017.  URL https://api.semanticscholar.org/CorpusID:28695052.
  * Sun et al. (2022) Jiao Sun, Swabha Swayamdipta, Jonathan May, and Xuezhe Ma.  Investigating the benefits of free-form rationales.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 5867–5882, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.432.  URL https://aclanthology.org/2022.findings-emnlp.432.
  * Tafjord et al. (2019) Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal.  Quarel: A dataset and models for answering questions about qualitative relationships.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 7063–7071, 2019.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2022) Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren.  Pinto: Faithful language reasoning using prompt-generated rationales.  _ArXiv_ , abs/2211.01562, 2022.  URL https://api.semanticscholar.org/CorpusID:253265114.
  * Wang et al. (2023a) PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren.  PINTO: Faithful language reasoning using prompt-generated rationales.  In _The Eleventh International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=WBXbRs63oVu.
  * Wang et al. (2023b) Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren.  SCOTT: Self-consistent chain-of-thought distillation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 5546–5558, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.304.  URL https://aclanthology.org/2023.acl-long.304.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022.
  * Wiegreffe et al. (2021) Sarah Wiegreffe, Ana Marasović, and Noah A. Smith.  Measuring association between labels and free-text rationales.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 10266–10284, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.804.  URL https://aclanthology.org/2021.emnlp-main.804.
  * Wiegreffe et al. (2022) Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi.  Reframing human-AI collaboration for generating free-text explanations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 632–658, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.47.  URL https://aclanthology.org/2022.naacl-main.47.
  * Wu et al. (2023) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  _ArXiv_ , abs/2306.01693, 2023.  URL https://api.semanticscholar.org/CorpusID:259064099.
  * Zelikman et al. (2022) Eric Zelikman, Jesse Mu, Noah D Goodman, and Yuhuai Tony Wu.  Star: Self-taught reasoner bootstrapping reasoning with reasoning.  2022\.

##  Appendix A Related Work

Self-rationalization and rationale-based distillation. There are two primary
methods adopted by prior works for generating free-text rationales. The first
set of approaches use gold human-written rationales to train a rationale
generation model (Camburu et al., 2018; Narang et al., 2020; Wiegreffe et al.,
2021). The second set of approaches prompt large LMs with the help of curated
templates with or without demonstrations containing examples of rationale
generation for the task at hand (Wei et al., 2022; Kojima et al., 2023; Li et
al., 2023c; Jung et al., 2022). Some approaches also leverage few-shot
training approaches with a handful of gold rationales (Marasovic et al., 2022;
Chen et al., 2023b). Recent approaches also leverage rationales generated by
large LMs to distill small LMs to be better at the task or better
rationalizers. (Pruthi et al., 2022; Li et al., 2023b; Chan et al., 2023; Wang
et al., 2023b; Saha et al., 2023; Hsieh et al., 2023)

Evaluating free-text rationales. Existing works have conducted human and
automatic evaluation of free-text rationales based on their association with
predicted labels (Wiegreffe et al., 2021), acceptability (Wiegreffe et al.,
2022), informativeness (Chen et al., 2023a), benefits and human utility (Sun
et al., 2022; Joshi et al., 2023), simulatability (Rajani et al., 2019; Hase
et al., 2020) and faithfulness (Atanasova et al., 2023; Wang et al., 2023a) to
name a few. Some recent works have also provided frameworks to evaluate
logical correctness of reasoning chains, that are similar to free-text
rationales (Golovneva et al., 2022; Prasad et al., 2023).

Reward-conditioned text generation. Reinforcement learning has proven to be a
reliable means to optimize language models towards a specific objective. One
such example, proximal policy optimization (PPO) (Schulman et al., 2017), has
been commonly used for a variety of tasks, spanning detoxification (Wu et al.,
2023; Lu et al., 2022), RLHF (Dubois et al., 2023; Bai et al., 2022),
improving commonsense reasoning capabilities (Liu et al., 2022), and more.
Adjacent to PPO, there are several other lighter-weight algorithms which
condition the policy language model _directly_ on the reward without the need
for a value function (Lu et al., 2022; Gulcehre et al., 2023; Dong et al.,
2023; Lu et al., 2023; Zelikman et al., 2022). These methods rely on
iterative, off-policy explorations at fixed intervals to continuously
aggregate new trajectories to learn from. Another line of work improves the
reward directly through iterative refinement on a frozen policy model (Madaan
et al., 2023).

##  Appendix B Quark and MaRio

Here, we describe Quark and MaRio in more technical detail (refer the top and
bottom pipelines in Figure 5 respectively).

quar

Figure 5: Optimizing properties with Quark (top) and MaRio (bottom)

Quark begins training with a pretrained trained language model $P_{0}(t|x)$;
Quark also requires a reference language model $P_{ref}(t|x)$ (which can be
the same as $P_{0}$, or different), and a reward function
$Rew(t,x)\rightarrow\mathds{R}$. Note that $x=[x_{0},x_{1},\dots,x_{m-1}]$
stands for the input text sequence, and $t=[t_{0},t_{1},\dots,t_{n-1}]$ stands
for the output sequence generation.

As we explain in Section 3.1, Quark operates in an iterative fashion:

  1. 1.

sampling $P_{0}$ to generate more training data: $D_{new}=(x,t_{new})$

  2. 2.

scoring the generated data using $Rew(x,t)$:
$D^{{}^{\prime}}_{new}=(x,t_{new},r)$

  3. 3.

using these instance-level scores to sort and bin the data into a fixed number
of bins $[b_{1},b_{2},\dots,b_{5}]$ each of which correspond to a unique
control token: $D^{{}^{\prime\prime}}_{new}=(x,t_{new},r,b)$,

  4. 4.

Adding the now control-token attached data to the (growing) training data
pool: $D=D\cup D^{{}^{\prime\prime}}_{new}$

During training, the model starts to associate each control token with its
corresponding quality of data (as given by $Rew(x,t)$), and to obtain the best
quality generations during inference, Quark samples the trained $P_{0}$ using
the control token corresponding to the highest reward measure. Quark is
trained using the following training objectives:

  * •

Reward-based learning using implicit reward signals based on control tokens
(which are obtained by sorting the reward $Rew(x,t)$ scores), as described
above,

  * •

Language model objective using supervised/cross-entropy loss with respect to
the target generation (as explained above, Quark samples training data in an
online manner from $P_{0}$; however, if gold or silver offline training data
is available, that can also be injected into the training pipeline by scoring
with $Rew(x,t)$)

  * •

Stable text generation using the KL divergence penalty of $P_{0}$’s generation
with respect to $P_{ref}$, and.

  * •

Entropy regularization of the generated text as in Meister et al. (2020)

We extend Quark to MaRio by using multiple sets of control tokens, each
corresponding to a distinct reward/property, i.e.,
$Rew_{1}(x,t),Rew_{2}(x,t),\dots,Rew_{k}(x,t)$; the Canonical and Additive
methods use these control tokens either together, or in a step-by-step fashion
as we explain in §3.2, 3.3. The rest of the training follows the same
iterative process and training objectives as Quark.

##  Appendix C Order of tokens

As we explain in the above two sections, the order of the control tokens
corresponding to each reward we use in training our self-rationalizing LM is a
design choice. Say for example, we have three properties, along with control
tokens corresponding to the task accuracy (as we do in this paper, refer
§4.2): this means that there are potentially $24$ orders of these properties
that we can use in Canonical MaRio, and $48$ possible variations that we can
use for Additive MaRio ($24$ orders x $2$ directions in which we can introduce
the property to the training – left or right of the existing control tokens,
assuming we keep the direction of addition consistent throughout training). It
is impractical and inefficient to experiment with all these possible orders to
pick the best possible one. Hence, we propose a simple way of picking the
order, based on the relative strengths of a (supervised-trained) self-
rationalizing LM in each of these properties.

For example, say we have four reward metrics $R_{1},R_{2},R_{3},R_{4}$, and we
determine through a predefined method which property the LM is relatively
stronger in (for example, say the LM is good at generating lexically diverse
statements, but is only moderately good at grammar, is broadly bad at
generating plausible statements, and even worse at producing concise
rationales). For example, we determine the relative strength of rewards based
on how good the supervised finetuned baseline Sft is on a particular metric on
the validation set, as opposed to the maximum and minimum value of the metric
itself.

$\text{strength}(R_{i})=\frac{\text{max}(R_{i})-r_{i}}{\text{max}(R_{i})-\text{min}(R_{i})}$
(5)

Here $R_{i}$ refers to the reward, $r_{i}$ refers to the value the Sft has on
the property $R_{i}$ on the validation set, and max/min$(R_{i})$ refer to the
maximum and minimum value taken by the reward metric $R_{i}$.

For example, let the relative order of the four reward metrics using the above
approach is $R_{2}<R_{1}<R_{4}<R_{3}$. Hence, we experiment with training the
LM with the order $R_{2},R_{1},R_{4},R_{3}$ if we want to allow the weaker
rewards to improve on their own, before the stronger rewards are introduced
into the mix. Additionally, we can also use the opposite order
$R_{3},R_{4},R_{1},R_{2}$, so that the LM can quickly optimize on the stronger
rewards and then try to be better with the weaker rewards.

##  Appendix D Dataset Splits

  * •

For StrategyQA, since labels are not available for evaluation sets, we split
the train set into training, validation and test sets (taken from Joshi et al.
(2023)), and report scores on this test set.

  * •

For OpenBookQA and QuaRel, we use the provided training dataset, tuned on the
validation set and report final performances on the test
set777https://huggingface.co/datasets/openbookqa,https://huggingface.co/datasets/QuaRel.

  * •

For NumerSense, we use the train, validation and test sets as in the official
GitHub888https://github.com/INK-USC/NumerSense/tree/main/data release.

  * •

For QASC, we split the original train set into train and validation ($900$
questions chosen randomly for validation), and use the original validation set
as the test set999https://huggingface.co/datasets/qasc.

##  Appendix E Hyperparameters and Evaluation

We use T5-large ($0.7B$ parameters) for Sft and all our MaRio experiments, and
we use T5-base for our Consistency models (as used in the original work
Wiegreffe et al. (2021)) - we always start training with the pretrained model
from
HuggingFace101010https://huggingface.co/t5-large,https://huggingface.co/t5-base.
Tables 6, 7 and 8 show the hyperparameters used to train Sft, Consistency and
MaRio respectively. Note that for our MaRio training, we use Sft as the
reference model ($P_{ref}(t|x)$ from Appendix B) for the KL divergence
penalty. Further, during inference, we always use greedy decoding. We run all
our experiments on NVIDIA Quadro RTX 8000 GPUs. For training Sft and
Consistency models, we use 1 GPU per experiment; for training MaRio, we use 2
GPUs per experiment - the first GPU to hold $P_{0},P_{ref}$ (notation from
Appendix B), and the second GPU to hold the Plausibility and Consistency
reward models.

Furthermore, we aggregate metrics using Normalized Relative Gain as mentioned
in Chan et al. (2022). NRG of a metric value $z_{i}$ (corresponding to the
general property $Z$) is formally defined as:

$\text{NRG}(z_{i})=\frac{z_{i}-\text{min}(Z)}{\text{max}(Z)-\text{min}(Z)}$
(6)

The average NRG of a set of metrics (such as with the four metrics in this
work) is a simple mathematical average of their individual NRG’s.

Table 6: Sft training details

Hyperparameter | Value
---|---
Optimizer | Adam
Adam epsilon |  $1e$-$8$
Adam initial learning-rate |  $3e$-$5$
Learning-rate scheduler | linear with warmup
Warmup steps | 1000
Gradient clipping | $0.5$
Train batch-size |  $4$ / $8$
Training time |  $\sim 4$ hours on 1 GPU

Table 7: Training details for the $\mathbb{M}_{QR}$ and $\mathbb{M}_{Q}$
models used for Consistency

Hyperparameter | Value
---|---
Optimizer | Adam
Adam epsilon |  $1e$-$8$
Adam initial learning-rate |  $3e$-$5$
Learning-rate scheduler | linear with warmup
Warmup steps | 1000
Gradient clipping | $0.5$
Train batch-size |  $4$ / $32$
Training time |  $\sim 4$ hours on 1 GPU

Table 8: Quark and MaRio training details

Hyperparameter | Value
---|---
Optimizer | Adam
Adam epsilon |  $1e$-$8$
Adam initial learning-rate |  $3e$-$5$
Learning-rate scheduler | linear with warmup
Warmup steps | 1000
Gradient clipping | $1.0$
Gradient accumulation |  $2$ steps
KL-divergence coef. |  $0.05$ / $0.1$
Entropy regularization coef. |  $0.05$ / $0.0$
Sampling rate |  $1$ (QuaRel, NumerSense, QASC)
or $2$ (StrategyQA, OpenBookQA)
samples for every train sample
Frequency of exploration | every $300$ (StrategyQA,
QuaRel) / $4000$ (OpenBookQA,
NumerSense, QASC) steps
Sampling strategy | Top-p ($0.7$) sampling
Temperature for sampling | $1.0$
Number of distinct reward-bins |  $5$ for rationale metrics,
$2$ for Task Accuracy
Train batch-size | $4$
Training time |  $\sim 1$ day on 2 GPUs
Order of rewards |  StrategyQA: strongest to weakest,
add to right
QuaRel: strongest to weakest,
OpenBookQA: weakest to strongest,
NumerSense: weakest to strongest,
QASC: strongest to weakest

Further, for our statistical significance tests, are done using one-tailed
independent t-tests (using scipy.stats.ttest_ind).

##  Appendix F Single reward Experiments

For completeness of analysis, we present single-reward Quark experiments,
where we focus on improving just one property. Table 9 shows results on the
same. We first note that in most of the cases, MaRio achieves an equivalent or
better improvement as compared to single-reward Quark. Further, we note that
even if individually some properties are better when trained under single
reward Quark as compared to MaRio, MaRio is the only experiment where all the
properties improve as compared to the Sft baseline. We also see that
sometimes, single reward Quark leads to improvement in other metrics as well;
this could be because the metrics are positively correlated for that dataset.
However, since we want to improve all metrics comprehensively, MaRio is a
deterministic way to achieve the same. (Note: We don’t run the experiment on
NumerSense Diversity, since Sft already achieves the best possible value of
$1.0$).

Table 9: Quark experiments on improving single rewards. For each dataset, the
best averaged NRG (across Task Accuracy, Plausibility, Diversity and
Consistency) is highlighted in bold, and each best individual metric is
underlined.

Method $\rightarrow$ |  | Baselines | Single Reward Quark | MaRio
---|---|---|---|---
Dataset $\downarrow$ | Metric | Sft | Product | Acc. | Plau. | Div. | Cons. | Canonical | Additive
StrategyQA | Acc. | 57.64 | 62.01 | 61.57 | 61.35 | 59.17 | 59.17 | 60.26 | 65.07
Plau. | 0.33 | 0.35 | 0.36 | 0.36 | 0.36 | 0.36 | 0.38 | 0.39
Div. | 0.95 | 0.92 | 0.92 | 0.93 | 0.96 | 0.95 | 0.95 | 0.97
Cons. | -0.02 | 0.00 | -0.01 | 0.01 | -0.04 | 0.01 | 0.01 | 0.04
Avg. NRG | 58.66 | 59.75 | 59.77 | 60.21 | 59.79 | 60.17 | 60.94 | 63.27
QuaRel | Acc. | 76.99 | 79.53 | 81.88 | 80.62 | 78.99 | 80.62 | 79.89 | 78.99
Plau. | 0.71 | 0.72 | 0.74 | 0.81 | 0.71 | 0.73 | 0.77 | 0.75
Div. | 0.95 | 0.95 | 0.95 | 0.93 | 0.97 | 0.95 | 0.97 | 0.97
Cons. | 0.18 | 0.21 | 0.23 | 0.20 | 0.20 | 0.22 | 0.19 | 0.20
Avg. NRG | 75.50 | 76.71 | 78.1 | 78.66 | 77.0 | 77.41 | 78.35 | 77.75
OpenBookQA | Acc. | 63.65 | 61.65 | 64.46 | 61.65 | 64.66 | 66.27 | 66.06 | 65.55
Plau. | 0.53 | 0.52 | 0.54 | 0.53 | 0.51 | 0.54 | 0.55 | 0.55
Div. | 0.98 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 | 0.98
Cons. | 0.05 | 0.07 | 0.09 | 0.07 | 0.07 | 0.11 | 0.09 | 0.09
Avg. NRG | 66.79 | 66.54 | 67.99 | 66.79 | 67.04 | 68.69 | 68.64 | 68.29
NumerSense | Acc. | 46.23 | 50.75 | 51.76 | 50.75 | - | 54.27 | 55.28 | 54.27
Plau. | 0.60 | 0.60 | 0.63 | 0.63 | - | 0.61 | 0.63 | 0.63
Div. | 1.00 | 1.00 | 0.99 | 1.00 | - | 1.00 | 1.00 | 0.99
Cons. | 0.17 | 0.20 | 0.21 | 0.21 | - | 0.22 | 0.23 | 0.23
Avg. NRG | 66.18 | 67.69 | 68.57 | 68.56 | - | 69.07 | 69.95 | 69.44
QASC | Acc. | 58.64 | 57.88 | 58.21 | 57.88 | 58.1 | 58.75 | 60.15 | 59.61
Plau. | 0.44 | 0.43 | 0.42 | 0.45 | 0.40 | 0.41 | 0.47 | 0.47
Div. | 0.96 | 0.95 | 0.96 | 0.97 | 0.98 | 0.96 | 0.99 | 0.99
Cons. | 0.19 | 0.17 | 0.17 | 0.17 | 0.17 | 0.20 | 0.19 | 0.19
Avg. NRG | 64.54 | 63.60 | 63.68 | 64.6 | 63.65 | 63.94 | 66.41 | 66.28

##  Appendix G Extended comparison with few-shot LLMs

In Table 10, we present the detailed performance metrics of different
reference LMs as opposed to MaRio. For QuaRel, MaRio beats all reference LLMs
except for GPT-3 on all four metrics. For NumerSense, MaRio beats all
reference LLMs except for Flan-T5-XXL and GPT-3 on all four metrics. The
results are more varied with StrategyQA, OpenBookQA and QASC; MaRio is better
than the reference LLMs (apart from GPT-3) in the case of Diversity for all
three datasets, and in cases of varying comparisons with the reference LLMs
(for example, MaRio is better at Consistency than Flan-T5-L and LLaMa-7B for
OpenBookQA). However, overall, we note that our model still needs to go
further with respect to Plausibility and Task Accuracy. We note that our
method MaRio has done a significant job in bridging the gap between LMs such
as the ones discussed in this section, and much smaller LMs such as T5-large.
We also note for Task Accuracy, Consistency and Diversity, MaRio beats
Flan-T5-L, a model of equal size which has been trained with instruction fine-
tuning for all 5 datasets (except for QASC and Consistency); and for all
datasets except for StrategyQA, MaRio also beats Plausibility of Flan-T5-L.

Table 10: We compare MaRio with strong few-shot reference LMs: Flan-T5, LLaMa
and GPT-3. Apart from Flan-T5-L (which we have included to show a model of
equivalent size that has been instruction finetuned), all these models are
much bigger than our T5-large trained with MaRio.

Method $\rightarrow$ |  | Flan-T5 | LLaMa | GPT-3 |  MaRio (0.7B)
---|---|---|---|---|---
Dataset $\downarrow$ | Metric | L | XL | XXL | 7B | 65B | T-D-003 | Canonical | Additive
StrategyQA | Acc. | 54.59 | 71.83 | 70.52 | 59.17 | 72.27 | 69.0 | 60.26 | 65.07
Plau. | 0.49 | 0.59 | 0.64 | 0.72 | 0.70 | 0.70 | 0.38 | 0.39
Div. | 0.88 | 0.82 | 0.86 | 0.88 | 0.93 | 0.95 | 0.95 | 0.97
Cons. | -0.01 | 0.02 | 0.05 | 0.00 | 0.06 | 0.09 | 0.01 | 0.04
Avg. NRG | 60.27 | 65.96 | 68.26 | 67.29 | 72.07 | 72.13 | 60.94 | 63.27
QuaRel | Acc. | 77.36 | 76.99 | 77.54 | 56.70 | 76.27 | 83.33 | 79.89 | 78.99
Plau. | 0.60 | 0.68 | 0.70 | 0.64 | 0.70 | 0.78 | 0.77 | 0.75
Div. | 0.93 | 0.90 | 0.92 | 0.94 | 0.96 | 0.95 | 0.97 | 0.97
Cons. | 0.14 | 0.13 | 0.10 | 0.00 | 0.17 | 0.23 | 0.19 | 0.20
Avg. NRG | 71.84 | 72.87 | 73.64 | 66.18 | 75.19 | 79.46 | 78.35 | 77.75
OpenBookQA | Acc. | 60.64 | 72.49 | 80.32 | 40.76 | 73.30 | 85.94 | 66.06 | 65.66
Plau. | 0.49 | 0.59 | 0.67 | 0.66 | 0.73 | 0.74 | 0.55 | 0.55
Div. | 0.87 | 0.84 | 0.93 | 0.95 | 0.97 | 0.99 | 0.99 | 0.98
Cons. | 0.05 | 0.13 | 0.22 | 0.01 | 0.16 | 0.25 | 0.09 | 0.09
Avg. NRG | 62.29 | 68.00 | 75.33 | 63.07 | 75.33 | 80.36 | 68.64 | 68.29
NumerSense | Acc. | 26.13 | 48.24 | 61.81 | 17.59 | 36.18 | 74.37 | 55.28 | 54.27
Plau. | 0.51 | 0.65 | 0.72 | 0.62 | 0.68 | 0.76 | 0.63 | 0.63
Div. | 0.97 | 0.92 | 0.98 | 0.98 | 0.99 | 1.00 | 1.00 | 0.99
Cons. | 0.03 | 0.19 | 0.35 | 0.2 | 0.36 | 0.46 | 0.23 | 0.23
Avg. NRG | 56.41 | 66.19 | 74.83 | 59.40 | 67.80 | 80.84 | 69.95 | 69.44
QASC | Acc. | 61.02 | 70.63 | 74.84 | 24.19 | 75.59 | 80.24 | 60.15 | 59.61
Plau. | 0.44 | 0.55 | 0.63 | 0.59 | 0.71 | 0.75 | 0.47 | 0.47
Div. | 0.78 | 0.63 | 0.89 | 0.74 | 0.98 | 0.97 | 0.99 | 0.99
Cons. | 0.23 | 0.32 | 0.37 | 0.10 | 0.31 | 0.38 | 0.19 | 0.19
Avg. NRG | 61.13 | 63.66 | 73.84 | 53.05 | 77.52 | 80.31 | 66.41 | 66.28

##  Appendix H Few-Shot Demonstrations

We include the full few-shot demonstrations used to prompt different models
for five datasets in Tables 11-15. For clarity, the rationalizations are
highlighted.

Table 11: The complete prompt of rationalization for StrategyQA. Demonstration
examples are collected from Wei et al., 2022

Q: Do hamsters provide food for any animals?
---
Hamsters are prey animals. Prey animals provide food for predators.
A: So the answer is yes.
Q: Could Brooke Shields succeed at University of Pennsylvania?
Brooke Shields graduated from Princeton University. Princeton is ranked as the
number 1 national college by US news. University of Pennsylvania is ranked as
number 6 national college by US news. Princeton only admits around 6 percent
of applicants as of 2018. University of Pennsylvania accepts around 9% of
applicants as of 2018.
A: So the answer is yes.
Q: Yes or no: Hydrogen’s atomic number squared exceeds number of Spice Girls?
Hydrogen is the first element and has an atomic number of one. To square a
number, you multiply it by itself. The Spice Girls has five members.
A: So the answer is no.
Q: Yes or no: Is it common to see frost during some college commencements?
College commencement ceremonies often happen during the months of December,
May, and sometimes June. Frost isn’t uncommon to see during the month of
December, as it is the winter.
A: So the answer is yes.
Q: Yes or no: Could a llama birth twice during War in Vietnam (1945-46)?
The War in Vietnam (1945-46) lasted around 6 months. The gestation period for
a llama is 11 months.
A: So the answer is no.
Q: Yes or no: Would a pear sink in water?
The density of a raw pear is about 0.59 g/cm3̂. The density of water is about
1 g/cm3̂. Objects only sink if they are denser than the surrounding fluid.
A: So the answer is no.

Table 12: The complete prompt of rationalization for QuaRel. Demonstration
examples are collected from Li et al., 2023a

Q: Mike was snowboarding on the snow and hit a piece of ice. He went much
faster on the ice because _____ is smoother.
---
(A) snow (B) ice
A: When something is smoother, it is easier to slide on. Thus, he could go
faster on the ice because ice is smoother. So the answer is: (B)
Q: I could hear then boy that say close to me clear as day, however I could
not hear the young lady sitting in the back of the room. Who am I able to hear
louder
(A) Boy (B) Lady
A: When someone is close, it is easier to hear them. I also could not hear the
young lady well. Thus, I am able to hear the boy louder. So the answer is: (A)
Q: I watched the snowflakes go from tiny specks in the sky to a nice size once
they fell on my face. When did the snowflakes seem bigger?
(A) in the sky (B) on my face
A: When something is closer, it seems bigger. The snowflakes are closer when
they are on my face. Thus, they seem bigger when they are on my face. So the
answer is: (B)
Q: When Tammy tried to slide the glass mixing bowl down the marble counter top
to her mom, it came to a dead stop when it reached the wooden cutting board.
The bowl came to a stop because the wooden cutting board has
(A) more resistance or (B) less resistance
A: When something has more resistance, it is harder to slide. Thus, the bowl
came to a stop because the wooden cutting board has more resistance. So the
answer is: (A)
Q: Sarah walked through the city and saw a tourist attraction she wanted to
visit. She had several blocks to go to get to it, and the attraction looked
very small. As she got close to it though, it towered over her. This is
because when she was close to it the attraction looked
(A) much bigger (B) much smaller
A: When something is closer, it looks bigger. Thus, the attraction looked much
bigger when she was close to it. So the answer is: (A)

Table 13: The complete prompt of rationalization for OpenBookQA. Demonstration
examples are collected from Wang et al., 2022

Q: The sun is responsible for
---
(a) puppies learning new tricks (b) children growing up and getting old (c)
flowers wilting in a vase (d) plants sprouting, blooming and wilting
A: A plant requires sunlight for photosynthesis, which accumulates resources
required for sprouting, blooming, and wilting. So the answer is: (d)
Q: When standing miles away from Mount Rushmore
(a) the mountains seem very close (b) the mountains are boring (c) the
mountains look the same as from up close (d) the mountains seem smaller than
in photographs
A: When an object is far away, it takes up less of your field of view, and so
seems smaller than in the photographs. So the answer is: (d)
Q: When food is reduced in the stomach
(a) the mind needs time to digest (b) take a second to digest what I said (c)
nutrients are being deconstructed (d) reader’s digest is a body of works
A: The stomach is part of the digestive system. The breaking down of food into
nutrients occurs in the digestive system. So the answer is: (c)
Q: Poison causes harm to which of the following?
(a) a Tree (b) a robot (c) a house (d) a car
A: A tree is a living thing. Poison causes harm to living things. So the
answer is: (a)
Q: A magnet will stick to
(a) a belt buckle (b) a wooden table (c) a plastic cup (d) a paper plate
A: A belt buckle is made of metal. If a magnet is attracted to a metal, then
that magnet will stick to that metal. So the answer is: (a)
Q: Deer are less safe in the woods because wolves
(a) have fur (b) howl (c) have claws (d) have tails
A: Claws are used by wolves to catch prey like deer. So the answer is: (c)
Q: An electric car causes
(a) more CO2 emissions (b) equal CO2 emissions (c) electric emissions (d) less
CO2 emissions
A: An electric car uses less gasoline than a regular car and thus causes less
CO2 emissions. So the answer is: (d)

Table 14: The complete prompt of rationalization for NumerSense. Demonstration
examples are collected from Liu et al., 2022

Q: penguins have <mask> wings.
---
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven
(J) eight (K) nine (L) ten
A: Birds have two wings. Penguin is a kind of bird. So the answer is (D).
Q: a parallelogram has <mask> sides.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven
(J) eight (K) nine (L) ten
A: A rectangular is a parallelogram. A square is a parallelogram. So the
answer is (F).
Q: there are <mask> feet in a yard.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven
(J) eight (K) nine (L) ten
A: A yard is three feet. So the answer is (E).
Q: water can exist in <mask> states.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven
(J) eight (K) nine (L) ten
A: There states for matter are solid, liquid, and gas. So the answer is (E).
Q: a typical human being has <mask> limbs.
(A) no (B) zero (C) one (D) two (E) three (F) four (G) five (H) six (I) seven
(J) eight (K) nine (L) ten
A: Human has two arms and two legs. So the answer is (F)

Table 15: The complete prompt of rationalization for QASC. Demonstration
examples are collected from Wang et al., 2023a

Q: How do you reduce pollution? (A) igniting fuel and oxidiser (B)
transportation technology (C) wasting (D) not recycling (E) burning fossil
fuels (F) converting electricity to heat (G) water conservation (H) using less
resources
---
A: Conserving resources has a positive impact on the environment. Use of
resources affects the environment such as pollution. So the answer is: (H)
Q: what will move to another area if their habitat will no longer support
them? (A) density (B) Birds (C) squids (D) humans (E) clouds (F) gravity (G)
cows (H) Whales
A: If a habitat can no longer support animals then those animals will move to
another area. Cows are social animals. So the answer is: (G)
Q: With the exception of allergies, what may cause a person to seek medical
attention? (A) Contact with latex (B) a tree falling (C) Organs within the
body. (D) Contact with baby chicks (E) prolactin release (F) Contact with
peanut butter (G) hypothyroidism (H) Contact with microorganisms
A: Microorganisms can cause infections. Infections usually require medical
treatment. So the answer is: (H)
Q: Lavender can induce (A) healing (B) energy (C) hormones (D) mutations (E)
Heart rate (F) growth (G) symptoms (H) warmth
A: Healing requires rest. Lavender induces restful sleep. So the answer is:
(A)
Q: what state is a liquid in when frozen? (A) vapor (B) dense (C) gas (D)
cooled (E) steam (F) solid (G) boiling (H) cold
A: Freezing means changing from a liquid into a solid by reducing heat energy.
Liquids freeze when they change to the solid state. So the answer is: (F)
Q: what unites to form a diploid zygote? (A) plant reproduction (B) Most
plants (C) orchids (D) sperm and ova (E) salt and pepper (F) predator and prey
(G) honeybees (H) diploids and zygotes
A: Gametes then unite in fertilization and form a diploid zygote.
Collectively, the sperm and the ova are also referred to as gametes. So the
answer is: (D)
Q: What absorbs all visible light? (A) apples (B) coal (C) Green (D) coral (E)
skin (F) bamboo (G) glass (H) eyes
A: If an object is black then that object absorbs all visible light. Light
grains are quartz, Black grains are coal. So the answer is: (B)

##  Appendix I Crowdsourcing for Human Evaluations

In this section, we describe the MTurk experiment setup. Each MTurk annotator
is paid above minimum wage. Since the dataset we used is carefully annotated
by human, we can assure there is no toxic content and our experiment setup was
submitted to IRB for ethical review. We limited our Turkers to English
speaking nations - United States, Canada, Australia, New Zealand and United
Kingdom. To ensure the quality of evaluation, we conduct a round of
qualification tasks which include a small set of evaluations. Turkers need to
finish the qualification task first and get results of it, then we will show
them the whole task.

####  I.0.1 Worker Selection and Quality Control

Here, we describe details about how workers are selected and how annotations
are ensured to be clean. First, we employ multiple rounds of trials before
deploying the actual task so as to get feedback from annotators whether they
understand the task correctly. This includes in-house tests, tested via Amazon
Turk Sandbox 111111https://requester.mturk.com/developer/sandbox and small
batches tested on Turk. Second, we create a set of medium to hard
qualification tasks for verifying preference, plausibility and consistency
annotations that the annotators have to work on. These tasks are hand curated
that cater certain parts of the instruction – whether the annotators are
reading the rationale correctly, or whether they are able to make appropriate
connections between the rationale and the question. This weeds out a lot of
annotators who do not understand the task or are cheating. We also weed out
workers who are too ‘fast’ (completing the task in less than $5$ seconds,
which is indicative of potential slacking in the task). Third, we constantly
monitor task responses and feedback provided to annotators about their task.
We also collect feedback from them which we adapt in new versions of the task.

The final MTurk instructions and template that we land upon after the
qualifications is shown in Figure 6 and 7

Figure 6: MTurk Instructions. We show these instructions to turkers, along
with a sample HIT, and more examples that contain special cases of each of the
annotation questions. Figure 7: MTurk Template. Given a question and two
explanations, we ask annotators to choose which explanation they prefer,
followed by questions about their plausibility and consistency.

##  Appendix J Limitations

MaRio demonstrates promising improvements on the self-rationalization
capability of small language models; we note that using MaRio on a small LM
like T5-large leads to considerable bridging of the gap between the quality of
its rationales versus the quality of rationales generated by much larger
language models. However, we note that the results are still very much
dependent on the initially available data (since we heavily depend upon silver
standard rationales generated by GPT-3 to give our model a warm start). Our
method is also dependent upon the mathematical rewards that we use: as we
discuss in Section 5, this is a very new and active area of research, and we
as a research community are still figuring out what properties we need, and
how to efficiently implement a good mathematical metric for them.