11institutetext: Institute for Bioengineering of Catalonia (IBEC), Barcelona,
Spain
11email<EMAIL_ADDRESS>22institutetext: Universitat Pompeu Fabra
(UPF), Barcelona, Spain 33institutetext: Barcelona Institue of Science and
Technology (BIST), Barcelona, Spain 44institutetext: Catalan Institute for
Research and Advanced Studies (ICREA), Barcelona, Spain

# Towards sample-efficient policy learning with DAC-ML

Ismael T. Freire 112233 Adrián F. Amil 112233 Vasiliki Vouloutsi 1133
Paul F.M.J. Verschure ITF and AFA contributed equally to this work11223344

###### Abstract

The sample-inefficiency problem in Artificial Intelligence refers to the
inability of current Deep Reinforcement Learning models to optimize action
policies within a small number of episodes. Recent studies have tried to
overcome this limitation by adding memory systems and architectural biases to
improve learning speed, such as in Episodic Reinforcement Learning. However,
despite achieving incremental improvements, their performance is still not
comparable to how humans learn behavioral policies. In this paper, we
capitalize on the design principles of the Distributed Adaptive Control (DAC)
theory of mind and brain to build a novel cognitive architecture (DAC-ML)
that, by incorporating a hippocampus-inspired sequential memory system, can
rapidly converge to effective action policies that maximize reward acquisition
in a challenging foraging task.

###### keywords:

Cognitive Architectures, Sample-Inefficiency Problem, Sequence Learning,
Reinforcement Learning, Distributed Adaptive Control

##  1 Introduction

With the advent of Deep Reinforcement Learning (DRL), the last decade has been
marked by several historical landmarks in the field of Artificial Intelligence
(AI), in terms of both scientific and societal impact. Reaching human-level
performance in historically considered uniquely-human intellectual domains
(such as chess [1], Go [2], and real-time strategy video games [3]) has led to
renewed claims about its resemblance to human cognition. Some researchers have
even proposed that DRL could be a valid metaphor to understand the brain [4].
However, criticisms to this view have pointed out to at least one great
fundamental difference between human learning and deep learning approaches:
sample efficiency [5].

Sample efficiency refers to the number of data samples required for a learning
system to attain a target level of performance (see [6] for a more detailed
discussion on the topic). Current DRL models are sample inefficient because
they require large amounts of training episodes to achieve human-level
performance. An example of this limitation is the research on emergent tool-
use in a hide-and-seek multi-agent game; where DRL agents took up to 380
million episodes to learn some of the more sophisticated behaviors [7].

Recent research in the field has sought to find solutions to the sample-
inefficiency problem by introducing algorithms that make use of a memory
system to bootstrap policy learning, such as in Episodic Reinforcement
Learning (ERL) [8]. In ERL, stored episodes are retrieved from past
interactions with the environment to support decision-making when the agent
encounters similar states. However, states are treated as being independent of
one another. Therefore, ERL cannot capitalize on the sequential nature of
experience that gives rise to highly informative state-transition statistics.

In this paper, we present a novel cognitive architecture well suited for
state-of-the-art Machine Learning (ML) environments, whereby policy learning
and action selection are based on a sequential memory system inspired by the
human hippocampus [9]. In particular, building upon the architectural
principles of the Distributed Adaptive Control (DAC) theory of mind and brain,
we derive a biologically-inspired architecture (DAC-ML) that leverages the
continuous nature of experience to bootstrap policy learning in a sample-
efficient manner. We demonstrate the learning efficiency of DAC-ML in a
challenging foraging task and show how the sequential feature of the memory
system is key for achieving such fast learning dynamics.

##  2 Cognitive architecture

DAC-ML is a biologically-inspired cognitive architecture composed of three
layers (Reactive, Adaptive, and Contextual, see Fig. 1). It is built based on
the architectural principles of DAC of layered and hierarchical organization,
that predicate that cognition is hierarchically distributed in different
layers of control [10], [11], [12].

The Reactive layer represents the agent’s prewired reflexes and provides
predefined responses that allow for simple but robust interaction with the
environment. In DAC-ML, the Reactive layer incorporates a pseudo-random-walk
algorithm that drives the agent to explore the environment and facilitates the
input sampling initially required by the learning in the upper layers of the
architecture (see Fig. 1, bottom panel).

The Adaptive Layer, where the perceptual learning takes place, contains a
convolutional autoencoder that uses the sensory input sampled by the Reactive
layer to build compressed representations of the state space (see Fig. 1,
middle panel). These state representations are then linked with the agent’s
action at every step to build the state-action couplets to be conveyed to the
Contextual layer for policy learning. Moreover, the transmission of the
couplets to the Contextual layer depends on the compressed representations
surpassing a quality threshold, assessed by the convolutional autoencoder’s
corresponding reconstruction error.

Figure 1: Representation of the three layers of the DAC-ML architecture:
Reactive (bottom), Adaptive (center), and Contextual (top). The Reactive layer
implements a pseudo-random-walk generator that allows an efficient random
exploration of the environment. The Adaptive layer comprises a convolutional
autoencoder that generates compressed representations of the visual input to
produce the state-action couplets to be conveyed to the STM buffer in the
Contextual layer. The Contextual layer incorporates the STM and LTM buffers,
along with an action selection algorithm, that allows the agent to perform
policy learning and decision-making based on the stored episodic sequences.
RE: reconstruction error; e: compressed state vectors; m: action; V: reward
value; G: episodic sequence.

The Contextual layer uses stored sequential episodic memories to perform
policy learning and to guide real-time decision-making. It is composed of
short-term memory (STM) and long-term memory (LTM) buffers, and an action
selection algorithm (see Fig. 1, top panel). The STM transiently stores the
most recent sequence of state-action couplets generated by the agent’s
interaction with the environment, and it is updated following a first-in,
first-out (FIFO) rule. Once a goal is encountered, the current sequence
maintained in the STM is conveyed and stored in the LTM with its associated
reward value. In the current implementation, the STM has a fixed length of 50
state-action couplets, while the capacity of LTM is limited to 100 sequences.

###  2.0.1 Action selection with a sequential memory bias for rapid policy
learning.

The Contextual layer’s action selection algorithm, while being invariant in
itself, selects an action based on the recent history of observed states and
performed actions. Concretely, at every time step, the observed compressed
state is compared, based on a similarity metric (i.e., Euclidean distance),
with all the state representations stored in LTM. The similarity scores are
then weighted by an additional memory system storing the trigger values, which
keep track of the recent history of selected state-action couplets (as in
[13]). This procedure increases the eligibility of the couplets that come
immediately after the recently selected ones. Hence, the resulting eligibility
scores (i.e., similarity scores weighted by the trigger values) effectively
bias the action selection towards continuing on recently visited sequences in
LTM.

Furthermore, a set of state-action couplets in LTM is selected based on their
eligibility scores surpassing both absolute and proportional thresholds. This
procedure enforces a winner-takes-all mechanism that has been shown to operate
in the hippocampus [14]. The selected set of state-action couplets are then
assigned the reward values associated with their corresponding sequences,
weighted by an exponential decay based on each couplet’s normalized distance
to the end of the sequence. This mechanism favors the selection of actions
that are closer in time to bigger rewards. Moreover, the reward values
assigned to each couplet are normalized to the maximum reward associated with
the selected couplets, thus implementing a relative reward valuing function.
Lastly, the final action to be performed is randomly drawn from the
probability distribution generated over the discrete action space by
normalizing the sums of all the relative reward values associated with the
actions of each selected couplet. Then, the trigger values are updated based
on the selected couplets so that the couplets selection (and thus action
selection) in the next time step is biased towards following the same
sequences. This biasing mechanism is further reinforced by behavioral feedback
[12], whereby the input sampling caused by the selected action favors the
storage of similar sequences of state-action couplets. At the same time, our
implementation of the action selection as a probability distribution over the
discrete action space keeps the Bayesian optimality feature of previous DAC
models operating in a continuous action space [11].

##  3 Results

Figure 2: Experimental setup. Left panel: a top-view of the foraging task
modeled in Animal-AI. The small blue sphere represents the agent. The big
green sphere at the center represents the reward. Top right panels: first-
person view of the agent (left) and the reconstructed image generated by the
Adaptive layer (right). Bottom right panel: representation of the selected
couplets of the LTM in the Contextual layer.

We used the Animal-AI framework [15] to create the simulation environment for
testing the capabilities of the DAC-ML cognitive architecture. Animal-AI is an
AI experimentation and evaluation platform specially designed to allow for
cognitive testing in reinforcement learning and spatial navigation tasks. This
framework imposes several realistic constraints regarding the type of input an
agent can receive since the agents have to learn to navigate the 3D
environment only guided by first-person observations in the form of raw pixel
data.

To test the efficacy of DAC-ML in obtaining rewards by acquiring stable
behavioral sequences, we tested it in a foraging task that requires the
sequential encoding and retrieval of adequate visual cues in order to reach
the reward location in an optimal amount of time. The environment is a colored
maze (see Fig. 2) with only one reward source in the center. The walls are
painted with different colors to serve as visual cues for the agent. At the
beginning of each episode, the agent is randomly allocated in one of the four
corners of the maze and has to reach the center to acquire the reward. The
reward value is initially set to 3, and it decays linearly with time until the
end of the episode. An episode finishes either when the agent reaches the
reward or when the time limit is reached.

We ran 20 simulations of 1.000 episodes each and compared the results of DAC-
ML against two control agents: (1) a Reactive agent that randomly explores the
environment, and (2) a version of DAC-ML with the same set of parameters but
without incorporating the sequential memory bias, and thus treating the state-
action couplets as independent units. The results show the capacity of DAC-ML
to efficiently acquire rewards in the foraging task in relatively few
episodes, as compared with the control agents. Fig. 3-top shows the difference
in reward acquisition between the three agents. We show that DAC-ML is able to
maximize the reward acquisition in less than 1.000 episodes. In addition, Fig.
3-center shows the number of steps (time) taken by the agents in every episode
to reach the reward. These results demonstrate that DAC-ML can optimize the
time it requires to reach the reward and that it generates rapid learning
dynamics compared to a more standard cognitive architecture that does not
incorporate a memory system based on episodic sequences. Finally, we also
estimated the stability over time of the action selection (i.e., policy
learning) by computing the mean entropy across the probability distributions
over the action space throughout each episode. In the bottom panel of Fig. 3,
we observe how the policy entropy also steadily decreases over time for DAC-ML
(but not for the control agent), indicating convergence towards a stable
action policy. Overall, these results demonstrate the advantage of
incorporating a sequential memory bias in achieving rapid and stable policy
learning in a cognitive architecture optimized by reinforcement learning.

Figure 3: DAC-ML rapidly stabilizes and maximizes reward acquisition in a
foraging task. Top panel: mean reward per episode scores of DAC-ML (green),
DAC-ML with no sequencing bias (red), and a reactive agent (blue). Center
panel: mean steps per episode of the same agents. Bottom panel: mean entropy
per episode for both DAC-ML agents. Entropy is computed over the probability
distribution of actions at every step so that lower values indicate policy
convergence and stability. All metrics have been computed by an averaged
sliding window of 20 episodes over time.

##  4 Discussion and Future Work

In this paper, we have presented DAC-ML, a novel cognitive architecture based
on the organizational principles of the Distributed Adaptive Control theory,
that can efficiently maximize reward acquisition in a challenging foraging
task. DAC-ML is a memory-guided decision-making architecture inspired by the
hippocampus, that stores complete episodic sequences of state-action couplets
to guide action selection. We showed that this biologically-inspired
architectural bias promotes the rapid generation of stable and efficient goal-
directed behavior, thus affording sample-efficient policy learning.

In contrast with classical RL where the policy is directly updated by the
learning algorithm, in DAC-ML, policy learning is achieved through the
interaction of the different components of its layered architecture. Since
action-selection in DAC-ML follows a fixed rule, the main driver of policy
learning is the constant acquisition and retrieval of new sequential memories.
In other words, the behavioral policy of a DAC-ML agent is implicitly updated
by the memories it forms in its interaction with the environment. Moreover,
DAC-ML emphasizes the learning of complete behavioral sequences, whereas ERL
usually treats its memory units as separate disconnected events. The
incorporation of this hippocampus-inspired architectural bias bootstraps the
formation of a behavioral feedback loop [12], allowing DAC-ML to rapidly
transition from an initial exploration phase to a more exploitative phase in
which the behavior of the agent stabilizes over time while maximizing reward
acquisition.

With this work, we seek to contribute to the available proposed solutions to
the sample-inefficiency problem in reinforcement learning by taking
inspiration from empirical and theoretical research from the cognitive
sciences. In future work, we plan to extend the results presented in this
paper by directly comparing DAC-ML with state-of-the-art ERL models, and test
it in multi-agent environments, such as in game-theoretic dilemmas [16] [17]
[18].

##  5 Acknowledgements

This research received funding from the H2020-EU project HR-Recycler, ID:
820742.

## References

  * [1] Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., … Lillicrap, T. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419), 1140-1144.
  * [2] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., … Dieleman, S. (2016). Mastering the game of Go with deep neural networks and tree search. nature, 529(7587), 484-489.
  * [3] Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., … Oh, J. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350-354.
  * [4] Marblestone, A. H., Wayne, G., Kording, K. P. (2016). Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience, 10, 94.
  * [5] Marcus, G. (2018). Deep learning: A critical appraisal. arXiv preprint arXiv: 1801.00631.
  * [6] Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., Hassabis, D. (2019). Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5), 408-422.
  * [7] Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.
  * [8] Pritzel, A., Uria, B., Srinivasan, S., Puigdomenech, A., Vinyals, O., Hassabis, D., … Blundell, C. (2017). Neural episodic control. arXiv preprint arXiv:1703.01988.
  * [9] Buzsáki, G., Tingley, D. (2018). Space and time: The hippocampus as a sequence generator. Trends in cognitive sciences, 22(10), 853-869.
  * [10] Verschure, P.F., Pennartz, C.M., Pezzulo, G.: The why, what, where, when and how of goal-directed choice: neuronal and computational principles. Philosophical Transactions of the Royal Society B: Biological Sciences369(1655), 20130483(2014)
  * [11] Verschure, P. F., Althaus, P. (2003). A real‐world rational agent: unifying old and new AI. Cognitive science, 27(4), 561-590.
  * [12] Verschure, P.F., Voegtlin, T., Douglas, R.J.: Environmentally mediated synergy between perception and behaviour in mobile robots. Nature, 425(6958), 620–624(2003)
  * [13] Marcos, E., Ringwald, M., Duff, A., Sánchez-Fibla, M., Verschure, P. F. (2013). The hierarchical accumulation of knowledge in the distributed adaptive control architecture. In Computational and robotic models of the hierarchical organization of behavior (pp. 213-234). Springer, Berlin, Heidelberg.
  * [14] de Almeida, L., Idiart, M., Lisman, J. E. (2009). A second function of gamma frequency oscillations: an E%-max winner-take-all mechanism selects which cells fire. Journal of Neuroscience, 29(23), 7497-7503.
  * [15] Crosby, M., Beyret, B., Halina, M. (2019). The animal-ai olympics. Nature Machine Intelligence, 1(5), 257-257.
  * [16] Freire, I.T., Moulin-Frier, C., Sanchez-Fibla, M., Arsiwalla, X.D., Verschure, P.F. Modeling the formation of social conventions from embodied real-time interactions.PloS one15(6), e0234434 (2020)
  * [17] Freire, I. T., Arsiwalla, X. D., Puigbò, J. Y., Verschure, P. (2019). Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control. arXiv preprint arXiv:1905.13225.
  * [18] Freire, I. T., Puigbò, J. Y., Arsiwalla, X. D., Verschure, P. (2018, July). Modeling the opponent’s action using control-based reinforcement learning. In Conference on Biomimetic and Biohybrid Systems (pp. 179-186). Springer, Cham.


# How Much Consistency Is Your Accuracy Worth?

Jacob K. Johnson  Ana Marasović
Kahlert School of Computing
University of Utah
<EMAIL_ADDRESS>
###### Abstract

Contrast set consistency is a robustness measurement that evaluates the rate
at which a model correctly responds to all instances in a bundle of minimally
different examples relying on the same knowledge. To draw additional insights,
we propose to complement consistency with _relative consistency_ — the
probability that an equally accurate model would surpass the consistency of
the proposed model, given a distribution over possible consistencies. Models
with 100% relative consistency have reached a consistency peak for their
accuracy. We reflect on prior work that reports consistency in contrast sets
and observe that relative consistency can alter the assessment of a model’s
consistency compared to another. We anticipate that our proposed measurement
and insights will influence future studies aiming to promote consistent
behavior in models.

##  1 Introduction

Annotators introduce data shortcuts that allow models to solve tasks in
unintended ways Gururangan et al. (2018). In response, it has been proposed to
measure whether a model correctly responds to a bundle (or a _contrast set_)
of slightly modified instances that rely on the same knowledge Gardner et al.
(2020); Kaushik et al. (2020). The rate at which a model accomplishes this is
termed _consistency_. We propose an additional measurement —  _relative
consistency_ — that facilitates discussion about achievable consistency
scores, enabling a more nuanced comparison.

|  |  |  |
---|---|---|---|---
|  |  |  |
(a) Accuracy=4/10, Consistency=2/5, RelConsistency=$100\%$

|  |  |  |
---|---|---|---|---
|  |  |  |
(b) Accuracy=7/10, Consistency=2/5, RelConsistency=$66.7\%$

|  |  |  |
---|---|---|---|---
|  |  |  |
(c) Accuracy=7/10, Consistency=3/5, RelConsistency=$100\%$

|  |  |  |
---|---|---|---|---
|  |  |  |
(d) Accuracy=8/10, Consistency=3/5, RelConsistency=$88.9\%$

|  |  |  |
---|---|---|---|---
|  |  |  |
(e) Accuracy=8/10, Consistency=4/5, RelConsistency=$100\%$

Table 1: Tables depict a dataset of 10 examples, where each column showcases a
bundle of an original instance paired with its perturbed version.  denotes
that the instance is correctly predicted by a model. The relative consistency
is the measurement we propose to complement the standard consistency.

To demonstrate why this is desired, consider situations that are illustrated
in Table 1. Both 1(a)–1(b) correctly solve two bundles, i.e., have the same
consistency. 1(b) solves three additional instances but in a way that does not
promote consistency; 1(c) shows that a higher consistency can be gained with
the same accuracy. In contrast, although 1(a) is less accurate, everything it
handled was done consistently, and higher consistency cannot be achieved with
the same accuracy. This analysis sheds light on an upside of 1(a) and a
limitation of 1(b) that might go unnoticed if we solely compare
accuracy/consistency. Let us turn to examples 1(d). Although it represents a
model with an improved consistency relative to 1(a), we could have achieved
better consistency for the same accuracy (see 1(e)).111Because this is a toy
example, relative consistency is high, though not perfect, even in less-than-
ideal cases 1(b) and 1(d).

Relative consistency (§2) measures whether the consistency of our model would
likely be outperformed by an equally accurate model, relative to the
distribution of possible consistencies; see Eq. (5). Specifically, it is the
probability that our model’s consistency is (in most cases) higher or equal to
the consistency scores that are achievable with the same accuracy. If relative
consistency is 100% then our model is the most consistent it can be given its
accuracy, as a more consistent, equally accurate model exists only with near-
zero probability. In practice, the goal should be to increase the “standard
consistency” while also achieving 100% relative consistency.

In light of this additional consistency metric, in §4 we revisit the findings
of three publications that report consistency as a metric for their
evaluations and point out some additional conclusions we might draw from these
reported consistencies. Our code is available at
https://github.com/jacobkj314/relative-consistency.

##  2 Relative Consistency

We first introduce background terminology (§2.1), then derive elements we need
for defining relative consistency: (i) achievable consistency scores for a
given accuracy (§2.2) and (ii) a distribution over achievable consistency
scores (2.3).

###  2.1 Background

A _contrast set_ or _bundle_ is a set of minimally different instances that
might admit different answers, thus testing a model across/near its decision
boundary.222Sometimes “contrast set” is used to refer to contrastive instances
only (without the original ones). For example, these two HotpotQA instances
Yang et al. (2018) represent a contrast set:

  * •

Q: Is the Marsilea or the Brabejum the genus of more individual species of
plants? A: Marsilea

  * •

Q: Is the Marsilea or the Brabejum the genus of less individual species of
plants? A: Brabejum

The model is required to answer both of them correctly to be considered
consistent in that bundle. Evaluation with contrast sets makes it harder for
simple and inadequate models to perform highly (e.g, a model that has just
learned a spurious correlation between the word “Marsilea” and “more”).
Related studies construct bundles of paraphrases that have the same, not
contrastive, labels Elazar et al. (2021).

The term _consistency_ is overloaded in NLP and refers to different concepts
Li et al. (2019); Jang et al. (2022); Wang et al. (2023). In this work, we
study _contrast set consistency_ defined as the proportion of bundles where a
model accurately labels every instance in a bundle:

$\operatorname{consistency}=\frac{|B\in\mathcal{B}:\forall x\in
B,y_{p}(x)=y(x)|}{|\mathcal{B}|},$ (1)

where $\mathcal{B}$ is a set of all bundles of related instances in a given
dataset, $x$ is an example, $y_{p}(x)$ is the predicted label for $x$, and
$y(x)$ is its gold label.

(a) Distributions of consistency scores.

(b) Relative consistency scores.

Figure 1: On the left is a heatmap of distributions of consistency at each
accuracy for 100 bundles of 2 instances: each vertical slice corresponds to a
separate distribution of different consistencies. Fig. 2 (Appendix) shows the
$\log_{10}$ of this plot that better highlights the long tails of these
distributions. On the right are relative consistency scores given a model’s
accuracy and consistency, i.e., the CDF of the figure on the left. Note that
for a different number of bundles, these plots would look slightly different.

###  2.2 Achievable Consistency Scores

Consider a contrastive test set formed from $n$ original instances, plus a
contrastive instance derived from each original instance by varying along some
pertinent dimension. There are $2n+1$ possible accuracies $a$ that a model
could achieve on this test set, namely $A=\\{0,1,\ldots,2n-1,2n\\}$.333While
accuracy is typically denoted as a proportion of correct instances, reporting
absolute numbers simplifies our notation. It is easy to translate a quantity
$a$ to a corresponding proportion $\alpha$ via the identity $a=2n\alpha$,
while a consistency quantity $c$ relates to the consistency proportion
$\gamma$ via $c=n\gamma$. Similarly, there are $n+1$ possible consistencies
$c$ that a model could achieve, namely $C=\\{0,1,\ldots,n-1,n\\}$.

Furthermore, for a given accuracy $a\in A$, only a subset $C_{a}\subseteq C$
of consistencies is achievable. Trivially, for $a=0$, $C_{a}=\\{0\\}$ (because
a model cannot consistently respond to a bundle without correctly responding
to at least the instances within that bundle) and for $a=2n$, $C_{a}=\\{n\\}$
(because a model that correctly responds to all instances has also
consistently responded to all the bundles those instances comprise). $C_{a}$
can then be defined in terms of $n$ and $a$:

$C_{a}=\\{c\in C:c_{min}^{(a)}\leq c\leq c_{max}^{(a)}\\}$ (2)

where $c_{min}^{(a)}$ and $c_{max}^{(a)}$ are defined as:

$\displaystyle c_{min}^{(a)}$ $\displaystyle=\begin{cases}0&\text{if }a\leq
n\\\ a-n&\text{if }a>n\end{cases}$ (3) $\displaystyle c_{max}^{(a)}$
$\displaystyle=\left\lfloor\frac{a}{2}\right\rfloor$ (4)

Intuitively, if $a\leq n$ then it is possible that all bundles have one of
their constituent instances incorrectly answered, in which case,
$c_{min}^{(a)}=0$. However, if $a>n$, then at least $a-n>0$ of bundles must be
fully correctly answered. Indeed, for a bundle to be inconsistent at least one
item must be incorrectly answered, so for a given $a$, the number of incorrect
items is $2n-a$. Thus, at most $2n-a$ bundles can be inconsistent, and
$c_{min}^{(a)}=n-(2n-a)=n-2n+a=a-n$.

The definition of $c_{max}^{(a)}$ follows from the observation that a
maximally consistent model will consistently respond to the maximum number of
bundles for which it is possible that both instances are correctly answered,
and that equals $\left\lfloor\frac{a}{2}\right\rfloor$.

###  2.3 Distribution of Achievable Consistencies

Given an accuracy $a$, we construct a distribution of achievable consistencies
$c\in C_{a}$ with:

$\mathbb{P}(c|a)=\frac{m(c,a)}{M(a)}$ (5)

where $M(a)$ is the number of ways a model can achieve accuracy $a$ and is
given by:

$M(a)={2n\choose a}$ (6)

because there are $2n$ total instances, of which any $a$ might be the ones to
which a model correctly responds.444It is possible to consider consistency to
be the more underlying property of a model’s behavior and compute a
distribution over possible accuracies in the range $\left[2c,2n-n+c\right]$.
The corresponding accuracy by consistency distributions could then be computed
given the above-defined consistency by accuracy distributions. $m(c,a)$
represents the number of ways a model can achieve accuracy $a$ and consistency
$c$, and is given by:

$m(c,a)={n\choose c}{{n-c}\choose{a-2c}}{2^{a-2c}}$ (7)

where:

  * •

${n\choose c}$ corresponds to the number of ways that $c$ consistent bundles
can be selected from $n$,

  * •

${{n-c}\choose{a-2c}}$ corresponds to the number of ways the remaining $a-2c$
accurate instances can be distributed across the remaining $n-c$ bundles,
giving each selected bundle only one correct instance (to avoid creating an
additional consistent bundle),

  * •

$2^{a-2c}$ represents the number of ways that these partially correct bundles
could have either instance correct.

Using this, we can calculate $m(c,a)$ and $M(a)$ across all values of $c$ and
$a$ for reasonable sizes of $n$. These distributions can be extended for
bundle sizes above 2; see formulas in Appendix B. Figure 1(a) shows the
distributions of consistency scores for a dataset with 100 bundles of 2
instances.

Note that this distribution is not uniform for different consistencies at a
given accuracy. There will be some consistencies that have more ways to be
achieved for a given accuracy. This is why the formula $m(c,a)$ is crucial to
the computation of relative consistency that comes next.

This formulation assumes that all instances are equally difficult which is
known to not be the case in practice Swayamdipta et al. (2020). It also
disregards any inductive biases of models/datasets that could skew the
distribution.

#### Relative Consistency

We measure the tendency to be consistent exhibited by a model that achieved
accuracy $a$ and consistency $c$ on a contrastive set by computing the
cumulative probability distribution over achievable consistencies in $C_{a}$
up to $c$:

$\operatorname{RC}(c,a)=\sum_{\begin{subarray}{c}c_{i}\in C_{a}\\\ c_{i}\leq
c\end{subarray}}{\mathbb{P}(c_{i}|a)}$ (8)

Intuitively, $\operatorname{RC}(c,a)$ indicates how likely the model’s
consistency is to outperform an equally accurate model relative to the
distribution of achievable consistencies defined in (5). This allows us to
quantify whether model consistency is below, at, or above chance, given its
accuracy. In a good case, $\operatorname{RC}$ is high, meaning that it is
unlikely that an equally accurate model will have higher consistency.
Alternatively, if $\operatorname{RC}$ is low, then it is likely that an
equally accurate model will have higher consistency (which is unwanted).

Although other measurements which contextualize consistency scores within a
particular accuracy can be constructed — such as simply scaling the
consistency between $c_{min}^{(a)}$ and $c_{max}^{(a)}$, or reporting the
fraction of fully consistent of those that are at least partly correct — these
approaches lack the probabilistic interpretation underlying
$\operatorname{RC}$. §3–4 highlight circumstances in which this probabilistic
interpretation is useful, and Appendix C compares the score distributions
obtained via these measurements to the score distributions obtained via
$\operatorname{RC}$.

##  3 Analysis with Simulated Contrastive Set

Suppose you evaluate a model on a contrastive test set containing 100 bundles
of 2 instances. The distribution of consistencies for this dataset is shown in
Figure 1(a), with the CDF of that distribution (corresponding to the
$\operatorname{RC}$ score) in Figure 1(b).

Note that the highest-density region of the distribution moves upward as
accuracy increases, and takes up only a very thin band. This means that, for a
given accuracy, there is generally little room for improvement in consistency.
This can be useful when discussing results: if a particular training approach
yields a 5% improvement in consistency for an equally accurate model, that
represents a substantial change in how the model tends to respond to inputs.

It can still happen that improving accuracy and consistency decreases relative
consistency. As an example, consider comparing a model $M_{1}$, which achieves
$a=130,c=45$ ($65\%$ accuracy, $45\%$ consistency) against a model $M_{2}$
with $a=150,c=55$ ($75\%$ accuracy, $55\%$ consistency). Clearly, model
$M_{2}$ is more desirable for practical uses, if we are just comparing one
model to another, but if we are comparing two different training approaches,
and want to know which induces a stronger tendency for consistent responses,
then we would be interested to know that $M_{1}$ has
$\operatorname{RC}=93.0\%$, while $M_{2}$ has $\operatorname{RC}=37.1\%$. This
insight, that one model is below chance consistency, while another is well
above, is made possible by the probabilistic interpretation of
$\operatorname{RC}$.

##  4 Meta-Analysis of Prior Work

In this section, we discuss results reported by prior works that conduct
evaluation with contrast sets under the light of relative consistency.

Dataset | #Bundles | Acc | Cons | $\operatorname{RC}$
---|---|---|---|---
UD Parsing | 150 | 55.3 | 17.3 |  $\sim$0.0
PERSPECTRUM | 217 | 88.0 | 78.8 | 97.6
ROPES | 974 | 40.1 | 17.6 | 97.8
MC-TACO | 646 | 26.0 | 8.0 | 95.2

Table 2:  Relative consistency scores computed for results reported in Gardner
et al. (2020). In the 3rd column, we report the average of “Original Test”
(original only) and “Contrast” (contrastive only) columns in their Table 2.
That is the accuracy, $a$, we use in calculations in §2. Models with similar
consistency (UD Parsing and ROPES) have different tendencies to respond
consistently as revealed by their $\operatorname{RC}$ scores.

###  4.1 Gardner et al. (2020)

They construct contrast sets for several common test sets by modifying a
sample of the test set instances. They train a biaffine parser Dozat and
Manning (2017) with ELMo embeddings Peters et al. (2018) for UD parsing
(Zeldes, 2017, Silveira et al., 2014, Basili et al., 2015, Ahrenberg, 2007),
and RoBERTa Liu et al. (2019) for reading comprehension tasks: ROPES Lin et
al. (2019), and MC-TACO Zhou et al. (2019) and stance prediction: PERSPECTRUM
Chen et al. (2019). Table 2 shows the accuracy and consistency of these models
for four of their contrast sets.555We exclude contrast sets that do not have
the bundle size of 2. They report the accuracy of the original instances and
contrastive instances separately, so to obtain the accuracy in the contrast
set (that we need to calculate $\operatorname{RC}$) we average those. In doing
so, we assume that the accuracy of the full original test set is similar to
the accuracy of the sample of original test set instances. In the rightmost
column, we report the relative consistency scores that we introduce.

#### Analysis

We observe that the UD parsing and ROPES models have a similar consistency
score (17.3 and 17.6). However, the UD parsing model’s consistency has a near-
zero chance to outperform an equally accurate model. On the other hand, the
ROPES model is quite likely to do so.

Additionally, relative consistency shows that models with low consistency
could nonetheless have a large tendency to respond to bundles
consistently.666Note that high relative consistency does not guarantee that
such a model will continue to respond to bundles consistently with improved
accuracy. We see this with the results for MC-TACO, which, despite only
achieving 8.0% consistency, is more consistent than an equally accurate model
in 95.2% of cases. Intuitively, this means that the above chance model has at
least generalized well within the few cases to which it correctly responds.

###  4.2 Dua et al. (2021)

They investigate whether training approaches that consider a full bundle of
related instances together, instead of their constituent instances separately,
improve consistency. Table 3 shows their report results obtained with T5
Raffel et al. (2020) and the relative consistency scores we compute from their
results, on the contrastive version of ROPES — a reading comprehension dataset
for evaluating a model’s ability to reason about “effects of the relationships
in the background passage in the context of the situation”.

Loss | Accuracy | Consistency | $\operatorname{RC}$
---|---|---|---
MLE | 65.7 | 52.1 | 100.0
$\Lsh$ +UL | 68.3 | 55.6 | 100.0
$\Lsh$ +CE | 76.6 | 64.7 | 100.0

Table 3: A comparison of relative consistency scores computed from results
report in Dua et al. (2021) (in “Dev EM” and “Dev C” columns in their Table
3). The number of bundles is 844. The unlikelihood (UL) and contrastive
estimation (CE) objectives improved the accuracy and consistency over MLE,
_without decreasing relative consistency_. This is how consistency should be
improved in this case.

Size | Loss | Scope-Acc | Aff-Acc | Scope-Cons | Aff-Cons | Scope-RC | Aff-RC
---|---|---|---|---|---|---|---
Large | MLE | 66.84 | 67.09 | 42.86 | 42.35 | 17.10 | 10.06
CE | 64.80 | 66.84 | 40.31 | 43.37 | 20.10 | 26.64
$\lambda_{1}$MLE + $\lambda_{2}$CE |  |  |  |  |  |
$\Lsh$ $\lambda_{1},\lambda_{2}=1.0,1.0$ | 66.33 | 68.11 | 42.86 | 44.39 | 30.43 | 19.37
$\Lsh$ $\lambda_{1},\lambda_{2}=0.33,1.0$ | 66.58 | 68.37 | 43.37 | 44.90 | 42.44 | 16.01
3B | MLE | 74.23 | 76.79 | _56.12_ | 60.71 | _80.76_ | 88.68
CE | 74.23 | 77.55 | 56.12 | 61.73 | 80.76 | 92.03
$\lambda_{1}$MLE + $\lambda_{2}$CE |  |  |  |  |  |
$\Lsh$ $\lambda_{1},\lambda_{2}=0.33,1.0$ | 74.23 | 77.04 | 56.12 | 60.71 | 80.76 | 88.68
$\Lsh$ $\lambda_{1},\lambda_{2}=0.33,1.0$ | 76.02 | 78.57 | _58.67_ | 63.78 | _79.32_ | 98.60

Table 4: Results of UnifiedQA-v2 Khashabi et al. (2022) on the CondaQA
contrastive dataset, with the expectation that including the Contrastive
Estimation (CE) objective would improve consistency, as in Dua et al. (2021).
$\operatorname{RC}$ scores are reported here only for some of the edit
dimensions in CondaQA; see Table 5 for the rest.

#### Analysis

We observe that the baseline model trained with the maximum likelihood
estimation (MLE) is already at ceiling performance in terms of its tendency to
produce consistent responses (i.e., its $\operatorname{RC}$ scores). Combining
contrastive estimation (CE; Smith and Eisner, 2005), or unlikelihood training
(UL; Welleck et al., 2020), with MLE not only improves the accuracy and
consistency but also does so in a way that does not lower the relative
consistency, which is desired. This emphasizes the effectiveness of these
objectives.

###  4.3 Ravichander et al. (2022)

They introduce CondaQA, a contrastive dataset for studying reading
comprehension models’ effectiveness in reasoning about the implications of
negation expressed in a given text. Each CondaQA instance comes with three
minimally varied versions: one paraphrases the negation, another modifies what
is negated (scope), and the last removes the negation. Ravichander et al.
(2022) use UnifiedQA-v2 Khashabi et al. (2022) as a backbone model. We explore
the factors that might influence the consistency of the large and 3B versions
of this model:

  * •

The training objective: MLE, CE, or combined $\lambda_{1}$MLE+$\lambda_{2}$CE.

  * •

The choice of hyperparemeters $\lambda_{1}$ and $\lambda_{2}$ (with UnifiedQA-
large).

Table 4 shows accuracy, consistency, and relative consistency we obtain for
bundles where the original instance is paired with its: (i) _scope_ -edited
version, and (ii) _affirmative_ version (without negation). In Table 5
(Appendix), we also include the results with paraphrase-edits.

#### Analysis

An increase in consistency does not necessarily indicate a heightened tendency
to consistently respond to bundles (unless the accuracy stays the same).
Compare CE with 1MLE+1CE (double underlined, in the upper part of the table).
In this case, by training with MLE and CE, affirmative consistency has gone up
slightly, however, the model’s chance of outperforming an equally accurate
model dropped down from 26% to 19%. This is an example of a suboptimal way of
improving consistency, and MLE+CE is not necessarily superior to the
standalone CE in this case. A similar, but less pronounced, situation occurs
when comparing MLE against .33MLE+1CE for scope consistency in the bottom part
of the table (italicized).

Conversely, even if standard consistency has not improved, a model’s tendency
to consistently respond to bundles may have. For example, compare MLE with
1MLE+1CE for scope consistency in the upper part of the table (wavy
underlined). In this case, scope accuracy lowered slightly but absolute scope
consistency remained the same, leading to a large improvement in
Scope-$\operatorname{RC}$. This may suggest that additional CE loss resulted
in the model unlearning a few individual instances without unlearning any
complete bundles it had learned. Similarly, 0.33MLE+1CE scope consistency in
the upper part of the table (underlined once) increased slightly but the scope
relative consistency has increased notably. If we compared only consistency we
would conclude that the choice of hyperparameters $\lambda_{1},\lambda_{2}$ is
not vital, where actually they can affect model consistency behavior as shown
by relative consistency.

##  5 Conclusion

We introduce relative consistency, which complements standard contrast
consistency by allowing an accuracy and consistency score pair to be examined
to determine whether a higher consistency was possible with that accuracy.
This facilitates the comparison of consistencies achieved by models that
achieved different levels of accuracy. We show that relative consistency
enriches conclusions we make about whether a model is more consistent than
another, and occasionally even leads us to different takeaways.

##  6 Limitations

This mathematical model is based on a simplified version of contrastive
datasets. Contrastive datasets may have more than two edits for each original
instance, which will result in a different distribution. Although we provide
formulas for distributions of arbitrary bundle size in Appendix B, these
distributions are less intuitive, more expensive to compute, and additionally
have the drawback that, if a model achieves high pairwise $\operatorname{RC}$
on two of the elements of the bundle, it is likely to achieve high bundle
$\operatorname{RC}$, even if the other elements of the test set do not achieve
high pairwise $\operatorname{RC}$. In general, we recommend formulating
questions of consistency in terms of bundles with one instance exhibiting a
feature and the other instance lacking that feature. Moreover, contrastive
datasets may include extra data that is not contrastive; e.g., CondaQA has a
small number of bundles with a single instance because other instances in the
bundle were filtered because they did not pass quality checks.

In §2.3, we state the drawbacks of the distribution (5). Namely, we do not
consider that the distribution might be skewed due to the varying example
difficulty and other inherent properties of datasets and models.

##  7 Acknowledgements

We thank anonymous reviewers for their thoughtful and constructive comments,
members of the UtahNLP group for helpful feedback, and Petar Bakić for
proofreading our formulas.

## References

  * Ahrenberg (2007) Lars Ahrenberg. 2007.  LinES: An English-Swedish parallel treebank.  In _Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA 2007)_ , pages 270–273, Tartu, Estonia. University of Tartu, Estonia.
  * Basili et al. (2015) Roberto Basili, Cristina Bosco, Rodolfo Delmonte, Alessandro Moschitti, and Maria Simi. 2015.  _Harmonization and Development of Resources and Tools for Italian Natural Language Processing within the PARLI Project_, volume 589.
  * Chen et al. (2019) Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019\.  Seeing things from a different angle:discovering diverse perspectives about claims.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 542–557, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.
  * Dua et al. (2021) Dheeru Dua, Pradeep Dasigi, Sameer Singh, and Matt Gardner. 2021.  Learning with instance bundles for reading comprehension.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7347–7357, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1307–1323, Online. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Jang et al. (2022) Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022.  BECEL: Benchmark for consistency evaluation of language models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3680–3696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard H. Hovy, and Zachary Chase Lipton. 2020.  Learning the difference that makes A difference with counterfactually-augmented data.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Khashabi et al. (2022) Daniel Khashabi, Yeganeh Kordi, and Hannaneh Hajishirzi. 2022.  Unifiedqa-v2: Stronger generalization via broader cross-format training.
  * Li et al. (2019) Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019.  A logic-driven framework for consistency of neural models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3924–3935, Hong Kong, China. Association for Computational Linguistics.
  * Lin et al. (2019) Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019.  Reasoning over paragraph effects in situations.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 58–62, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marasovic. 2022.  CONDAQA: A contrastive reading comprehension dataset for reasoning about negation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8729–8755, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Silveira et al. (2014) Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014.  A gold standard dependency corpus for English.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 2897–2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Smith and Eisner (2005) Noah A. Smith and Jason Eisner. 2005.  Contrastive estimation: Training log-linear models on unlabeled data.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 354–362, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural text generation with unlikelihood training.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Zeldes (2017) Amir Zeldes. 2017.  The gum corpus: creating multilayer resources in the classroom.  _Language Resources and Evaluation_ , 51:581–612.
  * Zhou et al. (2019) Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019.  “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3363–3369, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Numerical Stability of Relative Consistency

To avoid numerical instability, especially when comparing $\operatorname{RC}$
scores for two models, (i.e. to determine whether a training approach improves
a model’s tendency to produce consistent responses, or determine which of two
training approaches best improves a model’s tendency towards consistent
responses), we define:

$\mu(c,a)=\sum_{\begin{subarray}{c}c_{i}\in C_{a}\\\ c_{i}\leq
c\end{subarray}}{m(c_{i},a)}$ (9)

(i.e., the cumulative combinatoric mass) and then rephrase the definition of
$\operatorname{RC}$ as:

$\operatorname{RC}(c,a)=\frac{\mu(c,a)}{M(a)}$ (10)

which relies on only one division, so is less prone to floating-point rounding
errors.

This also allows us to compute:

$\frac{\mu(c_{1},a_{1})}{M(a_{1})}-\frac{\mu(c_{2},a_{2})}{M(a_{2})}$ (11)

(i.e., the improvement in $\operatorname{RC}(c_{1},a_{1})$ over
$\operatorname{RC}(c_{2},a_{2})$ scores) as:

$\frac{\mu(c_{1},a_{1})M(a_{2})-\mu(c_{2},a_{2})M(a_{1})}{M(a_{1})M(a_{2})}$
(12)

which allows for comparisons between models that are very close in their
$\operatorname{RC}$ scores, (i.e., in the long tail of consistency).

##  Appendix B Formulas for Bundle Sizes $b>2$

Let us consider a contrastive test set containing $n$ bundles of $b$ instances
each. There are $nb+1$ possible accuracies $a$, but still $n+1$ possible
consistencies $c$.

$C_{a}$ can then be defined in terms of $n$, $b$, and $a$ as follows:

$C_{a}=\\{c\in C:c_{min}^{(a)}\leq c\leq c_{max}^{(a)}\\}$ (13)

where $c_{min}^{(a)}$ and $c_{max}^{(a)}$ are defined as:

$c_{min}^{(a)}=\begin{cases}0&\text{if }a\leq n(b-1)\\\ a-n(b-1)&\text{if
}a>n(b-1\end{cases}$ (14) $c_{max}^{(a)}=\left\lfloor\frac{a}{b}\right\rfloor$
(15)

Intuitively, if $a\leq n(b-1)$ then it is possible that all bundles have at
least one of their constituent instances incorrectly answered, in which case,
$c_{min}^{(a)}=0$. However, if $a>n(b-1)$, then at least $a-n(b-1)>0$ of
bundles must be fully correctly answered. Indeed, for a bundle to be
inconsistent at least one item must be incorrectly answered, so for a given
$a$, the number of incorrect items is $nb-a$. Thus, at most $nb-a$ bundles can
be inconsistent, and $c_{min}^{(a)}=n-(nb-a)=n-nb+a=a-n(b-1)$.

The definition of $c_{max}^{(a)}$ follows from the observation that a
maximally consistent model will consistently respond to the maximum number of
bundles for which it is possible that all $b$ instances are correctly
answered, and that equals $\left\lfloor\frac{a}{b}\right\rfloor$.

Now, $M(a)$ (the number of ways a model can achieve accuracy $a$) is given by:

$M(a)={nb\choose a}$ (16)

and $m(c,a)$ (the number of ways a model can achieve accuracy $a$ and
consistency $c$) is given by:

$m(c,a)={n\choose c}\cdot G(n-c,b,a-cb)$ (17)

where the first factor in the product still intuitively corresponds to the
number of ways that $c$ consistent bundles can be selected out of $n$, but the
second refers to the number of ways the remaining correct instances could be
distributed within responses to the test set such that no additional
consistent bundles can be formed.

This second factor $G(m,b,k)$ is defined as:

$G(m,b,k)=\sum_{r=0}^{R}{\left(-1\right)^{r}{m\choose r}{(m-r)b\choose k-rb}}$
(18)

where $R=\min(m,\left\lfloor\frac{k}{b}\right\rfloor)$. This can be understood
as the number of ways to select $k$ elements of an $m\times b$ matrix such
that no row contains a complete $b$ elements selected. The first term (which
simplifies to ${mb\choose k}$) is the total number of ways these $k$ elements
could be selected, ignoring the restriction on complete rows, and the
remaining terms apply the principle of inclusion-exclusion to alternately
subtract and add the number of ways that at least $r$ rows could be filled (by
multiplying the number of ways that $r$ out of $m$ rows could be selected by
the number of ways the remaining $m-r$ rows and $b$ columns could be filled by
the remaining $k-rb$ items to select), up to the maximal number of rows $R$
that could be filled, whether that is determined by the total number of rows
available $m$ or the number of rows the items $k$ could fill.

In general, we do not recommend using this measurement for bundle sizes above
2 except for evaluating consistency on three-valued features, as many
consistency questions can be formulated as bundles with one instance
exhibiting a feature and one instance lacking that feature.

##  Appendix C Distributions of Alternative Approaches

Figures 3 and 4 plot the distributions of consistency scores (for a 100-bundle
dataset) obtained via simpler non-probabilistic alternatives and compare them
to the distributions obtained via $\operatorname{RC}$. Both of these
characterizations lower the scores for consistencies that are above chance and
raise the scores for consistencies that are below chance.

Figure 2:  The $\log_{10}$ of the distributions of consistency scores in
Figure 1(a).

Size | Loss | B-A | P-A | S-A | A-A | B-C | P-C | S-C | A-C | B-RC | P-RC | S-RC | A-RC
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Large | MLE | 67.22 | 66.33 | 66.84 | 67.09 | 27.04 | 58.16 | 42.86 | 42.35 | 99.92 | 100.00 | 17.10 | 10.06
CE | 67.35 | 67.35 | 64.80 | 66.84 | 28.57 | 61.22 | 40.31 | 43.37 | 99.99 | 100.00 | 20.10 | 26.64
$\lambda_{1}$MLE + $\lambda_{2}$CE |  |  |  |  |  |  |  |  |  |  |  |
$\Lsh$ $\lambda_{1},\lambda_{2}=1.0,1.0$ | 67.73 | 68.88 | 66.33 | 68.11 | 28.57 | 63.78 | 42.86 | 44.39 | 99.98 | 100.00 | 30.43 | 19.37
$\Lsh$ $\lambda_{1},\lambda_{2}=0.33,1.0$ | 68.24 | 68.37 | 66.58 | 68.37 | 30.10 | 63.27 | 43.37 | 44.90 | 100.00 | 100.00 | 42.44 | 16.01
3B | MLE | 75.64 | 76.28 | 74.23 | 76.79 | 44.39 | 71.43 | 56.12 | 60.71 | 100.00 | 100.00 | 80.76 | 88.68
CE | 75.38 | 75.51 | 74.23 | 77.55 | 43.88 | 70.41 | 56.12 | 61.73 | 100.00 | 100.00 | 80.76 | 92.03
$\lambda_{1}$MLE + $\lambda_{2}$CE |  |  |  |  |  |  |  |  |  |  |  |
$\Lsh$ $\lambda_{1},\lambda_{2}=1.0,1.0$ | 75.51 | 75.77 | 74.23 | 77.04 | 44.90 | 70.92 | 56.12 | 60.71 | 100.00 | 100.00 | 80.76 | 88.68
$\Lsh$ $\lambda_{1},\lambda_{2}=0.33,1.0$ | 76.53 | 77.55 | 76.02 | 78.57 | 45.92 | 73.47 | 58.67 | 63.78 | 100.00 | 100.00 | 79.32 | 98.60

Table 5: The full results of UnifiedQA-v2 Khashabi et al. (2022) on the
CondaQA contrastive dataset, with the expectation that including the
Contrastive Estimation (CE) objective would improve consistency, as in Dua et
al. (2021).

Figure 3:  In this figure, the interval
$\left[c_{min}^{(a)},c_{max}^{(a)}\right]$ is simply scaled to cover $[0,1]$
and the score is scaled accordingly. On the left is the score given a model’s
accuracy and consistency, on the right is shown the change in score when
moving from $\operatorname{RC}$ to this formulation.

Figure 4:  In this figure, of the bundles which are at least partially
correct, the proportion of fully consistent bundles is reported. On the left
is the score given a model’s accuracy and consistency, on the right is shown
the change in score when moving from $\operatorname{RC}$ to this formulation.


# Minimal lensing solutions in the singular perturbative approach.

Alard, C.,
IAP, 98bis Boulevard Arago, Paris

###### Abstract

This paper analyse the properties of minimal solutions for the reconstruction
of the lens potential in the singular perturbative approach. These minimal
solutions corresponds to an expansion with a minimal degree in Fourier
expansion of the perturbative fields. Using these minimal solutions prevent
spurious physically meaningless terms in the reconstruction of the fields. In
effect a perturbative analysis indicates that a small change in the source
model will corresponds to the higher order terms in the expansion of the
fields. The results of the perturbative analysis are valid not only for
slightly non-circular sources but also for more distorted sources to order
two. It is thus of crucial importance to minimize the number of terms used in
the modelling of the lens. Another important asset of the minimal solutions is
that they offers a de-coupling between the source and lens model and thus help
to break the source lens degeneracy issue. The possible drawback of minimal
solutions is to under-estimate the higher order terms in the solution. However
this bias has its merit since the detection of higher order terms using this
method will ensure that these terms are real. This type of analysis using
minimal solutions will be of particular interest when considering the
statistical analysis of a large number of lenses, especially in light of the
incoming satellite surveys.

###### keywords:

gravitational lensing: strong - (cosmology:) dark matter

††pubyear: 2022††pagerange: Minimal lensing solutions in the singular
perturbative approach.–LABEL:lastpage

##  1 Introduction

###  1.1 The general problem of the source shape degeneracy.

Modeling gravitational lenses does not only require the reconstruction of the
lens potential but also the reconstruction of the source. In practice the
problem of modeling gravitational lenses is operated in sequence. The first
step is to estimate a source model for a given potential model. When the
potential is fixed the reconstruction of the source is a linear least-square
problem (see Warren & Dye (2003)). Once a source model has been estimated the
quality of the reconstruction is estimated by calculating the $\chi^{2}$ from
the difference between the model and the data. Then in the next step the
potential is slightly modified in order to search for a better $\chi^{2}$.
This sequence is iterated until a minimum for the $\chi^{2}$ has been
obtained, leading to the best solution for the source lens system. However it
is important to note that in this process there is a complex interaction
between the source model and the lens model. As a consequence it is difficult
to understand if the convergence towards a given solution in this process
leads to well defined and unique solution, or if the lens and source solution
is degenerate. In order to understand the context of the source lens
degeneracy problem it is interesting to note the existence of some
sophisticated methods of reconstruction. These methods may provide some
improvements in the direction of breaking the source lens degeneracy. Some
example of these methods are, Brewer etal. (2016), Agnello etal. (2013), Saha
& Williams (2006), Meneghetti etal. (2006). Although these methods are of
interest for the source lens degeneracy problem, they do not offer of a full
solution to the degeneracy problem. This work will provide a different view of
this problem this time based on the minimal solutions for the lens. The
analysis will be based on the singular perturbative approach of gravitational
lenses (see Alard (2007)). This paper will begin with a perturbative analysis
of nearly circular sources to prove that a small change in source model
generates a large number of higher order terms in the corresponding model of
the perturbative fields. The analysis will be generalized to sources with
larger distortion of circularity. Then it will be shown that minimal solutions
are of great interest for breaking source lens degeneracy and that at lower
order in the expansion of the expansion of the perturbative fields the source
and lens solutions are almost de-coupled.

###  1.2 The singular perturbative approach in gravitational lensing.

In this paper we will work in the singular perturbative approach developed in
Alard (2007), Alard (2008), Alard (2009), Alard (2010), and Alard (2017). In
this approach the lens equation is simplified and depends only on two
functionals of the angular variable $\theta$, the fields $f_{1}(\theta)$ and
$\frac{df_{0}}{d\theta}$. For convenience it is useful to introduce the fields
$\tilde{f}_{1}(\theta)$ and $\frac{d\tilde{f}_{0}}{d\theta}$ which include the
additional contributions from the source impact parameters $(x_{0},y_{0})$
(see Eq. (1), and Alard (2007) Eq. 10).

$\tilde{f}_{i}=f_{i}+x_{0}\cos\theta+y_{0}\sin\theta~{}~{}~{}~{}{\rm
where}~{}i=0,1$ (1)

In the singular perturbative approach the source coordinates ${\bf r_{s}}$ are
related to the lens coordinates ${\bf r}$ by Eq. (2). To express the singular
perturbative lens equation it is useful to introduce the parameter $dr$ with
$r=1+dr$. Note that for convenience the Einstein radius has been renormalized
to unity bu using a proper choice for the coordinates system.

$\mathbb{r}_{s}=(\kappa_{2}dr-\tilde{f}_{1})\mathbb{u}_{r}-\frac{d\tilde{f}_{0}}{d\theta}$
(2)

An interesting application is to consider the image of a round source contour
with radius $R_{0}$ in this theory. This model is simple with direct
analytical formula for the images in singular perturbative expansion (see
Alard (2007), Eq. 12).

$dr=\frac{1}{\kappa_{2}}\left[\tilde{f}_{1}\pm\sqrt{R_{0}^{2}-\left(\frac{d\tilde{f}_{0}}{d\theta}\right)^{2}}\right]$
(3)

The circular contour is transformed into the images of the source by Eq. (3)
(originally Eq. (13) in Alard (2007)).This transformation uses a certain model
for the fields $\tilde{f}_{1}$ and $\frac{d\tilde{f}_{0}}{d\theta}$. In the
forthcoming perturbation theory presented in the next section these fields in
the circular source model will be the background fields, $f_{1}^{0}$ and
$df_{0}^{0}$.

##  2 General perturbation of the round source model.

In this section we will consider how changing the source model affects the
perturbative field solution for the lens. We will start from the un-perturbed
round solution (see Sec. 3) and its associated fields $\tilde{f}_{1}$ and
$\frac{d\tilde{f}_{0}}{d\theta}$. Let’s consider now that for the same
gravitational arc we change the model of the source, which will be now a
perturbed circular source contour. This change in the source contour leads to
new equations for the corresponding lens potential. To the perturbation of the
source model a perturbation of the singular perturbative field model will be
associated. Let’s define the circular contour source model perturbation
$G(x_{s},y_{s})$,

$x_{s}^{2}+y_{s}^{2}+\epsilon G(x_{s},y_{s})-R_{0}^{2}=0$ (4)

Here we define the perturbation of the field model when making a model based
on the perturbed circular contour defined in Eq. (4).

$\begin{cases}\tilde{f}_{1}=f_{1}^{0}+\epsilon\delta f_{1}\\\
\frac{d\tilde{f}_{0}}{d\theta}=\frac{df_{0}^{0}}{d\theta}+\epsilon\delta
f_{0}\end{cases}$ (5)

The left hand side of Eq. (4) represents the circular source contour, while
the right side represents the perturbation to the circular contour. The
corresponding perturbations of the fields in Eq. (4) are $\delta f_{1}$ and
$\delta f_{0}$. Here the source coordinates $\mathbb{r}_{s}=(x_{s},y_{s})$ are
related to the lens coordinates and perturbative fields by the perturbative
lens equation (see Eq. 2). In Cartesian coordinates Eq. (2) reads,

$\begin{cases}x_{s}=(dr-\tilde{f}_{1})\cos\theta+\frac{d\tilde{f}_{0}}{d\theta}\sin\theta\\\
y_{s}=(dr-\tilde{f}_{1})\sin\theta-\frac{d\tilde{f}_{0}}{d\theta}\cos\theta\end{cases}$
(6)

Note that in Eq. (6) $\kappa_{2}$ is set to unity. The mass sheet degeneracy
implies that $\kappa_{2}$ is a degenerate quantity, and for simplicity we set
$\kappa_{2}=1$. It is more convenient to express 4 in normalized source
coordinates units. We define the following re-scaling,

$\begin{cases}x_{s}\equiv\frac{x_{s}}{R_{0}}~{};~{}y_{s}\equiv\frac{y_{s}}{R_{0}}\\\
\tilde{f}_{i}\equiv\frac{\tilde{f}_{i}}{R_{0}}~{};~{}f_{i}^{0}\equiv\frac{f_{i}^{0}}{R_{0}}~{};~{}\delta
f_{i}\equiv\frac{\delta f_{i}}{R_{0}}~{};~{}i=0,1\\\
dr\equiv\frac{dr}{R_{0}}\\\
G(x_{s},y_{s})\equiv\frac{G(x_{S},y_{s})}{R_{0}}\end{cases}$ (7)

For convenience we will consider that Eq. (4) and Eq. (5) are renormalized
with the coordinates transformation of Eq. (7) The perturbative solution to
the first order in $\epsilon$ for Eq. (4) and Eq. (6) reads,

$\begin{cases}\delta
f_{1}=\frac{1}{4\sqrt{1-\left(\frac{df_{0}^{0}}{d\theta}\right)^{2}}}\left(G\left(\bf{u_{r}H},\bf{u_{t}H}\right)-G\left(\bf{\tilde{u}_{r}H},\bf{\tilde{u}_{t}H}\right)\right)\\\
\delta
f_{0}=\frac{1}{4\frac{df_{0}^{0}}{d\theta}}\left(G\left(\bf{u_{r}H},\bf{u_{t}H}\right)+G\left(\bf{\tilde{u}_{r}H},\bf{\tilde{u}_{t}H}\right)\right)\end{cases}$
(8)

With the definition of the vectors:

$\begin{cases}{\bf u_{r}}=(\cos\theta,\sin\theta)\\\ {\bf
u_{t}}=(-\sin\theta,\cos\theta)\\\
{\bf\tilde{u}_{r}}=(-\cos\theta,\sin\theta)\\\
{\bf\tilde{u}_{t}}=(\sin\theta,\cos\theta)\\\ {\bf
H}=\left(\sqrt{1-\left(\frac{df_{0}^{0}}{d\theta}\right)^{2}},\frac{df_{0}^{0}}{d\theta}\right)\end{cases}$
(9)

To illustrate the nature of the perturbative solution presented in Eq. (8) it
is interesting to consider simple models of perturbation of the circular
contour. One model of particular interest is the case where the function $G$
is a polynomial function of the coordinate of order $n$. The lowest order of
interest is $n=2$, the polynomial terms of order two are, $x_{S}^{2}$,
$x_{S}y_{S}$, $y_{S}^{2}$. As an example, the perturbation associated with the
$x_{S}^{2}$ term is,

$\begin{cases}G(x_{s},y_{s})=x_{S}^{2}\\\ \delta
f_{1}=-\frac{1}{2}\frac{\tilde{d}f_{0}}{d\theta}\sin 2\theta\\\ \delta
f_{0}=\left(\frac{df_{0}^{0}}{d\theta}\right)^{-1}\frac{1}{4}\left(\cos
2\theta+1\right)-\frac{1}{2}\left(\frac{df_{0}^{0}}{d\theta}\right)\cos
2\theta\end{cases}$ (10)

In the Fourier series expansion of the fields it is apparent in Eq. (10) that
the leading term is,
$\frac{df_{0}^{0}}{d\theta}\sin\left(2(\theta+\zeta)\right)$, with $\zeta=0$
or $\frac{\pi}{2}$. An elliptical model aligned with the coordinates system
axis corresponds to $G(x_{S},y_{S})=\eta\left(y_{S}^{2}-x_{S}^{2}\right)$. For
this particular elliptical model we retrieve the equation for the elliptical
contour in (Alard (2007)) when an expansion in first order in $\eta$ is
performed. In the same way an order $n$ polynomial in $(x_{S},y_{S})$ the
leading term $Q$ in the Fourier series expansion of the fields is,

$\begin{cases}G(x_{s},y_{s})=x_{S}^{m}y_{S}^{n-m}\\\
Q_{n}\propto\left(\frac{df_{0}^{0}}{d\theta}\right)^{n-1}\cos\left(n\theta+\zeta\right)~{}{\rm
with}~{}\zeta=0,\frac{\pi}{2}\end{cases}$ (11)

We define the order of the Fourier series expansion of the unperturbed field
$\frac{df_{0}^{0}}{d\theta}$, $n_{0}$. Then it is interesting to note that the
order $N_{F}$ of the Fourier expansion according to Eq. (11) is,

$N_{F}=(n-1)n_{0}+n$ (12)

It is clear that from Eq. ( 12) for $n\geq 2$, which is the lowest order of
source perturbation (ellipticity) the order of $Q_{n}$ is greater than
$n_{0}$. This implies clearly that for $n\geq 2$ the perturbative expansion
Fourier series order is larger that the order of the original unperturbed
term. The obvious consequence is that if the solution for the source is a
circular contour, considering a non-circular contour with order beyond $n=2$
(elliptical) will induce higher order terms in the solution for the fields
$\tilde{f}_{1}$ and $\frac{d\tilde{f}_{0}}{d\theta}$. Note that the order of
the Fourier expansion increase very quickly. Even for the lowest order source
perturbation ($n=2$), and for the lowest order lens model, ($n_{0}=2$), we go
from order 2 to order 4 in the Fourier expansion of the fields which is a very
significant increase (see Eq. 12). Considering perturbation of the source of
higher order will only increase further the higher order terms if the
expansion of the fields. It is also true that if the best solution is some
perturbed non-circular contour, a solution for the fields with a circular
contour will contain higher order terms. This is simply because an elliptical
contour solution is a circular contour solution plus a higher order
perturbation. Let $S_{1}$ be the true, most minimal non circular solution,
$C_{0}$ the circular solution, and $T_{1}$ the higher order terms associated
with the first order expansion. Since $S_{1}$ is a minimal non-circular
solution, then due to Eq. (13) the circular solution will contain additional
higher order terms.

$S_{1}=C_{0}+T_{1}\Rightarrow C_{0}=S_{1}-T_{1}$ (13)

Similarly if we were trying to make a model of a non-circular contour with a
different non-circular contour the effect would be additional higher order
terms in the Fourier series of the fields. Here this result is simply the
consequence that a non-circular solution is a circular solution plus
additional higher order terms, and that the difference between two non-
circular solutions is the difference between two circular perturbed models. In
similarity to Eq. (13) if $S_{2}$ is a non-circular, non-optimal solution with
associated terms $T_{2}$ we obtain,

$S_{2}=S_{1}+T_{2}-T_{1}$ (14)

It is clear from Eq. (14) that the solution $S_{2}$ contains the additional
higher order terms $T_{2}-T_{1}$ with respect to the optimal solution $S_{1}$.
As a consequence for a source model which is a general perturbation to the
first order of the circular source model, the proper solution is the solution
with the minimum degree expansion of the fields.

##  3 Second order expansion

In this section we will demonstrate that the results obtained in the former
section are still valid not only for nearly round models but also for more
distorted models. To prove this result we will study more elongated contours
by pushing the expansion to order 2. We modify Eq. ( 5) by expanding the
perturbative fields to order 2, leading to Eq ( 15).

$\begin{cases}\tilde{f}_{1}=f_{1}^{0}+\epsilon\delta f_{1}+\epsilon^{2}\delta
f_{1,2}\\\
\frac{d\tilde{f}_{0}}{d\theta}=\frac{df_{0}^{0}}{d\theta}+\epsilon\delta
f_{0}+\epsilon^{2}\delta f_{0,2}\end{cases}$ (15)

By using Eq’s ( 4), ( 6), and ( 15) it is possible to calculate the
perturbative solution to order 2. The calculations are heavier than for the
first order expansion and we will analyse the solution main properties and
behavior at higher order. The main result is that the second order terms,
$\delta f_{1,2}$ and $\delta f_{0,2}$, are of higher order than the first
order terms except when the perturbation to the contour is of the lowest order
($n=2$). For polynomial functionals of order $n$ corresponding to
$G(x_{x},y_{s})=x_{s}^{m}y_{s}^{n-m}$, and for $n>2$ the second order terms
are of higher order than the first order terms. To illustrate the structure of
the second order terms we express the solution for $n=2$,

$\begin{cases}G(x_{s},y_{s})=x_{S}^{2}\\\ \delta
f_{1,2}=-\frac{df_{0}^{0}}{d\theta}^{-1}\left[\left(\frac{1}{4}\frac{df_{0}^{0}}{d\theta}^{2}+\frac{1}{8}\right)\sin
2\theta+\frac{1}{16}\sin 4\theta\right]\\\ \delta
f_{0,2}=\frac{df_{0}^{0}}{d\theta}^{-3}\left[\left(-\frac{1}{64}\right)\cos
4\theta+\left(-\frac{1}{16}+\frac{1}{8}\frac{df_{0}^{0}}{d\theta}^{2}-\frac{1}{4}\frac{df_{0}^{0}}{d\theta}^{4}\right)\cos
2\theta-\frac{3}{64}+\frac{1}{8}\frac{df_{0}^{0}}{d\theta}^{2}+\frac{1}{8}\frac{df_{0}^{0}}{d\theta}^{4}\right]\end{cases}$
(16)

For $n>2$ the leading higher order therm $Q_{n}$ is,

$Q_{n}=\cos(2n\theta+\zeta)\frac{df_{0}^{0}}{d\theta}^{2n-3}$ (17)

As a consequence the expansion at order two is at least of the order of the
first order expansion. This result demonstrates the existence of higher order
terms for more elongated sources and generalized the result obtained for first
order small distortions of nearly circular sources.

##  4 Considering minimal solutions help breaking the lens source degeneracy.

If we consider a gravitational arc which is the image of an elliptical source
it is definitely possible to make a model of this arc with a circular source.
However as we have seen in the former sections (see Sec. 2 & Sec. 3) a change
in the source model will produce a number of higher order terms in the
expansion of the fields. As a consequence if we make a minimal model of the
arc we should retrieve the elliptical model and not the circular one which
would require many higher order Fourier terms in the expansion of the fields.
This is confirmed by analyzing the singular perturbative solution for an
elliptical source either from Eq. (15) in Alard (2007) or here in Eq. (8). In
effect by developing to the first order in the ellipticity of the source we
find that there is very little effect of the ellipticity of the source on the
expansion of the fields. For instance if we make a model of the unperturbed
circular model with a Fourier expansion at order 3 we find that to the first
order there is no terms corresponding to the ellipticity of the source in the
perturbed fields. The coupling between the ellipticity of the source and the
field expansion occur only when the potential has effectively significant
fourth order Fourier terms. However it is important to note that in practical
cases the higher order terms in the expansion of the potential have quickly
decreasing amplitude (see Alard (2009) Alard (2010) Alard (2017)) and that as
a result the coupling of the field expansion with source ellipticity should be
small. These results are illustrated by taking a simple lens model, an
elliptical isothermal lens potential and a cusp configuration (see Fig.
1).Basically the elliptical isothermal potential we choose is well described
by a second order Fourier expansion in the singular perturbative model. This
potential has only very weak terms beyond order 2. We then make the following
experiments, first we reconstruct the source by transforming the arc to the
source plane coordinates by using the true lens potential (see Fig. 2), and
second we reconstruct the lens potential by fitting a circular source model
and transform to source coordinates using this second potential (see Fig. 3).
The potential that we fit in the second reconstruction with a circular source
model is minimal which means that the expansion is limited to order 2 . It is
clear from Fig’s 2 and 3, that the difference between the two reconstructions
are very weak, which proves that there is very little coupling between the
source model and the potential. The difference in the axis ratio between the
two source reconstructions is less than one percent of the ellipticity. The
difference between Fig’s 2 and 3 are due to weak higher order terms in the
expansion of the fields which were not taken into account in the expansion
which is limited to order 2. The same analysis can be conducted for higher
order source distortion. The same calculations were performed by replacing the
elliptical source model by a source distorted by fourth order quartic terms.
We choose the following model for the source (see Eq. 4),

$G(x_{s},y_{s})\propto x_{s}^{4}$ (18)

With the source model of Eq. LABEL:Eq_G4 as before we reconstruct the source
by transforming to source coordinates using the true potential (see 4), and
for comparison we reconstruct the lens potential to order 2 using a circular
source model and use this potential to reconstruct the source by moving to
source coordinates (see Fig. 5). This time we observe some difference between
the two reconstructions of the source. However it is important to note that
the differences are not large and that the general properties of the source
are close in each case. For instance the difference between the coefficients
of the fourth order moment of the source in each case is of only of about five
percent. This is larger than what we observed for an elliptical source but
this is still quite close which shows that the coupling is not very large
between the source model and the field reconstruction even for higher order
source distortion. This discussion shows that in general we expect that the
coupling between the source model and the potential model will be quite small.
As a consequence it is clear that using an iterative method where the first
step is to make a circular source model estimate the potential and then
estimate the source, and then using the improved source model and modeling the
potential and so on will lead to a quick convergence towards the right source
lens solution. Provided of course that the potential model is minimal and does
not include un-necessary spurious higher order terms. Here we see all the
interest of the minimal solution for the perturbative fields. The numerical
experiments we described are designed to analyze the effect of potential
modeling on the source reconstruction with maximum accuracy, and as a
consequence do not include noise in the image of the gravitational arc. It is
interesting to make new numerical experiments with realistic noise and
sampling to see if the results obtained with the former numerical experiments
still holds in the presence of noise. To investigate this issue the same cusp
configuration was simulated but this time with Poisson noise and some under-
sampling in order to be consistent with real data (see Fig. 6). For this
simulation process an elliptical source was used. The simulation process
include a total of 1000 simulations. For each simulation a new image image is
simulated with a given expectation for the Poisson noise. Then a circular
source model with an order 2 potential is fitted to the data. As before the
fitting processes is performed in the multi parameter space using a
minimization method based on the amoeba method. To initiate the parameter
search the guess is the true solution for the fields plus a random fluctuation
of the parameters which is of the order of a fraction of the mean value of the
parameters. The fluctuation in the guess is performed to analyze the
sensitivity of the minimization process to the guess. The results presented in
Fig. (7) shows the stability of the solution and the general convergence of
the minimization process to a small area of the parameter space. In Fig. (7)
the stability of the solution and the general error is analyzed by evaluating
the axis ration of the reconstructed source. A small bias of the order of 1 %
is observed in the mean value of the axis ratio, while the scatter is of about
the same order. The result of these simulations show that once again the
source and potential model are de-coupled even in the presence of noise,
provided that a minimal model is used for the perturbative fields.

Figure 1: The contours of the image of the arc used for the numerical
experiments. This image corresponds to a cusp configuration for a an
elliptical isothermal potential. This arc is the image of an elliptical
source. This image is represented on a well sampled grid with no noise, it is
designed to reconstruct the source with maximum accuracy, it is not intended
to represent real conditions. The image presented if Fig. (6) is more
representative of real conditions. Figure 2: The reconstruction of the source
corresponding to the arc of Fig. (1). The source is reconstructed by re-
mapping the lens coordinates to the source coordinates using the lens
equation. The potential used in the lens equation corresponds to the true lens
potential to order 2 in the Fourier expansion of the fields. Figure 3: Same
reconstruction as Fig. (2) but this time the potential used in the lens
equation is estimated by fitting a circular source model to the data presented
in Fig. (1). The circular source model is adjusted by using a minimization
procedure in the parameter space. The minimization method is based of the
Amoeba method. The starting guess for the minimization is the true potential.
Some experiments were performed by adding random fluctuation of the initial
guess. These experiments show that the convergence to the solution is stable
and does not depend on the initial guess. Figure 4: Same as Fig. (2) except
for a change in the model of the source. In this case the source contour is
distorted by a fourth order term (See Eq. (LABEL:Eq_G4)). Figure 5: Same as
Fig. (3) except for a different source model. The source model is defined in
Eq. (LABEL:Eq_G4).

Figure 6: The configuration described in Fig. (1) represented in a coarser
grid with the addition of Poisson noise. The under-sampling and noise of this
new image are more representative of real data that Fig. (1). Figure 7:
Histogram of the relative axis ratio for 1000 simulations for the under-sample
arc with added Poisson noise presented in Fig. (6). The relative axis ratio
corresponds to the ratio between thes estimated axis ratio and the true axis
ratio. For each simulation the source is reconstructed with a potential
obtained by fitting a circular source model. The fitting procedure is
identical to the procedure used in the former numerical experiments. The
initial guess for the fitting procedure is different for each simulation. This
initial guess is obtained by adding a random fluctuation to the true potential
for each of the Fourier coefficients of the expansion of the perturbative
fields. The amplitude of the random fluctuation is a fraction of the mean
amplitude of the Fourier coefficients of the true potential Fourier expansion.

##  5 Conclusion.

In this paper new approach to the lens source degeneracy is proposed by
considering minimal lens solutions. In the perturbative approach the minimal
solution corresponds to the minimum degree in the Fourier expansion of the
perturbative fields. In the modeling of gravitational lenses it is essential
to consider these minimal solutions. The minimal solutions are a well defined
fixed point in the space of solutions. Trying to represent the potential of
gravitational lenses with unnecessary higher order expansions leads to a
quickly increasing number of spurious higher order terms. This is due to the
fact that in response to a small change of the source model from the model
corresponding to the optimal solution the perturbative response in the lens
solution grows very quickly, leading to a high number of higher order terms.
These higher order terms do not have any physical meaning or any value in the
interpretation of the lens model. Another very important asset of the minimal
solutions is that in general they offer some de-coupling between the source
and lens solution. As discussed in Sec. (4) the terms associated with the
distortion from circularity in the source model have only a weak influence in
the perturbative field model provided that the model for the fields is
minimal. In Sec. (4) a simple model with dominant order 2 term was explored.
We may ask now what would be the situation for a more complex lens model ?
This problem was discussed in Sec. 4) and the idea is that even if the lens
model contains significant terms beyond order 2 in practice these terms will
be of decreasing amplitude and thus should not have a major influence on the
lens source coupling. The only case where higher order Fourier terms can be
large is when the potential is perturbed locally with for instance the
presence of a substructure (see Alard (2008)). But it is clear that in the
case of a local structure the higher order terms are significant only in the
vicinity of the substructure, and that most of the arc will not require these
high amplitude higher order terms for its modeling. As a consequence we expect
that even in this case the lens source problem will be de-coupled to some
degree. The problem now is to discuss what is a minimal model for a given
gravitational arc. This minimal model should be explored carefully and
individually for each gravitational arc system. It is of crucial importance to
look for solutions with the least possible number of terms in order to avoid
introducing spurious terms in the solution. When reconstructing the solution
it is essential to ensure that at the lowest order the best minimization has
been achieved before any additional higher order terms are introduced. At
least in the beginning of the process it is essential to explore the most
minimal solution possible in order to ensure a proper de-coupling of the
source and the lens. This means that the minimization process and search for
the best solution at the lowest order for the potential has to explore
thoroughly the parameter space and ensure that the absolute minima for a given
order of the Fourier expansion has been found. Once a convergence has been
obtained for the most minimal model, only then can the fitting procedure can
be extended to include higher order terms. It is clear also that the search
for higher order terms should be done in the vicinity of the most minimal
former solution to avoid reaching a spurious higher order solution. The
additional higher order terms should be minimal or have the least possible
amplitude. A possible drawback of enforcing the minimal solution is that the
solution may be biased and minimize the higher order terms. But this drawback
is also an advantage since in the minimal approach if higher order terms are
found in the solution it means that these terms are real and not spurious
oscillations of the solution. This point is of special importance if we
consider the problem of substructure in dark halos. Since the presence of
substructures is related to the existence of of higher order terms in the
solution (see Alard Alard (2009)), it is of special importance to look for the
minimal solution. A statistical analysis based on minimal solutions will
ensure that the higher order signal present in the statistics of the solutions
is real and an effect of spurious terms. The same is true for other
manifestations of higher order terms like the complex geometry of the dark
matter halos which as a consequence may not follow the distribution of light
(see Alard (2010)). As a conclusion it is clear that looking for the minimal
solution should be the standard procedure in the reconstruction of
gravitational lenses in the singular perturbative approach. To conclude it is
clear that the minimal solutions have very interesting and promising
properties for the reconstruction of gravitational lenses. However more work
is required to explore the properties of these solutions in practical
applications to evaluate how the minimality of the solution can be enforced
for various lens systems. The problem of the practical application of the
method will be explored in more detail in a forthcoming paper.

## Data Availability

No data sets were generated or analyzed during the current study.

## References

  * Agnello etal. (2013) Agnello, A., Auger, M. W., Evans, N. W., 2013, MNRAS, 429L, 35
  * Alard (2007) Alard, C., 2007, MNRAS, 382, L58
  * Alard (2008) Alard C., 2008, MNRAS, 388, 375
  * Alard (2009) Alard C., 2009, A&A, 506, 609
  * Alard (2010) Alard C., 2010, A&A, 513, 39
  * Alard (2017) Alard C., 2017, MNRAS, 467, 1997
  * Brewer etal. (2016) Brewer, Brendon J.; Huijser, David; Lewis, Geraint F., 2016, MNRAS, 455, 1819
  * Meneghetti etal. (2006) Meneghetti, M., Bartelmann, M., Moscardini, L., 2003, MNRAS, 340, 105
  * Saha & Williams (2006) Saha, P., Williams, L., 2006, ApJ, 653, 936
  * Warren & Dye (2003) Warren, S., & Dye, S. 2003, ApJ, 590, 673