# Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large
Language Models

Nishanth Madhusudhan  ServiceNow<EMAIL_ADDRESS>,
Sathwik Tejaswi Madhusudhan  ServiceNow
<EMAIL_ADDRESS>,  Vikas Yadav  ServiceNow
<EMAIL_ADDRESS>and  Masoud Hashemi  ServiceNow
<EMAIL_ADDRESS>
(2024)

###### Abstract.

As Large Language Models (LLMs) achieve remarkable performance across various
NLP tasks, their reliability becomes essential for widespread adoption. This
paper focuses on _Abstention Ability_ ($\mathcal{AA}$), a critical yet
underexplored aspect of reliability - the ability of LLMs to refrain from
answering questions when they are uncertain or when a definitive answer is not
possible, while maintaining question-answering (QA) task performance. While
previous works have focused on understanding the recollection abilities of
LLMs or their ability to identify imponderable/unanswerable questions, we
believe there is a need for an effective $\mathcal{AA}$ evaluation method.
Therefore, we propose a black-box evaluation methodology to examine and
understand the $\mathcal{AA}$ of LLMs across a variety of multiple-choice QA
tasks. We measure $\mathcal{AA}$ by rewarding models for abstaining from
answering when their predictions are incorrect or when the questions are
inherently unanswerable. We investigate three strategies, Strict Prompting,
Verbal Confidence Thresholding, and Chain-of-Thought (CoT), to understand
their impact on abstention across different LLMs. Our findings reveal that
while even state-of-the-art LLMs like GPT-4 struggle with abstention,
strategic prompting, such as CoT, can significantly enhance this ability.
Furthermore, we demonstrate that improving $\mathcal{AA}$ also leads to better
overall QA task performance, underscoring the importance of evaluating
$\mathcal{AA}$ in LLMs.

Large Language Model, Abstention Ability

††copyright: acmcopyright††journalyear: 2024††doi: XXXXXXX.XXXXXXX††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Natural language
generation

##  1\. Introduction

Figure 1. (A) We introduce ‘Answerable-Unanswerable Confusion Matrix (AUCM)’
as a tailored approach to accurately quantify a model’s abstention ability
(see section 3.3). This matrix contrasts the types of model predictions (model
answered or abstained) with the nature of the questions (answerable or
unanswerable), to capture all potential outcomes. (B) To effectively evaluate
abstention ability, we assess the model in different categories of ‘question
types’, ‘domains’ or ‘data domains’, and ‘task types’. The selection of these
categories aims to challenge the model across diverse scenarios, testing its
robustness and adaptability in handling different types of information and
cognitive demands.

Large Language Models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Jiang
et al., 2024) have demonstrated impressive capabilities in various NLP tasks;
however, their reliability is paramount for practical use (Weidinger et al.,
2022; Lin et al., 2021), especially in critical domains like legal, medical,
security and alike (Xu et al., 2024a; Li et al., 2023a). It’s essential for
LLMs to possess effective Abstention Ability ($\mathcal{AA}$), as withholding
answers when lacking confidence or uncertain is preferable over dispensing
incorrect information, thereby improving their reliability. Despite previous
studies on LLMs’ abstention capabilities (Yin et al., 2023; Yang et al., 2023;
Balabanov and Linander, 2024), there is no standardized methodology to
evaluate $\mathcal{AA}$.

Previous approaches for improving $\mathcal{AA}$ in LLM responses include
prompting (Xiong et al., 2023; Yin et al., 2023) to abstain (e.g., respond
with “I don’t know”) and fine-tuning (Chen et al., 2023) to detect uncertain
responses. Popular prompting techniques such as chain of thought prompting
(CoT) (Wei et al., 2022), self-consistency prompting (Wang et al., 2022), and
chain of density prompting (Adams et al., 2023) improve task performance by
encouraging step-by-step thinking, generating multiple outputs, or refining
outputs iteratively, but do not directly address the issue of dispensing
erroneous information. Context-faithful prompting (Zhou et al., 2023) is an
example of prompt-based abstention using opinion-based prompts to reframe the
context as a narrator’s statement.

Soft prompt fine-tuning is used in (Chen et al., 2023) to enable LLMs to
distinguish between correct and incorrect answers. Post-abstention (Varshney
and Baral, 2023) uses a trained abstention model to improve the coverage
without degrading the accuracy by re-examining the abstained instances.
Quantifying performance in these studies involves F-score which does not
account for abstention or the coverage-accuracy curve that needs continuous
thresholding which is not possible for black-box LLMs. An alternative approach
for selective response generation is assessing uncertainty in LLM responses,
achievable through calibration and uncertainy-quantification techniques to
measure uncertainty (Duan et al., 2023; Azaria and Mitchell, 2023; Xu et al.,
2024b; Gui et al., 2024) and self-detecting techniques (Xiong et al., 2023)
where LLMs are prompted to elicit expressions of uncertainty. These methods
are typically evaluated using AUROC, which treats certain/uncertain as a
binary score but fails to capture the interplay between a model’s ability to
abstain and the correctness of its responses.

In this work, we propose an effective $\mathcal{AA}$ evaluation methodology.
We present a new evaluation dataset, Abstain-QA which is a collection of
diverse MCQA datasets (see figure 1-(B)), and formulate a metric called
Abstention-rate ($\mathcal{AR}$), to quantify the rate of abstention of LLMs.
To evaluate the effect of prompting on $\mathcal{AA}$, we explore three
different techniques applicable to any LLM including black-box models: Strict
Prompting, Verbal Confidence Thresholding and Chain-of-Thought.

Abstain-QA is designed based on the categories depicted in figure 1-(B) and
includes three diverse MCQA datasets with constrained output spaces: 1) Pop-QA
(Mallen et al., 2023), featuring questions on popular and unpopular simple
named entities, 2) MMLU (Hendrycks et al., 2020), testing specific subject
knowledge, and 3) CarnaticQA (CQA), assessing deep knowledge in the under
represented domain of Carnatic Classical Music. Each dataset includes “none-
of-the-above” as an additional option, and for half of the questions the
correct answers are omitted, rendering them unanswerable. Abstention-rate
quantifies $\mathcal{AA}$ by rewarding LLMs for abstaining in two key
scenarios: 1) when the question is answerable but the LLM’s are uncertain and
are prone to err, and 2) when the question is unanswerable (figure 2). Our
detailed study applies the proposed $\mathcal{AA}$ evaluation strategy to five
state-of-the-art LLMs from two different model families, i.e., OpenAI (Achiam
et al., 2023) and Mistral (Jiang et al., 2024). Our results reveal that these
state-of-the-art LLMs show poor $\mathcal{AA}$, raising concerns about their
use in sensitive domains like legal, medical and so on (Weidinger et al.,
2022). By implementing the three $\mathcal{AA}$ improvement techniques, we
show that enhancing $\mathcal{AA}$ not only boosts reliability by improving
Precision $\mathcal{P}$ but also increases overall QA task performance. It
should be noted that our abstention analysis methodology can be extended to
other NLP tasks and other properties, such as hallucination, and safety
(Tomani et al., 2024). Key contributions of this work are as follows:

(1)

We present an evaluation methodolody and detailed study evaluating
$\mathcal{AA}$ of LLMs by measuring Abstention-rate, a simple formulated
metric to quantify the rate of Abstention by leveraging the Answerable-
Unanswerable Confusion Matrix (section 3.3).

(2)

We introduce Abstain-QA, a diverse collection of 2900 Zero Shot MCQA samples.
This also includes Carnatic-QA, a dataset of 900 factoid and conceptual MCQA
questions, specifically created as part of this work (section 2).

(3)

We demonstrate that LLMs exhibit varying $\mathcal{AA}$ performance depending
on the question type. While they excel at abstaining on simple factoid-based
MCQs, their $\mathcal{AA}$ significantly deteriorates when requiring
reasoning, problem-solving, or when dealing with under-represented data.

(4)

Finally, we show that the use of techniques like Strict Prompting, and Chain-
of-Thought enhances $\mathcal{AA}$ of LLMs. Importantly, improvements in
$\mathcal{AA}$ also leads to improvement in the overall QA task performance,
further highlighting benefits of evaluating $\mathcal{AA}$.

##  2\. DATASET CONSTRUCTION

Figure 2. This highlights the LLM’s ability to avoid guessing when the answer
is unclear. On the left, the model incorrectly selects “Gemma Hayes” for an
unanswerable question (none of the options are correct). On the right, the
model correctly chooses “None of the Above” when instructed to abstain if
uncertain, i.e., FP changes to TN.

We propose using multiple-choice questions (MCQs) to evaluate the
$\mathcal{AA}$ of LLMs. Our approach involves constructing a dataset
comprising of an equal split between answerable and unanswerable questions,
both featuring an explicit “I Don’t Know/None of the Above” (IDK/NOTA) option
where the IDK/NOTA option serves as the correct response for unanswerable
questions. Following this recipe, we introduce “Abstain-QA” - a comprehensive
MCQ dataset with 2900 samples and five options for each sample - to evaluate
the $\mathcal{AA}$ of LLMs. We source these samples from Pop-QA (Mallen et
al., 2023), MMLU (Hendrycks et al., 2020), and our newly created dataset
Carnatic-QA. The unanswerable questions are crafted by replacing the correct
option with a relevant but incorrect one. Abstain-QA covers a diverse range of
QA tasks and categories, shown in figure 1-(B), from simple fact-based
questions to logical and conceptual reasoning and problem-solving, featuring
samples from both well-represented and under-represented domains. This
diversity allows for a thorough analysis of the LLMs’ abstention ability.

Pop-QA, comprises 14000 QA pairs from 16 diverse relationships, such as
Occupation, Producer, Composer, etc. Each sample includes the object entity,
subject entity, and the Wikipedia monthly page view for both entities, which
we use to ensure equal splits of well and under-represented entity-based
questions. We sample 100 questions from a subset of 10 relationships for
analysis.

MMLU, contains 15,908 MCQA pairs across 57 subjects. 1000 MCQA pairs are
chosen, with 100 questions from 10 subjects that best represent Problem-
Solving, Logical Reasoning, and Fact-based QA tasks. For example, High School
Mathematics, Professional Psychology, Virology, Management etc.

Carnatic-QA (CQA), Carnatic Music Raga (Krishna and Ishwar, 2012; Samsekai
Manjabhat et al., 2017) (akin to a scale in Western Music) recognition is a
popular and widely studied task in music information retrieval (Gulati et al.,
2016; Madhusudhan and Chowdhary, 2019; Sridhar and Geetha, 2009). Since
Carnatic music is an under-represented domain requiring subject matter
expertise, we believe it is a strong candidate for testing $\mathcal{AA}$ in
LLMs. To create CQA, we start with a web-scraped list of 930 known Carnatic
Music Ragas from Wikipedia (Wikipedia contributors, 2024) and ‘Grammar of
Carnatic Music’ (Vijayakrishnan, 2007). We then reduce this list to 272 ragas
with the assistance of two expert annotators, who are well versed in Carnatic
Music. Next, we leverage the theoretical aspects of Carnatic Music Raga
recognition to create distinct question generation templates and then generate
nine unique tasks with 100 questions each by inserting raga names in the
question generation templates. Three expert Carnatic musician volunteers
assess the data quality, with consensus achieved through majority voting. All
annotators and volunteers are full-time workers compensated according to local
wages.

##  3\. Evaluation methodology

Each sample in Abstain-QA has three parts: Task Prompt ($\phi$), Abstain
Clause ($\alpha$), and Ground Truth ($y$). $y$ is the expected answer for each
question. The model ($\mathcal{F}$) prediction,
$\hat{y}=\mathcal{F}(\phi,\alpha)$, is a function of the $\alpha$ and $\phi$.
$\phi$ contains the question to be answered, the options to select from, and
the output formatting requirements. $\alpha$ defines the sensitivity to
uncertainty and abstention. Our evaluation methodology evaluates the effect of
$\alpha$ and $\phi$ on the abstention ability of LLMs in a black-box setup,
with $y$ remaining the same.

###  3.1. Abstain Clause Variations

A key feature of our experimentation, ‘Abstain’ Clause ($\alpha$), can be of
three types: (1) Standard Clause serves as a baseline, where the model is not
explicitly instructed to abstain but is shown an IDK/ NOTA option. (2) In
contrast, the Abstain Clause ($\mathcal{AC}$) introduces a mechanism to
encourage the model to refrain from answering when uncertain, by prompting the
potential negative consequences of incorrect responses. (3) We also introduce
an Extreme Abstain Clause ($\mathcal{EAC}$), inspired by findings that LLMs
respond to emotional stimuli (Li et al., 2023b), which exerts even more severe
pressure on the model. By incorporating these three clauses, each sample is
expanded into three sub-samples, resulting in three distinct model
predictions: $\hat{y}_{s}$ (Standard), $\hat{y}_{abs}$ (Abstain), and
$\hat{y}_{eabs}$ (Extreme Abstain).

###  3.2. Task Prompt Variations

Three-pronged experimental setups are defined based on the task prompt $\phi$
definition: ‘Base’, ‘Verbal Confidence’ and ‘Chain of Thought’.

(1) Base Experiment, the task prompt solely defines the MCQ the model needs to
answer.

(2) Verbal Confidence Experiment, we extend the Base experiment’s setup by
including a Verbal Confidence clause within $\phi$. The clause instructs the
models to self-assess their confidence in their predictions and to provide a
confidence score, along with their answer, ranging from 1 (‘Least Confident’)
to 5 (‘Extremely Confident’). We then utilize these scores to implement
Confidence Thresholding (detailed in section 3.3).

(3) Chain of Thought (CoT) Experiment, multi-step reasoning behavior and CoT
prompting significantly improves task performance in LLMs (Wei et al., 2022).
Inspired by this, we incorporate CoT prompting to evaluate $\mathcal{AA}$ in
LLMs by extending the Base experiment’s configuration and introducing a CoT
clause within $\phi$. This mandates the models to verbalize their thought
process step-by-step, leading up to their response to a given question.

###  3.3. Evaluation Metrics

We use the Answerable-Unanswerable Confusion Matrix (AUCM) to compute the
metrics for the observed predictions (see figure 1-(A)). As illustrated in the
AUCM, the MCQs are either Answerable or Unanswerable (detailed in section 2)
and the LLM prediction is either an abstention option, i.e., IDK/NOTA, or a
candidate option. All answerable samples are considered positive as their
ground truth is a candidate option and all unanswerable samples are considered
negative as their ground truth is the ‘IDK/ NOTA’ option. For Answerable
questions, if the model prediction is a candidate option, and is correct it
creates True Positives ($TP$) whereas if is incorrect it creates False
Positives ($FP$). Abstaining to answer an Unanswerable question generates True
Negatives ($TN$) whereas failing to abstain from the same would generate $FP$.
If the LLM abstains for an answerable question, it creates False Negatives
($FN$). To quantify how often a model abstains, we define a new metric, called
Abstention Rate ($\mathcal{AR}$):

(1) $\mathcal{AR}=\frac{FN+TN}{|\mathcal{D}|}$

where $|\mathcal{D}|$ is the number of QAs in the dataset. Moreover, we define
the Answerable Accuracy, $\mathcal{AAC}$, measuring the accuracy of correct
option selection in answerable QAs and Unanswerable Accuracy, $\mathcal{UAC}$,
measuring the accuracy of abstention in unanswerable QA:

(2)
$\mathcal{AAC}=\frac{TP}{|\mathcal{A}|},\mathcal{UAC}=\frac{TN}{|\mathcal{U}|}$

where $|\mathcal{A}|$ is the number of answerable QAs and $|\mathcal{U}|$ is
the number of unanswerable QAs. We also use $\mathcal{P}=\frac{TP}{TP+FP}$ in
our evaluations. The combination of $\mathcal{AAC}$, $\mathcal{UAC}$ and
$\mathcal{AR}$ depicts an accurate picture of $\mathcal{AA}$ while precision
($\mathcal{P}$) highlights the user experience/ reliability of the LLMs. As
the model abstains, it is natural for $FN$ to increase thereby reducing
$\mathcal{AAC}$ but the goal to achieve effective $\mathcal{AA}$ is to
maximize $\mathcal{UAC}$ (higher $TN$) and $\mathcal{P}$, while keeping $FN$
to a minimum thereby maintaining/ improving $\mathcal{AAC}$.

To calculate the metrics in ‘Verbal Confidence Experiment’, Confidence
Thresholding is used based on the confidence score generated by the LLMs. We
posit that any prediction with a low confidence level can be treated as
abstained and hence any prediction, irrespective of which option it
represents, with a confidence score either less than or equal to the
confidence threshold will be abstained i.e., converted to the ‘IDK/ NOTA’
option. We experiment with confidence thresholding at four levels (1 to 4) and
assess performance using the four above-mentioned metrics.

##  4\. Results and Discussion

We use five state-of-the-art LLMs, namely, GPT-3.5-Turbo (OpenAI, 2023),
GPT-4-Turbo (OpenAI, 2024), GPT-4-32k (Achiam et al., 2023), Mixtral 8x7b
Instruct (Jiang et al., 2024) and Mixtral 8x22b Instruct (team, 2024).
Throughout our work, we refer to them as GPT-3.5 Turbo, GPT-4 Turbo, GPT-4
32K, Mixtral 8x7b and Mixtral 8x22b respectively. For each model, we conduct
the three experiments (see section 3), on all datasets (see section 2). All
experiments are performed under a zero-shot setting and three predictions
($\hat{y}_{s}$, $\hat{y}_{abs}$, $\hat{y}_{eabs}$) are generated for each
question. Note that, the verbal confidence results are reported with a
confidence threshold of three, as it draws out the best performance from all
models111We saw similar trends with verbal confidence threshold of 2 and 4..
Table 1 shows a summary of the evaluation results from our experiments for
CQA, MMLU, and PopQA datasets and every set of four rows showcases
$\mathcal{P}$, $\mathcal{AAC}$, $\mathcal{UAC}$ and $\mathcal{AR}$.

Clause |  | Base | Verbal Confidence (Threshold = 3) | CoT
---|---|---|---|---
Type | Metric | GPT-4 | GPT-4 | GPT-3.5 | Mixtral | Mixtral | GPT-4 | GPT-4 | GPT-3.5 | Mixtral | Mixtral | GPT-4 | GPT-4 | GPT-3.5 | Mixtral | Mixtral
|  | Turbo | 32K | Turbo | 8X7b | 8X22b | Turbo | 32K | Turbo | 8X7b | 8X22b | Turbo | 32K | Turbo | 8X7b | 8X22b
CQA
Standard | P | 30.1 | 28.3 | 14.3 | 15.6 | 20.4 | 29.4 | 29.2 | 13.7 | 16.2 | 18.9 | 43.0 | 37.4 | 13.8 | 17.5 | 24.7
AAC | 48.0 | 49.3 | 28.6 | 28.8 | 32.8 | 46.4 | 48.6 | 26.8 | 28.6 | 30.6 | 40.2 | 43.7 | 21.1 | 26.2 | 28.0
UAC | 32.8 | 22.8 | 0.6 | 10.8 | 27.3 | 33.3 | 26.4 | 2.8 | 12.4 | 24.6 | 67.1 | 56.8 | 25.7 | 26.2 | 39.7
AR | 20.3 | 12.8 | 0.3 | 6.1 | 19.7 | 21.1 | 16.8 | 2.5 | 9.4 | 19.2 | 53.5 | 41.2 | 23.5 | 20.8 | 38.0
Abstain | P | 32.3 | 32.7 | 13.1 | 16.7 | 25.5 | 32.1 | 30.8 | 13.1 | 16.2 | 22.0 | 53.2 | 46.0 | 13.7 | 17.5 | 27.1
AAC | 42.6 | 46.4 | 24.6 | 27.7 | 24.0 | 37.5 | 44.2 | 14.0 | 27.1 | 30.0 | 32.8 | 37.5 | 8.6 | 19.7 | 23.3
UAC | 50.8 | 44.8 | 10.2 | 23.5 | 65.1 | 56.6 | 42.8 | 39.1 | 21.5 | 40.6 | 84.8 | 75.7 | 75.1 | 40.4 | 59.7
AR | 34.0 | 29.1 | 6.1 | 16.6 | 53.0 | 41.5 | 28.2 | 42.8 | 15.9 | 32.0 | 69.1 | 59.2 | 68.4 | 37.4 | 54.5
Extreme-abstain | P | 35.9 | 32.2 | 13.9 | 16.0 | 23.6 | 34.6 | 31.9 | 12.8 | 15.4 | 21.8 | 52.7 | 44.7 | 16.1 | 17.9 | 25.8
AAC | 38.4 | 43.7 | 26.6 | 24.8 | 26.6 | 32.2 | 43.1 | 24.2 | 22.2 | 30.0 | 34.0 | 36.8 | 17.7 | 21.3 | 28.0
UAC | 62.2 | 48.4 | 7.1 | 31.1 | 54.2 | 68.0 | 47.7 | 8.6 | 25.7 | 38.8 | 85.3 | 75.1 | 51.1 | 37.3 | 49.5
AR | 46.5 | 32.2 | 4.2 | 22.2 | 43.6 | 53.4 | 32.5 | 5.7 | 21.5 | 31.3 | 67.7 | 58.7 | 44.8 | 34.9 | 44.4
MMLU
Standard | P | 44.8 | 43.4 | 31.8 | 32.7 | 38.0 | 44.4 | 41.4 | 31.8 | 34.0 | 34.8 | 51.7 | 49.6 | 35.0 | 40.0 | 45.1
AAC | 77.0 | 79.0 | 62.0 | 59.4 | 67.0 | 74.2 | 75.4 | 60.2 | 58.6 | 62.6 | 80.8 | 78.6 | 63.6 | 62.0 | 70.0
UAC | 26.0 | 16.8 | 2.4 | 13.4 | 18.6 | 30.6 | 16.4 | 6.0 | 16.0 | 17.2 | 39.4 | 34.6 | 15.6 | 25.6 | 33.6
AR | 14.2 | 8.9 | 1.5 | 8.9 | 11.9 | 16.3 | 8.9 | 5.0 | 10.5 | 10.2 | 21.7 | 20.0 | 9.0 | 17.2 | 19.6
Abstain | P | 47.5 | 46.0 | 31.4 | 39.4 | 44.5 | 47.5 | 43.5 | 32.5 | 37.9 | 38.6 | 59.7 | 56.1 | 46.5 | 44.4 | 49.6
AAC | 75.0 | 78.6 | 58.2 | 53.8 | 67.2 | 73.0 | 75.0 | 55.6 | 52.6 | 64.6 | 79.2 | 78.2 | 42.8 | 50.2 | 66.4
UAC | 38.6 | 26.4 | 9.2 | 46.2 | 40.0 | 42.8 | 24.8 | 19.0 | 41.0 | 28.6 | 59.0 | 50.8 | 65.4 | 46.8 | 47.8
AR | 20.9 | 14.4 | 7.3 | 31.5 | 24.5 | 23.2 | 13.7 | 14.6 | 30.2 | 16.3 | 33.4 | 30.1 | 53.8 | 36.2 | 31.1
Extreme-abstain | P | 50.2 | 44.6 | 30.5 | 35.7 | 43.1 | 49.6 | 43.4 | 29.7 | 37.6 | 39.1 | 58.7 | 55.1 | 38.3 | 44.1 | 49.2
AAC | 76.0 | 78.0 | 60.2 | 58.6 | 70.4 | 72.0 | 75.8 | 56.2 | 51.4 | 67.0 | 76.4 | 76.6 | 66.0 | 57.2 | 68.8
UAC | 44.2 | 23.4 | 1.6 | 23.8 | 31.4 | 47.4 | 22.6 | 8.0 | 20.0 | 26.2 | 57.4 | 50.8 | 21.2 | 40.8 | 40.2
AR | 24.4 | 12.6 | 1.4 | 16.3 | 18.4 | 27.5 | 12.7 | 5.5 | 15.6 | 14.3 | 34.7 | 30.2 | 13.5 | 30.3 | 26.2
Pop-QA
Standard | P | 84.2 | 79.1 | 52.2 | 61.7 | 65.0 | 88.6 | 84.2 | 62.7 | 71.3 | 65.0 | 94.3 | 92.9 | 59.3 | 72.7 | 81.9
AAC | 91.2 | 94.8 | 96.0 | 91.0 | 92.2 | 90.8 | 93.2 | 90.2 | 88.2 | 92.4 | 89.6 | 89.6 | 89.8 | 84.8 | 87.8
UAC | 83.6 | 76.0 | 16.2 | 47.8 | 53.6 | 89.6 | 83.6 | 48.8 | 66.8 | 54.4 | 95.8 | 94.0 | 42.8 | 69.2 | 82.6
AR | 45.9 | 40.0 | 8.1 | 26.1 | 28.81 | 48.8 | 44.7 | 28.1 | 38.0 | 29.0 | 52.4 | 51.7 | 24.2 | 40.0 | 46.4
Abstain | P | 97.5 | 91.6 | 71.0 | 86.9 | 86.3 | 97.1 | 91.7 | 83.6 | 93.0 | 78.4 | 98.3 | 98.5 | 94.1 | 91.6 | 95.0
AAC | 87.0 | 90.2 | 91.0 | 72.2 | 87.6 | 87.6 | 90.8 | 76.6 | 67.2 | 91.0 | 81.8 | 81.4 | 44.8 | 46.0 | 76.6
UAC | 97.8 | 92.2 | 66.4 | 90.2 | 88.6 | 98.2 | 93.0 | 86.0 | 96.2 | 78.0 | 99.8 | 99.4 | 98.2 | 93.2 | 97.2
AR | 55.4 | 50.8 | 35.7 | 58.5 | 49.3 | 54.9 | 50.5 | 54.2 | 63.9 | 42.0 | 58.3 | 58.7 | 76.2 | 73.8 | 59.7
Extreme-abstain | P | 97.2 | 91.2 | 53.4 | 73.0 | 83.3 | 97.9 | 93.1 | 54.6 | 79.6 | 79.8 | 98.5 | 98.5 | 70.3 | 85.0 | 88.1
AAC | 86.2 | 91.8 | 95.4 | 86.0 | 91.0 | 85.8 | 90.4 | 93.0 | 84.8 | 89.6 | 81.2 | 83.4 | 88.2 | 69.4 | 87.6
UAC | 97.8 | 92.2 | 20.8 | 70.4 | 83.4 | 98.8 | 94.4 | 27.4 | 78.6 | 80.8 | 99.8 | 99.4 | 65.0 | 85.2 | 89.8
AR | 55.7 | 49.7 | 10.7 | 41.0 | 45.4 | 56.2 | 51.5 | 14.9 | 46.2 | 43.9 | 58.7 | 57.6 | 37.3 | 57.4 | 50.2
Table 1.  Evaluation results for CQA, MMLU and PopQA. Best numbers for each
metric for each model (across clause types) in each dataset is highlighted in
bold.

Comparative Analysis on Question Type and Domain -  For Pop-QA, which
comprises of questions based on simple named entities from both well and
under-represented domains, models, especially GPT-4 Turbo and GPT-4 32k,
perform well in abstaining while retaining a high $\mathcal{AAC}$ and
$\mathcal{P}$. However, for MMLU and CQA, which involve more complex reasoning
and questions based on under-represented data respectively, all models show
poorer performance marked by lower $\mathcal{AAC}$, $\mathcal{UAC}$, and
$\mathcal{P}$. In CQA, using CoT significantly increased $\mathcal{AR}$ and
$\mathcal{UAC}$, but $\mathcal{AAC}$ often decreased, indicating a high rate
of abstention in response to answerable questions. We conjecture this happens
because the queries come from under-represented domains, making them less
familiar to the models.

Comparison of Task Prompt Types -  For standard, $\mathcal{AC}$ and
$\mathcal{EAC}$, our findings reveal that CoT outperforms both Base and Verbal
confidence settings in terms of $\mathcal{P}$, $\mathcal{UAC}$, and
$\mathcal{AR}$ across all benchmarks. In the same comparison, CoT boosts
$\mathcal{AAC}$ in all benchmarks except CQA. We posit that this is due to the
under-represented nature of the data from CQA making the models abstain more.
We find that Verbal confidence did not consistently exceed the performance of
the Base setting. This indicates that models struggle to quantify their
uncertainty accurately through verbal confidence outputs. However, GPT-4 Turbo
has a slightly more consistent improvement with verbal confidence, showing
higher capability in expressing uncertainty.

Effects of Enhanced Abstention Mechanisms -  Leveraging $\mathcal{AC}$ and
$\mathcal{EAC}$ generally improves models’ $\mathcal{AA}$, and QA task
performance on all benchmarks measured by $\mathcal{AAC}$, $\mathcal{P}$, and
$\mathcal{UAC}$, when compared with the Standard clause. However, GPT-3.5
Turbo and Mixtral 8x7b showed weaker performance with $\mathcal{EAC}$. We
observe that $\mathcal{EAC}$ generally enhances the performance of larger
models, particularly in Pop-QA and CQA. However, $\mathcal{AC}$ often
outperforms $\mathcal{EAC}$ in smaller or less capable models due to
$\mathcal{EAC}$’s increased complexity, which can hinder these models’
understanding and adherence to instructions.

Overall Performance Trends by Models -  GPT-4 Turbo consistently outperforms
other models, showing superior $\mathcal{P}$, $\mathcal{UAC}$, and
$\mathcal{AR}$. Mixtral 8x22b, while strong in some settings, shows a decline
in performance under increased complexity and verbal confidence experiments.
GPT-3.5 Turbo and Mixtral 8x7b generally lags behind other models,
particularly under enhanced abstention, $\mathcal{EAC}$. CoT consistently
outperforms Verbal Confidence which often lags behind the Base setting in most
scenarios, except with GPT-4 Turbo and GPT-4 32k models. Comparing Standard,
$\mathcal{AC}$, and $\mathcal{EAC}$, smaller models exhibit inconsistent
behavior especially with CQA, and better performance with $\mathcal{AC}$
compared to $\mathcal{EAC}$, except for larger models like GPT-4 Turbo,
demonstrating consistent improvements with $\mathcal{EAC}$. These results
indicate that $\mathcal{AA}$ heavily depends on the LLMs’ capabilities, type
of data and prompt complexity.

##  5\. CONCLUSION

In this work, we explored the Abstention Ability, $\mathcal{AA}$, of LLMs
using our dataset, Abstain-QA. We find that LLMs struggle to abstain from
responding to reasoning, conceptual, and problem-solving questions within
well-represented domains. Moreover, they have difficulty abstaining from
conceptual questions derived from under-represented data. Our observations
show that $\mathcal{AA}$ and QA task performance of LLMs can be significantly
enhanced by employing strategies like strict-prompting and Chain of Thought.
However, further research in this area is necessary to comprehensively
understand and improve the Abstention Abilities of LLMs. We believe that this
is truly an important direction as it would lead to the development of LLMs
and applications that are more reliable in terms of the dispensed information.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Adams et al. (2023) Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. 2023.  From sparse to dense: GPT-4 summarization with chain of density prompting.  _arXiv preprint arXiv:2309.04269_ (2023).
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The internal state of an llm knows when its lying.  _arXiv preprint arXiv:2304.13734_ (2023).
  * Balabanov and Linander (2024) Oleksandr Balabanov and Hampus Linander. 2024.  Uncertainty quantification in fine-tuned LLMs using LoRA ensembles.  _arXiv preprint arXiv:2402.12264_ (2024).
  * Chen et al. (2023) Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan O Arik, Tomas Pfister, and Somesh Jha. 2023.  Adaptation with self-evaluation to improve selective prediction in llms.  _arXiv preprint arXiv:2310.11689_ (2023).
  * Duan et al. (2023) Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2023.  Shifting attention to relevance: Towards the uncertainty estimation of large language models.  _arXiv preprint arXiv:2307.01379_ (2023).
  * Gui et al. (2024) Yu Gui, Ying Jin, and Zhimei Ren. 2024.  Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees.  _arXiv preprint arXiv:2405.10301_ (2024).
  * Gulati et al. (2016) Sankalp Gulati, Joan Serra, Vignesh Ishwar, Sertan Sentürk, and Xavier Serra. 2016.  Phrase-based rāga recognition using vector space modeling. In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 66–70.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_ (2020).
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024\.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ (2024).
  * Krishna and Ishwar (2012) TM Krishna and Vignesh Ishwar. 2012.  Carnatic music: Svara, gamaka, motif and raga identity. In _Serra X, Rao P, Murthy H, Bozkurt B, editors. Proceedings of the 2nd CompMusic Workshop; 2012 Jul 12-13; Istanbul, Turkey. Barcelona: Universitat Pompeu Fabra; 2012._ Universitat Pompeu Fabra.
  * Li et al. (2023b) Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023b.  Large language models understand and can be enhanced by emotional stimuli.  _arXiv preprint arXiv:2307.11760_ (2023).
  * Li et al. (2023a) Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023a.  Large language models in finance: A survey. In _Proceedings of the Fourth ACM International Conference on AI in Finance_. 374–382.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ (2021).
  * Madhusudhan and Chowdhary (2019) Sathwik Tejaswi Madhusudhan and Girish Chowdhary. 2019.  Deepsrgm-sequence classification and ranking in Indian classical music with deep learning. In _Proceedings of the 20th International Society for Music Information Retrieval Conference_. 533–540.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 9802–9822.
  * OpenAI (2023) OpenAI. 2023.  Models-OpenAI API.  https://platform.openai.com/docs/models/gpt-3-5-turbo
  * OpenAI (2024) OpenAI. 2024.  Models-OpenAI API.  https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4
  * Samsekai Manjabhat et al. (2017) S Samsekai Manjabhat, Shashidhar G Koolagudi, KS Rao, and Pravin Bhaskar Ramteke. 2017.  Raga and tonic identification in carnatic music.  _Journal of New Music Research_ 46, 3 (2017), 229–245.
  * Sridhar and Geetha (2009) Rajeswari Sridhar and TV Geetha. 2009.  Raga identification of carnatic music for music information retrieval.  _International Journal of recent trends in Engineering_ 1, 1 (2009), 571.
  * team (2024) Mistral AI team. 2024.  Cheaper, Better, Stronger, Faster — Mistral AI.  https://mistral.ai/news/mixtral-8x22b/
  * Tomani et al. (2024) Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. 2024.  Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations.  _arXiv preprint arXiv:2404.10960_ (2024).
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Varshney and Baral (2023) Neeraj Varshney and Chitta Baral. 2023.  Post-abstention: Towards reliably re-attempting the abstained instances in QA.  _arXiv preprint arXiv:2305.01812_ (2023).
  * Vijayakrishnan (2007) KG Vijayakrishnan. 2007.  _The grammar of Carnatic music_.  Mouton de Gruyter.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ (2022).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Weidinger et al. (2022) Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al. 2022\.  Taxonomy of risks posed by language models. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 214–229.
  * Wikipedia contributors (2024) Wikipedia contributors. 2024.  List of Janya ragas — Wikipedia, The Free Encyclopedia.  https://en.wikipedia.org/w/index.php?title=List_of_Janya_ragas&oldid=1213455818
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023.  Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs.  _arXiv preprint arXiv:2306.13063_ (2023).
  * Xu et al. (2024a) HanXiang Xu, ShenAo Wang, Ningke Li, Yanjie Zhao, Kai Chen, Kailong Wang, Yang Liu, Ting Yu, and HaoYu Wang. 2024a.  Large language models for cyber security: A systematic literature review.  _arXiv preprint arXiv:2405.04760_ (2024).
  * Xu et al. (2024b) Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao Gao. 2024b.  SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales.  _arXiv preprint arXiv:2405.20974v_ (2024).
  * Yang et al. (2023) Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. 2023.  Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning.  _arXiv preprint arXiv:2310.04782_ (2023).
  * Yin et al. (2023) Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023.  Do Large Language Models Know What They Don’t Know?  _arXiv preprint arXiv:2305.18153_ (2023).
  * Zhou et al. (2023) Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023.  Context-faithful prompting for large language models.  _arXiv preprint arXiv:2303.11315_ (2023).


morekeywords= abstract,case,catch, char,class, def,else,extends,final,
if,import, match,module,new,null,object, override,package,private,protected,
public,return,super,this,throw, trait,try,type,val,var,with,implicit,
macro,sealed , sensitive, morecomment=[l]//, morecomment=[s]/**/,
morestring=[b]”, morestring=[b]’, aboveskip=3mm, belowskip=3mm,
showstringspaces=false, columns=flexible, basicstyle=, numberstyle=,
numbers=left, keywordstyle=, commentstyle=, stringstyle=, breaklines=true,
breakatwhitespace=true, tabsize=3

# Classification of simulated radio signals using Wide Residual Networks for
use in the search for extra-terrestrial intelligence.

G. A. Cox IBM Watson Data Platform<EMAIL_ADDRESS>S. Egly Team Effsubsee
G. R. Harp The SETI Institute, 189 Bernardo Ave., Mountain View, CA, 94043 J.
Richards33footnotemark: 3 S. Vinodababu22footnotemark: 2 J.
Voien22footnotemark: 2

###### Abstract

We describe a new approach and algorithm for the detection of artificial
signals and their classification in the search for extraterrestrial
intelligence (SETI). The characteristics of radio signals observed during SETI
research are often most apparent when those signals are represented as
spectrograms. Additionally, many observed signals tend to share the same
characteristics, allowing for sorting of the signals into different classes.
For this work, complex-valued time-series data were simulated to produce a
corpus of 140,000 signals from seven different signal classes. A wide residual
neural network was then trained to classify these signal types using the gray-
scale 2D spectrogram representation of those signals. An average $F_{1}$ score
of 95.11% was attained when tested on previously unobserved simulated signals.
We also report on the performance of the model across a range of signal
amplitudes.

##  1 Deep Learning and SETI

Advances over the last two decades in neural network training algorithms,
increased computational power and available data have had astonishing success
with automatic image classification and similar applications. In this work, we
apply these techniques to the unique case of signal classification of time-
series radio signals.

In a typical ETI search at radio frequencies, a radio telescope observes
signals emanating from selected directions on the sky. After down-conversion
and digitization, the raw data output of the telescope is a time series of
digital voltage samples representing the electromagnetic field in the focal
plane of the telescope. At the Allen Telescope Array (ATA), an array of 42,
6-meter dual-polarity offset-Gregorian radio telescopes[11], a specialized
program, called SonATA, sifts through these time series data looking for weak
radio signals with telltale signs of artificial origin. At the heart of SonATA
is a sensitive algorithm (Doubling Accumulation Drift Detector or DADD) which
uses conventional digital signal processing techniques honed over decades of
effort[2]. DADD effectively detects just one kind of narrowband signal, a tone
with a frequency that drifts linearly with time. While DADD has a low
probability of generating false negatives, there are many different kinds of
narrowband signals that generate false positives[4]. However, for a human,
these signals are easily distinguishable from a drifting tone.

In this paper, we implement convolutional neural networks (CNN) for the
purposes of signal detection and classification, which may one day replace or
complement the use of the DADD algorithm. The CNN is a sensitive (low false
negative rate) detector of several types of narrowband signals and
additionally classifies those signals by type. The classifier makes our SETI
search even more effective since signals generating false positive detections
with DADD are correctly classified, effectively eliminating false positives
without additional telescope time. Moreover, the CNN is a reliable detector of
many different signal types (instead of just one). This expands our SETI
search many-fold by generalizing the search to include a number of different
signal types.

###  1.1 The Spectrogram

(a) Observation in time-domain

(b) Observation as a spectrogram

Figure 1: Observation of a signal from the ISEE3 satellite on August 20, 2013.
The raw data are shown in the time domain (a) and as a spectrogram (b).

A common approach to signal detection is to transform the demodulated time-
series signal to a spectrogram. The utility of a spectrogram is highlighted
with the following signal observed from the ISEE3 satellite on August 20,
2013. Figure 1(a) shows the modulus of the time series dominated by noise and
with a weak signal superimposed. The time series has been broken up (reshaped)
into rows of length 1000 points and then arranged in time order from top to
bottom. Visually, this representation of the time series is indistinguishable
from pure noise.

A spectrogram is a representation of a signal’s power across the available
frequency band, estimated by the squared modulus of the Fourier transform, as
a function of time. To produce the spectrogram of the time series in Fig.
1(a), the data in each row is passed through a frequency filter bank (based on
the fast fourier transform) and the squared modulus of the resultant rows are
arranged into a two dimensional array as before (Fig. 1(b)). Striking evidence
of the superimposed signal is observed in the spectrogram.

In both Fig. 1(a) and Fig. 1(b), the same data is presented as a 2D
representation, suitable for input to a machine vision algorithm. The
difference is that we have used our domain knowledge to choose the
representation where the signal features are most easily detected. The Fourier
transformation efficiently adds structure that CNNs can utilize, while we do
not know of any general purpose neural network architecture that can learn
from the complex time-series with a reasonable quantity of sample data or
within a reasonable training time. Training on data similar to Fig. 1(b) will
succeed in producing an effective data classifier with a modest training set
(demonstrated below).

It should be noted that deep neural networks have been used in other fields
for the purpose of classifing spectrogram representations of time-series data;
from music instrument recognition to noise detection at the gravitational-wave
observatory, at the Laser Interferometer Gravitational-Wave Observatory
(LIGO), and recently to discover new exoplanets with Kepler Space Telescope
data [9, 3, 10].

###  1.2 SETI Research with the Allen Telescope Array

For full context of this work, we briefly describe normal signal detection
operations at the ATA. For twelve hours each day, the SETI Institute searches
for persistent radio signals emanating from an extra-terrestrial source. The
telescope array combines observations from up to 42 dishes into three, dual-
polarization beams which can be pointed anywhere within the large ATA field of
view (FOV, frequency dependent with a FWHM of 3.5∘ at 1 GHz), in a process
called beamforming. These pencil-shaped synthetic beams measure only a small
region of the sky at one time (FWHM 0.1∘ at 1 GHz) centered on a target of
interest. After downconversion and digital sampling, each beam produces
voltage time series data at 104 million complex samples per second (MCSs),
representing about 70 MHz of useful radio frequency bandwidth and center
frequency tuned anywhere from 1-10 GHz.

The six data streams (three beams, two polarizations) are digitally processed
as follows. A cascade of polyphase filter banks divide each high-speed beam
stream into many parallel low-speed time series, each representing a sliver of
about 8 kHz bandwidth. From here, each low-speed time series is examined
independently for evidence of artificial signals. A spectrogram is created and
then searched with the DADD algorithm.

The full data acquisition system is managed by SonATA. It manages the nightly
observations, controls beamforming, maintains and uses an RFI database for
comparison with observed signals, and performs signal classifications based on
signal characteristics estimated from the DADD algorithm. SonATA also manages
the follow-up observation procedure when signals are classified as candidate
signals. A candidate signal means the data exhibits sufficient traits of a
narrowband signal.

After a candidate signal is found, SonATA performs a series of tests for
persistence. In particular, SonATA directs the ATA to make observations away
from the direction from which the candidate signal appeared in order to
determine that the candidate signal is not observed elsewhere, and then
reforms a beam toward the original location to affirm its continued presense.
This process repeats up to five times, at which point a human is notified.
These series of tests can take many tens of minutes to perform.

In comparison to a neural net classifier, which we show can classify many
different signal types at once, the DADD algorithm is limited. DADD can
reliably find narrowband signals in spectrograms (low false negative rate) but
can easily be fooled by signals that are clearly of another class (high false
positive rate). Observing time is wasted every time DADD identifies a false
positive, which must be followed up with additional observing. From this
perspective, the neural net enabled classifier described here is more flexible
and can potentially speed up the SETI search. On the other hand, a robust and
efficient multi-class classifier opens up the possibility for SETI research to
consider different signal classes beyond narrowband. For example, a signal
that is initially detected as narrowband by the DADD may later possess a
stochastically varying central frequency or amplitude modulations that make
DADD detection unreliable. A classifier trained on signals with these
different characteristics, however, would be more reliable.

###  1.3 Motiviation for Simulated Signals

Recently, attemps have been made to cluster and classify candidate signals
found by SonATA/DADD in the data set from 2013 to 2015, which contained
slightly more than 4 million candidate signals. Promising approaches included
one technique that utilized simple affine transformation followed by a
comparator to examples of known signal types, and another technique which
utilized an autoencoder to extract a subset of features from spectrogram and
then attempted to perform clustering with the t-SNE alorithm [8]. However,
neither approach resulted in satisfactory classification capabilities. As
such, this approach to develop a classifier based on a set of simulated data
was conceived.

##  2 Simulations

The simulated data set was designed to achieve a number of goals: 1) reproduce
real SonATA data in basic structure, 2) simulate a feasible number of
different signal classes, 3) test machine-learning model performance over a
range of signal strengths, 4) produce signals with distinct characteristics
between the classes, yet still produce signals that may be difficult to
classify. Ultimately, we simulated 140,000 time-series signals for the
training data. There were six signal classes, plus a set of simulations that
contained only background noise.

The data acquisition system at the ATA demodulates the observed signals from
the GHz range and digitizes with 8-bit digitizers. This produces a complex-
valued time-series data set with 8-bits for the real and imaginary components.
As such, our simulation program also outputs complex-valued (8-bit real, 8-bit
imaginary) time-series data. It should be noted that the ”digitization rate”
of the simulated data are unspecified, which means the time- and frequency-
resolution in the spectrograms is arbitrary. This has no effect, however, on
the algorithm used to classify signals.

###  2.1 Signal Classes and Characteristics

Based on domain knowledge from SETI Institute researchers, six signal classes
were chosen to be simulated. These classes represent, roughly, the more common
signal types that have been observed. These signal types were also relatively
simple to simulate, making them attractive choices for this study. Signal
types are specified by the apparant ”shape” of the signal when observed as a
gray-scale spectrogram. In the examples in this paper, white was chosen to
represent the largest amplitude values and black are zero (Fig. 2). The labels
for the six simulated signal classes were brightpixel, narrowband,
narrowbanddrd, squarepulsednarrowband, squiggle, and
squigglesquarepulsednarrowband. The label for the simulated data files that
were noise only was noise. In the full training set there were 20,000
simulations for each of the seven classes.

(a) narrowband

(b) narrowbanddrd

(c) squiggle

(d) squarepulsednarrowband

(e) squigglesquarepulsednarrowband

(f) brightpixel

Figure 2: Examples of the six simulated signal types. A seventh signal class
was also simulated, but contained only gaussian white noise and is not shown
here.

####  2.1.1 Simulation Algorithm

The simulated signals in the time-domain can all be described with a single
equation,

$s(t)=A(t)\,\exp(i\omega(t)\,t+\phi)+n(t)$ (1)

where $A(t)$ is the signal amplitude, $\omega(t)$ is the frequency, and $\phi$
is a random phase offset. The noise component of all simulated signals,
$n(t)$, was generated by random sampling from a Gaussian with zero mean and
width of $\sigma=13.0$ for each of the real and imaginary components at every
$t$. The seven signal types were generated using the following frequency and
amplitude functions. The frequency function is

$\omega(t)=\omega_{0}+(\omega_{1}+\dot{\omega}_{1}\,t)\,t+B\int_{0}^{t}\Omega(t)\,d\tau$
(2)

where $\omega_{0}$ is the starting frequency, $\omega_{1}$ is the drift rate,
and $\dot{\omega}_{1}$ is its derivative. The $\Omega(t)$ term is a uniformly
sampled random value between -1 and 1 that is updated at each time step. The
expression $\int_{0}^{t}\Omega(t)\,d\tau$ corresponds to a random walk in
frequency versus time and the arbitrary constant $B$ is referred to as the
squiggle amplitude. The time-dependent amplitude is

$A(t)=A_{o}\cdot W(t\,|T,\,D,\,\phi_{w})$ (3)

where $W(t\,|T,\,D,\,\phi_{w})$ is a square-wave modulation function that
depends on a period, $T$, a duty cycle, $D$ and starting phase, $\phi_{w}$.

Each signal class was defined by a combination of setting some parameters to
0, and allowing the others to be randomly sampled (LABEL:tab:simparams). For
all signals types, the starting frequency, $\omega_{0}$, was sampled between
[$-2\pi/3$, $2\pi/3$] and the drift rate, $\omega_{1}$, was sampled between
[-7.324e-6, 7.324e-6].

The narrowband class (Fig. 2(a)) is the simplest signal type, defined by
$\dot{\omega}_{1}=0$, $B=0$, and $W(t\,|T=L,\,D=1.0,\phi_{w}=0)=1$, where $L$
is the full length of the simulation, which was fixed at $L=196608$. The
equation for narrowband reduces to

$s(t)=A_{o}\,\exp(i(\omega_{0}+(\omega_{1}\,t)\,t+\phi)+n(t)$ (4)

The narrowbanddrd (DRD stands for “drift rate derivative”) was defined by
$\dot{\omega}_{1}\neq 0$, $B=0$, and $W(t\,|T=L,\,D=1.0,\phi_{w}=0)=1$ (Fig.
2(b)). The squiggle class is defined by $\dot{\omega}_{1}=0$, $B\neq 0$, and
$W(t\,|T=L,\,D=1.0,\phi_{w}=0)=1$ (Fig. 2(c)).

The squarepulsednarrowband (Fig. 2(d)), squigglesquarepulsednarrowband (Fig.
2(e))
and brightpixel (Fig. 2(f)) all have a time-dependent amplitude. For the
squarepulsednarrowband and squigglesquarepulsednarrowband, the square-wave
period, $T$, and duty cycle, $D$, were uniformly sampled within the range
specified in LABEL:tab:simparams. For the brightpixel, the period was fixed at
$T=L$ and the duty cycle was uniformly sampled within a much smaller range,
$D\in[0.0078125,0.03125]$. The square-wave phase $\phi_{w}$, which sets the
start-time of the modulation, was uniformily distributed between 7% and 93% of
the full length of the signal, preventing the possibility of the signal only
being found near the very beginning or end of the simulation. Finally, signals
of the noise class were defined by $A_{o}=0$.

The range of values for the simulation parameters were tuned, by hand, to
produce a set of simulated signals that appeared to match the real ATA data
sets. They were also tuned in such a way to generate signals from one class
that appeared very similar to signals from another. For example, narrowband
signals with very small values of $A_{o}$ appear much like noise (Fig. 3),
while narrowbanddrd signals with small values of $\dot{\omega}_{1}$, and
squiggle signals with small values of $B$ appear very much like narrowband
(Fig. 4(a) and Fig. 4(b), respectively).

As may be noticed by those with signal processing experience, this model would
generate spectrograms that would alias high-frequency signals to low
(negative) frequency once $\omega(t)+\phi>\pi$. Similarly, low-frequency
signals would alias up to high-frequencies in the spectrogram once
$\omega(t)+\phi<-\pi$. To prevent this aliasing effect, code was included in
the simulation that set $A_{o}=0$ when the frequency crossed either of these
thresholds.

(a) A moderately bright signal, with $A_{o}/13.0=0.25$

(b) The dimmest possible signal, with $A_{o}/13.0=0.05$

Figure 3: Two narrowband simulations at significantly different amplitudes,
with the same drift rate ${\omega_{1}=0.001}$. The allowed range of $A_{o}$
for narrowband is found in LABEL:tab:simparams. The low-amplitude narrowband
signal in Fig. 3(b) is not visible by eye in this rendering.

(a) A narrowbanddrd with the smallest allowed ${\dot{\omega}_{1}}$.

(b) A squiggle with the smallest allowed $B$.

Figure 4: A simulated narrowbanddrd with ${\dot{\omega}_{1}=-1.0\cdot
10^{-8}}$ and initial drift rate ${\omega_{1}=-9.542\cdot 10^{-4}}$ and
squiggle with ${B=1.0\cdot 10^{-4}}$ and initial drift rate
${\omega_{1}=3.714\cdot 10^{-3}}$.

##  3 Neural Network Signal Classifier

We frame the challenge of classifying radio signal time series of complex
amplitudes as an image recognition task on their 2D spectrograms and apply
Wide Residual Network with 34 convolutional layers and a widening factor of 2.

Figure 5: Classification Pipeline

###  3.1 Recasting as Image Classification

The motivation to recast signal classification as an image classification
problem comes from the fact that humans regularly classify these signals by
inspecting a spectrogram. For this image recognition task, a spectrogram for a
given time series of complex amplitude is created by first reshaping the data
to height and width more appropriate for our neural network classifier. Then a
Hanning window and Fourier transform are applied. The square of the absolute
value of the result provides the spectrogram, which we take the logarithm of
to produce one of the images to be supplied to our convolutional neural
network. The second is the phase of the Fourier transformed signal, which was
included for the purpose of providing additional useful features to the neural
network models tested. Although a followup investigation indicated it did not
increase the classification accuracy of our final model trained on simulated
signals, phase may be useful when using real signals that have properties not
included in our simulations, such as polarization, and is an area of future
research.

The reciprocally dependent time- and frequency-resolution of a spectrogram can
be arbitrarily chosen within the bounds of the total length of the signal.
This choice results in spectogram images of different shapes (height and
width). While these differences can change signal detectability as a human
perceives it, for the purposes of classification by a convolutional neural
network, we choose an aspect ratio close to square, since downsampling will
reduce both horizontal and vertical resolution significantly, and a reduction
to near zero would be unproductive.

###  3.2 Exploration of CNN Architectures

Deep learning is an active area of research in which the advances of deep
neural networks have produced near-human or superhuman accuracy in supervised
machine learning tasks, including image recognition tasks [7]. Convolutional
neural networks (aka convnets or CNNs) are at the core of these advances, and
the gains in state-of-the-art performance demonstrated by ResNet is a go-to
example[5]. To accurately classify these images generated from radio signal
time series, we test multiple convolutional neural network architectures in
the classification task using only the spectrogram and phase images as input
features.

The milestone advance in image classification performance seen in the 2015
ImageNet showing by the ResNet team suggests that this tried-and-true network
is a suitable baseline for conducting these experiments. Augmentations and
further improvements to the ResNet architecture have been presented since, so
we tested these in a benchmark against ResNet and each other to determine
which performed best on this particular dataset. Although these competing
networks were able to show state-of-the-art performance on various tasks and
datasets, their performances are close enough that for this task, somewhat
different from the typical benchmarks for image recognition, empirical
evidence is sought to determine which network best classifies simulated radio
signal spectrograms.

####  3.2.1 Residual Network (ResNet)

ResNet introduced the residual connection between convolutional layers in a
very deep convolutional neural network in order to combat the loss of signal
backpropagation. Previously, very deep networks tended not to train
successfully, but this skip connection that bypasses a layer’s nonlinearity
allows gradients to backpropagate further in the network and allow for deeper,
more expressive networks. Hence, we experimented with ResNets up to the limit
of computational constraints for these medium-resolution images.

####  3.2.2 Wide Residual Network (WRN)

The development of wide residual networks was based on the observations that
increasing the depth of ResNet provided diminishing returns on network
performance improvement, and offered shallower convolutional networks with
more convolutional filters at each layer as a solution[12]. Additionally, the
authors change the order of convolution, batch normalization, and activation,
and add dropout to tune their architecture to train faster and perform
competitively with deeper ResNet models. In our experiments, we used the
variations they had the most success with; namely, using the same 3x3 basic
convolutional block (as opposed to a bottleneck block), and a dropout rate of
0.3.

####  3.2.3 Densely Connected Residual Network (DenseNet)

DenseNet extends the idea of residual connections introduced by ResNet by
adding residual connections not only between consecutive convolutional layers,
but also between all subsequent convolutional layers[6]. The dense residual
block allows the gradient signal to skip more layers, and which more closely
ties the loss function to earlier layers of the network. The additional skip
connections also are thought to encourage feature reuse by sending signal from
multiple convolutional layers to later layers, leading to more expressive
power in a more compact network.

####  3.2.4 Dual Path Network (DPN)

Finally, we experimented with dual path networks, which integrate both
residual networks and densely connected residual networks to realize the
advantages of each, while sharing weights to maintain a reasonable model
complexity[1].

###  3.3 Comparison of Models

We trained several models for the ResNet, WRN, DenseNet, and DPN
architectures, since each architecture can be implemented with varying sizes.
The accuracy of a learned classifier depends on this size because less complex
networks cannot necessarily express as complex a pattern as a larger network,
but the larger networks do not necessarily succeed in learning the patterns
they can express. Table 1 lists the models trained for each architecture type,
with the best performing model shown in bold.

Architecture | Models Tested | Accuracy | Parameters
---|---|---|---
ResNet | ResNet-18, ResNet-50, ResNet-101 | 94.99 | 42.6M
| ResNet-152, and ResNet-203 |  |
Wide ResNet |  WRN-34-2, WRN-16-8, | 95.77 | 1.9M
| and WRN-28-10 |  |
DenseNet | DenseNet-161, DenseNet-201 | 94.80 | 18.3M
Dual Path Network |  DPN-92, DPN-98, and DPN-131 | 95.08 | 35.1M
Table 1: Comparison of model architectures. Accuracy is measured with a cross-
validation subset of the full training set. The number of parameters are given
for the best-performing model (bold) of each type. Figure 6: Basic Block: two
3x3 2D convolutions with batch normalization, ReLU nonlinearity, and a
residual connection Figure 7: Complete WRN-34-2: a series of Basic Blocks
grouped into three sections of increasing number of features planes and
decreasing image resolution

The accuracies included in Table 1 are validation accuracies on the same
train-validate split for all architectures. The performances of the four best
models are comparable, with the WRN-34-2 (Fig. 7) only slightly outperforming
the rest. However, a very significant basis for choosing the WRN-34-2 over the
other models is the size of the model. The WRN-34-2 contains only 1.9M
parameters (also called weights). In comparison, the second smallest model,
the DenseNet-201, is nearly ten times as large. The ResNet-101 is the largest,
with over twenty times as many parameters. For the purposes of quick training
and inference on-site at the ATA, it is ideal to have a small memory
footprint.

###  3.4 Wide Resnet Ensemble

The comparison of these four network architectures showed similar and
encouraging results, but the edge that WRN-34-2 had over the others led us to
continue working to improve its performance. A common practice to increase
generalizability of a machine learning model that threatens to overfit on
training data is to create an ensemble of models. For this task, ensemble
averaging was used. Five WRN-34-2 models were trained on different training
data but with the same hyperparameters and training strategy. While the
comparative study of architectures used the same four of five folds for
training and the fifth for validation, the ensemble members trained on the
five distinct four-fold subsets of data, as with k-fold validation for $k=5$.

To evaluate the ensemble model, each of the five member models outputs its
softmax predictions and the average of these scores is taken as the final
score. With no validation data left over, this ensemble model was evaluated on
new test data provided as part of the code challenge, which yielded an
accuracy of 94.99%.

##  4 Final Model Performance

###  4.1 Classification Report

The model with the best validation set accuracy, the WRN-34-2 using a 5-fold
averaging, was then tested using a separate test set withheld during the
training phase of all models. As this work was performed in the context of an
online code challenge, the other models described in the previous section were
not tested with this test data set.

Table 2 and Table 3 display the performance of the model on the test set of
2496 simulations.

| $N$ | precision | recall | $F_{1}$
---|---|---|---|---
brightpixel | 385 | 0.991 | 0.857 | 0.919
narrowband | 355 | 0.994 | 0.944 | 0.968
narrowbanddrd | 348 | 0.969 | 0.977 | 0.973
noise | 368 | 0.785 | 0.995 | 0.877
squarepulsednarrowband | 385 | 0.975 | 0.925 | 0.949
squiggle | 322 | 1.000 | 0.997 | 0.998
squigglesquarepulsednarrowband | 332 | 1.000 | 0.970 | 0.984
Table 2: Model Performance Scores. The number of simulations of each class in the test set is given by $N$. Actual / Predicted | bp | nb | drd | no | sqnb | sgl | sglsqnb
---|---|---|---|---|---|---|---
bp | 330 | 0 | 0 | 55 | 0 | 0 | 0
nb | 0 | 335 | 10 | 5 | 5 | 0 | 0
drd | 0 | 0 | 340 | 7 | 1 | 0 | 0
no | 1 | 0 | 0 | 366 | 1 | 0 | 0
sqPnb | 2 | 2 | 1 | 24 | 356 | 0 | 0
sgl | 0 | 0 | 0 | 1 | 0 | 321 | 0
sglsqPnb | 0 | 0 | 0 | 8 | 2 | 0 | 322
Table 3: Confusion Matrix. Actual classifications are along each row and
Predicted classification counts are along the columns. For example, there were
322 squiggle signals. Of those, 321 were predicted to be squiggles and one was
predicted to be noise. Labels have been abbreviated: bp for brightpixel, nb
for narrowband, drd for narrowbanddrd, no for noise, sqPnb for
squarepulsednarrowband, sgl for squiggle, and sglsqPnb for
squigglesquarepulsednarrowband.

Clearly the largest source of uncertainty was distinguishing brightpixel
signals from noise. As we’ll see in the next section, this was due to very
low-amplitude brightpixel signals, as one would intuitively expect.

###  4.2 Model Performance Characteristics

In order to briefly explore the performance characteristics of this trained
model in a controlled way, 14 new sets of test data were generated, each with
250 signals of each class. For each set, the signals were simulated with a
fixed signal amplitude, $A/13.0=$ 0.008, 0.01, 0.02, 0.04, 0.05, 0.06, 0.07,
0.08, 0.09, 0.1, 0.12, 0.16, 0.2, or 0.4. Many of these signal ampliutdes, it
should be noted, are below the amplitudes of signals found in the training
data set, allowing us to explore the model performance slightly outside of
range of signal amplitudes on which it was trained. All other parameters of
the test data, however, remained consistent with the training data.

For each of these test sets, we performed inference and recorded the model’s
multinomial cross-entropy loss, classification accuracy (Fig. 8(a)), and
$F_{1}$ score (Fig. 8(b)). The model performs as expected. Signals with
smaller amplitudes were more difficult to classify and tend to be classified
as noise. Also, the model does not appear to have any classification power
outside of the trained signal amplitude space.

Furthermore, the onset rise of each class’s $F_{1}$ score (Fig. 8(b)) can be
intuitively explained by considering the average amount of power per pixel in
each signal, akin to the apparent brightness of the signal in the spectrogram.
The brightpixel signals have lowest total power, overall. These signals have
zero amplitude for most of the simulation except for a brief moment at a
particular frequency. As such, the model struggles to recognize these signals
the most. The simulations that contain a non-zero $B$ (squiggle and
squigglesquarepulsednarrowband) also have a reduced apparent brightness in the
spectrogram. The stochastic flucutations of these signals result in the power
being spread across a larger bandwidth of frequencies during any particular
time range of the simulation, as compared with narrowband. As such, one
expects the classification and recall of those signals to be reduced relative
to the other signals that appear brighter in the spectrogram. The narrowband
and narrowbanddrd have the earliest onset of significant $F_{1}$ scores,
followed by squarepulsednarrowband, consistent with the argument that the
brighter signal types are more easily recognizable.

(a)

(b)

Figure 8: Multinomial cross-entropy loss scores, classification accuracy, and
$F_{1}$ scores versus the signal amplitude ($A/13.0$). The horizontally
hatched region represents the signal amplitudes in the training data with
amplitudes in the range [0.05, 0.4], while the vertically hatched region spans
the amplitudes from [0.1, 0.75]. Refer to LABEL:tab:simparams for the signal
classes that were simulated in these regions.

##  5 Conclusions

This work demonstrates the potential usefulness of applying contemporary
convolutional neural networks to SETI research. With further improvements,
these algorithms may soon become part of SETI research data acquisition
systems.

Subsequent work building on the results here could focus on a number of
aspects. Clearly, the precision and recall scores for the different signal
classes were quite good. The most immediate next step would be to build new
models trained on data generated using a larger range of simulation parameters
for each class (LABEL:tab:simparams). Additionally, more signal types could be
added to training and test sets. For example, a common signal type not
included in our repertoire is a short burst of power over all frequencies
sometimes identified with radar pulses.

The model’s performance falloff just at the lower-bound of the signal
amplitudes found in the training data (Section 4.2) leads to a particular
question: to what small signal amplitude can we train a model of the same
network architecutre and still retain robust classification accuracy? That is,
if we were to construct new training data with smaller signal amplitudes and
retrain new models, how small in ampliutude can we go before the models fail
to accuractly classify signals?

Besides new signal types and characeteristics, the noise component of the
signal, $n(t)$, could be more realistic. For this study, we used a very simple
gaussian white noise model. In a previous version of the training data set,
however, we used real observations from the Sun as the background component,
$n(t)$. Observations from the Sun do not have constant power at all
frequencies and are non-stationary. This background data was not used in this
work, but may be the basis for simulations in future work.

Real observations from the ATA do not always contain just one signal or signal
type in the spectrogram. Therefore, another future work would be to include a
more complete set of commonly-observed signal types within the same simulation
and build models that can find multiple signal types with techniques similar
to those used to perform object detection in ordinary daily human photographs.

The set of simulated data could also be improved through the use of some type
of generative network. This would require some effort, though not unreasonably
burdensome, to hand-label real spectrogram observations. An autoencoder and
t-SNE clustering approach was already taken with ATA data from 2013 to 2015 to
cluster signals [8]. Although these simulations from generative network model
would appear to be more realistic, the signals would not have controllable (or
learnable) parameters such as $A_{o}$, $B$, $\dot{\omega}_{1}$, etc.

Finally, the DADD algorithm currently in use at the ATA estimates a linear fit
to observed signals, with parameters including the signal power, initial
signal frequency, $\omega_{0}$, and drift rate, $\omega_{1}$. This regression
is important because it allows for prediction of the future signal frequency,
which is needed to identify the same signal in a subsequent observation. We
speculate that either the parameters of human-constructed models (Eqs. 1 \- 3)
for each signal class could be estimated through regression after signals
classification with the 2D CNN, or that some type of recurrent neural network,
or causal convolutional architecture would be able to estimate signal
characteristics and predict future signal frequency by using each row of the
spectrogram as the input vector at each time-step.

In conclusion, we report here the adaptation of a convolutional neural
network, specifically a wide residual network, to the problem of signal
discovery and classification relevant to SETI. We find that by treating
spectrograms as if they were images, we can train an image classifying network
and achieve very good results. The sensitivity of the neural network detector
is comparable to that of a finely honed conventional processing algorithm, and
that the signal classification accuracy is quite good. Using such signal
classifications will improve the efficiency of radio SETI, by providing extra
information that can be used to decide whether to follow-up on detected
signals. Compared to the conventional algorithm, the neural network approach
lacks the parameterization of signals necessary for extrapolation to later
times. We suggest that this limitation may be overcome with a traditional
approach or with additional neural networks.

##  6 Acknowledgements

We’d like to thank Galvanize, Skymind, Nimbix, and The SETI League for their
financial contribution and to the hard work put in by many employees of those
organizations that ensured a successful code challenge. Thanks to IBM for
providing significant compute and data storage. We acknowledge the helpful
suggestions from Francois Luus of IBM Research South Africa. Special thanks to
Graham Mackintosh for establishing the IBM-SETI Institute partnership.

## References

  * [1] Yunpeng Chen et al.  “Dual Path Networks”  In _CoRR_ abs/1707.01629, 2017  arXiv: http://arxiv.org/abs/1707.01629
  * [2] D.. Cullers, I.. Linscott and B.. Oliver  In _Commun ACM, 28, 1151_ , 1985
  * [3] D. George, H. Shen and E.. Huerta  “Deep Transfer Learning: A new deep learning glitch classification method for advanced LIGO”  In _ArXiv e-prints_ , 2017  arXiv:1706.07446 [gr-qc]
  * [4] G.. Harp, J. Richards and J.. Tarter  In _Ap J, 152, 181_ , 2016  URL: http://arxiv.org/abs/1607.04207
  * [5] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep Residual Learning for Image Recognition”  In _CoRR_ abs/1512.03385, 2015  arXiv: http://arxiv.org/abs/1512.03385
  * [6] Gao Huang, Zhuang Liu and Kilian Q. Weinberger  “Densely Connected Convolutional Networks”  In _CoRR_ abs/1608.06993, 2016  arXiv: http://arxiv.org/abs/1608.06993
  * [7] Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton  “ImageNet Classification with Deep Convolutional Neural Networks”, 2012  URL: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
  * [8] Francois Luus  “To Be Published”
  * [9] Taejin Park and Taejin Lee  “Musical instrument sound classification with deep convolutional neural network using feature fusion approach”  In _CoRR_ abs/1512.07370, 2015  arXiv: http://arxiv.org/abs/1512.07370
  * [10] C.. Shallue and A. Vanderburg  “Identifying Exoplanets with Deep Learning: A Five Planet Resonant Chain around Kepler-80 and an Eighth Planet around Kepler-90”  In _ArXiv e-prints_ , 2017  arXiv:1712.05044 [astro-ph.EP]
  * [11] J. Welch, D. Backer and L. Blitz  In _Proc IEEE, 97, 1438_ , 2009  URL: http://arxiv.org/pdf/0904.0762
  * [12] Sergey Zagoruyko and Nikos Komodakis  “Wide Residual Networks”  In _CoRR_ abs/1605.07146, 2016  arXiv: http://arxiv.org/abs/1605.07146