# Model Analysis & Evaluation for Ambiguous Question Answering

Konstantinos Papakostas
University of Amsterdam
Irene Papadopoulou11footnotemark: 1
University of Amsterdam  Equal contributions Correspondence email:
<EMAIL_ADDRESS>
###### Abstract

Ambiguous questions are a challenge for Question Answering models, as they
require answers that cover multiple interpretations of the original query. To
this end, these models are required to generate long-form answers that often
combine conflicting pieces of information. Although recent advances in the
field have shown strong capabilities in generating fluent responses, certain
research questions remain unanswered. Does model/data scaling improve the
answers’ quality? Do automated metrics align with human judgment? To what
extent do these models ground their answers in evidence? In this study, we aim
to thoroughly investigate these aspects, and provide valuable insights into
the limitations of the current approaches. To aid in reproducibility and
further extension of our work, we open-source our code here.

##  1 Introduction

Question Answering (QA) has been subject to great progress in the past years,
largely thanks to the representational capabilities of modern architectures
like the Transformer Vaswani et al. (2017), but also due to the curation of
large, high-quality datasets that enabled the effective training of these
models Joshi et al. (2017); Kwiatkowski et al. (2019).

At the same time, the presence of ambiguous questions has been a challenging
aspect of QA. In order to answer such questions, models are required to
generate long answers with fluency and cohesion, which is often referred to in
the literature as Long-Form Question Answering (LFQA). To tackle this, Min et
al. (2020) curated the AmbigQA dataset, which contains disambiguations for
various questions that were present in popular benchmarks. Extending this
work, Stelmakh et al. (2022) selected a subset of the questions and crowd-
sourced gold answers that cover all possible interpretations of each question,
resulting in the ASQA dataset.

Recently, Krishna et al. (2021) performed a case study on ELI5 Fan et al.
(2019), one of the largest collections available for LFQA, and pointed out
several issues that complicate the development and evaluation of suitable
models. In particular, the authors questioned whether the retrieved documents
are considered by the generative models when producing an answer, and the
correlation of the common evaluation metrics with human judgment.

In this work, we aim to investigate whether the baselines set on ASQA by
Stelmakh et al. (2022) suffer from the issues pointed out by Krishna et al.
(2021), but also to analyze the modeling choices that contribute to
performance. Concretely, we set out to answer the following research
questions:

  1. RQ1

Does scaling the size of the generative models affect the quality of the
generated answers?

  2. RQ2

Can an intermediate round of fine-tuning on non-ambiguous LFQA collections
improve performance in ambiguous QA?

  3. RQ3

When comparing models head-to-head, does human judgment reflect the difference
in the automated evaluation metrics?

  4. RQ4

Do models base their answers on the retrieved evidence, or could they be
hallucinating?

##  2 Methodology

We design a standard retrieval-augmented system, to identify the dimensions
that contribute to disambiguating questions and generating factual answers.

###  2.1 The LFQA Pipeline

#### Evidence Retrieval

The first step of the pipeline is to identify the documents that will form the
basis of the generated answers. Given a question $q$, we employ a retrieval
method $R$ that fetches the top-$k$ relevant documents
$\\{d_{i}\\}_{i=1}^{k}$. For a document $d_{i}$ to be considered relevant, it
needs to cover at least one aspect of $q$. To completely resolve the
ambiguity, the passages111We use the terms document and passage
interchangeably. in the index should suffice to collectively answer all
aspects of a given question.

#### Answer Generation

Once the evidence has been collected, we feed the retrieved passages to a
generative model to summarize them in a concise answer that disambiguates the
question at hand. As is the standard practice in contemporary literature, we
opt for a sequence-to-sequence model $G$ that follows an encoder-decoder
architecture, which first creates a dense representation of the concatenation
of the question and the passages, and then produces the answer by attending to
this latent representation:

$\mathrm{answer}=G([q;d_{1};\ldots;d_{k}])$

with $[\cdot~{};\cdot]$ being the concatenation of $\geq 2$ passages.

###  2.2 Modeling Choices

We expect more sophisticated pipelines to provide better answers for ambiguous
questions, and thus we make a distinction based on the complexity of the
interaction between the two components.

#### Naive

We implement the Question baseline, which repeats the ambiguous question a few
times in order to match the typical length of the answers in the dataset. This
is a lower bound on the task, as we are not truly answering the question at
hand.

#### Retrieval-Only

In this case, we rely exclusively on a retriever to fetch the top-$k$ passages
as a response to the ambiguous question. We experiment with different values
of $k$ to evaluate whether using more passages leads to an answer that covers
more of the disambiguated questions.222Employing some form of result
diversification has the potential to improve the performance of the retrieval
component, but we leave this direction as future work.

#### Sequence-to-Sequence

A generative language model is often used to produce a concise response that
summarizes all of the disambiguating answers. We analyze three scenarios:

  * •

Closed Book: In the most extreme approach, we assume that the model is not
conditioned on the results of a retriever, but rather only on the question
itself, and relies on its parametric knowledge Roberts et al. (2020) to
respond. We expect this to significantly harm performance, as the available
context to provide an accurate answer is limited.

  * •

Random Retrieval: In order to verify whether the model grounds its answers on
the retrieved documents Krishna et al. (2021), we design a controlled scenario
where we randomly sample passages from our index as evidence.

  * •

Open Book: In the most realistic setting, the model treats the top-$k$ results
of a retriever as context to respond appropriately. We expect stronger
retrieval methods to lead to more comprehensive answers, as the generative
model will be conditioned on more diverse and relevant information.

##  3 Experimental Setup

We aim to assess whether LFQA systems can generate concise answers that
disambiguate the provided questions. In this section, we present the datasets
and models used, as well as the evaluation metrics.

###  3.1 Datasets

We use the ASQA dataset to train and evaluate our systems. It is a subset of
the AmbigQA dataset, with long-form answers and additional context for each of
the selected samples. More specifically, it contains 6,316 ambiguous
questions, with each one being paired to a set of disambiguated questions, the
corresponding short answers, and the Wikipedia passages where the answers were
found. For each question, the dataset curators crowd-sourced a long-form
answer that resolves the ambiguity by summarizing all short answers. The
annotators provided one reference answer for all train (4,353) samples, and
two for all dev (948) and test (1,015) samples.333An ambiguity can be observed
even in a simple query like “Who was the ruler in France in 1830?”, which
presents a challenge due to the existence of two rulers during that period.

Although ASQA is a useful resource for LFQA, when going through the dataset
for preliminary analysis, we found cases where the ambiguity was to identify
when “last/this year” refers to. We argue that training a model on such
samples is counter-intuitive, as we generally assume that the information is
coming from a fixed snapshot of a knowledge base. We provide a few examples
that cover similar types of questions in Appendix A.

Additionally, given the limited size of the ASQA dataset, we follow one of the
proposed research directions posed by Stelmakh et al. (2022) and investigate
the impact of intermediate fine-tuning on a larger LFQA collection, before
training on ASQA. In particular, we use a processed version of the ELI5
dataset444Available on HuggingFace: lfqa & lfqa_support_docs. that addresses
some of the issues raised by Krishna et al. (2021) (226,147 train / 3,020 dev
samples), as well as the NLGEN set of the MS MARCO QA dataset (153,725 train /
12,467 dev samples).

###  3.2 Models

The typical QA pipeline comprises a retrieval and a generative model, which
can be pre-trained separately and then fine-tuned on the downstream task.
Although this can be done in an end-to-end fashion, we chose to keep the
retriever frozen, to avoid re-indexing of the support passages during
training, and only train the generative model. We provide detailed training
information in Appendix B.

#### Retrieval Models

We experiment with both sparse (lexical) and dense (neural) methods, in order
to investigate whether the type of the question encoder has an impact on the
relevancy of the retrieved passages. We chose BM25 Robertson et al. (1995) for
the former, and DPR Karpukhin et al. (2020) for the latter, as they were both
supported by the Pyserini Lin et al. (2021) toolkit and they constitute two of
the most explored options in their corresponding fields. In both cases, we use
the pre-built indices of Wikipedia provided by Pyserini to have a common
knowledge base that matches the one used by the dataset curators.

#### Generative Models

We use two of the most popular Transformer-based encoder-decoder models,
namely T5 Raffel et al. (2019) and BART Lewis et al. (2020). More
specifically, we experiment with three variants of these models, in increasing
parameter count: BART-base (140M params), T5-base (220M params), and BART-
large (400M params). By doing so, we aim to verify whether increasing the
capacity of the generative model corresponds to an increase in answer quality.

###  3.3 Metrics

#### Automated Evaluation

Evaluating the performance of generative models is one of the most challenging
aspects for LFQA. For recall-oriented QA systems, the most common metric used
is Rouge-L Lin (2004), which identifies the longest common sub-sequence
between the generated answer and a reference (gold) answer. Sequences are
penalized proportionally to their length to prevent generating a longer output
to artificially increase the overlap with the reference text. However, a
recent study by Krishna et al. (2021) revealed that Rouge-L did not always
correlate with human judgment, pointing to the need for a more diverse
evaluation setup. Specifically for ambiguous LFQA, Stelmakh et al. (2022)
proposed two metrics to quantify the disambiguation ability of the model:

  1. 1.

Str-EM (String Exact Match): the fraction of disambiguated answers that the
model includes verbatim in its output.

  2. 2.

Disambig-F1: the fraction of answers that can be deduced with a text
comprehension model, using the predicted long answer and the disambiguated
question:

$\textsc{Disambig-F1}=\frac{1}{N}\sum_{k}\frac{1}{n^{(k)}}\sum_{i}\phi(y_{i}^{(k)},\hat{y}_{i}^{(k)})$

where $N$ is the number of evaluation samples, $n^{(k)}$ the number of
disambiguations for the $k$-th question, $y_{i}^{(k)}$ its $i$-th ground-truth
short answer, $\hat{y}_{i}^{(k)}$ the predicted short answer555Generated from
a RoBERTa Liu et al. (2019) model trained on SQUAD2.0 Rajpurkar et al. (2018),
with the predicted long answer and the disambiguated question as input., and
$\phi$ a function that computes the token-level F1 score between them.

Finally, they define another metric, namely DR, which is the geometric mean of
Disambig-F1 and Rouge-L, as an overall estimate of the performance in both
disambiguation and answer overlap.

#### Human Evaluation

To better align the model’s performance with the overall satisfaction of the
human users that interact with the system, we follow Stelmakh et al. (2022) in
creating an anthropocentric evaluation pipeline. Our approach differs in that
it performs head-to-head comparisons between different models, aiming to draw
conclusions about the design choices that lead to a well-performing LFQA
system. For a pair of answers, we compare their comprehensiveness (Comp);
whether the answer suffices to understand both the source of ambiguity in the
question, and the relation between the individual answers, their fluency
(Flue); whether the answer is coherent and fluent from a human reading stance,
and the overall human impression (Over); which of the answers is prefered
overall.

##  4 Results

| Answer Length | Rouge-L | Str-EM | Disambig-F1 | DR
---|---|---|---|---|---
Question | 71.6 | 15.3 | 1.2 | 0.1 | 1.4
T5-base Closed Book | 38.1 | 30.7 | 3.7 | 2.7 | 9.1
BART-base Closed Book | 44.5 | 31.5 | 3.9 | 2.8 | 9.3
BART-large Closed Book | 50.2 | 33.4 | 7.1 | 4.5 | 12.2
BM25@1,3,5 | 103.7 / 310.9 / 518.0 | 28.6 / 20.3 / 15.2 | 18.6 / 30.0 / 36.4 | 10.6 / 14.7 / 17.4 |  17.4 / 17.3 / 16.2
DPR@1,3,5 | 103.5 / 310.4 / 517.3 | 31.4 / 22.1 / 16.4 | 29.3 / 44.0 / 50.8 | 17.5 / 22.8 / 25.7 |  23.5 / 22.5 / 20.5
T5-base DPR@1,3,5 | 51.4 / 57.8 / 57.8 | 31.6 / 33.7 / 33.9 | 23.9 / 26.4 / 26.2 | 17.6 / 17.8 / 17.9 | 23.6 / 24.5 / 24.6
T5-base NLGEN DPR@1,3,5 | 48.5 / 56.0 / 56.0 | 31.2 / 33.7 / 33.7 | 24.6 / 26.0 / 26.7 | 16.8 / 18.1 / 17.8 | 22.9 / 24.7 / 24.5
BART-base DPR@1,3,5 | 52.4 / 57.1 / 56.7 | 33.1 / 33.9 / 33.9 | 24.2 / 25.1 / 24.5 | 16.1 / 16.5 / 16.4 | 23.1 / 23.7 / 23.5
BART-large DPR@1,3,5 | 54.7 / 62.5 / 63.3 | 34.2 / 36.4 / 36.6 | 26.0 / 30.0 / 29.8 | 18.1 / 20.8 / 20.5 | 24.9 / 27.5 / 27.4
BART-large ELI5 DPR@1,3,5 | 55.2 / 59.7 / 59.9 | 35.1 / 36.6 / 37.0 | 26.7 / 30.3 / 29.7 | 19.0 / 21.0 / 21.0 | 25.8 / 27.7 / 27.6

Table 1: Performance comparison on the development set of ASQA, using the
metrics described in Section 3.3. For retrieval-augmented models, $@1,3,5$
indicates using $1/3/5$ retrieved passages as evidence. Bold scores indicate
best result among different number of retrieved passages, underlined scores
indicate best result in each setup.

#### Automated Evaluation

We evaluate our models on the development set of ASQA using the automated
metrics introduced in Section 3.3. Table 1 displays the results for the naive
baseline, the retrieval-only experiments using both BM25 and DPR, and the
closed/open book experiments with DPR. In cases where we use a retriever, we
fetch the top-$k$ relevant documents, for $k\in\\{1,3,5\\}$. As the level of
information contained in the retrieved passages has a direct impact on the
answer quality, we perform a short study on the upper bound of retrieval in
Appendix C.

Naturally, the naive Question baseline performs the worst, with a Rouge-L
score of 15.3 and a Disambig-F1 score of 0.1, leading to an overall DR score
of 1.4. Focusing on the retrieval-only experiments, we observe that DPR
consistently outperforms BM25 in all metrics, which is anticipated as semantic
matching allows us to retrieve passages that answer different aspects of the
question. Remarkably, although the closed-book variants surpass the retrieval-
only methods in terms of Rouge-L, we see the opposite trend for all other
metrics. The open-book variants outperform the rest, confirming our hypothesis
that augmenting the generative model with a retriever is crucial for
performance. We also notice that the increase in performance closely follows
the growth in parameter count, implying that [RQ1] model scaling improves the
quality of the answer.666To gain further insight into some of the common
errors, we showcase a few example model outputs in Appendix E. Surprisingly,
the impact of fine-tuning on larger collections on performance is marginal, as
the disambiguation metrics only narrowly improve. For completeness, we report
the intermediate fine-tuning results for the closed-book experiments in
Appendix D. In general, we deem that [RQ2] naively fine-tuning on non-
ambiguous LFQA datasets has limited added value, as the models are not
inclined to identify any uncertainty in the meaning of the questions asked.

#### Human Evaluation

We perform a head-to-head comparison between a selection of approaches to
determine which modeling aspects contribute more/less to the performance. For
each pair, we asked 2 assessors to compare the models blindly using the human
evaluation metrics defined in Section 3.3. We report our results in Table 2,
where we observe that most comparisons follow the trends we described
previously using the automated metrics. Noticeably, when comparing T5-base
DPR@3 vs BART-base DPR@3 and DPR@1 vs BART-base DPR@1, human annotators seem
to equally prefer both models overall. However, the difference in DR for these
pairs is merely a few decimal points, which justifies the inability to
distinguish between them. Hence, we confirm that [RQ3] the overall human
impression aligns with automated metrics.

| Comp | Flue | Over
---|---|---|---
|  |  |
T5-base vs BART-base∗ | 50% | 38% | 50%
BART-large vs BART-base∗ | 62% | 62% | 62%
BART-large ELI5 vs BART-large∗ | 62% | 50% | 62%
BART-large: DPR@3 vs DPR@1 | 75% | 50% | 75%
DPR@1 vs BART-base DPR@1 | 62% | 38% | 50%
BART-large DPR@1 vs DPR@1 | 38% | 50% | 75%

Table 2: Head-to-head comparison using human evaluation metrics. Pairs marked
with ∗ use DPR@3 for retrieval. Bars indicate the annotators’ preference for
either model.

#### Random Retrieval

Finally, we evaluate whether the generated answers are grounded on the
retrieved passages. Table 3 showcases the performance for the models used in
our open-book experiments, and we generally see a decrease in performance when
compared to fetching relevant documents. Interestingly, BART-base with random
retrieval performs better compared to its closed-book variant. As this is the
smallest model we tested, we speculate that for LFQA, under-parameterized
architectures may be relying more on the question and not the corresponding
evidence. Despite this, [RQ4] models for ambiguous LFQA appear to be well
grounded in the retrieved evidence overall.

| Rouge-L | Str-EM | Dis-F1 | DR
---|---|---|---|---
T5-base | 20.5 | 2.0 | 0.7 | 3.7
T5-base NLGEN | 20.8 | 1.6 | 0.8 | 4.1
BART-base | 29.8 | 5.5 | 3.5 | 10.3
BART-large | 24.3 | 4.4 | 2.2 | 7.2
BART-large ELI5 | 29.6 | 4.8 | 3.0 | 9.5

Table 3: Random Retrieval performance. We use $k=3$ random passages from the
DPR index as evidence.

##  5 Conclusion

In conclusion, by exploring the LFQA field in answering ambiguous questions,
we find that the ASQA dataset is a good foundation to develop and evaluate
models. We notice that larger generative models produce better answers, with
semantic matching for retrieval having a positive impact. To compensate for
the small size of the dataset, we experiment with intermediate fine-tuning on
larger collections and find that doing so only marginally improves the
results. By comparing the performance when using relevant versus random
documents, we show the models’ dependency on the provided context. Our human
evaluation confirms the general trends we observed using the automated
metrics, giving credit to their disambiguating ability.

## Limitations

We see two main limitations in our study. Primarily, given the clear trend of
larger generative models producing higher quality answers, an obvious question
is to investigate whether this continues to be the case indefinitely, or
whether it saturates after a critical amount of parameters. Despite this, due
to hardware restrictions, we were unable to experiment with models larger than
BART-large. Additionally, considering that the field of ambiguous QA
inherently requires complementary pieces of evidence, there is no doubt that
diversification methods are bound to yield better results in terms of
disambiguation quality. In this work, however, we limited ourselves to using a
typical neural retriever, shifting our focus toward the factuality and the
fluency of the generated answers.

## Ethics Statement

As we use a publicly available Wikipedia index as our knowledge base, it is
possible that the generated answers may contain some form of bias that
reflects the information submitted by the anonymous editors of the website. To
prevent this, follow-up work could examine how to detect misinformation or
hate speech in indexed passages, before using them as evidence for the
generative models.

## Acknowledgements

This work was carried out on the Dutch national e-infrastructure with the
support of the SURF Cooperative. We would also like to thank Evangelos
Kanoulas for his valuable feedback and guidance during this project.

## References

  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4940–4957, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021.  Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2356–2362, New York, NY, USA. Association for Computing Machinery.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled Weight Decay Regularization.  _arXiv preprint arXiv:1711.05101_.
  * Min et al. (2021) Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. 2021.  Joint passage ranking for diverse multi-answer retrieval.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6997–7008, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5783–5797, Online. Association for Computational Linguistics.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _CoRR_ , abs/1910.10683.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Robertson et al. (1995) Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995\.  Okapi at TREC-3.  In _Overview of the Third Text REtrieval Conference (TREC-3)_ , pages 109–126. Gaithersburg, MD: NIST.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022.  ASQA: Factoid Questions Meet Long-Form Answers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.

##  Appendix A Potential Issues with ASQA

During our experiments, we notice certain issues regarding the ASQA dataset.
In particular, we observed that in some cases the ambiguity in the question is
pedantic. For instance, there are questions where a date is not specified
(e.g., “What kind of car won the Daytona 500 this year?”), and in a typical
scenario, the system would assume that the question refers to the current
year. Instead, in ASQA this type of question is considered ambiguous, and the
short answers resolve the ambiguity by reformulating the question for
different years (e.g., 2017, 2016, 2015, etc.). In addition, we argue that
some of the disambiguated questions and their short answers are too specific.
This results in the model being penalized for not generating the exact correct
terms, even though its answer semantically covers some part of that
interpretation (e.g., the last example in Table 6).

##  Appendix B Implementation Details

We use the official T5-base, BART-base, and BART-large implementations from
HuggingFace. We train each model for $20$ epochs on the
ASQA777https://huggingface.co/datasets/din0s/asqa dataset with the AdamW
optimizer Loshchilov and Hutter (2017), using a weight decay of $0.01$, and a
learning rate of $10^{-5}$ for T5 and $5\cdot 10^{-6}$ for BART. Training is
stopped early if the validation loss doesn’t decrease after $5$ epochs. We use
a train batch size of $8$ for open-book experiments, and $16$ for closed-book,
with an evaluation batch size of $8$ in both cases. We train our models on one
NVIDIA Titan RTX GPU, and use 16-bit mixed precision to accelerate training.
All of the models converged within $\sim$30 minutes of training.

For our intermediate fine-tuning experiments, we first train T5-base on NLGEN
subset888https://huggingface.co/datasets/din0s/msmarco-nlgen of the MS MARCO
dataset for $1$ epoch with a learning rate of $10^{-4}$ and then continue
training on ASQA as described above. For BART, we use a publicly available
instance from HuggingFace that has been pre-trained on
ELI5999https://huggingface.co/vblagoje/bart_lfqa, and continue training on
ASQA.

Finally, we use beam decoding with $5$ beams and a max sequence length of
$100$ tokens. We force the model to not repeat the same trigram in its output
by using the option no_repeat_ngram_size=3 when generating answers.

##  Appendix C Upper Bound for Retrieval

To investigate the efficiency of our retrieval models, we perform an analysis
for the upper bound of the retrieved passages’ relevancy to the corresponding
gold answers. In particular, for each of the systems used, we count the number
of relevant short answers retrieved using an exact string match with the
dataset’s gold answers. In Table 4, we notice that even the best retrieval
setup at our disposal, namely DPR@5, only fetches $\sim 44\%$ of the relevant
answers on average or up to $58\%$ in cases where it identifies at least one
relevant passage. This emphasizes the need for a multi-hop retrieval system
like JPR Min et al. (2021) in order to fully utilize the power of the first
component of the pipeline. It is evident that a higher upper bound for
retrieval will increase the overall performance of the systems that tackle
ambiguous LFQA, making it a high priority for follow-up work.

| Avg # of short answers retrieved
---|---
| In all results |  | In results with
---
$\geq 1$ correct answer
BM25@1 | 0.52 (14.71%) | 1.61 (45.44%)
BM25@3 | 0.89 (24.10%) | 1.83 (49.35%)
BM25@5 | 1.11 (29.81%) | 1.96 (52.53%)
DPR@1 | 0.89 (24.70%) | 1.74 (48.38%)
DPR@3 | 1.39 (37.73%) | 2.00 (54.61%)
DPR@5 | 1.62 (43.79%) | 2.15 (58.15%)
Table 4: Number of relevant chunks of evidence identified using different
retrieval systems. This constitutes an upper bound to the generative model’s
ability to answer the various disambiguated questions.

##  Appendix D Effect of Intermediate Fine-Tuning for the Closed Book
Experiments

Table 5 displays the impact of the intermediate fine-tuning on a larger LFQA
collection in the closed-book setting, using automated evaluation metrics.
Similarly to the open-book setting, BART-large ELI5 performs the best, which
confirms the hypothesis that larger models are able to benefit the most when
given more training data.

| Rouge-L | Str-EM | Dis-F1 | DR
---|---|---|---|---
T5-base | 30.7 | 3.7 | 2.7 | 9.1
T5-base NLGEN | 30.0 | 2.9 | 2.8 | 9.2
BART-base | 31.5 | 3.9 | 2.8 | 9.3
BART-large | 33.4 | 7.1 | 4.5 | 12.2
BART-large ELI5 | 34.1 | 7.7 | 4.8 | 12.8

Table 5: Closed-book performance, showcasing the effect of intermediate fine-
tuning on a large LFQA dataset.

##  Appendix E Answer Demonstration

We demonstrate a brief selection of answers generated by our best-performing
model (BART-large ELI5) in Table 6. We observe that the main reason for our
model’s inability to disambiguate all parts of the question is that the
passages retrieved with DPR simply do not cover all possible interpretations.
In most cases, the first passage retrieved suffices to answer at least one of
the disambiguated questions, and the rest passages are either not relevant or
answer the same question again. Contrary to the results of Stelmakh et al.
(2022), our model does not hallucinate or repeat itself. We hypothesize that
since their model (T5-large) is almost twice as big as BART-large in terms of
parameter count, it can better utilize its context but at the same time be
overconfident in its answers.

Question, provided disambiguations, and answers |  Generated model output: BART-large ELI5 DPR@3 |  DPR@3 retrieved passages |  Answer analysis
---|---|---|---
Q: Who was the ruler of France in 1830?

Q1: Who was the ruler of France until 2 August 1830?
A1a: Charles Philippe
A1b: Charles X

Q2: Who was the ruler of France after 9 August 1830?
A2: Louis-Philippe I
|  the french revolution of 1830, also known as the july revolution or in french, the second french revolution, led to the overthrow of king charles x, the french bourbon monarch, and the ascent of his cousin louis philippe, duke of orléans, who himself, after 18 precarious years on the throne, would be overthrown in 1848. |  July Revolution The French Revolution of 1830, also known as the July Revolution ("révolution de Juillet"), Second French Revolution or in French ("Three Glorious [Days]"), led to the overthrow of King Charles X, the French Bourbon monarch, and the ascent of his cousin Louis Philippe, Duke of Orléans, who himself, after 18 precarious years on the throne, would be overthrown in 1848.. July Monarchy 1840, to general indifference. The publicś attention was concentrated on the trial of Marie Lafarge.. July Revolution ..led to the overthrow of King Charles X, the French Bourbon monarch, and the ascent of his cousin Louis Philippe, Duke of Orléans, who himself.. |  We do not notice any hallucination or repetition in this example. The model manages to correctly answer the ambiguous question with fluency and coherency. In addition, through reading the answer we understand why the question asked was ambiguous.
Q: The repetition of phrases or sentences with similar structures or meanings
is called?

Q1:The repetition of phrases or sentences with similar structures is called?
A1a: parallel construction
A1b: parallelism
A1c: Parallel syntax

Q2: The repetition of phrases or sentences with similar meanings is called?
A2: Schesis onomaton
|  parallelism, also known as parallel structure or parallel construction, is a balance within one or more sentences of similar phrases or clauses that have the same grammatical structure. the application of parallelism affects readability and may make texts easier to process. |  Parallelism (grammar) In grammar, parallelism, also known as parallel structure or parallel construction, is a balance within one or more sentences of similar phrases or clauses that have the same grammatical structure.. Metathesis (linguistics) Metathesis (; from Greek , from "I put in a different order"; Latin: "trānspositiō") is the transposition.. Resultative Resultative In linguistics, a resultative (abbreviated ) is a form that expresses that something or someone has undergone a change in state as the result of the completion of an event.. |  In this example, the model only manages to get one of the two answers that disambiguate the question. We do not observe any hallucinations or repetition.

Q: When does callie come in grey’s anatomy?

Q1: What date does Callie Torres first appear in Grey’s Anatomy?
A1: February 26, 2006

Q2: Which episode does Callie Torres first appear in Grey’s Anatomy?
A2: "What Have I Done to Deserve This?" (2.19)

Q3: What date does Callie Torres start to appear in Grey’s Anatomy as a series
regular?
A3: September 21, 2006

Q4: What episode Callie Torres start to appear in Grey’s Anatomy as a series
regular?
A4: "Time Has Come Today" (3.01)
|  grey’s anatomy is an american medical drama television series created by shonda rhimes, which airs on the abc in the united states. the character callie torres was introduced in season two, as a senior orthopedic resident as a love interest for intern george o’malley (t.r. knight). she was originally contracted to appear on a recurring basis, but received star billing in the third season. at the end of the show’s twelfth season, ramirez |  Callie Torres …"Callie" Torres, M.D. is a fictional character from the medical drama television series "Greyś Anatomy"… She was introduced in season two, as a senior orthopedic resident, as a love interest for intern George OḾalley (T.R. Knight). Eventually becoming an attending orthopedic surgeon, the character was originally contracted to appear on a recurring basis, but received star billing.. Callie Torres ..Eventually becoming an attending orthopedic surgeon, the character was originally contracted to appear on a recurring basis, but received star billing in the third season..’ Callie Torres Ramirez was nominated for several awards for her portrayal of Torres, including the.. |  Here the model is not able to get any of the answers correctly. However, one could argue that two of the four disambiguated questions were partially answered since the model outputs that the character was introduced in season two and became a series regular in the third season. We also do not notice any hallucinations or repetition.
Table 6:  Qualitative analysis. Green/orange text highlights
correct/partially-correct parts of the answer.


# Hardy-type paradoxes for an arbitrary symmetric bipartite Bell scenario

Kai-Siang Chen  Department of Physics and Center for Quantum Frontiers of
Research & Technology (QFort), National Cheng Kung University, Tainan 701,
Taiwan  Shiladitya Mal  Department of Physics and Center for Quantum Frontiers
of Research & Technology (QFort), National Cheng Kung University, Tainan 701,
Taiwan  Physics Division, National Center for Theoretical Sciences, Taipei
10617, Taiwan  Centre for Quantum Science and Technology, Chennai Institute of
Technology, Chennai 600069, India  Gelo Noel M. Tabia  Department of Physics
and Center for Quantum Frontiers of Research & Technology (QFort), National
Cheng Kung University, Tainan 701, Taiwan  Physics Division, National Center
for Theoretical Sciences, Taipei 10617, Taiwan  Yeong-Cherng Liang
<EMAIL_ADDRESS>Department of Physics and Center for Quantum
Frontiers of Research & Technology (QFort), National Cheng Kung University,
Tainan 701, Taiwan  Physics Division, National Center for Theoretical
Sciences, Taipei 10617, Taiwan

(September 1, 2024)

###### Abstract

As with a Bell inequality, Hardy’s paradox manifests a contradiction between
the prediction given by quantum theory and local-hidden variable theories. In
this work, we give two generalizations of Hardy’s arguments for manifesting
such a paradox to an arbitrary, but symmetric Bell scenario involving two
observers. Our constructions recover that of Meng et al. [Phys. Rev. A. 98,
062103 (2018)] and that first discussed by Cabello [Phys. Rev. A 65, 032108
(2002)] as special cases. Among the two constructions, one can be naturally
interpreted as a demonstration of the failure of the transitivity of
implications (FTI). Moreover, a special case of which is equivalent to a
ladder-proof-type argument for Hardy’s paradox. Through a suitably generalized
notion of success probability called degree of success, we provide evidence
showing that the FTI-based formulation exhibits a higher degree of success
compared with all other existing proposals. Moreover, this advantage seems to
persist even if we allow imperfections in realizing the zero-probability
constraints in such paradoxes. Explicit quantum strategies realizing several
of these proofs of nonlocality without inequalities are provided.

##  I Introduction

In the thought-provoking paper by Einstein, Podolksy, and Rosen [1], the
strong correlations between measurement outcomes have led them to suspect that
quantum theory could be somehow completed (with additional variables). This
was eventually shown to be untenable by Bell [2], who proved that no local-
hidden-variable (LHV) models can reproduce all quantum-mechanical predictions.
In particular, he demonstrated how, with the help of so-called Bell
inequalities, one can experimentally falsify the predictions of LHV models.
Nowadays, we know that Bell nonlocality not only opens the door to answer
fundamental questions in physics but also serves as an important resource for
device-independent quantum information [3, 4].

Interestingly, Bell inequalities are not the only way to manifest Bell
nonlocality [3]. Indeed, Greenberger, Horne, and Zeilinger (GHZ) [5] showed in
their seminal work that a logical contradiction can be demonstrated between
the quantum mechanical prediction on a four-qubit GHZ state and that of any
deterministic LHV model (DLHVM). Soon after, such a contradiction was also
provided for a three-qubit GHZ state [6] and a two-qubit singlet state [7].
This last construction, in particular, was adapted to give the well-known
Peres-Mermin game [8] for showing quantum pseudo-telepathy.

A common feature of these logical proofs is that they rely strongly on the
perfect correlation of maximally entangled states. In contrast, Hardy [9]
provided a different type of logical proof of “nonlocality without inequality”
for a partially entangled two-qubit state. In Hardy’s proof, a contradiction
comes about only when a certain event is observed, see Fig. 1. The probability
at which this event occurs is thus commonly called the success probability, as
it facilitates (initiates) the chain of logical reasoning in Hardy’s
arguments.

Hardy’s original proof was soon generalized to cater for certain bipartite
quantum states of arbitrary local Hilbert space dimension [10], an arbitrary
number of qubit systems [11] (see also [12, 13]), an arbitrary partially
entangled two-qubit state [14], and later to an experimental scenario
involving an arbitrary number of binary-outcome measurement settings [15]. In
the meantime, Stapp [16] showed that Hardy’s argument (which leads to the so-
called Hardy paradox) can also be interpreted as the failure of the
transitivity implications (FTI), thereby demonstrating Bell nonlocality (see
also [17]).

$A_{1}=1$$B_{1}=1$$A_{0}=0$$B_{0}=0$sometimesalwaysalwaysnever (a) Hardy’s
argument

$A_{1}=1$$B_{1}=1$$A_{0}=1$$B_{0}=0$sometimesalwaysneveralways (b) Stapp’s
argument

Figure 1: Schematic representation of (a) Hardy’s argument and (b) Stapp’s
reformulation [16] for demonstrating the inadequacy of DLHVMs in reproducing
quantum predictions. Here, $A_{x}$ and $B_{y}$ represent, respectively, the
outcome ($0$ or $1$) observed by Alice and Bob when she performs the $x$-the
measurement and he performs the $y$-th measurement. In a DLHVM, the
implications (black arrows) and the forbidden event (red) imply that the event
$A_{1}=B_{1}=1$ is also forbidden, yet quantum theory defies this implication.

Several years later, relaxations of Hardy’s original formulation were also
proposed. For example, motivated by Kar’s observation [18] that no mixed two-
qubit entangled states exhibit Hardy’s paradox, Liang and Li [19, 20]
generalized Hardy’s argument by relaxing one of the equality constraints to an
inequality constraint (see also[21]). Indeed, their construction allowed them
to demonstrate a Hardy-type logical contradiction for certain mixed two-qubit
states via a generalized notion of success probability, called degree of
success in [22]. Subsequently, Kunkri et al. [23] showed that this
generalization could give a higher degree of success compared to the original
formulation in [14]. A brief discussion of a further generalization from
Cabello and that of Liang-Li to a scenario with an arbitrary number of
measurement settings was subsequently given in [24].

Indeed, a noticeably higher success probability (or degree of success) can be
obtained if we are willing to consider a Bell scenario with more measurement
settings [15, 24], outcomes [25], or both [26]. As with [26], in this work, we
propose two generalizations of Hardy’s arguments applicable to an arbitrary
(symmetric) bipartite Bell scenario, which recovers, respectively, that of
[26] and that discussed by [20, 23, 24] as a special case. We provide evidence
showing that the one that can be interpreted as a demonstration of FTI leads
to a degree of success higher than all those offered by other existing
proposals, even in the presence of noise.

##  II Hardy’s paradox and its generalization in the simplest Bell scenario

###  II.1 Hardy’s original formulation

Consider the simplest Clauser-Horne-Shimony-Holt (CHSH) Bell scenario, i.e.,
one in which two observers each perform two binary-outcome measurements. Let
$x$ and $y$ ($a$ and $b$) represent, respectively, the setting/ input
(outcome/ output) of Alice and Bob side, and $A_{x}$ ($B_{y}$) $=0,1$ denotes
the outcome of Alice (Bob) when given input $x$ ($y$) $=0,1$. The probability
distribution $\\{P(a,b|x,y)=P(A_{x},B_{y})\\}$ admissible in LHV models can be
described by convex mixtures of local deterministic strategies
$\\{A_{x}=f_{A}(x,\lambda),B_{y}=f_{B}(y,\lambda)\\}$, where $f_{A}$ ($f_{B}$)
is a deterministic function of the input $x$ ($y$) and LHV $\lambda$. The
Hardy paradox of [14] is encapsulated by:

$\begin{split}&P(0,0|0,0)=0,\quad P(1,1|0,1)=0,\\\ &P(1,1|1,0)=0,\quad
P(1,1|1,1)=q>0.\end{split}$ (1)

For DLHVMs, the equality constraints of Eq. 1 imply $P(1,1|1,1)=0$, which
contradicts the inequality constraint of Eq. 1. In other words, together with
the equality constraints, the occurrence of the event $x=y=a=b=1$ contradicts
the prediction of any DLHVM. Consequently, the quantity $q\equiv P(1,1|1,1)$
is also known as the success probability. In quantum theory, it is known [27]
that the maximal attainable success probability is
$\frac{5\sqrt{5}-11}{2}\approx 9.02\%$. Finally, note that a general LHV model
can always be seen mathematically as a convex mixture of DLHVM. Thus, the
observation of Eq. 1 also rules out a general LHV model.

###  II.2 Generalization due to Cabello-Liang-Li

Cabello’s [21] relaxation of Hardy’s argument, originally proposed for a
tripartite scenario and subsequently applied in the bipartite scenario by
Liang and Li [20], takes the form:

$\begin{split}P(0,0|0,0)=p,\quad P(1,1|0,1)=0,\\\ P(1,1|1,0)=0,\quad
P(1,1|1,1)=q.\end{split}$ (2)

Hereafter, we refer to this as the Cabello-Liang-Li (CLL) argument. Compared
with Eq. 1, we see that in this argument, $P(0,0|0,0)$ is allowed to take
nonzero value. From Fig. 1(a), we see that for any DLHVM, if the event
$x=y=a=b=1$ occurs, so must the event $x=y=a=b=0$. However, there exist other
local deterministic strategies (e.g., one where $a=b=0$ regardless of $x$ and
$y$) where the latter event occurs while the former does not. Thus, for the
prediction of a general LHV model, we must have $p\equiv P(0,0|0,0)\geq
q\equiv P(1,1|1,1)$. In other words, one may take the positive value of the
quantity

$q-p=P(1,1|1,1)-P(0,0|0,0)$ (3)

as a witness for successfully demonstrating a logical contradiction based on
such an argument. In [23], the authors refer to $q-p$ as the success
probability of such an argument. However, since $q-p$ is the difference
between two conditional probabilities, we shall follow [22] and refer to $q-p$
instead as the degree of success. In [23], the authors showed that this degree
of success can reach $\approx 10.79\%$.

###  II.3 Our generalization based on FTI

Using Stapp’s reformulation [16] of Eq. 1 as an FTI, we now provide a
different relaxation of Hardy’s paradox via:

$\begin{split}P(0,0|0,0)=0,\quad P(1,1|0,1)=r,\\\ P(1,1|1,0)=0,\quad
P(1,1|1,1)=q,\end{split}$ (4)

From Fig. 1(b), we see that for any DLHVMs, if the event $x=y=a=b=1$ occurs,
so must the event for $x=0$ and $y=a=b=1$. However, there are local
deterministic strategies where the converse does not hold. Thus, for a general
LHV model (obtained by averaging the deterministic ones), we must have
$r\equiv P(1,1|0,1)\geq q\equiv P(1,1|1,1)$. Hence, in analogy with the CLL
argument, we refer to

$q-r=P(1,1|1,1)-P(1,1|0,1)$ (5)

as the degree of success of such an argument. In [28], it was shown that in
quantum theory, the largest value of $q-r$ is $\frac{1}{8}=12.5\%$, attainable
by performing projective measurements on a two-qubit pure state and higher
than that achievable with Eq. 2.

####  II.3.1 Maximal degree of success for $2$-qubit pure states

In fact, the general two-qubit pure state and observables satisfying the zero-
probability constraints of Eq. 4 are [28]:

$\displaystyle\ket{\psi}=\sin\theta\left(\cos{\alpha}\ket{0}-\sin\alpha\ket{1}\right)\\!\ket{1}+\cos\theta\ket{1}\\!\ket{0},$
(6a) $\displaystyle\begin{split}A_{0}=\sigma_{z},\quad
A_{1}=\cos{2\alpha}\,\sigma_{z}-\sin{2\alpha}\,\sigma_{x},\\\
B_{0}=\sigma_{z},\quad
B_{1}=\cos{2\beta}\,\sigma_{z}-\sin{2\beta}\,\sigma_{x},\end{split}$ (6b)

where $\theta,\alpha,\beta\in[0,\pi]$. From here, the corresponding degree of
success of Eqs. 5 and 6, as a function of the parameters $\theta$, $\alpha$,
and $\beta$ can be shown to be:

$\begin{split}P_{\text{succ}}(\theta,\alpha,\beta)=&\frac{1}{2}\sin{\alpha}\left[\left(\cos{2\beta}\cos{2\theta}-1\right)\sin{\alpha}\right.\\\
&\left.+\sin{2\beta}\sin{2\theta}\right]\end{split}$ (7)

Naturally, one may wonder which entangled two-qubit pure state gives the
largest value of $P_{\text{succ}}$. To this end, note that the entanglement of
the two-qubit state of Eq. 6a, as measured according to the concurrence [29],
is

$C(\ket{\psi})=|\sin{2\theta}\cos{\alpha}|.$ (8)

Using variational techniques, the largest degree of success that we have found
for given concurrence $C$ is

$P^{*}_{\text{succ}}(C)=\frac{\sqrt{1-C^{2}}}{2}\left(1-\sqrt{1-C^{2}}\right),$
(9)

for which $\theta=\beta\in\\{\frac{\pi}{4},\frac{3\pi}{4}\\}$. From Fig. 2, it
is clear that for any given concurrence, this degree of success is always
larger that from the CLL argument, which, in turn, is larger than that from
Hardy’s original formulation.

Figure 2: The maximal degree of success (which reduces to the success
probability in the Hardy argument) in demonstrating a proof of nonlocality
without inequality for given concurrence [29]. From top to bottom, we plot
$q-r$ of Eq. 4 for our FTI argument (red, solid), $q-p$ of Eq. 2 for CLL’s
argument (green, dashed, see [23]), and $q$ of Eq. (1) for Hardy’s argument
(blue, dashed-dotted, see [14]). Note that the entangled state that gives the
largest degree of success differs from one formulation to the other.

####  II.3.2 FTI argument with noise

Evidently, imperfections in any realistic experimental scenario make it
essentially impossible to realize the zero-probability equality constraints in
all these different formulations. To understand the impact of these
imperfections, we now relax Eq. 4 and consider

$\begin{split}P(0,0|0,0)\leq\epsilon,\quad P(1,1|0,1)=r,\\\
P(1,1|1,0)\leq\epsilon,\quad P(1,1|1,1)=q,\end{split}$ (10)

where $\epsilon$ is the error bound. For the CLL argument, the same discussion
has been made in [22]. In Fig. 3, we show the corresponding maximal degree of
success, i.e., $q-r$ from Eq. 10, when the noise $\epsilon\in(0,0.5)$. Our
results clearly show that for any amount of noise in this range, the FTI
argument gives a higher maximal degree of success compared to the CLL one.

Note that for our relaxed FTI argument of Eq. 10, upper bounds on the maximum
degree of success (based on level-3 of the semidefinite programming (SDP)
hierarchy introduced in [30]) coincide with the lower bounds (based on two-
qubit pure state with rank-1 projective measurements) to within a numerical
precision of $10^{-7}$. However, if we consider only qubit strategies, then as
already noted in [22] (see their Fig. 1), there appears to be a gap between
the maximal degree of success achievable with such strategies and the upper
bound obtained from SDP for some values of $\epsilon$. Upon closer inspection,
we find that for $0.158\lesssim\epsilon<0.5$, the SDP upper bound is
attainable to within the same precision by considering a convex mixture of the
qubit strategy for $\epsilon=0.158$ and $\epsilon=1$, or equivalently a
ququart strategy obtained from their direct sum.

Figure 3: Degree of success as a function of the deviation $\epsilon$ from the
zero-probability constraints. The top curve (red, solid) shows a lower bound
on the degree of success for our FTI argument (obtained by optimizing over
two-qubit states and projective measurements); the middle curve (green,
dashed) shows an upper bound on the degree of success for CLL’s argument (see
also [22]) obtained using level-3 of the semidefinite programming (SDP)
hierarchy described in Ref. [30]. The inset is to highlight the difference
between the two degrees of success (provided, respectively, by our FTI
argument and the CLL argument), which is always positive within a numerical
precision of $10^{-7}$. Note that two probabilities coincide when
$\epsilon\approx 0.0730$.

##  III Generalization of Hardy’s proof beyond the simplest Bell scenario

The ladder proof [15]Cereceda [24]Meng et al. [26]Chen et al. [25]Generalized
CLL [Eq. 11]Generalized FTI [Eq. 15]Cabello-Liang-Li (CLL) [19, 20, 21]Hardy’s
paradox [14]The FTI [Eq.
4]$p=0$$p=0$$r=0$$d=2$$k=2$$d=2$$k=2$$d=k=2$$k=2$$d=2$$p=0$$r=0$ Figure 4:
Summary of the relationships between the various Hardy-type paradoxes
discussed in this work. Our new constructions are printed in boldface.

Having understood how Hardy and Hardy-type paradoxes work in the simplest Bell
scenario, the time is now ripe to discuss their generalization to more complex
Bell scenarios. In this section, we propose, respectively, a generalization of
both the Hardy-type paradox of CLL, Eq. 2, and that based on FTI, Eq. 4, to an
arbitrary bipartite $k$-input $d$-output Bell scenario, i.e., one in which
both party has a choice over $k$ alternative $d$-outcome measurements.

###  III.1 Generalization of CLL Hardy-type paradox

Specifically, for the CLL Hardy-type paradox, the following conditions on the
joint conditional probabilities:

$\displaystyle\left\\{\begin{aligned}
&P(A_{k-1}<B_{k-1})=q,\leavevmode\nobreak\ \text{if}\leavevmode\nobreak\
k\in\text{odd},\\\ &P(A_{k-1}>B_{k-1})=q,\leavevmode\nobreak\
\text{if}\leavevmode\nobreak\ k\in\text{even},\end{aligned}\right.$ (11a)
$\displaystyle P(A_{i}>B_{i-1})=0,\leavevmode\nobreak\
\forall\leavevmode\nobreak\ i\in\text{odd}\cap\\{1,\cdots,k-1\\},$ (11b)
$\displaystyle P(A_{i}<B_{i-1})=0,\leavevmode\nobreak\
\forall\leavevmode\nobreak\ i\in\text{even}\cap\\{1,\cdots,k-1\\},$ (11c)
$\displaystyle P(A_{i-1}>B_{i})=0,\leavevmode\nobreak\
\forall\leavevmode\nobreak\ i\in\text{odd}\cap\\{1,\cdots,k-1\\},$ (11d)
$\displaystyle P(A_{i-1}<B_{i})=0,\leavevmode\nobreak\
\forall\leavevmode\nobreak\ i\in\text{even}\cap\\{1,\cdots,k-1\\},$ (11e)
$\displaystyle P(A_{0}<B_{0})=p,$ (11f)

together with $q>p$ define our generalization of this paradox, where the
outcomes $A_{x},B_{y}$ may take $d$ possible values, say, from
$\\{0,1,\cdots,d-1\\}$. For the special case of $p=0$, one obtains a
generalization of the original Hardy paradox to an arbitrary bipartite
$k$-input $d$-output Bell scenario. If we further set $d=2$, then the
construction reduces to one equivalent (under relabeling of inputs and
outputs) to the ladder proof of nonlocality [15]. If, instead, we take $d=2$
in Eq. 11 without setting $p=0$, one obtains the argument briefly discussed in
[24]. All these relations are summarized in Fig. 4.

$A_{k-1}=s^{A}_{k-1}$$B_{k-1}=s^{B}_{k-1}$$A_{k-2}=s^{A}_{k-2}$$B_{k-2}=s^{B}_{k-2}$$A_{1}=s^{A}_{1}$$B_{1}=s^{B}_{1}$$A_{0}=s^{A}_{0}$$B_{0}=s^{B}_{0}$$q$$p$
Figure 5: Logical structure of the generalized CLL arguments in the $k$-input
$d$-output Bell scenario. The generalization of Hardy’s original argument to
these cases is recovered by setting $p=0$.

To see that the constraints of Eq. 11 with $q-p>0$ indeed constitute a proof
of nonlocality without inequality, we begin by restricting our attention to a
DLHVM where the measurement outcomes take definite values, denoted by
$\\{A_{i}=s^{A}_{i}\\}$ and $\\{B_{i}=s^{B}_{i}\\}$, where
$i\in\\{0,1,\dots,k-1\\}$. We depict the logical structure behind this
argument schematically in Fig. 5. Let us now consider the case of even and odd
$k$ separately, starting with odd $k$. Then, in order for a DLHVM to reproduce
LABEL:Extend_Hardy_a, i.e., $P(A_{k-1}<B_{k-1})=q>0$, the model must produce
events $\\{A_{i}=s^{A}_{i}\\}$ and $\\{B_{i}=s^{B}_{i}\\}$ such that
$s^{A}_{k-1}<s^{B}_{k-1}$. Similarly, the other constraints of Eq. 11 imply
constraints on the relationship between $\\{s^{A}_{i}\\}$ and
$\\{s^{B}_{i}\\}$, where $i\in\\{0,1,\dots,k-1\\}$. For example, together with
the conditions of Eq. 11c and Eq. 11e for $i=k-1$, i.e.,
$P(A_{k-1}<B_{k-2})=0$ and $P(A_{k-2}<B_{k-1})=0$, we get

$s^{B}_{k-2}\leq s^{A}_{k-1}<s^{B}_{k-1}\leq s^{A}_{k-2}.$ (12)

By considering the other zero-probability constraints one at a time for the
remaining $i$, we arrive at

$s^{A}_{0}\leq\dots\leq s^{B}_{k-2}\leq s^{A}_{k-1}<s^{B}_{k-1}\leq
s^{A}_{k-2}\leq\dots\leq s^{B}_{0}.$ (13)

This means that for any DLHVMs that give $s^{A}_{k-1}<s^{B}_{k-1}$, the
constraints of Eq. 13 imply that they must also give $s^{A}_{0}<s^{B}_{0}$.
However, there can be other DLHVMs where $s^{A}_{0}<s^{B}_{0}$ holds even
though $s^{A}_{k-1}\not<s^{B}_{k-1}$. Thus, for a general LHV model, the
conditions of Eq. 13 imply that $p\geq q$. In other words, a nonzero value of
the degree of success $q-p$ witness Bell nonlocality without resorting to a
Bell inequality.

Similarly, for even $k$, starting from $P(A_{k-1}>B_{k-1})=q>0$ and by
considering the other zero-probability constraints lead to, for any DLHVMs,

$s^{A}_{0}\leq\dots\leq s^{A}_{k-2}\leq s^{B}_{k-1}<s^{A}_{k-1}\leq
s^{B}_{k-2}\leq\dots\leq s^{B}_{0}.$ (14)

Again, this observation implies that $p\geq q\Leftrightarrow p-q\geq 0$ for
any LHV model for all $k\geq 2$ and $d\geq 2$.

###  III.2 Generalization of the FTI-based Hardy-type paradox

Next, let us describe our generalization of the FTI-based Hardy-type paradox
from Eq. 4, which consists of the following conditions:

$\displaystyle P(A_{k-1}<B_{k-1})=q,$ (15a) $\displaystyle
P(A_{i}<B_{i-1})=0,\leavevmode\nobreak\ \forall\leavevmode\nobreak\
i\in\\{1,\cdots,k-1\\},$ (15b) $\displaystyle
P(B_{i-1}<A_{i-1})=0,\leavevmode\nobreak\ \forall\leavevmode\nobreak\
i\in\\{1,\cdots,k-1\\},$ (15c) $\displaystyle P(A_{0}<B_{k-1})=r,$ (15d)

and the requirement of $q>r$. The special case of $r=0$, which can be seen as
a generalization of Stapp’s argument [16], has been proposed and discussed in
[26]. To recover Eq. 4 from Eq. 15, one sets $k=d=2$ and apply the relabeling
$A_{i}=0\leftrightarrow A_{i}=1$ for all $i\in\\{0,1\\}$. In Fig. 6, we depict
schematically the logical structure of this paradox.

As with our explanation to Eq. 11, for any DLHVM satisfying
$P(A_{k-1}<B_{k-1})=q>0$, the model must produce events
$\\{A_{i}=s^{A}_{i}\\}$ and $\\{B_{i}=s^{B}_{i}\\}$ such that
$s^{A}_{k-1}<s^{B}_{k-1}$. At the same time, the other inequality constraints
from Eq. 15 imply $s^{B}_{k-2}\leq s^{A}_{k-1}$, $s^{A}_{k-2}\leq
s^{B}_{k-2}$, etc., leading to

$s^{A}_{0}\leq s^{B}_{0}\leq\dots\leq s^{A}_{k-2}\leq s^{B}_{k-2}\leq
s^{A}_{k-1}<s^{B}_{k-1},$ (16)

which implies $s^{A}_{0}<s^{B}_{k-1}$. This means that with the zero-
probability constraints, a DLHVM equipped with a strategy giving
$s^{A}_{k-1}<s^{B}_{k-1}$ must also give $s^{A}_{0}<s^{B}_{k-1}$. Again, other
DLHVM may give $s^{A}_{0}<s^{B}_{k-1}$ even though
$s^{A}_{k-1}\not<s^{B}_{k-1}$. Thus, from Eq. 15d, we conclude that for a
general LHV model (obtained by averaging over local deterministic strategies),
we must have $r\geq q\Leftrightarrow r-q\geq 0$ for all $k,d\geq 2$.

$A_{k-1}=s^{A}_{k-1}$$B_{k-1}=s^{B}_{k-1}$$A_{k-2}=s^{A}_{k-2}$$B_{k-2}=s^{B}_{k-2}$$A_{1}=s^{A}_{1}$$B_{1}=s^{B}_{1}$$A_{0}=s^{A}_{0}$$B_{0}=s^{B}_{0}$$q$$r$
Figure 6: Logical structure of our FTI arguments in the $k$-input $d$-output
Bell scenario. Generalized Stapp’s arguments as introduced in [26] are
recovered by setting $r=0$.

###  III.3 Proof of equivalence of generalized Stapp’s proof and generalized
ladder proof of nonlocality

Interestingly, the authors of [26] proved that in the $k$-input $2$-output
scenario, the generalized Stapp’s argument, Eq. 15 with $r=0$, and the ladder
proof of nonlocality, cf. Eq. 11 with $p=0$, are equivalent. In other words,
these two sets of conditions can be obtained from each other via an
appropriate relabeling of inputs and outputs. In what follows, we show that
this equivalence also holds for arbitrary $k,d\geq 2$.

######  Theorem III.1.

For any symmetric bipartite Bell scenario, the set of conditions given in Eq.
11 with $p=0$ is equivalent to the set of conditions of Eq. 15 with $r=0$.

###### Proof.

Let us rewrite Alice’s and Bob’s measurement outcomes in Eq. 11, respectively,
as $A^{\prime}_{x^{\prime}}$ and $B^{\prime}_{y^{\prime}}$, and let

$\displaystyle\mathscr{L}_{1k-}:=\\{1,2,\ldots,\lfloor\tfrac{k}{2}\rfloor-1\\},\quad\mathscr{L}_{1k}:=\mathscr{L}_{1k-}\cup\\{\lfloor\tfrac{k}{2}\rfloor\\}$
$\displaystyle\mathscr{L}_{0k-}:=\\{0\\}\cup\mathscr{L}_{1k-},\quad\mathscr{L}_{0k}:=\mathscr{L}_{0k-}\cup\\{\lfloor\tfrac{k}{2}\rfloor\\}$
(17)

For odd $k$, one may verify that the following relabeling

$\begin{split}A^{\prime}_{x^{\prime}}=\left\\{\begin{array}[]{ll}A_{\ell-1},&x^{\prime}=k-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{1k}\\\
A_{k-1-\ell},&x^{\prime}=k-1-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{0k}\end{array}\right.,\\\
B^{\prime}_{y^{\prime}}=\left\\{\begin{array}[]{ll}B_{k-1},&y^{\prime}=k-1\\\
B_{k-1-\ell},&y^{\prime}=k-2\ell\leavevmode\nobreak\ \land\leavevmode\nobreak\
\ell\in\mathscr{L}_{1k}\\\
B_{\ell-1},&y^{\prime}=k-1-2\ell\leavevmode\nobreak\ \land\leavevmode\nobreak\
\ell\in\mathscr{L}_{1k}\end{array}\right.,\\\ \end{split}$ (18)

transforms the $2k$ conditions of Eq. 11 to those of Eq. 15. To see this, note
that under this transformation, the condition of LABEL:Extend_Hardy_a stays as

$P(A^{\prime}_{k-1}<B^{\prime}_{k-1})=P(A_{k-1}<B_{k-1})=q,$ (19)

which is Eq. 15a. With the transformation, the conditions of Eq. 11b, Eq. 11d,
Eq. 11c, and Eq. 11f with $p=0$, respectively, become the requirements that
each of the following probabilities vanish:

$\displaystyle
P(A^{\prime}_{k-2\ell}>B^{\prime}_{k-2\ell-1})=P(A_{\ell-1}>B_{\ell-1}),\,\,\ell\in\mathscr{L}_{1k},$
$\displaystyle
P(A^{\prime}_{k-2\ell-1}>B^{\prime}_{k-2\ell})=P(A_{k-1-\ell}>B_{k-1-\ell}),\,\,\ell\in\mathscr{L}_{1k};$
$\displaystyle
P(A^{\prime}_{k-2\ell-1}<B^{\prime}_{k-2\ell-2})=P(A_{k-1-\ell}<B_{k-2-\ell}),\,\,\ell\in\mathscr{L}_{0k-};$
$\displaystyle
P(A^{\prime}_{0}<B^{\prime}_{0})=P(A_{\frac{k-1}{2}}<B_{\frac{k-1}{2}-1}).$
(20)

In addition, the conditions of Eq. 11e become

$\displaystyle
P(A^{\prime}_{k-2\ell-2}<B^{\prime}_{k-2\ell-1})=P(A_{\ell}<B_{\ell-1}),\,\,\ell\in\mathscr{L}_{1k-}\text{
and}$ $\displaystyle P(A^{\prime}_{k-2}<B^{\prime}_{k-1})=P(A_{0}<B_{k-1}),$
(21)

To summarize, the requirement that the probabilities in the first two lines of
Section III.3 vanish is identical to the condition of Eq. 15b, the requirement
that the probabilities in the last two lines of Section III.3 and the first
line of Section III.3 vanish is identical to the condition of Eq. 15c, and the
requirement that the probability in the last line of Section III.3 vanishes is
identical to the condition of Eq. 15d.

Similarly, for even $k$, one can verify that the following relabeling

$\begin{split}A^{\prime}_{x^{\prime}}=\left\\{\begin{array}[]{ll}d-1-A_{\ell-1},&x^{\prime}=k-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{1k}\\\
d-1-A_{k-1-\ell},&x^{\prime}=k-1-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{0k}\end{array}\right.,\\\
B^{\prime}_{y^{\prime}}=\left\\{\begin{array}[]{ll}d-1-B_{k-1},&y^{\prime}=k-1\\\
d-1-B_{k-1-\ell},&y^{\prime}=k-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{1k}\\\
d-1-B_{\ell-1},&y^{\prime}=k-1-2\ell\leavevmode\nobreak\
\land\leavevmode\nobreak\ \ell\in\mathscr{L}_{1k}\end{array}\right.,\\\
\end{split}$ (22)

transforms the $2k$ conditions of Eq. 11 to those of Eq. 15. ∎

For completeness, we show in Footnote 2 bounds on the the maximal degree of
success found for different Hardy-type paradoxes in several $k$-input
$2$-output and $k$-input $3$-output Bell scenarios; analogous results for a
larger number of outputs are shown in Table 2. One worth noticing thing is
that for all these numerical results, we observe that the degree of success
from our FTI arguments is always higher than that obtained from all these
other proposals.

$k$ | $2$ | $3$ | $4$ | $5$ | $6$
---|---|---|---|---|---
Boschi [15] | UB | $0.09020$ | $0.17459$ | $0.23132$ | $0.27095$ | $0.29999$
LB | $0.09017^{*}$ | $0.17455$ | $0.23126$ | $0.27088$ | $0.29995$
Cereceda [24] | UB | $0.10785$ | $0.18523$ | $0.23801$ | $0.27546$ | $0.30327$
LB | $0.10781^{*}$ | $0.18519$ | $0.23796$ | $0.27542$ | $0.30321$
FTI-based | UB | $0.12501$ | $0.20713$ | $0.25976$ | $0.29579$ | $0.32192$
LB | $0.125^{*}$ | $0.20711$ | $0.25973$ | $0.29576$ | $0.32190$
(a) Results for binary-output Bell scenarios. From top to bottom, we list the
results for Hardy paradox given by the ladder proof of nonlocality without
inequality [15] (see also [26]), generalized CLL’s argument due to [24], and
our FTI-based formulation [Eq. 15]. The quantum strategies corresponding to
the LBs are provided in [31]. Entries marked with * are known to be tight
quantum bound.

$k$ | $2$ | $3$ | $4$ | $5$
---|---|---|---|---
Meng [26] | UB | $0.14194$ | $0.26782$ | $0.34823$ | $0.40196$
LB | $0.14189$ | $0.26779$ | $0.34816$ | $0.40184$
CLL-type | UB | $0.16794$ | $0.28272$ | $0.35706$ | $0.40763$
LB | $0.16791$ | $0.28265$ | $0.35698$ | $0.40753$
FTI-based | UB | $0.19313$ | $0.31230$ | $0.38467$ | $0.43225$
LB | $0.19309$ | $0.31226$ | $0.38460$ | $0.43216$
(b) Results for ternary-output Bell scenarios. From top to bottom, we list the
results for generalized Hardy paradox given by Meng et al. [26], our
generalized CLL formulation [Eq. 11], and our FTI-based formulation [Eq. 15].
The quantum strategies corresponding to the LBs are provided in [31].

Table 1: Comparison of the best upper bound (UB) and the best lower bound (LB)
found on the degree of success for three different Hardy and Hardy-type
paradoxes beyond the CHSH scenarios with $k\geq 2$ inputs. See also Fig. 4 for
the relationship between all these paradoxes. The UBs were obtained by
considering level-1222Due to numerical issues, we find worse upper bounds when
considering higher-level of the SDP hierarchy. of the SDP hierarchy introduced
in Ref. [30]. The LBs, obtained numerically, may be attained using the
strategies provided in [31] (see also Appendix A).

$d$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$
---|---|---|---|---|---|---
Chen [25] | UB | $0.09020$ | $0.14194$ | $0.17769$ | $0.20470$ | $0.22623$ | $0.24406$
LB | $0.09017^{*}$ | $0.14189$ | $0.17657$ | $0.20489$ | $0.22425$ | $0.13768$
CLL-type | UB | $0.10785$ | $0.16794$ | $0.20945$ | $0.24116$ | $0.26643$ | $0.28726$
LB | $0.10781^{*}$ | $0.16791$ | $0.20883$ | $0.23987$ | $0.25903$ | $0.14923$
FTI-based | UB | $0.12501$ | $0.19313$ | $0.23844$ | $0.27181$ | $0.29787$ | $0.31904$
LB | $0.125^{*}$ | $0.19309$ | $0.23839$ | $0.27175$ | $0.29526$ | $0.15888$

Table 2: Comparison of the best upper bound (UB) and the best lower bound (LB)
found on the degree of success for the Hardy paradox due to [25], our
generalized CLL formulation [Eq. 11], and our FTI-based formulation [Eq. 15]
for Bell scenarios with two inputs and $d\geq 2$ outputs. See also Fig. 4 for
the relationship between all these paradoxes. The UBs were obtained by
considering level-1 of the SDP hierarchy introduced in Ref. [30] (see also
Footnote 2). The LBs, obtained numerically, may be attained using the
strategies provided in [31]. Entries marked with * are known to be tight
quantum bound.

##  IV Discussion

Hardy and Hardy-type paradoxes are fascinating proofs of Bell nonlocality
without resorting to Bell inequalities. Aside from fundamental interests (see,
e.g., [32, 33, 34, 28]), they are also known to be relevant in the task of
randomness amplification [35] (see, e.g., [36, 37]). In this work, we propose
a Hardy-type paradox that can be naturally understood via the failure of the
transitivity of implications (FTI), cf. [16, 17].

As with the Hardy-type paradoxes formulated by Cabello-Liang-Li (CLL) [21,
20], we show that a degree of success generalizing the notion of success
probability—whose non-negative value witnesses Bell-nonlocality—may be
introduced for the FTI-based Hardy-type paradox. In the simplest Bell scenario
with two inputs and two outputs, we show that the new FTI-based formulations
give the highest degree of success among all existing (i.e., Hardy, CLL, and
FTI-based) formulations. Moreover, this advantage persists even when the zero-
probability constraints required in all these formulations are relaxed.

Then, we provide—as with [26] for the original Hardy paradox—a generalization
of the FTI-based formulation and the CLL-type formulation, to symmetric Bell
scenarios involving an arbitrary number of inputs and outputs. In turn, this
allows us to show that a ladder-type, cf. [15], and an FTI-based proof of
nonlocality without inequality are equivalent for an arbitrary symmetric Bell
scenario, thereby generalizing the result of [26] for the binary-outcome Bell
scenarios.

For several simple Bell scenarios, we further observe (see Footnote 2 and
Table 2) numerically that our FTI-based generalizations provide the largest
value of the degree of success. A natural question left open from the current
work is to determine if this trend continues to hold for an arbitrary,
symmetric Bell scenario. Another closely related problem concerns the observed
monotonically-increasing behavior of the degree of success when one increases
either the number of inputs $k$ or the number of outputs $d$ involved within
each type of logical argument — an analytic proof of this observation will be
more than welcome.

On the other hand, given the close connection found [38, 28, 39] between the
optimizing strategy for a Hardy paradox and its self-testing [40] property, it
would also be interesting to see if the optimizing correlations found for
these new generalizations are also self-testing (and non-exposed [33]). From
an application perspective, one may also be interested in the potential of
such correlations for device-independent applications, especially in
randomness amplification [35], and proofs of Bell-nonlocality in the presence
of measurement independence [32].

###### Acknowledgements.

We thank Ashutosh Rai for helpful discussions. This work is supported by the
National Science and Technology Council, Taiwan (Grants No.
109-2112-M006-010-MY3, 112-2628-M006-007-MY4).

##  Appendix A Quantum Strategies

In this Appendix, we give some further information about the quantum
strategies that reproduce our best lower bound (LB) on the various degrees of
success shown in Footnote 2 and Table 2. The actual quantum strategies for
each case are available at [31]. For convenience, we refer to the various
generalizations of Hardy [14] due to Boschi et al. [15], Chen et al. [25], and
Meng et al. [26] as Hardy paradoxes. Indeed, in all these three cases, the
degree of success is exactly the success probability of observing such a
paradox.

Next, notice that the best quantum strategies we have found for the Hardy,
CLL-type, and FTI-based arguments for a $k$-input, $d$-output Bell scenario,
denoted by $(k,d)$, are always attained by performing rank-1 projective
measurements on a pure quantum state $\ket{\psi}$ residing in the two-qudit
Hilbert space $\mathbb{C}^{d}\otimes\mathbb{C}^{d}$. However, due to numerical
imprecisions, the zero-probability constraints are not always strictly
enforced in all these optimal strategies found. For completeness, we list in
Table 3 the largest deviation found among all the zero-probability constraints
for each of these “optimal" strategies.

$(k,d)$ | Hardy | CLL | FTI
---|---|---|---
$(3,2)$ | $2.4065\times 10^{-14}$ | $1.1241\times 10^{-14}$ | $1.0108\times 10^{-15}$
$(4,2)$ | $3.7173\times 10^{-14}$ | $1.5651\times 10^{-15}$ | $7.3663\times 10^{-16}$
$(5,2)$ | $1.31\times 10^{-15}$ | $3.207\times 10^{-10}$ | $1.5557\times 10^{-12}$
$(6,2)$ | $2.268\times 10^{-15}$ | $1.6756\times 10^{-11}$ | $4.0153\times 10^{-12}$
$(2,3)$ | $1.4965\times 10^{-6}$ | $1.3404\times 10^{-16}$ | $2.9385\times 10^{-16}$
$(3,3)$ | $2.4822\times 10^{-9}$ | $4.9713\times 10^{-16}$ | $3.8448\times 10^{-10}$
$(4,3)$ | $1.6744\times 10^{-11}$ | $2.075\times 10^{-14}$ | $4.8237\times 10^{-14}$
$(5,3)$ | $9.9977\times 10^{-14}$ | $4.7704\times 10^{-14}$ | $4.8075\times 10^{-12}$
$(2,4)$ | $7.2538\times 10^{-7}$ | $1.9799\times 10^{-16}$ | $3.3075\times 10^{-16}$
$(2,5)$ | $7.9181\times 10^{-5}$ | $1.2006\times 10^{-6}$ | $5.0784\times 10^{-8}$
$(2,6)$ | $1.2162\times 10^{-4}$ | $4.1223\times 10^{-4}$ | $6.5519\times 10^{-5}$
$(2,7)$ | $2.6871\times 10^{-3}$ | $3.6214\times 10^{-3}$ | $3.8616\times 10^{-4}$
Table 3:  Summary of the largest zero-probability-constraint violation for
each optimal strategy [31] used to give the best lower bound on the degree of
success presented in Footnotes 2 and 2.

## References

  * Einstein  _et al._ [1935] A. Einstein, B. Podolsky, and N. Rosen, Phys. Rev. 47, 777 (1935).
  * Bell [1964] J. S. Bell, Physics 1, 195 (1964).
  * Brunner  _et al._ [2014] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, Rev. Mod. Phys. 86, 419 (2014).
  * Scarani [2012] V. Scarani, Acta Physica Slovaca 62, 347 (2012).
  * Greenberger  _et al._ [1989] D. M. Greenberger, M. A. Horne, and A. Zeilinger, “Going beyond bell’s theorem,” in  _Bell’s Theorem, Quantum Theory and Conceptions of the Universe_, edited by M. Kafatos (Springer Netherlands, Dordrecht, 1989) pp. 69–72.
  * Greenberger  _et al._ [1990] D. M. Greenberger, M. A. Horne, A. Shimony, and A. Zeilinger, Am. J. Phys. 58, 1131 (1990).
  * Peres [1990] A. Peres, Phys. Lett. A 151, 107 (1990).
  * Mermin [1990] N. D. Mermin, Phys. Rev. Lett. 65, 3373 (1990).
  * Hardy [1992] L. Hardy, Phys. Rev. Lett. 68, 2981 (1992).
  * Clifton and Niemann [1992] R. Clifton and P. Niemann, Phys. Lett. A 166, 177 (1992).
  * Pagonis and Clifton [1992] C. Pagonis and R. Clifton, Phys. Lett. A 168, 100 (1992).
  * Jiang  _et al._ [2018] S.-H. Jiang, Z.-P. Xu, H.-Y. Su, A. K. Pati, and J.-L. Chen, Phys. Rev. Lett. 120, 050403 (2018).
  * Luo  _et al._ [2018] Y.-H. Luo, H.-Y. Su, H.-L. Huang, X.-L. Wang, T. Yang, L. Li, N.-L. Liu, J.-L. Chen, C.-Y. Lu, and J.-W. Pan, Sci. Bull. 63, 1611 (2018).
  * Hardy [1993] L. Hardy, Phys. Rev. Lett. 71, 1665 (1993).
  * Boschi  _et al._ [1997] D. Boschi, S. Branca, F. De Martini, and L. Hardy, Phys. Rev. Lett. 79, 2755 (1997).
  * Stapp [1993] H. P. Stapp,  _Mind, Matter and Quantum Mechanics_ (Springer Verlag, 1993).
  * Liang  _et al._ [2011] Y.-C. Liang, R. W. Spekkens, and H. M. Wiseman, Phys. Rep. 506, 1 (2011).
  * Kar [1997] G. Kar, Phys. Lett. A 228, 119 (1997).
  * Liang and Li [2003] L.-M. Liang and C.-Z. Li, Phys. Lett. A 318, 300 (2003).
  * Liang and Li [2005] L.-M. Liang and C.-Z. Li, Phys. Lett. A 335, 371 (2005).
  * Cabello [2002] A. Cabello, Phys. Rev. A 65, 032108 (2002).
  * Rai  _et al._ [2021] A. Rai, M. Pivoluska, M. Plesch, S. Sasmal, M. Banik, and S. Ghosh, Phys. Rev. A 103, 062219 (2021).
  * Kunkri  _et al._ [2006] S. Kunkri, S. K. Choudhary, A. Ahanj, and P. Joag, Phys. Rev. A 73, 022346 (2006).
  * Cereceda [2017] J. L. Cereceda, Quantum Studies: Mathematics and Foundations 4, 205 (2017).
  * Chen  _et al._ [2013] J.-L. Chen, A. Cabello, Z.-P. Xu, H.-Y. Su, C. Wu, and L. C. Kwek, Phys. Rev. A 88, 062116 (2013).
  * Meng  _et al._ [2018] H.-X. Meng, J. Zhou, Z.-P. Xu, H.-Y. Su, T. Gao, F.-L. Yan, and J.-L. Chen, Phys. Rev. A 98, 062103 (2018).
  * Rabelo  _et al._ [2012] R. Rabelo, L. Y. Zhi, and V. Scarani, Phys. Rev. Lett. 109, 180401 (2012).
  * Chen  _et al._ [2023] K.-S. Chen, G. N. M. Tabia, C. Jebarathinam, S. Mal, J.-Y. Wu, and Y.-C. Liang, Quantum 7, 1054 (2023).
  * Wootters [1998] W. K. Wootters, Phys. Rev. Lett. 80, 2245 (1998).
  * Moroder  _et al._ [2013] T. Moroder, J.-D. Bancal, Y.-C. Liang, M. Hofmann, and O. Gühne, Phys. Rev. Lett. 111, 030501 (2013).
  * [31] Download ancillary files at arXiv for the explicit quantum strategies leading to these success probabilities.
  * Pütz  _et al._ [2014] G. Pütz, D. Rosset, T. J. Barnea, Y.-C. Liang, and N. Gisin, Phys. Rev. Lett. 113, 190402 (2014).
  * Goh  _et al._ [2018] K. T. Goh, J. Kaniewski, E. Wolfe, T. Vértesi, X. Wu, Y. Cai, Y.-C. Liang, and V. Scarani, Phys. Rev. A 97, 022104 (2018).
  * Rai  _et al._ [2019] A. Rai, C. Duarte, S. Brito, and R. Chaves, Phys. Rev. A 99, 032106 (2019).
  * Colbeck and Renner [2012] R. Colbeck and R. Renner, Nat. Phys. 8, 450 (2012).
  * Ramanathan  _et al._ [2018] R. Ramanathan, M. Horodecki, H. Anwer, S. Pironio, K. Horodecki, M. Grünfeld, S. Muhammad, M. Bourennane, and P. Horodecki, arXiv:1810.11648 (2018).
  * Zhao  _et al._ [2023] S. Zhao, R. Ramanathan, Y. Liu, and P. Horodecki, Quantum 7, 1114 (2023).
  * Rai  _et al._ [2022] A. Rai, M. Pivoluska, S. Sasmal, M. Banik, S. Ghosh, and M. Plesch, Phys. Rev. A 105, 052227 (2022).
  * Liu  _et al._ [2023] Y. Liu, H. Y. Chung, and R. Ramanathan, “Investigations of the boundary of quantum correlations and device-independent applications,” arXiv:2309.06304 (2023).
  * Šupić and Bowles [2020] I. Šupić and J. Bowles, Quantum 4, 337 (2020).