# Secret-Keeping in Question Answering

Nathaniel W. Rollings* Kent O’Sullivan* Sakshum Kulshrestha
Department of Computer Science
University of Maryland
{nrolling, osullik<EMAIL_ADDRESS>
* equal contribution by the authors

###### Abstract

Existing question answering research focuses on unanswerable questions in the
context of always providing an answer when a system can…but what about cases
where a system should not answer a question. This can either be to protect
sensitive users or sensitive information. Many models expose sensitive
information under interrogation by an adversarial user. We seek to determine
if it is possible to teach a question-answering system to keep a specific fact
secret. We design and implement a proof-of-concept architecture and through
our evaluation determine that while possible, there are numerous directions
for future research to reduce system paranoia (false positives), information
leakage (false negatives) and extend the implementation of the work to more
complex problems with preserving secrecy in the presence of information
aggregation.

##  1 Introduction

###  1.1 Protecting secrets and serving answers

Answering Questions. It is evident from the survey completed by Rogers et al.
that the majority of work in Question Answering (QA) seeks to improve
accuracy. For a long time, the focus has been maximizing the availability of
an answer given text. Work into the answerability of questions is improving
the integrity of returned information. No work we have found has explicitly
considered the confidentiality of answers in question answering. The focus has
been exhaustively on whether a QA system can answer a question accurately. In
this paper, we ask should a question be answered accurately, and how do we
prevent secret information from being disclosed by a QA system? Rogers et al.
contend that work in QA systems is increasingly driven by commercial demand,
emphasizing the impact of the assertion from Roy and Anand that the objectives
of a commercial system may differ from the broader goal of building a QA
system with complex and higher reasoning ability. The requirement to provide
confidentiality features in QA systems with access to sensitive corporate data
is almost certain, and there has been a lack of work examining the problem.
Worryingly, a 2022 study by Jagielski et al. indicates that memorization of
training data is more likely on recently observed examples in Large Language
Models (LLMs). As the focus of QA shifts towards answer generation Baradaran
et al. (2022), systems like ChatGPT OpenAI (2022) are increasingly likely to
be deployed in a corporate setting. Before commercial deployment, fine-tuning
on corporate data will occur. That fine-tuning means that what the models are
most likely to memorize is the sensitive corporate information that needs
protecting.

Protecting Information. Current approaches to protect against the leaking of
secrets are inadequate. Censoring secrets in the potential answer context are
suboptimal. Work by Song and Shmatikov in 2019 showed that censoring training
data degrades performance and, in some cases, is reversible, exposing the
sensitive material. Complete redaction is an option to protect secrets;
however, a counter-factual analysis by Kandpal et al. suggests that
(unsurprisingly) the overall performance of a generative QA model drops when
the context is redacted. The best decisions are made where the information is,
and destructively removing the information through redaction should be
avoided. A detailed exploration of related work is in section 6, but to build
intuition about why we should care about this problem of whether a computer
should be able to answer every question, consider the following fictional
scenario:

Motivation. After years of complaints from Wile E. Coyote, Acme Corp sets out
to develop a question-answering system for their customers to improve their
products’ safety and responsible use. Acme expects most users to ask how-to
questions before or first-aid questions after using a product. To ensure that
the system can answer as many questions as possible, they make available to it
every official document ever written by Acme - technical reports, patents,
product designs, investigations, reviews, meeting minutes etc. Initial testing
is promising. The system can accurately answer most of the common questions
like "How do I decrease the thrust on the Acme rocket skates" and "How to
treat burns from Acme little-giant firecracker". Unfortunately, it also
happily provides answers containing proprietary (and dangerous) information.
Testers found that asking "how do I build an Acme Self-Guided Aerial Bomb"
yields detailed instructions on how to build the device. Not wanting to give
away confidential designs or be liable for a spate of DIY bomb-makers, Acme
tried to remove the underlying information about the Acme Self-Guided Aerial
Bomb. They found that redacting the references to the Bomb’s construction also
degraded the system’s ability to answer questions about whether or not the
Self Guided Aerial bomb is made of ethically sourced materials and what type
of fuel needs to be put in it. So, how does Acme Corp stop their system from
leaking damaging or dangerous information into the wrong hands while releasing
other information? How do they maximize their ability to share information
while protecting their secrets?

QA Systems. The generation of succinct answers to questions over increasingly
heterogeneous modalities is achieved by question answering (QA). QA systems
seek to provide a direct answer to an information need posed by a user in
natural language Roy and Anand (2021). QA systems can be characterized by
their question input, context input and output. Input questions can be
information seeking, where the user is trying to gain knowledge they do not
yet have, or probing, where the user is confirming what knowledge a system has
Rogers et al. (2023). The source of the material a QA system will answer
questions about is called the context. The context of a QA system is typically
sourced from a structured knowledge base or an unstructured collection.
Unstructured collections can cover any modality but are most heavily
concentrated on unstructured text Roy and Anand (2021). Systems optimized for
unstructured text are also known as reading comprehension or machine reading
systems. Outputs of a QA system can be extractive returning a span of text or
knowledge base entry located within the context to satisfy the information
need, generative, creating a novel answer to the information requirement,
multi-choice selecting from a list of possibilities or categorical for
example, yes/no Rogers et al. (2023).

Evaluating QA Systems. Current QA evaluation focuses on measuring the
’accuracy’ of returned answers: did the provided answer satisfy the
information requirement of the question with respect to the context? These
measures are typically evaluated against a ’gold standard’ dataset. These gold
standards contain, at a minimum: contexts, questions and acceptable answers.
Extractive, multi-choice and categorical systems are typically measured using
exact match accuracy, F1 score or one of a few specialized metrics outlined by
Baradaran et al.. Generative systems also use specialized measures primarily
based on n-gram comparisons; detail is available in work by Baradaran et al..
The 2023 survey paper by Rogers et al. gives a detailed overview of available
datasets for evaluation. Most follow the gold-standard pattern with variations
to the source of questions and context. Most relevant to protecting
information is work exploring answerability, the measure of whether or not a
QA system is capable of answering a given question. Roy and Anand offer three
common reasons for questions being unanswerable: low confidence in the answer,
the requested information being not in the context and when the correct answer
is null. Rajpurkar et al. published the SQUAD2.0 dataset in 2018, allowing
systems to evaluate against unanswerable questions. Work by Wallace et al. in
2019 shows how deliberately constructing adversarial questions can make
questions unanswerable, and the QuAIL dataset published in 2020 by Rogers et
al. expands further on answerability, noting that the systems they tested were
rarely able to identify if a question was unanswerable with an accuracy
greater than chance. All of these metrics focus on whether a system can answer
a question, but none of them addresses if an answer it provided should have
been provided.

Research Question. How can we implement a secret-keeping system capable of
protecting secret information from disclosure without inflicting unacceptable
degradation on the system’s ability to provide non-secret answers to
questions?

Organization. The remainder of the paper gives key definitions, explains the
design of our system in section 2, our experiments in section 3 and the
results in section 4. Sections 5, 6 and 7 outline the ethics, related work and
future work, respectively.

Contributions. Our key contributions are:

  * •

Problem Definition. We identify the gaps in assuring confidentiality in QA
systems and introduce secret-keeping as a solution.

  * •

Architecture. We develop a flexible architecture that can be easily adapted to
different question-answering systems to protect secret information from
unauthorized disclosure.

  * •

Evaluation We develop evaluation metrics to assess the effectiveness of a
secret-keeping model.

###  1.2 What is a secret?

Before building systems that prevent the leaking of secret information, we
need to define what a secret is, how we imagine users interacting with the
system that can access secret and non-secret material, and what can go wrong:

  * •

A secret is an atomic fact or a relationship between entities that should not
be disclosed. We define the metric of secrecy as the proportion of answers
containing secrets that were correctly identified by the secret-keeper.

  * •

Interrogation is an adversarial interaction with the question-answering entity
where questions are deliberately constructed to induce information leakage. An
effective interrogator can introduce information leakage by crafting
adversarial in-band interactions with the system111See what happened to
ChatGPTsamczsun (2022); Johnson (2022) or by employing out-of-band privacy
attacks exploiting the memorization of training examples like those described
by Kandpal et al. in their 2022 paper.

  * •

Information leakage we use similarly to Bakhtin et al. 2022as use in 2022,
referring to situations where the [question answering] agent reveals
compromising information about its plan [secret]. We define the information
leakage metric as the proportion of answers containing secrets not identified
by the secret-keeper (.i.e. False Negatives).

  * •

Paranoia describes when the QA agent refuses to answer a question that does
not contain a secret. We define the paranoia metric as the proportion of
answers not containing secrets that were incorrectly identified as containing
secrets by the secret-keeper (i.e. False Positives).

##  2 Designing a system to keep secrets

Design Principles. Our review of the literature in section 6 identifies
several areas of related work that have informed the design of our system. Key
influences are noted in our list of design principles used to develop the
secret keeper approach below:

  1. 1.

The system must minimize information leakage as work in agent based systems
reviewed in section 6.2 identifies as a priority.

  2. 2.

The system should minimize paranoia so that it performs effectively as a QA
system for non-secret information as described in section 6.1.

  3. 3.

The system should not destroy context to prevent information loss as explored
in section 6.5.

  4. 4.

The system should generalize to extractive, generative, multi-choice and
categorical QA.

  5. 5.

The sanitizing should be invisible to users, concealing omission balances
ethics and efficacy as discussed in section 5.

As an initial proof-of-concept, we deliberately limit our scope to extractive,
machine-reading QA systems. Our design principles prioritize an approach that
focuses on filtering the output of a system rather than redacting or censoring
the input. The filtering of output is referred to as sanitization by Carlini
et al. in their 2019 paper, asserting that it is the best practice for
processing sensitive and private data. We implement the following secret
keeping architecture:

Secret Keeper Architecture. We designed the output-sanitization architecture
shown in figure 2 to offer significant flexibility to a practitioner. By
focusing exclusively on the output we make our system agnostic to the
underlying model, QA method and context. The secret keeper only stops secrets
from being disclosed. It determines if a QA answer is a secret by asking the
same question the QA system is answering of its own secret context. The secret
context contains only secrets. The secret keeper takes uses the cosine
similarity of the QA answer. If the similarity is >0.5, the QA answer is
determined to be secret and the output is sanitized.

##  3 Verifying that secrets are being kept

We approach the experimental design in three phases.

###  3.1 Phase 1. Baseline Assessment

The purpose of our baseline assessment is to measure the performance of each
of our QA models on an unmodified context without interference from the
secret-keeper. The input questions and context are both sourced from the SQUAD
Dev Set222Squad Dev set Rajpurkar et al. (2016). The secrets are a non-
disjoint subset of the SQUAD Dev Set. The context has not been redacted and
the secret keeper is not active. We measure the Accuracy of the QA using the
SQUAD gold standard answers, and the information leakage and paranoia of the
system using the gold standard answers and membership of the secret subgroup.
The models we use are:

  * •

distilbert-base-cased-distilled-squad333Huggingface: ditsilbert-base-cased-
distilled-squad

  * •

roberta-base-squad2444Huggingface: roberta-base-squad2

Figure 1: Redaction architecture. Secrets are identified and removed from the
data store prior to access by the QA system. At runtime, the QA system
interacts directly with this clean data store.

###  3.2 Phase 2. Redacted Context Assessment

The purpose of our redacted context assessment is to understand how severe the
information loss is should destructive redaction to remove secrets occur. The
input questions and secrets are unchanged from Phase 1, and we reuse the
roberta-base-squad2 model for the QA sytsem. We implement aggressive redaction
of secret material from the context in a pre-processing phase. The pre-
processing work involves generating sentence-level embeddings for both the
secret context and the full SQUAD Dev set context. Unlike the sanitization
approach, this redaction must make some assumptions about the size and
structure of potential secrets because it has no question and QA model to
identify potential question answers. Those embeddings are compared, and only
sentences from the context that do not have a corresponding embedding in the
secret context are added to the redacted context store. After the pre-
processing is complete, the user can ask questions drawing answers from the
redacted context. The process is depicted in Figure 1

###  3.3 Phase 3. Secret Keeping

The purpose of the secret-keeping assessment is to understand the tradeoff
between information leakage, paranoia and QA accuracy that our proposed
secret-keeping model offers. The input questions, contexts and secrets are
replicated from Phase 1. The secret keeping is implemented as an output
sanitization function following the QA module of the system. The process is
outlined in figure 2.

Figure 2: The output-sanitization system architecture. Both the question-
answering and secret-keeping subsystems receive the question and employ a QA
model to generate answers. The secret-keeping system has access to a data
store containing only secret information, while the question-answering system
has access to the full (secret and non-secret) data. The results are passed
through a sentence encoder, and the cosine similarity between the embeddings
is then compared. If over a threshold determined by the user risk profile, the
question-answering subsystem’s result is flagged as secret and is not returned
to the user.

To compare the models in the three phases, we conducted our experiment by
adjusting three key settings to explore the associated hypotheses:

Design | Model | Samples | Accuracy $\uparrow$ | Paranoia $\downarrow$ | Leakage $\downarrow$
---|---|---|---|---|---
Baseline | RoBERTa | 27,000 | 0.91 | 0.00 | 0.43
Baseline | DistilBERT | 27,000 | 0.86 | 0.00 | 0.43
Sanitization | RoBERTa | 27,600 | 0.65 | 0.03 | 0.25
Sanitization | DistilBERT | 27,000 | 0.67 | 0.03 | 0.26
Sanitization | Custom Model | 27,250 | 0.65 | 0.04 | 0.23
Secret Remover | RoBERTa | 27,250 | 0.67 | 0.00 | 0.26

Table 1: Macro Results show Secret Remover is the most accurate non-baseline
model and minimizes paranoia, but the fine-tuned output-sanitization model
minimizes secret leakage. The baselines set the upper bound on accuracy for
the sanitization and secret remover systems employing these QA models.
Removing secrets necessitates that some correct answers are no longer
available and results in lower accuracy. However, they are unable to identify
secrets and therefore provide any secret information requested by the user.

H1 - As the number of secrets kept increases, paranoia rates will increase and
QA accuracy will decrease. To test our theory, we varied the number of
passages to flag as ’secret’ between 0 and 32. In the Acme scenario, this
represents the amount of information they mark as secret, with higher values
representing more topics in their system being protected.
H2 - As the amount of context available to the Secret Keeper decreases,
information leakage will increase. The real world has incomplete and changing
contextual knowledge. To simulate these conditions, we varied the percentage
of the text from a secret we provided to the secret keeper between 25% and
100%, expecting its performance to degrade as the amount of contextual
information decreases. While Acme may be able to mark all secret information
as such in our previous scenario, this will not always be the case. Some
documents may slip past their initial review, and more documents may be added
without updating the secret data store. The context ratio here represents how
much of the information on these secrets have been marked as such, and
therefore is accessible to the secret keeper.
H3 - As the number of questions about secrets increases, information leakage
will increase. To simulate varying intensities of questioning, we change the
ratio of questions about secrets in the evaluation set. Our limited proxy
implementation of interrogation used randomly selected questions over the
secret context, but a more likely real-world scenario would involve targeted
questions related to the topic of interest. However, this approach is intended
to provide some insights into how the model behaves as the focus of questions
becomes more tightly centred on secrets. We expect the model’s accuracy to
approach that of the underlying reader QA system as we decrease the focus on
secrets. Conversely, we expect the simulation of an attacker homing in on the
secret by increasing the rate of questions to result in overall accuracy and
an increase in leakage. This approach represents increasingly aggressive
questioning by Wile E. Coyote as he tries to get secret information about the
failures of Acme products (which they have marked as secret).

Our experiment framework for the number of secrets, context ratio, and
question ratio was applied to the three models we implemented into the output-
sanitization architecture and the single model implemented in the secret
removal architecture. This process yielded a total of 325 discrete
experiments, generating over 190,000 question attempts in our evaluation set.

##  4 Evaluating the effectiveness of secret-keeping

The evaluation of these experiments consists of a quantitative evaluation of
runtimes and overall accuracy, paranoia, and leakage of each model as well as
a qualitative investigation of failure cases to gain a deeper understanding of
weaknesses in these approaches to the secret-keeping problem.

(a) Runtime and accuracy by model

(b) Runtime and accuracy by number of secrets

Figure 3: Comparison of model runtime and accuracy. We see that having more
secrets to protect degrades the runtime, but not accuracy, of secretRemover
and customCheckpoint, but the other output-sanitization models remain
unaffected.

###  4.1 Quantitative Results

Adding a secret keeper to a model along with a secret context will reduce the
system’s overall question-answering accuracy since the intent is to not return
some correct answers as they are secret. As a result, we see in Table 1 that
the accuracy of each secret-keeping approach is notably lower than that of the
base QA system without a secret keeper. We also observe a substantial decrease
in leakage when using all of these models instead of the base system, but the
output-sanitization models also introduce paranoia not present in the base
models – they are unable to identify secrets, so they will never mistakenly
classify a non-secret as one.

Our results supported our hypothesis to varying extents:

H1: Keeping many secrets causes (mild) paranoia. We varied the number of
secrets that the system was required to keep between 1 and 32 to test H1. Our
results in Figure 5(b) show a weak correlation in the number of secrets kept
to the occurrence of false positives, steadily increasing from 0.01 to 0.04 as
the number of secrets increases. Figure 4(b) shows that this effect is most
evident when the majority of questions do not target a secret and false
positives are more likely. Our qualitative analysis in 4.2 suggests that this
phenomenon is a result of our cosine similarity metric relating numerical
words, dates, or names with similar patterns together. The exception is the
secret remover which has a false positive rate of zero. We attribute the lack
of a false positive to its use of full sentences in its embeddings rather than
just the phrase returned by a QA model.

H2: A lack of context causes information leakage. As we varied the amount of
secret context available to the secret keeper, simulating keeping secrets on
domains with incomplete knowledge. Our results in Figure 5(c) show that when
the secret keeper doesn’t have access to all the information it needs to
explicitly keep secret, its ability to identify secrets across the domain
decreases. However, we do see some generalization as access only to a quarter
of the secret context still allows nearly 70% of secrets to be identified, and
full access to the secret context still fails to catch nearly 10% of the
secrets returned by the QA system due to the selection of incorrect secret
answers.

H3: Interrogation causes information leakage Our experiment varied the ratio
of questions asked that were attempting to compromise the secret to simulate
what we call the interrogation mode. Figures 4(a) and 5(a) show when asked
more questions pursuing a secret, the probability of leaking the secret
increases substantially. Increasingly aggressive interrogation has a much more
pronounced impact than that observed in other experimental changes, as shown
in Figures 5(c) and 5(b). At the most extreme, nearly half of all secrets were
leaked. This experiment opens the door for developing an adversarial question
set to conduct interrogation of a secret and provide a basis for testing more
advanced systems.

(a) Output-Sanitization RoBERTa with 0.5 secret context

(b) Output-Sanitization RoBERTa with 1.0 secret context

Figure 4: Results using the RoBERTa Output-Sanitization system with varying
secret context coverage. Accuracy in the output-sanitization secret model
decreases linearly as the frequency of questions touching on the secret
increases with partial context but remains consistent as the secret question
frequency increases when provided access to the full secret context.

We also found some additional valuable results across the models. While these
were not directly related to our original hypotheses, they provide useful
insights into comparisons between the different approaches.

The secret removal approach scales poorly with more secrets while the output-
sanitization models see consistent performance. Considering Figure 3, we see
this method takes up to 16 times longer than the output-sanitization models as
the number of secrets increases. Its performance degrades because the secret
removal system must compute embeddings for all sentences in the secret context
and all sentences in the QA context and then conduct comparisons on the cosine
similarity of all combinations of those embeddings. While it only has to do
this once, the time requirement grows significantly as more secrets are added.

The underlying model drives performance for the output-sanitization designs.
From Figure 3, we can conclude the runtime performance is heavily dependent on
the underlying model for the output sanitization architecture. Two of our
three models were pre-trained with no modification, and both outperformed the
runtime of the fine-tuned model and showed minimal variation as the number of
secrets increased. While fine-tuning remains a viable approach and may provide
valuable improvements with additional refinement, our initial experimentation
reveals that unmodified pre-trained models perform remarkably well on this
task.

The secret remover is more accurate than the output-sanitization systems. As
shown in Table 1, the secret remover has higher accuracy and lower paranoia
than any of the output-sanitization systems. However, it struggles with
leakage, and its time requirements and more fragile design present additional
concerns. In particular, it relies on secrets being sentences, an assumption
that does not hold true outside of this experiment and one that is not
required by the output-sanitization secret-keeping approach. While it could
use other structures instead of sentences, it relies on comparing only these
structures, unlike the more flexible QA systems which can select answers of
varying lengths and formats based on the question asked and the context
supplied. Additionally, adding new secrets requires further expensive
computation and declassification becomes impossible as currently designed
because the information no longer exists. Finally, in cases where the original
QA context for an answer contained both secrets and non-secrets, this approach
may remove critical contextual information for identifying a correct but non-
secret answer.

(a) Failure cases when varying rate of interrogative questions. The increasing
focus on questions targeting secret information drives down false positives
(paranoia) across models but results in dramatic increases in false negatives
(leakage).

(b) Failure cases when varying the number of secrets kept. Increasing the
number of secrets kept drives a small but consistent increase in false
positives (paranoia).

(c) Failure cases when varying the amount of available secret context.
Increasing the amount of secret context available to models improves
performance, but we still see some generalization with only a quarter of the
secret context provided to the system.

Figure 5: Failure cases showing the impact of each variable. All results are
averaged across all models for every other combination of variables.

###  4.2 Qualitative Evaluation

Our investigation of the failure cases identified a number of intriguing
scenarios that provide some insights into the causes of these failure modes.

Question: What mountain has snow on it all year round? (With secret passage on
Kenya)
QA Answer: Mount Kenya
Secret Answer: Mount Wilson
Result: No Match
Example 1: Secrets leak when the secret keeper gets the answer wrong:
Occasionally, the secret keeper model will select an incorrect secret answer
from the secret context while the QA system finds the correct answer in its
context that is also a secret. As a result, the QA answer, which reveals
secret information, slips through undetected. As seen here, this may happen
when the QA answer is heavily dependent on the specific context provided. The
secret keeper model is not wrong: Mount Wilson does have snow on it all year,
but so does Mount Kenya. When multiple secret contexts may provide correct
answers to a question, this system is likely to make mistakes. Additionally,
it highlights that the underlying model used for the secret QA provides an
upper bound on the ability to prevent leakage. If this model selects incorrect
answers, it may not be able to select a valid answer from the secret context
in order to compare it against the original QA system’s output.

Question: How many weight rooms are in the Malkin Athletic Center (With secret
passage on Chemistry)
QA Answer: three
Secret Answer: two
Result: Match
Example 2: Numbers and dates cause paranoia (false positives): Numeric
responses always produce high similarity, even if there is little overlap in
the original context, causing false positives.

Question: What was the name of the Florida Huguenot colony? (With secret
passage on Jacksonville Beaches)
QA Answer: Fort Caroline
Secret Answer: Fort Caroline
Result: Match
Example 3: Domain collision can cause paranoia (false positives): The above is
an example of when both models work correctly, but the contexts have unrelated
matching information. The secret keeper is drawing from a context about the
area around Jacksonville, while the QA model is drawing from a context
discussing Huguenots, which have overlapping information since some of these
Huguenots settled near Jacksonville. In this case, providing this answer as
the name of the colony in the context of a question about the Huguenots rather
than about Jacksonville would not have revealed secret information, causing a
false positive.

Question: What type of medicine did Otachi focus on? (With secret passage on
Pharmacists)
QA Answer: herbal remedies
Secret Answer: herbal medicine
Result: Match
Example 4: Differing contexts may determine if the same answer reveals a
secret and sometimes leads to paranoia (false positives): Again, both models
work correctly, but the coincidental similarity in the contexts produces a
false positive. The QA model was drawing from a context on the Yuan dynasty
and a well-known doctor at the time, Otachi. The secret context is on
pharmacists. However, one listed specialization of pharmacists is herbal
medicine. Unlike the previous example, both the QA and secret models are
referencing the same entity (herbal medicine/remedies), but the difference in
source contexts separates the secret related to pharmacists from a historical
reference to a doctor in China.

This analysis highlights some key weaknesses of this approach, but it also
suggests some avenues for improvement. First, tracking entities within the
context around the QA and secret keeper answers may help resolve the issues
highlighted in Examples 2 and 3. If we are able to confirm that the "two"
response secret keeper is referring to chemical compounds and the "three" from
the QA system refers to the number of rooms in an athletic center, then we may
choose to return the QA result despite the high cosine similarity. However,
this approach also introduces some additional concerns as we ultimately care
about what the user can learn from the answer. If a user asked the question
"What is the name of a beach in Florida?" with the QA system still returning
Fort Caroline (from the colony context) as an answer, the user would still be
in possession of a secret despite the accidental domain collision. In our
experiments, the questions were tied to relevant contexts as part of the
dataset, but in the wild, that may not always be the case. Consideration not
only of the context of the QA and secret keeper answers but also the question
being asked will be critical in further improving these models.

##  5 Is it right for machines to keep secrets?

Is it ethical for a QA system to withhold information from a user, or lie to
them? A strong argument exists that the good stemming from enforcing a
requirement for honest and truthful AI far outweighs any potential harms
caused by lies or negligent falsehoods (Evans et al., 2021). Reflecting on our
motivating example, the question of whether it is more harmful for Acme to
release the components and assembly instructions for the Acme Self-Guided
Aerial Bomb or for a computer to deliberately withhold that information from a
user suggests that there are some scenarios where withholding of information
may be in the ’right’ thing to do. Proceeding under the premise that there
exist circumstances where information should be secret, what is the best
method?

Omission versus Lying Neither humans nor computers are particularly effective
at detecting deception (Peskov et al., 2020), but humans tend to be better at
detecting lies that can be verified by checking another source. Obvious
omission is suspicious, but covert omission is more effective than the use of
outright lies (Van Swol et al., 2012). The effectiveness of omission over
lying about the answer is likely linked to the ability to fact-check incorrect
statements but the inability to fact-check that which is unknown. From an
ethical perspective, the use of omission or glomarisation to conceal
information is seen as less problematic than outright lying (Evans et al.,
2021). While out of scope for this paper, we identify significant future work
in approaches of satisficing or glomarization to concealing information from a
user. Preventing a user from commencing interrogative interactions with the
system increases the overall security of information and ameliorates some
ethical concerns about active deception.

Risks of a Secret File An obvious risk of our approach to secret-keeping is
that the system owner is required to maintain a central repository of all
secrets. The repository will be a highly desirable target for attackers. Our
current implementation requires the secrets to be in plaintext, but future
work should examine how to improve the security of the secret store without
unacceptably detracting from the system’s performance.

##  6 Related Work in protecting sensitive information

Although the problem of secret keeping in QA systems is new, there are related
efforts in QA, agent-based systems, content moderation, LLM Privacy, spoiler
detection, censorship, and sanitization that have informed our work.

###  6.1 Question Answering

Recent work in extractive question answering focuses on answerability, not
protecting secrets. We examined modern QA datasets including Comqa (Abujabal
et al., 2018), HotpotQA (Yang et al., 2018), and Natural Questions
(Kwiatkowski et al., 2019). They focus on maximizing the availability of
answers rather than preventing answers from being given. Notably, both
SQUAD2.0 (Rajpurkar et al., 2018) and QuAC (Choi et al., 2018) include the
notion of unanswerable questions. Unfortunately, these are focused on poorly
formed questions and do not explore deliberate secret-keeping.

The removal of examples from training data reduces the ability of the LLMs to
correctly answer questions. Kandpal et al. (2022a) found a correlation between
the frequency of examples in training and the likelihood of its generation by
the trained model. They also found that the removal of training material
significantly degrades the ability of both LLM and retrieval-based QA models
to answer factoid questions. The negative impact on QA accuracy caused by
removing concepts from training data drives our design choice for a system
that checks for secret leakage at query-time, rather than during training.

###  6.2 Agent-based systems and reasoning

Adversarial inputs could induce the disclosure of strategically sensitive
information that undermines an agent’s objectives. A 2022 work by Bakhtin et
al. develops CICERO, a system that can play the game Diplomacy. Their
supplementary material notes that by conditioning on intent they are able to
improve the perplexity of generated text (Bakhtin et al., 2022b). The impact
on performance suggests that an understanding of what and why a piece of
information is important and improves the ability to reason about its value.
They also found that post-generation filtering of messages sent to other
players to be a useful approach in protecting the agent’s strategic
intentions. We take a similar approach in checking for secret leaks after the
generation of responses.

###  6.3 Content Moderation

Traditional content moderation using blacklist keyword matching or ’harmful’
or ’not harmful’ text classification suffers from domain dependence and
lexical fragility. A 2020 Report from the Transatlantic Working Group, led by
Llansó et al. asserts that the rapid defeat of the Jigsaw Perspective API was
due to an inability to handle variations in the lexical inputs. Keyword-based
content moderation for the Cicero system had to be supplemented with domain-
specific terminology to make it functional Bakhtin et al. (2022b). Active
learning approaches like that described by Markov et al. in their 2022 paper
showed they are more robust to lexical variances in user behavior attempting
to bypass content moderation and warrants further investigation for secret-
keeping.

###  6.4 Spoiler Detection

Spoiler detection is conceptually aligned with secret keeping but does not
protect secrets. Early work in spoiler detection sought to use classification
approaches to determine if spans of text are spoilers or not spoilers. Work by
Boyd-Graber et al. demonstrated it is possible to detect spoilers through
classification. Further work by Wan et al. in 2019 applies a neural approach
and Chang et al. in 2021 develop a Graph Neural Network that leverages
dependency relations to improve their performance. While conceptually similar,
spoiler detection is like content moderation in that it tries to detect the
language of spoilers as an open-ended problem, rather than focusing on
protecting a discrete secret as we propose here.

###  6.5 Memorization and Forgetting

Recent work in memorization by Jagielski et al. (2022) suggests that recently
seen training examples are more likely to be memorized, and so fine-tuning an
LLM on domain-specific data means it is more likely to leak that domain-
specific information. Memorization refers to when a trained neural network
reveals the presence of training data. In 2019 Carlini et al. observed that
rare examples can still be memorized and that it is very difficult to detect
and prevent unintentional memorization from occurring. In 2021 Carlini et al.
extended their prior work to show how an attacker can cause the indiscriminate
extraction of training examples from large language models. That result was
further extended by Tramèr et al. in 2022 who showed that by poisoning the
training input they are able to increase information leakage using what they
term active inference attacks. Effective, indiscriminate attacks to leak
training examples are very problematic for secret-keeping. These approaches
focus on ’private’ or ’sensitive’ categories of information rather than a
specific protected secret fact. Additionally, they are effectively side-
channel attacks that an interrogator may resort to in an attempt to have a
secret disclosed. We focus our secret-keeping system on in-band interactions
but note that in a pipeline design like the one we propose, it is possible to
detect the likely disclosure of a secret and prevent it, even if the example
is memorized by the model.

###  6.6 Censorship, Sanitization and Anonymization

Sanitizing the output of a system is superior to censoring inputs or
anonymizing text for preventing information leakage, but the methods available
to sanitize outputs are lacking. Mozes and Kleinberg assert that current
approaches to text anonymization by removing PII from text offer insufficient
guarantees of success. Ultimately, their proposed framework to address the
guarantee requires human review and as a result, is not scalable enough for
general use. Carlini et al. state that sanitizing outputs based on blacklists
is unable to handle the many variations of PII, finding that their attempt to
implement a sanitizer by training multiple models to sanitize output did not
reliably capture all PII. While the sanitization approach may not work for the
problem of protecting any PII, we believe it is likely to be the most
effective approach for protecting specific secrets because of the
substantially reduced domain variance. We largely derive the idea to use of
one QA model to sanitize the output of another from their work, though we
implement them in a different manner. Song and Shmatikov (2019) suggest
censorship of inputs as an approach to combat the memorization of training
examples in LLMs. They observe that while the memorization of sensitive
material decreases, so does the model accuracy. Further, they present a de-
censorship attack which is able to reveal some elements of censored input
data. Based on their results we contend that allowing the model access to all
training material and then effectively sanitizing the output of secret
material is a superior approach to censoring input or complete redaction.

##  7 Conclusion and Future Work

We have introduced the task of secret-keeping as an important, and under-
explored problem in question answering. We identify a lack of suitable secret-
keeping metrics and define secrecy, paranoia and information leakage to
address the gap. We design and implement a secret-keeping approach that is
model-agnostic, only requiring access to pre-defined secrets, and the output
of a QA system to, detect the disclosure of secrets.

We have identified a rich field for future work in secret-keeping including:

  * •

Reducing Paranoia and Information Leakage. How do we reduce the rate of
paranoia against dates, names and other patterns with high cosine similarity
but low semantic similarity?

  * •

Gold-Standard Dataset. We need to generate a gold-standard dataset for secret-
keeping, or extend current datasets to support secret-keeping to benchmark
performance for future efforts in the area.

  * •

Other QA Methods. Our architecture is flexible and so can support any QA
answer as input, but formal testing is needed on the ability to handle
generative, multi-choice and categorical systems. Categorical in particular
will be challenging as yes/no answers will likely require inspection of the
question as well.

  * •

Memorization attacks. As with Other QA methods, we expect that the results of
memorization attacks that disclose secrets can be addressed with secret-
keeping, but formal experimentation is required.

  * •

Resistance to Interrogation Extending the work by Markov et al. to apply
active learning to reduce the information leakage induced by interrogation.

  * •

Information Aggregation. Identifying atomic facts and relationships that are
secrets is feasible, but how can a secret-keeper track the aggregate knowledge
of an interrogator who asks many innocuous questions and combines the answers
themselves?

  * •

Interrogation detection. How can a QA system detect adversarial patterns of
questions that indicate an interrogation might be underway? Can the system
adjust its strictness of secret-keeping in response to defeat it?

  * •

Satisficing and Glomarization. Can the QA system generate answers using
techniques like those described by Evans et al. that will appease them or
distract them so that they will not commence an interrogation to help protect
information, supporting efforts by Bakhtin et al. and Tabatabaei and Jamroga
to protect intent-based systems?

  * •

Secret Security How can we protect secrets without storing them in a way that
they are vulnerable to direct access by a threat actor?

Our experiments demonstrate that Secret information leaks from unprotected QA
systems, and complete redaction of secret material is time-consuming,
destructive and still prone to information leakage. Secret-Keeping offers
users the ability to trade-off between paranoia and information leakage
without disproportionately sacrificing accuracy in addition to providing
extensive flexibility in selecting QA models.

##  8 Acknowledgments

The authors gratefully acknowledge the technical contributions of Ryan Van
Voorhis and Ben Moskowitz.

## References

  * Abujabal et al. (2018) Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and Gerhard Weikum. 2018\.  Comqa: A community-sourced dataset for complex factoid question answering with paraphrase clusters.  _arXiv preprint arXiv:1809.09528_.
  * Bakhtin et al. (2022a) Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022a.  Human-level play in the game of diplomacy by combining language models with strategic reasoning.  _Science_ , page eade9097.
  * Bakhtin et al. (2022b) Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. 2022b.  Supplementary materials to: Human-level play in the game of diplomacy by combining language models with strategic reasoning.  _Science_.
  * Baradaran et al. (2022) Razieh Baradaran, Razieh Ghiasi, and Hossein Amirkhani. 2022.  A survey on machine reading comprehension systems.  _Natural Language Engineering_ , 28(6):683–732.
  * Boyd-Graber et al. (2013) Jordan Boyd-Graber, Kimberly Glasgow, and Jackie Sauter Zajac. 2013.  Spoiler alert: Machine learning approaches to detect social media posts with revelatory information.  _Proceedings of the American Society for Information Science and Technology_ , 50(1):1–9.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019\.  The secret sharer: Evaluating and testing unintended memorization in neural networks. In _USENIX Security Symposium_ , volume 267.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650.
  * Chang et al. (2021) Buru Chang, Inggeol Lee, Hyunjae Kim, and Jaewoo Kang. 2021.  " killing me" is not a spoiler: Spoiler detection model using graph neural networks with dependency relation-aware attention mechanism.  _arXiv preprint arXiv:2101.05972_.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  Quac: Question answering in context.  _arXiv preprint arXiv:1808.07036_.
  * Evans et al. (2021) Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021.  Truthful ai: Developing and governing ai that does not lie.  _arXiv preprint arXiv:2110.06674_.
  * Jagielski et al. (2022) Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. 2022.  Measuring forgetting of memorized training examples.  _arXiv preprint arXiv:2207.00099_.
  * Johnson (2022) Arianna Johnson. 2022.  Here’s what to know about openai’s chatgpt—what it’s disrupting and how to use it.  Technical report.
  * Kandpal et al. (2022a) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022a.  Large language models struggle to learn long-tail knowledge.  _arXiv preprint arXiv:2211.08411_.
  * Kandpal et al. (2022b) Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022b.  Deduplicating training data mitigates privacy risks in language models.  In _International Conference on Machine Learning_ , pages 10697–10707. PMLR.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Llansó et al. (2020) Emma Llansó, Joris Van Hoboken, Paddy Leerssen, and Jaron Harambam. 2020.  Artificial intelligence, content moderation, and freedom of expression.
  * Markov et al. (2022) Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2022.  A holistic approach to undesired content detection in the real world.  _arXiv preprint arXiv:2208.03274_.
  * Mozes and Kleinberg (2021) Maximilian Mozes and Bennett Kleinberg. 2021.  No intruder, no validity: Evaluation criteria for privacy-preserving text anonymization.  _arXiv preprint arXiv:2103.09263_.
  * OpenAI (2022) OpenAI. 2022.  Chatgpt: Optimizinglanguage modelsfor dialogue.  Technical report.  Accessed 08 Dec 2022 1725.
  * Peskov et al. (2020) Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Barrow, Christian Danescu-Niculesu-Mizil, and Jordan Boyd-Graber. 2020.  It takes two to lie: One to lie, and one to listen.  In _Proceedings of ACL_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _arXiv preprint arXiv:1806.03822_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023.  Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension.  _ACM Computing Surveys_ , 55(10):1–45.
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020.  Getting closer to ai complete question answering: A set of prerequisite real tasks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8722–8731.
  * Roy and Anand (2021) Rishiraj Saha Roy and Avishek Anand. 2021.  Question answering for the curated web: Tasks and methods in qa over knowledge bases and text collections.  _Synthesis Lectures onSynthesis Lectures on Information Concepts, Retrieval, and Services_ , 13(4):1–194.
  * samczsun (2022) samczsun. 2022.  bypassing chatgpt’s content filter.  twitter threat.  Accessed 8 Dec 2022.
  * Song and Shmatikov (2019) Congzheng Song and Vitaly Shmatikov. 2019.  Overlearning reveals sensitive attributes.  _arXiv preprint arXiv:1905.11742_.
  * Tabatabaei and Jamroga (2023) Masoud Tabatabaei and Wojciech Jamroga. 2023.  Playing to learn, or to keep secret: Alternating-time logic meets information theory.  _arXiv preprint arXiv:2303.00067_.
  * Tramèr et al. (2022) Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. 2022.  Truth serum: Poisoning machine learning models to reveal their secrets.  _arXiv preprint arXiv:2204.00032_.
  * Van Swol et al. (2012) Lyn M Van Swol, Deepak Malhotra, and Michael T Braun. 2012.  Deception and its detection: Effects of monetary incentives and personal relationship history.  _Communication Research_ , 39(2):217–238.
  * Wallace et al. (2019) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019\.  Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering.  _Transactions of the Association for Computational Linguistics_ , 7:387–401.
  * Wan et al. (2019) Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian McAuley. 2019.  Fine-grained spoiler detection from large-scale review corpora.  _arXiv preprint arXiv:1905.13416_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_.