# Language models are not naysayers:
An analysis of language models on negation benchmarks

Thinh Hung Truong1 Timothy Baldwin1,3 Karin Verspoor2,1 Trevor Cohn1,
1University of Melbourne 2RMIT University 3MBZUAI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Now at Google DeepMind.

###### Abstract

Negation has been shown to be a major bottleneck for masked language models,
such as BERT. However, whether this finding still holds for larger-sized auto-
regressive language models (“LLMs”) has not been studied comprehensively. With
the ever-increasing volume of research and applications of LLMs, we take a
step back to evaluate the ability of current-generation LLMs to handle
negation, a fundamental linguistic phenomenon that is central to language
understanding. We evaluate different LLMs — including the open-source GPT-neo,
GPT-3, and InstructGPT — against a wide range of negation benchmarks. Through
systematic experimentation with varying model sizes and prompts, we show that
LLMs have several limitations including insensitivity to the presence of
negation, an inability to capture the lexical semantics of negation, and a
failure to reason under negation.

##  1 Introduction

Despite being a core linguistic phenomenon, negation remains a major stumbling
block for modern NLP architectures (Kassner and Schütze, 2020; Hossain et al.,
2022). A reason for this could be that texts containing negation are
underrepresented in training data of language models, as humans tend to
express themselves using affirmative rather than negative expressions
(Ettinger, 2020). Regardless, negation has been shown to be challenging even
for humans to correctly interpret due to the diversity of forms across domains
(Truong et al., 2022a). For instance, in clinical documents, many acronyms are
used to denote negation such as NAD (no abnormality detected), and implicit
negation abounds, such as normal chest x-ray scan, which implies the absence
of an abnormality. Even more complex is the use of negation in combination
with other linguistic phenomena such as quantifiers, gradable adjectives (not
unattractive does not imply attractive) (Truong et al., 2022b); licensing
context (negative polarity items, e.g. any, either, yet, normally appear in
certain negative grammatical contexts Warstadt et al. (2019)); downward
entailment (A man owns a dog entails A man owns an animal but A man does not
own a dog does not entail A man does not own an animal) (Geiger et al., 2020).

Traditionally, negation has been treated as a standalone problem, e.g. as
negation detection (Chapman et al., 2001). The investigation of the impact of
negation in various downstream tasks (Hossain et al., 2022; Hossain and
Blanco, 2022a), or through probing (Ettinger, 2020) has revealed several
limitations of modern large language models (“LLMs”) in handling negation.
Given that LLMs are being adopted in an ever-growing range of tasks and have
been shown to display emergent abilities for high-level tasks that require
complex reasoning (Wei et al., 2022a), we are interested in exploring how the
handling of negation has progressed.

In this work, we investigate the performance of auto-regressive language
models on different negation-focused benchmarks. Instead of just looking at
samples containing negation in common NLP datasets, we consider datasets in
which negation plays an important role in making the correct judgement. In
particular, we classify the benchmarks into three categories corresponding to
the requisite negation reasoning abilities: (1) sensitivity to negation
through cloze completion (fill-in-the-blank) queries of factual statements;
(2) lexical semantics of negation through classification of antonym/synonym
relationships; and (3) ability to reason with negation through language
inference tasks.

Benchmark |  Task |  # Samples |  Example
---|---|---|---
MKR-NQ |  Completion |  3360 |  Query: Iburofen isn’t a kind of [MASK]. Wrong completions: NSAID, painkiller, drug, medicine.
MWR |  Completion |  27546 |  Query: Demand is an antonym of [MASK]. Wrong completions: necessitate, demands, request, requirement, imposition, need, demand.
SAR |  NLI |  2000 |  Word 1: Superiority / Word 2: Inferiority / Label: Antonym
NegNLI |  NLI |  4500 |  P: They watched me constantly for weeks. / H: They did not leave me on my own for weeks. / Label: Entailment
NaN-NLI |  NLI |  258 |  P: Not all people have had the opportunities you have had. / H: Some people have not had the opportunities you have had. / Label: Entailment
MoNLI |  NLI |  200 |  P: The man does not own a dog. / H: The man does not own a mammal. / Label: Not Entailment
Table 1: Summary of the negation-related benchmark datasets used in this
paper.

We conduct extensive experiments using prompt-based learning to facilitate
zero- and few-shot evaluation of LLMs, and find the following:

  * •

larger LMs are more insensitive to negation compared to smaller ones (Section
3);

  * •

LLMs lack lexical semantic knowledge about negation, yielding almost random
performance for synonym/antonym classification (Section 3);

  * •

LLMs have limited ability to reason under negation, performing worse than
random across most NLI datasets (Section 3). Only with the latest instruction
fine-tuned model (Ouyang et al., 2022; Chung et al., 2022) do we observe
above-chance performance (Section 3);

  * •

For each dataset, we also experiment with prompt variations and find that in
most cases, providing more information (context, instruction, simple wording)
leads to a degradation in performance.

##  2 Experimental settings

In this section, we outline the settings that , including benchmark datasets,
models to evaluate, and the prompts that were used. Our code is available at
https://github.com/joey234/llm-neg-bench.

Task |  Prompt name |  Example
---|---|---
MKR-NQ |  Default |  An expectorant isn’t a type of
|  Contrasting |  An expectorant is a type of medicine. An expectorant isn’t a type of
|  Discourse |  An expectorant is a type of medicine. Therefore, an expectorant isn’t a type of
|  Mask |  An [MASK] is a type of medicine. An [MASK] isn’t a type of
MWR |  Default |  Greed is an antonym of
|  Quote |  The word “greed” is an antonym of the word “
SAR |  Default |  Choose the correct answer: bad and good are antonyms or synonyms? Answer:
|  Simple |  Choose the correct answer: bad and good are opposite or similar? Answer:
|  Negation |  Antonyms are words with opposite meaning. Synonyms are words with similar meaning. Choose the correct answer: bad and good are antonyms or synonyms? Answer:
NLI |  Default |  Not all people have had the opportunities you have had.
|  |  Question: Some people have not had the opportunities you have had. True, False, or Neither?
|  |  Answer:
|  Negation |  The question requires reasoning about negation.
|  |  Not all people have had the opportunities you have had.
|  |  Question: Some people have not had the opportunities you have had. True, False, or Neither?
|  |  Answer:
Table 2: Prompts used for each task

###  2.1 Benchmarks

We use a range of benchmark datasets that exhibit the effects of negation
across a wide range of tasks, in the form of either cloze completion or
classification tasks. An overview of the datasets is presented in Table 1,
categorized according to purpose and the type of negation they contain.
Specifically, we focus on: (1) investigating whether LLMs are sensitive to the
presence of negation in factual statements; (2) testing whether LLMs capture
negation in lexical semantics relations (synonym/antonym relations); and (3)
investigating whether LLMs are able to reason under negation through multiple
natural language inference benchmarks. We discuss the datasets in greater
detail in Section 3.

###  2.2 Models

For the LLMs, we primarily focus on open-source auto-regressive LLMs with up
to 6.7B parameters, including GPT-Neo Black et al. (2021), and OPT (Zhang et
al., 2022), which are claimed to be comparable in performance to similar-sized
GPT-3 class models. Architecture-wise, they are both decoder-only PLMs pre-
trained with a causal LM objective, with the main difference being in their
pre-training corpora: GPT-neo was trained solely on the Pile dataset (Gao et
al., 2020) consisting of 22 sub-datasets spanning different sources, whereas
OPT was trained on the combination of datasets used in RoBERTa (Liu et al.,
2019), Pile, and PushShift Reddit (Baumgartner et al., 2020). We use the
official model checkpoints from HuggingFace
hub,111https://huggingface.co/models as detailed in Appendix A. We experiment
with smaller-sized variants of these two classes of models to observe the
effect of scaling on their performance over different benchmarks.

We also consider base GPT-3 (175B) (Brown et al., 2020), and its instruction
fine-tuned variant InstructGPT (Ouyang et al., 2022), as well as a strong
open-source instruction-tuned model FLAN-T5-XXL (11B) (Chung et al., 2022), to
examine how recent commercial LLMs perform on negation.

###  2.3 Prompts

We adopt prompt-based learning to enable zero- and few-shot evaluation of LLMs
(Radford et al., 2019). Given that LLMs have been found to be sensitive to
prompt variation (Wei et al., 2021), and that more natural-sounding prompts
correlate with model performance (Gonen et al., 2022), we also experiment with
different types of prompts (see Table 2).

We use GPT-3 style prompts (Brown et al., 2020) as the Default setting. As
handling negation plays an important role in all tasks, we also design prompts
to prime the LLMs to focus more on the negation context, by introducing
modifications specific to each task. In detail, for the cloze completion task
MKR-NQ, we investigate whether a given model can detect the difference between
two contrasting sentences (with/without negation). To achieve this, we prepend
the prompt with the corresponding sentence without negation (Contrasting
prompt). In addition, we also evaluate alternative prompts where we connect
the two sentences with a discourse marker (Discourse prompt), or mask the main
subject to encourage the model to attend more to negation cues (Mask prompt).

For antonym/synonym-related tasks (MWR, SAR), we also experiment with
simplifying the prompt and use descriptive terms rather than the formal names
of the relations (e.g. antonyms, synonyms $\rightarrow$ opposite of, similar
to), based on the intuition that these terms will appear more frequently in
the pre-training data.

Finally, for classification tasks, we propose negation-aware prompting
(Negation prompt) by modifying the prompts to explicitly state that the task
involves reasoning about negation. Note that we explicitly include class
options in the prompts to help reduce the effect of the surface form
competition causing LLMs to assign lower probabilities to the correct answers
(Holtzman et al., 2021).

For datasets with an accompanying training set (SAR, MoNLI), we also
experiment with few-shot evaluation formulated as in-context learning by
prepending the input prompts with 10 random samples from the training set.

###  2.4 Metrics

To evaluate cloze completion tasks, we employ Weighted Hit Rate (WHR) (Jang et
al., 2022b), which measures the number of matches between the top-k predicted
words and a given set of target wrong predictions, taking into account the
prediction probabilities:

$\textit{WHR}_{k}{(x,W)}=\frac{\sum^{k}_{i=1}{c_{i}\times\mathbb{1}(w_{i}\in
W_{x})}}{\sum^{k}_{i=1}c_{i}}$ (1)

where $W_{x}$ is the wrong prediction set of the input query $x$, and $w_{i}$
is the top $i$-th prediction with confidence score $c_{i}$, obtained by taking
the softmax of log probabilities $p(w_{i}|x)$ from the LM. Note that the model
performance is better if there are fewer matches between models’ predictions
and wrong completions, WHR is an error metric (lower is better). One problem
with the WHR metric is that we can only evaluate using a fixed set of wrong
predictions. Regardless, we believe the relative performance numbers are
indicative of model performance.

For classification tasks, we evaluate using Accuracy, noting that all datasets
are reasonably balanced.

##  3 Main findings

We summarize the main findings in this section. In general, the performance of
GPT-neo and OPT follows a similar trend across all benchmarks (we present GPT-
neo results; results of OPT models are in Appendix B).

### Finding 1: Larger LMs are more insensitive to negation

#### MKR-NQ (Jang et al., 2022b)

Masked Knowledge Retrieval – Negated Query (MKR-NQ) is a negated version of
the LAMA dataset (Petroni et al., 2019), which contains lexicalized statements
of triples in ConceptNet (Speer et al., 2017). This dataset contains factual
statements with verbal negations (i.e. negators not, don’t are associated with
the main verb of the sentence), e.g. Iburofen is a type of medicine.
$\rightarrow$ Iburofen isn’t a type of medicine.

Each sample contains the query along with a set of wrong word completions,
supporting the evaluation of the sensitivity of the model to negation by
measuring how likely a model will generate incorrect completions. Note that
MKR-NQ only considers sample sentences that contain a single verb, making it
trivial to negate the original sentences.

Figure 1: Zero-shot performance of GPT-neo on MKR-NQ using different prompts
under the Weighted Hit Rate (WHR) metrics (lower scores are better). Note the
different scale for the left-most plot.

#### Findings

From Figure 1, which is based on LLMs with a negated factual statement
(Default prompt), we observe relatively low hit rates ($<$ 0.15) across all
model sizes, and a clear inverse scaling trend between model sizes and their
performance. The smallest variant (GPT-neo-125M) has the best performance,
which is comparable to that of masked language model of a similar size (BERT-
base, 110M parameters) (Jang et al., 2022b). This phenomenon reflects the
finding that larger models tend to memorize the training data more (McKenzie
et al., 2022; Jang et al., 2022a). Moreover, higher hit rates for top-1
predictions suggest that models predict wrongly with high confidence.

For Contrasting prompts, in which we prepend the negated statement with its
non-negated version, we notice a drastic increase in WHR, showing that models
are prone to repeating what is presented in the prior context, confirming the
finding of Kassner and Schütze (2020). When a discourse term is added to
connect the two sentences (Discourse prompt), we do not observe any
improvement, and the performance of the largest model is even worse. To
investigate whether this phenomenon is attributable to models not being able
to detect the presence/absence of negation, we experiment with masking out the
main noun/verb of the queries (Mask prompt). We observed even higher WHR,
especially for the top-1 prediction in this setting. The results suggest that
repetitions are caused more by LLMs being easily primed by repeating what is
present in the previous context, than by generating words that are closely
associated with the main subject of interest. This again shows that the models
cannot differentiate between identical contexts, differing only on whether
negation is present or absent (i.e., outputs tend to be similar with or
without negation).

To further analyze the outputs, we calculate the perplexity (PPL) of the
generated predictions to determine their plausibility (Wilcox et al., 2020).
Here, we choose the model with the best $\textit{WHR}_{5}$ score on the MKR-NQ
benchmark, and calculate the mean perplexity over all queries for each prompt
type (5 completions for each query). PPL is calculated as the exponentiated
average negative log-likelihood of a sequence, with exponent base $e$. As a
point of reference, we calculated the average perplexity of the provided
completion of the original non-negated dataset (denoted Corpus). From the
reported perplexities (Table 3), we can see that Default output are the most
plausible (with PPL markedly lower than Corpus), while Contrasting is the
least natural. The remaining prompts types (Discourse, Mask) are comparable to
Corpus. These results show that LLMs can indeed generate plausible and human-
like output for this task.

Setting |  Example |  Mean PPL$\downarrow$
---|---|---
Corpus |  [Baseball is a type of sport.] |  434.42
Default |  [Baseball isn’t a type of sport.] |  288.94
Contrasting |  Baseball is a type of sport. [Baseball isn’t a type of sport.] |  533.56
Discourse |  Baseball is a type of sport. Therefore, [baseball isn’t a type of sport.] |  477.44
Mask |  MASK is a type of sport. [MASK isn’t a type of sport.] |  448.23
Table 3: Mean perplexity (PPL) calculated using the GPT-J-6B model. Only the
strings enclosed in square brackets are considered during calculation in order
to provide a fair comparison with similar token length. For Corpus, PPL is
calculated using the provided gold completion.

### Finding 2: LMs fail to capture synonym/antonym lexical relations

#### MWR (Jang et al., 2022b)

To test the ability of LMs to capture negative lexical semantics, we use MWR
dataset, where models are asked to predict the antonym/synonym of a target
word. The dataset was constructed by using the most frequent nouns,
adjectives, and adverbs that appear in SNLI (Bowman et al., 2015), then
choosing their corresponding synonyms and antonyms from ConceptNet (Speer et
al., 2017). The dataset also contains different wordings for antonym-asking
and synonym-asking queries (e.g. is the opposite of, is different from and is
similar to, is a rephrasing of) to test model sensitivity to prompt
variations.

Figure 2: Zero-shot performance of GPT-neo on MWR using different prompts (WHR
metrics; lower is better)

#### Findings

From Figure 2, we can observe the same inverse scaling trend as for MKR-NQ
using the Default prompt, where the hit rate of the smallest model is around
0.02, better than previously-reported SOTA results (Jang et al., 2022b). With
a more natural query with more focus on the target words via quotation marks
(Quote prompt), surprisingly, we noticed a drastic jump in hit rates. However,
MWR may not be a good indicator of model performance due to how the task is
framed. One problem is that models can generate words that are not in the
given wrong prediction set, but are also irrelevant, and are also neither
antonyms nor synonyms of the given target words, as demonstrated in Table 4.

Query |  Wrong completions |  Top-5 predictions
---|---|---
Greed is an antonym of |  greed, avarice, desire, greeds, gluttony |  altruism, self-sacrifice, self-denial, self-abnegation, gods
Finale is an antonym of |  conclusion, finish, finales, finale |  last, epiphany, finality, anti-climax, anti-climactic
Table 4: Example output of GPT-J-6B on MWR. bolded words are related to target
words, but are neither synonyms nor antonyms. underlined are wrong antonyms
but are not in the given set of wrong completions. Figure 3: Zero-shot
performance of GPT-neo on SAR dataset using different prompts (accuracy
metric; higher is better)

#### SAR (Jang et al., 2022b)

To further investigate the ability of LLMs to capture negative lexical
semantics, we consider the antonym/synonym relation classification task (SAR).
Different from the MWR cloze-style synonym/antonym prediction task, this
benchmark is framed as a binary classification task of predicting the correct
antonym or synonym relationship between two given words. Data is once again
taken from ConceptNet, where triplets with synonym and antonym relations are
extracted in equal numbers (1000 samples for each relation).

#### Findings

In contrast to the high results for MWR, we find that for this task, model
performance is equivalent to random, with accuracy fluctuating around 0.5
(Figure 3). For prompt variants, we do not observe any meaningful improvement,
in that Simple follows a similar trend to Default and Negation performs better
for larger models (2.7B and 6B). This is a huge degradation from previous
fully fine-tuned results over encoder models. For instance, Jang et al.
(2022b) reported that BERTlarge achieves 92.5% accuracy on SAR. We argue that
this is a specific task that is not captured in the next token prediction
training objective of LLMs and thus, requires explicit supervision.

Figure 4: Zero-shot performance of GPT-neo on NLI datsets using different prompts (higher is better). The dashed line denotes a random baseline. Note that RTE-neg and MoNLI are 2-way classification tasks while the rest are 3-way. Benchmark |  |  GPT-J-6B |  GPT-3 |  InstructGPT |  InstructGPT w/ Neg. prompt |  FLAN-T5-XXL w/ Neg. prompt
---|---|---|---|---|---|---
MKR-NQ |  WHR5 $\downarrow$ |  0.083 |  0.172 |  0.195 |  NA |  NA
MWR |  |  0.125 |  0.488 |  0.504 |  NA |  NA
SAR |  Accuracy $\uparrow$ |  0.490 |  0.501 |  0.687 |  0.780 |  0.507
SNLI-neg |  |  0.316 |  0.267 |  0.640 |  0.673 |  0.477
MNLI-neg |  |  0.275 |  0.359 |  0.548 |  0.625 |  0.354
RTE-neg |  |  0.211 |  0.525 |  0.767 |  0.807 |  0.770
NaN-NLI |  |  0.298 |  0.469 |  0.647 |  0.682 |  0.376
MoNLI |  |  0.500 |  0.540 |  0.470 |  0.400 |  0.500
Table 5: Zero-shot results on the different benchmarks. “NA” denotes that
Negation prompts are not applicable to MKR-NQ and MWR. The best results are
bolded for each task (row).

### Finding 3: LLMs are unable to reason under negation

#### NegNLI (Hossain et al., 2020)

NegNLI contains 4500 premise–hypothesis pairs with important negation, where
negation is essential in making the correct judgement about the relationship
between the premise–hypothesis pairs. Samples are extracted from the commonly-
used NLI datasets (RTE Dagan et al. (2005), SNLI Bowman et al. (2015), MNLI
Williams et al. (2018)), then the negator not is added to the main verb either
in the premise, hypothesis, or both. Here, we consider each subset separately,
as the number of classes are not the same, and denote them SNLI-neg, MNLI-neg,
RTE-neg.

#### MoNLI (Geiger et al., 2020)

MoNLI is an NLI dataset focused on lexical entailment and negation.
Specifically, the dataset investigates the downward monotonicity property
where negation reverses entailment relations (e.g. dance entails move, but not
move entails not dance). MoNLI was created by extending samples from SNLI by
substituting the nouns by their hypernyms/hyponyms from WordNet (Miller,
1998).

#### NaN-NLI (Truong et al., 2022b)

NaN-NLI is a test suite which focuses on sub-clausal negation, in which only
part of the sentence’s meaning is negated, thus making it harder to correctly
determine the correct negation scope (e.g. in Not the first time that they
pulled that off the negation scope is only Not the first time and the main
clause of the sentence they pulled that off is not negated). Each
premise–hypothesis pair is constructed so that the corresponding hypotheses
are constructed to reflect different interpretations that the negated instance
in the premise are likely to be misunderstood for.

#### Findings

Similar to the antonym/synonym classification task, the performance for most
negation-focused NLI benchmarks is low. In particular, for all NLI datasets,
the performance is generally lower than baseline. As shown in Figure 4,
scaling up model size has almost no effect, and the largest model performs
worse in many cases, even when the prompt explicitly states that the task
requires reasoning about negation (Negation prompt). For datasets which
include a training set (SAR, MoNLI), we also experimented with few-shot
learning but did not observe any noticable improvement (Figure 5). One
exception is that the 2.7B model seems to pick up some signal from the
provided MoNLI training samples, but falls back again when we increase the
model size to 6B.

Even with general NLI datasets, zero-shot applications of LLMs were previously
shown to be roughly equivalent to a random baseline (Wei et al., 2021). When
negation is involved, the task becomes even more complex. As pointed out in
Brown et al. (2020), one possible reason that LLMs struggle with NLI is that
the samples consist of two disjoint sentences, which are unlikely to appear
naturally in standard training corpora. We hypothesise that NLI is a generally
hard task that requires substantially more supervision in order for models to
detect meaningful patterns.

Figure 5: 10-shot performance of GPT-neo on SAR and MoNLI using Default prompt
(higher is better)

### Finding 4: Instruction fine-tuning improves reasoning under negation

We further evaluate with GPT-3 class models of significantly larger scale
(175B), which have been shown to achieve strong results in zero- and few-shot
settings across a wide range of tasks (Brown et al., 2020). In detail, we
benchmark the largest GPT-3 model (text-davinci-001: Brown et al. (2020)) and
its variant InstructGPT, which is trained to follow human instructions using
reinforcement learning (text-davinci-003: Ouyang et al. (2022)). The results
can be found in Table 5.

For the base GPT-3 model, the results over most benchmarks are no better than
much smaller language models (GPT-neo-125M). For cloze-completion tasks,
consistent with the earlier-observed trend of larger models performing worse,
we observe higher (worse) WHR scores compared to that of smaller language
models, confirming our finding that larger models are more insensitive to the
presence of negation. Results get even worse with using the instruction fine-
tuned model.

On the other hand, for most classification tasks, InstructGPT achieves better
zero-shot results than other models. In addition, using this model in
combination with explicit instruction about negation (Negation prompt) further
improves performance, which we did not observe for other LLMs. It is, however,
unclear what data the instruction-tuning process was performed on. Thus, the
huge gain in performance could be attributed to the existence of similar
patterns in the training set (i.e. explicit supervision over similar tasks).
Interestingly, InstructGPT performance on MoNLI did not increase (it
underperfomed other models). We hypothesize that this is due to an inductive
bias from model’s ability to reason with hypernymy. For instance, the model
can understand that “dog is an animal” (and therefore own an animal entails
own a dog), but incorrectly generalizes this logic to a similar sample
containing negation (not own a dog entails not own an animal). This is indeed
true when we look at the explanation generated by ChatGPT, the subsequent
model to InstructGPT (Figure 6).

Figure 6: A ChatGPT-generated output of a failed negative monotonicity
reasoning sample. The output was generated using ChatGPT Feb 13 Version

We also experiment with the instruction-tuned FLAN-T5-XXL model (Chung et al.,
2022) and find that the results are better than GPT-3 for most NLI tasks,
despite being $\sim$16x smaller. These results suggest that instruction fine-
tuning has much greater impact than model scaling in terms of models
developing the ability to perform reasoning tasks under negation.

##  4 Related work

Our work builds upon previous research on negation. In particular, we were
inspired by the pioneering works of Kassner and Schütze (2020) and Ettinger
(2020), which reveal that pre-trained language models have a major issue in
being insensitive to the presence of negation, based on evaluation over a set
of cloze-style queries. Following this line of research, Jang et al. (2022b)
also explored negation in a cloze completion context by negating factual
statements extracted from ConceptNet and come to a similar finding.

In a broader context, Hossain et al. (2020, 2022) investigated the performance
of BERT-based methods on samples containing negation in the GLUE (Wang et al.,
2018) and SuperGLUE (Wang et al., 2019) datasets. Their main finding is that
the results for the subsets containing only negation are lower than those
without, as well as the whole test set, showing that models struggle with
negation, even when fine-tuned on relevant training data. Ravichander et al.
(2022) proposed the challenging CONDAQA dataset to test the ability of models
to reason about the implications of negation. The authors conducted
comprehensive analysis of different types of LLMs under different settings,
and found that the best-performing models were still well below human
performance. Negation has also been investigated as part of psycholinguistic
probing datasets (Lialin et al., 2022; Jumelet et al., 2021; Staliūnaitė and
Iacobacci, 2020). Contrasting previous finding, Gubelmann and Handschuh (2022)
found that the ability to understand negation of LMs is underestimated in
previous studied. Through designing a controlled dataset with minimal pairs
varying in syntactic structure, gender, profession, and first name, they
concluded that the models are indeed sensitive to negation and thus, their
struggle comes more from the contextualization of the tasks.

As part of the analysis on emergent abilities of LMs, negation has been shown
to be one of the tasks that displays a flat scaling curve (Wei et al., 2022a)
or even inverse-scaling (McKenzie et al., 2022). This behaviour was later
shown to be alleviated by instruction fine-tuning (Wei et al., 2022b). The
effectiveness of instruction fine-tuning is further supported in Jang and
Lukasiewicz (2023). The authors investigated the logical consistency of
ChatGPT and found that ChatGPT understands negation and antonyms much better
than previous models.

Beside probing and evaluation, there have also been works on making language
models more robust to negation, including unlikelihood training (Hosseini et
al., 2021), adaptive pre-training on relevant data (Truong et al., 2022a),
leveraging affirmative interpretations from negation Hossain and Blanco
(2022b), and learning better representation of negation through contrastive
learning (Jiang et al., 2022; Wang et al., 2022).

##  5 Conclusion

We have shown that LLMs still struggle with different negation benchmarks
through zero- and few-shot evaluations, implying that negation is not properly
captured through the current pre-training objectives. With the promising
results from instruction-tuning, we can see that rather than just scaling up
model size, new training paradigms are essential to achieve better linguistic
competency. Through this investigation, we also encourage the research
community to focus more on investigating other fundamental language phenomena,
such as quantification, hedging, lexical relations, and downward entailment.

##  6 Limitations

First, regarding the experimental settings, the WHR metrics used to evaluate
cloze completion tasks are imperfect, as we discussed. Framing cloze
completion tasks in the style of multiple-choice question answering to limit
the options that models are evaluated on would be a good direction to follow
(Robinson et al., 2022). In addition, the prompt engineering in this work is
in no way exhaustive, and could be extended using different prompt engineering
strategies such as soft prompt tuning (Lester et al., 2021), or mining- and
paraphrasing-based methods to generate high quality prompts (Jiang et al.,
2020).

Second, due to computational constraints, we could not perform an extensive
set of experiments for larger models like PaLM (with up to 540B parameters)
(Chowdhery et al., 2022). Recent work by Wei et al. (2022b) has shown that the
inverse scaling trend on several benchmarks can be alleviated using the large
instruction fine-tuned models such as FLAN-PaLM-540B, which is largely in line
with our findings regarding InstructGPT and FLAN-T5. With a small-scale
experiment, we found that ChatGPT displayed strong performance on challenging
samples in the investigated benchmark, so the main findings of the paper may
not hold true for newer LLMs.

Finally, this work only considers negation in the English language. There is
every reason to believe that negation is an equally challenging problem in
other languages. As this is a linguistically-intensive task, and requires
native speakers to conduct thorough analysis of the results, we leave this for
future work.

## Acknowledgement

The authors would like to thank the anonymous reviewers for their detailed,
kind, and constructive reviews. This research was conducted by the Australian
Research Council Training Centre in Cognitive Computing for Medical
Technologies (project number ICI70200030) and funded by the Australian
Government. This research was undertaken using the LIEF HPC-GPGPU Facility
hosted at the University of Melbourne. This Facility was established with the
assistance of LIEF Grant LE170100200.

## References

  * Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020.  The PushShift Reddit dataset.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 830–839.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chapman et al. (2001) Wendy W Chapman, Will Bridewell, Paul Hanbury, Gregory F Cooper, and Bruce G Buchanan. 2001.  A simple algorithm for identifying negated findings and diseases in discharge summaries.  _Journal of Biomedical Informatics_ , 34(5):301–310.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The Pascal Recognising Textual Entailment challenge.  In _Machine learning challenges workshop_ , pages 177–190. Springer.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020.  Neural natural language inference models partially embed theories of lexical entailment and negation.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 163–173, Online. Association for Computational Linguistics.
  * Gonen et al. (2022) Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2022\.  Demystifying prompts in language models via perplexity estimation.  _arXiv preprint arXiv:2212.04037_.
  * Gubelmann and Handschuh (2022) Reto Gubelmann and Siegfried Handschuh. 2022.  Context matters: A pragmatic study of PLMs’ negation understanding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4602–4621, Dublin, Ireland. Association for Computational Linguistics.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021\.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hossain and Blanco (2022a) Md Mosharaf Hossain and Eduardo Blanco. 2022a.  Leveraging affirmative interpretations from negation improves natural language understanding.  _arXiv preprint arXiv:2210.14486_.
  * Hossain and Blanco (2022b) Md Mosharaf Hossain and Eduardo Blanco. 2022b.  Leveraging affirmative interpretations from negation improves natural language understanding.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5833–5847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hossain et al. (2022) Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022.  An analysis of negation in natural language understanding corpora.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021.  Understanding by understanding not: Modeling negation in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1301–1312, Online. Association for Computational Linguistics.
  * Jang et al. (2022a) Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2022a.  Can large language models truly understand prompts? a case study with negated prompts.  _arXiv preprint arXiv:2209.12711_.
  * Jang and Lukasiewicz (2023) Myeongjun Jang and Thomas Lukasiewicz. 2023.  Consistency analysis of chatgpt.  _arXiv preprint arXiv:2303.06273_.
  * Jang et al. (2022b) Myeongjun Jang, Frank Mtumbuka, and Thomas Lukasiewicz. 2022b.  Beyond distributional hypothesis: Let language models learn meaning-text correspondence.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2030–2042, Seattle, United States. Association for Computational Linguistics.
  * Jiang et al. (2022) Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. 2022.  PromptBERT: Improving BERT sentence embeddings with prompts.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8826–8837, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know? _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Jumelet et al. (2021) Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021.  Language models use monotonicity to assess NPI licensing.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4958–4969, Online. Association for Computational Linguistics.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lialin et al. (2022) Vladislav Lialin, Kevin Zhao, Namrata Shivagunde, and Anna Rumshisky. 2022.  Life after BERT: What do other muppets understand about language? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3180–3193, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McKenzie et al. (2022) Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez. 2022.  Inverse scaling prize: Round 1 winners.
  * Miller (1998) George A Miller. 1998.  _WordNet: An electronic lexical database_.  MIT press.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_.
  * Ravichander et al. (2022) Abhilasha Ravichander, Matt Gardner, and Ana Marasović. 2022.  CONDAQA: A contrastive reading comprehension dataset for reasoning about negation.  _arXiv preprint arXiv:2211.00295_.
  * Robinson et al. (2022) Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022.  Leveraging large language models for multiple choice question answering.  _arXiv preprint arXiv:2210.12353_.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  ConceptNet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Staliūnaitė and Iacobacci (2020) Ieva Staliūnaitė and Ignacio Iacobacci. 2020.  Compositional and lexical semantics in RoBERTa, BERT and DistilBERT: A case study on CoQA.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7046–7056, Online. Association for Computational Linguistics.
  * Truong et al. (2022a) Thinh Truong, Timothy Baldwin, Trevor Cohn, and Karin Verspoor. 2022a.  Improving negation detection with negation-focused pre-training.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4188–4193, Seattle, United States. Association for Computational Linguistics.
  * Truong et al. (2022b) Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, and Karin Verspoor. 2022b.  Not another negation benchmark: The NaN-NLI test suite for sub-clausal negation.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 883–894, Online only. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _CoRR_ , abs/1905.00537.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2022) Hao Wang, Yangguang Li, Zhen Huang, Yong Dou, Lingpeng Kong, and Jing Shao. 2022\.  Sncse: Contrastive learning for unsupervised sentence embedding with soft negative samples.
  * Warstadt et al. (2019) Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019.  Investigating BERT’s knowledge of language: Five analysis methods with NPIs.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2877–2887, Hong Kong, China. Association for Computational Linguistics.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.
  * Wei et al. (2022b) Jason Wei, Yi Tay, and Quoc V Le. 2022b.  Inverse scaling can become U-shaped.  _arXiv preprint arXiv:2211.02011_.
  * Wilcox et al. (2020) Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020\.  On the predictive power of neural language models for human real-time comprehension behavior.  In _Proceedings of the 42nd Annual Meeting of the Cognitive Science Society_ , page 1707–1713.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  OPT: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.

##  Appendix A Model checkpoints

For open-sourced LMs, we consider the official released checkpoints on the
HuggingFace hub at:

  * •

https://huggingface.co/EleutherAI/x

  * •

https://huggingface.co/facebook/y

where x in {gpt-neo-125M,gpt-neo-1.3B,gpt-neo-2.7B,gpt-j-6B}, and y in
{opt-125m,opt-350m,opt-1.3b,opt-2.7b,opt-6.7b}.

For GPT-3 models, we access them through the official API at
https://openai.com/api/, using the Text completion endpoint. The considered
model identifiers along with their sizes are:

  * •

text-ada-001: 350M

  * •

text-babbage-001: 1.3B

  * •

text-curie-001: 6.7B

  * •

text-davinci-001: 175B

  * •

text-davinci-003: 175B

Figure 7: Zero-shot performance of OPT on MWR using different prompts Figure
8: Zero-shot performance of OPT on SAR using different prompts

##  Appendix B OPT results

Figure 9: Zero-shot performance of OPT on MKR using different prompts

For MWR, although we observe improvements with increasing model sizes, the WHR
scores are much higher than those of GPT-neo, showing that OPT is worse at
predicting antonyms and synonyms of words. The gap in performance may lie in
differences in training data between the two types of models.

Figure 10: Zero-shot performance of OPT on NLI tasks using different prompts

##  Appendix C Model outputs

Prompt | Model | Output
---|---|---
Paracetamol isn’t a kind of | GPT-neo-125M | muscle
| GPT-J-6B | painkiller
| OPT-125M | pain
| OPT-6.7B | medicine
| GPT-3 | medication
| InstructGPT | NSAID
Entrance is an antonym of | GPT-neo-125M | interest
| GPT-J-6B | entrance
| OPT-125M | entrance
| OPT-6.7B | exit
| GPT-3 | departure
| InstructGPT | entrance
Choose the correct answer: flimsy and sturdy are synonyms or antonyms? | GPT-neo-125M | Synonyms
GPT-J-6B | Synonyms
OPT-125M | Antonyms
OPT-6.7B | Synonyms
GPT-3 | Antonyms
InstructGPT | Antonyms
I can not think of a few reasons for the allergy to substance. Question: There are not reasons why there’s an allergy. True, False, or Neither? Answer: | GPT-neo-125M | True
GPT-J-6B | True
OPT-125M | True
OPT-6.7B | Neither
GPT-3 | False
InstructGPT | Neither
The man does not own a dog. Question: the man does not own a mammal. True or Not true? Answer: | GPT-neo-125M | True
GPT-J-6B | True
OPT-125M | True
OPT-6.7B | True
GPT-3 | True
InstructGPT | Not True
Table 6: Example outputs of models. Wrong answers are highlighted


(d) Uncertainty-Health

(e) Uncertainty-Politics

(f) Uncertainty-Policy

(g) Negative Sentiment-Economics

(h) Negative Sentiment-Health

(i) Negative Sentiment-Politics

(j) Negative Sentiment-Policy

Figure 9: Coefficient Estimates and Confidence Intervals (% 95) of interaction
between the treatment variable and time dummies. The dependent variable is
Share of tweets classified as _Uncertainty_ and _Negative Sentiment_
aggregated and grouped by Topic. The baseline period starts from the 1rst of
February and ends on the 19th of February.

###  F.2 Sensitivity analysis with bounds on pre-trends

Next, we assess the robustness of the estimates of $\delta_{0}$ in equation 10
– i.e. the effect of the first lockdown on those living in the _red zone_ , as
we allow for some violation of the parallel trends conditions following [77].
With this approach, the size of post-treatment violation of parallel trends is
bounded, as we assume that it cannot be larger than $\bar{M}$ times the
largest pre-treatment violation [79].

We concentrate on the effect of the lockdown on those first treated by the
measures, therefore we drop all observations from after the extension of the
measure at the national level to avoid dealing within a staggered setting - as
all units become treated on March 9 and so the control group is left empty. We
perform sensitiviy analysis on $\delta_{0}$ using the HonestDiD
package424242https://github.com/asheshrambachan/HonestDiD, that is, we allow
for different values of $\bar{M}$ and report the confidence intervals of
$\delta_{0}$ for changing the value of $\bar{M}$. As the effect becomes
insignificant for some values of $\bar{M}$, then we obtain a significant
result that is robust to violation of parallel trends that can as large as
$\bar{M}$ times the pre-treatment violation in trends.

The results of all outcomes of interest are reported in Figure 12 and Figure
13. The test returns 95% _robust_ confidence intervals for different values of
$\bar{M}$434343We refer to the original paper by [77] for a discussion of the
method..

By allowing up to 1.5 times the maximal violation in pre-treatment trends we
still retain a robust significant effect for uncertainty and negatives
sentiments towards health and the policy, and up to 1 time for negative
sentiments towards politics.

On the other hand, the effect on uncertainty related to politics is not robust
to any violations of parallel trends, as depicted in Figure 10(d). Therefore,
the effect is only significant as we assume zero violations of parallel
trends, in line with the usual parallel trend assumption of the DiD model.

(a) Uncertainty

(b) Uncertainty-Economics

(c) Uncertainty-Health

(d) Uncertainty-Politics

(e) Uncertainty-Policy

Figure 10: Sensitivity analysis using relative magnitudes restrictions of
$\delta_{0}$ of equation 10 for _Uncertainty_ aggregated and grouped by Topic.
The baseline period starts from the 1rst of February and ends on the 19th of
February.

(a) Negative Sentiment

(b) Negative Sentiment-Economics

(c) Negative Sentiment-Health

(d) Negative Sentiment-Politics

(e) Negative Sentiment-Policy

Figure 11: Sensitivity analysis using relative magnitudes restrictions of
$\delta_{0}$ of equation 10 for _Negative Sentiment_ aggregated and grouped by
Topic. The baseline period starts from the 1rst of February and ends on the
19th of February.

###  F.3 SUTVA

SUTVA is another implicit assumption of the DiD model. A violation of the
SUTVA means that we have multiple treatment effects, depending on who is
treated [doi:10.1080/01621459.1986.10478355]. Given a randomly drawn sample
and Assumptions 1 and 2, the DID estimator for the ATT becomes biased and can
be decomposed as:

$ATT=\delta^{tot}_{1}+\delta^{spill}_{1}(0)=\delta_{1}+\delta^{spill}_{1}(1)-\delta^{spill}_{1}(0)$
(11)

where $\delta^{spill}_{1}(1)$ is the average spillover effect on treated units
and $\delta^{spill}_{1}(0)$ the average spillover non-treated units in period
1.

In our model, there could be a violation of the SUTVA as one could potentially
argue that the outcome of the control group is positively correlated with the
reaction of those under lockdown through social and virtual connections. The
more the people under lockdown express distress, the more should also those in
the neighboring cities. Similarly, messages describing a cool and collected
situation inside the red zone should be reassuring to those outside of the
lockdown area.

One can try to disentangle the spillovers from the treatment effect by making
assumptions on the exposure effects. In our setting, it seems reasonale to
assume that spillovers will affect nearby units of the treatment area, while
exposure to externalities of the treatment could be ignored for units located
above a certain distance from the lockdown area. Under the assumption that
spillovers are local, SUTVA will be violated only for units within a certain
distance from the treated units [23].

Following [23], we create dummy identifiers for _distance from treated units_
, sort of _rings_ surrounding the treatment units with increasing distance,
and interact them with the treatment indicator.

Our data features units coming from a small geographical area surrounding the
red zone, encompassing all cities red zone cities and orange zone cities in
the surrounding area. Considering only the first post-treatment period, i.e.
for which we are able to identify a not-yet-treated set of units that can be
used as _counterfactuals_ , define $Distance_{i}(d)$ set of binary identifiers
indicating whether unit i is within $d\in Dist:=\\{(0,20],(20,25],(25,30],\\}$
kilometers from the treated area.

To identify the total treatment effect, we modify our model specification as
follows:

$Y_{i,t}=\alpha+\gamma
D_{i}+\sum_{t\in\\{1,2\\}}\lambda_{t}\text{1}[T_{i}=t]+\sum_{t\in\\{1,2\\}}\delta_{t}(\text{1}[T_{i}=t]\times
D_{i})+\sum_{d\in Dist}\eta_{d}Distance_{i}(d)(1-\text{1}[t=1]\times
D_{i})+\epsilon_{i,t}$ (12)

The estimates on coefficients $\eta_{d}$ will return the average spillover
effect on the control units within distance d. Here we assume that units
located at more than 30 km from the red zone are not affected by
spillovers444444Because the maximum radius allowed by the geographical
filtering option of the Twitter API is given by 42km from the center
coordinates – in our case Codogno, we use as counterfactuals units within 30
and 42 km from Codogno..

This allows us to capture the bias on the treatment effect estimate coming
from average spillover effects. Because we cannot reasonably argue that some
treated units are unaffected by spillover effect, we are only able to identify
the _total_ average effect of the treatment $\delta^{tot}_{1}$, as estimating
the _direct_ effect $\delta_{1}$ requires zero exposure for some treated
units. The ring’s method does not require parametrization of the exposure
function.

In Table 6 we report the estimated coefficients of equation 12. The results
are in line with the baseline estimates of the effect of the local lockdown of
February 23, 2020. We still retrieve a significant impact of
$\delta^{tot}_{1}$ ($redzone=1\times post=1$) on uncertainty related to
economics, health and politics, as well as negative sentiments towards health
and politics.

Table 6: Estimates of spillover effect and total treat effect for
_Uncertainty_ and _Negative Sentiment_ , with clustered standard errors.

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)
Aggregate Economics Health Politics Policy Aggregate Economics Health Politics
Policy red zone=1 0.0505 0.00866 -0.00605 -0.0147∗ -0.00345 0.138∗ 0.0224∗
-0.0137 -0.0452∗∗∗ -0.00873 (0.81) (1.43) (-0.50) (-2.54) (-0.64) (2.52)
(2.38) (-1.31) (-4.58) (-1.48) post=1 0.0538∗∗∗ 0.00855∗∗∗ 0.0637∗∗∗ -0.00181
0.0207∗∗∗ -0.0527 -0.00692∗∗∗ 0.0322∗∗∗ -0.0137 0.00562∗∗ (4.56) (5.09) (7.49)
(-0.43) (11.36) (-1.34) (-5.11) (4.90) (-1.54) (2.93) post=2 0.0659∗∗∗
0.0153∗∗∗ 0.0718∗∗∗ -0.00206 0.0200∗∗∗ -0.0694 -0.000510 0.0382∗∗∗ -0.0220∗∗
0.00752∗ (5.10) (5.81) (9.27) (-0.54) (7.01) (-1.93) (-0.17) (6.35) (-2.76)
(2.50) red zone=1 $\times$ post=1 0.0274 -0.0136∗∗∗ 0.0349∗∗ 0.0159∗ 0.00316
-0.0178 -0.0184 0.0545∗∗∗ 0.0526∗∗∗ 0.0128 (0.30) (-4.16) (2.73) (2.58) (0.24)
(-0.44) (-1.61) (3.61) (4.90) (0.93) red zone=1 $\times$ post=2 -0.0198
-0.0118 -0.0108 0.00675 -0.00856∗ 0.0259 -0.00614 0.0452∗ 0.0736∗∗∗ 0.00839
(-0.40) (-1.26) (-0.55) (1.65) (-2.36) (0.49) (-0.33) (2.11) (5.19) (1.97)
$\eta_{(0,20]}$ -0.0217 -0.00325 -0.00929 0.00369 -0.00220 -0.0336 -0.00275
-0.0103 0.00330 -0.00751 (-0.75) (-0.63) (-1.32) (0.84) (-0.65) (-0.55)
(-0.69) (-1.02) (0.37) (-1.64) $\eta_{(20,25]}$ -0.0102 -0.0115∗ -0.0340∗∗∗
-0.0112∗∗ -0.0129∗∗∗ 0.0444 -0.00682 -0.0182∗∗ -0.0145∗ -0.0122∗∗ (-0.90)
(-2.21) (-4.39) (-2.92) (-3.83) (1.79) (-1.78) (-2.70) (-2.28) (-2.87)
$\eta_{(25,30]}$ 0.0202 0.00153 0.0118 0.00540 -0.00205 0.0212 0.00331
-0.0148∗ 0.00509 -0.00815 (1.55) (0.30) (1.49) (1.24) (-0.55) (0.51) (0.58)
(-2.36) (0.47) (-1.91) Constant 0.202∗∗∗ 0.0170∗∗∗ 0.0146 0.0190∗∗∗ 0.00772∗
0.345∗∗∗ 0.0246∗∗∗ 0.0222∗∗ 0.0452∗∗∗ 0.0130∗∗ (13.57) (3.39) (1.57) (3.57)
(2.09) (8.82) (6.38) (3.18) (4.58) (2.91) Observations 28370 28370 28370 28370
28370 28370 28370 28370 28370 28370 t statistics in parentheses ∗ $p<0.05$, ∗∗
$p<0.01$, ∗∗∗ $p<0.001$

(a) Uncertainty

(b) Uncertainty-Economics

(c) Uncertainty-Health

(d) Uncertainty-Politics

(e) Uncertainty-Policy

Figure 12: Sensitivity analysis using relative magnitudes restrictions of
$\delta_{0}$ of equation 10 for _Uncertainty_ aggregated and grouped by Topic.
The baseline period starts from the 1rst of February and ends on the 19th of
February.

(a) Negative Sentiment

(b) Negative Sentiment-Economics

(c) Negative Sentiment-Health

(d) Negative Sentiment-Politics

(e) Negative Sentiment-Policy

Figure 13: Sensitivity analysis using relative magnitudes restrictions of
$\delta_{0}$ of equation 10 for _Negative Sentiment_ aggregated and grouped by
Topic. The baseline period starts from the 1rst of February and ends on the
19th of February.

(a) Uncertainty

(b) Negative Sentiment

(c) Uncertainty-Economics

(d) Uncertainty-Health

(e) Uncertainty-Politics

(f) Uncertainty-Policy

(g) Negative Sentiment-Economics

(h) Negative Sentiment-Health

(i) Negative Sentiment-Politics

(j) Negative Sentiment-Policy

Figure 14: Coefficient Estimates and Confidence Intervals (% 95) of
interaction between the treatment variable and time dummies. The dependent
variable is Share of tweets classified as _Uncertainty_ and _Negative
Sentiment_ aggregated and grouped by Topic. The baseline period starts from
the 1rst of February and ends on the 19th of February.

##  Appendix G Multiple Test Adjusted P-values

We report the Benjamini-Hochberg adjusted pvalues [14] for multiple hypothesis
testing for false discovery rate (with uncorrected p-values in parenthesis).
We jointly test the p-values of the model for Aggregate uncertainty
(sentiment) with additional regressions for two new binary variables
representing the correlated categories of _certainty_ and _medium_ uncertainty
(_positive_ and _medium_ sentiment), as there are likely correlations amongst
the p-values of the three regression models that may have been inflating our
significance. Considering uncertainty and negative sentiment grouped by
possible topics, the p-values of the four topic-related independent
regressions are adjusted together.

In Table 8, we confirm all main findings of the baseline estimation as the
treatment effect (_red_ $zone$ x $post=1$) retains its statistical
significance on uncertainty towards economics, health and politics and
negative sentiment on health and politics. This indicates that the original
p-value of the policy effect on public reaction was not inflated by
correlations among the regression models, leading to an incorrect inference.

Table 7: Benjamini Hochberg (1995) adjusted p-values (original p-values in
parenthesis)

Uncertainty Negative Sentiment Aggregate Economics Health Politics Policy
Aggregate Economics Health Politics Policy post = 1 0.01 0.02 0.00 0.11 0.00
0.10 0.95 0.00 0.11 0.00 (0.00) (0.01) (0.00) (0.07) (0.00) (0.08) (0.95)
(0.00) (0.07) (0.01) post = 2 0.01 0.00 0.00 0.72 0.00 0.00 0.28 0.00 0.02
0.00 (0.00) (0.00) (0.00) (0.54) (0.00) (0.03) (0.26) (0.00) (0.01) (0.00) red
zone = 1 x post = 1 0.03 0.84 0.00 0.00 0.01 0.63 0.17 0.00 0.00 0.02 (0.02)
(0.78) (0.00) (0.00) (0.00) (0.63) (0.14) (0.00) (0.00) (0.01) red zone = 1 x
post = 2 0.57 0.72 0.94 0.41 0.99 0.63 0.19 0.10 0.00 0.12 (0.49) (0.59)
(0.88) (0.10) (0.29) (0.99) (0.16) (0.07) (0.00) (0.09)

Table 8: Benjamini Hochberg (1995) adjusted p-values (original p-values in
parenthesis)

Uncertainty Negative Sentiment Aggregate Economics Health Politics Policy
Aggregate Economics Health Politics Policy red zone = 1 0.53 0.00 0.97 0.00
0.92 0.04 0.02 0.92 0.00 0.92 (0.39) (0.00) (0.97) (0.00) (0.87) (0.01) (0.01)
(0.92) (0.00) (0.90) post = 1 0.01 0.01 0.00 0.66 0.00 0.38 0.00 0.00 0.17
0.02 (0.00) (0.00) (0.00) (0.53) (0.00) (0.23) (0.00) (0.00) (0.12) (0.01)
post = 2 0.00 0.00 0.00 0.63 0.00 0.18 0.78 0.00 0.02 0.03 (0.00) (0.00)
(0.00) (0.47) (0.00) (0.08) (0.66) (0.00) (0.01) (0.02) red zone = 1 x post =
1 0.74 0.00 0.01 0.02 0.85 0.72 0.17 0.00 0.00 0.42 (0.74) (0.00) (0.01)
(0.01) (0.76) (0.67) (0.13) (0.00) (0.00) (0.34) red zone = 1 x post = 2 0.74
0.36 0.85 0.16 0.09 0.72 0.86 0.06 0.00 0.07 (0.73) (0.25) (0.72) (0.10)
(0.06) (0.66) (0.77) (0.03) (0.00) (0.05)

##  Appendix H Placebo Test

We perform a _placebo_ test to see if we can retrieve the effects of the
lockdown when the implementation of quarantine measures is not unexpected by
the public.

We argue that in those cities where the perceived risk of the danger of the
virus is _similar_ to the first 10 cities under quarantine, after the national
lockdown there should not be any particular different reaction compared to
cities with a totally dissimilar perceived risk, since the announcement of the
nation-wide measures were easily anticipated and expected. We take the
percentage increase of total deaths at the monthly and city level available on
the Istat archives454545Istat data available at
https://www.istat.it/it/archivio/240401, under “4 maggio 2020” as a proxy
variable for the perceived risk related to the pandemics. For each city, the
percentage increase over a given month in 2020, is computed as the total
number of deaths in that month minus the mean of the total number of deaths in
the same month over the years 2015-2019 divided by the mean of the total
deaths of that month over the period 2015-2019. For instance, if we take the
month of January 2020, we compute the total number of deaths in January 2020
minus the mean of the total number of deaths of January over the years
2015-2019, divided the mean of the total deaths of January over 2015-2019.

We select tweets geo-located in municipalities from the North of Italy only
(from a larger dataset of tweets at the national level), and exclude those
municipalities used as controls for the Diff-In-Diff models.

For each city, we consider the percentage increase of deaths of January and
February 2020. Each location is therefore represented by a two-dimensional
vector and we wish to compare this vector to another two-dimensional array of
the monthly percentage increases averaged over the ten cities undergoing the
first lockdown. The overall distance between a city and the averages over the
ten under the first lockdown is computed based on the Euclidean distance. This
allows us to define the binary variable _sim_ indicating whether the
perception of risk in the city is close to that of the ten under quarantine:
the positive label 1 indicates all those cities with Euclidean distance less
or equal to the 10th percentile of the distribution of Euclidean distance
values in the tweets sample (i.e. less or equal to 316.8821), the negative
label 0 stands for all cities with distance equal or above the 90th percentile
(i.e. above 424.1583). The cities and therefore observations in the middle are
removed to obtain a clean effect and not end up comparing two groups with
gradually homogeneous increase in total deaths (Figure 15). The cities with
positive (negative) similarity are 231 (295) and account for 6 967 (7 082)
observations.

Figure 15: Placebo test cities under lockdown (the treated group - in _red_)
and controls (in _yellow_).

We compare the national lockdown effect on public interests displayed on
Twitter between these set of cities with $sim=1$ and the municipalities
featuring $sim=0$ coming from the same regions. We include a time dummy $Post$
that equals 1 starting from March 9, following the government’s decision to
extend the measures nation-wide. Again, we drop all observations from March 8,
as some municipalities enter quarantine one day before the national lockdown.
We also remove March 7, the day of the announcement of the one-day lockdown
for 14 provinces in Northern Italy. To control for unobserved time-invariant
heterogeneity arising from the large number of different and far-apart
municipalities included in the sample, we include municipality level fixed-
effects.

From Table 9, we have no significant effect of the treatment, suggesting that
indeed the significant estimates we found in the baseline models are robust to
any anticipatory effect of the treatment.

Table 9: Placebo Test regression table for _Uncertainty_ and _Negative
Sentiment_ , aggregated and grouped by topics, with municipality level fixed
effects (omitted).

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)
Aggregate Economics Health Politics Policy Aggregate Economics Health Politics
Policy post=1 -0.0960∗∗∗ -0.00796 -0.0771∗∗∗ -0.0134∗ -0.0248∗∗ -0.0434∗∗
-0.00159 -0.0397∗∗ -0.0127 0.00225 (-6.73) (-1.12) (-5.38) (-2.15) (-3.20)
(-2.89) (-0.29) (-3.20) (-1.70) (0.49) sim=1 $\times$ post=1 0.0475 0.0105
0.0476 0.00491 0.0122 -0.0322 -0.0102 -0.0145 -0.00556 -0.000633 (1.76) (0.93)
(1.89) (0.53) (1.17) (-1.61) (-1.24) (-0.77) (-0.62) (-0.10) Constant 0.564∗∗∗
0.00530 0.385∗∗∗ 0.00894∗ 0.0166∗∗ 0.862∗∗∗ 0.00106 0.360∗∗∗ 0.00850 -0.00150
(59.25) (1.12) (40.32) (2.15) (3.20) (86.05) (0.29) (43.54) (1.70) (-0.49)
Observations 14049 14049 14049 14049 14049 14049 14049 14049 14049 14049
Clustered SE Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes t statistics in
parentheses ∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

Table 10: Placebo Test regression table for _Uncertainty_ and _Negative
Sentiment_ , aggregated and grouped by topics.

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)
Aggregate Economics Health Politics Lockdown Pol. Aggregate Economics Health
Politics Lockdown Pol. sim=1 -0.00494 -0.00889 0.00501 -0.000674 -0.0147∗
0.00621 0.00698 0.00227 0.00283 0.000678 (-0.21) (-1.83) (0.24) (-0.07)
(-1.99) (0.27) (1.35) (0.13) (0.33) (0.20) post=1 -0.0989∗∗∗ -0.00751
-0.0839∗∗∗ -0.0144∗ -0.0275∗∗∗ -0.0375∗ -0.000978 -0.0355∗∗ -0.0135 0.00255
(-8.11) (-1.15) (-7.05) (-2.50) (-4.00) (-2.29) (-0.19) (-2.61) (-1.75) (0.61)
sim=1 $\times$ post=1 0.0427 0.00763 0.0457∗ 0.00552 0.0152 -0.0347 -0.0108
-0.0188 -0.00308 -0.000927 (1.68) (0.72) (1.98) (0.63) (1.56) (-1.65) (-1.42)
(-0.99) (-0.32) (-0.15) Constant 0.430∗∗∗ 0.0523∗∗∗ 0.335∗∗∗ 0.0434∗∗∗
0.0736∗∗∗ 0.323∗∗∗ 0.0282∗∗∗ 0.236∗∗∗ 0.0491∗∗∗ 0.0181∗∗∗ (42.63) (13.05)
(39.74) (12.29) (19.44) (20.97) (9.49) (15.96) (8.01) (9.45) Observations
14049 14049 14049 14049 14049 14049 14049 14049 14049 14049 Clustered SE Yes
Yes Yes Yes Yes Yes Yes Yes Yes Yes t statistics in parentheses ∗ $p<0.05$, ∗∗
$p<0.01$, ∗∗∗ $p<0.001$

Table 11: _Uncertainty_ and _Negative Sentiment_ Placebo test results.

(1) (2) Uncertainty Negative Sentiment sim=1 0.000666 0.00841 (0.04) (0.38)
post=1 -0.0745∗∗∗ -0.0622∗∗∗ (-6.52) (-6.58) sim=1 $\times$ post=1 0.0269
0.000252 (1.31) (0.02) Constant 0.453∗∗∗ 0.377∗∗∗ (52.92) (44.08) Observations
14904 14904 t statistics in parentheses ∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

Table 12: Placebo test for Uncertainty and Negative Sentiment grouped by
topic.

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) Economics
Health Politics Lockdown pol Economics Health Politics Lockdown pol sim=1
-0.0115∗∗ 0.0143 0.00489 -0.0146∗ -0.00382 0.00785 -0.00186 -0.00127 (-2.90)
(0.71) (0.38) (-2.49) (-0.69) (0.44) (-0.20) (-0.29) post=1 -0.00193
-0.0651∗∗∗ -0.0185∗∗∗ -0.0250∗∗∗ -0.00283 -0.0558∗∗∗ -0.0221∗∗∗ -0.00946∗
(-0.38) (-5.31) (-4.42) (-5.51) (-0.67) (-4.66) (-3.91) (-2.57) sim=1 $\times$
post=1 0.00409 0.0283 0.00635 0.0148 0.00326 0.0184 0.00495 0.00605 (0.46)
(1.49) (0.60) (1.95) (0.63) (1.15) (0.65) (1.18) Constant 0.0513∗∗∗ 0.365∗∗∗
0.0474∗∗∗ 0.0648∗∗∗ 0.0399∗∗∗ 0.288∗∗∗ 0.0612∗∗∗ 0.0396∗∗∗ (20.33) (39.88)
(13.96) (22.88) (14.27) (36.82) (13.97) (15.21) Observations 14904 14904 14904
14904 14904 14904 14904 14904 t statistics in parentheses ∗ $p<0.05$, ∗∗
$p<0.01$, ∗∗∗ $p<0.001$

##  Appendix I Model estimates with White Standard Errors

We report the DID model estimates with White standard errors robust to
heteroskedasticy, assuming to within cluster correlation of the errors.

Once the estimated standard errors do not account for within-city
correlations, we can no longer find a significant treatment effect on
political uncertainty. The variance _within_ clusters is much different from
the variance _between_ clusters, even as we correct for violation of
homoskedastic error variance.

Table 13: DiD Regression table for _Uncertainty_ and _Negative Sentiment_ ,
aggregated and grouped by topics with user fixed effects (omitted) with White
Standard Errors.

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)
Aggregate Economics Health Politics Policy Aggregate Economics Health Politics
Policy post=1 0.0378∗∗∗ 0.00819∗∗ 0.0592∗∗∗ -0.00679∗ 0.0198∗∗∗ -0.0337∗∗∗
-0.000346 0.0365∗∗∗ -0.0108∗∗ 0.00736∗∗∗ (4.64) (3.00) (16.59) (-2.50) (9.40)
(-3.94) (-0.12) (11.74) (-2.78) (4.34) post=2 0.0500∗∗∗ 0.0169∗∗∗ 0.0673∗∗∗
-0.00253 0.0191∗∗∗ -0.0405∗∗∗ 0.00614 0.0401∗∗∗ -0.0159∗∗∗ 0.00783∗∗∗ (5.69)
(5.43) (16.21) (-0.87) (8.25) (-4.45) (1.92) (11.09) (-3.97) (4.50) red zone=1
$\times$ post=1 0.145∗∗∗ 0.00222 0.0606∗∗ 0.0158 0.0281∗ -0.0168 -0.0176
0.0614∗∗ 0.0281∗∗ 0.0389∗∗ (3.65) (0.16) (3.09) (1.86) (2.12) (-0.40) (-1.04)
(3.12) (2.82) (2.91) red zone=1 $\times$ post=2 0.0475 -0.00495 0.00294
-0.00647 -0.000174 -0.0224 -0.0115 0.0419∗ 0.0358∗∗∗ 0.0233∗ (1.11) (-0.28)
(0.15) (-0.81) (-0.02) (-0.49) (-0.59) (2.13) (3.54) (2.00) Constant 0.902∗∗∗
-0.0119 -0.0702∗∗∗ 0.00900 -0.0189 0.0629 0.00535 -0.0819∗∗∗ -0.0199∗
-0.0311∗∗ (21.47) (-0.69) (-3.59) (1.21) (-1.67) (1.42) (0.28) (-4.23) (-2.14)
(-2.70) Observations 28370 28370 28370 28370 28370 28370 28370 28370 28370
28370 t statistics in parentheses ∗ $p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$

Table 14: DID Regression table for _Uncertainty_ and _Negative Sentiment_ ,
aggregated and grouped by topics, with White Standard Errors.

Uncertainty Negative Sentiment (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) All
Economics Health Politics Policy All Economics Health Politics Policy red
zone=1 0.0533 0.0115 -0.000325 -0.0160∗∗∗ 0.000634 0.134∗∗∗ 0.0236 -0.000780
-0.0455∗∗∗ -0.000504 (1.84) (1.10) (-0.05) (-3.35) (0.15) (4.02) (1.68)
(-0.13) (-14.47) (-0.11) post=1 0.0508∗∗∗ 0.00744∗∗∗ 0.0598∗∗∗ -0.00311
0.0198∗∗∗ -0.0518∗∗∗ -0.00781∗∗ 0.0318∗∗∗ -0.0152∗∗∗ 0.00523∗∗∗ (7.09) (3.38)
(22.40) (-1.28) (12.16) (-6.27) (-3.08) (13.88) (-4.35) (3.82) post=2
0.0635∗∗∗ 0.0143∗∗∗ 0.0683∗∗∗ -0.00335 0.0192∗∗∗ -0.0678∗∗∗ -0.00128 0.0382∗∗∗
-0.0235∗∗∗ 0.00736∗∗∗ (8.50) (5.87) (22.51) (-1.35) (10.94) (-7.98) (-0.47)
(14.78) (-6.78) (4.86) red zone=1 $\times$ post=1 0.0304 -0.0125 0.0388∗∗
0.0172∗∗ 0.00404 -0.0187 -0.0176 0.0548∗∗∗ 0.0542∗∗∗ 0.0132∗ (0.92) (-1.08)
(3.20) (2.61) (0.57) (-0.50) (-1.18) (4.72) (7.46) (1.99) red zone=1 $\times$
post=2 -0.0174 -0.0108 -0.00736 0.00805 -0.00780 0.0243 -0.00537 0.0452∗∗
0.0751∗∗∗ 0.00854 (-0.48) (-0.81) (-0.53) (1.21) (-1.04) (0.59) (-0.32) (2.98)
(6.81) (1.06) Constant 0.199∗∗∗ 0.0141∗∗∗ 0.00887∗∗∗ 0.0202∗∗∗ 0.00364∗∗∗
0.348∗∗∗ 0.0234∗∗∗ 0.00933∗∗∗ 0.0455∗∗∗ 0.00478∗∗∗ (33.03) (7.93) (6.27)
(9.53) (4.01) (48.49) (10.27) (6.43) (14.47) (4.59) Observations 28370 28370
28370 28370 28370 28370 28370 28370 28370 28370 t statistics in parentheses ∗
$p<0.05$, ∗∗ $p<0.01$, ∗∗∗ $p<0.001$


for $\arg\lambda\in(\frac{k-1}{2}\pi,\frac{k}{2}\pi)$, $k=1,2,3$. Here, the
constant matrices

$J_{0}=\left(\begin{array}[]{cc}1&0\\\ j_{0}&1\\\ \end{array}\right),\quad
J_{1}=\left(\begin{array}[]{cc}1&j_{1}\\\ 0&1\\\ \end{array}\right),\quad
J_{2}=\left(\begin{array}[]{cc}1&0\\\ -j_{0}e^{-2\pi i\nu}&1\\\
\end{array}\right),$ (7.17)

with $j_{0}=-i\frac{\sqrt{2\pi}}{\Gamma(\nu)}$ and
$j_{1}=-\frac{\sqrt{2\pi}}{\Gamma(1-\nu)}e^{i\pi\nu}$. We also denote

$J_{3}=(J_{0}J_{1}J_{2})^{-1}=\left(\begin{array}[]{cc}1&-j_{1}e^{2\pi
i\nu}\\\ 0&1\\\ \end{array}\right)e^{2\pi i\nu\sigma_{3}}.$ (7.18)

Then, it is direct to see that $\widehat{\Psi}(\lambda)$ satisfies the
following RH problem.

### RH problem for $\widehat{\Psi}(\lambda)$

  * (1)

$\widehat{\Psi}(\lambda)$ is analytic in $\mathbb{C}\setminus\Xi_{k}$, where
$\Xi_{k}=e^{i\frac{k}{2}\pi}\mathbb{R}_{+}$, $k=0,1,2,3$.

  * (2)

$\widehat{\Psi}(\lambda)$ satisfies the jump conditions

$\widehat{\Psi}_{+}(\lambda)=\widehat{\Psi}_{-}(\lambda)\left(\begin{array}[]{cc}1&0\\\
2\pi i\gamma-i\frac{\sqrt{2\pi}}{\Gamma(\nu)}&1\\\
\end{array}\right),\quad\lambda\in\Xi_{0},$ (7.19)

and

$\widehat{\Psi}_{+}(\lambda)=\widehat{\Psi}_{-}(\lambda)J_{k},\quad\lambda\in\Xi_{k},$
(7.20)

with $J_{k}$, $k=1,2,3$ given in (7.17) and (7.18).

  * (3)

$\widehat{\Psi}(\lambda)$ has the asymptotic behavior as $\lambda\to\infty$

$\widehat{\Psi}(\lambda)=\left(\mathbf{I}+\frac{\widehat{\Psi}_{1}}{\lambda}+O\left(\frac{1}{\lambda^{2}}\right)\right)\lambda^{-\nu\sigma_{3}}\exp\left((\lambda^{2}/2+x\lambda)\sigma_{3}\right).$
(7.21)

  * (4)

$\widehat{\Psi}(\lambda)$ possesses the behavior near the origin

$\widehat{\Psi}(\lambda)=O\left(\ln\lambda\right).$ (7.22)

It comes out that the RH problem for $\widehat{\Psi}(\lambda)$ is the same as
the RH problem for the Jimbo-Miwa Lax pair of the PIV equation with the
parameters $\theta_{0}=0$ and $\theta_{\infty}=\nu$; cf. [20, (C.30)-(C.31)],
see also [14, Chapter 5.1]. It is shown in [20, (C.34)-(C.37)] that

$\sigma_{\nu}(x)=-2(\widehat{\Psi}_{1})_{11}-2\nu x$ (7.23)

satisfies the equation (1.15). Using (7.15) and (7.16), we find

$2(Y_{1})_{11}=-2(\widehat{\Psi}_{1})_{11}-2\nu x.$ (7.24)

Thus, we have

$2(Y_{1})_{11}=\sigma_{\nu}(x).$ (7.25)

Recalling (7.13), we have derived (1.31).

Next, we prove (1.32). It is seen from (1.4) that the parabolic cylinder
function $D_{\nu}(x)$ decay exponentially fast as $x$ tends to positive
infinity. We get from the series expansion of the Fredholm determinant that

$\det(\mathbf{I}-\gamma K_{\nu,x})\sim
1-\gamma\int_{0}^{+\infty}K_{\nu,x}(\lambda,\lambda)d\lambda.$ (7.26)

Thus, we have

$\frac{d}{dx}\ln\det(\mathbf{I}-\gamma K_{\nu,x})\sim\gamma
K_{\nu,x}(0,0),~{}~{}~{}x\to+\infty.$ (7.27)

The asymptotic (1.32) then follows from (1.4), (1.30) and (7.27). We complete
the proof of Theorem 1.6.

## Acknowledgements

The authors are grateful to the editor and the referee for their valuable
suggestions and comments. The work of Shuai-Xia Xu was supported in part by
the National Natural Science Foundation of China under grant numbers 11571376
and 11971492, and by Guangdong Basic and Applied Basic Research Foundation
(Grant No. 2022B1515020063). Yu-Qiu Zhao was supported in part by the National
Natural Science Foundation of China under grant numbers 11571375 and 11971489.

##  Appendix A Local parametrix models

###  A.1 Airy parametrix

Introduce

$\Phi^{(\mathrm{Ai})}(\zeta)=\mathbf{N}\left\\{\begin{aligned}
&\begin{pmatrix}\mathrm{Ai}(\zeta)&\mathrm{Ai}(w^{2}\zeta)\\\
\mathrm{Ai}^{\prime}(\zeta)&w^{2}\mathrm{Ai}^{\prime}(w^{2}\zeta)\end{pmatrix}e^{-i\frac{\pi}{6}\sigma_{3}},&\zeta&\in\mathrm{I},\\\
&\begin{pmatrix}\mathrm{Ai}(\zeta)&\mathrm{Ai}(w^{2}\zeta)\\\
\mathrm{Ai}^{\prime}(\zeta)&w^{2}\mathrm{Ai}^{\prime}(w^{2}\zeta)\end{pmatrix}e^{-i\frac{\pi}{6}\sigma_{3}}\begin{pmatrix}1&0\\\
-1&1\end{pmatrix},&\zeta&\in\mathrm{II},\\\
&\begin{pmatrix}\mathrm{Ai}(\zeta)&-w^{2}\mathrm{Ai}(w\zeta)\\\
\mathrm{Ai}^{\prime}(\zeta)&-\mathrm{Ai}^{\prime}(w\zeta)\end{pmatrix}e^{-i\frac{\pi}{6}\sigma_{3}}\begin{pmatrix}1&0\\\
1&1\end{pmatrix},&\zeta&\in\mathrm{III},\\\
&\begin{pmatrix}\mathrm{Ai}(\zeta)&-w^{2}\mathrm{Ai}(w\zeta)\\\
\mathrm{Ai}^{\prime}(\zeta)&-\mathrm{Ai}^{\prime}(w\zeta)\end{pmatrix}e^{-i\frac{\pi}{6}\sigma_{3}},&\zeta&\in\mathrm{IV},\end{aligned}\right.$
(A.1)

where $\mathrm{Ai}(\zeta)$ denotes the Airy function (cf. [24, Chapter 9]),
$w=e^{2\pi i/3}$ and

$\mathbf{N}=\sqrt{2\pi}\,e^{\frac{1}{6}\pi i}\begin{pmatrix}1&0\\\
0&-i\end{pmatrix}.$

The regions I-IV are illustrated in Figure 9. Then,
$\Phi^{(\mathrm{Ai})}(\zeta)$ solves the following RH problem (cf. [11]).

### RH problem for $\Phi^{(\mathrm{Ai})}(\zeta)$

(1) $\Phi^{\mathrm{(Ai)}}(\zeta)$ is analytic for
$\zeta\in\mathbb{C}\setminus\bigcup^{4}_{k=1}\Sigma_{k}$, where
$\Sigma_{1}=\mathbb{R}_{+}$, $\Sigma_{2}=e^{\frac{2\pi i}{3}}\mathbb{R}_{+}$,
$\Sigma_{3}=\mathbb{R}_{-}$ and $\Sigma_{4}=e^{-\frac{2\pi
i}{3}}\mathbb{R}_{+}$ with orientations indicated in Figure 9.

(2) We have the jump relation

$\Phi^{\mathrm{(Ai)}}_{+}(\zeta)=\Phi^{\mathrm{(Ai)}}_{-}(\zeta)\left\\{\begin{aligned}
&\begin{pmatrix}1&1\\\ 0&1\end{pmatrix},&&\zeta\in\Sigma_{1},\\\
&\begin{pmatrix}1&0\\\ 1&1\end{pmatrix},&&\zeta\in\Sigma_{2}\cup\Sigma_{4},\\\
&\begin{pmatrix}0&1\\\
-1&0\end{pmatrix},&&\zeta\in\Sigma_{3}.\end{aligned}\right.$

(3) $\Phi^{\mathrm{(Ai)}}(\zeta)$ possesses the following asymptotic behavior
as $\zeta\to\infty$

$\Phi^{(\mathrm{Ai})}(\zeta)=\zeta^{-\frac{\sigma_{3}}{4}}\frac{1}{\sqrt{2}}\begin{pmatrix}1&i\\\
i&1\end{pmatrix}\left(\mathbf{I}+O\left(\zeta^{-\frac{3}{2}}\right)\right)e^{-\frac{2}{3}\zeta^{\frac{3}{2}}\sigma_{3}}.$
(A.2)

Figure 9: The jump contour and regions for $\Phi^{(\mathrm{Ai})}$

###  A.2 Parabolic cylinder parametrix

Let ${\beta}$ be a fixed real or complex number. Define

$\mathbf{D}(\zeta)=2^{-\frac{\sigma_{3}}{2}}\begin{pmatrix}D_{-{\beta}-1}(i\zeta)&D_{{\beta}}(\zeta)\\\
iD_{-{\beta}-1}^{\prime}(i\zeta)&D_{{\beta}}^{\prime}(\zeta)\end{pmatrix}\begin{pmatrix}e^{i\frac{\pi}{2}({\beta}+1)}&0\\\
0&1\end{pmatrix},$ (A.3)

where $D_{{\beta}}(\zeta)$ denotes the standard parabolic cylinder function
(cf. [24, Chapter 12]). Denote

$H_{0}=\begin{pmatrix}1&0\\\ h_{0}&1\end{pmatrix},\quad
H_{1}=\begin{pmatrix}1&h_{1}\\\ 0&1\end{pmatrix},\quad
H_{k+2}=e^{i\pi\left({\beta}+\frac{1}{2}\right)\sigma_{3}}H_{k}e^{-i\pi\left({\beta}+\frac{1}{2}\right)\sigma_{3}},\
k=0,1,$

where

$h_{0}=-i\frac{\sqrt{2\pi}}{\Gamma({\beta}+1)},\quad
h_{1}=\frac{\sqrt{2\pi}}{\Gamma(-{\beta})}e^{i\pi{\beta}},\quad
1+h_{0}h_{1}=e^{2\pi i{\beta}}.$ (A.4)

We consider the function

$\Phi^{(\mathrm{PC})}(\zeta)=\left\\{\begin{aligned}
&\mathbf{D}(\zeta),\quad&\arg\zeta\in&\left(-\frac{\pi}{4},0\right),\\\
&\mathbf{D}(\zeta)H_{0},\quad&\arg\zeta\in&\left(0,\frac{\pi}{2}\right),\\\
&\mathbf{D}(\zeta)H_{1},\quad&\arg\zeta\in&\left(\frac{\pi}{2},\pi\right),\\\
&\mathbf{D}(\zeta)H_{2},\quad&\arg\zeta\in&\left(\pi,\frac{3\pi}{2}\right),\\\
&\mathbf{D}(\zeta)H_{3},\quad&\arg\zeta\in&\left(\frac{3\pi}{2},\frac{7\pi}{4}\right).\end{aligned}\right.$

It is direct to check that $\Phi^{(\mathrm{PC})}(\zeta)$ solves the following
RH problem (cf. [7, 14]).

### RH problem for $\Phi^{(\mathrm{PC})}(\zeta)$

(1) $\Phi^{\mathrm{(PC)}}(\zeta)$ is analytic for all
$\zeta\in\mathbb{C}\setminus\bigcup^{4}_{k=0}\Upsilon_{k}$, where

$\Upsilon_{k}=\left\\{\zeta\in\mathbb{C}~{}\Big{|}~{}\arg\zeta=\frac{k\pi}{2}\right\\},~{}k=0,1,2,3,\quad\Upsilon_{4}=\left\\{\zeta\in\mathbb{C}~{}\Big{|}~{}\arg\zeta=-\frac{\pi}{4}\right\\};$

see Figure 10.

(2) $\Phi^{\mathrm{(PC)}}(\zeta)$ satisfies the jump conditions

$\Phi^{(\mathrm{PC})}_{+}(\zeta)=\Phi^{(\mathrm{PC})}_{-}(\zeta)\left\\{\begin{aligned}
&H_{k},\quad&&\zeta\in\Upsilon_{k},~{}k=0,1,2,3,\\\ &e^{2\pi
i\beta\sigma_{3}},\quad&&\zeta\in\Upsilon_{4}.\end{aligned}\right.$

(3) $\Phi^{\mathrm{(PC)}}(\zeta)$ satisfies the following asymptotic behavior
as $\zeta\to\infty$

$\displaystyle\Phi^{\mathrm{(PC)}}(\zeta)=\begin{pmatrix}0&1\\\
1&-\zeta\end{pmatrix}2^{\frac{\sigma_{3}}{2}}\begin{pmatrix}1+O\left(\zeta^{-2}\right)&\frac{{\beta}}{\zeta}+O\left(\zeta^{-3}\right)\\\
\frac{1}{\zeta}+O\left(\zeta^{-3}\right)&1+O\left(\zeta^{-2}\right)\end{pmatrix}e^{\frac{\zeta^{2}}{4}\sigma_{3}}\zeta^{-{\beta}\sigma_{3}}.$
(A.5)

Figure 10: The jump contour and jump matrices for $\Phi^{(\mathrm{PC})}$

###  A.3 Bessel parametrix

Consider the following RH problem.

### RH problem for $\Phi^{(\mathrm{Bes})}(\zeta)$

(1) $\Phi^{\mathrm{(Bes)}}(\zeta)$ is analytic for all
$\zeta\in\mathbb{C}\setminus\bigcup^{8}_{k=1}\Gamma_{k}$, where

$\Gamma_{k}=\left\\{\zeta\in\mathbb{C}~{}\Big{|}~{}\arg\zeta=\frac{(k-1)\pi}{4}\right\\},~{}k=1,\cdots,8;$

see Figure 11.

(2) We have the jump conditions

$\Phi_{+}^{(\mathrm{Bes})}(\zeta)=\Phi_{-}^{(\mathrm{Bes})}(\zeta)\left\\{\begin{aligned}
&\begin{pmatrix}0&1\\\
-1&0\end{pmatrix},\quad&\zeta&\in\Gamma_{1}\cup\Gamma_{5},\\\
&\begin{pmatrix}1&0\\\ e^{-2\pi
i\alpha}&1\end{pmatrix},\quad&\zeta&\in\Gamma_{2}\cup\Gamma_{6},\\\
&\begin{pmatrix}e^{\pi i\alpha}&0\\\ 0&e^{-\pi
i\alpha}\end{pmatrix},\quad&\zeta&\in\Gamma_{3}\cup\Gamma_{7},\\\
&\begin{pmatrix}1&0\\\ e^{2\pi
i\alpha}&1\end{pmatrix},\quad&\zeta&\in\Gamma_{4}\cup\Gamma_{8}.\end{aligned}\right.$
(A.6)

(3) As $\zeta\rightarrow\infty$,

$\Phi^{\mathrm{(Bes)}}(\zeta)=\frac{1}{\sqrt{2}}\begin{pmatrix}1&-i\\\
-i&1\end{pmatrix}(\mathbf{I}+O(\zeta^{-1}))e^{\frac{\pi
i\sigma_{3}}{4}}e^{-i\zeta\sigma_{3}}\left\\{\begin{aligned}
&e^{-\frac{\alpha\pi i\sigma_{3}}{2}},&\zeta&\in\Lambda_{1}\cup\Lambda_{2},\\\
&e^{\frac{\alpha\pi i\sigma_{3}}{2}},&\zeta&\in\Lambda_{3}\cup\Lambda_{4},\\\
&e^{\frac{\alpha\pi
i\sigma_{3}}{2}}\sigma_{1}\sigma_{3},&\zeta&\in\Lambda_{5}\cup\Lambda_{6},\\\
&e^{-\frac{\alpha\pi
i\sigma_{3}}{2}}\sigma_{1}\sigma_{3},&\zeta&\in\Lambda_{7}\cup\Lambda_{8}.\end{aligned}\right.$
(A.7)

Figure 11: The jump contour and regions for $\Phi^{(\mathrm{Bes})}$

From [26], we see that the above RH problem can be constructed in terms of the
modified Bessel function $I_{\alpha\pm\frac{1}{2}}(\zeta)$ and
$K_{\alpha\pm\frac{1}{2}}(\zeta)$:

$\Phi^{\mathrm{(Bes)}}(\zeta)=\begin{pmatrix}\sqrt{\pi}\zeta^{\frac{1}{2}}I_{\alpha+\frac{1}{2}}(\zeta
e^{-\frac{\pi
i}{2}})&-\frac{1}{\sqrt{\pi}}\zeta^{\frac{1}{2}}K_{\alpha+\frac{1}{2}}(\zeta
e^{-\frac{\pi i}{2}})\\\
-i\sqrt{\pi}\zeta^{\frac{1}{2}}I_{\alpha-\frac{1}{2}}(\zeta e^{-\frac{\pi
i}{2}})&-\frac{i}{\sqrt{\pi}}\zeta^{\frac{1}{2}}K_{\alpha-\frac{1}{2}}(\zeta
e^{-\frac{\pi i}{2}})\end{pmatrix}e^{-\frac{1}{2}\alpha\pi i\sigma_{3}}$ (A.8)

for $\zeta\in\Lambda_{2}$. The explicit expressions of
$\Phi^{\mathrm{(Bes)}}(\zeta)$ in other sectors are determined by (A.8) and
the jump relation (A.6).

Using the series expansion of the modified Bessel function [24, (10.25.2)]

$I_{\mu}(\zeta)=\left(\frac{\zeta}{2}\right)^{\mu}\sum_{k=0}^{\infty}\frac{(\frac{\zeta^{2}}{4})^{k}}{k!\,\Gamma(\mu+k+1)},$

and the connection formula [24, (10.27.4)]

$K_{\mu}(\zeta)=\frac{\pi}{2}\frac{I_{-\mu}(\zeta)-I_{\mu}(\zeta)}{\sin(\pi\mu)},\quad\mu\not\in\mathbb{Z},$

it is seen from (A.8) that

$\Phi^{\mathrm{(Bes)}}(\zeta)=(I+O(\zeta))C_{\alpha}^{\sigma_{3}}\begin{pmatrix}0&-1\\\
1&0\end{pmatrix}\zeta^{\alpha\sigma_{3}}\begin{pmatrix}1&\frac{1}{1+e^{-2\pi
i\alpha}}\\\ 0&1\end{pmatrix},\quad\zeta\to 0,~{}~{}\zeta\in\Lambda_{2},$
(A.9)

where $\alpha-\frac{1}{2}\not\in\mathbb{Z}$ and the constant $C_{\alpha}$ is
given by

$C_{\alpha}=\pi^{-\frac{1}{2}}2^{\alpha-\frac{1}{2}}e^{\pi
i(\alpha+\frac{1}{4})}\Gamma(\alpha+\frac{1}{2}).$ (A.10)

If $\frac{1}{2}-\alpha\in\mathbb{N}$, using the connection formulas [24,
(10.27.1)] and [24, (10.27.3)]

$I_{-n}(\zeta)=I_{n}(\zeta),\quad K_{-\mu}(\zeta)=K_{\mu}(\zeta),$

and the small-$z$ asymptotics [24, (10.30.1)-(10.30.3)]

$I_{\mu}(\zeta)\sim\frac{\zeta^{\mu}}{2^{\mu}\Gamma(\mu+1)},\quad
K_{\mu}(\zeta)\sim\frac{2^{\mu-1}\Gamma(\mu+1)}{\zeta^{\mu}},\quad
K_{0}(\zeta)\sim-\ln\zeta,$

we obtain

$\Phi^{\mathrm{(Bes)}}(\zeta)=\Phi^{\mathrm{(Bes)}}_{0}(\zeta)\zeta^{-\alpha\sigma_{3}},\quad\zeta\to
0,\quad\zeta\in\Lambda_{2},$ (A.11)

where $\Phi^{\mathrm{(Bes)}}_{0}(\zeta)$ is analytic near the origin.

The behaviors of $\Phi^{\mathrm{(Bes)}}(\zeta)$ near the origin in the other
regions can be determined by (A.9), (A.11) and the jump relations (A.6).

## References

  * [1] A.S. Abdullayev, Justification of asymptotic formulas for the fourth Painlevé equation, Stud. Appl. Math., 99 (1997), 255-283.
  * [2] J. Baik, R. Buckingham and J. DiFranco, Asymptotics of Tracy-Widom distributions and the total integral of a Painlevé II function, Comm. Math. Phys., 280 (2008), 463-497.
  * [3] J. Baik, R. Buckingham, J. DiFranco and A.R. Its, Total integrals of global solutions to Painlevé II, Nonlinearity, 22 (2009), 1021-1061.
  * [4] A.P. Bassom, P.A. Clarkson and A.C. Hicks, Bäcklund transformations and solution hierarchies for the fourth Painlevé equation, Stud. Appl. Math., 95 (1995), 1-71.
  * [5] A.P. Bassom, P.A. Clarkson and A.C. Hicks, Numerical studies of the fourth Painlevé equation, IMA J. Appl. Math., 50 (1993), 167-193.
  * [6] A.P. Bassom, P.A. Clarkson, A.C. Hicks and J.B. McLeod, Integral equations and exact solutions for the fourth Painlevé equation, Proc. R. Soc. A, 437 (1992), 1-24.
  * [7] T. Bothner and A.R. Its, The nonlinear steepest descent approach to the singular asymptotics of the second Painlevé transcendent, Phys. D, 241 (2012), 2204-2225.
  * [8] Y. Chen and G. Pruessner, Orthogonal polynomials with discontinuous weights, J. Phys. A, 38 (2005), L191-L198.
  * [9] P.A. Clarkson and J.B. McLeod, Integral equations and connection formulae for the Painlevé equations, Painlevé Transcendents, their Asymptotics and Physical Applications, Ed. P. Winternitz and D. Levi, 1992, 1-31.
  * [10] D. Dai, S.-X. Xu and L. Zhang, On integrals of the tronquée solutions and the associated Hamiltonians for the Painlevé II equation, J. Differential Equations, 269 (2020), 2430–2476.
  * [11] P. Deift, Orthogonal Polynomials and Random Matrices: A Riemann-Hilbert Approach, Courant Lecture Notes, vol. 3, New York University, 1999.
  * [12] P. Deift and X. Zhou, A steepest descent method for oscillatory Riemann-Hilbert problems. Asymptotics for the MKdV equation, Ann. Math., 137 (1993), 295-368.
  * [13] P. Deift and X. Zhou, Asymptotics for the Painlevé II equation, Comm. Pure Appl. Math., 48 (1995), 277-337.
  * [14] A.S. Fokas, A.R. Its, A.A. Kapaev and V.Y. Novokshenov, Painlevé transcendents: The Riemann-Hilbert approach, Math. Surv. Monog., Vol. 128, Amer. Math. Soc., Providence, RI, 2006.
  * [15] P. Forrester and N. Witte, Application of the $\tau$-function theory of Painlevé equations to random matrices: PIV, PII and the GUE, Comm. Math. Phys., 219 (2001), 357-398.
  * [16] W.Y. Hu, Singular asymptotics for solutions of the inhomogeneous Painlevé II equation, Nonlinearity, 32 (2019), 3843-3872.
  * [17] A.R. Its, A.G. Izergin, V.E. Korepin, N.A. Slavnov, Differential equations for quantum correlation functions, Int. J. Mod. Phys., B 4 (1990), 1003-1037.
  * [18] A.R. Its and A.A. Kapaev, Connection formulae for the fourth Painlevé transcendent: Clarkson-McLeod solution, J. Phys. A, 31 (1998), 4073-4113.
  * [19] M. Jimbo, T. Miwa and K. Ueno, Monodromy preserving deformation of linear ordinary differential equations with rational coefficients. I. General theorey and $\tau$-function, Phys. D, 2 (1981), 306-352.
  * [20] M. Jimbo and T. Miwa, Monodromy preserving deformation of linear ordinary differential equations with rational coefficients. II., Phys. D, 2 (1981), 407-448.
  * [21] A.A. Kapaev, Connection formulae for the degenerated asymptotic solutions of the fourth Painlevé equation, arXiv:solv-int/9805011v1.
  * [22] P. Kokocki, Total integrals of Ablowitz-Segur solutions for the inhomogeneous Painlevé II equation, Stud. Appl. Math., 144 (2020), 504-547.
  * [23] P.D. Miller, On the increasing tritronquée solutions of the Painlevé-II equation, SIGMA, 14 (2018), 125.
  * [24] F.W.J. Olver, A.B. Olde Daalhuis, D.W. Lozier, B.I. Schneider, R.F. Boisvert, C.W. Clark, B.R. Miller, B.V. Saunders (Eds.), 2020, NIST Digital Library of Mathematical Functions, http://dlmf.nist.gov/.
  * [25] C. Tracy and H. Widom, Fredholm determinants, differential equations and matrix models, Comm. Math. Phys., 159 (1994), 151-174.
  * [26] M. Vanlessen, Strong asymptotics of the recurrence coefficients of orthogonal polynomials associated to the generalized Jacobi weight, J. Approx. Theory, 125 (2003), 198-237.
  * [27] R. Wong and H.Y. Zhang, On the connection formulas of the fourth Painlevé transcendent, Anal. Appl., 4 (2009), 419-448.
  * [28] J. Xia, S.-X. Xu and Y.-Q. Zhao, Singular asymptotics for the Clarkson-McLeod solutions of the fourth Painlevé equation, Phys. D, 434 (2022), 133254.
  * [29] J.-R. Zhou, S.-X. Xu and Y.-Q. Zhao, Uniform asymptotics of a system of Szegő class polynomials via the Riemann-Hilbert approach, Anal. Appl., 9 (2011), 447-480.
  * [30] J.-R. Zhou and Y.-Q. Zhao, Uniform asymptotics of the Pollaczek polynomials via the Riemann-Hilbert approach, Proc. R. Soc. Lond. Ser. A, 464 (2008), 2091-2112.