# A one-shot texture-perceiving generative Adversarial network for
unsupervised surface inspection

###### Abstract

Visual surface inspection is a challenging task owing to the highly diverse
appearance of target surfaces and defective regions. Previous attempts heavily
rely on vast quantities of training examples with manual annotation. However,
in some practical cases, it is difficult to obtain a large number of samples
for inspection. To combat it, we propose a hierarchical texture-perceiving
generative adversarial network (HTP-GAN) that is learned from the one-shot
normal image in an unsupervised scheme. Specifically, the HTP-GAN contains a
pyramid of convolutional GANs that can capture the global structure and fine-
grained representation of an image simultaneously. This innovation helps
distinguishing defective surface regions from normal ones. In addition, in the
discriminator, a texture-perceiving module is devised to capture the spatially
invariant representation of normal image via directional convolutions, making
it more sensitive to defective areas. Experiments on a variety of datasets
consistently demonstrate the effectiveness of our method.

Index Terms—  One-shot learning, texture-perceiving module, visual surface
inspection, generative adversarial network

##  1 Introduction

Due to the rapid development of deep neural networks [1, 2, 3, 4], visual
surface inspection [5, 6] has attracted increasing attention as an important
technology in many intelligent industrial applications. Visual surface
inspection aims to detect the abnormal regions on the surface of material
using visual images. It is a challenging task owing to various image noises,
texture variations of the target surface, and highly diversified appearance of
abnormal regions.

Typical visual inspection approaches can be categorized into two main groups:
traditional methods [6, 7] and learning-based methods [5, 8, 9, 10].
Traditional methods adopt handcrafted features to perform surface inspection,
which cannot be well generalized to new scenarios. Learning-based approaches
achieve significant performance when equipped with large amounts of manually
annotated training data. However, in some practical cases, such as surface
inspection under planes [11], only a small number of normal samples, or even a
single sample, are available. Therefore, how to design the one-shot surface
inspection method is very important for practical scenarios.

We present the problem of one-shot unsupervised surface inspection: given an
example of a normal image as training data, all defective regions should be
detected and segmented for arbitrary images with the same texture category as
the normal image. The challenge lies in 1) how to design an adaptable
perceiving model that is prone to handle the texture variations; 2) how to
perform a more generalized surface inspection model in the one-shot way.

To deal with the challenge, we propose a novel hierarchical texture-perceiving
generative adversarial network (HTP-GAN) that is learned from a one-shot
normal image in an unsupervised scheme. Specifically, HTP-GAN model contains a
pyramid of convolutional GANs which utilize a single image to simultaneously
extract the global and fine-grained representation of the image. This guides
the model to indeed learn the normal texture representation of the category of
the corresponding image and can distinguish the defective surface regions. In
addition, in the discriminator of HTP-GAN, texture-perceiving module is
devised to capture the spatially invariant representation of normal image via
directional convolutions, making it more sensitive to defective areas.
Experiments on a variety of datasets consistently demonstrate the
effectiveness of our method.

Fig. 1: Overview of the proposed hierarchical texture-perceiving generative
adversarial network (HTP-GAN). In the training phase, the HTP-GAN is devised
to capture the spatially invariant representation of a single normal image via
directional convolutions at multiple scales. In the testing phase, the well-
trained discriminators can indeed memorize the latent distribution properties
of normal texture representation and distinguish the defective surface
regions.

We summarize our main contributions as follows:

  * •

We introduce a novel one-shot surface inspection network with a pyramid of
convolutional GANs, achieving unsupervised surface inspection with a single
example of normal image.

  * •

A texture-perceiving module is devised to capture the spatially invariant
representation of normal image via directional convolutions, making it more
sensitive to defective areas.

  * •

Experimental results achieve the state-of-the-art performance on two public
datasets, which demonstrate the effectiveness of the proposed approaches.

##  2 Hierarchical Texture-perceiving Generative Adversarial Network

Our goal is to detect the abnormal regions on the surface of material from a
single image. When large amounts of training data are available, a well-
trained generative adversarial network (GAN) can easily learn a representation
of the distribution of the target samples via a generator $G$ and a
discriminator $D$. But in the case of the limited number of samples,
conventional GAN is hard to learn such a representation due to insufficient
training data [12]. In the context of learning from a single image, inspired
by the SinGAN [1], we adopt the downsampling strategy to generate different
scales of images from a single image as training samples. Then the network is
able to learn internal distributions at different scales. At the same time, a
pyramid of convolutional GANs is utilized to simultaneously capture the
statistics of complex image structures and perceive the fine-grained
representation of the normal image. To our best knowledge, this is the first
time that the SinGAN being applied in the one-shot surface inspection task.

###  2.1 Hierarchical Fully Convolution Architecture

In this section, we will detail the overall architecture of the proposed HTP-
GAN. As shown in Fig. 1, the network is based on SinGAN [1] to achieve the
one-shot learning task. We change the traditional GAN from two perspectives:
1) we resize the input image to generate the multi-scale images. Specifically,
given an image of normal surface $x_{0}$, we adopt the down-sampled strategy
to generate an image pyramid of $x:\\{x_{0},\ldots,x_{N}\\}$. Multi-scale
inputs contain more fine details and texture information. 2) a pyramid of
convolutional GANs are utilized to simultaneously capture the characteristics
of complex image structures and perceive fine-grained representation of the
training image. So we need to design a pyramid of generators
$\\{G_{0},\ldots,G_{N}\\}$ and discriminators $\\{D_{0},\ldots,D_{N}\\}$ to
handle the multi-scale inputs.

At the training phase, the proposed HTP-GAN is a multi-stage training process
from coarse-grain to fine-grain. Firstly, a noise map is injected into the
generator $G_{N}$ at the coarsest scale to generate the $\tilde{x}_{N}$. Then
a combination between the generated image and the noise map sequentially
passes through all other generators up to the finest scale:

$\leavevmode\resizebox{422.77661pt}{}{$\tilde{x}_{n}=G_{n}\left(z_{n},\left(\tilde{x}_{n+1}\right)\uparrow^{r}\right)=\left(\tilde{x}_{n+1}\right)\uparrow^{r}+\psi_{n}\left(z_{n}+\left(\tilde{x}_{n+1}\right)\uparrow^{r}\right)$},$
(1)

where $\psi_{n}$ is a fully convolutional net with 5 conv-blocks and
$\uparrow^{r}$ denote to the up-sampling operation. As shown in Fig. 1, the
first 4 conv-blocks consist of a $3\times 3$ convolutional layer following
with a BatchNorm and a LeakyReLU. The last conv-block consist of a $3\times 3$
convolutional layer and a tanh activation layer. In addition, the $\psi_{n}$
is able to learn the residual feature of a generated image at a finer scale.

###  2.2 Textual-perceiving Discriminator

A textual-perceiving module in the discriminator is devised to capture the
spatially invariant representation of a normal image via directional
convolutions, making it more sensitive to defective areas. Specifically, the
textual-perceiving discriminator consists of the several conv-blocks $f_{n}$,
the textual-perceiving module [13] ($c_{n}$, $g_{n}$) and a sigmoid activation
layer. As shown in Fig. 1, the several conv-blocks $f_{n}$ are used to extract
preliminary features $F_{n}$ from the generated image. Then, the $c_{n}$ and
$g_{n}$ are designed to capture the spatially invariant representation of
normal image via eight directional texture features under the guidance of
corresponding directional feature map $T_{i}$. Specifically,

$P_{n}=c_{n}\left(\operatorname{cat}\left(f_{n}(\tilde{x}_{n}),T_{i}\right)\right)(i=1,2\cdots
8),$ (2)

where $c_{n}$ denotes several convolutions which contains a $3\times 3$
convolution layer with a BatchNorm and a ReLU activation layer, $cat$ denotes
the concatenation operation. In our design, total eight directions are
adopted, including top, bottom, left, right, top left, bottom left, top right,
and bottom right. Each directional map $T_{i}$ is a generated trend square
matrix that decreases from 1 to 0 in a certain direction. Finally, the output
features of different branches are concatenated to form the whole spatial
invariant feature $M_{n}$, that is,

$M_{n}=Sigmoid(g_{n}\left(\operatorname{cat}\left(P_{1},P_{2}\cdots
P_{8}\right)\right)),$ (3)

where $g_{n}$ represents a set of standard convolution block,
$M_{n}\in\mathbb{R}^{1\times H\times W}$ is the distinguish map that is used
to calculate the loss of discriminator. Since the proposed directional
convolution unit is sensitive to local variations of the image along each
direction, it can make the network well adaptable to spatial distortions and
scale variations. Intuitively, the discriminator of a well-trained GAN
memories the patch distribution of a normal image. So it should be insensitive
to the normal regions but varies drastically in the abnormal regions.

###  2.3 Hierarchical Fusion for Surface Inspection

In this section, we introduce a hierarchical fusion strategy to fuse the
multi-scale distinguish maps for producing the final segmented result.

Inspired by [5], the information entropy is a suitable metric to represent the
output of discriminator for abnormal region segmentation. Thus, the
information entropy of multi-scale distinguish maps are expressed as
following:

$H_{n}=M_{n}*\log M_{n},$ (4)

$H=\\{H_{n}\\}|_{n=0}^{N}$ are a set of information entropy of multi-scale
distinguish maps and we refer to $H$ as the coarse inspection map set, which
reveals the coarse abnormal regions of the input image.

The GANs have small receptive fields and limited capacity, preventing them
from representing the single image. To capture global textual structure and
fine-grained texture information, we fuse the multi-scale coarse inspection
maps together to better distinguish the abnormal regions. Thus, the fusion
process expresses as following:

$R=\sum_{n=0}^{N}\alpha_{n}*H_{n},$ (5)

where $R$ denotes the final fusion and $\alpha_{n}=\frac{1}{N+1}$ are
weighting factors.

##  3 experiments

To evaluate the effectiveness of HTP-GAN in one-shot surface inspection task,
we design extensive experiments based on WOOD Defect Database (WOOD) [14] and
Road Crack Database (CRACK) [15]. Furthermore, we compare our method with four
representative methods as following: (1) Unsupervised visual surface
inspection method proposed in [5] (ICASSP 2018); (2) Surface defect detection
method based on positive samples and artificial defects [16] (Prical 2018);
(3) Automated surface inspection method proposed in [17] (JOIM 2019); (4)
Semantic image segmentation method proposed in [18] (ECCV 2018).

###  3.1  Datasets and Evaluation Metrics

Dataset and Setting: WOOD Defect Database (WOOD) and Road Crack Database
(CRACK) are two surface inspection datasets which are annotated with
segmentation labels of abnormal regions [5]. For our HTP-GAN, a single image
with normal surface is adopted as the training data, which is the embodiment
of the one-shot surface inspection. To compare our method with unsupervised
methods: [5] and [16], we adopt the same image with normal surface to train
the models. For supervised methods: [17] and [18], several abnormal samples
and normal samples are used to train the networks. All images are resized to
256 $\times$ 256 size.

Evaluation Metrics: We adopt two evaluation metrics to compare and analyze
experimental results: Intersection-over-union (IOU) and Pixel Accuracy (pixel
acc).

###  3.2 Comparison with the State-of-the-art Approaches

Fig. 2: Visualization comparison with the state-of-the-art methods for the surface inspection task.  Table 1: Quantitative comparisons. (IoU(%) / pixel acc(%)) | [5] | [18] | [17] | Ours
---|---|---|---|---
WOOD | 44.83/81.96 | 47.23/94.45 | 47.37/93.82 | 59.83/96.54
CRACK | 31.06/58.21 | 48.68/90.46 | 33.02/61.56 | 56.97/96.32

Tab. 1 shows the performance comparison of our HTP-GAN against the other four
methods in terms of IoU and pixel acc on the WOOD and CRACK dataset. As we can
see in Tab. 1, our method achieved higher IoU and pixel acc than the four
methods.

In the one-shot setting, when the single normal image or an image pair (a
normal image and an abnormal image) is given as training data, all four
methods can not predict any abnormal regions or produce a noise map. This
result demonstrates that these methods rely on a tremendous amount of training
data and these models are easy to cause overfitting on one-shot surface
inspection task. Thus, we try to add the training data on these methods in the
supervised learning methods or adopt the fully convolution network to improve
the unsupervised learning methods [1].

Fig. 2 shows the visualization of different methods for surface inspection.
[17] has a good performance until we adopt 10 normal-abnormal image pairs for
training while [18] performs well on 4 abnormal images. Although improved [5]
detects the more accurate abnormal surface regions than other methods, our
proposed model has a better performance. More importantly, HTP-GAN only needs
a normal image as training data and achieves the best performance.

###  3.3 Ablation Studies

Fig. 3: Visualization of influence of different scales and textual-perceiving
module in our proposed HTP-GAN. “TM” refers to the textual-perceiving module.

Effectiveness of Scale Number. The number of scales in HTP-GAN architecture
has a strong influence on the results. A small number of scales only can
capture the local textures, leading to poor segmentation results. As the
number of scales increases, Fig. 3 (a)-(f) demonstrates that our proposed
method manages to capture larger structures as well as the fine-grained
representation of the image. It indicates that a strong representation can
help the model to segment most of the anomalous regions very well.

Effectiveness of Textual-perceiving Module. As shown in Fig. 3 (g) and (f), by
adding the textual-perceiving module, our method segments more abnormal
regions and alleviates to split normal regions into defects. It demonstrates
that the proposed textual-perceiving module can learn spatially invariant
representation of normal images and be more sensitive to defective areas.

Influence of Different Image Variations. Our proposed method can handle most
variations (i.e.Translation, Mirror, Scaling). However, for rotation
variation, when the training image contains mostly horizontal textures (such
as the wood in Fig. 3, a 45-degree change in rotation only brings a slight
performance drop 3% in term of IOU score, and no performance drop in term of
pixel accuracy.

##  4 Conclusion

In this paper, we propose a hierarchical texture-perceiving generative
adversarial network (HTP-GAN) to achieve the one-shot surface inspection task.
By applying a pyramid of convolutional GANs, HTP-GAN can indeed learn the
global and fine-grained representation of normal surface from a single image.
Thus, it enables discriminators in well-trained HTP-GAN to exhibit more active
responses for the defective regions. Furthermore, the texture-perceiving
module is devised to capture the spatially invariant representation of normal,
making the discriminator more sensitive to defective areas. More importantly,
the training of the HTP-GAN only relies on a single image, which explores a
more general and practical solution to the surface inspection task.

## Acknowledgments

This work was supported by the Key Laboratory of Intelligent Infrared
Perception, Chinese Academy of Sciences.

## References

  * [1] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli,  “Singan: Learning a generative model from a single natural image,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 4570–4580.
  * [2] Kecheng Zheng, Wu Liu, Jiawei Liu, Zheng-Jun Zha, and Tao Mei,  “Hierarchical gumbel attention network for text-based person search,”  in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3441–3449.
  * [3] Kecheng Zheng, Wu Liu, Lingxiao He, Tao Mei, Jiebo Luo, and Zheng-Jun Zha,  “Group-aware label transfer for domain adaptive person re-identification,”  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [4] Kecheng Zheng, Cuiling Lan, Wenjun Zeng, Zhizheng Zhan, and Zheng-Jun Zha,  “Exploiting sample uncertainty for domain adaptive person re-identification,”  Association for the Advancement of Artificial Intelligence, 2021\.
  * [5] W. Zhai, J. Zhu, Y. Cao, and Z. Wang,  “A generative adversarial network based framework for unsupervised visual surface inspection,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 1283–1287.
  * [6] Choon-Woo Kim and Antti J Koivo,  “Hierarchical classification of surface defects on dusty wood boards,”  Pattern Recognition Letters, vol. 15, no. 7, pp. 713–721, 1994\.
  * [7] W Wen and Aihua Xia,  “Verifying edges for visual inspection purposes,”  Pattern recognition letters, vol. 20, no. 3, pp. 315–328, 1999\.
  * [8] R. Liu, Q. Gu, X. Wang, and M. Yao,  “Region-convolutional neural network for detecting capsule surface defects,”  Boletin Tecnico/Technical Bulletin, vol. 55, no. 3, pp. 92–100, 2017.
  * [9] Kecheng Zheng, Zheng-Jun Zha, and Wei Wei,  “Abstract reasoning with distracting features,”  in Advances in Neural Information Processing Systems, 2019, pp. 5842–5853.
  * [10] Zhiyang Yu, Xiaojun Wu, and Xiaodong Gu,  “Fully convolutional networks for surface defect inspection in industrial environment,”  in International Conference on Computer Vision Systems. Springer, 2017, pp. 417–426.
  * [11] Igor Jovančević, Huy-Hieu Pham, Jean-José Orteu, Rémi Gilblas, Jacques Harvent, Xavier Maurice, and Ludovic Brèthes,  “3d point cloud analysis for detection and characterization of defects on airplane exterior surface,”  Journal of Nondestructive Evaluation, vol. 36, no. 4, pp. 74, 2017\.
  * [12] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han,  “Differentiable augmentation for data-efficient gan training,”  Advances in Neural Information Processing Systems, vol. 33, 2020\.
  * [13] Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao,  “Oriented response networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 519–528.
  * [14] Olli Silvén, Matti Niskanen, and Hannu Kauppinen,  “Wood inspection with non-supervised clustering,”  Machine Vision and Applications, vol. 13, no. 5-6, pp. 275–285, 2003.
  * [15] Henrique Oliveira and Paulo Lobato Correia,  “Crackit—an image processing toolbox for crack detection and characterization,”  in IEEE International Conference on Image Processing. IEEE, 2014, pp. 798–802.
  * [16] Zhixuan Zhao, Bo Li, Rong Dong, and Peng Zhao,  “A surface defect detection method based on positive samples,”  in Pacific Rim International Conference on Artificial Intelligence. Springer, 2018, pp. 473–481.
  * [17] Domen Tabernik, Samo Šela, Jure Skvarč, and Danijel Skočaj,  “Segmentation-based deep-learning approach for surface-defect detection,”  Journal of Intelligent Manufacturing, vol. 31, no. 3, pp. 759–776, 2020.
  * [18] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam,  “Encoder-decoder with atrous separable convolution for semantic image segmentation,”  in Proceedings of the European Conference on Computer Vision, 2018, pp. 801–818.


# MedNLI Is Not Immune:
Natural Language Inference Artifacts in the Clinical Domain

Christine Herlihy
Department of Computer Science
University of Maryland
College Park, MD
<EMAIL_ADDRESS>
&Rachel Rudinger
Department of Computer Science
University of Maryland
College Park, MD
<EMAIL_ADDRESS>

###### Abstract

Crowdworker-constructed natural language inference (NLI) datasets have been
found to contain statistical artifacts associated with the annotation process
that allow hypothesis-only classifiers to achieve better-than-random
performance (Poliak et al., 2018; Gururangan et al., 2018; Tsuchiya, 2018). We
investigate whether MedNLI, a physician-annotated dataset with premises
extracted from clinical notes, contains such artifacts (Romanov and Shivade,
2018).

We find that entailed hypotheses contain generic versions of specific concepts
in the premise, as well as modifiers related to responsiveness, duration, and
probability. Neutral hypotheses feature conditions and behaviors that co-occur
with, or cause, the condition(s) in the premise. Contradiction hypotheses
feature explicit negation of the premise and implicit negation via assertion
of good health. Adversarial filtering demonstrates that performance degrades
when evaluated on the _difficult_ subset. We provide partition information and
recommendations for alternative dataset construction strategies for knowledge-
intensive domains.

##  1 Introduction

In the clinical domain, the ability to conduct natural language inference
(NLI) on unstructured, domain-specific texts such as patient notes, pathology
reports, and scientific papers, plays a critical role in the development of
predictive models and clinical decision support (CDS) systems.

Considerable progress in domain-agnostic NLI has been facilitated by the
development of large-scale, crowdworker-constructed datasets, including the
Stanford Natural Language Inference corpus (SNLI), and the Multi-Genre Natural
Language Inference (MultiNLI) corpus (Bowman et al., 2015; Williams et al.,
2017). MedNLI is a similarly-motivated, healthcare-specific dataset created by
a small team of physician-annotators in lieu of crowdworkers, due to the
extensive domain expertise required (Romanov and Shivade, 2018).

Poliak et al. (2018), Gururangan et al. (2018), Tsuchiya (2018), and McCoy et
al. (2019) empirically demonstrate that SNLI and MultiNLI contain lexical and
syntactic annotation artifacts that are disproportionately associated with
specific classes, allowing a hypothesis-only classifier to significantly
outperform a majority-class baseline model. The presence of such artifacts is
hypothesized to be partially attributable to the priming effect of the example
hypotheses provided to crowdworkers at annotation-time. Romanov and Shivade
(2018) note that a hypothesis-only baseline is able to outperform a majority
class baseline in MedNLI, but they do not identify specific artifacts.

We confirm the presence of annotation artifacts in MedNLI and proceed to
identify their lexical and semantic characteristics. We then conduct
adversarial filtering to partition MedNLI into _easy_ and _difficult_ subsets
(Sakaguchi et al., 2020). We find that performance of off-the-shelf fastText-
based hypothesis-only and hypothesis-plus-premise classifiers is lower on the
_difficult_ subset than on the _full_ and _easy_ subsets (Joulin et al.,
2016). We provide partition information for downstream use, and conclude by
advocating alternative dataset construction strategies for knowledge-intensive
domains.111See https://github.com/crherlihy/clinical_nli_artifacts for code
and partition ids.

##  2 The MedNLI Dataset

MedNLI is domain-specific evaluation dataset inspired by general-purpose NLI
datasets, including SNLI and MultiNLI (Romanov and Shivade, 2018; Bowman et
al., 2015; Williams et al., 2017). Much like its predecessors, MedNLI consists
of premise-hypothesis pairs, in which the premises are drawn from the Past
Medical History sections of a randomly selected subset of de-identified
clinical notes contained in MIMIC-III (Johnson et al., 2016; Goldberger et
al., 2000 (June 13). MIMIC-III was created from the records of adult and
neonatal intensive care unit (ICU) patients. As such, complex and clinically
severe cases are disproportionately represented, relative to their frequency
of occurrence in the general population.

Physician-annotators were asked to write a definitely true, maybe true, and
definitely false set of hypotheses for each premise, corresponding to
entailment, neutral and contradiction labels, respectively. The resulting
dataset has cardinality: $n_{\text{train}}=11232;\ n_{\text{dev}}=1395;\
n_{\text{test}}=1422$.

##  3 MedNLI Contains Artifacts

To determine whether MedNLI contains annotation artifacts that may
artificially inflate the performance of models trained on this dataset, we
train a simple, premise-unaware, fastText classifier to predict the label of
each premise-hypothesis pair, and compare the performance of this classifier
to a majority-class baseline, in which all training examples are mapped to the
most commonly occurring class label (Joulin et al., 2016; Poliak et al., 2018;
Gururangan et al., 2018). Note that since annotators were asked to create an
entailed, contradictory, and neutral hypothesis for each premise, MedNLI is
class-balanced. Thus, in this setting, a majority class baseline is equivalent
to choosing a label uniformly at random for each training example.

The micro F1-score achieved by the fastText classifier significantly exceeds
that of the majority class baseline, confirming the findings of Romanov and
Shivade (2018), who report a micro-F1 score of 61.9 but do not identify or
analyze artifacts:

| dev | test
---|---|---
majority class | 33.3 | 33.3
fastText | 64.8 | 62.6
Table 1: Performance (micro F1-score) of the fastText hypothesis-only
classifier.

As the confusion matrix for the test set shown in Table 2 indicates, the
fastText model is most likely to misclassify entailment as neutral, and
neutral and contradiction as entailment. Per-class precision and recall on the
test set are highest for contradiction (73.2; 72.8) and lowest for entailment
(56.7; 53.8).

| entailment | neutral | contradiction
---|---|---|---
entailment | 255 | 151 | 68
neutral | 126 | 290 | 58
contradiction | 69 | 60 | 345

Table 2: Confusion matrix for fastText classifier.

##  4 Characteristics of Clinical Artifacts

In this section, we conduct class-specific lexical analysis to identify the
clinical and domain-agnostic characteristics of annotation artifacts
associated with each set of hypotheses in MedNLI.

###  4.1 Preprocessing

We cast each hypothesis string in the MedNLI training dataset to lowercase. We
then use a scispaCy model pre-trained on the en_core_sci_lg corpus for
tokenization and clinical named entity recognition (CNER) (Neumann et al.,
2019a). One challenge associated with clinical text, and scientific text more
generally, is that semantically meaningful entities often consist of spans
rather than single tokens. To mitigate this issue during lexical analysis, we
map each multi-token entity to a single-token representation, where sub-tokens
are separated by underscores.

###  4.2 Lexical Artifacts

Following Gururangan et al. (2018), to identify tokens that occur
disproportionately in hypotheses associated with a specific class, we compute
token-class pointwise mutual information (PMI) with add-50 smoothing applied
to raw counts, and a filter to exclude tokens appearing less than five times
in the overall training dataset. Table 3 reports the top 15 tokens for each
class.

$\noindent\texttt{PMI}(\text{token, class})=log_{2}\frac{p(\text{token,
class})}{p(\text{token},\cdot)p(\cdot,\text{class})}$

entailment | % | neutral | % | contradiction | %
---|---|---|---|---|---
just | 0.25% | cardiogenic_shock | 0.33% | no_history_of_cancer | 0.27%
high_risk | 0.26% | pelvic_pain | 0.30% | no_treatment | 0.27%
pressors | 0.25% | joint_pain | 0.30% | normal_breathing | 0.27%
possible | 0.26% | brain_injury | 0.32% | no_history_of_falls | 0.27%
elevated_blood_pressure | 0.26% | delerium | 0.30% | normal_heart_rhythm | 0.28%
responsive | 0.25% | intracranial_pressure | 0.30% | health | 0.26%
comorbidities | 0.26% | smoking | 0.42% | normal_head_ct | 0.26%
spectrum | 0.27% | obesity | 0.41% | normal_vision | 0.26%
steroid_medication | 0.25% | tia | 0.32% | normal_aortic_valve | 0.27%
longer | 0.26% | acquired | 0.31% | bradycardic | 0.26%
history_of_cancer | 0.26% | head_injury | 0.31% | normal_blood_sugars | 0.27%
broad | 0.26% | twins | 0.30% | normal_creatinine | 0.28%
frequent | 0.25% | fertility | 0.30% | cancer_history | 0.26%
failed | 0.26% | statin | 0.30% | cardiac | 0.33%
medical | 0.29% | acute_stroke | 0.30% | normal_chest | 0.28%
Table 3: Top 15 tokens by PMI(token, class); % of _class_ training examples
that contain the token.

#### Entailment

Entailment hypotheses are characterized by tokens about: (1) patient status
and response to treatment (e.g., _responsive_ ; _failed_ ; _longer_ as in _no
longer intubated_); (2) medications and procedures which are common among ICU
patients (e.g., _broad_spectrum_ ; _antibiotics_ ; _pressors_ ;
_steroid_medication_ ; _underwent_ ; _removal_); (3) generalized versions of
specific words in the premise (e.g., _comorbidities_ ;
_multiple_medical_problems_), which Gururangan et al. (2018) also observe in
SNLI; and (4) modifiers related to duration, frequency, or probability (e.g.,
_frequent_ , _possible_ , _high_risk_).

#### Neutral

Neutral hypotheses feature tokens related to: (1) chronic and acute clinical
conditions (e.g., _obesity_ ; _joint_pain_ ; _brain_injury_); (2) clinically
relevant behaviors (e.g., _smoking_ ; _alcoholic_ ; _drug_overdose_); and (3)
gender and reproductive status (e.g., _fertility_ ; _pre_menopausal_).
Notably, the most discriminative conditions tend to be commonly occurring
within the general population and generically stated, rather than rare and
specific. This presumably contributes to the relative difficulty that the
hypothesis-only fastText model has distinguishing between the entailment and
neutral classes.

#### Contradiction

Contradiction hypotheses are characterized by tokens that convey normalcy and
good health. Lexically, such sentiment manifests as: (1) explicit negation of
clinical severity, medical history, or in-patient status (e.g., _denies_pain_
; _no_treatment_ ; _discharged_home_), or (2) affirmation of clinically
unremarkable findings (e.g., _normal_heart_rhythm_ ; _normal_blood_sugars_),
which would generally be rare among ICU patients. This suggests a heuristic of
inserting negation token(s) to contradict the premise, which Gururangan et al.
(2018) also observe in SNLI.

###  4.3 Syntactic Artifacts

#### Hypothesis Length

In contrast to Gururangan et al. (2018)’s finding that entailed hypotheses in
SNLI tend to be shorter while neutral hypotheses tend to be longer, hypothesis
sentence length does not appear to play a discriminatory role in MedNLI,
regardless of whether we consider merged- or separated-token representations
of multi-word entities, as illustrated by Table 4:

| entailment | neutral | contradiction
---|---|---|---
| mean | median | mean | median | mean | median
separate | 5.6 | 5.0 | 5.2 | 5.0 | 5.6 | 5.0
merged | 5.3 | 5.0 | 4.9 | 5.0 | 5.3 | 5.0

Table 4: Average and median hypothesis length by class and entity
representation.

##  5 Physician-Annotator Heuristics

In this section, we re-introduce premises to our analysis to evaluate a set of
hypotheses regarding latent, class-specific annotator heuristics. If
annotators _do_ employ class-specific heuristics, we should expect the
semantic contents, $\upvarphi$, of a given hypothesis, $h\in\mathcal{H}$, to
be influenced not only by the semantic contents of its associated premise,
$p\in\mathcal{P}$, but also by the target class, $c\in\mathcal{C}$.

To investigate, we identify a set of heuristics parameterized by
$\upvarphi(p)$ and $c$, and characterized by the presence of a set of
heuristic-specific Medical Subject Headings (MeSH) linked entities in the
premise and hypothesis of each heuristic-satisfying example. These heuristics
are described below; specific MeSH features are detailed in the Appendix.

#### Hypernym Heuristic

This heuristic applies when the premise contains clinical condition(s),
medication(s), finding(s), procedure(s) or event(s), the target class is
_entailment_ , and the generated hypothesis contains term(s) that can be
interpreted as super-types for a subset of elements in the premise (e.g.,
clindamycin <: antibiotic).

#### Probable Cause Heuristic

This heuristic applies when the premise contains clinical condition(s), the
target class is _neutral_ , and the generated hypothesis provides a plausible,
often subjective or behavioral, causal explanation for the condition, finding,
or event described in the premise (e.g., associating altered mental status
with drug overdose).

#### Everything Is Fine Heuristic

This heuristic applies when the premise contains condition(s) or finding(s),
the target class is _contradiction_ , and the generated hypothesis negates the
premise or asserts unremarkable finding(s). This can take two forms:
repetition of premise content plus negation, or inclusion of modifiers that
convey good health.

#### Analysis

We conduct a $\chi^{2}$ test for each heuristic to determine whether we are
able to reject the null hypothesis that pattern-satisfying premise-hypothesis
pairs are uniformly distributed over classes.

heuristic | $\chi^{2}$ | p-value | top class
---|---|---|---
hypernym | 59.15 |  $1.410-13$$\ddagger$ | entail (45.2%)
probable cause | 111.05 |  $7.710-25$$\ddagger$ | neutral (57.8%)
everything fine | 874.71 |  $1.110-190$$\ddagger$ | contradict (83.8%)
Table 5: Results of $\chi^{2}$ test statistic by heuristic, computed using the
combined MedNLI dataset ($\ddagger$ $p<0.001$, $\dagger$ $p<0.01$, * $p<0.5$).
Top class presented with % of heuristic-satisfying pairs.

The results support our hypotheses regarding each of the three heuristics.
Notably, the percentage of heuristic-satisfying pairs accounted for by the top
class is lowest for the hypernym hypothesis, which we attribute to the high
degree of semantic overlap between entailed and neutral hypotheses.

##  6 Adversarial Filtering

To mitigate the effect of clinical annotation artifacts, we employ AFLite, an
adversarial filtering algorithm introduced by Sakaguchi et al. (2020) and
analyzed by Bras et al. (2020), to create _easy_ and _difficult_ partitions of
MedNLI.

AFLite requires distributed representations of the full dataset as input, and
proceeds in an iterative fashion. At each iteration, an ensemble of $n$ linear
classifiers are trained and evaluated on different random subsets of the data.
A score is then computed for each premise-hypothesis instance, reflecting the
number of times the instance is correctly labeled by a classifier, divided by
the number of times the instance appears in any classifier’s evaluation set.
The top-$k$ instances with scores above a threshold, $\tau$, are filtered out
and added to the _easy_ partition; the remaining instances are retained. This
process continues until the size of the filtered subset is $<k$, or the number
of retained instances is $<m$; retained instances constitute the _difficult_
partition.

To represent the full dataset, we use $\texttt{fastText}_{\text{MIMIC-III}}$
embeddings, which have been pretrained on deidentified patient notes from
MIMIC-III (Romanov and Shivade, 2018; Johnson et al., 2016). We represent each
example as the average of its component token vectors.

We proportionally adjust a subset of the hyperparameters used by Sakaguchi et
al. (2020) to account for the fact that MedNLI contains far fewer examples
than Winogrande222MedNLI’s training dataset contains $14049$ examples when the
training, dev, and test sets are combined, while Winogrande contains $47$K
after excluding the $6$K used for fine-tuning.: specifically, we set the
training size for each ensemble, $m$, to $5620$, which represents
$\approx\frac{2}{5}$ of the MedNLI combined dataset. The remaining
hyperparameters are unchanged: the ensemble consists of $n=64$ logistic
regression models, the filtering cutoff, $k$ = 500, and the filtering
threshold $\tau=0.75$.

We apply AFLite to two different versions of MedNLI: (1) $\mathcal{X}_{h,m}$:
hypothesis-only, multi-token entities merged, and (2) $\mathcal{X}_{ph,m}$:
premise and hypothesis concatenated, multi-token entities merged. AFLIte maps
each version to an _easy_ and _difficult_ partition, which can in turn be
split into training, dev, and test subsets. We report results for the fastText
classifier trained on the original, hypothesis-only (hypothesis + premise)
MedNLI training set, and evaluated on the _full_ , _easy_ and _difficult_ dev
and test subsets of $\mathcal{X}_{h,m}$ ($\mathcal{X}_{ph,m}$), and observe
that performance decreases on the _difficult_ partition:

| model | eval dataset | full |  easy ($\Delta$) |  difficult ($\Delta$)
---|---|---|---|---|---
no premise | majority class | dev | 0.33 | 0.34 (+0.01) | 0.35 (+0.02)
no premise | majority class | test | 0.33 | 0.35 (+0.02) | 0.37 (+0.04)
no premise | fastText | dev | 0.65 | 0.67 (+0.02) | 0.46 (-0.19)
no premise | fastText | test | 0.63 | 0.65 (+0.02) | 0.4 (-0.23)
with premise | majority class | dev | 0.33 | 0.45 (+0.12) | 0.36 (+0.03)
with premise | majority class | test | 0.33 | 0.48 (+0.15) | 0.37 (+0.04)
with premise | fastText | dev | 0.53 | 0.6 (+0.07) | 0.43 (-0.1)
with premise | fastText | test | 0.51 | 0.55 (+0.04) | 0.4 (-0.11)

Table 6: Performance (micro F1-score) for the majority class baseline and
fastText classifiers, with and without premise, by partition (e.g., _full,
easy, difficult_).

##  7 Discussion

###  7.1 MedNLI is Not Immune from Artifacts

In this paper, we demonstrate that MedNLI suffers from the same challenge
associated with annotation artifacts that its domain-agnostic predecessors
have encountered: namely, NLI models trained on $\\{$Med, S, Multi$\\}$NLI can
perform well even without access to the training examples’ premises,
indicating that they often exploit shallow heuristics, with negative
implications for out-of-sample generalization.

Interestingly, many of the high-level lexical characteristics identified in
MedNLI can be considered domain-specific variants of the more generic, class-
specific patterns identified in SNLI. This observation suggests that a set of
abstract design patterns for inference example generation exists across
domains, and may be reinforced by the prompts provided to annotators. Creative
or randomized priming, such as Sakaguchi et al. (2020) ’s use of anchor words
from WikiHow articles, may help to decrease reliance on such design patterns,
but it appears unlikely that they can be systematically sidestepped without
introducing new, “corrective” artifacts.

###  7.2 A Prescription for Dataset Construction

To mitigate the risk of performance overestimation associated with annotation
artifacts, Zellers et al. (2019) advocate adversarial dataset construction,
such that benchmarks will co-evolve with language models. This may be
difficult to scale in knowledge-intensive domains, as expert validation of
adversarially generated benchmarks is typically required. Additionally, in
high-stakes domains such as medicine, information-rich inferences should be
preferred over correct but trivial inferences that time-constrained expert
annotators may be rationally incentivized to produce, because entropy-reducing
inferences are more useful for downstream tasks.

We advocate the adoption of a mechanism design perspective, so as to develop
modified annotation tasks that reduce the cognitive load placed on expert
annotators while incentivizing the production of domain-specific NLI datasets
with high downstream utility (Ho et al., 2015; Liu and Chen, 2017). An
additional option is to narrow the generative scope by defining a set of
inferences deemed to be useful for a specific task. Annotators can then map
(premise, relation) tuples to relation-satisfying, potentially fuzzy subsets
of this pool of useful inferences, or return partial functions when more
information is needed.

##  8 Ethical Considerations

When working with clinical data, two key ethical objectives include: (1) the
preservation of patient privacy, and (2) the development of language and
predictive models that benefit patients and providers to the extent possible,
without causing undue harm. With respect to the former, MedNLI’s premises are
sampled from de-identified clinical notes contained in MIMIC-III Goldberger et
al. (2000 (June 13); Johnson et al. (2016), and the hypotheses generated by
annotators do not refer to specific patients, providers, or locations by name.
MedNLI requires users to complete Health Insurance Portability and
Accountability Act (HIPAA) training and sign a data use agreement prior to
being granted access, which we have complied with.

Per MedNLI’s data use agreement requirements, we do not attempt to identify
any patient, provider, or institution mentioned in the de-identified corpus.
Additionally, while we provide AFLite _easy_ and _difficult_ partition
information for community use in the form of split-example ids and a checksum,
we do not share the premise or hypothesis text associated with any example.
Interested readers are encouraged to complete the necessary training and
obtain credentials so that they can access the complete dataset Romanov and
Shivade (2018); Goldberger et al. (2000 (June 13).

With respect to benefiting patients, the discussion of natural language
artifacts we have presented is intended to encourage clinical researchers who
rely on (or construct) expert-annotated clinical corpora to train domain-
specific language models, or consume such models to perform downstream tasks,
to be aware of the presence of annotation artifacts, and adjust their
assessments of model performance accordingly. It is our hope that these
findings can be used to inform error analysis and improve predictive models
that inform patient care.

## Acknowledgments

We thank the four anonymous reviewers whose feedback and suggestions helped
improve this manuscript. The first author was supported by the National
Institute of Standards and Technology’s (NIST) Professional Research
Experience Program (PREP). This research was also supported by the DARPA
KAIROS program. The views and conclusions contained in this publication are
those of the authors and should not be interpreted as representing official
policies or endorsements of NIST, DARPA, or the U.S. Government.

## References

  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bras et al. (2020) Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, and Yejin Choi. 2020.  Adversarial filters of dataset biases.
  * Goldberger et al. (2000 (June 13) A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. 2000 (June 13).  PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals.  _Circulation_ , 101(23):e215–e220.  Circulation Electronic Pages: http://circ.ahajournals.org/content/101/23/e215.full PMID:1085218; doi: 10.1161/01.CIR.101.23.e215.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ho et al. (2015) Chien-Ju Ho, Aleksandrs Slivkins, Siddharth Suri, and Jennifer Wortman Vaughan. 2015\.  Incentivizing high quality crowdwork.  In _Proceedings of the 24th International Conference on World Wide Web_ , WWW ’15, page 419–429, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  MIMIC-III, a freely accessible critical care database.  _Scientific data_ , 3:160035.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_.
  * Liu and Chen (2017) Yang Liu and Yiling Chen. 2017.  Machine-learning aided peer prediction.  In _Proceedings of the 2017 ACM Conference on Economics and Computation_ , EC ’17, page 63–80, New York, NY, USA. Association for Computing Machinery.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Neumann et al. (2019a) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019a.  ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 319–327, Florence, Italy. Association for Computational Linguistics.
  * Neumann et al. (2019b) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019b.  ScispaCy: Fast and robust models for biomedical natural language processing.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 319–327, Florence, Italy. Association for Computational Linguistics.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Romanov and Shivade (2018) Alexey Romanov and Chaitanya Shivade. 2018.  Lessons from Natural Language Inference in the Clinical Domain.  _CoRR_ , abs/1808.06752.
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  Winogrande: An adversarial winograd schema challenge at scale.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8732–8740.
  * Tsuchiya (2018) Masatoshi Tsuchiya. 2018.  Performance impact caused by hidden bias of training data for recognizing textual entailment.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  _CoRR_ , abs/1704.05426.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Hypothesis-only Baseline Analysis

To conduct the analysis presented in Section 3, we take the MedNLI training
dataset as input, and exclude the premise text for each training example. We
cast the text of each training hypothesis to lowercase, but do not perform any
additional preprocessing. We use an off-the-shelf fastText classifier, with
all model hyperparameters set to their default values with the exception of
wordNgrams, which we set equal to 2 to allow the model to use bigrams in
addition to unigrams Joulin et al. (2016). We evaluate the trained classifier
on the hypotheses contained in the MedNLI dev and test datasets, and report
results for each split.

###  A.2 Lexical Artifact Analysis

To perform the analysis presented in Section 4, we cast each hypothesis string
in the MedNLI training dataset to lowercase. We then use a scispaCy model pre-
trained on the en_core_sci_lg corpus for tokenization and clinical named
entity recognition (CNER) (Neumann et al., 2019a). Next, we merge multi-token
entities, using underscores as delimiters—e.g., “brain injury” $\rightarrow$
“brain_injury”.

When computing token-class pointwise mutual information (PMI), we exclude
tokens that appear less than five times in the overall training dataset’s
hypotheses. Then, following Gururangan et al. (2018), who apply add-100
smoothing to raw counts to highlight particularly discriminative token-class
co-occurrence patterns, we apply add-50 smoothing to raw counts. Our approach
is similarly motivated; our choice of 50 reflects the smaller state space
associated with a focus on the clinical domain.

###  A.3 Semantic Analysis of Heuristics

To perform the statistical analysis presented in Section 5, we take the
premise-hypothesis pairs from the MedNLI training, dev, and test splits, and
combine them to produce a single corpus. We use a scispaCy model pre-trained
on the en_core_sci_lg corpus for tokenization and entity linking Neumann et
al. (2019b), and link against the Medical Subject Headings (MeSH) knowledge
base. We take the top-ranked knowledge base entry for each linked entity.
Linking against MeSH provides a unique concept id, canonical name, alias(es),
a definition, and one or more MeSH tree numbers for each recovered entity.
Tree numbers convey semantic type information by embedding each concept into
the broader MeSH hierarchy 333https://meshb.nlm.nih.gov/treeView. We
operationalize each of our heuristics with a set of MeSH-informed semantic
properties, which are defined as follows:

  1. 1.

Hypernym Heuristic: a premise-hypothesis pair satisfies this heuristic if
specific clinical concept(s) appearing in the premise appear in a more general
form in the hypothesis. Formally:
$\\{(p,h)|\upvarphi(p)\subsetneq\upvarphi(h)\\}$. MeSH tree numbers are
organized hierarchically, and increase in length with specificity. Thus, when
a premise entity and hypothesis entity are left-aligned, the hypothesis entity
is a hypernym for the premise entity if the hypothesis entity is a substring
of the premise entity. To provide a concrete example: _diabetes mellitus_ is
an _endocrine system disease_ ; the associated MeSH tree numbers are C19.246
and C19, respectively.

  2. 2.

Probable Cause Heuristic: a premise-hypothesis pair satisfies this heuristic
if: (1) the premise contains one or more MeSH entities belonging to high-level
categories C (diseases), D (chemicals and drugs), E (analytical, diagnostic
and therapeutic techniques, and equipment) or F (psychiatry and psychology);
and (2) the hypothesis contains one or more MeSH entities that can be
interpreted as providing a plausible causal or behavioral explanation for the
condition, finding, or event described in the premise (e.g., smoking,
substance-related disorders, mental disorders, alcoholism, homelessness,
obesity).

  3. 3.

Everything Is Fine Heuristic: a premise-hypothesis pair satisfies this
heuristic if the hypothesis contains one or more of the same MeSH entities as
the premise (excluding the _patient_ entity, which appears in almost all
notes) and also contains: (1) a negation word or phrase (e.g., _does not have_
, _no finding_ , _no_ , _denies_); or (2) a word or phrase that affirms the
patient’s health (e.g., _normal_ , _healthy_ , _discharged_).

For each heuristic, we subset the complete dataset to find pattern-satisfying
premise-heuristic pairs. We use this subset when performing the $\chi^{2}$
tests.

###  A.4 Adversarial Filtering

When implementing AFLite, we follow Sakaguchi et al. (2020). We use a smaller
training set size of $m=5620$, but keep the remaining hyperparameters
unchanged, such that the ensemble consists of $n=64$ logistic regression
models, the filtering cutoff, $k=500$, and the filtering threshold
$\tau=0.75$.


EFI 24-4

arXiv:2406.05920

Possible mixing of a diquark-antidiquark with a $p\bar{p}$ hadronic molecule

Marek<EMAIL_ADDRESS>and Jonathan L.
<EMAIL_ADDRESS>
a School of Physics and Astronomy

Tel Aviv University, Tel Aviv 69978, Israel

b Enrico Fermi Institute and Department of Physics

University of Chicago, 5640 S. Ellis Avenue, Chicago, IL 60637, USA

ABSTRACT

> We discuss the possibility that the two nearby resonances observed by BESIII
> partially below the $p\bar{p}$ threshold might be due to mixing between two
> metastable states with the same $J^{PC}=0^{-+}$ quantum numbers, but rather
> different internal structure. One is a $p\bar{p}$ hadronic molecule and the
> other a bound state of a light-quark diquark and an antidiquark, both with
> spin 1 and isospin 0, a composite color antitriplet and triplet,
> respectively. The doubling of resonances, one of which may be interpreted as
> a hadronic molecule, while the other arises from $q\bar{q}$ annihilation in
> a state with vacuum quantum numbers may be a more general feature than the
> specific case considered here.

PACS codes: 12.38.Aw,12.39.Jh,14.40.Rt,36.10.Gv

##  1 Introduction

Ref. [1] reports an anomalous line shape of the $X(1840)$ resonance observed
in the invariant mass recoiling against the photon in the decay
$\hbox{$J\kern-1.49994pt/\kern-1.49994pt\psi\kern
1.49994pt$}\rightarrow\gamma\,3(\pi^{+}\pi^{-})$. A significant distortion at
1.84 GeV in the line-shape of the $3(\pi^{+}\pi^{-})$ invariant mass spectrum
is observed for the first time, which could be resolved by two overlapping
resonant structures, $X(1840)$ and $X(1880)$. The new state $X(1880)$ is
observed with a statistical significance larger than $10\sigma$. The mass and
width of $X(1880)$ are determined to be $1882.1\pm 1.7\pm 0.7$ MeV and
$30.7\pm 5.5\pm 2.4$ MeV, respectively. The region of $\pm 15$ MeV around the
central value of 1882 MeV extends down to 1867 MeV, i.e., about 10 MeV below
the $p\bar{p}$ threshold at 1877 MeV. This might indicate the existence of a
$p\bar{p}$ bound state.

Our purpose in this note is to try to understand the physical interpretation
of the two overlapping resonances, $X(1840)$ and $X(1880)$. We suggest that
our interpretation may have a broader context, applying to general cases of
two-hadron molecules such as those discussed in Ref. [2]. In addition to the
states in $3(\pi^{+}\pi^{-})$ there is a broad state $X(1835)$ in the
$\eta^{\prime}\pi^{+}\pi^{-}$ channel [3] with mass
$M(X)=1826.5^{+15.0}_{-3.4}$ MeV, $\Gamma(X)=242^{+14}_{-15}$ MeV, whose
participation in the two-state interference we do not consider.

##  2 Quantum numbers

We begin by identifying the likely $J^{PC}$ quantum numbers of the two
resonances. Both $J\kern-1.49994pt/\kern-1.49994pt\psi\kern 1.49994pt$ and the
photon have charge-conjugation eigenvalue C-parity C=-1. Therefore both
$X(1840)$ and $X(1880)$ must have $C={+}1$. The $J^{PC}$ of $X(1840)$ is well-
established to be $0^{-+}$ [3]. If one interprets $X(1880)$ as a $p\bar{p}$
resonance, its charge parity is given by

$C_{p\bar{p}}=({-}1)^{L+S}$ (1)

where $L$ and $S$ are the orbital and spin angular momentum of the $p\bar{p}$
system, respectively. $X(1880)$ is very close to the $p\bar{p}$ threshold, so
$L$ must be zero, because orbital excitations cost significant energy. We have
$L+S=S=\hbox{even}$. For $p\bar{p}$ one can have $S=0\ \hbox{or}\ S=1$. It
follows that $S=0$ and therefore $J=0$. The parity of a $p\bar{p}$ system is
$(-1)^{L+1}={-}1$. So finally

$J^{PC}[X(1840)]=J^{PC}[X(1880)]=0^{-+}~{}.$ (2)

##  3 Two-state interpretation

If one interprets either $X(1840)$ or $X(1880)$ as a metastable $p\bar{p}$
bound state, a question arises immediately as to the physical nature of the
other state. The point is that if one of the two states is a hadronic
molecule, it is very hard to come up with a plausible scenario in which the
other state, split from the first by 40 MeV, is also a hadronic molecule. With
only light quarks and antiquarks in the game, it is hard to generate such a
large splitting between below-threshold states. We are therefore led to search
for some completely different configuration for the state which is not a
hadronic molecule, yet has the same quantum numbers and a similar mass. In
this search we rely on intuition motivated by another system, the
$\chi_{c1}(3872)$ meson, a.k.a. $X(3872)$. It has been noted (see, e.g., the
discussion in Ref. [4]) that the unusual properties of this meson can be
understood if one assumes that the physical $\chi_{c1}(3872)$ is a mixture of
two very different objects which have the same $J^{PC}=1^{++}$ quantum numbers
and happen to be close in mass: a $\bar{D}D^{*}$ hadronic molecule and a
$P$-wave charmonium, a.k.a. $\chi_{c1}(2P)$. (One estimate of the
$\chi_{c1}(2P)$ mass is 3920.5 MeV [5].)

We now argue that in the $X(1840)$-$X(1880)$ system the analogues are a
$p\bar{p}$ hadronic molecule and a $P$-wave excitation of a narrow diquark-
antidiquark system. The possibility of narrow high-mass exotic states has been
raised in [6, 7, 8]. There it was pointed out that narrow high-mass states can
arise despite large phase space when two nearly degenerate states are coupled
to the same dominant decay mode. A necessary condition for such a proposal is
that the $P$-wave diquark-antidiquark system must have the same quantum
numbers and a mass close to the $p\bar{p}$ threshold. Conservation of angular
momentum demands that the quark-antiquark pair which annihilate to leave the
diquark-antidiquark system must have the quantum numbers $S=1$, $L=1$, and
$J=0$, or ${}^{3}P_{0}$ in a flavor singlet. The role of production of a
flavor-singlet${}^{3}P_{0}$ pair in quarkonium decay has been recognized for
many years [9, 10, 11]. In the following we check the quantum numbers and
estimate the mass of the diquark-antidiquark ground state using the standard
toolbox of the nonrelativistic quark model.

Let us start from the quantum numbers. We have two spin-1 diquarks, which are
each other’s antiparticles: $ud$ (or $uu$) $\bar{u}\bar{d}$ (or
$\bar{u}\bar{u}$), obviously with isospin=1, a.k.a. “bad diquarks”. Their
color representations are $\boldmath\bar{3}_{c}$ and $\boldmath 3_{c}$,
respectively. It is clear that they can be combined to a color and isospin
singlet.

Next, we have to make sure that we can obtain a bound state with
$J^{PC}=0^{-+}$. The intrinsic parity of the two diquark system is ${+}1$. To
get a negative parity, we need to put them in a $P$-wave, i.e., $L=1$. Then to
get the total angular momentum $J=0$, we combine the two vectors to $S=1$, so
that $\vec{J}=\vec{L}+\vec{S}=0$. The consistency check is whether the charge
parity comes out right. For two spin-1 bosons with $S=1$ and $L=1$ we indeed
have $C=1$, as illustrated by, e.g., $\eta(1270)$ with $J^{PC}=0^{-+}$ which
has a $\rho^{+}\rho^{-}$ decay mode [3].

##  4 Masses

The calculation of the mass of the diquark-antidiquark system is
straightforward. The mass of each “bad” diquark is just the constituent mass
of the two light quarks (we ignore isospin breaking), plus color-hyperfine
repulsion in spin-1 state:

$M_{diq,S=1}=2m_{q}+a_{HF}=2\times 363+50=776\ \hbox{MeV}.$ (3)

The mass of the diquark-antidiquark state is just the sum of the masses of the
two diquarks, plus the cost of the $P$-wave excitation. The latter is
estimated using a method previously employed in Ref. [12]. The residual energy
difference $\Delta E_{R}$ after accounting for quark masses is found to be
approximately linear in reduced mass $\mu_{R}$ of the two-body system:

$\Delta E_{R}=417.37-0.2141\mu_{R}$ (4)

The reduced mass of the diquark-antidiquark system is $\mu_{R}=(1/2)(776)=388$
MeV, so $\Delta E_{R}=334$ MeV and we estimate the mass of the diquark-
antidiquark configuration with $J^{PC}=0^{-+}$:

$M_{diq-diq}=2\times 776+334=1886\ \hbox{MeV}.$ (5)

In order for the molecular proton-antiproton state to be near the threshold of
$2m_{p}=2(938.27)=1876.5$ MeV, we must identify it with the $X(1880)$, leaving
the $X(1840)$ to be the state predicted in Eq. (5). This is probably
acceptable, given the crude nature of the approximation (4). The opposite
order, with $X(1840)$ identified as the $p\bar{p}$ molecule and $X(1880)$
taken as the diquark-antidiquark candidate, implies an unexpectedly large
molecular binding energy.

##  5 Other systems

The doubling of resonances, one of which may be interpreted as a hadronic
molecule while the other arises from $q\bar{q}$ annihilation in a state with
vacuum quantum numbers, i.e., ${}^{3}P_{0}$, may be a more general feature
than the case considered here. One would then expect such resonances as
$f_{0}(980)$ and $D_{s}(2317)$ (see Table I of Ref. [2]) to be accompanied by
a partner at a nearby mass.

###  5.1 $f_{0}(980)$

The $f_{0}(980)$ is a spin-zero isospin-zero resonance with couplings to
$\pi\pi$, $K\bar{K}$, $u\bar{u}$, $d\bar{d}$, $s\bar{s}$, $\eta\pi\pi$,
nucleon-antinucleon, $\gamma\gamma$, and tetraquark configurations. These
final states are sorted out in an extensive section of the Particle Data
Group’s Review of Particle Physics [13]. Analogy with the doubled-resonance
picture would regard it as a combination of a $K\bar{K}$ molecule with small
binding energy and a quark-antiquark ${}^{3}P_{0}$ state with a mixture of
${}^{3}P_{0}$ configurations $(u\bar{u}+d\bar{d})/\sqrt{2}$ and $s\bar{s}$.
The doubled-resonance structure of the $f_{0}(980)$ is consistent with the
analysis of Ref. [14].

###  5.2 $D_{s0}(2317)$

A relativistic quark model of charmed nonstrange and strange meson masses [15]
predicts all $S$-wave and $P$-wave masses to a satisfactory extent with the
exception of the spin-zero $D_{s0}(2317)$ and spin-1 $D_{s1}(2460)$. Its
prediction of $M(D_{s0}~{}(^{3}P_{0}))$ is compared with others in Table III
of Ref. [16], summarized in Table 1.

Table 1: Predictions of $M(D_{s0}~{}(^{3}P_{0}))$ ([16])(MeV)  2409 | 2484 | 2509 | 2380 | 2344 | 2409
---|---|---|---|---|---
[16] | [17] | [15] | [18] | [19] | [20]

Ref. [15] finds the relativistic $D_{s0}$ mass higher than any other
calculation, and farther from the observed value of $D_{s0}$. We propose that
$D_{s0}(2317)$ be accepted as the analog of $X(1840)$ while there should exist
a distinct molecular state near $D^{0}K^{+}$ threshold (1864.8 + 493.7 = 2359
MeV) which would be the analog of $X(1880)$.

##  6 Conclusions

Our treatment of the doubled resonance around 1840 and 1880 MeV has several
features:

(a) The $J^{PC}$ quantum numbers work out for both proton-antiproton and
diquark-antidiquark.

(b) The masses work out almost by construction for proton-antiproton and
rather nontrivially for diquark-antidiquark.

(c) We have identified a simple and non-trivial mechanism allowing the
transition between the two configurations.

We have identified the narrow molecular $p\bar{p}$ state reported in Ref. [1]
with the $X(1880)$ and the diquark-antidiquark resonance with the $X(1840)$. A
consequence of our conjecture is that the spin-parity of the
$3(\pi^{+}\pi^{-})$ system must be determined to be $0^{-+}$ without
interference from the broad $\eta^{\prime}\pi^{+}\pi^{-}$ state or others
around 1835 MeV.

## Acknowledgements

We thank Changzheng Yuan and Shuangshi Fang for useful discussions. This
research was supported in part by the ISF-NSFC joint research program (grant
No. 3166/23).

## References

  * [1] M. Ablikim et al. [BESIII Coll.], “Observation of the Anomalous Shape of $X(1840)$ in $J/\psi\rightarrow\gamma 3(\pi^{+}\pi^{-})$ Indicating a Second Resonance Near $p\bar{p}$ Threshold”, Phys. Rev. Lett. 132, 151901 (2024) [[arXiv:2310.17937 [hep-ex]].
  * [2] F. K. Guo, C. Hanhart, U. G. Meißner, Q. Wang, Q. Zhao and B. S. Zou,“Hadronic molecules,” Rev. Mod. Phys. 90, 015004 (2018) [erratum: Rev. Mod. Phys. 94, 029901 (2022)] [arXiv:1705.00141 [hep-ph]].
  * [3] R. L. Workman et al. [Particle Data Group], “Review of Particle Physics,” PTEP 2022, 083C01 (2022).
  * [4] M. Karliner and J. L. Rosner, “$X(3872)$, $X_{b}$, and the $\chi_{b1}(3P)$ state”, Phys. Rev. D 91, 014014 (2015) [arXiv:1410.7729 [hep-ph]].
  * [5] E. J. Eichten, K. Lane and C. Quigg, “New states above charm threshold,” Phys. Rev. D 73, 014014 (2006) [erratum: Phys. Rev. D 73, 079903 (2006)] [arXiv:hep-ph/0511179 [hep-ph]].
  * [6] M. Karliner and H. J. Lipkin, “Possibility of Narrow High-Mass Exotic States,” [arXiv:0710.4055 [hep-ph].
  * [7] M. Karliner and H. J. Lipkin, “Possibility of Exotic States in the Upsilon system,” [arXiv:0802.0649 [hep-ph]].
  * [8] M. Karliner and J. L. Rosner, “Configuration mixing in strange tetraquarks Zcs,” Phys. Rev. D 104, 034033 (2021) [arXiv:2107.04915 [hep-ph]].
  * [9] L. Micu, “Decay rates of meson resonances in a quark model,” Nucl. Phys. B 10, 521 (1969).
  * [10] E. W. Colglazier and J. L. Rosner, “Quark graphs and angular distributions in positive parity meson decays,” Nucl. Phys. B 27, 349 (1971).
  * [11] A. Le Yaouanc, L. Oliver, O. Pene and J. C. Raynal, “Naive quark pair creation model of strong interaction vertices,” Phys. Rev. D 8, 2223 (1973).
  * [12] M. Karliner and J. L. Rosner, “Scaling of $P$-wave excitation energies in heavy-quark systems,” Phys. Rev. D 98, 074026 (2018), [arXiv:1808.07869 [hep-ph]].
  * [13] T. Gutsche, C. Hanhart, R. E. Mitchell, and S. Spanier, “Scalar Mesons below 1 GeV,” Section 64 in [3].
  * [14] K. L. Au, M. R. Pennington and D. Morgan, “Experimental Evidence for Dynamics Beyond the Quark Model: An Extra I=0 Scalar Meson Near 1-GeV,” Phys. Lett. B 167, 229 (1986).
  * [15] D. Ebert, R. N. Faustov and V. O. Galkin, “Heavy-light meson spectroscopy and Regge trajectories in the relativistic quark model,” Eur. Phys. J. C 66, 197 (2010) [arXiv:0910.5612 [hep-ph]].
  * [16] R. H. Ni, Q. Li and X. H. Zhong, “Mass spectra and strong decays of charmed and charmed-strange mesons,” Phys. Rev. D 105, 056006 (2022) [arXiv:2110.05024 [hep-ph]].
  * [17] S. Godfrey and K. Moats, “Properties of Excited Charm and Charm-Strange Mesons,” Phys. Rev. D 93, 034035 (2016) [arXiv:1510.08305 [hep-ph]].
  * [18] J. Zeng, J. W. Van Orden and W. Roberts, “Heavy mesons in a relativistic model,” Phys. Rev. D 52, 5229 (1995) [arXiv:hep-ph/9412269 [hep-ph]].
  * [19] D. M. Li, P. F. Ji and B. Ma, “The newly observed open-charm states in quark model,” Eur. Phys. J. C 71, 1582 (2011) [arXiv:1011.1548 [hep-ph]].
  * [20] T. A. Lahde, C. J. Nyfalt and D. O. Riska, “Spectra and M1 decay widths of heavy light mesons,” Nucl. Phys. A 674, 141 (2000) [arXiv:hep-ph/9908485 [hep-ph]].