#  version1.4
Study of triangular flow $v_{3}$ in Au+Au and Cu+Cu collisions with a
multiphase transport model

Kai Xiao  Institute of Particle Physics, Central China Normal University,
Wuhan, Hubei, 430079, China  The Key Laboratory of Quark and Lepton Physics
(Central China Normal University), Ministry of Education, Wuhan, Hubei,
430079, China  Na Li<EMAIL_ADDRESS>Department of Physics, Huazhong
University of Science and Technology, Wuhan 430074, China  Shusu Shi
<EMAIL_ADDRESS>Institute of Particle Physics, Central China Normal
University, Wuhan, Hubei, 430079, China  The Key Laboratory of Quark and
Lepton Physics (Central China Normal University), Ministry of Education,
Wuhan, Hubei, 430079, China  Feng Liu  Institute of Particle Physics, Central
China Normal University, Wuhan, Hubei, 430079, China  The Key Laboratory of
Quark and Lepton Physics (Central China Normal University), Ministry of
Education, Wuhan, Hubei, 430079, China

###### Abstract

We studied the relation between the initial geometry anisotropy and the
anisotropic flow in a multiphase transport model (AMPT) for both Au+Au and
Cu+Cu collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV. It is found that unlike the
elliptic flow $v_{2}$, little centrality dependence of the triangular flow
$v_{3}$ is observed. After removing the initial geometry effect,
$v_{3}/\varepsilon_{3}$ increases with the transverse particle density, which
is similar to $v_{2}/\varepsilon_{2}$. The transverse momentum ($p_{T}$)
dependence of $v_{3}$ from identified particles is qualitatively similar to
the $p_{T}$ dependence of $v_{2}$.

###### pacs:

25.75.Ld, 25.75.Dw

##  I Introduction

A novel state of matter called quark-gluon plasma (QGP) composed by deconfined
partons is believed to be created experimentally in heavy ion collisions at
RHIC QGP . The discovery of large elliptic flow indicates that the partonic
collectivity is built up during the collisions, and the number-of-quark
scaling suggests that the partonic degrees of freedom are active nqscaling .

The anisotropic flow is usually described by a Fourier decomposition of the
azimuthal distribution with respect to the reaction plane ArtPRC . The second
harmonic coefficient, $v_{2}$, so called elliptic flow, has the biggest
magnitude at high energy collisions largeflow . It is believed that the
observed anisotropy in the momentum space is caused by the anisotropy in the
coordinate space in the initial condition. Lots of attention has been put on
the relation between $v_{2}$ and spatial eccentricity to see the hydrodynamics
behavior of the created system v2ecc ; PRCrun4 ; cucu_STAR .

Recent studies show that the event-by-event fluctuation of the initial
geometry v3ecc  may play an important role in the study of collective flow.
The triangular shape in the initial geometry will be transferred to the
momentum space as the system expands, and finally leads to the none zero value
of the third harmonic coefficient, $v_{3}$. It is found that the triangular
flow $v_{3}$ is responsible for the ridge and shoulder structures and the
broad away-side of two-particle azimuthal correlation v3ridge . Besides, it is
also considered to be a good probe to study the viscous hydrodynamics behavior
of the colliding system v3hydro .

Lots of properties of the triangular flow $v_{3}$ have been studied in
hydrodynamic and transport models v3hydro ; v3ampt . However, since the
reaction plane can not be directly measured in the experiment, those
anisotropic parameters can not be directly obtained. It is found that
different methods may cause up to $20\%$ discrepancy on $v_{2}$ Artv2Review ,
thus it should also be carefully evaluated for the $v_{3}$ study. Besides,
$v_{3}$ is directly related with the initial fluctuation, it is interesting to
see its system size dependence.

In this paper, we will study the triangular flow $v_{3}$ in both Au+Au and
Cu+Cu collisions in a multiphase transport model (AMPT) AMPT . The relation
between $v_{3}$ and $\varepsilon_{3}$ is studied as a function of number of
participants and transverse momentum. The paper is organized as follows: In
Sec. II, the observables and technical methods are introduced. A brief
description of AMPT model is given in Sec. III. The results and discussions
are presented in Sec. IV. Finally, a summary is given in Sec. V.

##  II OBSERVABLES

In a non-central collisions, the overlap region of two nuclei is an almond
shape. Since the position of nucleons may fluctuate event by event, as
discussed in Ref epart2 ; definition ; partecc , those initial geometric
irregularities of the colliding system can be described by $\varepsilon_{n}$:

$\varepsilon_{n}=\frac{{\sqrt{\left\langle{r^{2}\cos(n\varphi)}\right\rangle^{2}+\left\langle{r^{2}\sin(n\varphi)}\right\rangle^{2}}}}{{\left\langle{r^{2}}\right\rangle}},$
(1)

where $r$ and $\varphi$ are the polar coordinate position of participating
nucleons and $\langle\cdots\rangle$ is the average over all the participants
in an event. $n$ refers to the $n$-th harmonic, i.e., $\varepsilon_{2}$
describes the elliptic shape and $\varepsilon_{3}$ describes the triangular
shape.

As the system evolves, the anisotropy in the coordinate space is transferred
to the anisotropy in the momentum space due to the pressure gradient. The
particle distribution then can be written as

$\frac{dN}{d\phi}\propto 1+2\sum_{n=1}v_{n}\cos[n(\phi-\Psi_{n})],$ (2)

where $\phi$ is the azimuthal angle, and $\Psi_{n}$ is the $n$-th event plane
angle reconstructed by the final state particles:__

$\Psi_{n}=\frac{1}{n}\left[\tan^{-1}\frac{\sum\limits_{i}\sin(n\phi_{i})}{\sum\limits_{i}\cos(n\phi_{i})}\right].$
(3)

The observed anisotropic flow is defined as the $n$-th Flourier coefficient
$v_{n}$:

$v^{\rm obs}_{n}=\left\langle\cos[n(\phi-\Psi_{n})]\right\rangle.$ (4)

Here $\langle\cdots\rangle$ is taking the average over all the particles in
the sample.

This is the so-called event plane method of calculating $v_{n}$. The
reconstructed event plane fluctuates around the reaction plane. The observed
signals need to be revised by the corresponding resolution ArtPRC :

$v_{n}=\frac{v^{\rm obs}_{n}}{\mathscr{R}_{n}}.$ (5)

Due to the finite multiplicity of final state particles, the resolution

$\mathscr{R}_{n}=\langle\cos[n(\Psi_{n}-\Psi_{\rm nR})]\rangle$ (6)

is usually smaller than 1. $\Psi_{\rm nR}$ represents the nth real event plane
angle.

##  III AMPT Model

There are four main components in AMPT model: the initial conditions, parton
interactions, hadronization and hadron interactions. The initial conditions
are obtained from the HIJING model HIJING , which includes the spatial and
momentum distributions of minijet partons from hard processes and strings from
soft processes. The time evolution of partons is then treated according to the
ZPC ZPC  parton cascade model. After partons stop interacting, a combined
coalescence and string fragmentation model are used for the hadronization of
partons. The scattering among the resulting hadrons is described by a
relativistic transport (ART) model ART  which includes baryon-baryon, baryon-
meson and meson-meson elastic and inelastic scattering.

In our study, we analyzed the events from AMPT with the parton cross section
equals to 3 mb and 10 mb. As all the conclusions are independent on the parton
cross section, only the results from 3 mb are shown in this paper. There are
about 8 million events in $\mathrm{Au}+\mathrm{Au}$ collisions and 19 million
events in $\mathrm{Cu}+\mathrm{Cu}$ collisions at $\sqrt{s_{{}_{NN}}}$= 200
used. The string melting AMPT version is used since the previous study shows
that the string melting AMPT version agrees with the experimental results
better AMPT . The centrality is defined by the impact parameter.

##  IV Results and Discussions

Figure 1: The second and third harmonic event plane resolution calculated by
the particles with pseudo-rapidty region of $|\eta|>2$ as a function of
centrality in both Au+Au and Cu+Cu collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV
in AMPT model. Figure 2: $v_{n}$ as a function of centrality in both Au+Au and
Cu+Cu collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV in AMPT model.

In order to be comparable with the experimental data, the event plane method
is used to calculate $v_{n}$. The procedure is slightly different between ours
and Ref. v3ampt , in which the event plane was reconstructed by initial
partons. Charged particles with $p_{T}\leq 2$ GeV/$c$, $|\eta|>2$ are chosen
to reconstruct the event plane according to the Eq. 3. The particles used for
the $v_{n}$ measurements are within the $|\eta|<1$. The $\eta$ gap used here
is to reduce the auto-correlation between the particles used to reconstruct
the event plane and the particles of interest. In the following, the observed
$v_{n}$ are all corrected by the corresponding resolution. Fig. 1 shows the
resolution of $v_{2}$ and $v_{3}$ in both Au+Au and Cu+Cu collisions. The
resolution of $v_{2}$ shows a peak in mid-central collisions which is
consistent with the experimental result PRCrun2 . This is because the
resolution of $v_{2}$ is affected by both of the $v_{2}$ signal and the
multiplicity used to reconstruct the event plane. While the resolution of
$v_{3}$ only depends on the multiplicity, and keeps decreasing as the
multiplicity drops.

In Fig. 2, $v_{2}$ and $v_{3}$ are shown as functions of centrality in both
Au+Au and Cu+Cu collisions. We can see that $v_{2}$ shows strong centrality
dependence since it is mainly coming from the elliptic anisotropy in the
initial geometry. Unlike $v_{2}$, the dependence of $v_{3}$ on centrality and
system size are much smaller. The trend of $v_{3}$ observed is the same as
that in Ref v3ampt . However, the event plane angle $\Psi_{n}$ in Ref v3ampt
is obtained from the initial parton distribution, which is not observed in the
experiment, while in our study it is from the final state particle
distribution. The results indicate that the triangular flow is less sensitive
to the centrality and system size compared with the elliptic flow. It could be
understood as a result of combined effects from initial geometrical
fluctuation and collective dynamics which requires the size of bulk to
interact among themselves.

Figure 3: (Color online) $v_{2}$ as a function of $\varepsilon_{2}$ in Au+Au
collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV in AMPT model. The black points are
the average $v_{2}$ in the selected $\varepsilon_{2}$ bin. The pad in the
right down corner is the average of $v_{2}$ with smaller scale. Figure 4:
(Color online) $v_{3}$ as a function of $\varepsilon_{3}$ in Au+Au collisions
at $\sqrt{s_{{}_{NN}}}$=200 GeV in AMPT model. The black points are the
average $v_{3}$ in the selected $\varepsilon_{3}$ bin. The pad in the right
down corner is the average of $v_{3}$ with smaller scale.

It is commonly assumed that the harmonic flow coefficients $v_{n}$ linearly
depends on the $\varepsilon_{n}$. This assumption is supported by hydrodynamic
simulations v3hydro  as long as one probes deformed initial profiles with only
a single non-vanishing harmonic eccentricity coefficient. In Fig. 3 and Fig.
4, we investigate the feasibility of this assumption for $v_{2}$ and $v_{3}$
respectively. The relations between $v_{n}$ and $\varepsilon_{n}$ are drawn
event by event in the two-dimension plots. The black points are the average
values of $v_{n}$ in an selected $\varepsilon_{n}$ bin, and the curves are the
connection of points to guide our eyes. The pads in the right down corners are
the average values of $v_{n}$ with smaller scale. In Fig. 3, the $v_{2}$
increases with $\varepsilon_{2}$, which is consistent with the ideal
hydrodynamic calculation Ulrichv3hydro . While in Fig. 4, the triangular flow
$v_{3}$ firstly increases with $\varepsilon_{3}$ up to 0.17, and then
decreases. Based on our study, the higher $\varepsilon_{3}$ bin corresponds to
the more peripheral collisions. It is known that $v_{3}$ is caused by initial
geometrical fluctuation, and built up by the interactions of constituents. The
less interactions in the higher $\varepsilon_{3}$ bin may cause the less
converting efficiency from $\varepsilon_{3}$ to $v_{3}$. That could be the
reason of decreasing trend of $v_{3}$ when $\varepsilon_{3}$ is larger than
0.17. Both the trend and the value of $v_{3}$ show discrepancy to the ideal
hydrodynamic calculations Ulrichv3hydro . As discussed in Ref. v3hydro , the
viscosity causes the decrease of $v_{3}$, however, the effects to $v_{3}$
versus $\varepsilon_{3}$ is not shown in the viscous hydrodynamic calculation.

Figure 5: $v_{n}/\varepsilon_{n}$ as a function of transverse particle density
in both Au+Au and Cu+Cu collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV in AMPT
model. Figure 6: $v_{3}$ as a function of transverse momentum in (a) Au+Au and
(b) Cu+Cu collisions at $\sqrt{s_{{}_{NN}}}$=200 GeV in AMPT model.

The ratio of elliptic flow to eccentricity $v_{2}/\varepsilon_{2}$ gains lots
of interests by comparing with the hydrodynamic model definition ;
Ollitrault_etaOverS . Recently, the behavior of triangular flow $v_{3}$ in
ideal hydrodynamics is also discussed v3hydro . In Fig. 5, we study the
$v_{n}/\varepsilon_{n}$ as a function of transverse particle density. From the
plot we can see that $v_{n}/\varepsilon_{n}$ from Au+Au and Cu+Cu are
consistent with each other very well. As the transverse particle density
increases, $v_{3}/\varepsilon_{3}$ rises with smaller value than
$v_{2}/\varepsilon_{2}$. It implies that as the particle density increases,
the initial geometry asymmetry transfers to momentum asymmetry more
efficiently while the system expands. Besides, the second order harmonic is
more efficient than the third order.

At last, the transverse momentum dependence of $v_{3}$ for $\pi$, $K$, p and
$\Lambda$ is also studied in Au+Au and Cu+Cu collisions. In Fig. 6, we can see
that $v_{3}$ shows quite similar trend to $v_{2}$. At low $p_{T}$, the mass
ordering phenomena is observed. The lighter particles are found with larger
$v_{3}$. It indicates that although $v_{3}$ is driven by $\varepsilon_{3}$,
its transverse momentum dependence is dominated by the hydrodynamics behavior
of the system. While when $p_{T}\geq 1.5$ GeV/$c$, baryons and mesons are
separated into two groups. The $p_{T}$ dependence of $v_{3}$ from identified
particles is qualitatively similar to the $p_{T}$ dependence of $v_{2}$
PRCrun4 ; cucu_STAR . The $v_{3}$ results of identified particles from AMPT
model are similar to the STAR preliminary results Yadav .

##  V Summary

In summary, we studied the relation between initial geometry parameter
$\varepsilon_{n}$ and anisotropic flow $v_{n}$ in Au+Au and Cu+Cu collisions
using the AMPT Monte-Carlo model. We find that the triangular flow $v_{3}$ is
less sensitive to the centrality and system size compared with the elliptic
flow $v_{2}$. The $v_{2}$ displays an increasing trend as a function of
$\varepsilon_{2}$, which is qualitatively consistent with hydrodynamic
calculation. We found that $v_{3}$ shows an increasing trend when
$\varepsilon_{3}$ is less than 0.17, and then decreases beyond
$\varepsilon_{3}$ = 0.17. It may be because of the lower converting efficiency
from $\varepsilon_{3}$ to $v_{3}$ in the higher $\varepsilon_{3}$ bin. This
decreasing trend is in contrast to the results of ideal hydrodynamic
calculation. Both $v_{2}/\varepsilon_{2}$ and $v_{3}/\varepsilon_{3}$ increase
with the transverse particle density, and the second harmonic asymmetry in the
initial geometry seems to transfer to the momentum asymmetry more efficiently
than the third harmonic. The triangular flow $v_{3}$ of identified particles
shows a mass ordering in low $p_{T}$ and meson-baryon splitting at
intermediate $p_{T}$ in both Au+Au and Cu+Cu collisions which is similar to
the $p_{T}$ dependence of $v_{2}$.

##  VI Acknowledgments

We wish to thank Prof. Fuqiang Wang for useful suggestions, and Dr. Kejun Wu
for useful discussions on the AMPT model. This work was supported in part by
the National Natural Science Foundation of China under grant No. 10775060,
11105060, 11135011, 11147146 and ‘the Fundamental Research Funds for the
Central Universities’, Grant No. HUST: 2011QN195.

## References

  * (1) I. Arsene et al. (BRAHMS Collaboration), Nucl. Phys. A 757, 1 (2005); B. B. Back et al. (PHOBOS Collaboration), Nucl. Phys. A 757, 28 (2005); J. Adams et al. (STAR Collaboration), Nucl. Phys. A 757, 102 (2005); S. S. Adcox et al. (PHENIX Collaboration), Nucl. Phys. A 757, 184 (2005).
  * (2) J. Adams et al (STAR Collaboration), Phys. Rev. Lett. 95 122301 (2005).
  * (3) A. M. Poskanzer and S. A. Voloshin, Phys. Rev. C 58, 1671 (1998).
  * (4) K. H. Achermann et al. (STAR Collaboration), Phys. Rev. Lett. 86, 402 (2001).
  * (5) B. I. Abelev et al (STAR Collaboration), Phys. Rev. C 77, 54901 (2008).
  * (6) B. I. Abelev et al (STAR Collaboration), Phys. Rev. C 81, 044902 (2010).
  * (7) S. A. Voloshin and A. M. Poskanzer, Phys. Lett. B 474, 27 (2000). P. F. Kolb, J. Sollfrank and U. W. Heinz, Phys. Rev. C 62, 054909 (2000).
  * (8) B. Alver et al., Phys. Rev. Lett. 98, 242302 (2007); B. Alver et al. PHOBOS Collaboration, Phys. Rev. C 81, 034915 (2010).
  * (9) B. Alver and G. Roland, Phys. Rev. C 81, 054905 (2010); Jun Xu, Che Ming Ko, Phys. Rev. C 83, 021903 (2011);G.-L. Ma, X.-N. Wang, Phys. Rev. Lett. 106, 162301 (2011).
  * (10) B. H. Alver, C. Gombeaud, M. Luzum, and J. Y. Ollitrault, Phys. Rev. C 82, 034913 (2010).
  * (11) L.-X. Han et al., arXiv:1105.5415
  * (12) Z.-W. Lin et al., Phys. Rev. C59, 2716 (1999).
  * (13) B. Alver et al. (PHOBOS Collaboration), Phys. Rev. C 77, 014906 (2008).
  * (14) J. Y. Ollitrault, Phys. Rev. D46 229 (1992).
  * (15) B. Alver et al. (PHOBOS Collaboration), Phys. Rev. Lett. 98, 242302 (2007).
  * (16) S. A. Voloshin, A. M. Poskanzer, R. Snellings arXiv:0809.2949 (2008) and Voloshin, Sergei A., Poskanzer, Arthur M., Snellings, Raimond: Collective Phenomena in Non-Central Nuclear Collisions. Stock, R. (ed.). SpringerMaterials - The Landolt-Bornstein Database (http://www.springermaterials.com). Springer-Verlag Berlin Heidelberg, 2010. $DOI:10.1007/978-3-642-01539-7_{1}0$
  * (17) X.-N. Wang, Phys. Rev. D 43 104 (1991).
  * (18) B. Zhang, Comput. Phys. Commun. 109, 193 (1998).
  * (19) B.A. Li and C.M. Ko, Phys. Rev. C 52 2037 (1995).
  * (20) J. Adams et al (STAR Collaboration), Phys. Rev. C 72, 14904 (2005).
  * (21) Z. Qiu and U. Heinz, Phys. Rev. C 84, 024911 (2005).
  * (22) H. J. Drescher, A. Dumitru, C. Gombeaud and J. Y. Ollitrault, Phys. Rev. C76 024905 (2007).
  * (23) Y. Pandit for the STAR collaboration, poster, RHIC and AGS Annual Users’ Meeting, Upton, NY, Jun. 20-24, 2011.


11institutetext: State University of New York at Buffalo, Buffalo NY, USA
11email: {souvikda, sougatas<EMAIL_ADDRESS>

# Medical Literature Mining and Retrieval in a Conversational Setting

Souvik Das equal contribution11 Sougata Saha ††footnotemark:  11 Rohini K.
Srihari 11

###### Abstract

The Covid-19 pandemic has caused a spur in the medical research literature.
With new research advances in understanding the virus, there is a need for
robust text mining tools which can process, extract and present answers from
the literature in a concise and consumable way. With a DialoGPT based multi-
turn conversation generation module, and BM-25 & neural embeddings based
ensemble information retrieval module, in this paper we present a
conversational system, which can retrieve and answer coronavirus related
queries from the rich medical literature, and present it in a conversational
setting with the user. We further perform experiments to compare neural
embedding based document retrieval and the traditional BM25 retrieval
algorithm, and report the results.

###### Keywords:

Information Retrieval Conversational Systems Covid-19.

##  1 Introduction

The ongoing Covid-19 pandemic has seen a spur in medical research. With the
number of infections well over 25 million world wide, there has been a burst
of research for finding a cure of the highly infectious disease. With the
volume of publications related to coronaviruses increasing steeply, there is a
need for robust text mining tools which can aid medical researchers to survey
the existing biomedical publications effectively and exhaustively. Keeping
this use case in mind, we present a question answering socialbot that can not
only mine the existing medical literature, but also conduct open domain chit-
chat dialogue, thus acting as a personal assistant for a researcher. In the
field of natural language processing, there has been significant amount of
research in question answering(QA) systems. The introduction of transformer
architecture [17] has propelled the QA capabilities of NLP systems. Large pre-
trained transformer models like BERT [3] have attained state of the art
performance in standard open domain extractive QA datasets like SQuAD [13],
[14]. Although transformer based architectures attain state of the art
performance, they have a high time complexity in scenarios where answer is to
be extracted by processing multiple documents. [16] introduces a dense-sparse
phrase index to alleviate the inference time bottleneck, by generating and
indexing phrases from a corpus.

The Alexa prize socialbot grand challenge is testimony to the advances made in
conversational systems. With the goal of creating a conversational system that
can converse with a human for more than 20 minutes, and receive a user
satisfaction score of 4 out of 5, there has been significant research in
trying to attain this goal. The winners of the Alexa prize socialbot challenge
3 [4],and the runners up [11] leverage an array of state of the art NLP
techniques to develop a socialbot that converses like human. Using more
sophisticated techniques for bettering the NLU(natural language understanding)
module of a chatbot, the Alexa prize challenge has seen an increase in user
approval ratings of chatbot conversations. [12] introduced GPT-2, a
transformer based pre-trained language model which unlike BERT can only attain
to the seen context. With 1.5 billion parameters, GPT-2 achieves state of the
art performance in 7 out of 8 language modelling tasks in a zero-shot setting.
[19] introduces DialoGPT, a conversational response generation model which
extends GPT-2, and is trained on Reddit comment pairs.

In March 2020 Allen Institute for AI (AI2) released the first version of the
COVID-19 Open Research Dataset(CORD-19) dataset [18], which contains all
Covid-19 and coronavirus related research from PubMed’s PMC, WHO(World Health
Organization), bioRxiv and medRxiv pre-prints. The CORD-19 dataset has been
ever increasing, and has fuelled several IR competitions in Kaggle and the
TREC-Covid IR challenge [15]. In the past few months, several text mining and
retrieval systems have been implemented within the NLP research community with
the objective of making it easy for the medical research community to retrieve
information from coronavirus related publications. [10] proposed a language
model for automatically answering questions related to COVID-19. They utilized
the GPT-2 language model and applied transfer learning to retrain it on the
CORD-19 corpus. [6] proposes covidAsk, a question answering (QA) system that
combines biomedical text mining and QA techniques to provide answers to
questions in real-time. They leverage both supervised and unsupervised
approaches to provide informative answers using DenSPI [16] and BEST [8].

Inspired by the recent developments in NLP, and with the motive of enabling
medical research for finding a cure for Covid-19, we present a DialoGPT based
socialbot, which is not only capable of conducting open domain chat in a
multi-turn setting, but also able to detect user intent and leverage a
combination of traditional information retrieval models like BM25, and
transformer based models like BERT and bioBERT [7]. We leverage the
DailyDialog dataset [9] to fine tune DialoGPT on multi-turn open domain
conversations. For document retrieval, we implement an ensemble of rank-bm25,
a python implementation of Okapi BM25, and bioBERT embeddings. For mining and
extracting the answer span from a relevant document, we leverage a pre-trained
BERT large model which is fine tuned on the SQuAD dataset. In the next
sections we detail our implementation and share the results. Finally we
perform error analysis to understand the pitfalls of the system and propose
potential future research.

##  2 Methodology and Experiments

###  2.1 Datasets

####  2.1.1 Chatbot

We use the dailydialogue dataset [9] for fine-tuning the Dialog-GPT pretrained
model. The dataset consists of manually written multi-turn dialogues, with
manually labelled communication intent and emotion. The dataset consists of
conversations pertaining to 10 different day to day topics like ordinary life,
school life, culture & education, attitude & emotion, relationship, tourism ,
health, work, politics, and finance. The dataset captures 7 types of
emotion:anger, disgust, fear, happiness, sadness, surpris eand no emotion, and
4 distinct dialogue acts: in-form, questions, directives and commissive.

####  2.1.2 COVID Turn Classifier

For building the COVID text classifier we have used COQB-19 dataset [1], this
dataset consists of a total of 3,924 questions. Each question is assigned with
a Question ID. Questions of the same ID are regarded as similar questions and
grouped together in the dataset. The dataset is further divided into 2 parts,
(1) Crowd-Sourced: This set consists of a total of 2,341 questions with 280
unique Question IDs. (2) Auto-generated: This set consists of a total of 1,583
questions with 68 unique Question IDs. For our experimentation we have only
considered the crowd-sourced questions. The negative examples for the
classifier we have randomly sampled 3593 unique turns from the daily dialog
dataset.

####  2.1.3 Information Retrieval system:

With the aim of providing text mining capabilities to medical researchers, we
perform information retrieval and question answering on the the CORD-19 open
research dataset. Specifically we use the 19th June 2020 dump of the dataset.
The CORD-19 dataset consists of the latest medical literature related to
Covid-19 and coronaviruses. We further leverage the relevance judgement scores
from the 4th round of the TREC-COVID information retrieval challenge for
experimenting with different retrieval systems.

###  2.2 Chatbot Architecture

Figure 1: Architecture of chat processing and generation pipeline.

####  2.2.1 Dialog Generation Using DialoGPT

We have fine-tuned the DialoGPT pre-trained model on the Daily-Dialog dataset.
During training, we have conditioned the model to classify emotion and dialog-
act of the reply. Our model inherits the overall DialoGPT architecture, which
is based on GPT2. For generating the response for general queries we employ a
top-k sampling approach.

####  2.2.2 Natural Language Understanding(NLU) Module

NLU module ingests all the user turns and performs co-reference resolution on
the most recent user turn, with the uber goal of being able to resolve COVID
related terms. The resolved text is passed through a transformer based text
classifier for classifying whether the text contains any Covid related text.
If the text is classified as Covid text the we run this text through a keyword
analyzer, which matches the text with some predefined keywords related to
Covid, MERS and SARS, and enriches the query by attaching keywords related to
the disease. The processed text is then passed to the IR system to fetch
potential information related to the need of the user. For general queries we
generate responses using the fine-tuned Dialo GPT model.

###  2.3 Information Retrieval Architecture

We implement a hierarchical IR architecture for retrieving the correct
response to a user query. We utilize an ensemble of neural embedding based and
traditional IR based retrieval for retrieving top documents, which can
potentially answer the user query. We perform question answering on the body
text of the retrieved documents to extract the required user information, and
return a ranked set of responses to the user. Fig. 2 illustrates the document
pre-processing and retrieval architecture.

Figure 2: Architecture of the document pre-processing, and the inference time
document retrieval pipeline.

####  2.3.1 Document pre-processing & Indexing:

For our analysis, we only consider documents which have both an abstract and
body in the CORD-19 dataset. The CORD-19 dataset contains data from multiple
sources. In order to maintain the most complete representation of a document,
we keep the data from the source which yields higher tokens. We further filter
out documents which have either more than 300 tokens in abstract or more than
100 paragraphs of text in the body. Post processing, our document collection
stands at 53,000 documents with abstract and body text present. We store the
documents in 3 different ways: i) We tokenize the document abstracts and
create an inverted index over the tokens. ii) We generate 768 dimensional
bioBERT embeddings of the abstracts, and maintain a lookup of the document
abstracts. iii) We implement a sliding window of 220 tokens with an overlap of
50 tokens, and generate continuous chunks from the document body, and maintain
a lookup between the abstract inverted index and embeddings.

####  2.3.2 Document Retrieval:

Since an abstract is a succinct summary of a scientific document, we determine
the relevance of a document to a query, by calculating the relevancy between
the abstract and the user query. Given a user query, we simultaneously search
the previously created inverted index for the most relevant abstract using the
Okapi BM25 algorithm, and also calculate the cosine similarity between the
bioBERT embedding of the user query and the abstract embeddings to retrieve
the most similar abstracts to the query. We apply thresholds on the cosine
similarity score and BM25 score to determine if a document should be
considered to be relevant. The thresholds are determined by experimentation on
the TREC dataset, which is discussed in section 3.3. Post thresholding, we
normalize(between 0 and 1) and aggregate the two relevance scores, and re rank
the documents by the aggregated score. We consider the top 20 ranked
documents, and pass the body text to the downstream question answering module,
for extracting the exact information from the relevant document.

####  2.3.3 Question & Answering(QA):

Given a list of most relevant documents, the QA module performs BERT question
answering on the chunks of body text of a document, and extracts the most
relevant span. We lookup the body chunks of the top 20 documents retrieved by
the document retrieval module, and use pre-trained BERT large model which is
fine tuned on the SQUAD dataset to perform question answering on each of the
body chunks. Given a query and a text passage, the BERT question answering
model generates the log likelihood of each of the tokens in the passage, as
being a start and end token in the answer span. We extract the most probable
span from a text chunk, and leverage the log likelihood of the starting token
of the span for ranking and filtering the answers, which is discussed below.

####  2.3.4 Ranking & Response Generation:

We apply the following filtration criteria to the predicted spans to check
their eligibility of being an answer. i) We observed that longer spans were
associated with non favourable answers, hence applied a criteria that the span
should not be more than 15 tokens. ii) The answer should not contain any
special BERT specific tokens like ‘CLS’, ‘SEP’ and ‘PAD’. Since we perform
question answering on multiple chunks of the same document, we select one
answer per relevant document by selecting the answer whose start token has the
highest log likelihood. Finally, we rank the answers from the relevant
documents based on the start token log likelihood and the normalized &
aggregated BM25 and cosine similarity score, and yield the top 5 answers along
with the document name(paper name) to the user. If none of the answers pass
the filtration criteria, we return a ranked list of 5 most relevant document
names(paper names) to the user based on the document retrieval score(BM25 +
cosine similarity score). In the case of no document being retrieved by the
retrieval module, we respond with a response seeking clarification from the
user.

##  3 Experiments

###  3.1 Fine Tuning Dialo-GPT

The model has 125M trainable parameters, and we have trained the model using a
training batch size of 32, gradient accumulation step of 8 and a learning rate
of 1e-5. We have used Adam optimizer [5] with max grad norm set to 1. The
model was trained for 5000 optimization steps with 2000 warmup steps. The
weight-decay for all model parameters except for biases and LayerNorm [2] was
set to 0.01.

###  3.2 COVID Turn Classifier

We trained a transformer based classification model using the COQB-19 dataset
for the positive examples and randomly sampling DailyDialog utterances as
negative examples. The model consists of a transformer layer followed by a
linear layer to make a binary prediction of whether the text is coronavirus
related or open domain. For our experimentation we have used BERT base pre-
trained model, we apply dropout with a probability of 0.3 before passing
through the linear layer. A softmax layer was used to calculate the class
probabilities. Loss is calculated using binary cross entropy function. The
entire dataset was split into 70:15:15 ratio for training, validation and test
sets respectively. The model is trained for 2 epochs with maximum length of
the text equal to 250 characters, batch size of 32 and learning rate of 2e-5.
We have used Adam optimizer with max grad norm set to 1.

###  3.3 IR System

We compared 3 different retrieval strategies: i) Using Okapi BM-25 for scoring
and retrieving relevant document abstracts. ii) Cosine similarity between
bioBERT query and abstract embedding. iii) Using Okapi BM-25 score and
embedding cosine similarity and retrieving the union of the two strategies.
Between April to July 2020, TREC conducted an information retrieval challenge,
where the task was to retrieve the most relevant documents from the CORD-19
dataset, for a set of 45 Covid-19 related queries. They also provided a set of
human judged query relevance judgement scores(TREC paper relevance judgement).
The relevance judgement scores indicated whether a document was highly
relevant, mildly relevant or not relevant to a query. In order to determine
the optimal document retrieval strategy, we leveraged the relevance judgement
scores and performed a comparison study. We converted the three class problem
to binary problem (relevant or not relevant) by collapsing the highly and
mildly relevant classes into one, and tested our document retrieval strategies
by comparing the F1 score. Since BM25 and cosine similarity generates scores
and not classes, we had to determine thresholds on the scores to classify a
document as relevant or not relevant, and then calculate the F1 score. For
each retrieval strategy, we performed an extensive grid search to determine
the optimal threshold.

##  4 Results

We attain a perplexity score of 13.125 by jointly fine tuning DialoGPT to
generate responses as well as predicting the dialogue act and emotion of the
turns. Mathematically perplexity it is defined as the exponentiation of
entropy ($2^{H(p)}$), and a lower score is preferred. Table 1 shows the test
set performance of the Covid Turn Classifier. Though we observe a high
performance, the model fails to understand a few ambiguous cases when a
speaker talks about a random virus. We believe implementing a way to better
negative samples will yield better results.

We report the results of the different document retrieval strategies in Table
2. We observe that retrieving documents based on the cosine similarity of the
abstract bioBERT embedding performs worse than BM25 based retrieval. We reason
that although embedding based approaches are superior for many NLP tasks,
where it is important to represent near similar or semantically similar
documents nearby in the embedding space, in search the presence of exact
keywords matters more. We further notice that retrieving documents based on a
union of embedding cosine similarity and BM25 score yields best results, as
this captures the best of both retrieval methods.

Table 1: Testing performance of the COVID Turn Classifier Evaluation Metric | Score
---|---
Accuracy | 0.9833
F1 | 0.9850
Table 2: Comparison of F1 score between different IR approaches. Retrieval Method | F1 | Threshold
---|---|---
Okapi BM 25 | 0.6527 | 2.77
Cosine Similarity | 0.6315 | 0.89
Okapi BM 25 + Cosine Similarity | 0.6713 | 2.77, 0.89

##  5 Discussion

(a) Actual chat conducted with the conversational system.

(b) Correct documents retrieved by the IR module with a reformulated query.
Note how the last user query in chat (a) retrieves non-relevant documents.

Figure 3: Sample of chats conducted with the conversational system. Please
zoom in for better viewability.

Fig.3 depicts an example chat conducted with the conversational system. We
have annotated the chatbot responses with the module that was responsible for
the reply. Keeping the DialoGPT based response generation module aside, there
are 3 main kinds of errors related to information retrieval that the system
makes: i) Retrieval error: When the IR system retrieves a wrong document. As
shown in the experiments, a ensemble approach of embedding based and
traditional based retrieval works better, which can be made more efficient by
implementing a hierarchical index of the document body text instead of the
abstract. ii) Answer span selection error: When the BERT QA model predicts
spurious span as answer to a query. This error can be reduced by creating a
question answering dataset for the medical literature, and fine tuning BERT
for question answering in a supervised learning setting. iii) Query
formulation error: When the query is ill formed by the user or the NLU, which
can result in retrieval of the wrong documents. This can be reduced by
reformulating the user query to adhere to a specific template which can be
passed to the IR system. Fig. 3 depicts a similar scenario.

##  6 Conclusion

In this paper we present a document retrieval and text mining system, which
can mine the biomedical scientific literature for answers, and have
implemented a GPT-2 based conversational system as an interface for
interacting with a user. We use a natural language understanding unit for
deciding whether to send a query to the IR system for retrieving answers from
the biomedical literature, or to generate an open domain response based on the
previous context of the conversation. In future, we intend to implement a
paraphrasing and natural text generation module, which can take as input the
most relevant chunk of body text, and generate a response in a more
conversational way.

## References

  * [1] Coqb-19: Covid-19 question bank, https://www.newvoicesnasem.org/data-downloads
  * [2] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016)
  * [3] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Proceedings of the 2019 Conference of the North (2019). https://doi.org/10.18653/v1/n19-1423, http://dx.doi.org/10.18653/v1/N19-1423
  * [4] Finch, J.D., Ahmadvand, A., Dong, X., Qi, R., Sahijwani, H., Volokhin, S., Wang, Z., Wang, Z., Choi, J.D.: Emora: An inquisitive social chatbot who cares for you (2020)
  * [5] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014)
  * [6] Lee, J., Yi, S.S., Jeong, M., Sung, M., Yoon, W., Choi, Y., Ko, M., Kang, J.: Answering questions on covid-19 in real-time (2020)
  * [7] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (Sep 2019). https://doi.org/10.1093/bioinformatics/btz682, http://dx.doi.org/10.1093/bioinformatics/btz682
  * [8] Lee, S., Kim, D., Lee, K., Choi, J., Kim, S., Jeon, M., Lim, S., Choi, D., Kim, S., Tan, A.C., Kang, J.: Best: Next-generation biomedical entity search tool for knowledge discovery from biomedical literature. PLOS ONE 11(10), 1–16 (10 2016). https://doi.org/10.1371/journal.pone.0164680, https://doi.org/10.1371/journal.pone.0164680
  * [9] Li, Y., Su, H., Shen, X., Li, W., Cao, Z., Niu, S.: Dailydialog: A manually labelled multi-turn dialogue dataset (2017)
  * [10] Oniani, D., Wang, Y.: A qualitative evaluation of language models on automatic question-answering for covid-19 (2020)
  * [11] Paranjape, A., See, A., Kenealy, K., Li, H., Hardy, A., Qi, P., Sadagopan, K.R., Phu, N.M., Soylu, D., Manning, C.D.: Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations (2020)
  * [12] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
  * [13] Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: Unanswerable questions for squad. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (2018). https://doi.org/10.18653/v1/p18-2124, http://dx.doi.org/10.18653/v1/P18-2124
  * [14] Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016). https://doi.org/10.18653/v1/d16-1264, http://dx.doi.org/10.18653/v1/D16-1264
  * [15] Roberts, K., Alam, T., Bedrick, S., Demner-Fushman, D., Lo, K., Soboroff, I., Voorhees, E., Wang, L.L., Hersh, W.R.: TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19. Journal of the American Medical Informatics Association (07 2020). https://doi.org/10.1093/jamia/ocaa091, https://doi.org/10.1093/jamia/ocaa091, ocaa091
  * [16] Seo, M., Lee, J., Kwiatkowski, T., Parikh, A., Farhadi, A., Hajishirzi, H.: Real-time open-domain question answering with dense-sparse phrase index. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4430–4441. Association for Computational Linguistics, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1436, https://www.aclweb.org/anthology/P19-1436
  * [17] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2017)
  * [18] Wang, L.L., Lo, K., Chandrasekhar, Y., Reas, R., Yang, J., Eide, D., Funk, K., Kinney, R.M., Liu, Z., Merrill, W., Mooney, P., Murdick, D., Rishi, D., Sheehan, J., Shen, Z., Stilson, B., Wade, A., Wang, K., Wilhelm, C., Xie, B., Raymond, D., Weld, D.S., Etzioni, O., Kohlmeier, S.: Cord-19: The covid-19 open research dataset. ArXiv (2020)
  * [19] Zhang, Y., Sun, S., Galley, M., Chen, Y.C., Brockett, C., Gao, X., Gao, J., Liu, J., Dolan, B.: Dialogpt: Large-scale generative pre-training for conversational response generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020). https://doi.org/10.18653/v1/2020.acl-demos.30, http://dx.doi.org/10.18653/v1/2020.acl-demos.30


# STABLE TETRAQUARKS

CHRIS QUIGG 111Email<EMAIL_ADDRESS>FERMILAB–CONF–18/099–T Theoretical
Physics Department, Fermi National Accelerator Laboratory
P.O.Box 500, Batavia, Illinois 60510 USA

###### Abstract

For very heavy quarks, relations derived from heavy-quark symmetry imply novel
narrow doubly heavy tetraquark states containing two heavy quarks and two
light antiquarks. We predict that double-beauty states will be stable against
strong decays, whereas the double-charm states and mixed beauty+charm states
will dissociate into pairs of heavy-light mesons. Observing a new double-
beauty state through its weak decays would establish the existence of
tetraquarks and illuminate the role of heavy color-antitriplet diquarks as
hadron constituents.

##  1 Introduction

Since the BELLE collaboration’s discovery of the charmonium-associated state
$X(3872)$,[1] hadron spectroscopy has been reinvigorated and recast.[2] Many
of the newly observed states invite identification with compositions beyond
the traditional quark–antiquark meson and three-quark baryon schemes,
possibilities foreseen in the foundational quark-model papers.[3] Tetraquark
states composed of a heavy quark and antiquark plus a light quark and
antiquark have attracted much attention. All the observed candidates fit the
form $c\bar{c}q_{k}\bar{q}_{l}$, where the light quarks $q$ may be
$u,d,\hbox{or }s$. The putative tetraquarks typically have strong decays to
$c\bar{c}$ charmonium + light mesons. None is observed significantly below
threshold for strong decays into two heavy–light meson states
$\bar{c}q_{k}+c\bar{q}_{l}$.

Estia Eichten and I have examined the possibility of unconventional tetraquark
configurations for which all strong decays are kinematically forbidden.[4] In
the heavy-quark limit, stable—hence exceedingly
narrow—$Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ mesons must exist. To apply this
insight, we take into account corrections for finite heavy-quark masses to
deduce which tetraquark states containing $b$ or $c$ quarks might be stable.
The most promising candidate is a $J^{P}=1^{+}$ isoscalar double-$b$ meson,
$\mathcal{T}^{\\{bb\\}-}_{[\bar{u}\bar{d}]}$. I will sketch our derivation and
results, emphasizing the consequences for experiment, and indicate areas in
which experimental and theoretical work can be productive.

##  2 Heavy-quark symmetry implies stable heavy tetraquark mesons
$Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$

One-gluon-exchange between a pair of color-triplet heavy quarks is attractive
for $(QQ)$ in a color-$\mathbf{\bar{3}}$ configuration and repulsive for the
color-$\mathbf{6}$ configuration. The strength of the $\mathbf{\bar{3}}$
attraction is half that of the corresponding $(Q\bar{Q})$ in a
color-$\mathbf{1}$. This means that in the limit of very heavy quarks, we may
idealize the color-antitriplet $(QQ)$ diquark as a stationary, structureless
color charge, as depicted in the leftmost panel in Figure 1.

Figure 1: Schematic evolution of a $Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ state as
the heavy-quark masses decrease (and the mean separation between the heavy
quarks increases) from left to right.

We can separate the strong dynamics binding the diquark from the long-range
color interaction by which the light antiquarks interact with each other and
are bound to the diquark “nucleus.”

For sufficiently heavy quarks $Q$, a $Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$
tetraquark meson is stable against strong decays, as we can show by
considering possible decay modes. First, we note that dissociation into two
heavy–light mesons is kinematically forbidden. The $\mathcal{Q}$ value for the
decay is $\mathcal{Q}\equiv
m(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})-[m(Q_{i}\bar{q}_{k})+m(Q_{j}\bar{q}_{l})]=\Delta(q_{k},q_{l})-{{\textstyle\frac{1}{2}}}\\!\left({\textstyle\frac{2}{3}}\alpha_{\mathrm{s}}\right)^{2}\\![1+O(v^{2})]\overline{M}+O(1/\overline{M})$,
where $\Delta(q_{k},q_{l})$, the contribution due to light dynamics, becomes
independent of the heavy-quark masses,
$\overline{M}\equiv(1/{m_{Q}}_{i}+1/{m_{Q}}_{j})^{-1}$ is the reduced mass of
$Q_{i}$ and $Q_{j}$, and $\alpha_{\mathrm{s}}$ is the strong coupling. The
velocity-dependent hyperfine corrections, here negligible, are calculable in
the nonrelativistic QCD formalism.[5] For large enough values of
$\overline{M}$, the middle term on the right-hand side dominates, so the
tetraquark is stable against decay into two heavy-light mesons.

What of the other possible decay channel, a doubly heavy baryon plus a light
antibaryon,
$(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})\to(Q_{i}Q_{j}q_{m})+(\bar{q}_{k}\bar{q}_{l}\bar{q}_{m})$?
For very heavy quarks, the contributions of $Q$ motion and spin to the
tetraquark mass are negligible. Since the $(QQ)$ diquark is a color-
antitriplet, heavy-quark symmetry tells us that
$m(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})-m(Q_{i}Q_{j}q_{m})=m(Q_{x}q_{k}q_{l})-m(Q_{x}\bar{q}_{m})$.
The flavored-baryon–flavored-meson mass difference on the right-hand side has
the generic form $\Delta_{0}+\Delta_{1}/{M_{Q}}_{x}$. Using the observed mass
differences, $m(\Lambda_{c})-m(D)=416.87\hbox{ MeV}$ and
$m(\Lambda_{b})-m(B)=340.26\hbox{ MeV}$, and choosing effective quark masses
$m_{c}\equiv m(J\\!/\\!\psi)/2=1.55\hbox{ GeV}$, $m_{b}\equiv
m(\Upsilon)/2=4.73\hbox{ GeV}$, we find $\Delta_{1}=176.6\hbox{ MeV}^{2}$ and
$\Delta_{0}=303\hbox{ MeV}$, hence the mass difference in the heavy-quark
limit is $303\hbox{ MeV}$. The right-hand side is in every case smaller than
the mass of the lightest antibaryon, $m(\bar{p})=938.27\hbox{ MeV}$, so no
decay to a doubly heavy baryon and a light antibaryon is kinematically
allowed.

_With no open channels in the heavy-quark limit, stable
$Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ mesons must exist._ To assess the
implications for the real world, we must first test whether it makes sense to
idealize the $(QQ)$ diquark as a tiny, structureless, color-antitriplet color
source.222See Ref. 6 for a thoughtful critical assessment. As the separation
between the heavy quarks increases, the light-antiquark cloud screens the
$Q_{i}Q_{j}$ interaction, altering the $\mathbf{\bar{3},6}$ mix, and
eventually leading to the division of the $(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})$
state into a pair of heavy–light mesons. These changes are indicated in the
progression from left to right in Figure 1. Using a half-strength
Coulomb$+$linear quarkonium potential, we verified that the rms core radii are
small on the expected tetraquark scale: $\langle r^{2}\rangle^{1/2}=0.28\hbox{
fm}\,(cc);0.24\hbox{ fm}\,(bc);0.19\hbox{ fm}\,(bb)$. This conclusion is
supported by exploratory lattice QCD studies.[7]

To ascertain whether stable tetraquark mesons might be observed, we must
estimate masses of the candidate configurations. Numerous model calculations
exist in the literature,333A useful compilation appears in Table IX of Ref. 8.
but heavy-quark symmetry makes it possible to compute the
$Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ tetraquark masses directly, through the
relation
$m(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})-m(Q_{i}Q_{j}q_{m})=m(Q_{x}q_{k}q_{l})-m(Q_{x}\bar{q}_{m})$,
with due attention to spin configurations and finite-mass corrections that
arise from hyperfine interactions and kinetic-energy shifts for the light
degrees of freedom.444The arithmetic is made explicit in Ref. 4. Experiments
have determined nearly all the information about heavy baryons and heavy–light
mesons needed to evaluate the right-hand side in every case of interest, i.e.,
for tetraquarks based on $bb,bc,\hbox{and }cc$ diquarks.555The lifetime
($\approx 0.4\hbox{ ys}$) of the top quark is too short to permit the
formation of hadrons containing $t$. The doubly heavy baryons have been more
elusive: for the moment, the strongest evidence we have is for the
$\Xi_{cc}^{++}$ candidate reported by the LHC$b$ experiment at a mass of
$3621.40\pm 0.78\hbox{ MeV}$.[9] With this input, we compute the mass of the
lightest $(cc)$ tetraquark as $m(\\{cc\\}[\bar{u}\bar{d}])=3978\hbox{ MeV}$,
which lies $102\hbox{ MeV}$ above the threshold for decay into
$D^{+}D^{*0}$.666An earlier sighting by the SELEX Collaboration[10] of a
$\Xi_{cc}^{+}$ candidate at $3519\hbox{ MeV}$ would imply
$m(\\{cc\\}[\bar{u}\bar{d}])=3876\hbox{ MeV}$, coincident with the threshold
for dissociation into a heavy-light pseudoscalar and heavy-light vector.
Signatures for weak decay would include $D^{+}K^{-}\ell^{+}\nu$ and
$\Xi_{c}^{+}\bar{n}$. The $D^{0}D^{+}\gamma$ channel opens at $3734\hbox{
MeV}$. This would be a $J^{P}=1^{+}$ axial-vector meson, symmetric in $cc$
flavor and antisymmetric in the light antiquark flavors.

In the absence of comprehensive experimental information about the other
doubly heavy baryons, we rely for now on model calculations of their masses
[11] as inputs to our tetraquark mass calculation. Our results for the lowest-
lying levels are given in Table 1.

Table 1: Expectations for ground-state tetraquark masses, in MeV. State | $J^{P}$ | $m(Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l})$ | Example Decay Channel |  $\mathcal{Q}$ [MeV]
---|---|---|---|---
$\\{cc\\}[\bar{u}\bar{d}]$ | $1^{+}$ | $3978$ |  $D^{+}{D}^{*0}$ 3876 | $102$
$\\{cc\\}[\bar{q}_{k}\bar{s}]$ | $1^{+}$ | $4156$ |  $D^{+}{D}^{*+}_{s}$ $3977$ | $179$
$\\{cc\\}\\{\bar{q}_{k}\bar{q}_{l}\\}$ | $0^{+},1^{+},2^{+}$ | $4146,4167,4210$ |  $D^{+}{D^{0}},D^{+}{D}^{*0}$ $3734,3876$ | $412,292,476$
$[bc][\bar{u}\bar{d}]$ | $0^{+}$ | $7229$ |  $B^{-}D^{+}/B^{0}D^{0}$ $7146$ | $83$
$[bc][\bar{q}_{k}\bar{s}]$ | $0^{+}$ | 7406 |  $B_{s}D$ $7236$ | 170
$[bc]\\{\bar{q}_{k}\bar{q}_{l}\\}$ | $1^{+}$ | $7439$ |  $B^{*}D/BD^{*}$ $7190/7290$ | $249$
$\\{bc\\}[\bar{u}\bar{d}]$ | $1^{+}$ | $7272$ |  $B^{*}D/BD^{*}$ $7190/7290$ | $82$
$\\{bc\\}[\bar{q}_{k}\bar{s}]$ | $1^{+}$ | 7445 |  $DB_{s}^{*}$ 7282 | 163
$\\{bc\\}\\{\bar{q}_{k}\bar{q}_{l}\\}$ | $0^{+},1^{+},2^{+}$ | $7461,7472,7493$ |  $BD/B^{*}D$ $7146/7190$ | $317,282,349$
$\\{bb\\}[\bar{u}\bar{d}]$ | $1^{+}$ | $10482$ |  $B^{-}\bar{B}^{*0}$ $10603$ | $\mathbf{-121}$
$\\{bb\\}[\bar{q}_{k}\bar{s}]$ | $1^{+}$ | $10643$ |  $\bar{B}\bar{B}_{s}^{*}/\bar{B}_{s}\bar{B}^{*}$ $10695/10691$ | $\mathbf{-48}$
$\\{bb\\}\\{\bar{q}_{k}\bar{q}_{l}\\}$ | $0^{+},1^{+},2^{+}$ | $10674,10681,10695$ |  $B^{-}{B^{0}},B^{-}{B}^{*0}$ $10559,10603$ | $115,78,136$

We find two real-world candidates for stable tetraquarks: the axial vector
$\\{bb\\}[\bar{u}\bar{d}]$ meson, $\mathcal{T}^{\\{bb\\}-}_{[\bar{u}\bar{d}]}$
bound by $121\hbox{ MeV}$, and the axial vector $\\{bb\\}[\bar{u}\bar{s}]$ and
$\\{bb\\}[\bar{d}\bar{s}]$ mesons bound by $48\hbox{ MeV}$. Given the
provisional doubly heavy baryon masses, we expect all the other
$Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ tetraquarks to lie at least $78\hbox{ MeV}$
above the corresponding thresholds for strong decay.777In model calculations,
Karliner and Rosner[12] estimate somewhat deeper binding, and so point to
additional $bc$ and $cc$ candidates. We note that exploratory lattice studies
also suggest that double-beauty tetraquarks should be stable.[13, 14]
Promising final states include
$\mathcal{T}^{\\{bb\\}}_{[\bar{u}\bar{d}]}(10482)^{-}\\!\to\Xi^{0}_{bc}\bar{p}$,
$B^{-}D^{+}\pi^{-}$, and $B^{-}D^{+}\ell^{-}\bar{\nu}$ (which establishes a
weak decay),
$\mathcal{T}^{\\{bb\\}}_{[\bar{u}\bar{s}]}(10643)^{-}\\!\to\Xi^{0}_{bc}\bar{\Sigma}^{-}$,
$\mathcal{T}^{\\{bb\\}}_{[\bar{d}\bar{s}]}(10643)^{0}\\!\to\Xi^{0}_{bc}(\bar{\Lambda},\bar{\Sigma}^{0})$,
and so on.

If they should lie near enough to threshold, the _unstable_ doubly heavy
tetraquarks might be observed in “wrong-sign” (double flavor) combinations
bearing $DD,DB,\hbox{or }BB$ quantum numbers. For example, a
$J^{P}=1^{+}\;\mathcal{T}^{\\{cc\\}}_{[\bar{d}\bar{s}]}(4156)^{++}\\!\to
D^{+}D_{s}^{*+}$ resonance would constitute _prima facie evidence_ for a
non-$q\bar{q}$ level carrying double charge and double charm. This would be a
new kind of resonance, for which no attractive force is present at the
meson–meson level. Other nearly bound candidates include
$1^{+}\;\mathcal{T}^{\\{bb\\}}_{\\{\bar{q}_{k}\bar{q}_{l}\\}}(10681)^{0,-,--}$
($\mathcal{Q}=+78\hbox{ MeV}$),
$1^{+}\;\mathcal{T}^{\\{bc\\}}_{[\bar{u}\bar{d}]}(7272)^{0}$
($\mathcal{Q}=+82\hbox{ MeV}$),
$0^{+}\;\mathcal{T}^{[bc]}_{[\bar{u}\bar{d}]}(7229)^{0}$
($\mathcal{Q}=+83\hbox{ MeV}$), and
$1^{+}\;\mathcal{T}^{\\{cc\\}}_{[\bar{u}\bar{d}]}(3978)^{+}$
($\mathcal{Q}=+102\hbox{ MeV}$).

The production of stable doubly heavy tetraquarks (or their nearly bound
counterparts) is undoubtedly a rare event, since it entails—at a start—the
production of two heavy quarks and two heavy antiquarks. We have no rate
calculation to offer, but note the large yield of $B_{c}$ mesons in the LHC$b$
experiment:[15] $8995\pm 103$ $B_{c}\to J\\!/\\!\psi\mu\nu_{\mu}X$ candidates
in $2\hbox{ fb}^{-1}$ of $pp$ collisions at $8\hbox{ TeV}$, and the CMS
observation[16] of double-$\Upsilon$ production in 8-TeV $pp$ collisions:
$\sigma(pp\to\Upsilon\Upsilon+\hbox{ anything})=68\pm 15\hbox{ pb}$. These
suggest that the Large Hadron Collider experiments should be the first focus
of searches for novel tetraquark mesons. The ultimate search instrument might
be a future electron–positron Tera-$Z$ factory, for which the branching
fractions [17] $Z\to b\bar{b}=15.12\pm 0.05\%$ and $Z\to
b\bar{b}b\bar{b}=(3.6\pm 1.3)\times 10^{-4}$ encourage the hope of many events
containing multiple heavy quarks.

Two recent investigations go beyond the kinds of arguments I have presented
here. Beginning from a situation in which all the constituents are taken to be
heavy, so that one-gluon exchange prevails, Czarnecki and collaborators have
proposed a figure of merit that governs the color-($\mathbf{\bar{3},6}$)
admixture in the putative diquark system.[18] They conclude that no stable
$QQ\bar{Q}\bar{Q}$ (equal-mass) tetraquarks are to be expected in very-heavy-
quark limit, and they find support for the binding of $bb\bar{q}\bar{q}$, in
agreement with our conclusions. A generalization allows them to explore how
the result depends on $N_{\mathrm{c}}$, the number of colors. A lattice–NRQCD
study of the $bb\bar{b}\bar{b}$ system reveals no tetraquark with mass below
$\eta_{b}\eta_{b}$, $\eta_{b}\Upsilon$, $\Upsilon\Upsilon$ thresholds in the
$J^{PC}=0^{++},1^{+-},2^{++}$ channels.[19]

##  3 Some tasks to advance our understanding

_Homework for Experiment._ The most straightforward request is to look for
double-flavor resonances of two heavy–light mesons near threshold. The
ingredients for such searches should already exist in experiments that have
reconstructed many $D$, $D_{s}$, $B$, and $B_{s}$ mesons. Next, extend to
$\sqrt{s}=13\hbox{ TeV}$ the measurement of representative cross sections for
final states containing two heavy quarks and two heavy antiquarks. Then we
need to discover and determine the masses of doubly-heavy baryons. These
masses are essential “engineering information” for our purposes, as they are
needed to implement the heavy-quark–symmetry calculation of tetraquark
masses.888Doubly heavy baryons are of considerable interest in their own
right. A light quark bound to a doubly heavy diquark has much in common—in
both color configuration and dynamics—with a heavy–light meson. A further goal
is to observe excitations of the diquark core, along with the energy levels of
the bound light quark. An important element of the study of doubly heavy
baryons is to resolve the conundrum of the large mass difference between the
$\Xi_{cc}^{+}$ and $\Xi_{cc}^{++}$ candidates reported by SELEX and LHC$b$,
respectively. The ultimate experimental goal is to find stable tetraquarks
through their weak decays.

_Homework for Theory._ An important challenge is to develop expectations for
the production of final states containing
$Q_{i},\bar{Q}_{i},Q_{j},\bar{Q}_{j}$, and eventually for the anticipated
stable tetraquarks. For the stable $Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ states
we discuss here, refine lifetime estimates beyond the simplest guess-by-
analogy of $\tau\approx 1/3\hbox{ ps}$. Extend the considerations of Refs. 6,
18 to understand how color configurations evolve with $QQ$ (and
$\bar{q}\bar{q}$) masses. Continue to explore how diquarks influence hadron
spectroscopy, by analyzing the stability of different body plans in the heavy-
quark limit. A notable example is a possible
$(Q_{i}Q_{j})(Q_{k}Q_{l})(Q_{m}Q_{n})$ dibaryon, with
$\bar{Q}_{p}\bar{Q}_{q}\bar{Q}_{r}$ color structure.

##  4 Summary

In the limit of very heavy quarks $Q$, novel narrow doubly heavy tetraquark
states must exist. Heavy-quark symmetry relates the doubly heavy tetraquark
mass to the masses of a doubly heavy baryon, heavy-light-light baryon, and
heavy-light meson. In the future, when we have more complete experimental
knowledge of the doubly heavy baryon spectrum, the heavy-quark–symmetry
relations should provide the most reliable predictions of doubly heavy
tetraquark masses. Our current mass estimates—which must rely on plausible
model inputs for the doubly heavy baryon masses—lead us to expect that the
lightest $J^{P}=1^{+}$ $\\{bb\\}[\bar{u}\bar{d}]$, $\\{bb\\}[\bar{u}\bar{s}]$,
and $\\{bb\\}[\bar{d}\bar{s}]$ states should be exceedingly narrow, decaying
only through the charged-current weak interaction. The observation of these
novel tetraquark mesons would herald a new form of stable matter, in which the
doubly heavy color-$\mathbf{\bar{3}}$ $(Q_{i}Q_{j})$ diquark is a basic
building block. Unstable $Q_{i}Q_{j}\bar{q}_{k}\bar{q}_{l}$ tetraquarks with
small $\mathcal{Q}$-values may be observable as resonant pairs of heavy-light
mesons in channels with double flavor: $DD,DB,BB$.

## Acknowledgments

Je souhaite dire un très grand merci aux gentils organisateurs, à nos amies
sauvetrices du secretariat, au personnel du Planibel, à Kim et Van, et à tous
les fondateurs des Rencontres de Moriond. I am grateful to the Delta Institute
for Theoretical Physics and Nikhef, the National Institute for Subatomic
Physics, for generous hospitality in Amsterdam, where this note was prepared.
I thank Estia Eichten for collaboration on the work reported here. This
manuscript has been authored by Fermi Research Alliance, LLC under Contract
No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of Science,
Office of High Energy Physics.

## References

## References

  * [1] S. K. Choi et al. [Belle], Phys. Rev. Lett. 91 (2003) 262001 [hep-ex/0309032].
  * [2] The extensive literature may be traced from the recent review by S. L. Olsen, T. Skwarnicki, and D. Zieminska, Rev. Mod. Phys. 90 (2018) 015003 [arXiv:1708.04012].
  * [3] G. Zweig, CERN-TH-401 (1964); CERN-TH-412 (1964). M. Gell-Mann, Phys. Lett. 8 (1964) 214.
  * [4] E. J. Eichten and C. Quigg, Phys. Rev. Lett. 119 (2017) 202002 [arXiv:1707.09575]. For a very early indication that such states could arise, see J.-P. Ader, J.-M. Richard and P. Taxil, Phys. Rev. D 25 (1982) 2370.
  * [5] W. E. Caswell and G. P. Lepage, Phys. Lett. 167B (1986) 437.
  * [6] J.-M. Richard, A. Valcarce, & J. Vijande, Phys. Rev. C 97 (2018) 035211 [arXiv:1803.06155].
  * [7] A. Peters et al., PoS LATTICE 2015 (2016) 095 [arXiv:1508.00343].
  * [8] S. Q. Luo et al., Eur. Phys. J. C 77 (2017) 709 [arXiv:1707.01180].
  * [9] R. Aaij et al. [LHC$b$], Phys. Rev. Lett. 119 (2017) 112001 [arXiv:1707.01621].
  * [10] M. Mattson et al. [SELEX], Phys. Rev. Lett. 89 (2002) 112001 [hep-ex/0208014].
  * [11] M. Karliner and J. L. Rosner, Phys. Rev. D 90 (2014) 094007 [arXiv:1408.5877].
  * [12] M. Karliner and J. L. Rosner, Phys. Rev. Lett. 119 (2017) 202001 [arXiv:1707.07666].
  * [13] P. Bicudo et al., Phys. Rev. D 93 (2016) 034501 [arXiv:1510.03441].
  * [14] A. Francis et al., Phys. Rev. Lett. 118 (2017) 142001 [arXiv:1607.05214].
  * [15] R. Aaij et al. [LHC$b$ Collaboration], Eur. Phys. J. C 74 (2014) 2839 [arXiv:1401.6932].
  * [16] V. Khachatryan et al. [CMS Collaboration], JHEP 1705 (2017) 013 [arXiv:1610.07095].
  * [17] C. Patrignani et al. [Particle Data Group], Chin. Phys. C 40 (2016) 100001.
  * [18] A. Czarnecki, B. Leng, & M. B. Voloshin, Phys. Lett. B 778 (2018) 233 [arXiv:1708.04594].
  * [19] C. Hughes, E. Eichten, & C. T. H. Davies, Phys. Rev. D 97 (2018) 054505 [arXiv:1710.03236].


# Evaluating Induced CCG Parsers on Grounded Semantic Parsing

Yonatan Bisk1 Siva Reddy211footnotemark: 1 John Blitzer3 Julia Hockenmaier4
Mark Steedman2
1ISI, University of Southern California
2ILCC, School of Informatics, University of Edinburgh
3Google, Mountain View
4Department of Computer Science, University of Illinois at Urbana-Champaign
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS> Equal contribution

###### Abstract

We compare the effectiveness of four different syntactic CCG parsers for a
semantic slot-filling task to explore how much syntactic supervision is
required for downstream semantic analysis. This extrinsic, task-based
evaluation provides a unique window to explore the strengths and weaknesses of
semantics captured by unsupervised grammar induction systems. We release a new
Freebase semantic parsing dataset called SPADES (Semantic PArsing of
DEclarative Sentences) containing 93K cloze-style questions paired with
answers. We evaluate all our models on this dataset. Our code and data are
available at https://github.com/sivareddyg/graph-parser.

##  1 Introduction

The past several years have seen significant progress in unsupervised grammar
induction [Carroll and Charniak (1992, Yuret (1998, Klein and Manning (2004,
Spitkovsky et al. (2010, Garrette et al. (2015, Bisk and Hockenmaier (2015].
But how useful are unsupervised syntactic parsers for downstream NLP tasks?
What phenomena are they able to capture, and where would additional annotation
be required? Instead of standard intrinsic evaluations – attachment scores
that depend strongly on the particular annotation styles of the gold treebank
– we examine the utility of unsupervised and weakly supervised parsers for
semantics. We perform an extrinsic evaluation of unsupervised and weakly
supervised CCG parsers on a grounded semantic parsing task that will shed
light on the extent to which these systems recover semantic information. We
focus on English to perform a direct comparison with supervised parsers
(although unsupervised or weakly supervised approaches are likely to be most
beneficial for domains or languages where supervised parsers are not
available).

$\begin{array}[t]{@{}*{5}{c}}{\rm Google}&{\rm acquired}&{\rm Nest}&{\rm
in}&\rm{2014}\\\
\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\
\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(S\backslash
NP)/NP}$}&\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{((S\backslash
NP)\backslash(S\backslash NP))/NP}$}&\mbox{$\mathsf{NP}$}\\\
&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\
&\lx@intercol\hfil\mbox{$\mathsf{S\backslash
NP}$}\hfil\lx@intercol&\lx@intercol\hfil\mbox{$\mathsf{(S\backslash
NP)\backslash(S\backslash NP)}$}\hfil\lx@intercol\\\
&\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\
&\lx@intercol\hfil\mbox{$\mathsf{S\backslash NP}$}\hfil\lx@intercol\\\
\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\
\lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\end{array}$

(a) in 2014 modifies acquired Nest

$\begin{array}[t]{@{}*{5}{c}}{\rm Google}&{\rm acquired}&{\rm Nest}&{\rm
in}&\rm{2014}\\\
\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\
\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(S\backslash
NP)/NP}$}&\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(S\backslash
S)/NP}$}&\mbox{$\mathsf{NP}$}\\\
&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\
&\lx@intercol\hfil\mbox{$\mathsf{S\backslash
NP}$}\hfil\lx@intercol&\lx@intercol\hfil\mbox{$\mathsf{S\backslash
S}$}\hfil\lx@intercol\\\ \lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\
\lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\\\
\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\
\lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\end{array}$

(b) in 2014 modifies Google acquired Nest

$\begin{array}[t]{@{}*{5}{c}}{\rm Google}&{\rm acquired}&{\rm Nest}&{\rm
in}&\rm{2014}\\\
\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\
\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{((S\backslash
NP)/PP)/NP}$}&\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{PP/NP}$}&\mbox{$\mathsf{NP}$}\\\
&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\
&\lx@intercol\hfil\mbox{$\mathsf{(S\backslash
NP)/PP}$}\hfil\lx@intercol&\lx@intercol\hfil\mbox{$\mathsf{PP}$}\hfil\lx@intercol\\\
&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\
&\lx@intercol\hfil\mbox{$\mathsf{S\backslash NP}$}\hfil\lx@intercol\\\
\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\
\lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\end{array}$

(c) acquired Google takes the argument in 2014

Figure 1: Example of multiple valid derivations that can be grounded to the
same Freebase logical form (Eq. 1) even though they differ dramatically in
performance under parsing metrics (5, 4, or 3 “correct” supertags).

Specifically, we evaluate different parsing scenarios with varying amounts of
supervision. These are designed to shed light on the question of how well
syntactic knowledge correlates with performance on a semantic evaluation. We
evaluate the following scenarios (all of which assume POS-tagged input): 1) no
supervision; 2) a lexicon containing words mapped to CCG categories; 3) a
lexicon containing POS tags mapped to CCG categories; 4) sentences annotated
with CCG derivations (i.e., fully supervised). Our evaluation reveals which
constructions are problematic for unsupervised parsers (and annotation efforts
should focus on). Our results indicate that unsupervised syntax is useful for
semantics, while a simple semi-supervised parser outperforms a fully
unsupervised approach, and could hence be a viable option for low resource
languages.

##  2 CCG Intrinsic Evaluations

CCG [Steedman (2000] is a lexicalized formalism in which words are assigned
syntactic types, also known as supertags, encoding subcategorization
information. Consider the sentence Google acquired Nest in 2014, and its CCG
derivations shown in Figure 1. In (a) and (b), the supertag of acquired,
$\mathsf{(S\backslash NP)/NP}$, indicates that it has two arguments, and the
prepositional phrase in 2014 is an adjunct, whereas in (c) the supertag
$\mathsf{((S\backslash NP)/PP)/NP}$ indicates acquired has three arguments
including the prepositional phrase. In (a) and (b), depending on the supertag
of in, the derivation differs. When trained on labeled treebanks, (a) is
preferred. However note that all these derivations could lead to the same
semantics (e.g., to the logical form in Equation 1). Without syntactic
supervision, there may not be any reason for the parser to prefer one analysis
over the other. One procedure to evaluate unsupervised induction methods has
been to compare the assigned supertags to treebanked supertags, but this
evaluation does not consider that multiple derivations could lead to the same
semantics. This problem is also not solved by evaluating syntactic
dependencies. Moreover, while many dependency standards agree on the head
direction of simple constituents (e.g., noun phrases) they disagree on the
most semantically useful ones (e.g., coordination and relative
clauses).111Please see ?) for more details.

##  3 Our Proposed Evaluation

The above syntax-based evaluation metrics conceal the real performance
differences and their effect on downstream tasks. Here we propose an extrinsic
evaluation where we evaluate our ability to convert sentences to Freebase
logical forms starting via CCG derivations. Our motivation is that most
sentences can only have a single realization in Freebase, and any derivation
that could lead to this realization is potentially a correct derivation. For
example, the Freebase logical form for the example sentence in Figure 1 is
shown below, and none of its derivations are penalized if they could result in
this logical form.

$\displaystyle\lambda e.\,\mathrm{business.acquisition}(e)$ (1)
$\displaystyle\wedge\mathrm{acquiring\\_company}(e,\textsc{Google})$
$\displaystyle\wedge\mathrm{company\\_acquired}(e,\textsc{Nest})$
$\displaystyle\wedge\mathrm{date}(e,\textsc{2014})$

$\begin{array}[t]{@{}*{5}{c}}{\rm Google}&{\rm acquired}&\langle{\tt blank}\rangle&{\rm which}&\rm{was\,founded\,in\,PA}\\\ \lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\ \mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(S\backslash NP)/NP}$}&\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(NP\backslash NP)/(S\backslash NP)}$}&\mbox{$\mathsf{S\backslash NP}$}\\\ &&&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\ &&&\lx@intercol\hfil\mbox{$\mathsf{NP\backslash NP}$}\hfil\lx@intercol\\\ &&\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\ &&\lx@intercol\hfil\mbox{$\mathsf{NP}$}\hfil\lx@intercol\\\ &\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\ &\lx@intercol\hfil\mbox{$\mathsf{S\backslash NP}$}\hfil\lx@intercol\\\ \lx@intercol\hfil\hrulefill_{<}\hfil\lx@intercol\\\ \lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\end{array}$ |
---|---
$\lambda e_{1}.\exists
xe_{2}.\,\textsc{target}(x)\wedge\mathrm{acquired}(e_{1})\wedge
arg_{1}(e_{1},\mathrm{Google})\wedge arg_{2}(e_{1},x)$
$\wedge\,\mathrm{founded}(e_{2})\wedge\mathbf{arg_{2}(e_{2}},x)\wedge\mathrm{in}(e_{2},\mathrm{PaloAlto})$
$\begin{array}[t]{@{}*{5}{c}}{\rm Google}&{\rm acquired}&\langle{\tt blank}\rangle&{\rm which}&\rm{was\,founded\,in\,PA}\\\ \lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol&\lx@intercol\hrulefill\hfil\hfil\lx@intercol\\\ \mbox{$\mathsf{NP}$}&\mbox{$\mathsf{(S\backslash NP)/NP}$}&\mbox{$\mathsf{NP}$}&\mbox{$\mathsf{((S\backslash NP)\backslash(S\backslash NP))/(S\backslash NP)}$}&\mbox{$\mathsf{S\backslash NP}$}\\\ &\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol&\lx@intercol\hfil\hrulefill{}_{>}\hfil\lx@intercol\\\ &\lx@intercol\hfil\mbox{$\mathsf{S\backslash NP}$}\hfil\lx@intercol&\lx@intercol\hfil\mbox{$\mathsf{(S\backslash NP)\backslash S\backslash NP}$}\hfil\lx@intercol\\\ &\lx@intercol\hfil\hrulefill{}_{<}\hfil\lx@intercol\\\ &\lx@intercol\hfil\mbox{$\mathsf{S\backslash NP}$}\hfil\lx@intercol\\\ \lx@intercol\hfil\hrulefill_{<}\hfil\lx@intercol\\\ \lx@intercol\hfil\mbox{$\mathsf{S}$}\hfil\lx@intercol\end{array}$ |
$\lambda e_{1}.\exists
xe_{2}.\,\textsc{target}(x)\wedge\mathrm{acquired}(e_{1})\wedge
arg_{1}(e_{1},\mathrm{Google})\wedge
arg_{2}(e_{1},x)\wedge\,\mathrm{founded}(e_{2})\wedge\mathbf{arg_{2}(e_{2}},\mathrm{Google})\wedge\mathrm{in}(e_{2},\mathrm{PaloAlto})$
Figure 2: The lexical categories for which determine the relative clause
attachment and therefore the resulting ungrounded logical form. The top
derivation correctly executes a query to retrieve companies founded in Palo
Alto and acquired by Google. The bottom incorrectly asserts that Google was
founded in Palo Alto.

Since grammar induction systems are traditionally trained on declarative
sentences, we would ideally require declarative sentences paired with Freebase
logical forms. But such datasets do not exist in the Freebase semantic parsing
literature [Cai and Yates (2013, Berant et al. (2013]. To alleviate this
problem, and yet perform Freebase semantic parsing, we propose an entity slot-
filling task.

#### Entity Slot-Filling Task.

Given a declarative sentence containing mentions of Freebase entities, we
randomly remove one of the mentions to create a blank slot. The task is to
fill this slot by translating the declarative sentence into a Freebase query.
Consider the following sentence where the entity Nest has been removed:

Google acquired  which was founded in Palo Alto

To correctly fill in the blank, one has to query Freebase for the entities
acquired by Google (constraint 1) and founded in Palo Alto (constraint 2). If
either of those constraints are not applied, there will be many entities as
answers. For each question, we execute a single Freebase query containing all
the constraints and retrieve a list of answer entities. From this list, we
pick the first entity as our predicted answer, and consider the prediction as
correct if the gold answer is the same as the predicted answer.

##  4 Sentences to Freebase Logical Forms

CCG provides a clean interface between syntax and semantics, i.e. each
argument of a word’s syntactic category corresponds to an argument of the
lambda expression that defines its semantic interpretation (e.g., the lambda
expression corresponding to the category $\mathsf{(S\backslash NP)/NP}$ of the
verb acquired is $\lambda f.\lambda g.\lambda e.\exists x.\exists
y.\mathrm{acquired}(e)\wedge f(x)\wedge g(y)\wedge arg_{1}(e,y)\wedge
arg_{2}(e,x)$), and the logical form for the complete sentence can be
constructed by composing word level lambda expressions following the syntactic
derivation [Bos et al. (2004]. In Figure 2 we show two syntactic derivations
for the same sentence, and the corresponding logical forms and equivalent
graph representations derived by GraphParser [Reddy et al. (2014]. The graph
representations are possible because GraphParser assumes access to co-
indexations of input CCG categories. We provide co-indexation for all induced
categories, including multiple co-indexations when an induced category is
ambiguous. For example, $\mathsf{(S\backslash N)/(S\backslash N)}$ refers to
either $\mathsf{(S_{x}\backslash N_{y})/(S_{x}\backslash N_{y})}$ indicating
an auxiliary verb or $\mathsf{(S_{x}\backslash N_{y})/(S_{z}\backslash
N_{y})}$ indicating a control verb. Initially, the predicates in the
expression/graph will be based entirely on the surface form of the words in
the sentence. This is the “ungrounded” semantic representation.

Our next step is to convert these ungrounded graphs to Freebase graphs.222Note
that there is one-to-one correspondence between Freebase graphs and Freebase
logical forms. Like ?), we treat this problem as a graph matching problem.
Using GraphParser we retrieve all the Freebase graphs that are isomorphic to
the ungrounded graph, and select only the graphs that could correctly predict
the blank slot, as candidate graphs. Using these candidate graphs, we train a
structured perceptron that learns to rank grounded graphs for a given
ungrounded graph.333Please see Section 4.3 of ?) for details. We use
ungrounded predicate and Freebase predicate alignments as our features.

##  5 Experiments

###  5.1 Training and Evaluation Datasets

Our dataset SPADES (Semantic PArsing of DEclarative Sentences) is constructed
from the declarative sentences collected by ?) from ClueWeb09 [Gabrilovich et
al. (2013] based on the following constraints: 1) There exists at least one
isomorphic Freebase graph to the ungrounded representation of the input
sentence; 2) There are no variable nodes in the ungrounded graph (e.g., Google
acquired a company is discarded whereas Google acquired the company Nest is
selected). We split this data into training (85%), development (5%) and
testing (10%) sentences (Table 1). We introduce empty slots into these
sentences by randomly removing an entity. SPADES can be downloaded at
http://github.com/sivareddyg/graph-parser.

| Sentences | Tokens | Types | Entities
---|---|---|---|---
Train | 79,247 | 685,922 | 69,095 | 37,606
Dev | 4,763 | 41,102 | 9,306 | 4,358
Test | 9,309 | 80,437 | 15,180 | 7,431
Table 1: SPADES Corpus Statistics |  | CCGbank (Syntax) | Slot Filling (Semantics)
---|---|---|---
|  | LF1 | UF1 | 2 | 3 | 4 | Overall
| Sentences |  |  |  $\sim$6K |  $\sim$3K |  $\sim$600 |  $\sim$10K
| Bag-of-Words | – | – | 34.1 | 28.1 | 16.5 | 31.4
Syntax |  Unsupervised | 37.1 | 64.2 | 27.7 | 20.0 | 16.8 | 24.8
Semi-Supervised-PoS | 53.0 | 68.5 | 30.5 | 21.8 | 18.8 | 27.3
Semi-Supervised-Word | 53.5 | 68.9 | 30.7 | 25.5 | 18.7 | 28.4
Supervised | 84.2 | 91.0 | 32.7 | 29.0 | 20.2 | 30.9
Table 2: Syntactic and semantic evaluation of the parsing models. Left:
Simplified labeled F1 and undirected unlabeled F1 on CCGbank, Section 23.
Right: Slot filling performance (by number of entities per sentence).  Slot-
filling results are updated after the camera-ready submission. In the previous
version, instead of evaluating if the gold entity is same as the first
predicted entity, we mistakenly evaluated if the gold entity is present in the
list of predicted answer entities. However, the initial claims are still
valid. All other results and discussion are revised.

There has been other recent interest in similar datasets for sentence
completion [Zweig et al. (2012] and machine reading [Hermann et al. (2015],
but unlike other corpora our data is tied directly to Freebase and requires
the execution of a semantic parse to correctly predict the missing entity.
This is made more explicit by the fact that one third of the entities in our
test set are never seen during training, so without a general approach to
query creation and execution there is a limit on a system’s performance.

###  5.2 Our Models

We use different CCG parsers varying in the amounts of supervision. For the
Unsupervised scenario, we use ?)’s parser which exploits a small set of
universal rules to automatically induce and weight a large set of lexical
categories. For the semi-supervised, we explore two options – Semi-Supervised-
Word and Semi-Supervised-PoS. We use Bisk et al. in both settings but we
constrain its lexicon manually rather than inducing it from scratch. In the
former, we restrict the top 200 words in English to occur only with the CCG
categories that comprise 95% of the occurrences of a word’s use in Section 22
of WSJ/CCGbank. In the latter, we restrict the POS tags instead of words. For
the Supervised scenario, we use EasyCCG [Lewis and Steedman (2014] trained on
CCGbank.

Finally, in order to further demonstrate the amount of useful information
being learned by our parsers, we present a competitive Bag-of-Words baseline,
which is a perceptron classifier that performs “semantic parsing” by
predicting either a Freebase or a null relation between the empty slot and
every other entity in the sentence, using the words in the sentence as
features. This naive approach is competitive on simple sentences with only two
entities, rivaling even the fully supervised parser, but falters as complexity
increases.

###  5.3 Results and Discussion

Our primary focus is a comparison of intrinsic syntactic evaluation with our
extrinsic semantic evaluation. To highlight the differences we present Section
23 parsing performance for our four models in Table 2. Dependency performance
is evaluated on both the simplified labeled F1 of ?) and Undirected Unlabeled
F1.

Despite the supervised parser performing almost twice as well as the semi-
supervised parsers on CCGbank LF1 (53.5 vs 84.2), in our semantic evaluation
we see a comparatively small gain in performance (28.4 vs 30.9). It is
interesting that such weakly supervised models are able to achieve over 90% of
the performance of a fully supervised parser. To explore this further, we
break down the semantics performance of all our models by the number of
entities in a sentence. Each sentence has two, three, or four entities, one of
which will be dropped for prediction. The more entities there are in a
sentence, the more likely the models are to misanalyze a relation leading to
their making the wrong prediction. These results are presented on the right
side of Table 2. There are still notable discrepancies in performance, which
we analyze more closely in the next section.

Another interesting result is the drop in performance by the Bag-of-Words
Model. As the number of entities in the sentence increase, the model weakens,
performing worse than the unsupervised parser on sentences with four entities.
It becomes non-trivial for it to isolate which entities and relations should
be used for prediction. This seems to indicate that the unsupervised grammar
is capturing more useful syntactic/semantic information than what is available
from the words alone. Ensemble systems that incorporate syntax and a Bag-of-
Words baseline may yield even better performance.

###  5.4 The Benefits of Annotation

Figure 3: The effect of increasing number of lexical types in Semi-Supervised-
Word on syntax and semantics. When the lexicon grows past 200 lexical types
both syntax and semantics begins to degrade. We also observe there is a
correlation between syntactic parsing and semantic parsing performance.

The performance of Semi-Supervised-PoS and Semi-Supervised-Word suggests that
when resources are scarce, it is beneficial to create a even a small lexicon
of CCG categories. We analyze this further in Figure 3. Here we show how
performance changes as a function of the number of labeled lexical types. Our
values range from 0 to 1000 lexical types. We see syntactic improvements of
16pts and semantic gains of 4.33pts (16.8%) with 200 words, before performance
degrades. It is possible that increasing annotation may only benefit fully
supervised models. Finally, when computing the most frequent lexical types we
excluded commas. We found a drop in performance when restricting commas to the
category , (they are commonly $\mathsf{conj}$ in our data). Additional in-
domain knowledge might further improve performance.

| Error | Example
---|---|---
Prevalent | Incorrect conjunction | Stockholm, Sweden
Appositive | , a chemist ,
Introductory clauses | In Frankfurt, …
Reduced relatives | … , established in 1909, …
B&H 15 | Verb chains | is also headquartered
Possessive | Anderson ’s Foundation
PP Attachment | of the foundation in Vancouver
Table 3: Causes of semantic grounding errors with examples not previously
isolated via intrinsic evaluation.

###  5.5 Common Errors

?) performed an in-depth analysis of the types of categories learned and
correctly used by their models (the same models as this paper). Their analysis
was based on syntactic evaluation against CCGbank. In particular, they found
the most egregious “semantic” errors to be the misuse of verb chains,
possessives and PP attachment (bottom of Table 3). Since we now have access to
a purely semantic evaluation, we can therefore ask whether these errors exist
here, and how common they are. We do this analysis in two steps. First, we
manually analyzed parses for which the unsupervised model failed to predict
the correct semantics, but where the supervised parser succeeded. The top of
Table 3 presents several of the most common reasons for failure. These
mistakes were more mundane (e.g. incorrect use of a conjunction) than failures
to use complex CCG categories or analyze attachments.

Second, we can compare grammatical decisions made by the semi-supervised and
unsupervised parsers against EasyCCG on sentences they successfully grounded.
?) found that their unsupervised parser made mistakes on many very simple
categories. We found the same result. When evaluating our parsers against the
treebank we found the unsupervised model only correctly predicted transitive
verbs 20% of the time and adverbs 39% of the time. In contrast, on our data,
we produced the correct transitive category (according to EasyCCG) 65% of the
time, and the correct adverb 68% of the time. These correct parsing decisions
also lead to improved performance across many other categories (e.g.
prepositions). This is likely due to our corpus containing simpler
constructions. In contrast, auxiliary verbs, relative clauses, and commas
still proved difficult or harder than in the treebank. This implies that
future work should tailor the annotation effort to their specific domain
rather than relying on guidance solely from the treebank.

##  6 Conclusion

Our goal in this paper was to present the first semantic evaluation of induced
grammars in order to better understand their utility and strengths. We showed
that induced grammars are learning more semantically useful structure than a
Bag-of-Words model. Furthermore, we showed how minimal syntactic supervision
can provide substantial gains in semantic evaluation. Our ongoing work
explores creating a syntax-semantics loop where each benefits the other with
no human (annotation) in the loop.

## Acknowledgments

This paper is partly based on work that was done when the first and second
authors were interns at Google, and on work that that was supported by NSF
grant 1053856 to JH, and a Google PhD Fellowship to SR.

## References

  * [Berant et al. (2013]  Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.  2013\.  Semantic parsing on Freebase from question-answer pairs.  In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October.
  * [Bisk and Hockenmaier (2013]  Yonatan Bisk and Julia Hockenmaier.  2013\.  An HDP Model for Inducing Combinatory Categorial Grammars.  Transactions of the Association for Computational Linguistics, pages 75–88.
  * [Bisk and Hockenmaier (2015]  Yonatan Bisk and Julia Hockenmaier.  2015\.  Probing the linguistic strengths and limitations of unsupervised grammar induction.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, Beijing,China, July.
  * [Bos et al. (2004]  Johan Bos, Stephen Clark, Mark Steedman, James R Curran, and Julia Hockenmaier.  2004\.  Wide-coverage semantic representations from a CCG parser.  In Proceedings of the 20th international conference on Computational Linguistics, page 1240.
  * [Cai and Yates (2013]  Qingqing Cai and Alexander Yates.  2013\.  Semantic parsing freebase: Towards open-domain semantic parsing.  In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 328–338, Atlanta, Georgia, USA, June.
  * [Carroll and Charniak (1992]  Glenn Carroll and Eugene Charniak.  1992\.  Two Experiments on Learning Probabilistic Dependency Grammars from Corpora.  Working Notes of the Workshop Statistically-Based NLP Techniques, pages 1–15, March.
  * [Gabrilovich et al. (2013]  Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya.  2013\.  FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0), June.
  * [Garrette et al. (2015]  Dan Garrette, Chris Dyer, Jason Baldridge, and Noah A Smith.  2015\.  Weakly-Supervised Grammar-Informed Bayesian CCG Parser Learning.  In Proceedings of the Association for the Advancement of Artificial Intelligence.
  * [Hermann et al. (2015]  Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom.  2015\.  Teaching machines to read and comprehend.  In Advances in Neural Information Processing Systems 28, pages 1693–1701.
  * [Klein and Manning (2004]  Dan Klein and Christopher D Manning.  2004\.  Corpus-Based Induction of Syntactic Structure: Models of Dependency and Constituency.  In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume, pages 478–485, Barcelona, Spain, July.
  * [Lewis and Steedman (2014]  Mike Lewis and Mark Steedman.  2014\.  A* CCG Parsing with a Supertag-factored Model.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 990–1000, Doha, Qatar, October.
  * [Reddy et al. (2014]  Siva Reddy, Mirella Lapata, and Mark Steedman.  2014\.  Large-scale Semantic Parsing without Question-Answer Pairs.  Transactions of the Association for Computational Linguistics, pages 1–16, June.
  * [Reddy et al. (2016]  Siva Reddy, Oscar Täckström, Michael Collins, Tom Kwiatkowski, Dipanjan Das, Mark Steedman, and Mirella Lapata.  2016\.  Transforming Dependency Structures to Logical Forms for Semantic Parsing.  Transactions of the Association for Computational Linguistics, 4:127–140.
  * [Spitkovsky et al. (2010]  Valentin I Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky.  2010\.  From Baby Steps to Leapfrog: How “Less is More” in Unsupervised Dependency Parsing.  In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 751–759, Los Angeles, California, June.
  * [Steedman (2000]  Mark Steedman.  2000\.  The Syntactic Process.  The MIT Press, September.
  * [Yuret (1998]  Deniz Yuret.  1998\.  Discovery of Linguistic Relations Using Lexical Attraction.  Ph.D. thesis, Massachusetts Institute of Technology.
  * [Zweig et al. (2012]  Geoffrey Zweig, John C. Platt, Christopher Meek, Christopher J.C. Burges, Ainur Yessenalina, and Qiang Liu.  2012\.  Computational approaches to sentence completion.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 601–610, Jeju Island, Korea, July.