# Magnetic dipole moments of the $T_{cc}^{+}$ and $Z_{V}^{++}$ tetraquark
states

K. Azizi<EMAIL_ADDRESS>Department of Physics, University of Tehran,
North Karegar Avenue, Tehran 14395-547, Iran  Department of Physics, Dogus
University, Acibadem-Kadikoy, 34722 Istanbul, Turkey  U. Özdem
<EMAIL_ADDRESS>Health Services Vocational School of Higher Education,
Istanbul Aydin University, Sefakoy-Kucukcekmece, 34295 Istanbul, Turkey

###### Abstract

Inspired by the observation of the doubly charmed state $T_{cc}$ and following
theoretical studies on its spectroscopic parameters, we investigate its
magnetic dipole moment assigning it the quantum numbers $J^{P}=1^{+}$ and both
the compact diquark-antidiquark and molecular structures in the framework of
the light-cone QCD. We also calculate the magnetic dipole moment of the
theoretically predicted singly charmed state, $Z_{V}^{++}$, with two units of
electric charge and the quantum numbers $J^{P}=1^{-}$ in diquark-antidiquark
picture. The numerical results are obtained as
$\mu_{T_{cc}^{+}-Di}=0.66^{+0.34}_{-0.23}~{}\mu_{N}$,
$\mu_{T_{cc}^{+}-Mol}=0.43^{+0.23}_{-0.22}~{}\mu_{N}$ and
$\mu_{Z_{V}^{++}}=3.35^{+0.89}_{-0.73}~{}\mu_{N}$. These results may be
checked via other phenomenological approaches. The obtained results may be
useful in exact determinations of the natures of these states.

Magnetic dipole moment, $T_{cc}^{+}$ and $Z_{V}^{++}$ states, Light-cone QCD

##  I Introduction

Besides the standard mesons and baryons/antibaryons, formed of $q\bar{q}$ and
$qqq$/$\bar{q}\bar{q}\bar{q}$, there exist hadrons made of more than three
quarks/antiquarks or valence gluons. These categories of hadrons contain
exotic states including glueballs, hybrids, tetraquarks, pentaquarks,
hexaquarks, etc. In last two decades, a large number of these exotic states
have been observed in the particle factories. Although some of them are now
well established, there are some doubts on existence of some other members.
Exact determinations of the nature, structure and quantum numbers of these
states need more experimental efforts. Following the experiments, and
sometimes before the experimental results, these states are investigated by
many theoretical and phenomenological models and approaches. There are many
exotic states, proposed in theory, waiting to be confirmed by the experiments.
For the recent experimental and theoretical progresses on the exotic states
see, for instance, Refs. Faccini:2012pj ; Esposito:2014rxa ; Chen:2016qju ;
Ali:2017jda ; Esposito:2016noz ; Olsen:2017bmm ; Lebed:2016hpi ; Guo:2017jvc ;
Nielsen:2009uh ; Brambilla:2019esw ; Liu:2019zoy ; Agaev:2020zad ;
Dong:2021juy . Let us make a note here that most of heavy multi-quark
particles discovered in the experiment, to date, have hidden-charm or hidden-
bottom quark structure, i.e., they contain $c\bar{c}$ or $b\bar{b}$ in their
inner structures.

Very recently, the LHCb Collaboration reported observation of a first doubly
charmed tetraquark state $T_{cc}^{+}$ in the $D^{0}D^{0}\pi^{+}$ mass spectrum
with over 10$\sigma$ significance LHCb:2021vvq ; LHCb:2021auc . The simplest
assumption on its valence quark component is $cc\bar{u}\bar{d}$ and its spin-
parity was suggested by the experiment as $J^{P}=1^{+}$. Its mass, with
respect to the $D^{0}D^{\ast+}$ threshold, and width have been measured to be

$\displaystyle\delta m$
$\displaystyle=m_{T_{cc}^{+}}-(m_{D^{0}}+m_{D^{\ast+}})$ (1)
$\displaystyle=-273\pm 61\pm 5_{-14}^{+11}~{}\mathrm{KeV},$
$\displaystyle\Gamma$ $\displaystyle=410\pm 165\pm
43_{-38}^{+18}~{}\mathrm{KeV}.$ (2)

As we see, its decay width is very small and its decay to $D^{0}D^{0}\pi^{+}$
occur very slowly and with delay. Because of that this particle is the longest
living exotic state discovered till now. The discovery of the first doubly
charmed tetraquark state will, undoubtedly, usher in a new era in the study of
hadron spectroscopy and improve our understanding of the non-perturbative
nature of the strong interaction. Before and after the experimental discovery,
some spectroscopic properties and decay channels of the $T_{cc}^{+}$ state
have been investigated within different theoretical models Carames:2011zz ;
Richard:2018yrm ; Hernandez:2019eox ; Liu:2019stu ; Agaev:2021vur ; Li:2021zbw
; Yan:2021wdl ; Dong:2021bvy ; Feijoo:2021ppq ; Meng:2021jnw ; Ling:2021bir ;
Xin:2021wcr . Note that the spectroscopic parameters and different decay modes
of the scalar and pseudoscalar $T_{cc}$ states as well as the $T_{bc}$ and
$T_{bb}$ states of different quantum numbers were already investigated in
Refs. Agaev:2020mqq ; Agaev:2020zag ; Agaev:2020dba ; Agaev:2019lwh ;
Agaev:2019kkz ; Agaev:2019qqn ; Agaev:2018khe , which may be in agenda of
future experiments. Another important subclass of doubly charmed tetraquarks
includes particles that bear two units of electric charge. Spectroscopic
parameters and decay widths of such exotic states with $cc\bar{d}\bar{s}$ and
$cc\bar{s}\bar{s}$ quark contents were investigated in Ref. Agaev:2018vag .
These particles have not been observed experimentally yet, but their existence
is important for understanding the nature of exotic states. We calculate the
magnetic dipole moment of $T_{cc}^{+}$ state newly detected by LHCb
collaboration LHCb:2021vvq ; LHCb:2021auc  in order to shed light on its
nature and physical properties.

Recently, the LHCb collaboration announced observation of two new structures
$X_{0}(2900)$ and $X_{1}(2900)$ in the process $B^{+}\rightarrow
D^{+}D^{-}K^{+}$ Aaij:2020hon ; Aaij:2020ypa . These resonances appear as the
intermediate states of the decay chain $B^{+}\rightarrow D^{+}X\rightarrow
D^{+}D^{-}K^{+}$, and they are neutral four-quark states. These decays take
place due to color-favored and color-suppressed modes of the $B^{+}$ meson, as
also stated in Ref. Burns:2020xne . But weak decays of $B^{+}$ may provoke
also the decays $B^{+}\rightarrow D^{-}Z^{++}\rightarrow D^{-}D^{+}K^{+}$,
where $Z^{++}$ is a doubly charged exotic state with inner structure of
$cu\overline{s}\overline{d}$ (for more information see Agaev:2021jsz ). It is
expected that the LHCb collaboration observe the doubly charged tetraquarks
$Z^{++}=[cu][\overline{s}\overline{d}]$ with different spin-parities in the
above processes. Thus, the possible scalar and vector four-quark resonances
$Z_{\mathrm{S}}^{++}$ and $Z_{\mathrm{V}}^{++}$ may be found as resonances in
the $D^{+}K^{+}$ invariant mass. Motivated by this, such states are proposed
in Ref. Agaev:2021jsz  and the spectroscopic properties and possible decay
channels of $Z_{\mathrm{V}}^{++}$ state are investigated using two-point QCD
sum rules. Because of the importance of these states, in the second part of
this study, we investigate the electromagnetic properties of
$Z_{\mathrm{V}}^{++}$ state. Because of two units of electric charge, the
magnetic dipole moment of this state is expected to be large and easily
accessible in the future experiments. Note that the the magnetic dipole moment
of the charmed-strange $Z_{cs}(3985)$ tetraquark was calculated in Ref.
Ozdem:2021yvo  using the light-cone QCD sum rule formalism.

The electromagnetic multipole moments of hadrons, besides their spectroscopic
parameters, can help us determine their exact nature, substructure, and
quantum numbers. These parameters give useful information about the charge and
magnetization distributions inside the hadrons that help us gain information
on their geometric shapes. Recall that the magnetic and higher order moments
of particles that contain information about the spatial distributions of
charge and magnetization within the particles, are directly related to the
spatial distributions of quarks and gluons in hadrons.

In the present study, we calculate the magnetic dipole moments (MDM) of
$T_{cc}^{+}$ and $Z_{V}^{++}$ states in the compact diquark-antidiquark
picture using the light-cone QCD sum rule formalism. For the case of
$T_{cc}^{+}$, we also extract its magnetic dipole moment considering it as the
$D^{*}D+DD^{*}$ molecular state. We employ the distribution amplitudes (DAs)
of the on-shell real photon state at light cone, which are available in terms
of different twists.

The paper is organized in the following way: In sec. II, we calculate the MDM
of the $T_{cc}^{+}$ and $Z_{V}^{++}$ states. In section III, we numerically
analyze the MDM of the $T_{cc}^{+}$ and $Z_{V}^{++}$ states. The last section
is devoted to the summary of the results and conclusions.

##  II Formalism

In the light-cone QCD sum rule method we calculate a correlation function, as
building block of the method, once in terms of hadronic parameters such as
electromagnetic form factors, and second, in terms the QCD parameters and DAs
of the on-shell photon. By matching the coefficients of suitable Lorentz
structures from both sides and using the assumption of quark-hadron duality,
we will be able to evaluate the related hadronic observables in terms of QCD
degrees of freedom as well as auxiliary parameters entering the calculations
at different steps.

###  II.1 MDM of the $T_{cc}^{+}$ state

To evaluate the MDM of the $T_{cc}^{+}$ state within the LCSR, we start with
the correlation function

$\Pi_{\mu\nu}^{T_{cc}^{+}}(p,q)=i\int d^{4}xe^{ip\cdot x}\langle
0|\mathcal{T}\\{J_{\mu}^{T_{cc}^{+}}(x)J_{\nu}^{T_{cc}^{+}\dagger}(0)\\}|0\rangle_{\gamma},$
(3)

where the sub-index $\gamma$ represents the background electromagnetic field
and $J_{\mu}(x)$ is the interpolating current of the $T_{cc}^{+}$ state with
the quantum numbers $J^{P}=1^{+}$ and the quark content $cc\bar{u}\bar{d}$.
The $T_{cc}^{+}$ state can be interpolated by either compact axial vector
diquark-light scalar antidiquark or molecular $D^{*}D+DD^{*}$ structures as

$\displaystyle J_{\mu}^{T_{cc}^{+}-Di}(x)$ $\displaystyle=$ $\displaystyle
c_{a}^{T}(x)\gamma_{\mu}Cc_{b}(x)\bar{u}_{a}(x)C\gamma_{5}\bar{d}_{b}^{T}(x),$
(4) $\displaystyle J_{\mu}^{T_{cc}^{+}-Mol}(x)$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{2}}\\{[\bar{u}_{a}(x)i\gamma_{5}c_{a}(x)][\bar{d}_{b}(x)\gamma_{\mu}c_{b}(x)]+[\bar{u}_{a}(x)\gamma_{\mu}c_{a}(x)][\bar{d}_{b}(x)i\gamma_{5}c_{b}(x)]\\}.$

In Eq. (4), $u(x)$, $c(x)$ and $d(x)$ are the quark fields, and $C$ stands for
the charge-conjugation operator. In this equation the currents
$J_{\mu}^{T_{cc}^{+}-Di}(x)$ and $J_{\mu}^{T_{cc}^{+}-Mol}(x)$ stand for
compact diquark-antidiquark and molecular structures, respectively. In the
following, we will explicitly present the calculations only for
$J_{\mu}^{T_{cc}^{+}-Di}(x)$ case and for simplicity we will omit the label
$Di$ from the current. However, we do all the calculations in this subsection
in both pictures and will extract the numerical values of magnetic dipole
moment of $T_{cc}^{+}$ in both pictures.

In hadronic side, the correlation function is evaluated by its saturation with
the intermediate hadronic states. By performing the four-integral over $x$ we
get

$\displaystyle\Pi_{\mu\nu}^{Had-T_{cc}^{+}}(p,q)={\frac{\langle 0\mid
J_{\mu}^{T_{cc}^{+}}\mid
T_{cc}^{+}(p)\rangle}{p^{2}-m_{T_{cc}^{+}}^{2}}}\langle T_{cc}^{+}(p)\mid
T_{cc}^{+}(p+q)\rangle_{\gamma}\frac{\langle
T_{cc}^{+}(p+q)\mid{J^{\dagger}}_{\nu}^{T_{cc}^{+}}\mid
0\rangle}{(p+q)^{2}-m_{T_{cc}^{+}}^{2}}+\cdots,$ (5)

where q is the photon’s momentum and dots stand for the contributions coming
from the higher states and continuum. The matrix element $\langle 0\mid
J_{\mu}^{T_{cc}^{+}}\mid T_{cc}^{+}\rangle$ is parameterized as

$\displaystyle\langle 0\mid J_{\mu}^{T_{cc}^{+}}\mid
T_{cc}^{+}\rangle=\lambda_{T_{cc}^{+}}\varepsilon_{\mu}^{\theta}\,,$ (6)

with $\lambda_{T_{cc}^{+}}$ being the residue of the $T_{cc}^{+}$ state and
$\varepsilon_{\mu}^{\theta}\ $ is its polarization vector.

In the existence of the external electromagnetic background field, the matrix
element $\langle T_{cc}^{+}(p)\mid T_{cc}^{+}(p+q)\rangle_{\gamma}$ can be
expressed in terms of the Lorentz invariant form factors as follows
Brodsky:1992px :

$\displaystyle\langle T_{cc}^{+}(p,\varepsilon^{\theta})\mid
T_{cc}^{+}(p+q,\varepsilon^{\delta})\rangle_{\gamma}$
$\displaystyle=-\varepsilon^{\tau}(\varepsilon^{\theta})^{\alpha}(\varepsilon^{\delta})^{\beta}\Bigg{[}G_{1}(Q^{2})~{}(2p+q)_{\tau}~{}g_{\alpha\beta}+G_{2}(Q^{2})~{}(g_{\tau\beta}~{}q_{\alpha}-g_{\tau\alpha}~{}q_{\beta})$
$\displaystyle-\frac{1}{2m_{T_{cc}^{+}}^{2}}G_{3}(Q^{2})~{}(2p+q)_{\tau}~{}q_{\alpha}q_{\beta}\Bigg{]},$
(7)

where $\varepsilon^{\delta}$ and $\varepsilon^{\theta}$ are the polarization
vectors of the initial and final $T_{cc}^{+}$ states and $\varepsilon^{\tau}$
is the polarization vector of the photon. Here, $G_{1}(Q^{2})$, $G_{2}(Q^{2})$
and $G_{3}(Q^{2})$ are invariant form factors, with $Q^{2}=-q^{2}$.

Using Eqs. (5)-(II.1), the correlation function takes the form,

$\displaystyle\Pi_{\mu\nu}^{Had-T_{cc}^{+}}(p,q)$
$\displaystyle=\frac{\varepsilon_{\rho}\lambda_{T_{cc}^{+}}^{2}}{[m_{T_{cc}^{+}}^{2}-(p+q)^{2}][m_{T_{cc}^{+}}^{2}-p^{2}]}\Bigg{[}G_{2}(Q^{2})\Bigg{(}q_{\mu}g_{\rho\nu}-q_{\nu}g_{\rho\mu}-\frac{p_{\nu}}{m_{T_{cc}^{+}}^{2}}\big{(}q_{\mu}p_{\rho}-\frac{1}{2}Q^{2}g_{\mu\rho}\big{)}+$
$\displaystyle+\frac{(p+q)_{\mu}}{m_{T_{cc}^{+}}^{2}}\big{(}q_{\nu}(p+q)_{\rho}+\frac{1}{2}Q^{2}g_{\nu\rho}\big{)}-\frac{(p+q)_{\mu}p_{\nu}p_{\rho}}{m_{T_{cc}^{+}}^{4}}\,Q^{2}\Bigg{)}$
$\displaystyle+\mbox{other independent structures}\Bigg{]}\,+\cdots.$ (8)

To determine the MDM, the value of the form factor $G_{2}(Q^{2})$ is needed
only at $Q^{2}=0$. The magnetic form factor $F_{M}(Q^{2})$ is written as

$\displaystyle F_{M}(Q^{2})=G_{2}(Q^{2})\,,$ (9)

the value of which at $Q^{2}=0$, i.e. $F_{M}(0)$, is proportional to the MDM,
$\mu_{T_{cc}^{+}}$ :

$\displaystyle\mu_{T_{cc}^{+}}=\frac{e}{2\,m_{T_{cc}^{+}}}\,F_{M}(0).$ (10)

In QCD side, the correlation function in Eq. (3), is calculated in deep
Euclidean region in terms of QCD degrees of freedom as well as the DAs of the
photon. To this end, we substitute the explicit forms of the interpolating
currents in the correlation function and contract the corresponding quark
fields with the help of the Wick’s theorem. As a result, we get

$\displaystyle\Pi_{\mu\nu}^{\mathrm{QCD-T_{cc}^{+}}}(p,q)$ $\displaystyle=$
$\displaystyle i\int d^{4}xe^{ip\cdot x}\langle
0\mid\Big{\\{}\mathrm{Tr}[\gamma_{5}\widetilde{S}_{d}^{b^{\prime}b}(-x)\gamma_{5}S_{u}^{a^{\prime}a}(-x)]\mathrm{Tr}[\gamma_{\nu}\widetilde{S}_{c}^{aa^{\prime}}(x)\gamma_{\mu}S_{c}^{bb^{\prime}}(x)]$
(11)
$\displaystyle-\mathrm{Tr}[\gamma_{5}\widetilde{S}_{d}^{b^{\prime}b}(-x)\gamma_{5}S_{u}^{a^{\prime}a}(-x)]\mathrm{Tr}[\gamma_{\nu}\widetilde{S}_{c}^{ba^{\prime}}(x)\gamma_{\mu}S_{c}^{ab^{\prime}}(x)]\Big{\\}}\mid
0\rangle_{\gamma},$

where

$\widetilde{S}_{c(q)}^{ij}(x)=CS_{c(q)}^{ij\mathrm{T}}(x)C,$

with $S_{q(c)}(x)$ being the full light (charm) quark propagator. They are
given as

$\displaystyle S_{q}(x)$
$\displaystyle=i\frac{{x\\!\\!\\!/}}{2\pi^{2}x^{4}}-\frac{\langle\bar{q}q\rangle}{12}\Big{(}1-i\frac{m_{q}x\\!\\!\\!/}{4}\Big{)}-\frac{\langle\bar{q}q\rangle}{192}m_{0}^{2}x^{2}\Big{(}1-i\frac{m_{q}x\\!\\!\\!/}{6}\Big{)}-\frac{ig_{s}}{32\pi^{2}x^{2}}~{}G^{\mu\nu}(x)\Big{[}\hbox
to0.0pt{/\hss}{x}\sigma_{\mu\nu}+\sigma_{\mu\nu}\hbox
to0.0pt{/\hss}{x}\Big{]},$ (12) $\displaystyle S_{c}(x)$
$\displaystyle=\frac{m_{c}^{2}}{4\pi^{2}}\Bigg{[}\frac{K_{1}\Big{(}m_{c}\sqrt{-x^{2}}\Big{)}}{\sqrt{-x^{2}}}+i\frac{{x\\!\\!\\!/}~{}K_{2}\Big{(}m_{c}\sqrt{-x^{2}}\Big{)}}{(\sqrt{-x^{2}})^{2}}\Bigg{]}-\frac{g_{s}m_{c}}{16\pi^{2}}\int_{0}^{1}dv\,G^{\mu\nu}(vx)\Bigg{[}\big{(}\sigma_{\mu\nu}{x\\!\\!\\!/}+{x\\!\\!\\!/}\sigma_{\mu\nu}\big{)}$
$\displaystyle\times\frac{K_{1}\Big{(}m_{c}\sqrt{-x^{2}}\Big{)}}{\sqrt{-x^{2}}}+2\sigma_{\mu\nu}K_{0}\Big{(}m_{c}\sqrt{-x^{2}}\Big{)}\Bigg{]},$
(13)

where $\langle\bar{q}q\rangle$ is quark condensate, $m_{0}$ is defined through
the quark-gluon mixed condensate $\langle
0\mid\bar{q}\,g_{s}\,\sigma_{\alpha\beta}\,G^{\alpha\beta}\,q\mid
0\rangle=m_{0}^{2}\,\langle\bar{q}q\rangle$, $G^{\mu\nu}$ is the gluon field
strength tensor, $\sigma_{\mu\nu}=\frac{i}{2}[\gamma_{\mu},\gamma_{\nu}]$ and
$K_{i}$’s are modified Bessel functions of the second kind.

Finally, we choose the Lorentz invariant structure $q_{\mu}\varepsilon_{\nu}$
from the both sides and match its coefficients from both the hadronic and QCD
sides. To eliminate the contributions of the higher states and continuum, we
perform Borel transformation and continuum subtraction. The procedures are
lengthy but standard, we do not present the steps here and refer the reader
for instance to Ref. Azizi:2018duk . As a results, we obtain the light-cone
QCD sum rule for the MDM as

$\displaystyle\mu_{T_{cc}^{+}}\,\,\lambda_{T_{cc}^{+}}^{2}=e^{\frac{m_{T_{cc}^{+}}^{2}}{M^{2}}}\,\,\Pi^{QCD-
T_{cc}^{+}}(M^{2},s_{0}),$ (14)

where $M^{2}$ and $s_{0}$ are auxiliary parameters stemming from the
applications of the Borel transformation and continuum procedures. The
$\Pi^{QCD-T_{cc}^{+}}(M^{2},s_{0})$ function is quite lengthy, explicit
expression of which is not presented here.

###  II.2 MDM of the$Z_{V}^{++}$ state

As we mentioned in the previous subsection, calculations are started by
writing the appropriate correlation function,

$\Pi_{\mu\nu}^{Z_{V}^{++}}(p,q)=i\int d^{4}xe^{ip\cdot x}\langle
0|\mathcal{T}\\{J_{\mu}^{Z_{V}^{++}}(x)J_{\nu}^{Z_{V}^{++}\dagger}(0)\\}|0\rangle_{\gamma},$
(15)

where the $J_{\mu}^{Z_{V}^{++}}(x)$ is interpolating current of the
$Z_{V}^{++}$ state and it is given in the diquark-antidiquark picture with the
quantum number $J^{P}=1^{-}$ as

$J_{\mu}^{Z_{V}}(x)=\varepsilon\widetilde{\varepsilon}[u_{b}^{T}(x)C\gamma_{5}c_{c}(x)][\overline{s}_{d}(x)\gamma_{\mu}\gamma_{5}C\overline{d}_{e}^{T}(x)],$
(16)

where $\varepsilon\widetilde{\varepsilon}=\varepsilon^{abc}\varepsilon^{dec}$,
and $a$, $b$, $c$, $d$ and $e$ denote quark colors. In Eq. (16), $u(x)$,
$c(x)$, $s(x)$ and $d(x)$ are the quark fields, and $C$ stands for the charge-
conjugation operator.

To obtain required sum rules for parameters of $Z_{V}^{++}$ state, we have to
represent the correlation function $\Pi_{\mu\nu}^{Z_{V}^{++}}(p,q)$ with
respect to these parameters, and get the hadronic side of the sum rules
$\Pi_{\mu\nu}^{Had-Z_{V}^{++}}(p,q)$. In connection with the hadronic
parameters the correlation function has the subsequent form

$\displaystyle\Pi_{\mu\nu}^{Had-Z_{V}^{++}}(p,q)={\frac{\langle 0\mid
J_{\mu}^{{Z_{V}^{++}}}\mid{Z_{V}^{++}}(p)\rangle}{p^{2}-m_{{Z_{V}^{++}}}^{2}}}\langle{Z_{V}^{++}}(p)\mid{Z_{V}^{++}}(p+q)\rangle_{\gamma}\frac{\langle{Z_{V}^{++}}(p+q)\mid{J^{\dagger}}_{\nu}^{{Z_{V}^{++}}}\mid
0\rangle}{(p+q)^{2}-m_{{Z_{V}^{++}}}^{2}}+\cdots,$ (17)

where the dots stand for contributions of higher resonances and continuum
states. The matrix element of the electromagnetic current between the
$Z_{V}^{++}$ states is described by three form factors and it is presented in
Eq. (II.1). Using the corresponding equations, the result for the hadronic
side is obtained as follows

$\displaystyle\Pi_{\mu\nu}^{Had-Z_{V}^{++}}(p,q)$
$\displaystyle=\frac{\varepsilon_{\rho}\lambda_{Z_{V}^{++}}^{2}}{[m_{Z_{V}^{++}}^{2}-(p+q)^{2}][m_{Z_{V}^{++}}^{2}-p^{2}]}\Bigg{[}G_{2}(Q^{2})\Bigg{(}q_{\mu}g_{\rho\nu}-q_{\nu}g_{\rho\mu}-\frac{p_{\nu}}{m_{Z_{V}^{++}}^{2}}\big{(}q_{\mu}p_{\rho}-\frac{1}{2}Q^{2}g_{\mu\rho}\big{)}+$
$\displaystyle+\frac{(p+q)_{\mu}}{m_{Z_{V}^{++}}^{2}}\big{(}q_{\nu}(p+q)_{\rho}+\frac{1}{2}Q^{2}g_{\nu\rho}\big{)}-\frac{(p+q)_{\mu}p_{\nu}p_{\rho}}{m_{Z_{V}^{++}}^{4}}\,Q^{2}\Bigg{)}$
$\displaystyle+\mbox{other independent structures}\Bigg{]}\,+\cdots.$ (18)

The QCD side of the sum rules, $\Pi_{\mu\nu}^{\mathrm{QCD-Z_{V}^{++}}}(p,q)$,
is obtained by inserting the interpolating current $J_{\mu}(x)$ into Eq. (15),
and contracting proper quark fields via Wick’s theorem. After these steps, we
obtain

$\displaystyle\Pi_{\mu\nu}^{\mathrm{QCD-Z_{V}^{++}}}(p,q)=i\int
d^{4}xe^{ipx}\varepsilon\widetilde{\varepsilon}\varepsilon^{\prime}\widetilde{\varepsilon}^{\prime}\langle
0\mid\mathrm{Tr}[\gamma_{5}\widetilde{S}_{u}^{bb^{\prime}}(x)\gamma_{5}S_{c}^{cc^{\prime}}(x)]\mathrm{Tr}[\gamma_{\mu}\gamma_{5}\widetilde{S}_{d}^{e^{\prime}e}(-x)\gamma_{5}\gamma_{\nu}S_{s}^{d^{\prime}d}(-x)]\mid
0\rangle_{\gamma}.$ (19)

The sum rules for the MDM of the $Z_{V}^{++}$ state can be obtained by
matching the hadronic and QCD sides of the corresponding correlation function
and performing usual operations required in QCD sum rule calculations. As a
result of these processes, we get

$\displaystyle\mu_{Z_{V}^{++}}\,\,\lambda_{Z_{V}^{++}}^{2}=e^{\frac{m_{Z_{V}^{++}}^{2}}{M^{2}}}\,\,\Pi^{QCD-{Z_{V}^{++}}}(M^{2},s_{0}).$
(20)

The analytical calculations are ended here. Now, we are ready to move on the
numerical analysis part.

##  III Numerical analysis

In this section, we perform the numerical analysis of the sum rules for the
MDMs of the $T^{+}_{cc}$ and $Z_{V}^{++}$ states obtained in the previous
section. We use $m_{u}=m_{d}=0$, $m_{s}=96^{+8}_{-4}\,\mbox{MeV}$,
$m_{c}=(1.275\pm 0.025)\,$GeV, $m_{T_{cc}^{+}}=3868\pm 124~{}\mbox{MeV}$
Agaev:2021vur , $m_{Z_{V}^{++}}=3515\pm 125~{}\mbox{MeV}$ Agaev:2021jsz
$\langle\bar{s}s\rangle$= $0.8\langle\bar{u}u\rangle$ with
$\langle\bar{u}u\rangle$=$(-0.24\pm 0.01)^{3}\,$GeV3 Ioffe:2005ym ,
$m_{0}^{2}=0.8\pm 0.1$ GeV2 Ioffe:2005ym ,
$\langle\frac{\alpha_{s}}{\pi}G^{2}\rangle=(0.012\pm 0.004)$
$~{}\mathrm{GeV}^{4}$ Belyaev:1982cd , $\lambda_{T_{cc}^{+}-Di}=(1.96\pm
0.44)\times 10^{-2}$ GeV5 Agaev:2021vur , $\lambda_{T_{cc}^{+}-Mol}=(1.56\pm
0.22)\times 10^{-2}$ GeV5 Xin:2021wcr  and $\lambda_{Z_{V}^{++}}=(1.85\pm
0.45)\times 10^{-2}$ GeV5 Agaev:2021jsz . The wavefunctions inside the DAs of
the photon and all the related parameters are borrowed from Ref. Ball:2002ps .

The next step in the numerical analysis is to find the working regions of the
auxiliary parameters known as the continuum threshold $s_{0}$ and the Borel
parameter $M^{2}$. To this end, we enforce the following conditions:
Convergence of operator product expansion, pole dominance, as well as minimal
sensitivity of the resulting observables to auxiliary parameters. As a result
of these constraints, the following working regions are obtained for these
parameters:

$\displaystyle 4.0~{}\mbox{GeV}^{2}\leq M^{2}\leq
6.0~{}\mbox{GeV}^{2}~{}~{}\mbox{for}~{}T_{cc}^{+}~{}\mbox{state},$
$\displaystyle 4.0~{}\mbox{GeV}^{2}\leq M^{2}\leq
6.0~{}\mbox{GeV}^{2}~{}~{}\mbox{for}~{}Z_{V}^{++}~{}\mbox{state},$
$\displaystyle 19.5~{}\mbox{GeV}^{2}\leq s_{0}\leq
21.5~{}\mbox{GeV}^{2}~{}~{}\mbox{for}~{}T_{cc}^{+}~{}\mbox{state},$
$\displaystyle 15.0~{}\mbox{GeV}^{2}\leq s_{0}\leq
17.0~{}\mbox{GeV}^{2}~{}~{}\mbox{for}~{}Z_{V}^{++}~{}\mbox{state}.$

Having determined the working regions of $M^{2}$ and $s_{0}$, we now study the
dependence of MDM of the $T^{+}_{cc}$ and $Z_{V}^{++}$ states on $M^{2}$ , at
several fixed values of $s_{0}$ in Fig. 1. We notice that MDM of the
$T^{+}_{cc}$ and $Z_{V}^{++}$ states show good stability with respect to the
variation in $M^{2}$ in its working region, especially at lower values of
$s_{0}$. We also find that these MDMs exhibit residual dependence on the
variation of $s_{0}$, which appear as the main sources of uncertainties in the
results.

Figure 1:  The dependence of MDMs of the $T_{cc}^{+}$ and $Z_{V}^{++}$ states
on $M^{2}$ at three fixed values of $s_{0}$.

The final results obtained from the analyses for the MDMs, in both the compact
tetraquark of diquark-antidiquark and molecular pictures for $T_{cc}^{+}$
state and the compact tetraquark of diquark-antidiquark picture for
$Z_{V}^{++}$ state are given as

$\displaystyle\mu_{T_{cc}^{+}-Di}=0.66^{+0.34}_{-0.23}~{}\mu_{N},$ (21)
$\displaystyle\mu_{T_{cc}^{+}-Mol}=0.43^{+0.23}_{-0.22}~{}\mu_{N},$
$\displaystyle\mu_{Z_{V}^{++}}=3.35^{+0.89}_{-0.73}~{}\mu_{N}.$

Errors in the results are due to the uncertainties related to the calculations
of the working regions for $s_{0}$ and $M^{2}$ as well as those of other input
parameters. It should be noted again that the main source of errors in the
results is the variations of the results with respect to $s_{0}$. From Eq.
(21), we see that although the central values of the magnetic dipole moments
obtained for $T_{cc}^{+}$ state in two diquark-antidiquark and molecular
pictures differ from each other considerably, however they are consistent with
each other within the presented uncertainties. This is usual situation in the
case of tetraquarks against the standard baryons of three-quarks: the $\Sigma$
and $\Lambda$ baryons contain the same quark content and they are members of
the same octet, but they have very different magnetic dipole moments. This
means that the magnetic dipole moment is not able to distinguish between the
two pictures in $T_{cc}^{+}$ channel from theoretical point of view. Future
possible experimental or lattice results on the magnetic dipole moment of this
doubly charmed state and their comparison with the results obtained in the
present study as well as comparison of the existing and future experimental
results on the mass, width and other decay parameters, related to the various
interactions of this state with other particles, with the theoretical
predictions of different phenomenological models will shed light on the inner
structure and nature of this state and help us fix its quantum numbers and
quark-gluon organization. The result for the magnetic dipole moment of
$Z_{V}^{++}$ state, due to its double charge, is considerably large compared
to other tetraquarks. Any future measurements of its electromagnetic
properties and comparison of the results with theoretical predictions will be
very useful in our understanding of tetraquarks and as a result the QCD as
theory of strong interaction.

##  IV Discussion and concluding remarks

The magnetic dipole moments, together with other electromagnetic observables,
of the composite particles encode important information on their nature,
substructure and geometric shape. They contain information on the spatial
distributions of charge and magnetization inside the particles that directly
give information on the spatial distributions of quarks and gluons inside
hadrons.

In this study, the MDMs of the recently observed doubly charmed exotic state
$T_{cc}^{+}$ as well as the recently proposed doubly charged four-quark
candidate $Z_{V}^{++}$ were investigated within the framework of light-cone
QCD. We considered both the compact diquark-antidiquark and molecular pictures
for $T_{cc}^{+}$, but only diquark-antidiquark organization for the
$Z_{V}^{++}$ state. They were treated as states with $J^{P}=1^{+}$ and
$J^{P}=1^{-}$ quantum numbers, respectively.

We extracted the values of the MDMs as
$\mu_{T_{cc}^{+}-Di}=0.66^{+0.34}_{-0.23}~{}\mu_{N}$,
$\mu_{T_{cc}^{+}-Mol}=0.43^{+0.23}_{-0.22}~{}\mu_{N}$ and
$\mu_{Z_{V}^{++}}=3.35^{+0.89}_{-0.73}~{}\mu_{N}$. The obtained results for
the $\mu_{T_{cc}^{+}}$ in two pictures are consistent within the presented
uncertainties although their central values differ from each other
considerably. This indicates that the magnetic dipole moment is not able to
distinguish between the two pictures. The obtained results in this study for
the magnetic dipole moment of this state together with the results of other
theoretical models on the mass, width and other decay properties of this state
and comparison of the obtained results with the existing and future
experimental data will shed light on the nature of the newly seen $T_{cc}^{+}$
tetraquark. The result for the magnetic dipole moment of $Z_{V}^{++}$ state,
due to its double charge, is considerably large. The orders of MDMs,
especially for $\mu_{Z_{V}^{++}}$, indicate that they are accessible in the
experiment. Our result on MDM of $Z_{V}^{++}$ together with its spectroscopic
parameters calculated in Ref. Agaev:2021jsz , may help experimental groups in
the search for this state and its properties in the experiment. Our results on
MDMs may be checked via alternative phenomenological approaches.

## References

  * (1) R. Faccini, A. Pilloni, A. D. Polosa, Exotic Heavy Quarkonium Spectroscopy: A Mini-review, Mod. Phys. Lett. A 27 (2012) 1230025.  arXiv:1209.0107, doi:10.1142/S021773231230025X.
  * (2) A. Esposito, A. L. Guerrieri, F. Piccinini, A. Pilloni, A. D. Polosa, Four-Quark Hadrons: an Updated Review, Int. J. Mod. Phys. A 30 (2015) 1530002\.  arXiv:1411.5997, doi:10.1142/S0217751X15300021.
  * (3) H.-X. Chen, W. Chen, X. Liu, S.-L. Zhu, The hidden-charm pentaquark and tetraquark states, Phys. Rept. 639 (2016) 1–121.  arXiv:1601.02092, doi:10.1016/j.physrep.2016.05.004.
  * (4) A. Ali, J. S. Lange, S. Stone, Exotics: Heavy Pentaquarks and Tetraquarks, Prog. Part. Nucl. Phys. 97 (2017) 123–198.  arXiv:1706.00610, doi:10.1016/j.ppnp.2017.08.003.
  * (5) A. Esposito, A. Pilloni, A. D. Polosa, Multiquark Resonances, Phys. Rept. 668 (2017) 1–97.  arXiv:1611.07920, doi:10.1016/j.physrep.2016.11.002.
  * (6) S. L. Olsen, T. Skwarnicki, D. Zieminska, Nonstandard heavy mesons and baryons: Experimental evidence, Rev. Mod. Phys. 90 (1) (2018) 015003.  arXiv:1708.04012, doi:10.1103/RevModPhys.90.015003.
  * (7) R. F. Lebed, R. E. Mitchell, E. S. Swanson, Heavy-Quark QCD Exotica, Prog. Part. Nucl. Phys. 93 (2017) 143–194.  arXiv:1610.04528, doi:10.1016/j.ppnp.2016.11.003.
  * (8) F.-K. Guo, C. Hanhart, U.-G. Meißner, Q. Wang, Q. Zhao, B.-S. Zou, Hadronic molecules, Rev. Mod. Phys. 90 (1) (2018) 015004.  arXiv:1705.00141, doi:10.1103/RevModPhys.90.015004.
  * (9) M. Nielsen, F. S. Navarra, S. H. Lee, New Charmonium States in QCD Sum Rules: A Concise Review, Phys. Rept. 497 (2010) 41–83.  arXiv:0911.1958, doi:10.1016/j.physrep.2010.07.005.
  * (10) N. Brambilla, S. Eidelman, C. Hanhart, A. Nefediev, C.-P. Shen, C. E. Thomas, A. Vairo, C.-Z. Yuan, The $XYZ$ states: experimental and theoretical status and perspectives, Phys. Rept. 873 (2020) 1–154.  arXiv:1907.07583, doi:10.1016/j.physrep.2020.05.001.
  * (11) Y.-R. Liu, H.-X. Chen, W. Chen, X. Liu, S.-L. Zhu, Pentaquark and Tetraquark states, Prog. Part. Nucl. Phys. 107 (2019) 237–320.  arXiv:1903.11976, doi:10.1016/j.ppnp.2019.04.003.
  * (12) S. Agaev, K. Azizi, H. Sundu, Four-quark exotic mesons, Turk. J. Phys. 44 (2) (2020) 95–173.  arXiv:2004.12079, doi:10.3906/fiz-2003-15.
  * (13) X.-K. Dong, F.-K. Guo, B.-S. Zou, A survey of heavy-antiheavy hadronic molecules, Progr. Phys. 41 (2021) 65–93.  arXiv:2101.01021, doi:10.13725/j.cnki.pip.2021.02.001.
  * (14) R. Aaij, et al., Observation of an exotic narrow doubly charmed tetraquarkarXiv:2109.01038.
  * (15) R. Aaij, et al., Study of the doubly charmed tetraquark $T_{cc}^{+}$arXiv:2109.01056.
  * (16) T. F. Carames, A. Valcarce, J. Vijande, Doubly charmed exotic mesons: A gift of nature?, Phys. Lett. B 699 (2011) 291–295.  doi:10.1016/j.physletb.2011.04.023.
  * (17) J.-M. Richard, A. Valcarce, J. Vijande, Few-body quark dynamics for doubly heavy baryons and tetraquarks, Phys. Rev. C 97 (3) (2018) 035211.  arXiv:1803.06155, doi:10.1103/PhysRevC.97.035211.
  * (18) E. Hernández, J. Vijande, A. Valcarce, J.-M. Richard, Spectroscopy, lifetime and decay modes of the $T^{-}_{bb}$ tetraquark, Phys. Lett. B 800 (2020) 135073\.  arXiv:1910.13394, doi:10.1016/j.physletb.2019.135073.
  * (19) M.-Z. Liu, T.-W. Wu, M. Pavon Valderrama, J.-J. Xie, L.-S. Geng, Heavy-quark spin and flavor symmetry partners of the X(3872) revisited: What can we learn from the one boson exchange model?, Phys. Rev. D 99 (9) (2019) 094018.  arXiv:1902.03044, doi:10.1103/PhysRevD.99.094018.
  * (20) S. S. Agaev, K. Azizi, H. Sundu, Newly observed exotic doubly charmed meson $T^{+}_{cc}$arXiv:2108.00188.
  * (21) N. Li, Z.-F. Sun, X. Liu, S.-L. Zhu, Perfect $DD^{*}$ molecular prediction matching the $T_{cc}$ observation at LHCb, Chin. Phys. Lett. 38 (2021) 092001\.  arXiv:2107.13748, doi:10.1088/0256-307X/38/9/092001.
  * (22) M.-J. Yan, M. P. Valderrama, Subleading contributions to the decay width of the $T_{cc}^{+}$ tetraquarkarXiv:2108.04785.
  * (23) X.-K. Dong, F.-K. Guo, B.-S. Zou, A survey of heavy-heavy hadronic moleculesarXiv:2108.02673.
  * (24) A. Feijoo, W. t. Liang, E. Oset, $D^{0}D^{0}\pi^{+}$ mass distribution in the production of the $T_{cc}$ exotic statearXiv:2108.02730.
  * (25) L. Meng, G.-J. Wang, B. Wang, S.-L. Zhu, Probing the long-range structure of the $T_{cc}^{+}$ with the strong and electromagnetic decaysarXiv:2107.14784.
  * (26) X.-Z. Ling, M.-Z. Liu, L.-S. Geng, E. Wang, J.-J. Xie, Can we understand the decay width of the $T_{cc}^{+}$ state?arXiv:2108.00947.
  * (27) Q. Xin, Z.-G. Wang, Analysis of the axialvector doubly-charmed tetraquark molecular states with the QCD sum rulesarXiv:2108.12597.
  * (28) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, Semileptonic and nonleptonic decays of the axial-vector tetraquark $T_{bb;\overline{u}\overline{d}}^{-}$, Eur. Phys. J. A 57 (3) (2021) 106.  arXiv:2008.02049, doi:10.1140/epja/s10050-021-00428-5.
  * (29) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, A family of double-beauty tetraquarks: Axial-vector state $T_{bb;\bar{u}\bar{s}}^{-}$, Chin. Phys. C 45 (1) (2021) 013105.  arXiv:2002.04553, doi:10.1088/1674-1137/abc16d.
  * (30) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, Stable scalar tetraquark $T_{bb;\bar{u}\bar{d}}^{-}$arXiv:2001.01446, doi:10.1140/epja/s10050-020-00187-9.
  * (31) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, Heavy exotic scalar meson $T_{bb;\bar{u}\bar{s}}^{-}$, Phys. Rev. D 101 (9) (2020) 094026.  arXiv:1912.07656, doi:10.1103/PhysRevD.101.094026.
  * (32) S. S. Agaev, K. Azizi, H. Sundu, Double-heavy axial-vector tetraquark $T_{bc;\bar{u}\bar{d}}^{0}$, Nucl. Phys. B 951 (2020) 114890.  arXiv:1905.07591, doi:10.1016/j.nuclphysb.2019.114890.
  * (33) S. S. Agaev, K. Azizi, H. Sundu, Strong decays of double-charmed pseudoscalar and scalar $cc\overline{u}\overline{d}$ tetraquarks, Phys. Rev. D 99 (11) (2019) 114016.  arXiv:1903.11975, doi:10.1103/PhysRevD.99.114016.
  * (34) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, Weak decays of the axial-vector tetraquark $T_{bb;\bar{u}\bar{d}}^{-}$, Phys. Rev. D 99 (3) (2019) 033002.  arXiv:1809.07791, doi:10.1103/PhysRevD.99.033002.
  * (35) S. S. Agaev, K. Azizi, B. Barsbay, H. Sundu, The doubly charmed pseudoscalar tetraquarks $T_{cc;\bar{s}\bar{s}}^{++}$ and $T_{cc;\bar{d}\bar{s}}^{++}$, Nucl. Phys. B 939 (2019) 130–144.  arXiv:1806.04447, doi:10.1016/j.nuclphysb.2018.12.021.
  * (36) R. Aaij, et al., A model-independent study of resonant structure in $B^{+}\to D^{+}D^{-}K^{+}$ decays, Phys. Rev. Lett. 125 (2020) 242001.  arXiv:2009.00025, doi:10.1103/PhysRevLett.125.242001.
  * (37) R. Aaij, et al., Amplitude analysis of the $B^{+}\to D^{+}D^{-}K^{+}$ decay, Phys. Rev. D 102 (2020) 112003.  arXiv:2009.00026, doi:10.1103/PhysRevD.102.112003.
  * (38) T. J. Burns, E. S. Swanson, Discriminating among interpretations for the $X(2900)$ states, Phys. Rev. D 103 (1) (2021) 014004.  arXiv:2009.05352, doi:10.1103/PhysRevD.103.014004.
  * (39) S. S. Agaev, K. Azizi, H. Sundu, Doubly charged vector tetraquark $Z_{V}^{++}$=$[cu][\bar{s}\bar{d}]$, Phys. Lett. B 820 (2021) 136530.  arXiv:2105.00081, doi:10.1016/j.physletb.2021.136530.
  * (40) U. Özdem, K. Azizi, Magnetic dipole moment of the $Z_{cs}(3985)$ state: diquark-antidiquark and molecular pictures, Eur. Phys. J. Plus 136 (2021) 968\.  arXiv:2102.09231, doi:10.1140/epjp/s13360-021-01977-w.
  * (41) S. J. Brodsky, J. R. Hiller, Universal properties of the electromagnetic interactions of spin one systems, Phys. Rev. D 46 (1992) 2141–2149.  doi:10.1103/PhysRevD.46.2141.
  * (42) K. Azizi, A. R. Olamaei, S. Rostami, Beautiful mathematics for beauty-full and other multi-heavy hadronic systems, Eur. Phys. J. A 54 (9) (2018) 162.  arXiv:1801.06789, doi:10.1140/epja/i2018-12595-1.
  * (43) B. L. Ioffe, QCD at low energies, Prog. Part. Nucl. Phys. 56 (2006) 232–277.  arXiv:hep-ph/0502148, doi:10.1016/j.ppnp.2005.05.001.
  * (44) V. M. Belyaev, B. L. Ioffe, Determination of the baryon mass and baryon resonances from the quantum-chromodynamics sum rule. Strange baryons, Sov. Phys. JETP 57 (1983) 716–721.
  * (45) P. Ball, V. M. Braun, N. Kivel, Photon distribution amplitudes in QCD, Nucl. Phys. B 649 (2003) 263–296.  arXiv:hep-ph/0207307, doi:10.1016/S0550-3213(02)01017-9.


# KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative
Commonsense Reasoning

Haonan Li1 , Yeyun Gong2, Jian Jiao3, Ruofei Zhang3, Timothy Baldwin1, Nan
Duan2
1School of Computing and Information Systems, The University of Melbourne
2Microsoft Research Asia 3Microsoft
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done during an internship
at Microsoft Research Asia. Corresponding author.

###### Abstract

Pre-trained language models have led to substantial gains over a broad range
of natural language processing (NLP) tasks, but have been shown to have
limitations for natural language generation tasks with high-quality
requirements on the output, such as commonsense generation and ad keyword
generation. In this work, we present a novel Knowledge Filtering and
Contrastive learning Network (KFCNet) which references external knowledge and
achieves better generation performance. Specifically, we propose a BERT-based
filter model to remove low-quality candidates, and apply contrastive learning
separately to each of the encoder and decoder, within a general
encoder–decoder architecture. The encoder contrastive module helps to capture
global target semantics during encoding, and the decoder contrastive module
enhances the utility of retrieved prototypes while learning general features.
Extensive experiments on the CommonGen benchmark show that our model
outperforms the previous state of the art by a large margin: +6.6 points (42.5
vs. 35.9) for BLEU-4, +3.7 points (33.3 vs. 29.6) for SPICE, and +1.3 points
(18.3 vs. 17.0) for CIDEr. We further verify the effectiveness of the proposed
contrastive module on ad keyword generation, and show that our model has
potential commercial value.

##  1 Introduction

Pre-trained language models have achieved impressive results across a wide
range of NLP tasks (Devlin et al., 2019; Yang et al., 2019; Sun et al., 2019;
Liu et al., 2019; Lewis et al., 2020a; Qi et al., 2020; He et al., 2020b).
However, their ability to accurately reflect factual knowledge or perform
logical inference is still limited. To investigate the ability of systems to
capture commonsense knowledge, datasets such as CommonsenseQA (Talmor et al.,
2019), SWAG (Zellers et al., 2018), and WinoGrande (Sakaguchi et al., 2020)
have been proposed. Separate to these discriminative tasks that require models
to choose the correct option from multiple candidates, CommonGen (Lin et al.,
2020) is framed as a generation task, and requires the system to generate a
logical and coherent sentence describing an everyday scenario based on a
concept set. Experiments show that state-of-the-art generation models are not
adequate or accurate enough to generate plausible sentences or reflect
commonsense assumptions in this setting.

External knowledge provides not only information about the sorts of
relationships that hold between concepts, to potentially guide generation
models in capturing the implicit logic between concepts, but also
interpretability. Inspired by Lewis et al. (2020b) and Fan et al. (2020), we
adopt a retrieval-and-generation framework, and propose a BERT-filter and two
contrastive learning modules for retrieval and generation, respectively.

For retrieval, previous research (Lewis et al., 2020b) has shown that
traditional sparse vector space models, such as TF-IDF and BM25, perform
better than dense representation-based retrieval on heavily entity-centric
tasks such as FEVER (Thorne et al., 2018). However, while using sparse vector
space retrieval models can retrieve relevant prototypes that contain a set of
concepts, there can be significant domain mismatches between the retrieved
results and target distribution, making it difficult for generation models to
bridge between prototypes and targets. We argue that a two-stage retrieval
strategy alleviates this issue by combining sparse vector space search and
dense representation filters. First, a sparse vector retrieval model is used
to find passage candidates with high coverage of concept words, and then a
dense vector-based filter is applied to score the candidates, and filter out
low-quality prototypes.

For generation, we apply contrastive learning to each of the encoder and
decoder, in a general encoder–decoder architecture. The core idea of
contrastive learning is to construct positive and negative samples from an
anchor sample, and draw together the anchor and positive samples while pushing
away the anchor from all negative samples in the embedding space during
training. Given that high-quality prototypes can be used as clusters of
positive samples, we propose a decoder contrastive module that minimizes the
distance between decoded sentence representations with distinct prototypes
retrieved from the same concept set. Common scenario information and abstract
concept relationships can be learned based on the contrasts between different
prototypes. Moreover, we propose an encoder contrastive module to force the
encoder to learn sentence representations, and save it into a global token
which is visible to the decoder during decoding. In this way, global sentence-
level semantics can be captured better.

The main contributions of this work are threefold: (1) we demonstrate that
adding a high-quality matching model to the word overlap-based retriever
benefits entity-centric retrieval tasks; (2) we propose two contrastive
learning modules that can be applied to a general encoder–decoder generation
model; and (3) we conduct experiments on CommonGen and an ad keyword
generation task, and show that our method achieves large-scale improvements on
both tasks.

##  2 Related Work

###  2.1 Knowledge Enhanced Generation

There is significant work on incorporating external knowledge from knowledge
bases and incorporating retrieved information in language generation tasks
(Weston et al., 2018; Cao et al., 2018; Guan et al., 2019; Hossain et al.,
2020). Lewis et al. (2020b) explore a general-purpose fine-tuning recipe for
retrieval-augmented generation that combines a dense passage retriever
(Karpukhin et al., 2020) with a BART (Lewis et al., 2020a) generator. For
commonsense generation, Liu et al. (2020) propose a knowledge graph-augmented
language generation model that encompasses concepts from a knowledge graph,
and produces more logical and natural sentences. Fan et al. (2020) propose to
retrieve prototypes based on sparse vector similarity, and introduce a scaling
module and a prototype position indicator to explicitly deal with retrieval
noise.

This paper proposes a two-stage retrieval strategy and differs in applying
contrastive learning to make better use of prototypes for generation.

###  2.2 Contrastive Learning

Recently, contrastive learning has achieved remarkable results in many self-
supervised and supervised learning tasks, primarily for computer vision. The
two key elements of contrastive learning are: (1) the construction of positive
and negative samples; and (2) the learning framework.

####  2.2.1 Sample Construction

Usually in contrastive learning, positive samples are augmented forms of
anchor data points, and negative samples are augmented forms of other data
points. In NLP, Meng et al. (2021) create positive samples by masking and
cropping tokens from sentences; Gunel et al. (2020) and Fang and Xie (2020)
use back-translation to create positive augmentations of original sentences;
Chi et al. (2020) and Wei et al. (2021) regard parallel sentences distributed
in one or multiple languages as different views of the same semantics to learn
cross-lingual representations; and Gao et al. (2021) demonstrate that
constructing positive pairs with only standard dropout as minimal data
augmentation works surprisingly well on the NLI task. Distinct from these
methods, we propose to create positive sample pairs from retrieved results.

####  2.2.2 Learning Framework

Previous contrastive learning methods have required either specialized
architectures (Bachman et al., 2019; Hénaff, 2020) or a memory bank to store
large volumes of negative samples (Wu et al., 2018; Tian et al., 2020). Chen
et al. (2020) present a simple framework consisting of a feature extraction
module, and a non-linear transformation module, which outperforms previous
work on ImageNet (Russakovsky et al., 2015) without using a specialized
architecture or a memory bank. However, it requires a large batch size to
yield high performance, which is computationally prohibitive. Moco (He et al.,
2020a) addresses this issue by maintaining a queue of data samples as the
memory bank, and enqueuing encoded representations of the current mini-batch
and dequeuing the oldest representations on each iteration. They further
propose a momentum encoder to maintain the consistency of representations in
the queue. In this work, we use the Moco framework to train our contrastive
learning modules.

##  3 Method

In this section, we detail our method: Knowledge Filtering and Contrastive
learning Network (KFCNet). First, we introduce our prototype retrieval
strategy together with the knowledge filter model. Then we present our
generative model, based on an encoder–decoder architecture with two
contrastive learning modules. Finally, we show how we adapt the Moco
contrastive learning framework, and deal with multiple positive samples.

###  3.1 Task Formulation

We use $S$ to denote a set of concepts, where $S=\\{c_{1},c_{2},...,c_{m}\\}$,
$c_{i}\in\mathcal{C}$ and $\mathcal{C}$ is the concept vocabulary, and use
$\mathcal{X}$ to denote all possible concept sets. The commonsense generation
task is to learn a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ that maps
the concept set $S$ to a sentence $T$, where
$T=\\{t_{1},t_{2},...,t_{n}\\}\in\mathcal{Y}$, and $\mathcal{Y}$ is the target
sentence space. The generated sentence must be a plausible sentence that
describes a common scenario in our daily life based on the contents of $S$.

###  3.2 Prototype Retrieval and Filtering

In order to retrieve prototypes that contain the concepts in a given concept
set while keeping the retrieval results and target sentences semantically as
similar as possible, we use a two-stage retrieval strategy combining sparse
vector space search and dense representation matching. In Stage 1, a sparse
vector retrieval model is used to retrieve $N$ candidate prototypes from the
corpus $\mathcal{D}$, where $N\ll\left|\mathcal{D}\right|$. Then in Stage 2, a
dense representation-based scorer is used to score the candidates, and the
top-$k$ scored candidates are chosen as the final prototypes.

####  3.2.1 Stage 1

Given a concept set $S=\\{c_{1},c_{2},...,c_{m}\\}$, we first split corpus
$\mathcal{D}$ into $m+1$ parts $\\{d_{0},d_{1},...,d_{m}\\}$ according to the
number of concepts the sentence contains, where sentences in $d_{i}$ contain
$i$ concepts in $x$. Given that most concepts in $\mathcal{C}$ are verb and
noun lemmas, we pre-process based on lemmatization and stemming. Then we
choose $N$ sentences as candidates from the parts, prioritized such that
$d_{m}>d_{m-1}...>d_{1}$.

####  3.2.2 Stage 2

After retrieving $N$ candidate prototypes, a scorer is used to rank the
candidates and filter out candidates that are far from the targets in
embedding space. In this work, we use a BERT-based model as an encoder, and
use the hidden state of the [CLS] token as the sample representation. The
representation is then feed into a multi-layer perceptron with a scalar output
as follows:

$\displaystyle r_{cls}$ $\displaystyle=\operatorname{BERT}(S)$ (1)
$\displaystyle score$ $\displaystyle=\operatorname{MLP}(r_{cls})$ (2)

where $S=\texttt{[CLS]}+concept\ set+\texttt{[SEP]}+sentence+\texttt{[SEP]}$
is the training sample created from the concept set and candidates/targets,
$r_{cls}\in\mathbb{R}^{d}$ is the sample representation, and $score\in[0,1]$
is the final score of the sample. Theoretically, the label of the training set
can be any real number in the range $[0,1]$, but we find that it is sufficient
to train the scorer as a simple binary classifier. We create the positive
samples by combining a concept set with each corresponding target sentence,
and create the negative samples by combining the concept set with a different
candidate prototype or random sentence from $\mathcal{D}$. During inference,
we score all candidates and choose the candidate with the highest score as the
prototype.

For all experiments in this paper, we set $k=2$ and $N=8$. $k=2$ means that
for each input, we construct one positive sample which is widely used in
contrastive leaning work. $N=8$ is because experience shows that at least 2
high-quality prototypes can be retrieved with 8 candidates.

Figure 1: Generation model structure with contrastive learning.

###  3.3 Contrastive Learning for Generation

####  3.3.1 Encoder–decoder Architecture

The encoder–decoder architecture is widely used in generation tasks. Compared
to single decoder generation models such as GPT-2 (Radford et al., 2019) where
words are conditioned only on left context, models using an encoder–decoder
framework such as BART (Lewis et al., 2020a) and T5 (Raffel et al., 2020)
enable bidirectional interactions with an encoder, and auto-regressive
generation with a decoder. In this work, we use BART with an auto-regressive
objective for generation as shown in the middle of Figure 1, and propose
separate contrastive modules for the encoder and decoder, corresponding to the
right and left sub-structures in the figure.

Typical generation inputs only contain the source sentence, which is the
concept set $S$ in CommonGen. The difference here is that we append one of the
retrieved prototypes $p_{i}\in P$ to $S$ to enhance the input.

####  3.3.2 Encoder Contrastive Loss

Although BART learns bidirectional interactions using a transformer-based
encoder and implements cross-attention at each layer of the decoder, the
global target information is not explicitly learned during encoding, meaning
that the decoder needs to find important local information for the current
step during each timestep of decoding, without having access to the global
goal of generation. Here, we propose to force the model to learn global target
information during encoding and save it to a special token, using
source–target contrastive learning. The special token is visible to the
decoder via cross-attention during each timestep of decoding. Specifically,
given a concept set $S=\\{c_{1},c_{2},...,c_{m}\\}$ and a target sentence
$T=\\{t_{1},t_{2},...,t_{n}\\}$, where $c_{i}\in\mathcal{C}$, we denote the
retrieved prototypes as $\\{p_{1},p_{2},...,p_{k}\\}$, where each $p_{i}$ is a
complete prototype. We construct the input for encoder contrastive learning by
concatenating $S$ with $T$ and $S$ with $p_{i}$, respectively. As illustrated
in Figure 1, the concatenation of $S$ and $p_{i}$ will be used as the input to
the main encoder, which is followed by a decoder with gradient and auto-
regressive generation loss, and the concatenation of $S$ and $p_{i}$ will be
used as the input to another encoder without a decoder or gradient.

####  3.3.3 Decoder Contrastive Loss

At the retrieval step, multiple high-quality retrieval results are collected
as prototypes to augment generation. Although these retrieved results
substantially boost external information, they inevitably introduce noise. In
order to learn general information associated with the concept set and
eliminate noise in the prototypes, we propose a decoder contrastive learning
module, which we apply to the sentence representation at the decoder output.
Formally, we concatenate $S$ with $p_{j}$ ($j\neq i$), which is a different
prototype for $S$ from the one used in the main-branch BART model. Note also
that different from the main-branch model, here the gradient is not back-
propagated.

###  3.4 Momentum Contrast with Memory Bank

Most existing training methods greatly limit the number of in-batch negative
samples, limiting the potential of contrastive learning. To enable large-scale
interactions between negative samples, we follow Moco (He et al., 2020a) in
maintaining a dictionary as a queue of encoded/decoded data samples. The keys
of the dictionary are samples from data after encoding/decoding and the
queries are samples in current mini-batches after encoding/decoding during
training. Learning is formulated as minimizing the contrastive loss, which
makes the query similar with its matching key and dissimilar to other keys.

####  3.4.1 Memory Bank as Queues

We use two dictionaries to store the representations of the encoder and
decoder output, respectively. In each training iteration, the newest encoded
representations are enqueued and the oldest are dequeued, to maintain a fixed
queue size. For each sample, the number of contrast pairs is the size of the
queue, where usually only the matching key in the same mini-batch is positive,
and all others are negative.

####  3.4.2 Dealing with Multiple Positive Samples

In the CommonGen task, the mapping between source sequences and gold targets
can be many-to-many. $N$ independent sample pairs are created for 1-to-$N$ and
$N$-to-1 source–target pairs, which can be distributed across mini-batches.
However, these $N$ samples should be all regarded as mutually positive. To
enable interactions between positive samples intra- and inter-mini-batch, we
assign each source–target pair an identity, which indicates pairs that share
the same source or target. These identities are saved in another queue that is
synchronously updated with an encoder and decoder memory bank.

####  3.4.3 Momentum Updated Parameters

To keep the consistency of representations in the memory banks, we update the
parameters of the key encoder and decoder with momentum. Formally, denote the
parameters of the query encoder and decoder as ${\theta}_{q}^{e}$ and
${\theta}_{q}^{d}$, and the parameters of the key encoder and decoder as
${\theta}_{k}^{e}$ and ${\theta}_{k}^{d}$. The parameters ${\theta}_{q}^{e}$
and ${\theta}_{q}^{d}$ are updated by back-propagation, and the parameters
${\theta}_{k}^{e}$ and ${\theta}_{k}^{d}$ are updated by:

$\displaystyle{\theta}_{k}^{e}\leftarrow
m{\theta}_{k}^{e}+(1-m){\theta}_{q}^{e}$ (3)
$\displaystyle{\theta}_{k}^{d}\leftarrow
m{\theta}_{k}^{d}+(1-m){\theta}_{q}^{d}$ (4)

Here, $m\in[0,1)$ is a momentum coefficient which is set to be close to 1. In
this way, the parameters of the key encoder and decoder evolve more smoothly
than those of the query, which maintains the consistency of key
representations in the memory bank.

###  3.5 Training Objective

Consider a batch of query-key pairs
$\\{(q_{1},k_{1}),(q_{2},k_{2}),...,(q_{n},k_{n})\\}$, where there is only one
positive key $k_{i}$ for a given query $q_{i}$. After encoding, we fetch the
representation of the last <EOS> tokens and apply a projection to it as:

$\displaystyle a_{i}^{eos}$ $\displaystyle=\operatorname{Encoder}(q_{i})$ (5)
$\displaystyle z_{i}$ $\displaystyle=\operatorname{Proj}(a_{i}^{eos})$ (6)

The encoder contrastive loss function, called InfoNCE, is as follows:

$\mathcal{L}_{E_{i}}=-\log\frac{\exp(\operatorname{sim}({z_{q_{i}},z_{k_{i}}})/\tau)}{\sum_{j\in
M}\exp(\operatorname{sim}({z_{q_{i}},z_{k_{j}})/\tau})}$ (7)

where $\operatorname{sim}(,)$ denotes the similarity function, $\tau$ is a
temperature hyper-parameter, and $M$ denotes all indices in the memory bank.
The denominator has $|M|$ total terms, including one positive and $|M|-1$
negative samples. Intuitively, the loss function is the log loss of an
$|M|$-way softmax classifier that tries to classify $q_{i}$ according to the
positive $k_{i}$. Eqn (7) is only able to deal with the case of a single
positive key existing for each query. To generalize it to an arbitrary number
of positives, inspired by SupCon (Khosla et al., 2020), we consider the
following loss functions:

$\displaystyle\mathcal{L}_{E_{i}}^{out}$ $\displaystyle=-\sum_{p\in
P(i)}\frac{1}{|P(i)|}\log\mathcal{L}_{E_{i,p}}^{single}$ (8)
$\displaystyle\mathcal{L}_{E_{i}}^{in}$
$\displaystyle=-\log\left\\{\frac{1}{|P(i)|}\sum_{p\in
P(i)}\mathcal{L}_{E_{i,p}}^{single}\right\\}$ (9)
$\displaystyle\mathcal{L}_{E_{i,p}}^{single}$
$\displaystyle=\frac{\exp(\operatorname{sim}({z_{q_{i}},z_{k_{p}}})/\tau)}{\sum_{j\in
M}\exp(\operatorname{sim}({z_{q_{i}},z_{k_{j}})/\tau})}$ (10)

Here, $P(i)$ denotes all positive indices of the sample $i$, Eqn (8)
summarizes the positive samples outside of the log function, and Eqn (9)
summarizes those inside it.

The decoder contrastive loss $\mathcal{L}_{D}$ can be obtained in the same
way, except that the sentence representation is fetched from the <EOS> token
after decoding. During training, we try to minimize the sum of the encoder
contrastive loss, decoder contrastive loss, and the decoder auto-regressive
generation loss:

$\mathcal{L}=\mathcal{L}_{CE}+\lambda_{1}\mathcal{L}_{E}+\lambda_{2}\mathcal{L}_{D}$
(11)

Here, $\mathcal{L}_{CE}$ denotes the cross-entropy generation loss, and
$\lambda_{1}$ and $\lambda_{2}$ are tunable scalars.

During inference, we discard the momentum encoder and decoder, together with
the projection layers.

##  4 Experiments

$\mathcal{D}_{in}$ $\mathcal{D}_{out}$ # Sentences 4,118,993 70,245,048 #
Length (avg) 11.10 18.76 # Missing concept (avg) size=3 0.40 0.48 size=4 0.56
1.00 size=5 0.80 1.98

Table 1: Statistics of the two corpora. “Missing concept” indicates the number
of missing concepts in the top-2 retrieved sentences, broken down by concept-
set size.

###  4.1 Datasets

CommomGen (Lin et al., 2020) contains 32,651/993/1,497 unique
training/development/test concept sets, corresponding to 67,389 and 4,018
English target sentences in the training and development sets, meaning that
one concept-set can map to multiple target sentences. The percentage of
concept-sets in the development and test sets that are unseen in the training
set are 99.60%, and 100.00% respectively, making the dataset challenging for
compositional generalization.

###  4.2 Prototype Collection

####  4.2.1 In-domain Corpus

As CommonGen was created from visually-grounded caption datasets that describe
everyday scenarios, we build an in-domain corpus from datasets of image
captions, video captions, and natural language inference. In detail, we
extracted sentences from ActivityNet (Krishna et al., 2017), VaTeX (Wang et
al., 2019), Conceptual Captions (Sharma et al., 2018), SNLI (Bowman et al.,
2015), and MNLI (Williams et al., 2018) as the in-domain corpus
($\mathcal{D}_{in}$).

####  4.2.2 Out-of-domain Corpus

In addition to in-domain experiments, we create an out-of-domain corpus
($\mathcal{D}_{out}$) from Wikipedia,111English Wikipedia dump from May 01,
2020. using SpaCy222https://spacy.io/ as our sentence tokenizer.

For both corpora, sentences with fewer than 5 tokens or more than 100 tokens
were removed. Table 1 shows the basic statistics of the two corpora. Although
$\mathcal{D}_{out}$ is much larger than $\mathcal{D}_{in}$, sentences
retrieved from $\mathcal{D}_{in}$ contain more required concepts than those
from $\mathcal{D}_{out}$ on average. Specifically, for concept-sets of size 4
and 5, the retrieved sentences from $\mathcal{D}_{out}$ have 0.44 and 1.18,
respectively, more relevant concepts than $\mathcal{D}_{in}$.

###  4.3 Experimental Setup

####  4.3.1 Implementation Details

We employ the pre-trained BART-large model as the base generation model, and
initialize the momentum encoder and decoder by copying parameters from the
base model. We use the Adam optimizer with ${\beta}_{1,2}=(0.9,0.999)$,
$\epsilon=1e-6$, and 0.1 weight decay, with the initial learning rate setting
selected from $\\{8e-6,1e-5,3e-5,5e-5\\}$. We use the polynomial decay
learning rate scheduler with 500 warmup steps, and set dropout to 0.1. We set
the max tokens per batch to 3000 and max batch-size to 48, with 15k total
updates. For the auto-regressive generation loss, we use cross-entropy loss
with 0.1 label-smoothing penalty. During decoding, we use beam search with
size 5, and 1.0 length penalty.

For contrastive learning, we use an MLP as the projection network, with a
single hidden layer of 1024d and the output size of 128d. We use Eqn (8) as
the loss function, with similarity measured by dot-product, and set the
temperature to 0.1. The queue size of the memory bank is set to 4096, and the
momentum coefficient is set to 0.999.

####  4.3.2 Baselines

We use several state-of-the-art pre-trained language generation models as
baselines: GPT-2 (Radford et al., 2019), BERT-Gen (Bao et al., 2020), UniLM
(Dong et al., 2019), UniLM-v2 (Bao et al., 2020), T5 (Raffel et al., 2020),
and BART (Lewis et al., 2020a). All models are fine-tuned in seq2seq mode. We
also compare our model with two strong baselines that use external knowledge:
EKI (Fan et al., 2020) and KG-BART (Liu et al., 2020).

####  4.3.3 Evaluation Metrics

To evaluate generation performance, we use BLEU (Papineni et al., 2002), ROUGE
(Lin, 2004), and METEOR (Banerjee and Lavie, 2005), in addition to evaluation
metrics for captioning tasks, namely CIDEr (Vedantam et al., 2015) and SPICE
(Anderson et al., 2016). As all metrics score the output in the range
$[0,100]$, we also present the average score across all metrics.

##  5 Results

Model ROUGE-2/L BLEU-3/4 METEOR CIDEr SPICE Overall GPT-2 (Radford et al.,
2019) 17.18 39.28 30.70 21.10 26.20 12.15 25.90 24.64 BERT-Gen (Bao et al.,
2020) 18.05 40.49 30.40 21.10 27.30 12.49 27.30 25.30 UniLM (Dong et al.,
2019) 21.48 43.87 38.30 27.70 29.70 14.85 30.20 29.44 UniLM-v2 (Bao et al.,
2020) 18.24 40.62 31.30 22.10 28.10 13.10 28.10 25.93 T5 (Raffel et al., 2020)
22.01 42.97 39.00 28.60 30.10 14.96 31.60 29.89 BART (Lewis et al., 2020a)
22.23 41.98 36.30 26.30 30.90 13.92 30.60 28.89 EKI-out (Fan et al., 2020)
24.36 45.42 42.90 32.10 32.00 16.80 32.50 32.29 KFCNet-out 24.10 45.59 44.09
34.20 32.83 17.39 33.11 33.04 KG-BART (Liu et al., 2020) 23.38 44.54 42.10
30.90 32.40 16.83 32.70 31.83 EKI (Fan et al., 2020) 25.43 46.53 46.00 36.10
33.80 17.80 33.40 34.15 KFCNet w/o FC 25.16 46.13 50.22 41.97 36.22 18.85
35.90 36.35 KFCNet w/o C 25.91 46.81 54.75 47.33 38.19 20.21 38.20 38.77
KFCNet 26.81 47.52 57.33 51.46 38.92 20.98 39.15 40.31

Table 2: Overall performance of the different models on CommonGen (v1.0).
Models in the first block are fine-tuned pre-trained language models without
external knowledge; models in the second block use out-of-domain knowledge;
models in the last two blocks use in-domain knowledge, where the KG-BART uses
ConceptNet, and both EKI and KFCNet (our model) use the in-domain prototype
corpus as a knowledge base.

Table 2 presents the experimental results across all the metrics.333Note that
the latest test set (v1.1) adds one more human reference to each example in
the test set (v1.0), but is not publicly available. Additionally, EKI and KG-
BART were evaluated on v1.0, so this is what we use for our experiments.  We
observe the following: (1) Methods in the 2nd, 3rd, and 4th blocks of Table 2
that use external knowledge outperform the fine-tuned pre-trained language
models in the first block. This demonstrates that external knowledge benefits
commonsense reasoning and generation. (2) The overall performance of EKI and
our method (KFCNet) that both use natural sentences as prototypes is better
than KG-BART, which incorporates structured knowledge from knowledge bases. We
hypothesize that this is because pre-trained language models like BART can
more easily exploit natural language samples than structured information, even
with elaborate modules for information fusion. (3) Prototypes retrieved from
the in-domain corpus result in better performance than those from the out-of-
domain corpus. (4) Simply fine-tuning BART on our retrieved prototypes beats
previous published SOTA on several metrics, and using filtered prototypes
boosts the performance again. This on the one hand shows that the quality of
prototypes has a large impact on generation, and on the other hand, indicates
our retrieval method is better than that of EKI, and our filter helps in
selecting good prototypes. (5) Our KFCNet achieves new state-of-the-art
results which surpass all other methods by a large margin.

Model | BLEU-4 | CIDEr | SPICE
---|---|---|---
w/o Retrieval | 26.30 | 13.92 | 30.60
BM25 | 36.84 | 17.33 | 32.96
$+$Lemma & Stem | 41.97 | 18.85 | 35.90
$+$BERT Filter | 47.33 | 20.21 | 38.20
Table 3: Results for fine-tuning BART based on different retrieval strategies
over the test set.

###  5.1 Ablation study

To better understand the impact of the different modules in KFCNet, we perform
a number of ablation experiments.

####  5.1.1 Retrieval and Filter

Prototype retrieval is a key part of any retrieval-based generation model. To
assess the effectiveness of the retrieval-and-filter mechanism, we retrieve
prototypes from the in-domain corpus and run ablations on a single BART model.
Table 3 shows the results. Compared to models without retrieval, using
prototypes retrieved by a simple BM25 model improves generation performance,
which we suggest is due to the retrieved prototypes helping the model to
better capture relationships between concepts, and construct a coherent
scenario. With word lemmatization and stemming, the variety of the retrieval
results increases, resulting in better prototypes. Adding a BERT filter boosts
the performance again, achieving $+$5.38, $+$1.36, and $+$2.30 absolute
improvements for BLEU-4, CIDEr, and SPICE. This verifies the effectiveness of
using a high-quality matching model as an auxiliary module for a word overlap-
based retriever.

Model | BLEU-4 | CIDEr | SPICE
---|---|---|---
KFCNet | 36.10 | 17.96 | 33.89
$-C_{D}$ | 34.09 | 17.24 | 33.97
$-C_{E}-C_{D}$ | 30.82 | 16.20 | 33.16
Table 4: Contrastive ablation study on CommonGen development set. $C_{E}$ and
$C_{D}$ denote the encoder and decoder contrastive modules, respectively.

####  5.1.2 Contrastive Learning

The contrastive loss plays an important role in our model. We perform an
ablation study on the development set of CommonGen, by comparing the model
without the contrastive module, using only encoder contrastive learning, and
using both encoder and decoder contrastive learning. As shown in Table 4,
using only encoder contrastive learning leads to improvements over the
baseline BART model, and adding decoder contrastive learning further improves
results based on BLEU-4 and CIDEr.

###  5.2 Similarity Function and Summation Location

We further compare the performance of different similarity functions and
positive summation locations, as mentioned in Section 3.5. The results in
Table 5 demonstrate that the combination of dot-product similarity with
summation outside of the log function performs best, consistent with the
findings of Khosla et al. (2020).

Sim | Sum | BLEU-4 | CIDEr | SPICE
---|---|---|---|---
$(\cdot)$ | Out | 34.11 | 16.77 | 33.59
$(\cdot)$ | In | 32.45 | 16.54 | 33.79
$\cos(,)$ | Out | 32.49 | 16.62 | 33.73
$\cos(,)$ | In | 33.52 | 16.64 | 33.39
Table 5: Comparison of different similarity functions and positive sample
summation locations. $(\cdot)$ denotes dot-product similarity, and $\cos(,)$
denotes cosine similarity.

###  5.3 Model Efficiency

####  5.3.1 Retrieval

The prototype retrieval is done separately from the generation model, and the
retrieval time consists of 2 parts: (1) sparse vector matching time, in the
form of BM25 search; and (2) BERT filter inference, for fine-grained
selection, noting that only a few candidates (8 in our experiments) are left
after stage 1, which can be processed in a single mini-batch.

####  5.3.2 Contrastive Module

During training, the momentum encoder and decoder parameters are updated by
Eqn (3) and there are no gradients or back-propagation in these modules.
Therefore it takes no more than double the training time without contrastive
modules. During inference, the contrastive modules are disabled, and hence the
efficiency does not decrease.

###  5.4 Final Leaderboard Results

Model | BLEU-4 | CIDEr | SPICE
---|---|---|---
Human | 46.49 | 37.64 | 52.43
KFCNet | 42.45 | 18.37 | 33.27
RE-T5 | 40.86 | 17.66 | 31.07
KG-BART | 33.86 | 16.92 | 29.63
EKI-BART | 35.94 | 16.99 | 29.58
T5-Large | 31.96 | 15.12 | 28.85
BART | 31.82 | 13.97 | 27.99
UniLM | 30.61 | 14.88 | 27.42
BERT-Gen | 23.46 | 12.60 | 24.82
GPT-2 | 26.83 | 12.18 | 23.56
Table 6: Final CommonGen leaderboard results, using SPICE to rank the methods.

Table 6 shows the final evaluation results on the latest test set with
additional human references
(v1.1).444https://inklab.usc.edu/CommonGen/leaderboard.html Note that the
model in second place (RE-T5) expands the original training data and does
continuous pretraining before fine-tuning on CommonGen. Our method, KFCNet,
performs best on all metrics. Among all fine-tuned methods, KFCNet beats the
previous state-of-the-art by a large margin: $+$6.51 (18.11%) for BLEU-4,
$+$1.38 (8.12%) for CIDEr, and $+$3.64 (11.95%) for SPICE.

###  5.5 Experiment on Keyword Generation

To test the effectiveness of the proposed contrastive learning modules, we
constructed a real-world adword dataset, based on an advertising platform
(Edelman et al., 2007). The goal is to display a list of ads that matches the
user intent, for which the first step is to retrieve relevant keywords
provided by advertisers given a user query. The dataset contains 72,876
training samples, 10,000 dev samples, and 10,000 test samples from a major
search engine, with each sample corresponding to a query–keyword pair. Titles
of the top-two web search results of the query from the search engine are kept
as prototypes.

We fine-tune BART models following the same sequence generation experimental
design. The results are shown in Table 7.

From the first two lines, we see that directly appending the retrieved
information to the source does not lead to noticeable improvements, almost
certainly because of noise in the retrieved results. However, our contrastive
modules alleviate the effects of noise, and beat BART on all metrics.

Model | ROUGE-2/L | BLEU-3/4 | AVG
---|---|---|---
BART | 33.03/60.17 | 31.61/25.03 | 38.96
$+R$ | 33.68/60.31 | 31.69/24.75 | 39.70
$+C_{E,D}$ | 35.18/61.24 | 33.60/26.78 | 41.44
Table 7: Experimental results on ad keyword generation.

##  6 Conclusion

In this paper, we present KFCNet: a novel knowledge filtering and contrastive
learning model for retrieval-augmented sequence generation, which achieves
state-of-the-art results on the CommonGen benchmark. Two contrastive learning
modules are proposed to capture global target semantics and learn general
features from multiple retrieved prototypes. A prototype retrieval ablation
study showed the effectiveness of the proposed filter for filtering low-
quality candidates, and further experiments on ad keyword generation showed
that our model has potential commercial value. In the future, we plan to
extend the contrastive module to more general settings, such as natural
language understanding and representation learning.

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  SPICE: semantic propositional image caption evaluation.  In _ECCV - 14th European Conference, Part V_ , volume 9909, pages 382–398.
  * Bachman et al. (2019) Philip Bachman, R. Devon Hjelm, and William Buchwalter. 2019.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems, NeurIPS_ , pages 15509–15519.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: an automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL_ , pages 65–72.
  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon. 2020.  UniLMv2: Pseudo-masked language models for unified language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning, ICML_ , pages 642–652.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP_ , pages 632–642.
  * Cao et al. (2018) Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018.  Retrieve, rerank and rewrite: Soft template based neural summarization.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL_ , pages 152–161.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning, ICML_ , volume 119, pages 1597–1607.
  * Chi et al. (2020) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2020.  InfoXLM: An information-theoretic framework for cross-lingual language model pre-training.  _CoRR_ , abs/2007.07834.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_ , pages 4171–4186.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems, NeurIPS_ , pages 13042–13054.
  * Edelman et al. (2007) Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. 2007.  Internet advertising and the generalized second-price auction: Selling billions of dollars worth of keywords.  _American Economic Review_ , 97(1):242–259.
  * Fan et al. (2020) Zhihao Fan, Yeyun Gong, Zhongyu Wei, Siyuan Wang, Yameng Huang, Jian Jiao, Xuanjing Huang, Nan Duan, and Ruofei Zhang. 2020.  An enhanced knowledge injection model for commonsense generation.  In _Proceedings of the 28th International Conference on Computational Linguistics, COLING_ , pages 2014–2025.
  * Fang and Xie (2020) Hongchao Fang and Pengtao Xie. 2020.  CERT: contrastive self-supervised learning for language understanding.  _CoRR_ , abs/2005.12766.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  _CoRR_ , abs/2104.08821.
  * Guan et al. (2019) Jian Guan, Yansen Wang, and Minlie Huang. 2019.  Story ending generation with incremental encoding and commonsense knowledge.  In _The Thirty-Third AAAI Conference on Artificial Intelligence_ , pages 6473–6480.
  * Gunel et al. (2020) Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2020.  Supervised contrastive learning for pre-trained language model fine-tuning.  _CoRR_ , abs/2011.01403.
  * He et al. (2020a) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020a.  Momentum contrast for unsupervised visual representation learning.  In _Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 9726–9735.
  * He et al. (2020b) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020b.  DeBERTa: Decoding-enhanced BERT with disentangled attention.  _CoRR_ , abs/2006.03654.
  * Hénaff (2020) Olivier J. Hénaff. 2020.  Data-efficient image recognition with contrastive predictive coding.  In _Proceedings of the 37th International Conference on Machine Learning, ICML_ , volume 119, pages 4182–4192.
  * Hossain et al. (2020) Nabil Hossain, Marjan Ghazvininejad, and Luke Zettlemoyer. 2020.  Simple and effective retrieve-edit-rerank text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL_ , pages 2532–2538.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP_ , pages 6769–6781.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems, NeurIPS_.
  * Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017.  Dense-captioning events in videos.  In _International Conference on Computer Vision, ICCV_ , pages 706–715.
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL_ , pages 7871–7880.
  * Lewis et al. (2020b) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _Advances in Neural Information Processing Systems, NeurIPS_.
  * Lin et al. (2020) Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020.  CommonGen: A constrained text generation challenge for generative commonsense reasoning.  In _Conference on Automated Knowledge Base Construction, AKBC_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2020) Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S. Yu. 2020.  KG-BART: knowledge graph-augmented BART for generative commonsense reasoning.  _CoRR_ , abs/2009.12677.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Meng et al. (2021) Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021.  COCO-LM: correcting and contrasting text sequences for language model pretraining.  _CoRR_ , abs/2102.08473.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL_ , pages 311–318.
  * Qi et al. (2020) Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020.  ProphetNet: Predicting future n-gram for sequence-to-sequence pre-training.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP_ , pages 2401–2410.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. 2015.  ImageNet large scale visual recognition challenge.  _Int. J. Comput. Vis._ , 115(3):211–252.
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  WinoGrande: An adversarial Winograd schema challenge at scale.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_ , pages 8732–8740.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL_ , pages 2556–2565.
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019.  ERNIE: enhanced representation through knowledge integration.  _CoRR_ , abs/1904.09223.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_ , pages 4149–4158.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and verification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT_ , pages 809–819.
  * Tian et al. (2020) Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2020.  Contrastive multiview coding.  In _ECCV - 16th European Conference, Proceedings, Part XI_ , volume 12356, pages 776–794.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  CIDEr: Consensus-based image description evaluation.  In _Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 4566–4575.
  * Wang et al. (2019) Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019.  VaTeX: A large-scale, high-quality multilingual dataset for video-and-language research.  In _International Conference on Computer Vision, ICCV_ , pages 4580–4590.
  * Wei et al. (2021) Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021\.  On learning universal representations across languages.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander H. Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2nd International Workshop on Search-Oriented Conversational AI, SCAI@EMNLP_ , pages 87–92.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT)_ , pages 1112–1122.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 3733–3742.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  XLNet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems, NeurIPS_ , pages 5754–5764.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP_ , pages 93–104.