# Generalized three-intensity decoy state method for measurement device
independent quantum key distribution

Zong-Wen Yu1,2, Yi-Heng Zhou1, and Xiang-Bin Wang1,3111Email Address:
<EMAIL_ADDRESS>1State Key Laboratory of Low Dimensional Quantum
Physics, Tsinghua University, Beijing 100084, People s Republic of China
2Data Communication Science and Technology Research Institute, Beijing 100191,
China
3 Shandong Academy of Information and Communication Technology, Jinan 250101,
People s Republic of China

###### Abstract

We present improved results and more general results for the decoy-state
measurement device independent quantum key distribution (MDI-QKD) where the
intensities of all 3 pulses can be non-zero. We present a more tightened
explicit formula for the lower bound of the yield of two single-photon pulses
for this generalized 3-intensity protocol, which can be applied to the
recently proposed MDI-QKD with imperfect single-photon sources such as the
coherent states or the heralded states from the parametric down-conversion. We
strictly prove that our result is better than the prior art result. With this
result, the final key rate rises drastically.

###### pacs:

03.67.Dd, 42.81.Gs, 03.67.Hk

##  I Introduction

Recently, much attention has been spent on the security of real set-ups in
quantum key distribution (QKD) BB84 ; GRTZ02 . After the proposal of the decoy
state method ILM ; H03 ; wang05 ; LMC05 ; AYKI ; haya ; peng ; wangyang ; rep
; njp , the security of QKD with an imperfect single photon source is
guaranteed given whatever Eve’s attack to the channel, including the famous
photon-number splitting attack PNS1 ; PNS .

However, the guaranteed security assumes that Eve cannot attack device inside
Alice or Bob’s lab. As was demonstrated lyderson , given a lossy channel and
limited detection efficiency, Eve can attack the gated-mode detectors inside
Alice or Bob’s lab if there is no additional device to detect the abnormally
strong light at Alice or Bob’s lab. To reach the goal of ultimate security,
one can, in principle use the device independent protocol ind1 . However,
these protocols seem to be technically demanding at the moment.

Recently, measurement device independent QKD (MDI-QKD) was proposed based on
the idea of entanglement swapping ind3 ; ind2 . If we want to obtain a higher
key rate, we can choose to directly use an imperfect single-photon source such
as the coherent state ind2  with decoy-state method for this, say the MDI
decoy-state method. Calculation formulas for the practical decoy-state
implementation with only a few different states has been studied in, e.g.,
Refs. wangPRA2013 ; qing3 , and then further studied both experimentally
tittel1 ; tittel2 ; liuyang  and theoretically lopa ; han ; curty ; Wang2013 .
Theoretical study for the decoy-state MDI-QKD with only a few intensities are
of particularly interesting. The first formula for 3-intensity decoy-state
MDI-QKD was proposed in wangPRA2013 . It assumes that each user uses three
different states, one vacuum state and two non-vacuum states. The results can
apply to sources such as the coherent states, the parametric down conversion,
and so on. It was then extended to the case that the 3 intensities are all
non-vacuum coherent states lopa . One can see that the major formula there is
identical with the one in Ref.wangPRA2013  in the case when Alice and Bob use
coherent pulses and the weakest pulse is vacuum. In Ref.Wang2013 ; Wang201309
, we presented an improved explicit formulas of 3-state decoy-state method for
the MDI-QKD with the weakest pulse being vacuum.

Here in this work, we shall present an improved results for the decoy-state
MDI-QKD where all 3 intensities are non-vacuum. Compared with the prior-art
result lopa , our improved formulas here make a more tightened faithful
estimation of the yield and the phase flip rate of the single-photon pulse
pairs and hence can raise the final key rate drastically. Also, our result is
for a general type of sources with a simple condition, it does not limit to
the coherent-state source. In the second section, we present the improved
explicit formulas for 3-intensity decoy-state MDI-QKD and prove analytically
that our result is better than the previous lopa . In the third section, we
give the numerical simulations. The article is ended with a concluding remark.

##  II Generalized Three-intensity decoy state method for MDI-QKD

In the protocol, each time a pulse-pair (two-pulse state) is sent to the relay
for detection. The relay is controlled by an un-trusted third party (UTP). The
UTP will announce whether the pulse-pair has caused a successful event. Those
bits corresponding to successful events will be post-selected and further
processed for the final key. Since real set-ups only use imperfect single-
photon sources, we need the decoy-state method for security.

We assume Alice (Bob) has three sources, $v_{A},d_{A},s_{A}$
($v_{B},d_{B},s_{B}$) which can only emit three different states
$\rho_{v_{A}},\rho_{d_{A}},\rho_{s_{A}}$
($\rho_{v_{B}},\rho_{d_{B}},\rho_{s_{B}}$), respectively, in photon number
space. It’s worth pointing out that our protocol is applicable to the weakest
pulse being vacuum or not. Suppose

$\left\\{\begin{array}[]{lcr}\rho_{v_{A}}=\sum_{k}a^{v}_{k}|k\rangle\langle
k|,&&\rho_{v_{B}}=\sum_{k}b^{v}_{k}|k\rangle\langle k|,\\\
\rho_{d_{A}}=\sum_{k}a^{d}_{k}|k\rangle\langle
k|,&&\rho_{d_{B}}=\sum_{k}b^{d}_{k}|k\rangle\langle k|,\\\
\rho_{s_{A}}=\sum_{k}a^{s}_{k}|k\rangle\langle
k|,&&\rho_{s_{B}}=\sum_{k}b^{s}_{k}|k\rangle\langle k|.\end{array}\right.$ (1)

At each time, Alice will randomly select one of her 3 sources to emit a pulse,
and so does Bob. The pulse form Alice and the pulse from Bob form a pulse pair
and are sent to the UTP. We regard equivalently that each time a two-pulse
source is selected and a pulse pair (one pulse from Alice, one pulse from Bob)
is emitted. There are many different two-pulse sources used in the protocol.
We denote $\alpha\beta$ for the two pulse source when the pulse-pair is
produced by source $\alpha$ at Alice’s side and source $\beta$ at Bob’s side,
$\alpha$ can be one of $\\{v_{A},d_{A},s_{A}\\}$ and $\beta$ can be one of
$\\{v_{B},d_{B},s_{B}\\}$. For example, at a certain time $j$ Alice uses
source $v_{A}$ and Bob uses source $s_{B}$, we say the pulse pair is emitted
by source $v_{A}s_{B}$.

In the protocol, two different bases (e.g., $Z$ basis consisting of horizontal
polarization $|H\rangle\langle H|$ and vertical polarization $|V\rangle\langle
V|$, and $X$ basis consisting of $\pi/4$ and $3\pi/4$ polarizations) are used.
The density operator in photon number space alone does not describe the state
in the composite space. We shall apply the decoy-state method analysis in the
same basis (e.g., $Z$ basis or $X$ basis) for pulses from sources
$\alpha\beta(\alpha\in\\{v_{A},d_{A},s_{A}\\},\beta\in\\{v_{B},d_{B},s_{B}\\})$.
Therefore we only need consider the density operators in the photon number
space.

According to the decoy-state theory, the yield of a certain set of pulse pairs
is defined as the happening rate of a successful event (announced by the UTP)
corresponding to pulse pairs out of the set. We use capital letter
$Y_{\alpha\beta}$ to denote the nine yields for sources
$\alpha\beta(\alpha\in\\{v_{A},d_{A},s_{A}\\},\beta\in\\{v_{B},d_{B},s_{B}\\})$
and these values can be directly calculated from the observed experimental
data. We use $y_{mn}$ to denotes the yield given that Alice and Bob send
respectively an $m$-photon and $n$-photon pulses.

Our first major task is to deduce $y_{11}$ from the known values, i.e., to
formulate $y_{11}$, the yield of state $|1\rangle\langle
1|\otimes|1\rangle\langle 1|$ in the nine known values
$\\{Y_{\alpha\beta}|\alpha\in\\{v_{A},d_{A},s_{A}\\},\beta\in\\{v_{B},d_{B},s_{B}\\}\\}$.
Using the convex proposition, we have the following relations

$Y_{\alpha\beta}=\sum_{k,l\geq 0}a^{\alpha}_{k}b^{\beta}_{l}y_{kl},$ (2)

where $\alpha\in\\{v_{A},d_{A},s_{A}\\}$ and
$\beta\in\\{v_{B},d_{B},s_{B}\\}$. In practice, the imperfect single-photon
sources have the property that the probability of emitting an empty pulse is
greater than the probability of the no-empty pulse. With this fact, in order
to estimate the lower bound of $y_{11}$, we need to eliminate the unknown
values $y_{0l}(l\geq 0)$ and $y_{k0}(k\geq 0)$ firstly.

Without causing any ambiguity, we omit the subscripts A and B in the following
of this paper. Cancel out the terms $y_{0k}$ and $y_{j0}$ using Gaussian
elimination, we obtain the following four relations

$\displaystyle\left\\{\begin{array}[]{c}a_{0}^{v}b_{0}^{v}\hat{Y}_{dd}-a_{0}^{v}b_{0}^{d}\hat{Y}_{dv}-a_{0}^{d}b_{0}^{v}\hat{Y}_{vd}+a_{0}^{d}b_{0}^{d}\hat{Y}_{vv}=0,\\\
a_{0}^{v}b_{0}^{v}\hat{Y}_{ds}-a_{0}^{v}b_{0}^{s}\hat{Y}_{dv}-a_{0}^{d}b_{0}^{v}\hat{Y}_{vs}+a_{0}^{d}b_{0}^{s}\hat{Y}_{vv}=0,\\\
a_{0}^{v}b_{0}^{v}\hat{Y}_{sd}-a_{0}^{v}b_{0}^{d}\hat{Y}_{sv}-a_{0}^{s}b_{0}^{v}\hat{Y}_{vd}+a_{0}^{s}b_{0}^{d}\hat{Y}_{vv}=0,\\\
a_{0}^{v}b_{0}^{v}\hat{Y}_{ss}-a_{0}^{v}b_{0}^{s}\hat{Y}_{sv}-a_{0}^{s}b_{0}^{v}\hat{Y}_{vs}+a_{0}^{s}b_{0}^{s}\hat{Y}_{vv}=0,\end{array}\right.$
(7)

where $\hat{Y}_{ij}=Y_{ij}-\sum_{k,l\geq
1}a_{k}^{i}b_{l}^{j}y_{kl},(i,j\in\\{v,d,s\\})$. For the sake of convenience,
we denote

$\displaystyle\left\\{\begin{array}[]{c}\tilde{Y}_{dd}^{\prime}=a_{0}^{v}b_{0}^{v}{Y}_{dd}-a_{0}^{v}b_{0}^{d}{Y}_{dv}-a_{0}^{d}b_{0}^{v}{Y}_{vd}+a_{0}^{d}b_{0}^{d}{Y}_{vv},\\\
\tilde{Y}_{ds}^{\prime}=a_{0}^{v}b_{0}^{v}{Y}_{ds}-a_{0}^{v}b_{0}^{s}{Y}_{dv}-a_{0}^{d}b_{0}^{v}{Y}_{vs}+a_{0}^{d}b_{0}^{s}{Y}_{vv},\\\
\tilde{Y}_{sd}^{\prime}=a_{0}^{v}b_{0}^{v}{Y}_{sd}-a_{0}^{v}b_{0}^{d}{Y}_{sv}-a_{0}^{s}b_{0}^{v}{Y}_{vd}+a_{0}^{s}b_{0}^{d}{Y}_{vv},\\\
\tilde{Y}_{ss}^{\prime}=a_{0}^{v}b_{0}^{v}{Y}_{ss}-a_{0}^{v}b_{0}^{s}{Y}_{sv}-a_{0}^{s}b_{0}^{v}{Y}_{vs}+a_{0}^{s}b_{0}^{s}{Y}_{vv},\end{array}\right.$
(12)

and introduce the following notations

$h_{x_{k}}^{i}=\frac{x_{k}^{i}}{x_{0}^{i}},\quad(k\geq
0;x\in\\{a,b\\};i\in\\{v,d,s\\}).$ (13)

Divide the four relations in Eq.(7) by the positive factors
$a_{0}^{v}a_{0}^{d}b_{0}^{v}b_{0}^{d}$,
$a_{0}^{v}a_{0}^{d}b_{0}^{v}b_{0}^{s}$, $a_{0}^{v}a_{0}^{s}b_{0}^{v}b_{0}^{d}$
and $a_{0}^{v}a_{0}^{s}b_{0}^{v}b_{0}^{s}$ respectively, we obtain the
following simplified equation

$\displaystyle\left\\{\begin{array}[]{c}\tilde{Y}_{dd}=\sum_{k,l\geq
1}(h_{a_{k}}^{d}-h_{a_{k}}^{v})(h_{b_{l}}^{d}-h_{b_{l}}^{v})y_{kl},\\\
\tilde{Y}_{ds}=\sum_{k,l\geq
1}(h_{a_{k}}^{d}-h_{a_{k}}^{v})(h_{b_{l}}^{s}-h_{b_{l}}^{v})y_{kl},\\\
\tilde{Y}_{sd}=\sum_{k,l\geq
1}(h_{a_{k}}^{s}-h_{a_{k}}^{v})(h_{b_{l}}^{d}-h_{b_{l}}^{v})y_{kl},\\\
\tilde{Y}_{ss}=\sum_{k,l\geq
1}(h_{a_{k}}^{s}-h_{a_{k}}^{v})(h_{b_{l}}^{s}-h_{b_{l}}^{v})y_{kl}.\end{array}\right.$
(18)

where $\tilde{Y}_{ij}$ are defined as follows

$\tilde{Y}_{ij}=\frac{\tilde{Y}_{ij}^{\prime}}{a_{0}^{v}a_{0}^{i}b_{0}^{v}b_{0}^{j}},\quad(i,j\in\\{d,s\\}),$
(19)

with $\tilde{Y}_{ij}^{\prime}$ being presented in Eq.(12). Furthermore, if we
introduce

$\tilde{h}_{x_{k}}^{i}=h_{x_{k}}^{i}-h_{x_{k}}^{v},\quad(k\geq
0;x\in\\{a,b\\};i\in\\{v,d,s\\}),$ (20)

with $h_{x_{k}}^{i}$ defined in Eq.(13), we can write the relations about
variables $y_{kl}$ given by Eq.(18) into the following simplified form

$\displaystyle\tilde{Y}_{dd}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}y_{11}+\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{d}y_{12}+\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}y_{21}+J_{dd},$
(21) $\displaystyle\tilde{Y}_{ds}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{s}y_{11}+\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{s}y_{12}+\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{s}y_{21}+J_{ds},$
(22) $\displaystyle\tilde{Y}_{sd}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{d}y_{11}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{d}y_{12}+\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{d}y_{21}+J_{sd},$
(23) $\displaystyle\tilde{Y}_{ss}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{s}y_{11}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{s}y_{12}+\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}y_{21}+J_{ss},$
(24)

where $\tilde{Y}_{ij}$ is defined in Eq.(12), $\tilde{h}_{x_{k}}^{i}$ is given
by Eq.(20), and $J_{ij}=\sum_{(k,l)\in
J_{1}}{\tilde{h}_{a_{k}}^{i}\tilde{h}_{b_{l}}^{j}y_{kl}},(i,j\in\\{d,s\\})$
with $J_{1}=\\{(k,l)|k,l\geq 1;k+l\geq 4\\}$.

With these preparations, now we are going to introduce the following very
important condition:

$\frac{\tilde{h}_{x_{k}}^{s}}{\tilde{h}_{x_{k}}^{d}}\geq\frac{\tilde{h}_{x_{2}}^{s}}{\tilde{h}_{x_{2}}^{d}}\geq\frac{\tilde{h}_{x_{1}}^{s}}{\tilde{h}_{x_{1}}^{d}},\quad(x\in\\{a,b\\})$
(25)

for $k\geq 2$. We can easily prove that the imperfect sources used in practice
such as the coherent state source, the heralded source out of the parametric-
down conversion, satisfy the above restriction.

In order to get a lower bound of $y_{11}$, we should derive the expression of
$y_{11}$ with Eqs.(21-24) firstly. Combining Eqs.(21-23), we obtain the
expression of $y_{11}$ by eliminating $y_{12}$ and $y_{21}$ such that

$y_{11}=y_{11}^{(123)}+\sum_{(m,n)\in J_{1}}f_{11}^{(123)}(m,n)y_{mn},$ (26)

where $J_{1}=\\{(m,n)|m\geq 1,n\geq 1,m+n\geq 4\\}$,

$y_{11}^{(123)}=\frac{(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})\tilde{Y}_{dd}-\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})\tilde{Y}_{ds}-\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{d}(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})\tilde{Y}_{sd}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$
(27)

and

$f_{11}^{(123)}(m,n)=\frac{\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{n}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})+\tilde{h}_{a_{m}}^{d}\tilde{h}_{b_{1}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{2}}^{s}\tilde{h}_{b_{n}}^{d})}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})}.$
(28)

In these expressions, we use the superscript $*^{(123)}$ to denote the result
obtained with the first three equations from Eqs.(21-24). Under the conditions
presented in Eq.(25), we can easily find out that
$(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})\geq
0$,
$(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})\geq
0$,
$(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d})\geq
0$ for all $m\geq 1$ and
$(\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{2}}^{s}\tilde{h}_{b_{n}}^{d})\geq
0$ for all $n\geq 2$. Then we know that $f_{11}^{(123)}(m,n)\geq 0$ hold for
all $(m,n)\in J_{1}$. With this fact, we obtain a lower bound from Eq.(26) by
setting $y_{mn}=0,(m,n)\in J_{1}$ such that

$\underline{y_{11}}=y_{11}^{(123)}\leq y_{11},$ (29)

where $y_{11}^{(123)}$ is defined by Eq.(27). This and Eq.(27) are our major
formulas for the decoy-state method implementation for MDI-QKD in this
section.

Similarly, we can get other expressions with choosing any other three
equations from Eqs.(21-24). For example, we choose Eqs.(21-22,24). By
eliminating $y_{12}$ and $y_{21}$, we get another expression of $y_{11}$ such
that

$y_{11}=y_{11}^{(124)}+\sum_{(m,n)\in J_{1}}f_{11}^{(124)}(m,n)s_{mn},$ (30)

where

$y_{11}^{(124)}=\frac{\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})\tilde{Y}_{dd}+(\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})\tilde{Y}_{ds}-\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{d}(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})\tilde{Y}_{ss}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$
(31)

and

$f_{11}^{(124)}(m,n)=\frac{\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{n}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})+\tilde{h}_{a_{m}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{2}}^{s}\tilde{h}_{b_{n}}^{d})}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})}.$
(32)

Under the conditions presented in Eq.(25), we can also find out that
$f_{11}^{(124)}(m,n)\geq 0$ for all $(m,n)\in J_{1}$. Then we know that
$y_{11}^{(124)}$ is alos a lower bound of $y_{11}$. On the other hand, by
comparing $f_{11}^{(123)}(m,n)$ and $f_{11}^{(124)}(m,n)$, we have

$\displaystyle f_{11}^{(123)}(m,n)-f_{11}^{(124)}(m,n)$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{2}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{n}}^{d})}{-\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})}\leq
0,$

for all $(m,n)\in J_{1}$. Then we know that

$y_{11}^{(123)}\geq y_{11}^{(124)},$ (33)

with Eq.(26) and Eq.(30). With the relation presented in Eq.(33) we know that
the lower bound $y_{11}^{(123)}$ is tighter than the lower bound
$y_{11}^{(124)}$. In the same way, we can get another two lower bounds
$y_{11}^{(134)}$ and $y_{11}^{(234)}$ of $y_{11}$ with Eqs.(21,23-24) and
Eqs.(22-24) respectively. Furthermore, we can also prove that

$y_{11}^{(123)}\geq y_{11}^{(134)},\quad y_{11}^{(123)}\geq y_{11}^{(234)}.$
(34)

Now we only consider Eq.(21) and Eq.(24). By eliminating $y_{12}$ or $y_{21}$
respectively, we get two expressions of $y_{11}$ such that

$\displaystyle y_{11}$ $\displaystyle=$ $\displaystyle
y_{11}^{(14a)}+\sum_{(m,n)\in J_{2}}f_{11}^{(14a)}(m,n)y_{mn},$ (35)
$\displaystyle y_{11}$ $\displaystyle=$ $\displaystyle
y_{11}^{(14b)}+\sum_{(m,n)\in J_{2}}f_{11}^{(14a)}(m,n)y_{mn},$ (36)

where $J_{2}=\\{(m,n)|m\geq 1,n\geq 1,m+n\geq 3\\}$,

$\displaystyle y_{11}^{(14a)}$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{s}\tilde{Y}_{dd}-\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{d}\tilde{Y}_{ss}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{1}}^{s}(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$
(37) $\displaystyle y_{11}^{(14b)}$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}\tilde{Y}_{dd}-\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{Y}_{ss}}{\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})},$
(38)

and

$\displaystyle f_{11}^{(14a)}(m,n)$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{d}\tilde{h}_{a_{m}}^{s}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{s}\tilde{h}_{a_{m}}^{d}\tilde{h}_{b_{n}}^{d}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{1}}^{s}(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$
(39) $\displaystyle f_{11}^{(14b)}(m,n)$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{a_{m}}^{s}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}\tilde{h}_{a_{m}}^{d}\tilde{h}_{b_{n}}^{d}}{\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})}.$
(40)

For any sources used in the protocol, we must have either

$K_{a}=\frac{\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{s}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{d}}\leq\frac{\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}}{\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}}=K_{b}\quad\textrm{or}\quad
K_{a}>K_{b}.$

Suppose the former one holds, we can easily find out that
$f_{11}^{(14a)}(m,n)\geq 0$ for all $(m,n)\in J_{2}$ and $y_{11}^{(14a)}$ is a
lower bound of $y_{11}$. On the other hand, if $K_{a}\geq K_{b}$ holds, we
have $f_{11}^{(14b)}(m,n)\geq 0$ for all $(m,n)\in J_{2}$ and $y_{11}^{(14b)}$
is a lower bound of $y_{11}$. Considering the following two relations

$K_{a}-K_{b}=\frac{\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{d}},$
(41)

and

$\displaystyle f_{11}^{(14a)}(m,n)-f_{11}^{(14b)}(m,n)$ $\displaystyle=$
$\displaystyle\frac{\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{2}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{n}}^{d})(K_{a}-K_{b})}{\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{s}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$

we know that $K_{a}-K_{b}$ and $f_{11}^{(14a)}-f_{11}^{(14b)}$ have the same
sign which means that they are both positive or negative simultaneously. Then
we can write the lower bound of $y_{11}$ with Eq.(21) and Eq.(24) into the
following compact form

$y_{11}^{(14)}=\min\\{y_{11}^{(14a)},y_{11}^{(14b)}\\}.$ (42)

If both Alice and Bob use coherent pulses, the lower bound given by Eq.(42) is
just the result presented in Refs. lopa ; curty . In the coming, we will prove
that the lower bound $y_{11}^{(123)}$ given in Eq.(27) is more tightly than
$y_{11}^{(14)}$. Firstly, if we suppose $K_{a}\leq K_{b}$ holds, then we know
that
$\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}\leq\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d}$
and $y_{11}^{(14)}=y_{11}^{(14a)}$. For any $(m,n)\in J_{1}$ we have

$\displaystyle f_{11}^{(123)}(m,n)-f_{11}^{(14a)}(m,n)$ $\displaystyle=$
$\displaystyle-\frac{(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}})D_{a}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})},$

where
$D_{a}=(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{s}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{d}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}\tilde{h}_{b_{n}}^{d}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{d}\tilde{h}_{b_{n}}^{s})\geq\tilde{h}_{b_{2}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{2}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{2}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{n}}^{d})$.
Then we know that

$\displaystyle f_{11}^{(123)}(m,n)-f_{11}^{(14a)}(m,n)$ $\displaystyle\leq$
$\displaystyle-\frac{\tilde{h}_{b_{2}}^{d}(\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{m}}^{s}-\tilde{h}_{a_{1}}^{s}\tilde{h}_{a_{m}}^{d})(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{n}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{n}}^{d})}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{d}(\tilde{h}_{b_{1}}^{d}\tilde{h}_{b_{2}}^{s}-\tilde{h}_{b_{1}}^{s}\tilde{h}_{b_{2}}^{d})}\leq
0.$

We can easily know that $y_{11}^{(123)}\geq y_{11}^{(14)}$ when $K_{a}\leq
K_{b}$ with this equation. Secondly, if we suppose $K_{a}>K_{b}$ holds, we can
easily prove that $f_{11}^{(123)}(m,n)-f_{11}^{(14b)}(m,n)\leq 0$ for all
$(m,n)\in J_{1}$ within the same way. Then we get $y_{11}^{(123)}\geq
y_{11}^{(14)}$ when $K_{a}>K_{b}$. This completes the proof that
$y_{11}^{(123)}\geq y_{11}^{(14)}$.

In order to estimate the final key rate, we also need the upper bound of error
rate caused by the two single-photon pulses, say $e_{11}$. Similar to the
total gain, the total error rate with source $\alpha\beta$ chosen by Alice and
Bob can be written as ind2

$\displaystyle\tilde{T}_{dd}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}t_{11}+\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{d}t_{12}+\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{d}t_{21}+K_{dd},$
(43) $\displaystyle\tilde{T}_{ds}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{s}t_{11}+\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{2}}^{s}t_{12}+\tilde{h}_{a_{2}}^{d}\tilde{h}_{b_{1}}^{s}t_{21}+K_{ds},$
(44) $\displaystyle\tilde{T}_{sd}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{d}t_{11}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{d}t_{12}+\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{d}t_{21}+K_{sd},$
(45) $\displaystyle\tilde{T}_{ss}$ $\displaystyle=$
$\displaystyle\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{1}}^{s}t_{11}+\tilde{h}_{a_{1}}^{s}\tilde{h}_{b_{2}}^{s}t_{12}+\tilde{h}_{a_{2}}^{s}\tilde{h}_{b_{1}}^{s}t_{21}+K_{ss},$
(46)

where $T_{ij}=Y_{ij}E_{ij},(i,j\in\\{v,d,s\\})$, $t_{mn}=y_{mn}e_{mn}$, and

$\displaystyle\tilde{T}_{dd}$ $\displaystyle=$
$\displaystyle\frac{T_{dd}}{a_{0}^{d}b_{0}^{d}}-\frac{T_{dv}}{a_{0}^{d}b_{0}^{v}}-\frac{T_{vd}}{a_{0}^{v}b_{0}^{d}}+\frac{T_{vv}}{a_{0}^{v}b_{0}^{v}},$
(47) $\displaystyle\tilde{T}_{ds}$ $\displaystyle=$
$\displaystyle\frac{T_{ds}}{a_{0}^{d}b_{0}^{s}}-\frac{T_{dv}}{a_{0}^{d}b_{0}^{v}}-\frac{T_{vs}}{a_{0}^{v}b_{0}^{s}}+\frac{T_{vv}}{a_{0}^{v}b_{0}^{v}},$
(48) $\displaystyle\tilde{T}_{sd}$ $\displaystyle=$
$\displaystyle\frac{T_{sd}}{a_{0}^{s}b_{0}^{d}}-\frac{T_{sv}}{a_{0}^{s}b_{0}^{v}}-\frac{T_{vd}}{a_{0}^{v}b_{0}^{d}}+\frac{T_{vv}}{a_{0}^{v}b_{0}^{v}},$
(49) $\displaystyle\tilde{T}_{ss}$ $\displaystyle=$
$\displaystyle\frac{T_{ss}}{a_{0}^{s}b_{0}^{s}}-\frac{T_{sv}}{a_{0}^{s}b_{0}^{v}}-\frac{T_{vs}}{a_{0}^{v}b_{0}^{s}}+\frac{T_{vv}}{a_{0}^{v}b_{0}^{v}},$
(50)

and $K_{ij}=\sum_{(k,l)\in
J_{1}}{\tilde{h}_{a_{k}}^{i}\tilde{h}_{b_{l}}^{j}t_{kl}},(i,j\in\\{d,s\\})$
with $J_{1}=\\{(k,l)|k,l\geq 1,k+l\geq 4\\}$. According to Eq.(43), we can
find out the upper bound of $e_{11}$ such that

$e_{11}\leq
e_{11}^{(1)}=\frac{\tilde{T}_{dd}}{\tilde{h}_{a_{1}}^{d}\tilde{h}_{b_{1}}^{d}y_{11}}=\overline{e_{11}}.$
(51)

In the protocol, there are two different bases. We denote $y_{11}^{Z}$ and
$y_{11}^{X}$ for yields of single-photon pulse pairs in the $Z$ and $X$ bases,
respectively. Consider those post-selected bits cased by source $d_{A}d_{B}$
in the $Z$ basis. After an error test, we know the bit-flip error rate of this
set, say $T_{d_{A}d_{B}}^{Z}=Y_{d_{A}d_{B}}^{Z}E_{d_{A}d_{B}}^{Z}$. We also
need the phase-flip rate for the subset of bits which are caused by the two
single-photon pulse, say $e_{11}^{ph}$, which is equal to the flip rate of
post-selected bits caused by a single photon in the $X$ basis, say
$e_{11}^{X}$. Given this, we can now calculate the key rate by the well-know
formula. For example, for those post-selected bits caused by source
$s_{A}s_{B}$, it is

$R=a_{1}^{s}b_{1}^{s}y_{11}^{Z}[1-H(e_{11}^{X})]-Y_{s_{A}s_{B}}^{Z}fH(E_{s_{A}s_{B}}^{Z}),$
(52)

where $f$ is the efficiency factor of the error correction method used.

##  III Numerical Simulation

Figure 1: (Color online) The relative value between estimated parameter of
$y_{11}$ and the asymptotic limit of the infinite decoy-state method versus
the total channel transmission loss using 3-intensity decoy state MDI-QKD. In
the simulations, we assume that $\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$,
$\mu_{s}=\nu_{s}=0.5$ and the value of other parameters are presented in Table
1.
Figure 2: (Color online) The estimated parameter of $e_{11}$ versus the total
channel transmission loss using 3-intensity decoy state MDI-QKD. In the
simulations, we assume that $\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$,
$\mu_{s}=\nu_{s}=0.5$ and the value of other parameters are presented in Table
1.
Figure 3: (Color online) The estimated key rate $R$ versus channel
transmission using 3-intensity decoy state MDI-QKD. In the simulations, we
assume that $\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$,
$\mu_{s}=\nu_{s}=0.5$ and the value of other parameters are presented in Table
1.
Figure 4: (Color online) The relative value between the optimal key rate
obtained with different methods and the asymptotic limit of the infinite
decoy-state method versus the total channel transmission loss using
3-intensity decoy state MDI-QKD. In the simulations, we assume that
$\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$, $\mu_{s}=\nu_{s}$ and the value
of other parameters are presented in Table 1. The optimal key rate is just the
value with maximizing the key rate with $\mu_{s}=\nu_{s}\in(\mu_{d},1)$.
Figure 5: (Color online) The optimal intensities versus the total channel transmission loss using 3-intensity decoy state MDI-QKD. In the simulations, we assume that $\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$, $\mu_{s}=\nu_{s}$ and the value of other parameters are presented in Table 1. The optimal densities of the signal-state pulses are obtained by maximizing the key rate. Table 1: List of experimental parameters used in numerical simulations: $e_{0}$ is the error rate of background, $e_{d}$ is the misalignment-error probability; $p_{d}$ is the dark count rate per detector; $f$ is the error correction inefficiency. $e_{0}$ | $e_{d}$ | $p_{d}$ | $f$
---|---|---|---
0.5 | 1.5% | $3.0\times 10^{-6}$ | 1.16

Now, we present some numerical simulations to comparing our results with the
existing results lopa ; curty . Below for simplicity, we suppose that Alice
and Bob use the coherent-state sources. Here, we denote Alice’s sources
$\\{v_{A},d_{A},s_{A}\\}$ by their intensities $\\{\mu_{v},\mu_{d},\mu_{s}\\}$
and Bob’s sources $\\{v_{B},d_{B},s_{B}\\}$ by their intensities
$\\{\nu_{v},\nu_{d},\nu_{s}\\}$ respectively. The UTP locates in the middle of
Alice and Bob, and the UTP’s detectors are identical, i.e., they have the same
dark count rate and detection efficiency, and their detection efficiency does
not depend on the incoming signals. We shall estimate what values would be
probably observed for the gains and error rates in the normal cases by the
linear models as in ind2 ; Wang2013 ; Wang201309 :

$\displaystyle|n\rangle\langle
n|=\sum_{k=0}^{n}C_{n}^{k}\xi^{k}(1-\xi)^{n-k}|k\rangle\langle k|$

where $\xi^{k}$ is the transmittance for a distance from Alice to the UTB. For
fair comparison, we use the same parameter values used in ind2 ; Wang2013 ;
Wang201309  for our numerical evaluation, which follow the experiment reported
in UrsinNP2007 . For simplicity, we shall put the detection efficiency to the
overall transmittance $\eta=\xi^{2}\zeta$. We assume all detectors have the
same detection efficiency $\zeta$ and dark count rate $p_{d}$. The values of
these parameters are presented in Table 1. With this, the total gains
$Y_{\mu_{i},\nu_{j}}^{\omega},(\omega=X,Z)$ and error rates
$Y_{\mu_{i},\nu_{j}}^{\omega}E_{\mu_{i},\nu_{j}}^{\omega},(\omega=X,Z)$ of
Alice’s intensity $\mu_{i}(i\in\\{v,d,s\\})$ and Bob’s intensity
$\nu_{j}(j\in\\{v,d,s\\})$ can be calculated. By using these values, we can
estimate the lower bounds of yield $y_{11}^{Z}$ with Eq.(29) and Eq.(42).
Also, we can estimate the upper bounds of error rate $e_{11}^{X}$ with Eq.(51)
where the lower bound of $y_{11}^{X}$ being estimated by Eq.(29) and Eq.(42).
The estimated parameters of $y_{11}$ and $e_{11}$ are shown in Fig.1 and
Fig.2, respectively, which clearly shows that our methods are more tightly
than the pre-existed results. In order to see more clearly, in Fig.1, we plot
the relative value of $y_{11}$ to the result obtained with the infinite decoy-
state method. We can observe that our results are more close to the asymptotic
limit of the infinite decoy-state method than the pre-existed results lopa ;
curty . Furthermore, with these parameters, we can estimate the final key rate
$R$ of this protocol with Eq.(52) which is shown in Fig.3. In these figures,
the blue dashed line is obtained by Eq.(42) which is just the results
presented in Refs. lopa ; curty , the red solid line is obtained by the method
presented in section 2 with Eq.(29), the black dash-dot line is the result
obtained by the infinite decoy-state method. In the simulation, the densities
used by Alice and Bob are assigned to $\mu_{v}=\nu_{v}=0.01$,
$\mu_{d}=\nu_{d}=0.1$, $\mu_{s}=\nu_{s}=0.5$.

Furthermore, if we fix the densities of the decoy-state pulses used by Alice
and Bob, the final key rate will change with Alice and Bob taking different
densities for their signal-state pulses. Then we can find out the optimal
densities of their signal-state pulses with maximizing the final key rate.
Here, we also take $\mu_{v}=\nu_{v}=0.01$, $\mu_{d}=\nu_{d}=0.1$ and assume
that $\mu_{s}=\nu_{s}\in(\mu_{d},1)$. In Fig.4, we plot the relative value of
the optimal key rate to the result obtained with the infinite decoy-state
method. We can observe that our result is better than the pre-existed results.
The optimal densities with the optimal key rate versus the total channel
transmission loss is given in Fig.5.

##  IV Conclusion

We study the MDI-QKD in practice where the intensities of all 3 different
states can be nonzero. Our result here is the most general result for the
3-intensity method: Setting the weakest state to be zero, we obtain the result
in the special case where a vacuum state and two non-vacuum states are
usedWang2013 . The result here is not limited to the coherent states. It
applies to any states that satisfy Eq.(25), e.g., the heralded states from the
parametric down conversion. Our result is most efficient: It offers a more
tightened bound for $y_{11}$ and therefore a higher key rate than the prior
artlopa , as has been strictly proven and also numerically demonstrated. The
key rate can be further improved when it is combined with the method as shown
in Ref.Wang201309 .

Acknowledgement: We acknowledge the support from the 10000-Plan of Shandong
province, the National High-Tech Program of China Grants No. 2011AA010800 and
No. 2011AA010803 and NSFC Grants No. 11174177 and No. 60725416.

## References

  * (1) C.H. Bennett and G. Brassard, in Proc. of IEEE Int. Conf. on Computers, Systems, and Signal Processing (IEEE, New York, 1984), pp. 175-179.
  * (2) N. Gisin, G. Ribordy, W. Tittel, and H. Zbinden, Rev. Mod. Phys. 74, 145 (2002); N. Gisin and R. Thew, Nature Photonics, 1, 165 (2006); M. Dusek, N. Lütkenhaus, M. Hendrych, in Progress in Optics VVVX, edited by E. Wolf (Elsevier, 2006); V. Scarani, H. Bechmann-Pasqunucci, N.J. Cerf, M. Dusek, N. Lütkenhaus, and M Peev, Rev. Mod. Phys. 81, 1301 (2009).
  * (3) H. Inamori, N. Lütkenhaus, D. Mayers, European Physical Journal D, 41, 599 (2007), which appeared in the arXiv as quant-ph/0107017; D. Gottesman, H.K. Lo, N. Lütkenhaus, and J. Preskill, Quantum Inf. Comput. 4, 325 (2004).
  * (4) W.-Y. Hwang, Phys. Rev. Lett. 91, 057901 (2003).
  * (5) X.-B. Wang, Phys. Rev. Lett. 94, 230503 (2005).
  * (6) H.-K. Lo, X. Ma, and K. Chen, Phys. Rev. Lett. 94, 230504 (2005).
  * (7) Y. Adachi, T. Yamamoto, M. Koashi, and N. Imoto, Phys. Rev.Lett. 99, 180503 (2007).
  * (8) M. Hayashi, Phys. Rev. A 74, 022307 (2006); ibid 76, 012329 (2007).
  * (9) D. Rosenberg et al., Phys. Rev. Lett. 98, 010503 (2007); T. Schmitt-Manderbach et al., Phys. Rev. Lett. 98, 010504 (2007); Cheng-Zhi Peng et al. Phys. Rev. Lett. 98, 010505 (2007); Z.-L. Yuan, A. W. Sharpe, and A. J. Shields, Appl. Phys. Lett. 90, 011118 (2007); Y. Zhao, B. Qi, X. Ma, H.-K. Lo and L. Qian, Phys. Rev. Lett. 96, 070502 (2006); Y. Zhao, B. Qi, X. Ma, H.-K. Lo, and L. Qian, in Proceedings of IEEE International Symposium on Information Theory, Seattle, 2006, pp. 2094–2098 (IEEE, New York).
  * (10) X.-B. Wang, C.-Z. Peng et al. Phys. Rev. A 77, 042311 (2008); J.-Z. Hu and X.-B. Wang, Phys. Rev. A, 82, 012331(2010).
  * (11) X.-B. Wang, T. Hiroshima, A. Tomita, and M. Hayashi, Physics Reports 448, 1(2007).
  * (12) X.-B. Wang, L. Yang, C.-Z. Peng and J.-W. Pan, New J. Phys. 11, 075006 (2009).
  * (13) G. Brassard, N. Lütkenhaus, T. Mor, and B.C. Sanders, Phys. Rev. Lett. 85, 1330 (2000); N. Lütkenhaus, Phys. Rev. A 61, 052304 (2000); N. Lütkenhaus and M. Jahma, New J. Phys. 4, 44 (2002).
  * (14) B. Huttner, N. Imoto, N. Gisin, and T. Mor, Phys. Rev. A 51, 1863 (1995); H.P. Yuen, Quantum Semiclassic. Opt. 8, 939 (1996).
  * (15) L. Lyderson et al, Nature Photonics, 4, 686(2010); I. Gerhardt et al, Nature Commu. 2, 349 (2011).
  * (16) D. Mayers and A. C.-C. Yao, in Proceedings of the 39th Annual Symposium on Foundations of Computer Science (FOCS98) (IEEE Computer Society, Washington, DC, 1998), p. 503; A. Acin et al., Phys. Rev. Lett. 98, 230501 (2007); Scarani V and Renner R Phys. Rev. Lett. 100, 302008 (2008); Scarani V and Renner R 2008 3rd Workshop on Theory of Quantum Computation, Communication and Cryptography (TQC 2008), (University of Tokyo, Tokyo 30 Jan C1 Feb 2008) See also arXiv:0806.0120
  * (17) S.L. Braunstein and S. Pirandola, Phys. Rev. Lett. 108, 130502 (2012).
  * (18) H.-K. Lo, M. Curty, and B. Qi, Phys. Rev. Lett., 108,130503(2012), K. Tamaki et al, Phys. Rev. A, 85, 042307 (2012).
  * (19) Qin-Wang abd Xiang-Bin Wang, arXiv:1305.6480.
  * (20) Xiang-Bin Wang, Phys. Rev. A 87, 012320 (2013).
  * (21) Allison Rubenok, Joshua A. Slater, Philip Chan, Itzel Lucio-Martinez, Wolfgang Tittel, 1304.2463v1.
  * (22) P. Chan, J. A. Slater, I. Lucio-Martinez, A. Rubenok, W. Tittel, arxiv:1204.0738v1.
  * (23) Y. Liu et al, arXiv:1209.6178v1.
  * (24) Feihu Xu, Marcos Curty, Bing Qi, Hoi-Kwong Lo , arXiv:1305.6965v1.
  * (25) Chun Zhou, Wan-Su Bao, Wei Chen, Hong-Wei Li, Zhen-Qiang Yin, Yang Wang, Zheng-Fu Han, arXiv: 1308.3374v1.
  * (26) M. Curty et al, arXiv:1307.1081v1.
  * (27) Z.-W. Yu, Y.-H. Zhou, X.-B. Wang, arXiv:1308.5677.
  * (28) Z.-W. Yu, Y.-H. Zhou, X.-B. Wang, arXiv:1309.0471.
  * (29) R. Ursin, F. Tiefenbacher, T. Schmitt-Manderbach, H. Weier, T. Scheidl, M. Lindenthal, B. Blauensteiner, T. Jennewein, J. Perdigues, P.Trojek, B. Oemer, M. F uerst, M. Meyenburg, J. Rarity, Z. Sodnik, C. Barbieri, H. Weinfurther, and A. Zeilinger, Nat. Phys. 3, 481 (2007).


# LAViTeR: Learning Aligned Visual and Textual Representations
Assisted by Image and Caption Generation

Mohammad Abuzar Shaikh equal contributions. Department of Computer Science and
Engineering, University at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>Zhanghexuan
Ji00footnotemark: 0 Department of Computer Science and Engineering, University
at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>Dana Moukheiber
Department of Computer Science and Engineering, University at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>Yan Shen
Department of Computer Science and Engineering, University at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>Sargur Srihari
Department of Computer Science and Engineering, University at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>Mingchen Gao
Department of Computer Science and Engineering, University at Buffalo,
The State University of New York, Buffalo, NY, USA
<EMAIL_ADDRESS>
###### Abstract

Pre-training visual and textual representations from large-scale image-text
pairs is becoming a standard approach for many downstream vision-language
tasks. The transformer-based models learn inter- and intra-modal attention
through a list of self-supervised learning tasks. This paper proposes LAViTeR,
a novel architecture for visual and textual representation learning. The main
module, Visual Textual Alignment (VTA) will be assisted by two auxiliary
tasks, GAN-based image synthesis and Image Captioning. We also propose a new
evaluation metric measuring the similarity between the learnt visual and
textual embedding. The experimental results on two public datasets, CUB and
MS-COCO, demonstrate superior visual and textual representation alignment in
the joint feature embedding space. Our code is available at
https://github.com/mshaikh2/LaViTeR

##  1 Introduction

Learning cross-modal visual and textual representation is essential for
bridging the semantic gap between images and languages. It is the cornerstone
for a wide range of vision-language (V+L) tasks, such as image-text cross-
modal retrieval, visual question answering (VQA) [2], image captioning [2],
and so on.

Figure 1: An overview of the end-to-end LAViTeR network. VTA module is
assisted by ITM and TIM modules, which in-turn learns to better align the
corresponding visual and textual counterparts. The bidirectional arrows
indicate the alignment between words and their respective objects in the given
image. The intra-word arrows indicate the relationships between the input
words that the network learns.

Inspired by the success of BERT [9] and XLNet [48] using self-supervised
learning on natural language processing, there has been a surging research
interest in vision-language pre-training on image-text pairs. The learned
task-agnostic representation is shown to be effective for many image-language
applications after fine-tuning on specific downstream tasks. Self-supervised
learning is designed to explore the organization of the data as its own source
of supervision. This promising approach releases the burden of annotating data
with ground truth labels, provides an opportunity to explore a large amount of
unlabeled data such as image-text pairs, video-text pairs in free form format
from online platforms. This approach has been applied to radiology images
combined with their associated reports [24, 5] to leverage the abundance of
unlabeled medical data. This data can retrospectively be collected from
clinical routine, and has a lot of potential for self-supervised learning.

The representation alignment can be roughly classified into two categories,
one-to-one matching and many-to-many matching. One-to-one matching focuses on
the global representation from images and sentences, and then associates them
by exploiting visual-semantic embedding [43]. Many-to-many matching methods
incorporate relationship between regions of a image and words of a sentence to
capture fine-grained cross-modal matching [19].

Analogous to the pre-training task in BERT, some pre-training tasks for image-
text pairs include the Masked Language Modeling conditioned on image regions
and Masked Region Modeling conditioned on input text. Those approaches
randomly mask some words or regions from the input and use a Transformer model
to recover the words or regions. Many of the fine-grained region-word matching
rely on the modern object detectors [6, 23, 44], usually Faster R-CNN [33], to
detect salient regions and match them to words. However, the state-of-the-art
object detection, which needs to leverage large amount of annotated bounding
boxes for supervised learning, is not always available for domain-specific
datasets.

Motivated by the above discussion, we propose a model for Learning Aligned
Visual and Textual Representation (LAViTeR). As shown in Figure 1, the main
goal of LAViTeR is to learn the joint multi-modal embedding using visual
textual alignment (VTA) module, which is assisted by two other self-supervised
modules, the text-to-image module (TIM) and image-to-text module (ITM). The
method was inspired by CycleGAN [52] and its extension MirrorGAN [31] on image
synthesis. In our model, the given images and text are encoded to generate
corresponding text and images, respectively. The generated text and images are
trained to be mapped back to the original images and text in a cycle. Not only
the features learned from real image-text pairs are aligned in the VTA module,
the features learned from real-image-fake-text pairs and the fake-image-real-
text pairs will also assist the representation learning and alignment. These
generated images and texts can provide much more samples outside the training
set and make our model more diverse and robust to changes in real data. The
proposed method uses high-level features without any explicit supervision,
avoiding explicit object detection. Our approach is suitable for the
situations where the state-of-the-art object detection model is not feasible,
and where bounding box annotations are not available for training.

Our contributions are summarized as follows: 1) We introduce LAViTeR for the
image-text representation for V+L tasks. 2) We introduce two auxiliary pre-
training tasks, GAN-based image synthesis and image captioning, to assist the
representation learning. 3) We propose to use a new metric to quantitatively
evaluate the similarity between the image and text representation in the
embedded space.

##  2 Related Work

In this section, we discuss related work about multi-model representation
learning and alignment. We also briefly review two main tasks in our
architecture to pre-train the joint representation, the GAN-based image
generation and image captioning.

###  2.1 Multi Model Representation Learning

VilBERT [26] and LXMERT [37] are the two pioneering works in image-text joint
representation learning, utilizing two streams of Transformers to images and
text independently. Those image and text representations then fused by a self-
attention mechanism in the later stage. After those two pioneered work,
single-stream architecture has also been proposed using a single Transformer
to jointly learn image-text embedding, such as UNITER [6], VisualBERT [22],
Unicoder-VL [21], VL-VERT [35], B2T2 [1]. Typical self-supervised learning
tasks, such as masked language modeling, masked region modeling, image-text
matching, and word-region alignment are applied to pre-train the models. More
recently, VILLA has been proposed using adversarial training as a general
framework can be applied to any V+L models [11].

There are quite a lot of work trying to explicitly enforce the word and region
alignment, such as VisualBERT [22], UNITER [6], Oscar [23], MMAC [44]. For
example, UNITER [6] uses the Optimal Transport [30] to explicitly calculate
the minimum cost of transporting between the image embedding to word
embedding.

###  2.2 Text to Image Generation

Image synthesis from text is a fundamental task in multi-modal learning across
vision and language. Most proposed works in image generation are based on
conditional Generative Adversarial Networks (GAN). A common approach utilizes
a text encoder, mostly RNN-based text encoder before the popularity of
Transformer, to encode the text description to guide the image generator [32,
50]. The attention mechanism is widely used to guide the generator to focus on
certain words when generating specific regions [47]. The attention mechanism
is used to capture the similarity between the generated images and the
sentences in both the global level and fine-grained word/region level.
MirrorGAN [31], tries to learn better text-to-image generation by re-
describing the generated images.

###  2.3 Image Captioning

Image Captioning is the reverse process of text-to-image generation. It
typically consists of a CNN encoder and an RNN decoder to transfer the
information from images to the generated text description [41]. Attention
mechanism has been shown very effective focusing on salient objects while
generating the corresponding words [46]. Following the success of Faster-RCNN
[33] in object detection, bottom-up features provide informative regions in
the image, which are used for region level attention [2, 17] and visual scene
graph modeling [49]. Recently, transformer architecture [39] is also used for
image captioning, which further boosts the captioning performance with its
implicit self-attention mechanism [14, 15, 51].

Figure 2: The architecture of the proposed LAViTeR. The pipelines with dotted
outlines are the two assisting tasks, namely image to text and text to image
conversion. Feature vectors of real image regions are indicated by $r$ while
$v$ denotes the global image feature vector. Real text sentence level feature
vector is indicated by $s$ while $w$ denotes the word level feature vectors.
Similarly all $\hat{w},\hat{s},\hat{r},\hat{v}$ indicates the features
extracted from generated samples. $L$ stands for various losses. Dotted arrows
indicate the vectors that contribute the loss. Solid arrows indicate the
vectors are input to the subsequent network.

##  3 Learning Aligned Visual and Textual Representations

As shown in Figure 2, our proposed Learning Aligned Visual and Textual
Representations (LAViTeR) model consists of three modules: the visual text
alignment (VTA) module, which contains image encoder $F_{I}$ and text encoder
$F_{T}$; the text-to-image (TIM) module, which is the image generator $G$; and
image-to-text (ITM) module with the text generator $C$. VTA learns visual-text
representations via matching real image-text pairs as our main task. In
addition, we introduce two novel assisting tasks, illustrated in blue and
green dashed boxes, for a better representation learning: fake-image-real-text
matching with fake images generated from TIM; and real-image-fake-text
matching where fake texts are converted from ITM using VTA image
representations. Meanwhile, both TIM and ITM are also trained on their own
losses. We will introduce each of them in the following part of this section.

###  3.1 VTA: Visual Text Alignment

Given an image-text tuple $\langle I_{i},T_{i}\rangle$, we want to learn the
alignment between the words in $T_{i}$ and the parts of image $I_{i}$. For
this, we first extract the global features $v$ and local features $r$ using an
InceptionV3 [36] based image encoder, where $r\in\mathbb{R}^{D\times M}$ is
flattened from the intermediate feature map of ‘mixed_6e’ layer and
$v\in\mathbb{R}^{D}$ from the last average pooling layer. Both of them are
projected to the representation feature space with a projection layer. This is
denoted as a function $F_{I}$ in Figure 2, such that $r,v=F_{I}(I)$. Next, we
extract sentence and word level features $s$ and $w$ respectively using a
Transformer [39] based text encoder. In Figure 2, this text encoder is denoted
as a function $F_{T}$, such that, $w,s=F_{T}(T)$, where
$w\in\mathbb{R}^{D\times N}$ and $s\in\mathbb{R}^{D}$.

We define the $F_{T}$ as follows: Given text $T\in\mathbb{R}^{N}$, we use
word-based token to embed it as $e\in\mathbb{R}^{D\times N}$, which is summed
with positional encoding as the input of $F_{T}$. In transformer encoder
layer, $e$ is first transformed into queries $Q=W^{T}_{Q}e$, keys
$K=W^{T}_{K}e$ and values $V=W^{T}_{V}e$ within each attention head, where
$W_{Q},W_{K},Q_{V}\in\mathbb{R}^{D\times D_{k}}$ in our setting. Then “scaled
dot-product self-attention” is applied to $e$ as follows:

$\text{Attention}(Q,K,V)=f_{s}(\frac{QK^{T}}{\sqrt{D_{k}}})V$ (1)

where $f_{s}$ is the softmax function.

Multi-head attention is applied to the self-attention sub-layer and the
outputs from $h$ heads are concatenated:

$\displaystyle\text{head}_{i}=\text{Attention}(W^{T}_{Q_{i}}e,W^{T}_{K_{i}}e,W^{T}_{V_{i}}e)$
(2)
$\displaystyle\text{Multihead}(e)=W_{O}\text{Concat}(\text{head}_{1},...,\text{head}_{h})$

where $W_{O}\in\mathbb{R}^{D\times hD_{k}}$.

The output from the multi-head attention is then sent to a feed-forward
network. The residual mechanism [13] and layer normalization are applied to
multi-head attention and FFN outputs:

$\displaystyle\tilde{e}$
$\displaystyle=\text{LayerNorm}(e+\text{Multihead}(e))$ (3) $\displaystyle w$
$\displaystyle=\text{LayerNorm}(\tilde{e}+\text{FFN}(\tilde{e}))$

and $s=\bar{w}$ is used as the sentence representation feature.

Similar to [47] we align the word to image regions by implementing a word-
level attention mechanism. First, the word-region attention score $\alpha$ is
obtained by multiplying the query, $w$ with context, $r$, and then normalizing
the product using softmax. Next $\alpha$ is multiplied with the context $r$ to
obtain the contextual vector $c$.

$\displaystyle m$ $\displaystyle=w^{T}\cdot r$ (4) $\displaystyle\alpha$
$\displaystyle=f_{s_{M}}(\gamma_{1}f_{s_{N}}(m))$ $\displaystyle c$
$\displaystyle=r\odot\alpha$

Where $m\in\mathbb{R}^{N\times M}$ is the match vector; $f_{s_{N}}$ is the
soft-max operation along $N$ words in text $T_{i}$; $f_{s_{M}}$ is the soft-
max operation along $M$ sub-regions of image $I_{i}$; $\gamma_{1}$ is a hyper-
parameter to tune the required amount of visual attention for a word and
$\odot$ is a matrix multiplication operation. Next, we calculate the element
wise cosine similarity between $c$ and $w$ as $cos=(c^{T}w)/(||c||||w||)$ and
compute the image to text matching score $S$ by following the work done in [3,
45]:

$\displaystyle
S(I_{i},T_{i})=\log(\sum_{i=1}^{N-1}exp(\gamma_{2}cos))^{\frac{1}{\gamma_{2}}}$
(5)

where $\gamma_{2}$ is the importance magnification factor of the most relevant
word and image sub-region in the given pair $\langle I_{i},T_{i}\rangle$.

Finally, inspired by to [10, 18], we calculate the the posterior probability
$P$ of image $I_{i}$ matching with text $T_{i}$ in a batch of $B$ paired
samples:

$\displaystyle P(I_{i},T_{i})$
$\displaystyle=f_{s_{B}}(\gamma_{3}S(I_{i},T_{i}))$ (6) $\displaystyle
L_{m_{w}}^{IT}$ $\displaystyle=-log(P(I_{i},T_{i}))$

where $f_{s_{B}}$ is the soft-max of matching score $S$ over $B$ paired
samples and $\gamma_{3}$ is a hyper-parameter and $L_{m_{w}}^{IT}$ is the loss
when features of image sub-regions $r$ are matched to features of words $w$ in
text. Here, the text and image samples at different index are considered as
negative pairs and the samples at same index are considered positive pairs. To
maintain the symmetry, we also calculate $L_{m_{w}}^{TI}$ where the image and
text variables are switched:

$\displaystyle P(T_{i},L_{i})$
$\displaystyle=f_{s_{B}}(\gamma_{3}S(T_{i},L_{i}))$ (7) $\displaystyle
L_{m_{w}}^{TI}$ $\displaystyle=-log(P(T_{i},L_{i}))$

Furthermore, we calculate the sentence level matching loss $L_{m_{s}}^{IT}$ by
computing cosine similarity between global vectors $v$ and $s$ as
$cos=(v^{T}s)/(||v||||s||)$ and substituting the value of $cos$ in Equation 5,
6 and 8. Similar to before we can calculate $L_{m_{s}}^{TI}$ by switching $s$
and $v$. Thus we compute the total matching loss by adding all the losses.

$\displaystyle
L_{m}=L_{m_{s}}^{TI}+L_{m_{s}}^{IT}+L_{m_{w}}^{TI}+L_{m_{w}}^{IT}$ (8)

During the first phase of training the objective is to reduce $L_{m}$ for
pairs of real image $I$ and corresponding text $T$. For this training we
preset $\gamma_{1},\gamma_{2},\gamma 3$ as per the settings defined in [47]
and the batch size $B=8$.

###  3.2 TIM: Text to Image Module

To learn the parameters for transforming the textual domain to visual
counterpart we use Conditional Generative Adversarial Networks [12, 28] with
the sentence vector $s$ as the conditional input. Inspired by the AttnGAN
[47], we employ a cascade of GANs. Formally, our TIM task has $k$
discriminators with coupled generators $G\in\\{G_{1}\dots G_{k}\\}$ that
generate images $\hat{I}\in\\{I_{i1}\dots I_{ik}\\}$ of different scales,
where the suffix $i$ indicate the $i^{th}$ datapoint. We utilize only the
output of the $k^{th}$ generator for calculating the assisted losses explained
in Equation 15.

We first transform a real text sample $T_{i}$ using the transformer [39] based
text encoder $F_{T}$ to output a sentence vector $s$ and word vector
$w\in\mathbb{R}^{D\times N}$. We denote the entire image generation system,
including generators and discriminators as a function $G$ as displayed in
Figure 2. A 1-D uniformly sampled random noise vector $z$ along with $w$ as
condition, are merged and fed as input to $G$ which outputs an RGB image
$\hat{I_{i}}$ corresponding to $T_{i}$

The image generative loss $L_{G}$ is thus calculated in an adversarial setup
and is defined as follows:

$L_{G}=-\frac{1}{2}\mathbb{E}_{\hat{I}\sim
P_{G}}[\log(D(\hat{I}))]-\frac{1}{2}\mathbb{E}_{\hat{I}\sim
P_{G}}[\log(D(\hat{I},s))]$

(9)

We train the discriminators, to learn to distinguish between the real $I$ and
fake $\hat{I}$ samples, alternately with the generators while reducing the
cross-entropy loss as below:

$\begin{aligned} L_{D}=-\frac{1}{2}\mathbb{E}_{I\sim
P_{data}}[\log(D(I))]-\frac{1}{2}\mathbb{E}_{\hat{I}\sim
P_{G}}[\log(1-D(\hat{I}))]\\\ -\frac{1}{2}\mathbb{E}_{I\sim
P_{data}}[\log(D(I,s))]-\frac{1}{2}\mathbb{E}_{\hat{I}\sim
P_{G}}[\log(1-D(\hat{I},s))]\end{aligned}$

(10)

where $I$ is from the read data distribution $P_{data}$ and $\hat{I}$ is from
the generated data distribution $P_{G}$. In Equation 10 the first and the
second terms are unconditional and conditional losses as defined in [50].
Furthermore, we calculate the matching loss $L^{I}_{m}$ between fake image
vectors $\hat{r}$, $\hat{v}$ and real text vectors $w$, $s$ respectively,
similar to the process outlined in Subsection 3.1. Hence the total loss for
the TIM can be defined as:

$L_{TIM}=L_{G}+\lambda_{\hat{I}}L_{m}^{I}$ (11)

where $\lambda_{\hat{I}}$ is the hyper-parameter which is tuned to get better
performance and the matching loss $L_{m}^{I}$ between the generated image
$\hat{I}_{i}$ to real text $T_{i}$, as explained in Subsection 3.4.

###  3.3 ITM: Image to Text Module

The Image-to-Text branch (ITM) aims to generate fake text to assist VTA
training in 3.1. It has been shown that the attention mechanism leverages the
performance of image captioning models [46]. Inspired by the power and success
of the transformer layer in various vision tasks [17, 15, 14, 4], we decide to
use a transformer based image captioning model $C$ as ITM.

Following the implementation of [39], we use a stack of transformer layers for
the transformer encoder $C_{e}$ and decoder $C_{d}$. Similar to [46], in order
to pass image regional features into the transformer, we use the flattened
regional feature sequence $r\in\mathbb{R}^{D\times M}$ from $F_{I}$ as the
input to $C_{e}$, where each column of $r$ is a representation corresponding
to a certain part of the image. As for the decoder, given the caption
$T_{1:N}$ of length $N$, all the words prior to the target position $p$ are
embedded as $e_{1:p-1}\in\mathbb{R}^{D\times(p-1)}$, which is used as the
decoder inputs to predict the $p$th word $\hat{T}_{p}$.

According to [4], when passing the image features into the transformer, in
order to supplement the permutation-invariant issue, it’s better to add
positional encodings to the input of each self-attention sub-layer in $C_{e}$
and encoder-decoder cross-attention sub-layer in $C_{d}$ instead of only
applied at the bottom of encoder. Similarly, the positional embeddings for
text input are also added to the input of each self-attention sub-layer in
$C_{d}$.

Taking $r$ as input, $C_{e}$ applies the same operation as Eq. (1-3) in each
transformer layer and refines the regional visual features via self-attention
mechanism. Then it’s top output $r_{e}=C_{e}(r)$ is passed to the encoder-
decoder cross-attention sub-layer within each transformer layer of $C_{d}$ to
further introduce the visual-language cross-attention mechanism for image-to-
text generation. The sub-layer takes $r_{e}$ as keys along with the self-
attention sub-layer output $\tilde{e}_{1:p-1}$ as queries:

$\displaystyle\tilde{e}^{\prime}_{1:p-1}$
$\displaystyle=\text{Multihead}(W^{T}_{Q}\tilde{e}_{1:p-1},W^{T}_{K}r_{e},W^{T}_{V}r_{e})$
(12)

The decoder output is sent to MLP to predict the probability of word at
position $p$:

$p(\hat{T}_{p}|r,T_{1:p-1})=f_{s}(\text{MLP}(C_{d}(C_{e}(r),e_{1:p-1})))$ (13)

Given the ground truth caption $T_{1:N}$, we train the ITM with cross-entropy
loss:

$L_{ITM}=L_{C}=-\sum_{p=1}^{N}log(p(T_{p}|r,T_{1:p-1}))$ (14)

###  3.4 Assisting losses

Given the generated images $\hat{I}$ from TIM in Subsection 3.2 and generated
texts $\hat{T}$ from ITM in Subsection 3.3, we introduce two assisting
matching losses in the model training: fake-image-real-text matching loss
$L_{m}^{I}$ and fake-text-real-image matching loss $L_{m}^{T}$.

Similar to $L_{m}$ in Subsection 3.1, we input $\hat{I}$ to $F_{I}$ and
$\hat{T}$ to $F_{T}$ to get $\hat{r},\hat{v}=F_{I}(\hat{I})$ and
$\hat{w},\hat{s}=F_{T}(\hat{T})$. From Eq. (4-8), we can calculate the
symmetric matching posterior probabilities between $(\hat{I},T)$ and
$(I,\hat{T})$, thus $L_{m}^{I}$ and $L_{m}^{T}$ are as:

$\displaystyle L_{m}^{I}$
$\displaystyle=L_{m_{s}}^{T\hat{I}}+L_{m_{s}}^{\hat{I}T}+L_{m_{w}}^{T\hat{I}}+L_{m_{w}}^{\hat{I}T}$
(15) $\displaystyle L_{m}^{T}$
$\displaystyle=L_{m_{s}}^{\hat{T}I}+L_{m_{s}}^{I\hat{T}}+L_{m_{w}}^{\hat{T}I}+L_{m_{w}}^{I\hat{T}}$

Along with $L_{G}$ and $L_{C}$, our final multimodal loss for joint training
is:

$L=\lambda_{m}L_{m}+\lambda_{\hat{I}}L_{m}^{I}+\lambda_{\hat{T}}L_{m}^{T}+\lambda_{G}L_{G}+\lambda_{C}L_{C}$
(16)

where
$\lambda_{m},\lambda_{\hat{I}},\lambda_{\hat{T}},\lambda_{G},\lambda_{C}$ are
hyper-parameters to add weights for each loss above.

##  4 Experimental Results

In this section, we explain our experiment settings, evaluation metrics and
results to evaluate our proposed model.

###  4.1 Datasets

Our model is evaluated on two public datasets including CUB [42] and MS-COCO
[25], which are widely used in text-to-image generation and image captioning
tasks.

CUB is a dataset contains 200 bird species and is popular for classification,
text-based image generation and image captioning tasks. It has 8855 training
images and 2933 test images, where each image has 10 text descriptions. Since
CUB only contains bird images with various attributes, the semantic domain is
relative simple and narrow for image text representation, which is suitable
for evaluating our model in a specific semantic task. We preprocess the
dataset according to the method in [47].

MS-COCO is a challenging dataset which is popular for various image-text
related tasks including image captioning and image-text matching. It has 82783
training images which paired with 5 captions per image and 40504 images for
testing. It provides large amounts of common object classes in the images,
which can show the representation performance of our model in generalized
semantic space.

###  4.2 Evaluation Metric

We employ R-precision a technique proposed in [47] to rank the retrieval
results. To evaluate the task of reducing the heterogeneity gap between
relevant word and image representation, we also propose an Attribute to Image
Matching Cosine Score (AIMCoS) which matches the similarity of specific
textual attributes which are supposed to be found in the paired image.
Specifically, first for CUB dataset, we extract the attributes corresponding
to each image file from the validation dataset. Each attribute contains text
entries that list the features like color, bill length, shape, etc. of a bird.
For each experiment, we first extract the global features $v$ of the image
$I_{i}$ using $F_{I}$. Next, we extract the features $s$ of textual input of
each attribute associated with image. Afterwards, we compute the average of
cosine similarity $f_{cos}$ of $v$ with the representations of each attribute
corresponding to $I_{i}$. Finally, we find the mean of match scores for all
the images in the validation set. Mathematically if there are $K$ attributes
present in image $I_{i}$ then

$AIMCoS=\frac{1}{U}\sum_{i=0}^{i=U}\frac{1}{K}\sum_{k=0}^{k=K}f_{cos}(v,s_{k})$
(17)

where $U$ is the number of images in the validation set of CUB dataset.

For computing the AIMCoS using $F_{I}$ and $F_{T}$ trained on COCO dataset,
each of the 80 classes are considered as attributes. We then create a smaller
evaluation set (LAViTeRcocoeval), which contains 100 images, extracted from
ImageNet [7] dataset queried on the keyword. Next, similar to Eq. 17 we
calculate the mean of $f_{cos}$ between the representations of each 100 images
and the representation of text the respective class name. Lastly, this score
is averaged over all the classes to calculate AIMCoS for COCO dataset.

Figure 3: A T-SNE [27] visualization of 3200 image representations and 32
textual-label representations from LAViTeRcocoeval.

In Figure 3, the larger point with label present within each cluster is the
representations of the textual labels. The other points of the cluster are the
representations of image data-point. The LAViTeR model is able to bridge the
representational gap between textual and visual data effectively while still
performing at par with other techniques.

###  4.3 Implementation Details

The LAViTeR model is implemented in Pytorch [29] and all the experiments are
carried out on NVIDIA GTX 1080ti GPUs. For $F_{I}$, we use the InceptionV3
model pretrained on ImageNet [7] as our backbone initialization. We use one
transformer layer with 8 heads in $F_{T}$ and 6 transformer layers with 8
heads in both $C_{e}$ and $C_{d}$. The length of $r$ is 289 (flattened from
$17\times 17$ feature map in $F_{I}$) and 15 for sentence length. The
dimension of the word embedding and representation features $r,v,w,s$ from VTA
is 256. Three cascaded generators are used in TIM and generate fake images
with resolutions $64\times 64$, $128\times 128$ and $256\times 256$
progressively.

Three training phases are designed for LAViTeR in order to make the joint
training more stable and easier to converge. In phase 1, we pretrain VTA on
the training data so that $F_{I}$ and $F_{T}$ can generate acceptable image
and text representations for the following phases. We freeze $F_{I}$ except
the output layers for $r,v$, and train the transformer layer in $F_{T}$ and
the output layers in $F_{I}$ with a learning rate 0.0002. As for phase 2, $C$
in ITM and $G$ in TIM are pretrained separately with the $r,v,w,s$ features
from the pretrained VTA as inputs. The learning rate is set to 0.0002 for $G$,
and for pretraining $C$ it is first set to 0.0001 and then decay by 0.1 after
20 epochs. When the pretrainings are completed, we freeze the first 5 layers
of $F_{I}$ and jointly train all the modules together as our final LAViTeR
model in phase 3, with a small learning rate of $10^{-6}$. All the training
are optimized with the Adam optimizer [20] with a weight decay $0.0001$. The
batch size is set to 96 for phase 1, 32 and 14 for $C$ and $G$ pretraining in
phase 2, and 8 for the joint training due to the limitation of our GPU memory.
In all experiments, $\gamma_{1}=4,\gamma_{2}=5,\gamma_{3}=10$ are used for the
matching losses.

###  4.4 Performance

Hyper-parameter Selection We first adjust the $\lambda$ in our multi-model
loss (see Eq. 16) in order to test the effect of different loss terms in our
joint training and select the best $\lambda$ setting for our multi-modal loss.
The results on CUB and COCO are shown in Table 1.

$\lambda_{G}$ | $\lambda_{C}$ | $\lambda_{m_{T}}$ | $\lambda_{m_{I}}$ | $\lambda_{m}$ | R-precision | AIMCoS
---|---|---|---|---|---|---
0 | 0 | 0 | 0 | 1 | 89.76 | 0.5045
1 | 1 | 1 | 1 | 50 | 90.35 | 0.4931
0.01 | 0.1 | 1 | 1 | 50 | 90.71 | 0.5063
0.01 | 0.1 | 50 | 1 | 1 | 90.31 | 0.5063
0.01 | 0.1 | 1 | 50 | 1 | 90.49 | 0.496
0.01 | 0.1 | 10 | 1 | 10 | 90.05 | 0.498
0.01 | 0.1 | 10 | 1 | 1 | 90.19 | 0.501
0.001 | 0.1 | 1 | 0.1 | 10 | 90.16 | 0.501
0.01 | 0.1 | 1 | 1 | 10 | 90.79 | 0.504
1 | 1 | 1 | 1 | 1 | 63.4 | 0.203
0.01 | 0.1 | 1 | 5 | 1 | 62.2 | 0.198
0.01 | 0.1 | 5 | 1 | 1 | 64.6 | 0.197
0.01 | 0.1 | 1 | 1 | 1 | 63.0 | 0.2
0.01 | 0.1 | 1 | 1 | 10 | 66.4 | 0.202
0.01 | 0.1 | 1 | 1 | 5 | 66.8 | 0.201
Table 1: The best top-3 R-precision score of each LAViTeR model setting on
COCO (top nine rows) and CUB (bottom 6 rows) test set. AIMCoS score is also
reported.

As we can see, different $\lambda$ settings have influenced the performance of
the model. The best setting on CUB is from the bottom row in Table 1, which
gives the best top-3 R-precision as 66.8%, where we choose a relative large
$\lambda_{m}=5$ for loss $L_{m}$ while keeping small weights as
$\lambda_{G}=0.01$ and $\lambda_{C}=0.1$ for $L_{G}$ and $L_{C}$. Meanwhile,
row 9 in Table 1 with a similar setting
($\lambda_{m}=10,\lambda_{G}=0.01,\lambda_{C}=0.1$) also gives the best top-3
R-precision on COCO as 90.79%, and row 3 in Table 1 which adjusts
$\lambda_{m}$ to 50 gives the best AIMCoS score as 0.5063. These results show
that $L_{m}$ requires higher weight among all the loss terms, since it’s a
credible objective for real-image-real-text matching. On the other hand,
$L_{m}^{I}$ and $L_{m}^{T}$ are calculated from the fake image or fake text
matching, which may induce some noises in the gradients due to some low
quality generated images and captions, thus smaller $\lambda_{\hat{I}}$ and
$\lambda_{\hat{T}}$ gives better results. Moreover, since $L_{G}$ and $L_{C}$
is only used for necessary updates in $G$ and $C$ during joint training and
has limited contribution to VTA, thus we just keep tiny weights for them.

Ablation Studies and Analysis We also conduct necessary ablation studies in
Table 2 to further inspect the effect of each modules of LAViTeR. First, we
set all the $\lambda$s to 0 except $\lambda_{m}$, which results in the worst
R-precision among all the settings on COCO dataset (see the top row in Table
1). This case shows the importance of our assisting losses during the joint
training. Since $L_{m}$ trains the image and text encoders but has no effect
on $C$ and $G$, the updated feature outputs from $F_{I}$ and $F_{T}$ are no
longer working well for the captioning model and image generator, hence the
quality of generated images and captions degraded which also affects the
matching performance.

In Table 2, we test the performance of LAViTeR when different modules are
removed. Since symmetric global-local matching loss $L_{m}$ is also used to
train the image-text matching model (called DAMSM in their work) in [47, 31],
so we also evaluate their performance using our metrics, which refers to
AttnGAN/MirrorGAN in Table 2 and 3. DAMSM uses RNN-based text encoder in their
paper instead of the transformer-based model like ours, so it also plays a
role as one of our baseline. In our ablation settings, LAViTeR is our final
model. $L_{M}(F)$ and $L_{M}(T)$ mean that only the VTA module is trained in
phase 3 as our baseline without TIM and ITM branches, and the image encoder
backbone is frozen ($F$, same training method as DAMSM) and trainable ($T$)
during the joint training, respectively. LAViTeR-Img2Txt only keeps the ITM
module in the joint training to generate image captions for $L_{m}^{T}$ and
the image generator $G$ with $L_{m}^{I}$ is removed; As a complementary
setting, LAViTeR-Txt2Img keeps the image generator and $L_{m}^{I}$ and deducts
the captioning module $C$ along with $L_{m}^{T}$.

As it shows in Table 2, $L_{M}(T)$ improves the performance for 0.66% in
R-precision and 0.005 in AIMCoS, which shows that the image encoder requires
update during training for a better performance. In LAViTeR-Img2Txt where $G$
is trimmed, the R-precision drops by 0.57% from the full model and even lower
than the baseline $F_{M}(T)$. This might be related to the large amount of
unseen fake images no longer being generated and used for VTA training, thus
reduces the diversity of the training set and the matching accuracy.
Meanwhile, if $C$ is removed as in LAViTeR-Txt2Img setting, both the
R-precision and AIMCoS are reduced by 0.28% and 0.0023 respectively, which
could be caused by lack of fake captions. When no captions are generated for
matching, the training sentences and words become less various and thus affect
the image-text matching performance shown in R-precision as well as the word
level matching accuracy suggested by AIMCoS. On both CUB and COCO datasets,
our full model LAViTeR works much better than DAMSM in [47, 31], which
improves R-precision by 0.74% in COCO and 4.2% in CUB, and boosts AIMCoS from
0.44 to 0.5063 in COCO and 0.072 to 0.201 in CUB, almost 3 times better. This
result shows that image-to-text and text-to-image generation branches can
assist the image-text matching/representation model training by aligning
generated image with real text and generated text with real image.

We also find an interesting phenomenon in our baselines. When replacing the
original RNN-based text encoder in DAMSM with our transformer-based text
encoder, $L_{M}(F)$ has a slightly lower R-precision but much higher AIMCoS
than DAMSM baseline in both Tables. This actually shows one of the main
difference between transformer layer and RNN layer: due to the recurrence
feature and short memory of RNN, one layer transformer may have no obvious
advantage over RNN for captions with normal length around 15, which is implied
in R-precision for image-caption matching; however, RNN cannot get good
representations of short phrases with only 1-3 words, since it requires enough
context to give reasonable outputs. In contrast, transformer’s self-attention
mechanism is able to effectively handle any text length and align even a
single word with the corresponding image regions, which is observed in the
substantial increase in AIMCoS.

Model Name | R-precision | AIMCoS
---|---|---
AttnGAN/MirrorGAN[47, 31] | 89.97 | 0.44
LAViTeR-$L_{M}$ (F) | 89.81 | 0.501
LAViTeR-$L_{M}$ (T) | 90.37 | 0.5059
LAViTeR-Img2Txt | 90.14 | 0.506
LAViTeR-Txt2Img | 90.43 | 0.504
LAViTeR | 90.71 | 0.5063
Table 2: Top-3 R-precision and Attribute Image Match Cosine Score (AIMCoS) metrics calculated on COCO dataset. Model Name | R-precision | AIMCoS
---|---|---
AttnGAN/MirrorGAN [47, 31] | 62.6 | 0.072
LAViTeR-$L_{M}$ (F) | 59.2 | 0.184
LAViTeR | 66.8 | 0.201
Table 3: Top-3 R-precision and Attribute Image Match Cosine Score (AIMCoS)
metrics calculated on CUB dataset.

Visualization Analysis In addition, we illustrate some qualitative results
from image-to-text and text-to-image matching on COCO in Figure 4 and Figure 5
respectively. For image queries, the captions with top-5 similarity scores
from our model are retrieved. Most captions are correct matches. It is
observed that sentences that have “incorrect” match labels actually share
similar semantics with the image queries. In Figure 5, text queries with the
top-3 image matches from our model are listed. The correct matching is
retrieved with other similar images, and we find that all the high ranked
images are quite reasonable. These “incorrect” matching pairs with close
semantics data expose a drawback in the current evaluation metrics for multi-
modal matching: many images along with their paired captions in the test set
share similar semantics and are close to each other in the common semantic
space, however, these neighbor samples are treated as “mismatch” in the image-
to-text and text-to-image retrieval evaluation metrics and show no difference
with those distinctly mismatched samples with large semantic gap. This
evaluation defect motivates us to propose the new evaluation metric AIMCoS.

Figure 4: The top-5 image-to-text matching captions with descending similarity
scores. Blue captions are the correct matches, while red ones are incorrect
matches. Figure 5: The top-3 text-to-image matching images with descending
similarity scores from left to right. Green marks are the correct matches,
while red crosses are incorrect matches.

##  5 Conclusion

In this paper, we introduce a novel architecture for visual and textual
representation learning assisted by two auxiliary tasks, image-to-text
generation and text-to-image generation. The generated images and texts are
matched with real text and images to jointly train the representation model
with two assisting matching losses. A new evaluation metric AIMCoS is proposed
for measuring the similarity between the learnt visual and textual embedding.
The experimental results on two public datasets demonstrate the effectiveness
of the proposed architecture and evaluation metric.

## References

  * [1] C. Alberti, J. Ling, M. Collins, and D. Reitter.  Fusion of detected objects in text for visual question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2131–2140, 2019.
  * [2] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086, 2018.
  * [3] Biing-Hwang Juang, Wu Hou, and Chin-Hui Lee.  Minimum classification error rate methods for speech recognition.  IEEE Transactions on Speech and Audio Processing, 5(3):257–265, 1997.
  * [4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  arXiv preprint arXiv:2005.12872, 2020.
  * [5] G. Chauhan, R. Liao, W. Wells, J. Andreas, X. Wang, S. Berkowitz, S. Horng, P. Szolovits, and P. Golland.  Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 529–539. Springer, 2020.
  * [6] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu.  Uniter: Universal image-text representation learning.  In European Conference on Computer Vision, pages 104–120. Springer, 2020.
  * [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [8] M. Denkowski and A. Lavie.  Meteor universal: Language specific translation evaluation for any target language.  In Proceedings of the ninth workshop on statistical machine translation, pages 376–380, 2014.
  * [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [10] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt, C. L. Zitnick, and G. Zweig.  From captions to visual concepts and back, 2015.
  * [11] Z. Gan, Y.-C. Chen, L. Li, C. Zhu, Y. Cheng, and J. Liu.  Large-scale adversarial training for vision-and-language representation learning.  arXiv preprint arXiv:2006.06195, 2020.
  * [12] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 2672–2680, Cambridge, MA, USA, 2014. MIT Press.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [14] S. He, W. Liao, H. R. Tavakoli, M. Yang, B. Rosenhahn, and N. Pugeault.  Image captioning through image transformer.  arXiv preprint arXiv:2004.14231, 2020.
  * [15] S. Herdade, A. Kappeler, K. Boakye, and J. Soares.  Image captioning: Transforming objects into words.  In Advances in Neural Information Processing Systems, pages 11137–11147, 2019.
  * [16] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in neural information processing systems, pages 6626–6637, 2017.
  * [17] L. Huang, W. Wang, J. Chen, and X.-Y. Wei.  Attention on attention for image captioning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4634–4643, 2019.
  * [18] P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck.  Learning deep structured semantic models for web search using clickthrough data.  In Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, CIKM ’13, page 2333–2338, New York, NY, USA, 2013. Association for Computing Machinery.
  * [19] Y. Huang, W. Wang, and L. Wang.  Instance-aware image and sentence matching with selective multimodal lstm.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2310–2318, 2017.
  * [20] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang.  Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11336–11344. AAAI Press, 2020.
  * [22] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang.  Visualbert: A simple and performant baseline for vision and language, 2019\.
  * [23] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [24] Y. Li, H. Wang, and Y. Luo.  A comparison of pre-trained vision-and-language models for multimodal representation learning across medical images and reports.  arXiv preprint arXiv:2009.01523, 2020.
  * [25] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [26] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In Advances in Neural Information Processing Systems, pages 13–23, 2019.
  * [27] L. v. d. Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * [28] M. Mirza and S. Osindero.  Conditional generative adversarial nets, 2014.
  * [29] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [30] G. Peyré, M. Cuturi, et al.  Computational optimal transport: With applications to data science.  Foundations and Trends® in Machine Learning, 11(5-6):355–607, 2019.
  * [31] T. Qiao, J. Zhang, D. Xu, and D. Tao.  Mirrorgan: Learning text-to-image generation by redescription.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1505–1514, 2019.
  * [32] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee.  Generative adversarial text to image synthesis.  In 33rd International Conference on Machine Learning, pages 1060–1069, 2016.
  * [33] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [34] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans. arxiv 2016.  arXiv preprint arXiv:1606.03498.
  * [35] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, and J. Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  In International Conference on Learning Representations, 2019.
  * [36] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826, 2016.
  * [37] H. Tan and M. Bansal.  LXMERT: Learning cross-modality encoder representations from transformers.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100–5111, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
  * [38] S. Uppal.  CA$\vdots$TR: Image captioning with transformers.  https://github.com/saahiluppal/catr.
  * [39] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [40] R. Vedantam, C. Lawrence Zitnick, and D. Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [41] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan.  Show and tell: A neural image caption generator.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015.
  * [42] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [43] L. Wang, Y. Li, J. Huang, and S. Lazebnik.  Learning two-branch neural networks for image-text matching tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):394–407, 2018.
  * [44] X. Wei, T. Zhang, Y. Li, Y. Zhang, and F. Wu.  Multi-modality cross attention network for image and sentence matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10941–10950, 2020.
  * [45] Xiaodong He, Li Deng, and Wu Chou.  Discriminative learning in sequential pattern recognition.  IEEE Signal Processing Magazine, 25(5):14–36, 2008.
  * [46] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057, 2015.
  * [47] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [48] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In Advances in neural information processing systems, pages 5753–5763, 2019.
  * [49] T. Yao, Y. Pan, Y. Li, and T. Mei.  Exploring visual relationship for image captioning.  In Proceedings of the European conference on computer vision (ECCV), pages 684–699, 2018.
  * [50] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas.  Stackgan++: Realistic image synthesis with stacked generative adversarial networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8):1947–1962, 2019.
  * [51] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao.  Unified vision-language pre-training for image captioning and VQA.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 13041–13049. AAAI Press, 2020.
  * [52] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.

Below we mention some analysis and downstream tasks performed using our pre-
trained encoders.

##  Appendix A VTA Qualitative Results

Apart from the T-SNE visualization in main paper, we also plot the Image
representation vs Token representation similarity map in Figure 6.

Figure 6: Visualization of Image representation vs Token representation
similarity map

The x-axis denotes the image representations and the y-axis denotes the
category word token representation. Since there are 80 classes in COCO dataset
[25], the matrix size is $80\times 80$. There are 80 Tokens and 8000 images,
i.e. 100 images per token. We first compute similarity of each token
representation with all the 8000 images in LaViTeRcocoeval dataset. Then, the
average similarity value of match with each token plotted in Figure 6. The
dark color diagonal shows a high similarity between true matches, indicating
the efficacy of LAViTeR model.

From Figure 6, we observe that, the token “apple” matches highest with the
images of apple but also contains high similarity with images of another
fruits like “orange” because they are usually kept together. Similarly, the
token “microwave” is highly matched with images of “microwave” and also with
“oven” and in reality most of the times they appear together. More
interestingly, the token “person” is not matched with other images showing no
correlation, but is matched highly with images of “person” and also
“skateboard” as in many “skateboard” images there is a person in the frame.
Also, the token “spoon” is highest matched with “spoon” images and also
slightly matched with images of “fork” and “knife” as these items usually
appear together naturally. Thus, the joint training is able to reduce the
diversity gap between the nature of textual and visual representations
meaningfully.

##  Appendix B TIM Qualitative Results

Although our primary goal is not to improve GAN, we compare the quality of
generated images from LAViTeR model with the ground truth and the generated
images by [31, 47]. Those examples are selected from the visualization of the
previous state-of-the-art papers directly.

Figure 7: Examples of images generated by (a) AttnGAN [47], (b) MirrorGAN [31]
(c) LaViTeR network (d) the corresponding ground truth. Left four columns are
images from CUB [42] dataset.

As can be seen from Figure 7 and 8, the LAViTeR model is able to perform at-
par and in some cases better than the previous state-of-the-art models. We do
not report these images in the main paper as the major goal of LaViTeR is for
representation joint learning and alignment. The model is able to perform
better due to continuous training of VTA, TIM and ITM models cooperatively.
The generated images during the training process are able to provide a diverse
set of images, which further assists the joint learning and also the GAN
module.

Figure 8: Examples of images generated by (a) AttnGAN [47], (b) MirrorGAN [31] (c) LaViTeR network (d) the corresponding ground truth. Left four columns are images from COCO [25] dataset. Model |  IS $\uparrow$ |  FID $\downarrow$
---|---|---
StackGAN-v1 | 8.45 $\pm$ 0.03 | 74.05
StackGAN-v2 | 8.30 $\pm$ 0.10 | 81.59
AttnGAN | 25.89 $\pm$ 0.47 | -
MirrorGAN | 26.47 $\pm$ 0.41 | -
LAViTeR | 26.71 $\pm$ 0.39 | 75.5
Table 4: IS and FID scores, calculated on COCO 2014 validation set. Uparrow
means higher value is better, down arrow means lower value better.

We also compute the Inception Score (IS) [34] and Fréchet Inception Distance
(FID) [16] to compare the GAN module trained while joint learning. Table 4
shows that GAN in LaViTeR model performs better than all the previous models
in terms of IS Score while the FID score is still comparable to other text to
image network as [50] that report FID scores. However, better GAN performance
is a by-product of LAViTeR model, not it’s goal.

##  Appendix C ITM Qualitative Results

Figure 9: Examples of generated captions (FT (Fake Text) in blue under sample images) by LAViTeR ITM and the corresponding ground truth captions (GT in black). Figure 10: Examples of generated captions (FT (Fake Text) in blue under sample images) by LaViTeR ITM and the corresponding ground truth captions (in black). The objects or attributes shown in the sample image but not in its ground truth descriptions are marked in red. Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | CIDEr
---|---|---|---|---|---|---
Show-tell [41] | - | - | - | 27.7 | 23.7 | 85.5
Show-attend-tell [46] | 70.7 | 49.2 | 34.4 | 24.3 | 23.9 | -
LAViTeR | 69.15 | 51.75 | 37.68 | 27.44 | 24.20 | 86.72
Table 5: BLEU 1-4, METEOR[8] and CIDAr[40] scores of our ITM module calculated
on MSCOCO 2014 validation set. Results reported in [41] and [46] are also
listed here for reference.

Some examples of generated captions by our LAViTeR ITM branch are shown in
Figure 9 and Figure 10. As Figure 9 shows, the generated captions are able to
capture the main attributes (crown color, bill size, breast color, etc.) of
the birds in sample images from CUB test set. Figure 10 shows objects and
their corresponding actions and attributes in sample images from COCO test
set. Specifically, some of the generated captions can generate words (marked
in red) that are not included in the ground truth captions, but actually shown
up in the images. Some objects in the background are predicted in the
generated captions, e.g. for the first image in the third row, ‘trees’ is in
the background of the image which is never described in the paired ground
truth captions, but successfully predicted by ITM; for the last image, TV and
coffee table are shown in the background corner of the image and are ignored
by the ground truth texts, but are captured by the generated caption. Even
only a part of the object appears in an image, it still has a chance to show
up in our generated caption, e.g. in the image in the middle of the last row,
a tiny part of a fork tip appears on the left, our ITM can generate ‘a fork’
given such a small clue in the image. Missing attributes are also predicted
such as the attribute ‘smiling’ is generated in our generated text, which is
never described in the ground truths. These generated texts from ITM with the
missing objects and attributes in the original images provide further useful
information outside the training dataset and help the matching model learn
more word-region level matching pairs. This is exactly one of our motivations
for our LAViTeR method.

We also evaluate the captioning ability of our ITM branch from the joint
training, listed in Table 5. As we can see, our ITM can get a comparable
result with other captioning models, such as [46] and [41], which shows that
our ITM has the ability to generate reasonable captions for real-image-fake-
text matching training. Note that we are not aiming at coming up with a novel
and state-of-the-art captioning model, the main function of ITM branch is to
provide more reasonable fake text for training the VTA matching model.

##  Appendix D Code

Our code referenced from libraries open sourced by the work done in CATR [38]
and AttnGAN [47].