# Tetragonal modular quotients $X_{0}^{+}(N)$

Petar Orlić Petar Orlić
University of Zagreb
Bijenička Cesta 30
10000 Zagreb
Croatia<EMAIL_ADDRESS>
###### Abstract.

In this paper we determine all quotient curves $X_{0}^{+}(N)$ whose
$\mathbb{Q}$ or $\mathbb{C}$-gonality is equal to $4$. As a consequence, we
find several new cases when the modular curve $X_{0}(N)$ has
$\mathbb{Q}$-gonality equal to $8$.

###### Key words and phrases:

Modular curves, Gonality

###### 1991 Mathematics Subject Classification:

11G18, 14H35, 14H51

The author weas supported by the QuantiXLie Centre of Excellence, a project
co-financed by the Croatian Government and European Union through the European
Regional Development Fund - the Competitiveness and Cohesion Operational
Programme (Grant KK.01.1.1.01.0004) and by the Croatian Science Foundation
under the project no. IP-2018-01-1313.

##  1\. Introduction

Let $C$ be a smooth projective curve over a field $k$. The $k$-gonality of
$C$, denoted by $\textup{gon}_{k}C$, is the least degree of a non-constant
$k$-rational morphism $f:C\to\mathbb{P}^{1}$.

Morphisms from modular curves have been extensively studied. Zograf [25] gave
a linear lower bound for the $\mathbb{C}$-gonality for any modular curve.
Later, Abramovich [1] and Jeon, Kim and Park [16] improved the constant in
that bound.

Regarding the curve $X_{0}(N)$, Ogg [21] determined all hyperelliptic, Bars
[3] determined all bielliptic, Hasegawa and Shimura [11] determined all
trigonal curves $X_{0}(N)$ over $\mathbb{C}$ and $\mathbb{Q}$, and Jeon and
Park [17] determined all tetragonal curves $X_{0}(N)$ over $\mathbb{C}$.
Recently, Najman and Orlić [19] determined all curves $X_{0}(N)$ with
$\mathbb{Q}$-gonality equal to $4,5,$ or $6$, and for many of those curves
also determined their $\mathbb{C}$-gonality.

Much progress has been made in studying the $\mathbb{Q}$-gonality of the
modular quotients of $X_{0}(N)$ as well. Furumoto and Hasegawa [8] determined
all hyperelliptic quotients of $X_{0}(N)$, Hasegawa and Shimura [12, 13, 14]
determined all trigonal quotients of $X_{0}(N)$ over $\mathbb{C}$ and
partially determined the $\mathbb{Q}$-trigonal quotients (more precisely, they
solved all cases when the genus is not equal to $4$). Furthermore, Bars,
Gonzalez and Kamel [4] determined all bielliptic quotients of $X_{0}(N)$ for
squarefree levels, Jeon [15] determined all bielliptic curves $X_{0}^{+}(N)$,
and Bars, Kamel and Schweizer [5] determined all bielliptic quotients of
$X_{0}(N)$ for non-squarefree levels.

The next logical step is determining all tetragonal quotients of $X_{0}(N)$.
In this paper we will study the $\mathbb{Q}$ and $\mathbb{C}$-gonality of the
quotient curve $X_{0}^{+}(N)=X_{0}(N)/w_{N}$ ($w_{N}$ being the Atkin-Lehner
involution).

One of the reasons we study the gonality of $X_{0}^{+}(N)$, apart from being
an interesting question in itself, is that it can help us to determine the
gonality of $X_{0}(N)$. The reason for that is, of course, that we have a
natural rational quotient map $X_{0}(N)\to X_{0}^{+}(N)$ of degree $2$.
Therefore, any map $X_{0}^{+}(N)\to\mathbb{P}^{1}$ of degree $d$ induces a map
$X_{0}(N)\to\mathbb{P}^{1}$ of degree $2d$ defined over the same base field.

Also, the existence of rational degree $d$ maps to $\mathbb{P}^{1}$ is closely
linked to the problem of determining whether that curve has infinitely many
points of degree $d$, as can for example be seen in [2, 10], and, more
recently, [18, Theorem 1.4].

Our main results are the following theorems (though the first theorem was
already mostly proved by Hasegawa and Shimura).

######  Theorem 1.1.

The curve $X_{0}^{+}(N)$ has $\mathbb{Q}$-gonality equal to $3$ if and only if

$\displaystyle N\in\\{$ $\displaystyle
58,76,84,85,86,88,93,96,97,99,100,109,113,115,116,122,127,128,129,135,$
$\displaystyle
137,139,146,147,149,151,155,159,162,164,169,179,181,215,227,239\\}.$

######  Theorem 1.2.

The curve $X_{0}^{+}(N)$ has $\mathbb{Q}$-gonality equal to $4$ if and only if

$\displaystyle N\in\\{$ $\displaystyle
70,78,82,90,102,105,106,108,110,112,114,117,118,120,123,124,126,133,134,$
$\displaystyle
136,138,140,141,142,144,145,148,152,156,157,158,160,161,163,165,166,171,173,$
$\displaystyle
175,176,177,183,184,185,188,192,193,194,195,197,199,200,203,205,206,207,209,$
$\displaystyle
211,213,221,223,224,229,241,251,257,263,269,279,281,284,287,299,311,359\\}.$

######  Theorem 1.3.

The curve $X_{0}^{+}(N)$ has $\mathbb{Q}$-gonality greater than $4$ and
$\mathbb{C}$-gonality equal to $4$ if and only if $N\in\\{243,271\\}$.

As a consequence, we were able to determine the $\mathbb{Q}$-gonality of
$X_{0}(N)$ for several new levels $N$, not previously solved in [19].

It is perhaps surprising that we were able to determine all tetragonal curves
$X_{0}^{+}(N)$ using mostly previously known methods. One of the reasons for
that is that in many cases we can get the degree $4$ map from a degree $2$
quotient map to $X_{0}(N)/\left<w_{d},w_{N}\right>$, as presented in 3.9. If
one would, for example, search for all pentagonal curves $X_{0}^{+}(N)$, there
is no such natural source of degree $5$ maps.

A lot of the results in this paper rely on Magma computations. The codes that
verify all computations in this paper can be found on

https://github.com/orlic1/gonality_X0_quotients.

All computations were performed on the Euler server at the Department of
Mathematics, University of Zagreb with a Intel Xeon W-2133 CPU running at
3.60GHz and with 64 GB of RAM.

##  2\. Acknowledgements

Many thanks to Nikola Adžaga and Philippe Michaud-Jacobs for permission to use
their Magma codes as templates. I am also grateful to Peter Bruin, Maarten
Derickx, and Filip Najman for their helpful comments and suggestions.

##  3\. Results

In this section we list the known results that will be used to determine the
$\mathbb{Q}$-gonality of $X_{0}^{+}(N)$. We first mention two obvious lower
bounds for a curve $C$ defined over $\mathbb{Q}$:

$\displaystyle\textup{gon}_{\mathbb{C}}(C)$
$\displaystyle\leq\textup{gon}_{\mathbb{Q}}(C),$
$\displaystyle\textup{gon}_{\mathbb{F}_{p}}(C)$
$\displaystyle\leq\textup{gon}_{\mathbb{Q}}(C).$

Here $p$ is a prime of good reduction for $C$.

###  3.1. $\mathbb{F}_{p}$-gonality

An important tool for determining $\mathbb{Q}$-gonalities of modular curves
$X_{0}(N)$ is Ogg’s inequality [21, Theorem 3.1], stated in simpler form in
[11, Lemma 3.1].

######  Lemma 3.1.

Let $p$ be a prime not dividing $N$. Then the number of $\mathbb{F}_{p^{2}}$
points on $X_{0}(N)$ is at least

$L_{p}(N):=\frac{p-1}{12}\psi(N)+2^{\omega(N)}.$

Here $\psi(N)$ is the index of the congruence subgroup $\Gamma_{0}(N)$, equal
to $N\prod_{q\mid N}(1+\frac{1}{q})$, and $\omega(N)$ is the number of
different prime divisors of $N$.

######  Lemma 3.2.

[19, Lemma 3.5] Let $C$ be a curve, $p$ a prime of good reduction for $C$, and
$q$ a power of $p$. Suppose $\\#C(\mathbb{F}_{q})>d(q+1)$. Then
$\textup{gon}_{\mathbb{Q}}(C)>d$.

From these two results we can get a lower bound on $\mathbb{Q}$-gonality of
$X_{0}^{+}(N)$. Namely, if the quotient curve $X_{0}^{+}(N)$ is tetragonal,
then we have a rational composition map $X_{0}(N)\to
X_{0}^{+}(N)\to\mathbb{P}^{1}$ of degree $8$. Therefore, we must have

(1) $\displaystyle L_{p}(N)$ $\displaystyle\leq 8(p^{2}+1)$

for all primes $p\nmid N$. However, similarily as in [11, Lemma 3.2], we get
that for $N\geq 456$ there exists a prime $p$ for which this inequality does
not hold. This means that we have eliminated all but finitely many levels $N$.
This inequality can also be used to eliminate

$\displaystyle N\in\\{$ $\displaystyle
255,260,266,276,280,282,286,290,292,296,304,306,308,310,312,314,315,316,318,$
$\displaystyle
320,322,324,326,327,328,330,332,333,334,336,338,339,340,342,344,345,346,348,$
$\displaystyle
350,351,352,354,356,357,358,360,362-366,368,369,370,372,374,375,376,378,$
$\displaystyle
380,381,382,384-388,390,392-396,398,399,400,402-408,410-418,422-430$
$\displaystyle 432,434-438,440,441,442,444-448,450-455\\}.$

We can also use 3.2 directly to deal with some cases.

######  Proposition 3.3.

The curve $X_{0}^{+}(N)$ is not tetragonal over $\mathbb{Q}$ for the following
values of $N$:

$N$ | $p$ | $\\#X_{0}^{+}(N)(\mathbb{F}_{p^{2}})$ | $N$ | $p$ | $\\#X_{0}^{+}(N)(\mathbb{F}_{p^{2}})$ | $N$ | $p$ | $\\#X_{0}^{+}(N)(\mathbb{F}_{p^{2}})$
---|---|---|---|---|---|---|---|---
$268$ | $3$ | $46$ | $272$ | $3$ | $42$ | $273$ | $2$ | $26$
$274$ | $3$ | $48$ | $288$ | $5$ | $116$ | $291$ | $2$ | $21$
$297$ | $2$ | $27$ | $298$ | $3$ | $45$ | $301$ | $2$ | $21$
$305$ | $2$ | $24$ | $309$ | $2$ | $23$ | $323$ | $2$ | $23$
$325$ | $2$ | $23$ | $341$ | $2$ | $25$ | $343$ | $3$ | $43$
$347$ | $2$ | $21$ | $349$ | $2$ | $22$ | $353$ | $2$ | $22$
$355$ | $2$ | $22$ | $361$ | $2$ | $22$ | $367$ | $2$ | $46$
$371$ | $2$ | $25$ | $373$ | $2$ | $21$ | $377$ | $2$ | $24$
$379$ | $2$ | $22$ | $389$ | $2$ | $24$ | $391$ | $2$ | $24$
$397$ | $3$ | $41$ | $401$ | $2$ | $24$ | $409$ | $2$ | $25$
$419$ | $2$ | $23$ | $421$ | $2$ | $25$ | $433$ | $2$ | $23$
$439$ | $2$ | $22$ | $443$ | $2$ | $25$ | $449$ | $2$ | $26$

###### Proof.

Using Magma, we calculate the number of $\mathbb{F}_{p^{2}}$ points on
$X_{0}^{+}(N)$. It is now easy to check that
$\\#X_{0}^{+}(N)(\mathbb{F}_{p^{2}})>4(p^{2}+1)$ in all these cases. ∎

We continue with computing the $\mathbb{F}_{p}$-gonality of $X_{0}^{+}(N)$.
This is a finite task and can be done by checking the Riemann-Roch spaces of
degree $d$ effective $\mathbb{F}_{p}$-rational divisors. We can also use [19,
Lemma 3.1] to reduce the number of divisors that need to be checked and
quicken the computation.

######  Proposition 3.4.

The $\mathbb{F}_{p}$-gonality of $X_{0}^{+}(N)$ is bounded from below for the
following values of $N$:

$N$ | $p$ | $\textup{gon}_{\mathbb{F}_{p}}\geq$ | $N$ | $p$ | $\textup{gon}_{\mathbb{F}_{p}}\geq$ | $N$ | $p$ | $\textup{gon}_{\mathbb{F}_{p}}\geq$ | $N$ | $p$ | $\textup{gon}_{\mathbb{F}_{p}}\geq$
---|---|---|---|---|---|---|---|---|---|---|---
$70$ | $11$ | $4$ | $82$ | $3$ | $4$ | $90$ | $11$ | $4$ | $108$ | $5$ | $4$
$117$ | $7$ | $4$ | $130$ | $3$ | $5$ | $132$ | $5$ | $5$ | $150$ | $7$ | $5$
$154$ | $3$ | $5$ | $161$ | $2$ | $4$ | $168$ | $5$ | $5$ | $170$ | $3$ | $5$
$172$ | $3$ | $5$ | $173$ | $5$ | $4$ | $174$ | $5$ | $5$ | $178$ | $3$ | $5$
$180$ | $7$ | $5$ | $182$ | $3$ | $5$ | $187$ | $3$ | $5$ | $189$ | $2$ | $5$
$196$ | $3$ | $5$ | $198$ | $5$ | $5$ | $199$ | $5$ | $4$ | $201$ | $2$ | $5$
$202$ | $3$ | $5$ | $204$ | $5$ | $5$ | $208$ | $3$ | $5$ | $212$ | $3$ | $5$
$216$ | $5$ | $5$ | $217$ | $2$ | $5$ | $218$ | $3$ | $5$ | $219$ | $2$ | $5$
$225$ | $2$ | $5$ | $226$ | $3$ | $5$ | $228$ | $5$ | $5$ | $230$ | $3$ | $5$
$231$ | $2$ | $5$ | $232$ | $3$ | $5$ | $233$ | $2$ | $5$ | $234$ | $5$ | $5$
$235$ | $2$ | $5$ | $237$ | $2$ | $5$ | $240$ | $7$ | $5$ | $242$ | $3$ | $5$
$243$ | $7$ | $5$ | $244$ | $3$ | $5$ | $245$ | $2$ | $5$ | $247$ | $3$ | $5$
$250$ | $3$ | $5$ | $251$ | $2$ | $4$ | $253$ | $2$ | $5$ | $256$ | $3$ | $5$
$259$ | $2$ | $5$ | $261$ | $2$ | $5$ | $265$ | $2$ | $5$ | $271$ | $3$ | $5$
$275$ | $2$ | $5$ | $277$ | $2$ | $5$ | $283$ | $2$ | $5$ | $289$ | $2$ | $5$
$293$ | $2$ | $5$ | $307$ | $2$ | $5$ | $311$ | $2$ | $4$ | $313$ | $2$ | $5$
$317$ | $2$ | $5$ | $319$ | $2$ | $5$ | $331$ | $2$ | $5$ | $335$ | $2$ | $5$
$337$ | $2$ | $5$ | $383$ | $2$ | $5$ |  |  |  |  |  |

###### Proof.

Using Magma, we compute that there are no functions of degree $<d$ in
$\mathbb{F}_{p}(X_{0}^{+}(N))$. We are able to reduce the number of divisors
that need to be checked by noting the following: If there exists a function
$f$ over a field $k$ of a certain degree and if $c\in k$, then the function
$g(x):=\frac{1}{f(x)-c}$ has the same degree and its polar divisor contains a
$k$-rational point. ∎

In the following propositions we will use Poonen’s [22, Proposition A.1.],
stated below.

######  Proposition 3.5 (Poonen).

Let $X$ be a curve of genus $g$ over a field $k$.

  1. (i)

If $L$ is a field extension of $k$, then
$\textup{gon}_{L}(X)\leq\textup{gon}_{k}(X)$.

  2. (ii)

If $k$ is algebraically closed and $L$ is a field extension of $k$, then
$\textup{gon}_{L}(X)=\textup{gon}_{k}(X)$.

  3. (iii)

If $g\geq 2$, then $\textup{gon}_{k}(X)\leq 2g-2$.

  4. (iv)

If $g\geq 2$ and $X(k)\neq\emptyset$, then $\textup{gon}_{k}(X)\leq g$.

  5. (v)

If $k$ is algebraically closed, then $\textup{gon}_{k}(X)\leq\frac{g+3}{2}$.

  6. (vi)

If $\pi:X\to Y$ is a dominant $k$-rational map, then
$\textup{gon}_{k}(X)\leq\deg\pi\cdot\textup{gon}_{k}(Y)$.

  7. (vii)

If $\pi:X\to Y$ is a dominant $k$-rational map, then
$\textup{gon}_{k}(X)\geq\textup{gon}_{k}(Y)$.

Since all modular curves $X_{0}(N)$ and their quotients have at least one
rational cusp, this result implies that their $\mathbb{Q}$-gonality is bounded
from above by their genus.

######  Proposition 3.6.

The $\mathbb{F}_{p}$-gonality of $X_{0}^{+}(N)$ is at least $5$ for the
following values of $N$:

$N$ | $p$ | $d$
---|---|---
$246$ | $5$ | $3$
$264$ | $5$ | $3$

###### Proof.

Using Magma, we compute that there are no degree $4$ functions in
$F_{p}(X_{0}(N)/\left<w_{d},w_{N}\right>)$. Since there is a degree $2$
quotient map $X_{0}^{+}(N)\to X_{0}(N)/\left<w_{d},w_{N}\right>$, 3.5(vii)
tells us that $\mathbb{F}_{p}$-gonality of $X_{0}^{+}(N)$ is $\geq 5$. ∎

###  3.2. Castelnuovo-Severi inequality

This is a very useful tool for producing a lower bound on the gonality (see
[24, Theorem 3.11.3] for a proof).

######  Proposition 3.7 (Castelnuovo-Severi inequality).

Let $k$ be a perfect field, and let $X,\ Y,\ Z$ be curves over $k$. Let non-
constant morphisms $\pi_{Y}:X\rightarrow Y$ and $\pi_{Z}:X\rightarrow Z$ over
$k$ be given, and let their degrees be $m$ and $n$, respectively. Assume that
there is no morphism $X\rightarrow X^{\prime}$ of degree $>1$ through which
both $\pi_{Y}$ and $\pi_{Z}$ factor. Then the following inequality holds:

(2) $g(X)\leq m\cdot g(Y)+n\cdot g(Z)+(m-1)(n-1).$

Since $\mathbb{C}$ and $\mathbb{Q}$ are both perfect fields, we can use
Castelnuovo-Severi inequality to get lower bounds on both $\mathbb{C}$ and
$\mathbb{Q}$-gonalities.

######  Proposition 3.8.

The $\mathbb{C}$-gonality of $X_{0}^{+}(N)$ is at least $5$ for the following
values of $N$;

$N$ | $g(X_{0}^{+}(N))$ | $d$ | $g(X_{0}(N)/\left<w_{d},w_{N}\right>)$ | $N$ | $g(X_{0}^{+}(N))$ | $d$ | $g(X_{0}(N)/\left<d,N\right>)$
---|---|---|---|---|---|---|---
$186$ | $12$ | $3$ | $3$ | $190$ | $13$ | $2$ | $3$
$210$ | $19$ | $6$ | $6$ | $214$ | $12$ | $4$ | $4$
$220$ | $14$ | $4$ | $4$ | $222$ | $15$ | $2$ | $4$
$236$ | $10$ | $4$ | $3$ | $238$ | $15$ | $3$ | $3$
$248$ | $11$ | $8$ | $3$ | $249$ | $11$ | $3$ | $3$
$252$ | $17$ | $4$ | $5$ | $254$ | $12$ | $2$ | $4$
$258$ | $19$ | $3$ | $7$ | $262$ | $15$ | $2$ | $4$
$266$ | $14$ | $14$ | $5$ | $267$ | $13$ | $3$ | $4$
$270$ | $19$ | $2$ | $7$ | $276$ | $18$ | $12$ | $5$
$278$ | $14$ | $2$ | $5$ | $282$ | $21$ | $6$ | $6$
$285$ | $15$ | $3$ | $4$ | $286$ | $17$ | $2$ | $4$
$294$ | $18$ | $3$ | $7$ | $295$ | $11$ | $5$ | $3$
$300$ | $19$ | $4$ | $7$ | $302$ | $16$ | $2$ | $5$
$303$ | $12$ | $3$ | $3$ | $310$ | $21$ | $3$ | $8$
$312$ | $23$ | $8$ | $8$ | $316$ | $17$ | $4$ | $5$
$318$ | $23$ | $2$ | $7$ | $321$ | $13$ | $3$ | $4$
$329$ | $10$ | $7$ | $3$ | $330$ | $31$ | $3$ | $13$
$420$ | $39$ | $3$ | $17$ |  |  |  |

###### Proof.

[8] and [12] tell us that the quotient curves $X_{0}^{+}(N)$ are not
hyperelliptic nor trigonal over $\mathbb{C}$ for these levels $N$. If there
existed a degree $4$ map from $X_{0}^{+}(N)$ to $\mathbb{P}^{1}$, then, by the
Castelnuovo-Severi inequality, this map would have to factor through the
quotient map $X_{0}^{+}(N)\to X_{0}(N)/\left<w_{d},w_{N}\right>$ and the
quotient curve $X_{0}(N)/\left<w_{d},w_{N}\right>$ would need to be elliptic
or hyperelliptic. However, we can again use [8] to eliminate this possibility.
∎

###  3.3. Degree $4$ maps to $\mathbb{P}^{1}$

######  Proposition 3.9.

There exists a degree $4$ rational morphism from $X_{0}^{+}(N)$ to
$\mathbb{P}^{1}$ for the following values of $N$:

$N$ | $d$ | $g(X_{0}(N)/\left<w_{d},w_{N}\right>)$ | $N$ | $d$ | $g(X_{0}(N)/\left<w_{d},w_{N}\right>)$ | $N$ | $d$ | $g(X_{0}(N)/\left<w_{d},w_{N}\right>)$
---|---|---|---|---|---|---|---|---
$78$ | $2$ | $1$ | $102$ | $2$ | $2$ | $105$ | $3$ | $1$
$106$ | $2$ | $2$ | $110$ | $2$ | $1$ | $112$ | $7$ | $2$
$114$ | $3$ | $2$ | $118$ | $2$ | $1$ | $120$ | $8$ | $2$
$123$ | $3$ | $1$ | $124$ | $4$ | $1$ | $126$ | $2$ | $2$
$133$ | $7$ | $2$ | $134$ | $2$ | $2$ | $138$ | $6$ | $2$
$140$ | $4$ | $2$ | $141$ | $3$ | $1$ | $142$ | $2$ | $1$
$145$ | $5$ | $1$ | $153$ | $9$ | $2$ | $156$ | $4$ | $2$
$158$ | $2$ | $2$ | $165$ | $11$ | $3$ | $166$ | $2$ | $2$
$177$ | $3$ | $2$ | $184$ | $8$ | $2$ | $188$ | $4$ | $1$
$195$ | $5$ | $3$ | $205$ | $5$ | $2$ | $206$ | $2$ | $2$
$207$ | $9$ | $3$ | $209$ | $11$ | $2$ | $213$ | $3$ | $2$
$221$ | $13$ | $2$ | $279$ | $9$ | $5$ | $284$ | $4$ | $2$
$287$ | $7$ | $2$ | $299$ | $13$ | $2$ |  |  |

###### Proof.

In all these cases the quotient $X_{0}(N)/\left<w_{d},w_{N}\right>$ is
elliptic or hyperelliptic and the degree $4$ rational map can be realised as a
composition map $X_{0}^{+}(N)\to
X_{0}(N)/\left<w_{d},w_{N}\right>\to\mathbb{P}^{1}$. If the genus of the
quotient is higher than $2$, we can use [8] to prove its hyperellipticity. ∎

######  Proposition 3.10.

There exists a degree $4$ rational map from $X_{0}^{+}(N)$ to $\mathbb{P}^{1}$
for

$N\in\\{136,152,163,183,197,203,211,223,269,359\\}.$

###### Proof.

The curve $X_{0}^{+}(N)$ is of genus $6$ and we can use the Magma function
Genus6GonalMap(C) to get the desired map. ∎

######  Proposition 3.11.

There exists a degree $4$ rational map from $X_{0}^{+}(N)$ to $\mathbb{P}^{1}$
for

$N\in\\{144,148,157,171,175,176,185,193,194,200,263\\}.$

###### Proof.

We find a function of degree $4$ by searching the Riemann-Roch spaces of
divisors of the form $P_{1}+P_{2}+P_{3}+P_{4}$, where $P_{i}\in
X_{0}^{+}(N)(\mathbb{Q})$. ∎

######  Proposition 3.12.

There exists a degree $4$ rational map from $X_{0}^{+}(N)$ to $\mathbb{P}^{1}$
for

$N\in\\{160,192,224,229,241,257,281\\}.$

###### Proof.

In these cases we were not able to find a degree $4$ function whose polar
divisor is supported on rational points so we had to search for quadratic
points.

We searched for quadratic points by intersecting the curve $X_{0}^{+}(N)$ with
hyperplanes of the form

$b_{0}x_{0}+\ldots+b_{k}x_{k}=0,$

where $b_{0},\ldots,b_{k}\in\mathbb{Z}$ are coprime and chosen up to a certain
bound, a similar idea as in [6, Section 3.2]. We can improve this by noting
that, in a quadratic point $(x_{0},\ldots,x_{k})$, already its first three
coordinates must be linearly dependent over $\mathbb{Q}$. Therefore, it is
enough to check the hyperplanes

$b_{0}x_{0}+b_{1}x_{1}+b_{2}x_{2}=0.$

In all of these cases we found a function of degree $4$ lying in the Riemann-
Roch space of a divisor of the form $P_{1}+P_{2}+Q+\sigma(Q)$, where
$P_{1},P_{2}\in X_{0}^{+}(N)(\mathbb{Q})$, and $Q$ is one of the quadratic
points we found. ∎

It is worth mentioning here that the running time was $\sim 20$ minutes for
$N=192$ and $\sim 4.5$ hours for $N=224$. Most of that time was spent on
searching for points. Other computations in this paper were much faster.

###  3.4. $\mathbb{C}$-gonalities

In this section we will determine the cases when the curve $X_{0}^{+}(N)$ is
$\mathbb{C}$-tetragonal. The only levels $N$ we need to look at are those for
which $X_{0}^{+}(N)$ is not $\mathbb{Q}$-tetragonal and $g(X_{0}^{+}(N))\leq
9$. This due to the Tower Theorem [20, Theorem 2.1] and its corollary [19,
Corollary 4.6.] which says that for curves of genus $\geq 10$, the existence
of a map to $\mathbb{P}^{1}$ over $\mathbb{C}$ is equivalent with the
existence of a rational map to $\mathbb{P}^{1}$.

In order to bound the number of levels we need to check, we can use the
theorem by Jeon, Kim and Park, mentioned in the Introduction.

######  Theorem 3.13.

[16, Theorem 1.2.] Let $X_{\Gamma}$ be the algebraic curve corresponding to a
congruence subgroup $\Gamma\subseteq\operatorname{SL}_{2}(\mathbb{Z})$ of
index

$D_{\Gamma}=[\operatorname{SL}_{2}(\mathbb{Z}):\pm\Gamma].$

If $X_{\Gamma}$ is $d$-gonal, then $D_{\Gamma}\leq\frac{12000}{119}d$.

######  Corollary 3.14.

$X_{0}^{+}(N)$ is not tetragonal for $N\geq 807$.

###### Proof.

Suppose $X_{0}^{+}(N)$ is tetragonal. Then $X_{0}(N)$ has a degree $8$ map to
$\mathbb{P}^{1}$. Since $-I\in\Gamma_{0}(N)$, we have that
$\psi(N)=D_{\Gamma_{0}(N)}\leq\frac{12000}{119}\cdot 8$ (here
$\psi(N)=N\prod_{q\mid N}(1+\frac{1}{q})$, as mentioned in 3.1). ∎

Therefore, any curve $X_{0}^{+}(N)$ which is $\mathbb{C}$-tetragonal, but not
$\mathbb{Q}$-tetragonal must be of genus $\leq 9$ and level $N\leq 806$. This
leaves us with reasonably many cases to be solved manually. The only levels
$N$ we need to check are in the table below.

$N$ | $g(X_{0}^{+}(N))$ | $N$ | $g(X_{0}^{+}(N))$ | $N$ | $g(X_{0}^{+}(N))$ | $N$ | $g(X_{0}^{+}(N))$
---|---|---|---|---|---|---|---
$130$ | $8$ | $132$ | $8$ | $150$ | $8$ | $154$ | $9$
$170$ | $9$ | $172$ | $9$ | $178$ | $9$ | $187$ | $7$
$189$ | $7$ | $196$ | $7$ | $201$ | $8$ | $217$ | $8$
$219$ | $8$ | $225$ | $8$ | $231$ | $9$ | $233$ | $7$
$242$ | $9$ | $243$ | $7$ | $245$ | $8$ | $247$ | $8$
$256$ | $9$ | $259$ | $8$ | $271$ | $6$ | $275$ | $9$
$283$ | $9$ | $289$ | $7$ | $293$ | $8$ | $335$ | $8$
$341$ | $9$ | $361$ | $9$ | $383$ | $8$ | $419$ | $9$
$431$ | $9$ | $479$ | $8$ |  |  |  |
Table 1. Levels $N<915$ for which the curve $X_{0}^{+}(N)$ is not
$\mathbb{Q}$-tetragonal and $g(X_{0}^{+}(N))\leq 9$

We immediately see that for $N=271$ there exists a degree $4$ morphism since
the genus is $6$. For the other cases we will use graded Betti numbers
$\beta_{i,j}$. We will follow the notation in [17, Section 1.]. The results we
mention can be found there and in [19, Section 3.1.].

######  Definition 3.15.

For a curve $X$ and divisor $D$ of degree $d$, $g_{d}^{r}$ is a subspace of
the Riemann-Roch space $L(D)$ such that $\dim V=r+1$.

Therefore, we want to determine whether $X_{0}^{+}(N)$ has a $g_{4}^{1}$.
Green’s conjecture relates graded Betti numbers $\beta_{i,j}$ with the
existence of $g_{d}^{r}$.

######  Conjecture 3.16 (Green, [9]).

Let $X$ be a curve of genus $g$. Then $\beta_{p,2}\neq 0$ if and only if there
exists a divisor $D$ on $X$ of degree $d$ such that a subspace $g_{d}^{r}$ of
$L(D)$ satisfies $d\leq g-1$, $r=\ell(D)-1\geq 1$, and $d-2r\leq p$.

The ”if” part of this conjecture has been proven in the same paper.

######  Theorem 3.17 (Green and Lazarsfeld, Appendix to [9]).

Let $X$ be a curve of genus $q$. If $\beta_{p,2}=0$, then there does not exist
a divisor $D$ on $X$ of degree $d$ such that a subspace $g_{d}^{r}$ of $L(D)$
satisfies $d\leq g-1$, $r\geq 1$, and $d-2r\leq p$.

######  Corollary 3.18.

Let $X$ be a curve of genus $g\geq 5$ with $\beta_{2,2}=0$. Then
$\textup{gon}_{\mathbb{C}}(X)\geq 5$.

######  Corollary 3.19.

The curve $X_{0}^{+}(N)$ is not tetragonal for all $N$ in Table 1 except
$243,271$.

###### Proof.

For all these curves we compute $\beta_{2,2}=0$. The computation time for
$N=361$ was around $1$ hour, the other computations were much faster and took
less than $10$ minutes. ∎

To prove the next result, we need to introduce the Clifford index and Clifford
dimension.

######  Definition 3.20.

For a curve $X$, let $D$ be a divisor on $X$ and $K$ a canonical divisor on
$X$. The Clifford index of $D$ is the integer

$\textup{Cliff}(D):=\deg D-2(\ell(D)-1),$

and the Clifford index of $X$ is

$\textup{Cliff}(X):=\min\\{\textup{Cliff}(D)\mid\ell(D)\geq 2,\ell(K-D)\geq
2\\}.$

The Clifford dimension of $X$ is defined to be

$\textup{CD}(X):=\min\\{\ell(D)-1\mid\textup{Cliff}(D)=\textup{Cliff}(X)\\}.$

For every such divisor $D$ that achieves the minimum we say that it computes
the Clifford dimension.

The Clifford index gives bounds for the $\mathbb{C}$-gonality of $X$ [7]:

$\textup{Cliff}(X)+2\leq\textup{gon}_{\mathbb{C}}(X)\leq\textup{Cliff}(X)+3.$

######  Proposition 3.21.

The curve $X_{0}^{+}(243)$ is tetragonal over $\mathbb{C}$.

###### Proof.

We compute $g(X_{0}^{+}(243))=7$ and the Betti table. In particular, we get
$\beta_{2,2}=9$. By [23, Table 1.], this implies the existence of $g_{6}^{2}$.
Now we use a similar argument as in [17, Page 310, Case 3].

Since there exists a $g_{6}^{2}$, we can apply Riemann-Roch theorem to prove
that the Clifford index $\textup{Cliff}(X_{0}^{+}(243))$ is at most $2$. The
homogenous ideal of the canonical embedding of this curve is generated by
quadrics (Magma function X0NQuotient(243,[243]) gives the canonical embedding
for example). By [17, Section 1.4], this implies that
$\textup{Cliff}(X_{0}^{+}(243))\geq 2$ and we conclude that
$\textup{Cliff}(X_{0}^{+}(243))=2$.

We know that $\textup{CD}(X_{0}^{+}(243))\leq 2$ from the existence of
$g_{6}^{2}$. Therefore, it is enough to show $\textup{CD}=1$. However,
$\textup{CD}\neq 2$ since $X_{0}^{+}(243)$ is not a smooth plane curve of
degree $d\geq 5$ (a smooth plane curve of dimension $d$ has genus
$(d-1)(d-2)/2$). ∎

###  3.5. Proofs of the main theorems

###### Proof of 1.1.

As was mentioned before, Hasegawa and Shimura [12] already solved the cases
when $g(X_{0}^{+}(N))\neq 4$. When the genus is equal to $4$, we used the
Magma function Genus4GonalMap(C) to find the degree $3$ rational map to
$\mathbb{P}^{1}$. ∎

###### Proof of 1.2.

When $g(X_{0}^{+}(N))=4$, we used 3.4 to prove that there are no degree $3$
rational maps to $\mathbb{P}^{1}$. Therefore, $\mathbb{Q}$-gonality must be
equal to $4$ by 3.5.

If $g(X_{0}^{+}(N))\neq 4$, we know from [12] that the curve $X_{0}^{+}(N)$ is
not trigonal over $\mathbb{C}$. For levels $N$ in the list, in Section 3.3 we
find a rational degree $4$ map to $\mathbb{P}^{1}$, meaning that
$\textup{gon}_{\mathbb{Q}}X_{0}^{+}(N)=4$ in these cases. For other levels
$N$, we prove in Section 3.1 and Section 3.2 that there are no rational degree
$4$ maps to $\mathbb{P}^{1}$, and so $\textup{gon}_{\mathbb{Q}}X_{0}^{+}(N)>4$
in these cases. ∎

######  Corollary 3.22.

The $\mathbb{Q}$-gonality of $X_{0}(N)$ is equal to $8$ for

$N\in\\{193,194,207,224,229,241,257,281\\}.$

For $N\in\\{194,224,257,281\\}$ the $\mathbb{C}$-gonality of $X_{0}(N)$ is
also $8$.

###### Proof.

The composition map $X_{0}(N)\to X_{0}^{+}(N)\to\mathbb{P}^{1}$ is a degree
$8$ rational map and from [19, Tables 1.,2.,3.] we get that
$\textup{gon}_{\mathbb{Q}}(X_{0}(N))>7$. For $N=193,207,229,241$ this follows
from the bound on $\mathbb{F}_{p}$-gonality; codes for that can be found on

https://github.com/orlic1/gonality_X0/tree/main/Fp_gonality).

For $N=194,224,257,281$ we can use Castelnuovo-Severi inequality to prove it,
meaning that we also get the lower bound on $\mathbb{C}$-gonality in these
cases. ∎

## References

  * [1] D. Abramovich, A linear lower bound on the gonality of modular curves, Internat. Math. Res. Notices, (1996), pp. 1005–1011.
  * [2] D. Abramovich and J. Harris, Abelian varieties and curves in $W_{d}(C)$, Compos. Math., 78 (1991), pp. 227–238.
  * [3] F. Bars, Bielliptic modular curves, J. Number Theory, 76 (1999), pp. 154–165.
  * [4] F. Bars, J. González, and M. Kamel, Bielliptic quotient modular curves with n square-free, Journal of Number Theory, 216 (2020), pp. 380–402.
  * [5] F. Bars, M. Kamel, and A. Schweizer, Bielliptic quotient modular curves of $x_{0}(n)$, Mathematics of Computation, volume 92, number 340, March 2023, pages 895-929, (2022).
  * [6] J. Box, Quadratic points on modular curves with infinite Mordell-Weil group, Math. Comp., 90 (2021), pp. 321–343.
  * [7] M. Coppens and G. Martens, Secant spaces and Clifford’s theorem, Compos. Math., 78 (1991), pp. 193–212.
  * [8] M. Furumoto and Y. Hasegawa, Hyperelliptic Quotients of Modular Curves $X_{0}(N)$, Tokyo Journal of Mathematics, 22 (1999), pp. 105 – 125.
  * [9] M. L. Green, Koszul cohomology and the geometry of projective varieties. Appendix: The nonvanishing of certain Koszul cohomology groups (by Mark Green and Robert Lazarsfeld), J. Differ. Geom., 19 (1984), pp. 125–167, 168–171.
  * [10] J. Harris and J. H. Silverman, Bielliptic curves and symmetric products, Proceedings of the American Mathematical Society, 112 (1991), pp. 347–356.
  * [11] Y. Hasegawa and M. Shimura, Trigonal modular curves, Acta Arith., 88 (1999), pp. 129–140.
  * [12] , Trigonal modular curves $X_{0}^{+d}(N)$, Proc. Japan Acad., Ser. A, 75 (1999), pp. 172–175.
  * [13] , Trigonal modular curves $X_{0}^{*}(N)$, Proc. Japan Acad., Ser. A, 76 (2000), pp. 83–86.
  * [14] , Trigonal quotients of modular curves $X_{0}(N)$., Proc. Japan Acad., Ser. A, 82 (2006), pp. 15–17.
  * [15] D. Jeon, Bielliptic modular curves x0+(n), Journal of Number Theory, 185 (2018), pp. 319–338.
  * [16] D. Jeon, C. H. Kim, and E. Park, On the torsion of elliptic curves over quartic number fields, J. London Math. Soc. (2), 74 (2006), pp. 1–12.
  * [17] D. Jeon and E. Park, Tetragonal modular curves, Acta Arith., 120 (2005), pp. 307–312.
  * [18] B. Kadets and I. Vogt, Subspace configurations and low degree points on curves, 2022.
  * [19] F. Najman and P. Orlić, Gonality of the modular curve $x_{0}(n)$, Mathematics of Computation, (2023).
  * [20] K. V. Nguyen and M.-H. Saito, $d$-gonality of modular curves and bounding torsions.  preprint, arXiv:alg-geom/9603024.
  * [21] A. P. Ogg, Hyperelliptic modular curves, Bull. Soc. Math. France, 102 (1974), pp. 449–462.
  * [22] B. Poonen, Gonality of modular curves in characteristic $p$, Math. Res. Lett., 14 (2007), pp. 691–701.
  * [23] F.-O. Schreyer, Syzygies of canonical curves and special linear series, Mathematische Annalen, 275 (1986), pp. 105–137.
  * [24] H. Stichtenoth, Algebraic function fields and codes, vol. 254 of Grad. Texts Math., Berlin: Springer, 2nd ed. ed., 2009.
  * [25] P. G. Zograf, Small eigenvalues of automorphic laplacians in spaces of parabolic forms, Journal of Soviet Mathematics, 36 (1987), pp. 106–114.


# Open-book Video Captioning with Retrieve-Copy-Generate Network

Ziqi Zhang1,2,3 Zhongang Qi2 Chunfeng Yuan1 Corresponding author. Ying Shan2
Bing Li1 Ying Deng5 Weiming Hu1,3,4
1NLPR  Institute of Automation  Chinese Academy of Sciences
2Applied Research Center (ARC)  Tencent PCG
3School of Artificial Intelligence  University of Chinese Academy of Sciences
4CAS Center for Excellence in Brain Science and Intelligence Technology
5School of Aeronautical Manufacturing Engineering  Nanchang Hangkong
University
{ziqi.zhang,cfyuan,bli,wmhu}<EMAIL_ADDRESS>
###### Abstract

In this paper, we convert traditional video captioning task into a new
paradigm, , Open-book Video Captioning, which generates natural language under
the prompts of video-content-relevant sentences, not limited to the video
itself. To address the open-book video captioning problem, we propose a novel
Retrieve-Copy-Generate network, where a pluggable video-to-text retriever is
constructed to retrieve sentences as hints from the training corpus
effectively, and a copy-mechanism generator is introduced to extract
expressions from multi-retrieved sentences dynamically. The two modules can be
trained end-to-end or separately, which is flexible and extensible. Our
framework coordinates the conventional retrieval-based methods with orthodox
encoder-decoder methods, which can not only draw on the diverse expressions in
the retrieved sentences but also generate natural and accurate content of the
video. Extensive experiments on several benchmark datasets show that our
proposed approach surpasses the state-of-the-art performance, indicating the
effectiveness and promising of the proposed paradigm in the task of video
captioning.

##  1 Introduction

Video captioning is one of the most important vision-language tasks, and it
seeks to automatically describe what has happened in the video according to
the visual content. Recently, many promising methods [36, 22, 24, 34, 2] have
been proposed to address this task. These methods mainly focus on learning the
spatial-temporal representations of videos to fully tap visual information and
devising novel decoders to achieve visual-textual alignment or controllable
decoding. In general, there exist some drawbacks for most of the existing
work: first, since the video content is the only source of input, the
generation process lacks appropriate guidance, resulting in the generations of
more generic sentences; second, the memory or knowledge domain of the model is
fixed after training and cannot be expanded or revisited unless retraining.

Figure 1: Pipeline comparison of the existing methods and our method. Our
generation is produced based on not only video content but also the cues of
multi-retrieved sentences searched from text corpus by a cross-modal
retriever. Pluggable retriever provides guidance and expansion for the
generating model.

To address these issues, we propose an Open-book Video Captioning paradigm. We
first compare the two cross-modal tasks for better illustration: Video-Text
Retrieval (VTR) and Video Captioning (VC). VTR is a discriminative task that
can access all the information of visual and textual modalities all the time;
VC as a generative task can only produce words based on current generated
words and visual information, which is more challenging than VTR. Instead of
performing the VC task directly, we propose to convert it into two-stages: we
first perform VTR to search for sentences relevant to the given video from the
text corpus; then, we leverage the retrieval sentences as extra hints or
guidance for caption generation. During the inference, the generator can
generate words based on the video content or directly copy expressions from
retrieved sentences. The flexible VTR and the changeable corpus provide the
possibility for the model’s extension or revision.

The inspiration for the proposed paradigm comes from the Open-domain Question
Answering task [18, 10, 15], which requires a system to answer any questions
utilizing large-scale documents. Open means not providing the system with
documents containing the correct answers directly but requires the system to
retrieve documents related to the question from massive corpus and then
generate the correct answer based on them. We claim that the open-domain
mechanism is also effective in cross-modal interaction area and can thus be
applied to the video captioning task.

This mechanism essentially extends the knowledge domain of the model learned
only from the labeled data. It is known that producing large-scale, high-
quality labeled data is extremely laborious and time-consuming. Instead, a
model learns to collect related references, distinguish useful hints, abstract
and summarize information from external weakly labeled or unlabeled documents
breaks through the limitations of labeled data. This is valuable especially
for the industry-scale video platforms where hundreds of millions weakly
labeled or unlabeled data are generated everyday. Unlike traditional semi-
supervised learning that utilizes fixed weakly labeled or unlabeled examples
directly for training, the proposed paradigm makes the model learn to extract
useful information from changeable weakly labeled or unlabeled corpus directly
for inference.

To realize the aforementioned open-book video captioning, we introduce a novel
Retrieve-Copy-Generate (RCG) network. We introduce a Video-to-Text Retriever
to search for video-content-relevant sentences from the corpus containing the
whole sentences of the training set. Our retriever follows the Bi-encoders
[14] structure and utilizes both motion and appearance features to search
desired sentences efficiently and effectively. For the example in Fig.1, the
top retrieved sentences contain expressions “on a mat”, “does somersaults”,
and “someone watches”, which describe the given video accurately. Then, the
retrieved sentences and the visual features are passed to the generator. A
novel Copy-Mechanism Generator is introduced, which dynamically decides
whether to copy the expressions directly from multi-retrieved sentences or
generate new words from the video contents. The model combines the information
from the video content and the words copied from retrievals to generate the
final caption “A kid doing a somersault on a mat while a boy watches”, which
is much better than the generic caption “A little girl practices in a room”.

The contributions of this work are listed as follows:

  * (1)

We propose to solve the video captioning task with an open-book paradigm,
which generates captions under the guidance of video-content-retrieval
sentences, not limited to the video itself.

  * (2)

We introduce a novel Retrieve-Copy-Generate network to tackle this task, where
an improved cross-modal retriever is utilized to provide hints for generator,
and a copy-mechanism generator is proposed for dynamical copying and better
generation.

  * (3)

The extensive experimental results highlight the benefits of combining cross-
modal retrieval with copy-mechanism generation for the video caption task. The
proposed approach achieves state-of-the-art results on VATEX and superior
performance on MSR-VTT.

Figure 2: Overview of the proposed Retrieve-Copy-Generate Network for Open-
book Video Captioning. The left side is the pipeline of our method, which
consists of two components: the Video-to-Text Retriever that searches for the
video-content-relevant sentences from the corpus containing all the sentences
in the training set; the Copy-mechanism Generator produces the words by steps
under the hints or guidance of retrieved sentences and the visual features. In
the upper right corner, the Bi-encoders architecture is leveraged to
efficiently and effectively achieve the cross-modal retrieval. Here we only
show the similarity calculation process based on the motion features. In the
bottom right corner, a hierarchical caption decoder is used to generate the
fixed vocabulary based on the video content. Meanwhile, an improved multi-
pointer module directly copies the expressions from the retrieved sentences
for a better generation.

##  2 Related Works

Video Captioning. Reviewing the early retrieval-based visual description
methods can be roughly divided into two categories: [21, 9, 16] cast the
problem as retrieval from a visual space problem and transfer the retrieved
descriptions to a novel image/video; [5, 25, 11] cast the problem as retrieval
from a multimodal space problem. Although retrieval-based methods can find
human-like sentences with similar semantics to the video, it is challenging to
generate an entirely correct description due to limited retrieval samples.
With the advent of the encoder-decoder framework, most of the current work is
studying how to better use visual features [35, 23, 1, 36, 22, 24] and design
elaborate models [34, 2, 28, 13] to generate sentences directly. However, the
diversity and controllability of sentences generated in this way are not
satisfactory. We coordinate the classical retrieval-based method with modern
encoder-decoder method to generate descriptions with diverse expressions and
accurate video contents.

Video-Text Retrieval. Video-Text Retrieval is a fundamental discriminative
vision-language task that helps to learn the semantic alignment of different
modalities. In general, there are two kinds of model architectures, , Bi-
encoders [4, 6, 19, 20] and Cross-encoders [26, 30, 33, 3]. Bi-encoders map
the query and candidates into a common feature space with two separate
encoders. Since there is no interaction between their features, Bi-encoders
are lower-accuracy but very efficient during evaluation. Compared with it,
Cross-encoders yield rich interactions between query and candidates by
integrating features at an early stage. This helps to gain a higher-accuracy
but steep computational cost. We make efficient and effective improvements to
the Bi-encoders based model, which is more applicable to our paradigm.

Retrieval Augmented Generation Tasks. A series of NLP works utilize the
retrieved knowledge for better generation, such as open-domain question
answering (Open-QA), neural machine translation(NMT), and dialog generation.
REALM [10] and ORQA [17] show promising results on the Open-QA task by
combining masked language models with a differentiable retriever. DPR [15]
implements a dense embedding-based retriever to replace traditional sparse
retrievers and achieves significant performance. RAG [18] combines this
learnable retriever in an end-to-end pre-trained generator. Meanwhile, SEG-NMT
[7] and RER [12] leverage retrieval to assist the NMT system. DeepCopy [32]
also retrieves relevant unstructured sentences as external knowledge to assist
dialog generation. Inspirited from these works, we extend this pattern to
cross-modal generation task, , video captioning, which is quite different and
challenging.

##  3 Retrieve-Copy-Generate Network

We show the overall pipeline of the proposed Retrieve-Copy-Generate (RCG) for
Open-book video captioning in Fig.2. It consists of two components: (1) the
video-to-text retriever $p_{\eta}(z|x)$ with parameter $\eta$, which retrieves
the top-$k$ semantically similar sentences $z$ according to video $x$; and (2)
the copy-mechanism generator $p_{\theta}(y_{t}|z,x,y_{1:t-1})$ parametrized by
learnable $\theta$, which leverages the above additional retrieved sentences
$z$, the original visual information $x$, and the previous $t-1$ generated
tokens $y_{t-1}$ to generate the current target token $y_{t}$. Formally, the
conditional probability of producing caption given video for our proposed
approach are defined as follows:

$p(y|x)=\prod_{t=1}^{T}\sum_{i=1}^{topk}\underbrace{p_{\eta}(z_{i}|x)}_{retriever}\underbrace{p_{\theta}(y_{t}|z_{i},x,y_{1:t-1}}_{generator}),$
(1)

where $y$ is the target sentence with $N$ tokens. Since a dataset usually
contains videos with semantically similar content, the corresponding sentences
always have similar forms or expressions. Thus, the top-$k$ retrieved
sentences $z$ can provide information related to the video’s content $x$ to
help the generator to produce the target sentence more accurately. Meanwhile,
$p_{\eta}(z|x)$ can be treated as a soft-threshold, which represents the
confidence of whether the generator can copy words directly from the retrieval
sentences or not.

The theoretical part is organized as follows. We introduce how to construct a
simple but effective cross-modal retriever in Sec.3.1, and how to extract
representations related to video content from multi-retrieved sentences to
augment the generation in Sec.3.2. The iterative training procedure of the
whole system is described in Sec.3.3.

###  3.1 Effective Video-to-Text Retriever

The major function of our retriever is to find the top-$k$ most similar
sentences $z$ given video $x$ in a massive retrieval corpus $\mathcal{Z}$.
Note that $\left|\mathcal{Z}\right|$ can be large enough to cover all the
video contents ideally (the corpus contain nearly 0.3 million pieces of
sentences in our implement), and $k$ is usually small, such as $1\sim 30$.

The video-to-text retriever applies Bi-encoders architecture: the textual
encoder $E_{txt}(\cdot)$ maps all sentences in the corpus $\mathcal{Z}$ into
$d$-dimensional vectors and constructs a candidate dataset; the visual encoder
$E_{vis}(\cdot)$ maps the video $x$ into a $d$-dimensional vector as a query.
The whole retrieval model is trained by metric learning, which embeds visual
and textual modalities into a joint high-dimensional semantic space.
Subsequently, the similarity between the video and text is defined as the dot-
product of their embedding vectors:

$\mathrm{sim}(x,z)=E_{txt}(z)\cdot E_{vis}(x).$ (2)

In this way, the sentences whose embeddings are top-$k$ closest to the video
embedding are sorted by the retriever. We use a simple but effective way to
build these two encoders.

Textual Encoder. Given a sentence $\boldsymbol{z}=\\{s_{1},\cdots,s_{L}\\}$,
each word is first input to a bi-LSTM to generate a sequence of
$d$-dimensional context-aware word embeddings
$\boldsymbol{w}=\\{\boldsymbol{w_{1}},\cdots,\boldsymbol{w_{L}}\\}$:

$\displaystyle\overrightarrow{\boldsymbol{w_{t}}}$
$\displaystyle=\overrightarrow{\mathrm{LSTM}}(\boldsymbol{W_{s}}s_{t},\boldsymbol{w_{t-1}};\eta_{s}),$
(3) $\displaystyle\overleftarrow{\boldsymbol{w_{t}}}$
$\displaystyle=\overleftarrow{\mathrm{LSTM}}(\boldsymbol{W_{s}}s_{t},\boldsymbol{w_{t+1}};\eta_{s}),$
$\displaystyle\boldsymbol{w_{t}}$
$\displaystyle=(\overrightarrow{\boldsymbol{w_{t}}}+\overleftarrow{\boldsymbol{w_{t}}})/2,$

where $\boldsymbol{W_{s}}$ is a learnable word embedding matrix and $\eta_{s}$
denotes the parameters of LSTM.

Subsequently, all the embeddings are aggregated to a single vector as the
overall representation. For simplicity, we denote the aggregation function as
$\mathrm{Agg}(\cdot;\boldsymbol{v})$ which utilizes the multiplicative
attention mechanism, where the parameter $\boldsymbol{v}\in\mathbb{R}^{d}$ can
be viewed as a learnable core that gives higher weights to more discriminative
features:

$\mathrm{Agg}(\boldsymbol{w};\boldsymbol{v})=\sum_{t=1}^{L}\alpha_{t}\boldsymbol{w_{t}},$
(4) $\alpha_{t}=\mathrm{softmax}(\boldsymbol{v}^{T}\boldsymbol{w_{t}}).$ (5)

Thus, the word embeddings are aggregated to a single vector via
$\boldsymbol{\bar{e}_{w}}=\mathrm{Agg}(\boldsymbol{w};\boldsymbol{v_{s}})$,
where $\boldsymbol{v_{s}}$ is the parameters of the word aggregation function.

Visual Encoder. It is assumed that appearance features
$\boldsymbol{x^{a}}=\\{\boldsymbol{x^{a}_{1}},\cdots,\boldsymbol{x^{a}_{K}}\\}$
and motion features
$\boldsymbol{x^{m}}=\\{\boldsymbol{x^{m}_{1}},\cdots,\boldsymbol{x^{m}_{K}}\\}$
together constitute the representation of the video $x$. Each feature is the
original feature after linear transformation to $d$-dimensional. The video is
usually divided into $K$ key frames and segments.

For visual encoder, we directly aggregate features at different moments
according to their importance, since the sequential information has been
involved in motion features, and appearance features only provide
discrimination for the main objects in the video. We aggregate motion
embeddings via
$\boldsymbol{\bar{e}_{m}}=\mathrm{Agg}(\boldsymbol{x^{m}};\boldsymbol{v_{m}})$
and appearance embeddings via
$\boldsymbol{\bar{e}_{a}}=\mathrm{Agg}(\boldsymbol{x^{a}};\boldsymbol{v_{a}})$
by reusing the aggregation function defined by Eq.4, where
$\boldsymbol{v_{m},v_{a}}\in\mathbb{R}^{d}$ are the parameters of two
modalities’ aggregation functions.

We take the average of appearance and motion similarities as the final video-
text similarity:

$\mathrm{sim}(\boldsymbol{x},\boldsymbol{z})=\boldsymbol{\tilde{e}^{T}_{w}}(\boldsymbol{\tilde{e}_{m}}+\boldsymbol{\tilde{e}_{a}})/2,$
(6)

where
$\boldsymbol{\tilde{e}}=\boldsymbol{\bar{e}}/||\boldsymbol{\bar{e}}||_{2}$ is
the operation of $L_{2}$ normalization.

Pre-training and Retrieval. The training of our cross-modal retriever follows
the contrastive learning, where each positive pair $(x^{+},z^{+})$ should be
closer than any other negative pairs $(x^{+},z^{-})$ and $(x^{-},z^{+})$ in a
mini-batch. The max-margin ranking loss function pushes the hardest negative
pair $\Delta$ distance away from the positive pair, as follows:

$\begin{aligned}
\mathcal{L}_{ret}=[\Delta+\mathrm{sim}(x^{+},z^{-})-\mathrm{sim}(x^{+},z^{+})]_{+}\\\
+[\Delta+\mathrm{sim}(x^{-},z^{+})-\mathrm{sim}(x^{+},z^{+})]_{+}\end{aligned},$
(7)

where $[x]_{+}=max(x,0)$ and $\Delta$ is the slack coefficient. The sentences
belonging to other videos in the mini-batch are all negative samples of this
video and vice versa.

Due to the independent structure of Bi-encoders, the textual embeddings can be
calculated offline in advance for efficient evaluation. Given video $x$ as the
query, top-$k$ most relevant retrieved sentences can be found following Eq.6,
and the probability of retrieved sentence $\boldsymbol{z_{i}}$ is estimated
as:

$p_{\eta}(\boldsymbol{z_{i}}|\boldsymbol{x})=\mathrm{softmax}(\mathrm{sim}(\boldsymbol{x},\boldsymbol{z_{i}})),\
\boldsymbol{z_{i}}\in\\{\boldsymbol{z_{1}},\cdots,\boldsymbol{z_{topk}}\\}.$
(8)

###  3.2 Copy-mechanism Caption Generator

To generate captions based on the video content and the retrieved sentences,
we design a novel copy-mechanism caption generator, which consists of the
Hierarchical Caption Decoder and the Dynamic Multi-pointers Module. We
describe the proposed generator in detail as follows:

Hierarchical Caption Decoder. It consists of attention-LSTM and language-LSTM.
Formally, the attention-LSTM tries to focus on different visual features
$\boldsymbol{x}=\left[\boldsymbol{x^{m}};\boldsymbol{x^{a}}\right]$ according
to the current hidden state $\boldsymbol{h^{a}_{t}}$ to achieve the visual
context $\boldsymbol{c^{v}_{t}}$, where $\left[\cdot;\cdot\right]$ is the
concatenation of two sets of features in the feature dimension. The current
hidden state of the attention-LSTM $\boldsymbol{h^{a}_{t}}$ depends on the
previous hidden state $\boldsymbol{h^{a}_{t-1}}$ and generated word $y_{t-1}$:

$\boldsymbol{h^{a}_{t}}=\mathrm{LSTM}(\boldsymbol{W_{e}}y_{t-1},\boldsymbol{h^{a}_{t-1}};\theta_{a}),$
(9)
$\boldsymbol{c^{v}_{t}}=\mathrm{Att}(\boldsymbol{h^{a}_{t}},\boldsymbol{x},\boldsymbol{x};\theta_{v}),$
(10)

where, $\mathrm{Att}(query,key,value;\theta)$ denotes standard additive
attention module with parameters $\theta$ for simplify; $\boldsymbol{W_{e}}$
is the word embedding matrix.

Then, the language-LSTM aggregates the current state $\boldsymbol{h^{a}_{t}}$
and the visual context $\boldsymbol{c^{v}_{t}}$ to generate the probability
distribution of the fixed vocabulary $\boldsymbol{p_{voc}}$ at each time step:

$\boldsymbol{h^{l}_{t}}=\mathrm{LSTM}([\boldsymbol{h^{a}_{t}},\boldsymbol{c^{v}_{t}}],\boldsymbol{h^{l}_{t-1}};\theta_{l}),$
(11)
$\boldsymbol{p_{voc}}=\mathrm{softmax}(\boldsymbol{W_{voc}}\boldsymbol{h^{l}_{t}}+\boldsymbol{b_{voc}}),$
(12)

where $\boldsymbol{p_{voc}}\in\mathbb{R}^{1\times V}$;
$\theta_{v},\theta_{a},\theta_{l}$ are parameters of the visual attention
module and hierarchical-LSTMs; $\boldsymbol{W_{voc}}$ and
$\boldsymbol{b_{voc}}$ are all learnable parameters.

Dynamic Multi-pointers Module. In Sec.3.1, we get the top-$k$ most similar
retrieved sentences to the given video. We also leverage the bi-
LSTM$(\cdot;\theta_{z})$ to encode these retrieved sentences to
$\boldsymbol{z}=\\{\boldsymbol{z_{1}},\cdots,\boldsymbol{z_{topk}}\\}$. Each
retrieved sentence $\boldsymbol{z_{i}}$ consists of a set of words with their
embeddings
$\boldsymbol{z_{i}}=\\{\boldsymbol{z^{1}_{i}},\cdots,\boldsymbol{z^{L}_{i}}\\}$.

To draw on the expressions in multiple retrieved sentences, we propose the
multi-pointers module improved on Pointer-Networks [27] by extending the
single document copying to multiple’s. At each decoding step $t$, the multi-
pointers module acts on each retrieved sentence $\boldsymbol{z_{i}}$
separately, uses the hidden state $\boldsymbol{h^{l}_{t}}$ as the query to
attend $L$ words, and produces the word probability distribution
$\boldsymbol{p_{ret,i}}\in\mathbb{R}^{1\times L}$ of the corresponding
sentence:

$\boldsymbol{p_{ret,i}},\boldsymbol{c^{r}_{i,t}}=\mathrm{Att}(\boldsymbol{h^{l}_{t}},\boldsymbol{z_{i}},\boldsymbol{z_{i}};\theta_{r}),$
(13)

where $\mathrm{Att}(\cdot)$ is the additive attention module with parameters
$\theta_{r}$; $\boldsymbol{c^{r}_{i,t}}$ denotes the context of the retrieved
sentence which is the weighted summation of $\boldsymbol{z_{i}}$ by
$\boldsymbol{p_{ret,i}}$. Since not all the words in the retrieved sentence
are valid, the model needs to decide whether to copy or generate dynamically.
The probabilities of copying words from each retrieved sentence
$\boldsymbol{z_{i}}$ are determined by the semantic context of retrieved
sentence $\boldsymbol{c^{r}_{i,t}}$ and the decoder’s state
$\boldsymbol{h^{l}_{t}}$:

$p_{copy,i}=\sigma(\boldsymbol{W_{r}c^{r}_{i,t}}+\boldsymbol{W_{l}h^{l}_{t}}),$
(14)

Finally, we get the generation probability distribution
$\boldsymbol{p_{\theta}}\in\mathbb{R}^{topk\times V}$ , which is summarized by
accumulating the probability distribution of the fixed vocabulary
$\boldsymbol{p_{voc}}\in\mathbb{R}^{1\times V}$ and the dynamic words in
retrieved sentences $\boldsymbol{p_{ret}}\in\mathbb{R}^{topk\times V}$
conditioned on copy probability
$\boldsymbol{p_{copy}}\in\mathbb{R}^{topk\times 1}$ via broadcasting. Note
that $\boldsymbol{p_{ret}}$ has been extended to the same size of fixed
vocabulary.

$\boldsymbol{p_{\theta}}=(1-\boldsymbol{p_{copy}})\boldsymbol{p_{voc}}+\boldsymbol{p_{copy}}\boldsymbol{p_{ret}}.$
(15)

###  3.3 Training

Review the definition of the open-book video captioning, the final probability
of target word is jointly predicted by the similarities
$\boldsymbol{p_{\eta}}$ of the retrieved sentences and the generation
probabilities $\boldsymbol{p_{\theta}}$ with copy-mechanism by substituting
Eq.8 and Eq.15 into Eq.1. Our goal is to minimize the negative log-likelihood
of each target word $y_{t}$:

$\mathcal{L}_{gen}=-\sum_{t=1}^{T}log\sum_{i=1}^{topk}\boldsymbol{p_{\eta}}(\boldsymbol{z_{i}})\boldsymbol{p_{\theta,i}}(y_{t}).$
(16)

These two components can be trained separately. Assuming that given an off-
the-shelf retriever, our model can directly use the retrieval results for
generation. In this case, we keep the retriever fixed, only fine-tuning the
generator. This provides convenience for replacing better retrievers or
adapting to different datasets.

Additionally, the retriever and generator can be jointly trained end-to-end in
an iterative manner for better performance. However, updating the retriever
directly during training may decrease its performance drastically as the
generator has not been well trained to begin with. For a stable training, we
add the ranking loss mentioned in Sec3.2 to the generation loss as a
constraint , $\mathcal{L}=\mathcal{L}_{ret}+\mathcal{L}_{gen}$. Moreover, we
periodically (per epoch in our work) perform the retrieval process because it
is costly and frequently changing the retrieval results will confuse the
generator.

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We carry out all the experiments on MSR-VTT and recent VATEX, which
are two large-scale video-caption datasets. The MSR-VTT [31] contains 10,000
open-domain video clips from YouTube website. For each clip, there are 20
human descriptions and one of 20 categories (music, sports, ). We follow the
standard splits with 6,573 videos for training, 497 videos for validation, and
2,990 videos for testing. The most recent VATEX [29] dataset reuses the videos
from Kinetics-600 that contains 41,269 video clips with 10 English text
sentences. According to the official splits, the dataset is divided into
25,991 training, 3,000 validation, and 6,000 public testing.

Due to the strict quality control, VATEX has enricher, longer (average 16
tokens v.s. 9 tokens in MSRVTT) and more accurate annotations. Therefore, we
conduct and report most of the experiments on this dataset. Other experiments
can be seen in the supplementary materials.

Evaluation Metrics. We use standard captioning metrics, , BLEU-4, Meteor,
Rouge-L, and CIDEr, to evaluate the performance of video captioning. We pay
more attention to CIDEr during experiments, since only CIDEr weights the
n-grams that relevant to the video content, which can better reflect the
capability on producing novel expressions. Moreover, we introduce metrics in
information retrieval, including Recall at K (R@K), Median Rank (MedR), and
Mean Rank (MnR), to measure the performance of the video-text retrieval. R@K
measures the proportion of correct targets retrieved from $K$ samples. MedR
and MnR represent the median and average rank of correct targets in the
retrieved ranking list separately.

Implementation Details. For the extraction of visual features, we use C3D pre-
trained on Kinetics-400 and InceptionResNetV2 pre-trained on ImageNet to
extract motion and appearance features, respectively. We also conduct extra
experiments for VATEX with other features, , the I3D motion features pre-
trained on Kinetics-600 and ResNet152 appearance features pre-trained on
ImageNet. All the features above are extracted from 28 key-frames/segments of
video sampling at equal intervals. For the textual embedding, the sentences
longer than 40 words are truncated and initialized the word embedding with
GloVe by spaCy toolkit.

For the setting of retriever, the joint embedding size of Bi-encoders is 1024.
We set the margin $\Delta$ as 0.2, the batch size as 128 and the learning rate
as $2e^{-4}$. The retriever converges in around 10 epochs, and the best model
is selected from the best results on the validation. Note that during the
training and testing phase of RCG, the sentences are retrieved only from the
corpus of training set. Furthermore, the sentences belonging to their own
video should be excluded. Otherwise, the answer will be leaked, and the
training will be destroyed.

For the setting of generator, we keep the features in sync with the retriever.
The hidden size of the hierarchical-LSTMs is 1024, and the state size of all
the attention modules is 512. The model is optimized by Adam. The learning
rate is initialized with $2e^{-4}$ and decayed 0.5 times every 3 epochs. The
batch size is set to 64, and the training of the model can be converged with
no more than 20 epochs. During the validation, the beam-search (beam size is
3) is used for a better generation.

Table 1: Performance of video-text retrieval and video captioning tasks using
fixed retrievers trained by different features.

# |  Methods | Video-Text Retrieval | Video Captioning
---|---|---|---
R@1 | R@5 | MedR $\downarrow$ | MnR $\downarrow$ | C | B-4 | R | M
1 | w/o Retriever | - | - | - | - | 49.2 | 31.3 | 48.5 | 21.9
2 | Random | 0.1 | 0.1 | 7002 | 10328 | 49.1 | 31.0 | 48.6 | 21.7
3 | ResNet | 17.2 | 37.7 | 11 | 126.8 | 51.5 | 32.2 | 49.4 | 23.1
4 | I3D | 24.9 | 51.0 | 5 | 48.8 | 54.7 | 33.2 | 49.7 | 23.3
5 | ResNet+I3D | 29.4 | 56.9 | 4 | 36.5 | 56.8 | 33.4 | 50.1 | 23.6

Table 2: Performance of training the model with different numbers of retrieved
sentences. The model is tested via top-10 sentences. Fixed denotes whether the
retriever is fixed or jointly trained.

# |  | # Retrievals
---
Training
Fixed | CIDEr | BLEU-4 | Rouge-L | Meteor
1 | 1 | ✓ | 55.3 | 33.4 | 50.0 | 23.5
2 | $\times$ | 56.3 | 33.4 | 50.0 | 23.6
3 | 3 | ✓ | 56.8 | 33.4 | 50.1 | 23.6
4 | $\times$ | 57.5 | 33.9 | 50.2 | 23.7
5 | 5 | ✓ | 56.3 | 33.8 | 50.1 | 23.6
6 | $\times$ | 56.8 | 33.9 | 50.2 | 23.6
7 | 10 | ✓ | 56.4 | 33.6 | 50.0 | 23.5
8 | $\times$ | 57.1 | 33.7 | 50.1 | 23.6

###  4.2 Quantitative Analysis

The core of proposed open-book video captioning is to assist the caption
generation by introducing the cross-modal retrieval. We elaborate the
following Q&As to better illustrate the impact open-book captioning and prove
the effectiveness of our model. Unless otherwise specified, all the
experiments are carried out on VATEX.

Does the performance of the retriever affect the results? We leverage
different features to train the retrievers to simulate retrievers with
different capabilities. We report the performance of the video-text retrieval
and corresponding generation using different retrievers in Tab.1. In lines 3
to 5. we see that the retriever with better performance can significantly
improve the generation. An intuitive explanation is that a good retriever can
find sentences closer to the video content and provide better expressions.
Moreover, we find the results are similar between the model without retriever
in line 1 and the model with a randomly initialized retriever as the worst
retriever in line 2. In the worst case, the generator will not rely on the
retrieved sentences reflecting the robustness of our model.

Does the number of retrieved sentences affect the results? We analyze the
effect of using different numbers of retrieved sentences in training and
testing phases. In training phase, we explore $1\sim 10$ sentences for
training, and 10 sentences are used for testing. As illustrated in Tab.2, we
find that a moderate number of retrieved sentences (3 for VATEX) are helpful
for generation during training. This is because the retrieved sentence does
not exactly correspond to the video, and the noise is also introduced with
useful information. In testing phase, we select a well-trained model with
fixed retriever trained on 3 sentences and test with various $1\sim 30$
sentences retrieved from training set as hints. Compared with the results in
Tab.3 lines 4 to 6, it demonstrates that more sentences may bring more hints
for generation until saturation. In summary, too many retrieved sentences with
noise are not conducive to model training, and a trained model can adaptively
select useful cues from multiple sentences.

Figure 3: Illustration of the words copied from the retrieved sentences and
their probabilities during the generation process. Each block represents the
attention weights of the words in one retrieval sentence, and the first column
of each block denotes the probability of copying. Table 3: Performance of
testing the model with different numbers of retrieved sentences. The model is
trained via top-3 retrieved sentences.

# |  | # Retrievals
---
Testing
CIDEr | BLEU-4 | Rouge-L | Meteor
1 | 1 | 50.5 | 30.9 | 48.5 | 22.5
2 | 3 | 54.1 | 32.6 | 49.6 | 23.3
3 | 5 | 55.8 | 33.0 | 50.0 | 23.5
4 | 10 | 56.8 | 33.4 | 50.1 | 23.6
5 | 15 | 57.4 | 33.5 | 50.2 | 23.7
6 | 30 | 57.5 | 33.5 | 50.1 | 23.7

Table 4: Performance of using different qualities of the corpus for testing.
Different qualities are simulated through the corpus size.

# |  | Retrieval
---
Corpus
| Corpus
---
Size
CIDEr | BLEU-4 | Rouge-L | Meteor
1 | TrainSet | +0.1% | 39.9 | 29.4 | 47.7 | 21.9
2 | TrainSet | +1% | 48.9 | 31.4 | 49.0 | 22.9
3 | TrainSet | +10% | 56.0 | 33.2 | 49.9 | 23.5
4 | TrainSet | +100% | 56.8 | 33.4 | 50.1 | 23.6
5 | TestSet | +Oracle | 58.9 | 34.3 | 50.5 | 23.9

Does the quality of the retrieval corpus affect the results?  We conduct this
experiment by randomly selecting different proportions of sentences in
training set to simulate retrieval corpora of different quality. In Tab.4
lines 1 to 4, it illustrates large scale retrieval corpus is conducive to
producing better generation, which may be because the higher quality corpus
contains more semantically similar sentences, and it provides more hints
related to the video content for generation. Furthermore, assuming that our
retrieval corpus is good enough to contain sentences that correctly describe
the video. It can be seen in Tab.4 line 5, a significant improvement than ever
before if we combine training set and test set as the Oracle corpus for
testing. Note that the Oracle corpus is only intended to show that our model
can retrieve better sentences for generation and is not involved in the
training process.

Figure 4: The CIDEr scores of different models during training on VATEX.
FixRet and TrainRet denote the fixed retriever and jointly trained retriever
separately. The number after Ret means the number of sentences used for
training. RetOnly means directly selecting the top-$1$ retrieved sentence as
the caption. w/o Ret is our baseline model, which follows the standard
generation model without our proposed retriever.

Which is better, fixed or jointly trained retriever? We choose CIDEr as the
metric of caption performance since it reflects the generation associated with
video content. We plot the real-time test scores of fixed retriever with
different sentences, jointly trained retriever and without retriever during
training. As shown in Fig.4, the accuracy is significantly improved, and the
model converges faster after introducing our retriever. The jointly trained
retriever is also better than fixed in the term of accuracy. The same
conclusion can be drawn from Tab.2. This may be because joint trained
retriever can increase the probability of sentences that are more helpful for
generation.

For efficiency, we profile the model speed on a server with CPU E5-2650 v4 @
2.20GHz, 128 GB memory and GTX1080Ti. All tests are performed using a single
GPU. The retrieval speed is about 254 videos/sec, which is the same for both
retrievers. The fixed retriever’s training speed is 137.6 videos/sec, which is
slightly faster than the co-trained retriever of 127.2 videos/sec by 8.2%.

Figure 5: Visualizations of the proposed RCG. The attention weights and
similarities between the video of top-3 retrieved sentences are shown. The
baseline model can generate some correct words and wrong words. Compared with
it, our RCG can generate more diverse captions by copying the expressions from
retrieved sentences. Table 5: Experiments on cross-dataset video captioning.
The model is trained on VATEX training set and tested on MSRVTT test set. The
sentences are retrieved from training corpus. Ret and Gen denote using
retriever and using generator. $*$ means the retriever is trained on MSR-VTT.

# |  | Retrieval
---
Corpus
Ret | Gen | C | B-4 | R | M
1 | - | $\times$ | ✓ | 0.159 | 0.196 | 0.463 | 0.233
2 | VATEX | ✓ | $\times$ | 0.099 | 0.082 | 0.359 | 0.185
3 | VATEX | ✓ | ✓ | 0.202 | 0.217 | 0.482 | 0.242
4 | VATEX+MSRVTT | ✓ | $\times$ | 0.123 | 0.089 | 0.363 | 0.186
5 | VATEX+MSRVTT | ✓ | ✓ | 0.209 | 0.222 | 0.485 | 0.245
6 | VATEX+MSRVTT | ✓* | ✓ | 0.241 | 0.232 | 0.495 | 0.252

How is the generalization of the model for cross-dataset videos? In practical
applications, the input video distribution is not necessarily the same as that
of training data. This puts forward requirements for the generalization of the
model. For this experiment, we pre-train models on VATEX and measure the
performances on MSRVTT that is unseen in training. As shown in Tab.5, the
performance of our RCG in line 3 is better than the baseline generation model
in line 1. The comparison to line 3,5 shows that higher quality of the
retrieval corpus leads to better performance. Furthermore, we select a
retriever trained on MSR-VTT, and the comparison to line 5,6 shows a better
retriever can further improve performance. The above experiments also show
that our RCG can be extended by changing different retriever and retrieval
corpus.

Table 6: Performance comparison with state-of-the-art methods on MSR-VTT and
VATEX testing set. +RL denotes reinforcement learning, +Audio means
introducing audio features.

Dataset | Method | Ref. | CIDEr | BLEU-4 | Rouge-L | Meteor
---|---|---|---|---|---|---
MSR- VTT | POS-CG[28] | ICCV19 | 48.7 | 42.0 | 61.6 | 28.2
POS-VCT[13] | ICCV19 | 49.1 | 42.3 | 62.8 | 29.7
SAAT[37] | CVPR20 | 49.1 | 40.5 | 60.9 | 28.2
+RL |  | 51.0 | 39.9 | 61.2 | 27.7
STG-KD[22] | CVPR20 | 47.1 | 40.5 | 60.9 | 28.3
PMI-CAP[3] | ECCV20 | 49.4 | 42.1 | - | 28.7
+Audio |  | 50.6 | 43.9 | - | 29.5
ORG-TRL[36] | CVPR20 | 50.9 | 43.6 | 62.1 | 28.8
Baseline | Ours | 49.8 | 42.2 | 61.2 | 28.2
+FixRet | Ours | 52.3 | 43.1 | 61.9 | 29.0
+TrainRet | Ours | 52.9 | 42.8 | 61.7 | 29.3
VATEX | VATEX[29] | ICCV19 | 45.6 | 28.7 | 47.2 | 21.9
ORG-TRL[36] | CVPR20 | 49.7 | 32.1 | 48.9 | 22.2
NSA[8] | CVPR20 | 57.1 | 31.0 | 49.0 | 22.7
Baseline | Ours | 49.2 | 31.3 | 48.5 | 21.9
+FixRet | Ours | 56.8 | 33.4 | 50.1 | 23.6
+TrainRet | Ours | 57.5 | 33.9 | 50.2 | 23.7

###  4.3 Comparison to State-of-the-Arts

After exploring of our proposed model, we compare it with models published on
the most recent conferences. For a fair comparison, we use the standard motion
and appearance feature extractions used by most models. Reinforcement learning
and audio features are not used. The baseline model is a standard
hierarchical-LSTMs. We list the results of the fixed retriever model and
jointly trained retriever model. For MSR-VTT, we choose top-$3/10$ retrieved
sentences for training/inference. For VATEX, top-$3/10$ sentences are selected
for training/inference. Both the pre-training of the retriever and the fine-
tuning of RCG are performed on the training set. In Tab.6, compared with the
baseline, RCG achieves remarkable improvements, which proves the effectiveness
of our method. Moreover, it outperforms ORG-TRL model even without fine-
grained object features and external knowledge, which obtains 3.9% and 15.7%
relative gains on CIDEr metric for MSR-VTT and VATEX. We have a comparable
performance for the other metrics of MSR-VTT and have achieved the best
results in VATEX. In addition, compared to MSRVTT, the improvement in VATEX is
particularly obvious. It may be because the videos in VATEX are collected by
categories and have a large number of videos with the same semantics. This is
more consistent with our motivation and practical industrial applications.

###  4.4 Qualitative Analysis

We visualize the heatmap of the words copied from the retrieved sentences and
their probabilities during the generation process, as illustrated in Fig.3.
According to the heatmap, whether the words come from the retrieved sentences
and each sentence’s contribution can be seen intuitively. In Fig.5, we also
visualize the attention weights of retrieved sentences during the process of
video-text retrieval. Most of the weights are focused on keywords, , “scuba
diver”, “ocean water” and “masks”, which proves the effectiveness of our
retriever. Comparing the baseline model, the proposed RCG can acquire more
diverse expressions, , “breathing out of their masks” benefit from the coping
mechanism, and correct the expressions from “a game of frisbee” to “playing
catch” under the guidance of retrieved sentences.

##  5 Conclusion

In this paper, we have presented the RCG for open-book video captioning. RCG
efficiently retrieves video-content-relevant sentences from text corpus
through a cross-modal retriever, jointly copies cues from multi-retrieved
sentences and generates through a copy-mechanism caption generator, and is
optimized in a separately or end-to-end manner. Adequate experiments and
superior results on two large-scale video caption datasets demonstrate the
advantages of our method. Our results suggest that it is practical to copy
knowledge not limited to retrieved sentences, , video subtitles, text from
video and text from speech , for comprehensive information acquisition and
better generation. In the future, we will further explore the content above
and use more advanced retrieval to improve model efficiency.

Acknowledgment This work is supported by the National Key R&D Plan
(No.2018AAA0102802, 2018AAA0102803,2018AAA0102800,2018YFC0823003,
2017YFB1002801), Natural Science Foundation of China (No.62036011, 61721004,
61772225, 61972397), National Natural Science Foundation of China
(No.U2033210), Key Research Program of Frontier Sciences, CAS (No.QYZDJ-SSW-
JSC040), National Natural Science Foundation of Guangdong (No.2018B030311046),
Beijing Natural Science Foundation (No.L182058)

## References

  * [1] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian.  Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning.  In CVPR 2019, pages 12487–12496, 2019.
  * [2] Jia Chen and Qin Jin.  Better captioning with sequence-level exploration.  In CVPR 2020, pages 10887–10896, 2020.
  * [3] Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang.  Learning modality interaction for temporal sentence localization and event captioning in videos.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV 2020, volume 12349 of Lecture Notes in Computer Science, pages 333–351, 2020.
  * [4] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang.  Dual encoding for zero-example video retrieval.  In CVPR 2019, pages 9346–9355, 2019.
  * [5] Ali Farhadi, Seyyed Mohammad Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David A. Forsyth.  Every picture tells a story: Generating sentences from images.  In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, ECCV 2010, volume 6314 of Lecture Notes in Computer Science, pages 15–29, 2010.
  * [6] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV 2020, volume 12349 of Lecture Notes in Computer Science, pages 214–229, 2020.
  * [7] Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li.  Search engine guided neural machine translation.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, AAAI 2018, pages 5133–5140, 2018.
  * [8] Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu.  Normalized and geometry-aware self-attention network for image captioning.  In CVPR 2020, pages 10324–10333, 2020.
  * [9] Ankush Gupta, Yashaswi Verma, and C. V. Jawahar.  Choosing linguistics over vision to describe images.  In Jörg Hoffmann and Bart Selman, editors, AAAI 2012, 2012\.
  * [10] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: retrieval-augmented language model pre-training.  CoRR, abs/2002.08909, 2020.
  * [11] Micah Hodosh, Peter Young, and Julia Hockenmaier.  Framing image description as a ranking task: Data, models and evaluation metrics (extended abstract).  In Qiang Yang and Michael J. Wooldridge, editors, IJCAI 2015, pages 4188–4192, 2015.
  * [12] Nabil Hossain, Marjan Ghazvininejad, and Luke Zettlemoyer.  Simple and effective retrieve-edit-rerank text generation.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, ACL 2020, pages 2532–2538, 2020.
  * [13] Jingyi Hou, Xinxiao Wu, Wentian Zhao, Jiebo Luo, and Yunde Jia.  Joint syntax representation learning and visual cue translation for video captioning.  In ICCV 2019, pages 8917–8926, 2019.
  * [14] Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston.  Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring.  In ICLR 2020, 2020.
  * [15] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  CoRR, abs/2004.04906, 2020.
  * [16] Polina Kuznetsova, Vicente Ordonez, Alexander C. Berg, Tamara L. Berg, and Yejin Choi.  Collective generation of natural image descriptions.  In ACL 2012, pages 359–368, 2012.
  * [17] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.  Latent retrieval for weakly supervised open domain question answering.  In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, ACL 2019, pages 6086–6096, 2019.
  * [18] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  CoRR, abs/2005.11401, 2020.
  * [19] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman.  Use what you have: Video retrieval using representations from collaborative experts.  In BMVC 2019, page 279, 2019.
  * [20] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV 2019, pages 2630–2640, 2019.
  * [21] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.  Im2text: Describing images using 1 million captioned photographs.  In John Shawe-Taylor, Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q. Weinberger, editors, NIPS 2011, pages 1143–1151, 2011.
  * [22] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles.  Spatio-temporal graph for video captioning with knowledge distillation.  In CVPR 2020, pages 10867–10876, 2020.
  * [23] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai.  Memory-attended recurrent network for video captioning.  In CVPR 2019, pages 8347–8356, 2019.
  * [24] Botian Shi, Lei Ji, Zhendong Niu, Nan Duan, Ming Zhou, and Xilin Chen.  Learning semantic concepts and temporal alignment for narrated video procedural captioning.  In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, ACM MM 2020, pages 4355–4363, 2020.
  * [25] Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, and Andrew Y. Ng.  Grounded compositional semantics for finding and describing images with sentences.  Trans. Assoc. Comput. Linguistics, 2:207–218, 2014.
  * [26] Yale Song and Mohammad Soleymani.  Polysemous visual-semantic embedding for cross-modal retrieval.  In CVPR 2019, pages 1979–1988, 2019.
  * [27] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.  Pointer networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, NIPS 2015, pages 2692–2700, 2015.
  * [28] Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, and Wei Liu.  Controllable video captioning with POS sequence guidance based on gated fusion network.  In ICCV 2019, pages 2641–2650, 2019.
  * [29] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In ICCV 2019, pages 4580–4590, 2019.
  * [30] Michael Wray, Gabriela Csurka, Diane Larlus, and Dima Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In ICCV 2019, pages 450–459, 2019.
  * [31] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  MSR-VTT: A large video description dataset for bridging video and language.  In CVPR 2016, pages 5288–5296, 2016.
  * [32] Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tür.  Deepcopy: Grounded response generation with hierarchical pointer networks.  In Satoshi Nakamura, Milica Gasic, Ingrid Zuckerman, Gabriel Skantze, Mikio Nakano, Alexandros Papangelis, Stefan Ultes, and Koichiro Yoshino, editors, SIGdial 2019, pages 122–132, 2019.
  * [33] Youngjae Yu, Jongseok Kim, and Gunhee Kim.  A joint sequence fusion model for video question answering and retrieval.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, ECCV 2018, volume 11211 of Lecture Notes in Computer Science, pages 487–503, 2018.
  * [34] Yitian Yuan, Lin Ma, Jingwen Wang, and Wenwu Zhu.  Controllable video captioning with an exemplar sentence.  In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, ACM MM 2020, pages 1085–1093, 2020.
  * [35] Junchao Zhang and Yuxin Peng.  Object-aware aggregation with bidirectional temporal graph for video captioning.  In CVPR 2019, pages 8327–8336, 2019.
  * [36] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha.  Object relational graph with teacher-recommended learning for video captioning.  In CVPR 2020, pages 13275–13285, 2020.
  * [37] Qi Zheng, Chaoyue Wang, and Dacheng Tao.  Syntax-aware action targeting for video captioning.  In CVPR 2020, pages 13093–13102, 2020.