Let $\dot{P}_{\alpha/\alpha}$ be the $P_{\alpha}$-name of the trivial forcing
$(\\{\emptyset\\},\leq,\leq^{*})$. In $V^{P_{\alpha}}$, let
$\dot{C}_{\alpha/\alpha}$ be the $\dot{P}_{\alpha/\alpha}$-name of the empty
set. Now assume that $\beta<\alpha$. Define $\dot{P}_{\alpha/\beta}$ as the
following. Let $G$ be $P_{\beta}$-generic. Define
$P_{\alpha}[G]=\dot{P}_{\alpha/\beta}[G]$ as the forcing consisting of
conditions of the form
$\operatorname{stem}(p){}^{\frown}\operatorname{top}(p)$, where

  1. (1)

$\operatorname{stem}(p)$ is of the form

$(P_{\beta^{\prime}}[G],q^{\prime}){}^{\frown}(s_{0},\langle\dot{P}_{\beta_{0}/\alpha_{0}}[G],\dot{q}_{0}){}^{\frown}(s_{n-1},\langle\dot{P}_{\dot{\beta}_{n-1}/\alpha_{n-1}}[G],\dot{q}_{n-1}\rangle),$

for some $n$ (if $n=0$, then $\operatorname{stem}(p)$ is only
$(P_{\beta^{\prime}}[G],q^{\prime})$) such that

     * •

$P_{\beta^{\prime}}[G]=\dot{P}_{\dot{\beta}^{\prime}/\beta}[G]$, and
$q^{\prime}\in P_{\beta^{\prime}}[G]$.

     * •

if $n>0$, then $\alpha_{0}<\cdots<\alpha_{n-1}$, and for $i<n$,

       * –

if $\circ(\alpha_{i})=0$, $s_{i}=\langle f_{i}\rangle$, and if
$\circ(\alpha_{i})>0$, $s_{i}=\langle f_{i},\vec{f}_{i},A_{i},F_{i}\rangle$,
where $d_{i}=\operatorname{dom}(f_{i})$ is an $\alpha_{i}$-domain, $d_{i}\in
V$.

       * –

for $\zeta\in d_{0}$,
$\Vdash_{P_{\beta^{\prime}}[G]}``f_{0}(\zeta)<\alpha_{0}"$ and if $i>0$, then
for $\zeta\in d_{i}$,
$\Vdash_{P_{\alpha_{i-1}}[G]*\dot{P}_{\dot{\beta}_{i-1}/\alpha_{i-1}}[G]}``f_{i}(\zeta)<\alpha_{i}"$.

       * –

$\Vdash_{P_{\alpha_{i}}[G]}``\alpha_{i}\leq\dot{\beta}_{i}<\alpha_{i+1}"$,
where $\alpha_{n}=\alpha$.

       * –

$\Vdash_{P_{\alpha_{i}}[G]}``\dot{q}_{i}\in\dot{P}_{\dot{\beta}_{i}/\alpha_{i}}[G]"$.

       * –

if $\circ(\alpha_{i})>0$,

         * *

$A_{i}$ is a $d_{i}$-tree with respects to $\vec{E}_{\alpha_{i}}(d_{i})$ (in
the sense of $V$).

         * *

$\vec{f}_{i}=\langle f_{i,\nu}\mid\nu\in A_{i}(\alpha_{i})\rangle$.

         * *

for each $\nu$, $\operatorname{dom}(f_{i,\nu})=d_{i}$.

         * *

for $\zeta\in d_{i}$,
$\Vdash_{P_{\nu}[G]*\dot{P}_{\dot{\beta}_{\nu}/\nu}[G]}``f_{i,\nu}(\zeta)<\alpha_{i}"$.

         * *

$\operatorname{dom}(F_{i})=A_{i}(\alpha_{i})$.

         * *

for $\nu\in A_{i}(\alpha_{i})$,
$F_{i}(\nu)=\langle\dot{P}_{\dot{\beta}_{\nu}/\nu}[G],\dot{q}\rangle$,
$\Vdash_{P_{\nu}[G]}``\nu\leq\dot{\beta}_{\nu}<\alpha_{i}"$ and
$\Vdash_{P_{\nu}[G]}``\dot{q}\in\dot{P}_{\dot{\beta}_{\nu}/\nu}[G]"$.

  2. (2)

if $\circ(\alpha)=0$, then $\operatorname{top}(p)$ is $\langle f\rangle$, and
if $\circ(\alpha)>0$, then $\operatorname{top}(p)=\langle
f,\vec{f},A,F\rangle$, where there is a common domain $d$, which is an
$\alpha$-domain (in the sense of $V$) such that

     * •

If $\circ(\alpha)=0$, then $\operatorname{dom}(f)=d$ and for $\zeta\in d$,
$\Vdash_{P_{\alpha_{n-1}}[G]*\dot{P}_{\dot{\beta}_{n-1}/\alpha_{n-1}}[G]}``f(\zeta)<\alpha"$.

     * •

Assume $\circ(\alpha)>0$. Then,

       * –

$A$ is a $d$-tree with respects to $\vec{E}_{\alpha}(d)$ (in the sense of
$V$).

       * –

$\operatorname{dom}(F)=d$ and for $\nu\in\operatorname{dom}(F)$,
$F(\nu)=\langle\dot{P}_{\dot{\beta}_{\nu}/\nu}[G],\dot{q}\rangle$ where
$\Vdash_{P_{\nu}[G]}``\nu\leq\dot{\beta}_{\nu}<\alpha$ and
$\dot{q}\in\dot{P}_{\dot{\beta}_{\nu}/\nu}[G]"$.

       * –

$\operatorname{dom}(f)=d$, $\vec{f}=\langle f_{\nu}\mid\nu\in
A(\alpha)\rangle$, and for all $\nu$, $\operatorname{dom}(f_{\nu})=d$.

       * –

for $\zeta\in d$,
$\Vdash_{P_{\alpha_{n-1}}[G]*\dot{P}_{\dot{\beta}_{n-1}/\alpha_{n-1}}[G]}``f(\zeta)<\alpha"$
and for $\nu\in A(\alpha)$,
$\Vdash_{P_{\nu}[G]*\dot{P}_{\dot{\beta}_{\nu}/\nu}[G]}``f_{\nu}(\zeta)<\alpha"$.

Back in $V$. If $\dot{p}\in\dot{P}_{\alpha/\beta}$, then by density, the
collection of $p_{0}\in P_{\beta}$ such that $p_{0}$ decides $n$,
$\alpha_{0},\cdots,\alpha_{n-1}$,
$\operatorname{dom}(f_{0}),\cdots,\operatorname{dom}(f_{n-1})$, the common
domain, $A_{i}$, $A$, $q^{\prime}$ (as the equivalent
$\dot{P}_{\dot{\beta}^{\prime}/\beta}$-name, and so on), is open dense. In
this case, we say that $p_{0}$ interprets $\dot{p}$. All in all, for such
$p_{0}$ which interprets all the relevant components of $\dot{p}$, let $p_{1}$
be such the interpretation. Assume $\circ(\beta)>0$ and $\circ(\alpha)>0$ (the
other cases are simpler) write $p_{0}$ as $r_{0}{}^{\frown}\langle
g,\vec{g},B,H\rangle$ and by the interpretation, we may write

$p_{1}=(\langle\dot{P}_{\beta^{\prime}/\beta},\dot{q}^{\prime}){}^{\frown}(s_{0},\langle\dot{P}_{\dot{\beta}_{0}/\alpha_{0}},\dot{q}_{0}\rangle)\cdots{}^{\frown}(s_{n-1},\langle\dot{P}_{\dot{\beta}_{n-1}/\alpha_{n-1}},\dot{q}_{n-1}\rangle){}^{\frown}\langle
f,\vec{f},A,F\rangle.$

There is a natural concatenation $p_{0}$ with $p_{1}$, written by
$p_{0}{}^{\frown}p_{1}$, which is

$r=r_{0}{}^{\frown}(\langle
g,\vec{g},B,H\rangle,\langle\dot{P}_{\beta^{\prime}/\beta},\dot{q}^{\prime}\rangle){}^{\frown}\cdots{}^{\frown}(s_{n-1},\langle\dot{P}_{\dot{\beta}_{n-1}/\alpha_{n-1}},\dot{q}_{n-1}\rangle){}^{\frown}\langle
f,\vec{f},A,F\rangle.$

Then $r\in P_{\alpha}$ with $r\restriction P_{\beta}=p_{0}$ exists. Denote
$r/P_{\beta}$ the term $p_{1}$. For $P_{\beta}$-names $p_{0}$ and $p_{1}$ in
$\dot{P}_{\alpha/\beta}$, we say that $p_{0}\leq p_{1}$ if there is $p\in
G^{P_{\beta}}$ such that $p$ interprets $p_{0}$ and $p_{1}$, and
$p{}^{\frown}p_{0}\leq_{\alpha}p{}^{\frown}p_{1}$. Also define
$p_{0}\leq^{*}p_{1}$ if there is $p\in G^{P_{\beta}}$ such that $p$ interprets
$p_{0}$ and $p_{1}$, and
$p{}^{\frown}p_{0}\leq^{*}_{\alpha}p{}^{\frown}p_{1}$. One can check that the
map $\phi:\\{p\in P_{\alpha}\mid p\restriction P_{\beta}$ exists$\\}\to
P_{\beta}*\dot{P}_{\alpha/\beta}$ defined by $\phi(p)=(p\restriction
P_{\beta},p/P_{\beta})$ is a dense embedding, where $p\setminus P_{\beta}$ is
the obvious component of $p$ which is in $\dot{P}_{\alpha/\beta}$. Note that
if $G$ is $P_{\beta}$-generic and $H$ is $P_{\alpha}[G]$-generic, there is a
generic $I$ for $P_{\alpha}$ such that $V[G*H]=V[I]$, where $I$ is generated
by $\\{p\mid p\restriction P_{\beta}$ exists, $p\restriction P_{\beta}\in G$
and $(p/P_{\beta})[G]\in H\\}$. Conversely, if $I$ is $P_{\alpha}$-generic and
for some $p\in I$, $p\restriction P_{\beta}$ exists, we can get $G$ which is
$P_{\beta}$-generic and $H$ which is $P_{\alpha}[G]$-generic such that
$V[G*H]=V[I]$, where $G$ is generated by $\\{p\restriction P_{\beta}\mid p\in
I$ and $p\restriction P_{\beta}$ exists$\\}$ and $H=\\{(p/P_{\beta})[G]\mid
p\in I$ and $p\restriction P_{\beta}$ exists$\\}$.

In $V^{P_{\beta}}$, let $\dot{C}_{\alpha/\beta}$ be a
$\dot{P}_{\alpha/\beta}$-name of the set described as the following. Let $G$
be $P_{\beta}$-generic. and $H$ be generic over
$P_{\alpha}[G]=\dot{P}_{\alpha/\beta}[G]$. Then let $I=G*H$ be
$P_{\alpha}$-generic. $I$ derives the set $C_{\alpha}\subseteq\alpha+1$ and
$G$ derives the set $C_{\beta}\subseteq\beta+1$. Let
$C_{\alpha/\beta}=C_{\alpha}\setminus C_{\beta}$.

The following proposition has a similar proof as some previous propositions,
for example, Proposition 6.7 and Proposition 6.16.

######  Proposition 7.14.

  * •

$\Vdash_{\beta}``(\dot{P}_{\alpha/\beta},\leq)$ is $\alpha^{++}$-c.c.”

  * •

$\Vdash_{\beta}``\dot{P}_{\alpha/\beta},\leq,\leq^{*})$ has the Prikry
property.

  * •

$\Vdash_{\beta}``(\dot{P}_{\alpha/\beta},\leq^{*})$ is $\beta^{*}$-closed”,
where $\beta^{*}$ is the least inaccessible cardinal greater than $\beta$.

We conclude that from all the analysis, Proposition 4.1 holds for $P_{\alpha}$
and all relevant quotients at $\alpha$.

##  8\. The main forcing

We are now defining our main forcing $\mathbb{P}$. The forcing
$\mathbb{P}=\cup_{\\{\alpha<\kappa\mid\alpha\text{ is
inaccessible}\\}}P_{\alpha}$. For $p$ and $p^{\prime}$ in $\mathbb{P}$, define
$p\leq p^{\prime}$ if $p\in P_{\alpha}$, $p^{\prime}\in P_{\alpha^{\prime}}$,
$\alpha\geq\alpha^{\prime}$, $p\restriction P_{\alpha^{\prime}}$ exists, and
$p\restriction P_{\alpha^{\prime}}\leq_{\alpha^{\prime}}p$. The forcing is
$\kappa^{+}$-c.c. Let $G$ be $\mathbb{P}$-generic. Then if $p\in G$ is such
that $p\restriction P_{\alpha}$ exists, then $G\restriction P_{\alpha}$ is
$P_{\alpha}$-generic. We briefly describe $\mathbb{P}/P_{\alpha}$ for
$\alpha<\kappa$ inaccessible. Recall that for $\alpha\leq\eta<\kappa$,
$\Vdash_{\alpha}``\\{p/P_{\alpha}\mid p\in P_{\eta},p\restriction
P_{\alpha}\text{ exists}\\}$ is densely embedded in $\dot{P}_{\eta/\alpha}"$.
For $\alpha<\kappa$ inaccessible, let $\mathbb{P}/P_{\alpha}$ as the
collection $\\{p/P_{\alpha}\mid p\in\mathbb{P},p\restriction P_{\alpha}\text{
exists}\\}$. Define $p_{0}\leq p_{1}$ (in $V^{P_{\alpha}})$ if there is $p\in
P_{\alpha}$ such that $p{}^{\frown}p_{0}\leq_{\mathbb{P}}p\frown p_{1}$.

######  Remark 8.1.

$V^{P_{\alpha}}$, for every $p\in\mathbb{P}/P_{\alpha}$, there is $\eta$ such
that $p\in\dot{P}_{\eta/\alpha}$.

This introduces the set $C_{\alpha}$. Let $C=\cup_{\alpha}\\{C_{\alpha}\mid
G\restriction P_{\alpha}$ is $P_{\alpha}$-generic$\\}$. Then
$C\subseteq\kappa$ is a club. The next theorem shows that the cardinal
arithmetic should be as expected.

######  Theorem 8.2.

Let $\dot{f}$ be a $\mathbb{P}$-name of a function from $\beta$ to ordinals
such that $\beta<\kappa$ and $G$ is $\mathbb{P}$-generic. Then $f\in
V[G\restriction P_{\alpha}]$ for some $\alpha<\kappa$.

###### Proof.

We show by a density argument. Let $p\in\mathbb{P}$ and $\dot{f}$ be a
$\mathbb{P}$-name of functions from $\beta$ to ordinals, where $\beta<\kappa$.
For simplicity, assume $p$ is an empty condition. Let $M\prec H_{\theta}$ for
some sufficiently large regular $\theta$, $\beta\subseteq M$,
$\dot{f},p,\mathbb{P}\in M$, $V_{M\cap\kappa}\subseteq M$, and
$\circ(M\cap\kappa)\geq\beta$ (this is possible from Assumption 2.1, item
$(4)$). Say $\alpha=M\cap\kappa$. We are going to build $p^{*}\in P_{\alpha}$
of the form $p^{*}=\langle f,\vec{f},A,F\rangle$. Let $f,\vec{f}$, and $A$ be
any objects. Fix $\gamma<\beta$ and $\nu\in A(\alpha)$ such that
$\circ(\nu)=\gamma$. Let $Y_{\nu}$ be a maximal antichain of relevant
collections in $P_{\nu}$. For each $r\in Y_{\nu}$, let $G_{r}$ be
$P_{\nu}$-generic containing $r$. Since $V_{\alpha}\subseteq M$,
$M[G]\cap\kappa=M\cap\kappa$. Find $q\in\mathbb{P}/G$ such that $q$ decides
$\dot{f}(\gamma)[G]$. By elementarity, we may find such a $q$ in $M[G]$. Then
$q\in P_{\xi}/G$ for some $\xi<\alpha$. Back in $M$, let $\dot{\xi}$ and
$\dot{q}$ be the names for such $\xi$ and $q$. Define
$F(\nu)=\langle\dot{P}_{\dot{\xi}/\nu},\dot{q}\rangle$. For $\nu$ with
$\circ(\nu)\geq\beta$, we assign $F(\nu)$ to be any value. This completes the
construction of $F$. By our design, we have that $p^{*}$ decides $\dot{f}$,
and hence, $p^{*}\Vdash_{\mathbb{P}}\dot{f}\in V^{P_{\alpha}}$.

∎

######  Corollary 8.3.

Every cardinal is preserved in $V^{\mathbb{P}}$.

######  Corollary 8.4.

For $\beta<\kappa$ the value $2^{\beta}$ is determined in
$V^{\mathbb{P}_{\alpha}}$ for some $\alpha\in(\beta,\kappa)$.

######  Corollary 8.5.

$\kappa$ is inaccessible in $V^{\mathbb{P}}$.

###### Proof.

By Theorem 8.2, if $\kappa$ is collapsed, then the witness function has to be
in $V^{P_{\alpha}}$ for some $\alpha<\kappa$, but $\kappa$ is preserved in
$P_{\alpha}$, a contradiction. The same argument shows that $\kappa$ is
regular. Finally, for every $\beta<\kappa$, the value $2^{\beta}$ must be
determined in $V^{P_{\alpha}}$ for some sufficiently large $\alpha$ because
the forcing can be factored so that the quotient forcing after the stage
$\beta$ is $\beta^{+}$-closed under the direct extension, ∎

######  Theorem 8.6.

In $V^{\mathbb{P}}$, $\kappa$ is inaccessible, there is a club
$D\subseteq\kappa$ such that for $\alpha\in D$, $2^{\alpha}=\alpha^{++}$ and
for $\alpha\not\in D$, $2^{\alpha}=\alpha^{+}$.

###### Proof.

Let $C$ be the club derived from $\mathbb{P}$ and $D=\lim(C)$. Then $D$
satisfies the theorem. ∎

##  9\. Getting different cardinal behaviors on stationary classes

Assume GCH. Let $\kappa$ be a strongly inaccessible cardinal. For each
$\gamma<\kappa$, let $f_{\gamma}:\kappa\to\kappa$. Assume that for each
$\gamma$, there is a coherent sequence of extenders $\vec{E}_{\gamma}$, on a
set $X_{\gamma}\subseteq\kappa$ and $\circ^{\gamma}:X_{\gamma}\to\kappa$ such
that

  * •

$\vec{E}_{\gamma}=\langle
E_{\gamma}(\alpha,\beta)\mid\beta<\circ^{\gamma}(\alpha)\rangle$.

  * •

each $E_{\gamma}(\alpha,\beta)$ is an $(\alpha,\alpha^{+f_{\gamma}(\alpha)})$
extender witnesses $\alpha$ being $\alpha^{+f_{\gamma}(\alpha)}$-strong.

  * •

$\circ^{\gamma}(\alpha)<\alpha$.

  * •

for $\nu<\kappa$, $\\{\alpha\mid\circ^{\gamma}(\alpha)\geq\nu\\}$ is
stationary.

Then we can proceed a similar forcing construction, except that the
corresponding Cohen part at $\alpha$ will be
$C(\alpha^{+},\alpha^{f_{\gamma}(\alpha)})$. Let
$\mathbb{P}^{\langle\vec{f}_{\gamma}\mid\gamma<\kappa\rangle}$ be the
corresponded forcing.

######  Theorem 9.1.

In the forcing $\mathbb{P}^{\langle f_{\gamma}\mid\gamma<\kappa\rangle}$, all
the cardinals are preserved, the forcing produces a club
$C\subseteq\cup_{\gamma<\kappa}X_{\gamma}$ such that for each $0<\xi<\kappa$
regular and $\gamma<\kappa$, the collection of $\alpha$ with ${\rm
cf}(\alpha)\geq\xi$ and $2^{\alpha}=\alpha^{+f_{\gamma}(\alpha)}$ is
stationary.

###### Proof Sketch.

Fix $\xi>0$ and a $\mathbb{P}$-name of a club subset of $\kappa$ $\dot{D}$.
Let $p$ be a condition, $\dot{D}$ a name of a club subset of $\kappa$. Let
$M\prec H_{\theta}$ where $\theta$ is sufficiently large,
$\dot{D},p,\mathbb{P}^{\langle f_{\beta}\mid\beta<\kappa\rangle}\in M$,
$V_{M\cap\kappa}\subseteq M$, and $\circ^{\gamma}(M\cap\kappa)\geq\xi$. Let
$\alpha=M\cap\kappa$. We are now extending $p$ to a condition whose top level
is $\alpha$. Let $p=\langle f,\vec{f},A,F\rangle\in P_{\alpha}$, where
$f,\vec{f},A$ can be any sensible components. For each $\nu\in A(\alpha)$, let
$F(\nu)$ be a condition that decides an element $\dot{\xi}$ which is the
minimum of the interpretation of $\dot{D}\setminus(\nu+1)$. By elementarity,
$\dot{\xi}$ is decided to be below $\alpha$. Then the final condition forces
that $\alpha$ is in $\dot{C}\cap\dot{D}$, and forces that
$2^{\alpha}=\alpha^{f_{\gamma}(\alpha)}$, and ${\rm cf}(\alpha)\geq\xi$. ∎

######  Example 9.2.

Start from GCH, $\kappa$ carrying a $(\kappa,\kappa^{+\kappa})$-extender. Then
it is possible that for $\gamma<\kappa$, there is a sequence coherent sequence
of extenders $\vec{E}_{\gamma}$ on a stationary set
$X_{\gamma}\subseteq\kappa$ where each $E_{\gamma}(\alpha,\beta)$ witnesses
$\alpha$ being $\alpha^{+\gamma}$-strong. Let $f_{\gamma}$ be a constant
function with value $\gamma$. Then the forcing $\mathbb{P}^{\langle
f_{\gamma}\mid\gamma<\kappa\rangle}$ forces that $\kappa$ is inaccessible, and
in $V_{\kappa}$ and each $\gamma<\kappa$, there is a stationary class
$S_{\gamma}\subseteq\kappa$ such that for $\alpha\in S_{\gamma}$,
$2^{\alpha}=\alpha^{+\gamma}$. Also, in this situation, for each $\gamma$ and
$\xi$, the collection of $\alpha$ such that $\circ^{\gamma}(\alpha)=\xi$ is
stationary. A similar proof as in Theorem 9.1 shows that in the final model,
for every $\gamma<\kappa$ and $\xi<\kappa$ regular, there is a stationary set
of $\alpha$ such that $2^{\alpha}=\alpha^{+\gamma}$ and ${\rm
cf}(\alpha)=\xi$.

## References

  * [1] M. Gitik and S. Jirattikansakul, “Another method to add a closed unbounded set of former regulars,” submitted, 2022.  arXiv:2206.05693.
  * [2] M. Gitik and C. Merimovich, “Power function on stationary classes,” Annals of Pure and Applied Logic, vol. 140, no. 1, pp. 75–103, 2006.  Cardinal Arithmetic at work: the 8th Midrasha Mathematicae Workshop.
  * [3] C. Merimovich, “Extender-based magidor-radin forcing,” Israel Journal of Mathematics, vol. 182, pp. 439–480, 2011.
  * [4] S. Jirattikansakul, “Blowing up the power of a singular cardinal of uncountable cofinality with collapses,” Annals of Pure and Applied Logic, vol. 174, no. 6, 2023.


needs and values before or after the deployment?

  * •

How were you involved in the evaluation process?

  * •

How did community members collaborate and resolve disagreements during the
evaluation process?

  * •

Was the evaluation process effective or not? Why?

  * •

Are there forms of support that would be particularly helpful to have from
your perspective as a [participant’s role]?

  * •

Are there forms of support that would be particularly helpful to have for the
entire community?

  * •

Did the evaluation change the community’s perception and acceptance of AI
tools? How?

  * •

Have you ever participated in data labeling campaigns on Wikipedia? If so,
would you please describe your experience?

  * •

Can you envision ways to better support communities in evaluating AI tools
before they are deployed to make more informed decisions about whether or not
the community should adopt them?

###  A.2. Field Study

####  A.2.1. Exit interview questions

As described in Section 6.1.1, we conducted an exit interview with each
participant once they completed the field study to learn about their
experiences and gather feedback. Here, we provide our interview questions for
reference:

  * •

What is your overall experience using Wikibench?

  * •

What is your best and worst experience?

  * •

Does this process provide the community with agency over the curation of
evaluation datasets?

  * •

How well does this process align with Wikipedian’s norms for editing and
discussion?

  * •

How well does Wikibench support Wikipedians in discussing and resolving
disagreements?

  * •

Do you feel the primary labels are the result of community consensus?

  * •

How do you think Wikibench could be better designed to support consensus
building?

  * •

How well does Wikibench fit into Wikipedia’s interface and workflow?

  * •

How do you think Wikibench could fit better?

  * •

Were you able to get a good overview of the data curation progress using
Wikibench?

  * •

Do you feel Wikibench shows all the data and edits with transparency? Is it
good or bad?

  * •

Is there anything you would like to be able to keep track of?

  * •

How do you think we can improve Wikibench?

In addition to collecting feedback, we presented participants with a randomly
selected set of edits from Wikibench’s datasets, including their labels and
the predictions from the ORES and Revert-Risk models (mentioned in Section 3.1
and 8.2). Specifically, we presented these edits in a table, with each row
featuring an edit’s ID, a link to its entity page, its primary label in
Wikibench’s dataset, and the models’ predicted probabilities of the edit being
damaging or reverted. The table displayed a random set of ten edits at a time
and resampled each time as participants reloaded their browsers.

####  A.2.2. Exit interview thematic analysis

In Table 3, we provide a summary of the seven highest-level themes we
identified through a reflexive thematic analysis of the exit interviews. We
also list the sections in which each is discussed in the paper. Due to the
limitation of word counts, we do not include the 17 second-level themes, 64
first-level themes, and 249 codes in the table.

Table 3. The seven highest-level themes we identified through data analysis and sections where each is discussed in the paper. Highest-Level Themes |  Relevant Paper Sections
---|---
Participants perceive Wikibench as effective in surfacing disagreements and facilitating the development of a shared consensus. |  8.1.1, 8.1.2, 9.3.3
Participants appreciate Wikibench’s collaborative approach to data labeling because it allows contributors to build consensus and a stronger community. |  8.1.3, 9.3.1
Participants find data produced by Wikibench helpful in understanding gaps between different AI models’ predictions and community consensus. |  8.2
Participants perceive that Wikibench’s user interface and process fit naturally into Wikipedia’s existing interface, workflow, and community norms. |  9.1
Participants perceive that Wikibench provides them with the agency to collectively shape and reflect on the data curation process. |  9.2
Participants find Wikibench’s campaign and entity pages helpful for quickly pinpointing edits where more labels or discussions may be valuable. |  9.3.2
Participants believe the transparency Wikibench provides into the data and the process by which it is curated is essential for evaluation results to be trustworthy to the community. |  9.3.4
Table 4. Validation study participant demographics, including their self-identified experience and frequency of patrolling edits, registration year, edit count on English Wikipedia, and geographic location. A dash indicates that they chose not to provide that information. Participant ID | Patrol Experience | Patrol Frequency | Registered Since | Edit Count | Location
---|---|---|---|---|---
V1 | Years | Weekly | 2021 | 6.6k | –
V2 | Years | Monthly | 2018 | 10k | United States
V3 | Years | Daily | 2008 | 55k | United States
V4 | Years | Weekly | 2010 | 12k | United States
V5 | Months | Daily | 2022 | 20k | United Kingdom

###  A.3. Validation Study

####  A.3.1. Edit selection procedure

We selected the edits for the validation study based on a few considerations.
First, these edits should have labels from Wikilabels and Wikibench for
comparison. We achieved this by sampling existing, labeled edits from
Wikilabel’s dataset and having the field study participants provide initial
labels for these edits using Wikibench during onboarding sessions, so that
they would be added to Wikibench’s dataset. To ensure participants did not
focus on these edits more than they would otherwise, they were not told that
these edits would be used in a validation study. Secondly, due to the limited
onboarding time, we asked each participant to label only a small number of
edits, which in turn limited the total number of edits we could sample from
Wikilabels’ dataset in the first place. Finally, considering the total limit
and our goal of assessing Wikibench’s label quality resulting from
participants’ navigation of consensus, disagreement, and uncertainty, we
oversampled edits that were likely to spark discussion while also including
straightforward ones that were more likely to receive unanimous labels, as
described below. Given these three considerations, we sampled 90 edits from
Wikilabels’ datasets and had each of the 12 field study participants label a
random subset of 15 edits. This design ensured that each edit was labeled by
at least two participants, the minimum number needed to kickstart discussion.

In order to sample 90 edits from Wikilabels’
dataset383838https://labels.wmflabs.org/stats/enwiki/41 that were likely to
spark discussion, we first identified 4,407 edits with both edit damage and
user intent labels available in the dataset when we conducted the study. We
excluded 127 edits that were hidden from public view by Wikipedia
administrators and 18 edits that were labeled by more than one person due to
Wikilabels’ system race conditions. Among the remaining 4,262 edits, we
categorized edits into three categories: (1) potentially ambiguous edits, (2)
contested edits, and (3) other edits (cf. (Chen and Zhang, 2023; Gordon et
al., 2021)). We considered potentially ambiguous edits as those with the
”unsure” mark specified in Wikilabels’ dataset. We identified contested edits
as edits that had received higher-confidence labels (i.e., without the
“unsure” mark), which were different from their actual reversion outcomes on
Wikipedia (e.g., edits labeled as damaging in Wikilabels but didn’t get
reverted on Wikipedia). Given that these were cases where two Wikipedians had
historically disagreed, we expected that these edits were ones for which
Wikipedians are more likely differ in their perspectives. This categorization
led to 365 potentially ambiguous edits, 660 contested edits, and 3,237 other
edits. We sampled 30 edits from each category, resulting in 90 edits in total.

####  A.3.2. Participant demographics

The demographic information of the five additional Wikipedians we recruited
for the validation study is shown in Table 4.


# Neural Machine Translation with Phrase-Level Universal Visual
Representations

Qingkai Fang1,2, Yang Feng1,2
1 Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
2 University of Chinese Academy of Sciences, Beijing, China
<EMAIL_ADDRESS>
Corresponding author: Yang Feng.

###### Abstract

Multimodal machine translation (MMT) aims to improve neural machine
translation (NMT) with additional visual information, but most existing MMT
methods require paired input of source sentence and image, which makes them
suffer from shortage of sentence-image pairs. In this paper, we propose a
phrase-level retrieval-based method for MMT to get visual information for the
source input from existing sentence-image data sets so that MMT can break the
limitation of paired sentence-image input. Our method performs retrieval at
the phrase level and hence learns visual information from pairs of source
phrase and grounded region, which can mitigate data sparsity. Furthermore, our
method employs the conditional variational auto-encoder to learn visual
representations which can filter redundant visual information and only retain
visual information related to the phrase. Experiments show that the proposed
method significantly outperforms strong baselines on multiple MMT datasets,
especially when the textual context is limited.

††Code is publicly available at https://github.com/ictnlp/PLUVR.

##  1 Introduction

Multimodal machine translation (MMT) introduces visual information into neural
machine translation (NMT), which assumes that additional visual modality could
improve NMT by grounding the language into a visual space (Lee et al., 2018).
However, most existing MMT methods require additional input of images to
provide visual representations, which should match with the source sentence.
Unfortunately, in practice it is difficult to get this kind of pairwise input
of text and images which hinders the applications of MMT. What is worse, to
train an MMT model, the training data still involves the target sentence
besides the source sentence and the image, which is costly to collect. As a
result, the MMT model is usually trained on a small Multi30K (Elliott et al.,
2016) data set, which limits the performance of MMT. Therefore, it is
necessary to utilize the separate image data set to obtain visual
representations to break the constraints of pairwise input.

Towards this end, some researchers (Zhang et al., 2020; Wu et al., 2021)
propose to integrate a retrieval module into NMT, which retrieve images
related to the source sentence from existing sentence-image pairs as
complementary input, and then use a pre-trained convolutional neural network
(CNN) to encode the images. However, such sentence-level retrieval usually
suffers from sparsity as it is difficult to get the images that properly match
with the source sentence. Besides, visual features outputted by the CNN
contain richer information (_e.g._ , color, size, shape, texture, and
background) than the source text, thus encoding them in a bundle without any
filtering will introduce noise into the model.

To solve these problems, we propose a novel retrieval-based method for MMT to
learn phrase-level visual representations for the source sentence, which can
mitigate the aforementioned problems of sparse retrieval and redundant visual
representations. For the sparsity problem, our method retrieves the image at
the phrase level and only refers to the grounded region in the image related
with the phrase. For the redundancy problem, our method employs the
conditional variational auto-encoder to force the learned representations to
properly reconstruct the source phrase so that the learned representations
only retain the information related to the source phrase . Experiments on
Multi30K (Elliott et al., 2016) show that the proposed method gains
significant improvements over strong baselines. When the textual context is
limited, it achieves up to 85% gain over the text-only baseline on the BLEU
score. Further analysis demonstrates that the proposed method can obtain
visual information that is more related to translation quality.

##  2 Phrase-Guided Visual Representation

We use phrase-level visual representation to improve NMT. In this section, we
will introduce our proposed _phrase-guided visual representation_. We first
build a phrase-level image set, and then introduce a latent-variable model to
learn a phrase-guided visual representation for each image region.

###  2.1 Phrase-Level Image Set

Our phrase-level image set is built from the training set of Multi30K, which
contains about 29K bilingual sentence-image pairs. We only use the images
$\mathbf{e}$ and source descriptions $\mathbf{x}$ from them, which is denoted
as $\mathcal{D}=\\{(\mathbf{x}_{i},\mathbf{e}_{i})\\}_{i=1}^{N}$. We extract
<noun phrase, image region> pairs from <sentence, image> pairs in
$\mathcal{D}$ to build our phrase-level image set, which is denoted as
$\mathcal{D}_{p}$.

For each sentence $\mathbf{x}_{i}$, we use an open-source library _spaCy_
111https://spacy.io to identify the noun phrases, which is denoted as
$\mathbf{P}_{i}=(\mathbf{p}^{i}_{1},\mathbf{p}^{i}_{2},...,\mathbf{p}^{i}_{t_{i}})$,
where $t_{i}$ is the number of noun phrases in $\mathbf{x}_{i}$. For each noun
phrase $\mathbf{p}^{i}_{j}$, we detect the corresponding region
$\mathbf{r}^{i}_{j}$ from the paired image $\mathbf{e}_{i}$ using the visual
grounding toolkit (Yang et al., 2019). Then
$(\mathbf{p}^{i}_{j},\mathbf{r}^{i}_{j})$ is added to our phrase-level image
set $\mathcal{D}_{p}$. Figure 1 illustrates an example.

Finally, we obtain the phrase-level image set
$\mathcal{D}_{p}=\\{(\mathbf{p}_{i},\mathbf{r}_{i})\\}_{i=1}^{T}$, where
$T=\sum_{i=1}^{N}t_{i}$. It contains about 102K pairs in total.

###  2.2 Latent-Variable Model

For an image region $\mathbf{r}$, we can obtain the visual features
$\mathbf{v}$ with a pre-trained ResNet-101 Faster R-CNN (He et al., 2016; Ren
et al., 2015), which contains rich visual information (_e.g._ , color, size,
shape, texture, and background). However, we should not pay much attention to
the visual information not mentioned in the corresponding phrase, which will
introduce too much noise and even be harmful to NMT. Therefore, we further
introduce a continuous latent variable to explicitly model the semantic
information of image regions under the guidance of phrases. We adopt the
framework of conditional variational auto-encoder (CVAE) (Kingma and Welling,
2014; Sohn et al., 2015) to maximize the conditional marginal log-likelihood
$\log
p(\mathbf{p}|\mathbf{v})=\log\int_{z}p(\mathbf{p}|z,\mathbf{v})p(z|\mathbf{v})dz$
by maximizing the evidence lowerbound (ELBO):

$\displaystyle\mathcal{L}_{cvae}(\omega,\phi,\theta)=$
$\displaystyle\mathbb{E}_{z\sim
q_{\phi}(\mathbf{z}|\mathbf{p},\mathbf{v})}[\log
p_{\theta}(\mathbf{p}|\mathbf{z},\mathbf{v})]$ (1) $\displaystyle-{\rm
KL}[q_{\phi}(\mathbf{z}|\mathbf{p},\mathbf{v})\|p_{\omega}(\mathbf{z}|\mathbf{v})],$

where $p_{\omega}(\mathbf{z}|\mathbf{v})$ is the prior,
$q_{\phi}(\mathbf{z}|\mathbf{p},\mathbf{v})$ is an approximate posterior and
$p_{\theta}(\mathbf{p}|\mathbf{z},\mathbf{v})$ is the decoder. The prior
$p_{\omega}$ is modeled as a Gaussian distribution:

$\displaystyle
p_{\omega}(\mathbf{z}|\mathbf{v})=\mathcal{N}(\mathbf{z};\bm{\mu}_{p}(\mathbf{v}),\bm{\sigma}_{p}(\mathbf{v})^{2}\mathbf{I}),$
(2) $\displaystyle\bm{\mu}_{p}(\mathbf{v})={\rm Linear}(\mathbf{v}),$ (3)
$\displaystyle\bm{\sigma}_{p}(\mathbf{v})={\rm Linear}(\mathbf{v}),$ (4)

where ${\rm Linear}(\cdot)$ denotes linear transformation. The approximate
posterior $q_{\phi}$ is also modeled as a Gaussian distribution:

$\displaystyle
q_{\phi}(\mathbf{z}|\mathbf{p},\mathbf{v})=\mathcal{N}(\mathbf{z};\bm{\mu}_{q}(\mathbf{p},\mathbf{v}),\bm{\sigma}_{q}(\mathbf{p},\mathbf{v})^{2}\mathbf{I}),$
(5) $\displaystyle\bm{\mu}_{q}(\mathbf{p},\mathbf{v})={\rm Linear}([{\rm
RNN}(\mathbf{p}),\mathbf{v}]),$ (6)
$\displaystyle\bm{\sigma}_{q}(\mathbf{p},\mathbf{v})={\rm Linear}([{\rm
RNN}(\mathbf{p}),\mathbf{v}]),$ (7)

where ${\rm RNN}(\cdot)$ denotes a single-layer unidirectional recurrent
neural network (RNN). The final hidden state of RNN is used to compute the
mean and variance vectors.

Figure 1: Example of extracting <noun phrase, image region> pairs from
existing <sentence, image> pairs. Figure 2: Overview of our proposed method.

To be able to update the parameters using backpropagation, we use the
reparameterization trick (Kingma and Welling, 2014) to sample $\mathbf{z}$
from $q_{\phi}$:

$\mathbf{z}=\bm{\mu}_{q}+\bm{\sigma}_{q}\odot\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I}).$
(8)

The decoder $p_{\theta}(\mathbf{p}|\mathbf{z},\mathbf{v})$ is also implemented
by a single-layer unidirectional RNN. The initial hidden state of decoder RNN
is defined as:

$\mathbf{s}={\rm Linear}([\mathbf{z},\mathbf{v}]),$ (9)

and then the decoder will reconstruct the phrase $\mathbf{p}$ based on
$\mathbf{s}$. We refer to $\mathbf{s}$ as _phrase-guided visual
representation_ , since it pays more attention to the semantic information
mentioned in the phrase and filters out irrelevant information. We will
describe how to incorporate it into NMT in the next section.

##  3 NMT with Phrase-Level Universal Visual Representation

In this section, we will introduce our retrieval-based MMT method.
Specifically, we obtain visual context through our proposed phrase-level
visual retrieval, and then learn a universal visual representation for each
phrase in the source sentence, which is used to improve NMT. Figure 2 shows
the overview of our proposed method, which is composed of four modules: source
encoder, phrase-level visual retrieval module, multimodal aggregation module,
and target decoder. The source encoder and target decoder are the same as the
encoder and decoder of conventional text-only Transformer (Vaswani et al.,
2017). Therefore, we will introduce the phrase-level visual retrieval module
and multimodal aggregation module in detail in the rest of this section.

We denote the input source sentence as $\mathbf{x}=(x_{1},x_{2},...,x_{n})$,
the ground truth target sentence as
$\mathbf{y}^{*}=(y_{1}^{*},y_{2}^{*},...,y_{m}^{*})$ and the generated
translation as $\mathbf{y}=(y_{1},y_{2},...,y_{m})$. The input source sentence
$\mathbf{x}$ will be encoded with the source encoder to obtain source sentence
representation, which is denoted as
$\mathbf{H}=(\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{n})$.

###  3.1 Phrase-Level Visual Retrieval Module

To obtain the visual context of the source sentence without input paired
images, we design a phrase-level visual retrieval module. Specifically, for
the input sentence $\mathbf{x}=(x_{1},x_{2},...,x_{n})$, we identify the noun
phrases
$\mathbf{\bar{P}}=(\mathbf{\bar{p}}_{1},\mathbf{\bar{p}}_{2},...,\mathbf{\bar{p}}_{t})$
in $\mathbf{x}$. Each phrase
$\mathbf{\bar{p}}_{i}=(x_{l_{i}},x_{l_{i}+1},...,x_{l_{i}+d_{i}-1})$ is a
continuous list of tokens, where $l_{i}$ is the index of the first token and
$d_{i}$ is the length of $\mathbf{\bar{p}}_{i}$. For each noun phrase
$\mathbf{\bar{p}}_{i}$, we will retrieve several relevant <noun phrase, image
region> pairs from our phrase-level image set $\mathcal{D}_{p}$ according to
the semantic similarity between phrases, and then use the image regions as
visual context. We design a phrase encoder to compute the phrase embedding,
which is used to measure the semantic similarity between phrases.

#### Phrase Encoder

Our phrase encoder ${\rm Enc_{p}}(\cdot)$ is based on a pre-trained BERT
(Devlin et al., 2019). For a phrase $\mathbf{p}=(p_{1},p_{2},...,p_{l})$, we
first use BERT to encode it into contextual embeddings:

$\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{l}={\rm
BERT}(p_{1},p_{2},...,p_{l}),$ (10)

then the phrase embedding is the average embedding of all tokens:

${\rm Enc_{p}}(\mathbf{p})=\frac{1}{l}\sum_{i=1}^{l}\mathbf{c}_{i}.$ (11)

#### Visual Retrieval

For a given phrase $\mathbf{\bar{p}}$, we retrieve top-$K$ relevant <noun
phrase, image region> pairs from $\mathcal{D}_{p}$. For
$(\mathbf{p}_{i},\mathbf{r}_{i})\in\mathcal{D}_{p}$, the relevance score with
given phrase $\mathbf{\bar{p}}$ can be defined as the cosine distance between
their phrase embeddings:

${\rm RS}(\mathbf{\bar{p}},(\mathbf{p}_{i},\mathbf{r}_{i}))=\frac{{\rm
Enc_{p}}(\mathbf{\bar{p}})\cdot{\rm Enc_{p}}(\mathbf{p}_{i})}{\|{\rm
Enc_{p}}(\mathbf{\bar{p}})\|\|{\rm Enc_{p}}(\mathbf{p}_{i})\|},$ (12)

then we retrieve top-$K$ relevant pairs for $\mathbf{\bar{p}}$:

$\\{(\mathbf{p}_{i_{k}},\mathbf{r}_{i_{k}})\\}_{k=1}^{K}=\mathop{{\rm
top\text{-}}K}\limits_{i=1..T}({\rm
RS}(\mathbf{\bar{p}},(\mathbf{p}_{i},\mathbf{r}_{i}))).$ (13)

#### Universal Visual Representation

For every pair $(\mathbf{p}_{i_{k}},\mathbf{r}_{i_{k}})$, we can obtain the
phrase-guided visual representation $\mathbf{s}_{i_{k}}$ through our latent-
variable model as described in Section 2.2. Finally, the phrase-level
_universal visual representation_ of $\mathbf{\bar{p}}$ is defined as the
weighted sum of all $\\{\mathbf{s}_{i_{k}}\\}$:

$\mathbf{u}=\frac{1}{K}\sum_{k=1}^{K}{\rm
RS}(\mathbf{\bar{p}},(\mathbf{p}_{i_{k}},\mathbf{r}_{i_{k}}))\cdot\mathbf{s}_{i_{k}}.$
(14)

Our universal visual representation considers multi-view visual information
from several image regions, which avoids the bias caused by a single image
region. Finally, for all phrases
$\mathbf{\bar{P}}=(\mathbf{\bar{p}}_{1},\mathbf{\bar{p}}_{2},...,\mathbf{\bar{p}}_{t})$
in $\mathbf{x}$, we obtain the corresponding universal visual representation
$\mathbf{U}=(\mathbf{u}_{1},\mathbf{u}_{2},...,\mathbf{u}_{t})$.

###  3.2 Multimodal Aggregation Module

Inspired by the recent success of modality fusion in multimodal machine
translation (Yin et al., 2020; Zhang et al., 2020; Fang et al., 2022), we
design a simple _multimodal aggregation module_ to fuse the source sentence
representation $\mathbf{H}$ and phrase-level universal visual representation
$\mathbf{U}$. At first, we perform a phrase-level aggregation. For each phrase
$\mathbf{\bar{p}}_{i}=(x_{l_{i}},x_{l_{i}+1},...,x_{l_{i}+d_{i}-1})$, we fuse
the universal visual representation $\mathbf{u}_{i}$ and the textual
representation of corresponding tokens
$(\mathbf{h}_{l_{i}},\mathbf{h}_{l_{i}+1},...,\mathbf{h}_{l_{i}+d_{i}-1})$:

$\displaystyle\mathbf{m}_{i}={\rm
LayerNorm}(\mathbf{u}_{i}+\sum_{j=l_{i}}^{l_{i}+d_{i}-1}\mathbf{o}_{ij}\odot\mathbf{h}_{j}),$
(15) $\displaystyle\mathbf{o}_{ij}={\rm
sigmoid}(\mathbf{W}_{1}\mathbf{u}_{i}+\mathbf{W}_{2}\mathbf{h}_{j}),$ (16)

where $\odot$ denotes element-wise product. Now we obtain the multimodal
phrase representation
$\mathbf{M}=(\mathbf{m}_{1},\mathbf{m}_{2},...,\mathbf{m}_{t})$. Afterwards,
we apply a multi-head attention mechanism to append $\mathbf{M}$ to the source
sentence representation:

$\mathbf{\bar{S}}={\rm MultiHead}(\mathbf{H},\mathbf{M},\mathbf{M}).$ (17)

We then fuse $\mathbf{\bar{S}}$ and $\mathbf{H}$ with a gate mechanism:

$\displaystyle\mathbf{S}=\mathbf{H}+\bm{\lambda}\odot\mathbf{\bar{S}},$ (18)
$\displaystyle\bm{\lambda}={\rm
sigmoid}(\mathbf{W}_{3}\mathbf{H}+\mathbf{W}_{4}\mathbf{\bar{S}}).$ (19)

Finally, $\mathbf{S}$ is fed into our target decoder for predicting the
translation. The translation model is trained with a cross-entropy loss:

$\mathcal{L}_{trans}=-\sum_{i=1}^{m}\log
p(y_{i}^{*}|\mathbf{y}_{<i},\mathbf{x}).$ (20)

##  4 Experiments

Models | EN-DE | EN-FR
---|---|---
Test2016 | Test2017 | MSCOCO | Test2016 | Test2017 | MSCOCO
Transformer (Vaswani et al., 2017) | 39.87 | 31.78 | 29.36 | 60.51 | 52.44 | 42.49
Imagination (Elliott and Kádár, 2017) | 39.70-0.17 | 32.15+0.37 | 29.76+0.40 | 60.88+0.37 | 52.89+0.45 | 42.87+0.38
UVR-NMT (Zhang et al., 2020) | 38.19-1.68 | 31.85+0.07 | 28.55-0.81 | 60.02-0.49 | 51.50-0.94 | 43.22+0.73
Ours | 40.30+0.43 | 33.45+1.67** | 30.28+0.92 | 61.31+0.80* | 53.15+0.71* | 43.65+1.16*

Table 1: BLEU scores on Multi30K dataset. * and ** mean the improvements over
Transformer Vaswani et al. (2017) baseline is statistically significant
($p<0.05$ and $p<0.01$, respectively).

We conduct experiments on the following datasets:

#### Multi30K

Multi30K dataset contains bilingual parallel sentence pairs with image
annotations, where each image is paired with one English description and the
translations in German and French. Training, validation and test sets contain
29,000, 1,014, and 1,000 instances, respectively. We also report the results
on the WMT17 test set and the ambiguous MSCOCO test set, which contain 1,000
and 461 instances respectively.

#### WMT16 EN-DE

WMT16 EN-DE dataset contains about 4.5M sentence pairs. We choose
_newstest2013_ for validation and _newstest2014_ for test.

#### WMT16 EN-RO

WMT16 EN-RO dataset contains about 0.6M sentence pairs. We choose
_newsdev2016_ for validation and _newstest2016_ for test.

For all the above datasets, all sentences are tokenized and segmented into
subwords units using byte-pair encoding (BPE) (Sennrich et al., 2016). The
vocabulary is shared for source and target languages, with a size of 10K for
Multi30K, and 40K for WMT16 EN-DE and WMT16 EN-RO.

###  4.1 System Settings

#### Model Implementation

For the latent-variable model, the image region is encoded with a pre-trained
ResNet101 Faster-RCNN (He et al., 2016; Ren et al., 2015). Both the phrase
encoder and decoder are implemented using a single-layer unidirectional RNN
with 512 hidden states. The size of the latent variable is set to 64. The
batch size is 1024, and the learning rate is 5e-5. We train the model up to
200 epochs with Adam optimizer (Kingma and Ba, 2015). We adopt _KL cost
annealing_ and _word dropout_ tricks to alleviate the posterior collapse
problem following Bowman et al. (2016). The annealing step is set to 20000 and
the word dropout is set to 0.1. Note that the phrases are segmented using the
same BPE vocabulary as that for each source language.

For the translation model, we use Transformer (Vaswani et al., 2017) as our baseline. Both encoder and decoder contain 6 layers. The number of attention heads is set to 4. The dropout is set to 0.3, and the value of label smoothing is set to 0.1. For the visual retrieval module, we retrieve top-5 image regions for each phrase. We use Adam optimizer (Kingma and Ba, 2015) to tune the parameters. The learning rate is varied under a warm-up strategy with 2,000 steps. We train the model up to 8,000, 20,000, and 250,000 steps for Multi30K, WMT16 EN-RO, and WMT16 EN-DE, respectively. We average the checkpoints of last 5 epochs for evaluation. We use beam search with a beam size of 4. Different from previous work, we use sacreBLEU222https://github.com/mjpost/sacrebleu (Post, 2018) to compute the BLEU (Papineni et al., 2002) scores and the statistical significance of translation results with paired bootstrap resampling (Koehn, 2004) for future standard comparison across papers. Specifically, we measure case-insensitive detokenized BLEU for Multi30K (sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 | case:lc | eff:no | tok:13a | smooth:exp | version:2.0.0)333This is because the official pre-processing script of Multi30K dataset lowercases the corpus, see https://github.com/multi30k/dataset/blob/master/scripts/task1-tokenize.sh and case-sensitive detokenized BLEU for WMT datasets (sacreBLEU signature: nrefs:1 | bs:1000 | seed:12345 | case:mixed | eff:no | tok:13a | smooth:exp | version:2.0.0).

All models are trained and evaluated using 2 RTX3090 GPUs. We implement the
translation model based on _fairseq_ 444https://github.com/pytorch/fairseq
(Ott et al., 2019). We train latent-variable model and translation model
individually.

###  4.2 Baseline Systems

Our baseline is the text-only Transformer (Vaswani et al., 2017). Besides, we
implement Imagination (Elliott and Kádár, 2017) and UVR-NMT (Zhang et al.,
2020) based on Transformer, and compare our method with them. The details of
these methods can be found in Section 6. We use the same configuration for all
baseline systems as our model.

##  5 Results and Analysis

###  5.1 Results on Multi30K Dataset

Table 1 shows the results on Multi30K. Our proposed method significantly
outperforms the Transformer (Vaswani et al., 2017) baseline, demonstrating
that our proposed phrase-level universal visual representation can be helpful
to NMT. Our method also surpass Imagination (Elliott and Kádár, 2017) and UVR-
NMT (Zhang et al., 2020). We consider it is mainly due to the following
reasons. First, our phrase-level visual retrieval can obtain strongly
correlated image regions instead of weakly correlated whole images. Second,
our phrase-level universal visual representation considers visual information
from multiple image regions and pays more attention to the semantic
information mentioned in the phrases. Last, our phrase-level aggregation
module makes it easier for the translation model to exploit the visual
information.

Figure 3: Visualization of different visual representations.

###  5.2 Effects of Latent-Variable Model

In Section 2.2, we introduce a latent-variable model to learn a phrase-guided
visual representation for each image region. To understand how it improves the
model performance compared with original visual features, we visualize the
representations by reducing the dimension with Principal Component Analysis
(PCA). Specifically, for all <noun phrase, image region> pairs in
$\mathcal{D}_{p}$, we cluster the image regions by the
head555https://en.wikipedia.org/wiki/Head_(linguistics) of noun phrases. We
select top-8 clusters according to their size, and randomly sample 1000 image
regions for each cluster. As shown in Figure 3, the original visual features
of different clusters are mixed together, indicating that they contains too
much irrelevant information. In contrast, our proposed phrase-guided visual
representations, which pay more attention to the semantic information, form
several clusters according to their heads.

Combined with our visual retrieval module, we found that as the number of
retrieved image regions $K$ increases, the BLEU score keeps decreasing when we
use original visual features, while increasing when we use our proposed
phrase-guided visual representations, which is shown in Figure 4. We believe
the decrease of BLEU score is due to the irrelevant information in original
visual features, and thus directly sum them together will introduce too much
noise. Our method filters out those irrelevant information, and multiple image
regions could avoid the bias caused by a single one, which leads to the
increase of BLEU score. However, we don’t observe further improvements when
using more image regions.

Figure 4: BLEU scores with different number of retrieved image regions $K$.
Phrase-guided visual representations achieve better performance as $K$
increases. Figure 5: Example of different levels of retrieval. We denote the
index of retrieved image (regions) in the training set of Multi30K with #id.

###  5.3 Source-Degradation Setting

We further conduct experiments under source-degradation setting, to verify the
effectiveness of our method when the source textual context is limited.
Following Wu et al. (2021), we mask the visually grounded tokens in the source
sentence, which affects around 43% of tokens in Multi30K. As shown in Table 2,
our method achieves almost 85% improvements over the text-only Transformer
baseline. It means our proposed phrase-level universal visual representation
can fill in the missing information effectively.

Models | Test2016 | Test2017 | MSCOCO
---|---|---|---
Transformer | 10.42 | 8.59 | 7.08
Ours | 19.41+8.99 | 13.67+5.08 | 12.23+5.15

Table 2: BLEU scores on Multi30K En-De under source-degradation setting.

###  5.4 Phrase-Level vs. Sentence-Level Retrieval

To prove the effectiveness of phrase-level retrieval, we implement a sentence-
level variant of our method. In this variant, we switch the latent-variable
model, retrieval module and aggregation module from phrase-level to sentence-
level. In this way, we retrieve several images as visual context to help the
translation. As shown in Table 3, the sentence-level variant _Ours-sentence_
performs worse than _Ours_ , especially in the case of source-degradation
setting. We believe it is because phrase-level retrieval can obtain more
relevant image regions as visual context, which contain less noise and can be
integrated into textual representations more precisely. In contrast, sentence-
level retrieval leads to images with much irrelevant information, and makes it
difficult for the model to capture the fine-grained semantic correspondences
between images and descriptions. To understand this difference more
intuitively, we give an example in Figure 5. As we can see, for the input
sentence, phrase-level retrieval can obtain closely related image regions for
noun phrases _a person_ and _a black car_ , while the results of sentence-
level retrieval are actually weakly related with the input sentence.

Models | Test2016 | Test2016 (Mask)
---|---|---
Transformer | 39.87 | 10.42
Ours-sentence | 40.02+0.15 | 11.52+1.10*
Ours | 40.30+0.43 | 19.41+8.99**
Table 3: BLEU scores on Multi30K En-De Test2016. (Mask) indicates source-
degradation setting. * and ** mean the improvements over Transformer Vaswani
et al. (2017) baseline is statistically significant ($p<0.05$ and $p<0.01$,
respectively).

###  5.5 Results on WMT News Datasets

Finally, we conduct experiments on WMT16 EN-DE and WMT16 EN-RO datasets. As
shown in Table 4, we observe that both Zhang et al. (2020) and our method only
achieve marginal improvements compared with text-only Transformer baseline. We
consider that there are two main reasons. On the one hand, most of tokens in
such news text are not naturally related to specific visual contents. We found
that the percentage of visual grounded tokens in the training set of WMT16 EN-
DE is only 7% (vs. 43% in Multi30K), so the contribution of visual information
is indeed limited. On the other hand, the news text is far from the
descriptive text in Multi30K. In this way, the retrieved image regions are
actually weakly correlated with the source phrase. We did some analysis to
verify our hypotheses. As described in Section 3.1, we retrieve top-$K$ pairs
for each phrase according to the relevance scores. We define the average
relevance scores (ARS) as follows:

$\text{ARS}(k)=\mathbb{E}_{\mathbf{p}\in\mathcal{D}_{\rm val}}{\rm
RS}(\mathbf{p},(\mathbf{p}_{i_{k}},\mathbf{r}_{i_{k}})),$ (21)

which means the average relevance scores for all phrases in the validation
set. As shown in Figure 6, ARS on WMT news datasets are much lower than that
on Multi30K, which proves that the gap between news text and descriptive text
does exists.

Models | EN-DE | EN-RO
---|---|---
Transformer | 26.54 | 32.67
UVR-NMT | 26.89+0.35 | 32.93+0.26
Ours | 26.97+0.43 | 33.18+0.51
Table 4: BLEU scores on WMT16 EN-DE and WMT16 EN-RO dataset. Figure 6: Average
relevance scores (ARS) during visual retrieval for all phrases in the
validation set.

##  6 Related Work

Multimodal machine translation (MMT) aims to enhance NMT (Vaswani et al.,
2017; Zhang et al., 2019; Li et al., 2021) with additional visual context.
Since the release of Multi30K (Elliott et al., 2016) dataset, researchers have
proposed many MMT methods. Early methods (Huang et al., 2016; Calixto and Liu,
2017; Caglayan et al., 2016; Calixto et al., 2016; Caglayan et al., 2017;
Libovický and Helcl, 2017; Delbrouck and Dupont, 2017b, a; Zhou et al., 2018;
Calixto et al., 2017; Helcl et al., 2018; Caglayan et al., 2018) are mainly
based on the RNN-based encoder-decoder architecture with attention (Bahdanau
et al., 2015). Recent methods based on Transformer (Vaswani et al., 2017)
achieve better performance. Yao and Wan (2020); Yin et al. (2020); Liu et al.
(2021) design multimodal encoder to fuse the textual and visual information
during encoding. Ive et al. (2019); Lin et al. (2020) enhance the decoder with
deliberation networks (Xia et al., 2017) or capsule networks (Sabour et al.,
2017) to better utilize visual information during decoding. Caglayan et al.
(2021) propose a cross-lingual visual pre-training method and fine-tuned for
MMT. It is worth noting that some of previous works (Ive et al., 2019; Lin et
al., 2020; Yin et al., 2020; Wang and Xiong, 2021; Nishihara et al., 2020;
Zhao et al., 2021) adopt regional visual information like us, which shows
effectiveness compared with global visual features. The major difference
between our method and theirs is that our method is a retrieval-based method,
which breaks the reliance on bilingual sentence-image pairs, Therefore, our
method is still applicable when the input is text only (without paired
images), which is unfortunately not available with those previous methods.

In addition to focusing on model design, Yang et al. (2020); Nishihara et al.
(2020); Wang and Xiong (2021) propose auxiliary loss to allow the model to
make better use of visual information. Caglayan et al. (2019); Wu et al.
(2021) conduct systematic analysis to probe the contribution of visual
modality. Caglayan et al. (2020); Ive et al. (2021) focus on improving
simultaneous machine translation with visual context.

All of the above methods require a specific image as input to provide visual
context, which heavily restricts their applicability. To break this
bottleneck, Hitschler et al. (2016) propose target-side image retrieval to
help the translation. Elliott and Kádár (2017) propose a multitask learning
framework Imagination to decomposes the multimodal translation into learning
translation and learning visually grounded representation. Calixto et al.
(2019) introduce a latent variable and estimate a joint distribution over
translations and images. Long et al. (2020) predict the translation with
visual representation generated by a generative adversarial network (GAN)
(Goodfellow et al., 2014). The most closely related work to our method is UVR-
NMT (Zhang et al., 2020), which breaks the reliance on bilingual sentence-
image pairs. Like some retrieval-enhanced MT (Feng et al., 2017; Gu et al.,
2017) methods, they build a topic-image lookup table from Multi30K, and then
retrieve images related to the source sentence as visual context based on the
topic words. The central differences between Zhang et al. (2020) and our
method are as follows:

  * •

First, their method depends on the weak correlation between words and images,
which leads to much noise in the retrieved images, while our approach relies
on the strong correlation between noun phrases and image regions.

  * •

Second, our phrase-level retrieval can obtain more related visual context than
their sentence-level retrieval (Section 5.4).

  * •

Last, their method directly uses visual features extracted by ResNet (He et
al., 2016), which may introduce too much noise. We adopt a latent-variable
model to filter out irrelevant information and obtain a better representation.

##  7 Conclusion

In this paper, we propose a retrieval-based MMT method, which learns a phrase-
level universal visual representation to improve NMT. Our method not only
outperforms the baseline systems and most existing MMT systems, but also
breaks the restrictions on input that hinder the development of MMT in recent
years. Experiments and analysis demonstrate the effectiveness of our proposed
method. In the future, we will explore how to apply our method to other tasks.

## Acknowledgements

We thank all the anonymous reviewers for their insightful and valuable
comments. This work was supported by National Key R&D Program of China (NO.
2017YFE0192900).

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Bowman et al. (2016) Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016.  Generating sentences from a continuous space.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 10–21, Berlin, Germany. Association for Computational Linguistics.
  * Caglayan et al. (2017) Ozan Caglayan, Walid Aransa, Adrien Bardet, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, Marc Masana, Luis Herranz, and Joost van de Weijer. 2017.  LIUM-CVC submissions for WMT17 multimodal translation task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 432–439, Copenhagen, Denmark. Association for Computational Linguistics.
  * Caglayan et al. (2018) Ozan Caglayan, Adrien Bardet, Fethi Bougares, Loïc Barrault, Kai Wang, Marc Masana, Luis Herranz, and Joost van de Weijer. 2018.  LIUM-CVC submissions for WMT18 multimodal translation task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 597–602, Belgium, Brussels. Association for Computational Linguistics.
  * Caglayan et al. (2016) Ozan Caglayan, Loïc Barrault, and Fethi Bougares. 2016.  Multimodal attention for neural machine translation.  _CoRR_ , abs/1609.03976.
  * Caglayan et al. (2020) Ozan Caglayan, Julia Ive, Veneta Haralampieva, Pranava Madhyastha, Loïc Barrault, and Lucia Specia. 2020.  Simultaneous machine translation with visual context.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2350–2361, Online. Association for Computational Linguistics.
  * Caglayan et al. (2021) Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia. 2021.  Cross-lingual visual pre-training for multimodal machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1317–1324, Online. Association for Computational Linguistics.
  * Caglayan et al. (2019) Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019.  Probing the need for visual context in multimodal machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4159–4170, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Calixto et al. (2016) Iacer Calixto, Desmond Elliott, and Stella Frank. 2016.  DCU-UvA multimodal MT system report.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 634–638, Berlin, Germany. Association for Computational Linguistics.
  * Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017.  Incorporating global visual features into attention-based neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1003, Copenhagen, Denmark. Association for Computational Linguistics.
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-attentive decoder for multi-modal neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1913–1924, Vancouver, Canada. Association for Computational Linguistics.
  * Calixto et al. (2019) Iacer Calixto, Miguel Rios, and Wilker Aziz. 2019.  Latent variable model for multi-modal translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6392–6405, Florence, Italy. Association for Computational Linguistics.
  * Delbrouck and Dupont (2017a) Jean-Benoit Delbrouck and Stéphane Dupont. 2017a.  An empirical study on the effectiveness of images in multimodal neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 910–919, Copenhagen, Denmark. Association for Computational Linguistics.
  * Delbrouck and Dupont (2017b) Jean-Benoit Delbrouck and Stéphane Dupont. 2017b.  Multimodal compact bilinear pooling for multimodal neural machine translation.  _CoRR_ , abs/1703.08084.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30K: Multilingual English-German image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74, Berlin, Germany. Association for Computational Linguistics.
  * Elliott and Kádár (2017) Desmond Elliott and Ákos Kádár. 2017.  Imagination improves multimodal translation.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 130–141, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Fang et al. (2022) Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022.  STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
  * Feng et al. (2017) Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. 2017.  Memory-augmented neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1390–1399, Copenhagen, Denmark. Association for Computational Linguistics.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Gu et al. (2017) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2017.  Search engine guided non-parametric neural machine translation.  _CoRR_ , abs/1705.07267.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Helcl et al. (2018) Jindřich Helcl, Jindřich Libovický, and Dušan Variš. 2018.  CUNI system for the WMT18 multimodal translation task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 616–623, Belgium, Brussels. Association for Computational Linguistics.
  * Hitschler et al. (2016) Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016.  Multimodal pivots for image caption translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2399–2409, Berlin, Germany. Association for Computational Linguistics.
  * Huang et al. (2016) Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016.  Attention-based multimodal neural machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 639–645, Berlin, Germany. Association for Computational Linguistics.
  * Ive et al. (2021) Julia Ive, Andy Mingren Li, Yishu Miao, Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2021.  Exploiting multimodal reinforcement learning for simultaneous machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3222–3233, Online. Association for Computational Linguistics.
  * Ive et al. (2019) Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019.  Distilling translations with visual awareness.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6525–6538, Florence, Italy. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Lee et al. (2018) Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. 2018.  Emergent translation in multi-agent communication.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Li et al. (2021) Jicheng Li, Pengzhi Gao, Xuanfu Wu, Yang Feng, Zhongjun He, Hua Wu, and Haifeng Wang. 2021.  Mixup decoding for diverse machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 312–320, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Libovický and Helcl (2017) Jindřich Libovický and Jindřich Helcl. 2017.  Attention strategies for multi-source sequence-to-sequence learning.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 196–202, Vancouver, Canada. Association for Computational Linguistics.
  * Lin et al. (2020) Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020.  Dynamic context-guided capsule network for multimodal machine translation.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , MM ’20, page 1320–1329, New York, NY, USA. Association for Computing Machinery.
  * Liu et al. (2021) Pengbo Liu, Hailong Cao, and Tiejun Zhao. 2021.  Gumbel-attention for multi-modal machine translation.  _arXiv preprint arXiv:2103.08862_.
  * Long et al. (2020) Quanyu Long, Mingxuan Wang, and Lei Li. 2020.  Generative imagination elevates machine translation.  _CoRR_ , abs/2009.09654.
  * Nishihara et al. (2020) Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya, Yutaro Omote, and Hideki Nakayama. 2020.  Supervised visual attention for multimodal neural machine translation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4304–4314, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017.  Dynamic routing between capsules.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Sohn et al. (2015) Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015.  Learning structured output representation using deep conditional generative models.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang and Xiong (2021) Dexin Wang and Deyi Xiong. 2021.  Efficient object-level visual context modeling for multimodal machine translation: Masking irrelevant objects helps grounding.  _CoRR_ , abs/2101.05208.
  * Wu et al. (2021) Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021.  Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6153–6166, Online. Association for Computational Linguistics.
  * Xia et al. (2017) Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017.  Deliberation networks: Sequence generation beyond one-pass decoding.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Yang et al. (2020) Pengcheng Yang, Boxing Chen, Pei Zhang, and Xu Sun. 2020.  Visual agreement regularized training for multi-modal machine translation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):9418–9425.
  * Yang et al. (2019) Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. 2019\.  A fast and accurate one-stage approach to visual grounding.  In _ICCV_.
  * Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020.  Multimodal transformer for multimodal machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4346–4350, Online. Association for Computational Linguistics.
  * Yin et al. (2020) Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020.  A novel graph-based multi-modal fusion encoder for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3035, Online. Association for Computational Linguistics.
  * Zhang et al. (2019) Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019.  Bridging the gap between training and inference for neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4334–4343, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020) Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020.  Neural machine translation with universal visual representation.  In _International Conference on Learning Representations_.
  * Zhao et al. (2021) Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. 2021.  Neural machine translation with semantically relevant image regions.  _red_ , 2:c3.
  * Zhou et al. (2018) Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu. 2018.  A visual attention grounding neural model for multimodal machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3643–3653, Brussels, Belgium. Association for Computational Linguistics.