# Bridging the Gap between Synthetic and Authentic Images for
Multimodal Machine Translation

Author 1  Author n
Address line
…
Address line  Wenyu Guo1,2,3, Qingkai Fang1,2, Dong Yu3, Yang Feng1,2
1Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
2University of Chinese Academy of Sciences, Beijing, China
3Beijing Language and Culture University, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{fangqingkai21b<EMAIL_ADDRESS>
Corresponding author: Yang Feng.

###### Abstract

Multimodal machine translation (MMT) simultaneously takes the source sentence
and a relevant image as input for translation. Since there is no paired image
available for the input sentence in most cases, recent studies suggest
utilizing powerful text-to-image generation models to provide image inputs.
Nevertheless, synthetic images generated by these models often follow
different distributions compared to authentic images. Consequently, using
authentic images for training and synthetic images for inference can introduce
a distribution shift, resulting in performance degradation during inference.
To tackle this challenge, in this paper, we feed synthetic and authentic
images to the MMT model, respectively. Then we minimize the gap between the
synthetic and authentic images by drawing close the input image
representations of the Transformer Encoder and the output distributions of the
Transformer Decoder. Therefore, we mitigate the distribution disparity
introduced by the synthetic images during inference, thereby freeing the
authentic images from the inference process. Experimental results show that
our approach achieves state-of-the-art performance on the Multi30K En-De and
En-Fr datasets, while remaining independent of authentic images during
inference.††Code is publicly available at https://github.com/ictnlp/SAMMT.

##  1 Introduction

Multimodal machine translation (MMT) integrates visual information into neural
machine translation (NMT) to enhance language understanding with visual
context, leading to improvement in translation quality (Li et al., 2022a; Guo
et al., 2022; Fang and Feng, 2022). However, most existing MMT models require
an associated image with the input sentence, which is difficult to satisfy at
inference and limits the usage scenarios. Hence, overcoming the dependence on
images at inference stands as a pivotal challenge in MMT.

To tackle this challenge, some researchers have proposed to employ text-to-
image generation models (Long et al., 2021; Li et al., 2022b) to generate
synthetic images to use as the associated image for the source text at
inference, while during training MMT models still use the available authentic
images as the visual context to generate translation. Despite the remarkable
capabilities of text-to-image generation models in generating highly realistic
images from textual descriptions (Ramesh et al., 2022; Rombach et al., 2022),
the synthetic images may exhibit different distribution patterns compared to
the authentic (ground truth) images. As shown in Figure 1, the synthetic
images may depict counterfactual scenes, omit information in the text, or add
information irrelevant from the text. Therefore, training with authentic
images but inferring with synthetic images causing the distribution shift in
the images, leading to a decline in translation performance.

Figure 1: An example of the synthetic and authentic images of the same source
sentence.

In this paper, we embrace the utilization of text-to-image generation models
in MMT and propose a method to bridge the gap between synthetic and authentic
images. In our method, we also introduce synthetic images during training and
feed synthetic images and authentic images to the MMT model, respectively.
Then we minimize the gap between the two images by drawing close the following
two terms of the MMT model when the two images are inputted respectively: the
input image representations to the Transformer Encoder and the output
distributions from the Transformer Decoder. Regarding the input
representations, we leverage the Optimal Transport (OT) theory to mitigate the
disparity of the representations. Kullback-Leibler (KL) divergence is utilized
to ensure the consistency of the output distributions. Consequently, we
effectively eliminate the disparities introduced by synthetic images used
during the inference process, thereby freeing the authentic images from the
inference process.

Experimental results show that our approach achieves state-of-the-art
performance on the Multi30K En-De and En-Fr datasets, even in the absence of
authentic images. Further analysis demonstrates that our approach effectively
enhances the representation and prediction consistency between synthetic and
authentic images.

##  2 Background

###  2.1 Stable Diffusion

Stable Diffusion (Rombach et al., 2022) is a text-to-image generation model
based on the latent diffusion model (LDM), enabling the generation of highly
realistic images from textual descriptions. It mainly consists of a VAE model,
a U-Net model and a CLIP text encoder.

The training of the conditional LDM involves a forward diffusion process and a
backward denoising process. During the diffusion process, the images are
firstly compressed into the latent space using the VAE encoder, followed by
the addition of Gaussian noise to the latent representations. In the denoising
process, the U-Net is iteratively employed to progressively eliminate noise
from the noisy representations. To integrate textual guidance during the
denoising process, the text description is encoded using the CLIP text encoder
and incorporated into the U-Net through the cross-attention mechanism.
Finally, the VAE decoder reconstructs images from the latent representations.
By leveraging image-text pairs, the conditional LDM can be optimized by:

$\mathcal{L}_{ldm}=\mathbb{E}_{\mathcal{E}(a),x,\epsilon\sim\mathcal{N}(0,1),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t,\tau_{\theta}(x))\|_{2}^{2}\right],$
(1)

where $x$ and $a$ represent the input text and image, respectively. $\epsilon$
denotes the Gaussian noise. $t$ and $z_{t}$ refer to the time step and the
latent representation at the $t$-th time step. $\mathcal{E}$ represents the
VAE encoder. $\epsilon_{\theta}$ and $\tau_{\theta}$ represent the U-Net and
the CLIP text encoder, respectively. In this paper, we utilize the pre-trained
Stable Diffusion model to generate images for the source sentence.

###  2.2 Multimodal Transformer

Multimodal Transformer (Yao and Wan, 2020) is a powerful architecture designed
for MMT. It replaces the self-attention layer in the Transformer (Vaswani et
al., 2017) Encoder with a _multimodal self-attention_ layer, which can learn
multimodal representations from text representations under the guidance of the
image-aware attention.

Formally, given the input text $x$ and image $a$, the textual and visual
representations can be denoted as $\textbf{H}^{x}=(h^{x}_{1},..,h^{x}_{N})$
and $\textbf{H}^{a}=(h^{a}_{1},...,h^{a}_{P})$, respectively. Here, $N$
represents the length of the source sentence, and $P$ denotes the size of
visual features. In each multimodal self-attention layer, the textual and
visual representations are concatenated together as the query vectors:

$\widetilde{\textbf{H}}=[\textbf{H}^{x};\textbf{H}^{a}W^{a}]\in\mathbb{R}^{(N+P)\times
d},$ (2)

where $W^{a}$ are learnable weights. The key and value vectors are the textual
representations $\textbf{H}^{x}$. Finally, the output of the multimodal self-
attention layer is calculated as follows:

$\textbf{c}_{i}=\sum_{j=1}^{N}\widetilde{\alpha}_{ij}(h^{x}_{j}W^{V}),$ (3)

where $\widetilde{\alpha}_{ij}$ is the weight coefficient computed by the
softmax function:

$\widetilde{\alpha}_{ij}=\text{softmax}\left(\frac{(\widetilde{h}_{i}W^{Q})(h^{x}_{j}W^{K})^{\mathrm{T}}}{\sqrt{d_{k}}}\right),$
(4)

where $W^{Q},W^{K}$, and $W^{V}$ are learnable weights, and $d_{k}$ is the
dimension of the key vector. The output of the last Multimodal Transformer
Encoder layer is fed into the Transformer Decoder to generate the target
translation. In this paper, we use the Multimodal Transformer as our base
model architecture.

##  3 Method

The original Multimodal Transformer typically requires a paired image of the
input sentence to provide visual context, which is often impractical in many
scenarios. To address this limitation, we leverage the Stable Diffusion model
during inference to generate a synthetic image based on the source sentence.
However, as synthetic images may follow different distributions compared to
the authentic (ground truth) images, using authentic images for training and
synthetic images for inference could result in a performance decline in
translation quality due to the distribution shift in the images. Next, we will
introduce our proposed approach for bridging the gap between synthetic and
authentic images.

###  3.1 Training with both Synthetic and Authentic Images

To address the issue of distribution shifts in images, we propose training the
MMT model using a combination of synthetic and authentic images. Specifically,
we enhance the training set by augmenting each sentence with an additional
synthetic image, and train the model with both synthetic and authentic images.
By incorporating synthetic images during training, we can alleviate the
inconsistency between training and inference phases.

Technically, we utilize the Stable Diffusion model to generate an image $s$
corresponding to the input text $x$. We employ the pre-trained CLIP Radford et
al. (2021) image encoder to encode images into visual embeddings.
Subsequently, we incorporate a feed-forward network (FFN) to adjust the
dimension of visual embeddings to align with the dimension of word embeddings.

$\displaystyle\textbf{H}^{a}=\text{FFN}(\text{CLIPImageEncoder}(a)),$ (5)
$\displaystyle\textbf{H}^{s}=\text{FFN}(\text{CLIPImageEncoder}(s)),$ (6)

where $\textbf{H}^{a}\in\mathbb{R}^{1\times d}$ and
$\textbf{H}^{s}\in\mathbb{R}^{1\times d}$ denote visual representations of the
authentic and synthetic images, respectively, and $d$ is the dimension of word
embeddings. Next, $\textbf{H}^{a}$ and $\textbf{H}^{s}$ are fed into the
Multimodal Transformer to perform the MMT task.

Let us denote the target sentence as $y=(y_{1},...,y_{M})$. The loss functions
for training with synthetic and authentic images can be formulated as follows:

$\displaystyle\mathcal{L}_{syn}=-\sum_{j=1}^{M}\log
p\left(y_{j}|y_{<j},x,s\right),$ (7)
$\displaystyle\mathcal{L}_{aut}=-\sum_{j=1}^{M}\log
p\left(y_{j}|y_{<j},x,a\right).$ (8)

The final training objective is as follows:

$\mathcal{L}_{trans}=\frac{1}{2}\left(\mathcal{L}_{syn}+\mathcal{L}_{aut}\right).$
(9)

Figure 2: Overview of our proposed method.

###  3.2 Consistency Training

Incorporating synthetic images into the training process helps mitigate the
distribution shift between training and inference. However, the model’s
behaviour may still differ when handling synthetic and authentic images. To
enhance the inherent consistency of the model when dealing with these two
types of images, we introduce two consistency losses during training. Optimal
transport loss is a common training object used to encourage word-image
alignment in vision-language pretraining tasks (Chen et al., 2020; Kim et al.,
2021). The Kullback–Leibler divergence loss can also be used to improve visual
representation consistency in text-guided image inpainting tasks (Zhou and
Long, 2023). Following previous works, at the encoder side, we incorporate an
optimal transport-based training objective to encourage consistency between
representations of synthetic and authentic images. At the decoder side, we
encourage consistency between the predicted target distributions based on both
types of images. Figure 2 illustrates the overview of our proposed method.

#### Representation Consistency

Intuitively, we hope the visual representations of synthetic and authentic
images to be similar. However, directly optimizing the cosine similarity or L2
distance between $\mathbf{H}^{s}$ and $\mathbf{H}^{a}$ may not be optimal, as
the visual representations of two images may exhibit different distributions
across the feature dimension. Therefore, we propose measuring the
representation similarity between synthetic and authentic images based on the
optimal transport distance.

The Optimal Transport (OT) problem is a well-known mathematical problem that
seeks to find the optimal mapping between two distributions with minimum cost.
It has been widely employed as a loss function in various machine learning
applications. Following the principles of optimal transport theory (Villani,
2009), the two probability distributions $P$ and $Q$ can be formulated as:

$\begin{split}&P=\\{\left(w_{i},m_{i}\right)\\}_{i=1}^{K},\ \ \
s.t.\sum_{i}m_{i}=1;\\\
&Q=\\{\left(\hat{w}_{j},\hat{m}_{j}\right)\\}_{j=1}^{\hat{K}},\ \ \
s.t.\sum_{j}\hat{m}_{j}=1,\end{split}$ (10)

where each data point $w_{i}$ has a probability mass $m_{i}\in[0,\infty)$.
Given a transfer cost function $c\left(w_{i},w_{j}\right)$, and using
$\mathbf{T}_{ij}$ to represent the mass transported from $w_{i}$ to
$\hat{w}_{j}$, the transport cost can be defined as:

$\begin{split}&\mathcal{D}\left(P,Q\right)=\min_{\mathbf{T}\geq
0}\sum_{i,j}\mathbf{T}_{ij}c\left(w_{i},\hat{w}_{j}\right),\\\ &s.t.\ \ \
\sum_{j=1}^{\hat{K}}\mathbf{T}_{ij}=m_{i},\forall{i}\in\\{1,...,K\\},\\\ &\ \
\ \ \ \ \ \
\sum_{i=1}^{K}\mathbf{T}_{ij}=\hat{m}_{j},\forall{j}\in\\{1,...,\hat{K}\\}.\end{split}$
(11)

We regard two visual representations
$\textbf{H}^{s}=(h^{s}_{1},...,h^{s}_{d})$ and
$\textbf{H}^{a}=(h^{a}_{1},...,h^{a}_{d})$ as two independent distributions.
Each $h^{s}_{i}$ and $h^{a}_{j}$ here is a scalar value. We formulate the
distance between $\textbf{H}^{s}$ and $\textbf{H}^{a}$ as an optimal transport
problem, given by:

$\begin{split}&\mathcal{D}\left(\mathbf{H}^{s},\mathbf{H}^{a}\right)=\min_{\mathbf{T}\geq
0}\sum_{i,j}\mathbf{T}_{ij}c\left(h^{s}_{i},h^{a}_{j}\right),\\\ &s.t.\ \ \
\sum_{j=1}^{d}\mathbf{T}_{ij}=m_{i},\forall{i}\in\\{1,...,d\\},\\\ &\ \ \ \ \
\ \ \
\sum_{i=1}^{d}\mathbf{T}_{ij}=\hat{m}_{j},\forall{j}\in\\{1,...,d\\}.\end{split}$
(12)

We use the L2 distance as the cost function $c$, and the probability mass is
defined as:

$\begin{split}&m_{i}=\frac{|h^{s}_{i}|}{\sum\nolimits_{i}|h^{s}_{i}|},\\\
&\hat{m}_{j}=\frac{|h^{a}_{j}|}{\sum\nolimits_{j}|h^{a}_{j}|}.\end{split}$
(13)

Following Kusner et al. (2015), we remove the second constraint to obtain a
lower bound of the accurate OT solution. The relaxed OT improves the training
speed without performance decline, which can be defined as:

$\begin{split}&\mathcal{D}\left(\mathbf{H}^{s},\mathbf{H}^{a}\right)=\min_{\mathbf{T}\geq
0}\sum_{i,j}\mathbf{T}_{ij}c\left(h^{s}_{i},h^{a}_{j}\right),\\\ &s.t.\ \ \
\sum_{j=1}^{d}\mathbf{T}_{ij}=m_{i},\forall{i}\in\\{1,...,d\\}.\end{split}$
(14)

The final OT loss is defined as follows:

$\mathcal{L}_{ot}=\frac{1}{2}\left(\mathcal{D}\left(\mathbf{H}^{s},\mathbf{H}^{a}\right)+\mathcal{D}\left(\mathbf{H}^{a},\mathbf{H}^{s}\right)\right),$
(15)

#### Prediction Consistency

In addition to ensuring the consistency of representations between synthetic
and authentic images, we also aim to enhance the model’s consistency in the
predicted probability distributions based on both two types of images.
Inspired by previous works on speech translation (Fang et al., 2022; Zhou et
al., 2023; Fang and Feng, 2023), we introduce a prediction consistency loss,
defined as the Kullback-Leibler (KL) divergence between the two probability
distributions:

$\mathcal{L}_{kl}=\sum_{j=1}^{M}\mathrm{KL}\left[p(y_{j}|y_{<j},x,s)\|p(y_{j}|y_{<j},x,a)\right].$
(16)

Finally, the training objective is as follows:

$\mathcal{L}=\mathcal{L}_{trans}+\lambda\mathcal{L}_{kl}+\gamma\mathcal{L}_{ot},$
(17)

where $\lambda$ and $\gamma$ are the hyperparameters that control the
contribution of the KL divergence loss and the optimal transport loss.

# | Models | En-De | En-Fr | Average
---|---|---|---|---
Test2016 | Test2017 | MSCOCO | Test2016 | Test2017 | MSCOCO
Text-only Transformer
1 | Text-only | 40.69 | 34.26 | 30.52 | 62.84 | 54.35 | 44.81 | 44.58
Previous Image-must Systems
2 |  †DCCN (Lin et al., 2020) | 39.70 | 31.00 | 26.70 | 61.20 | 54.30 | 45.40 | 43.05
3 |  †RMMT (Wu et al., 2021) | 41.45 | 32.94 | 30.01 | 62.12 | 54.39 | 44.52 | 44.24
4 |  †Doubly-ATT (Calixto et al., 2017) | 41.45 | 33.95 | 29.63 | 61.99 | 53.72 | 45.16 | 44.32
5 |  †Gated Fusion (Wu et al., 2021) | 41.96 | 33.59 | 29.04 | 61.69 | 54.85 | 44.86 | 44.33
6 | Selective Attention (Li et al., 2022a) | 41.84 | 34.32 | 30.22 | 62.24 | 54.52 | 44.82 | 44.66
7 | Noise-robust (Ye et al., 2022) | 42.56 | 35.09 | 31.09 | 63.24 | 55.48 | 46.34 | 45.63
8 | VALHALLA (Li et al., 2022b) | 42.60 | 35.10 | 30.70 | 63.10 | 56.00 | 46.40 | 45.65
Our Image-must Systems
9 |  MultiTrans (A) (Yao and Wan, 2020) | 41.46 | 34.36 | 30.07 | 62.57 | 54.92 | 44.53 | 44.65
10 |  Integrated (A) | 42.21 | 33.94 | 31.05 | 62.76 | 54.73 | 45.18 | 44.98
11 |  Ours (A) | 42.50 | 36.04 | 31.95 | 63.71 | 56.17 | 46.43 | 46.13
Previous Image-free Systems
12 |  †UVR-NMT (Zhang et al., 2020) | 40.79 | 32.16 | 29.02 | 61.00 | 53.20 | 43.71 | 43.31
13 |  †Imagination (Elliott and Kádár, 2017) | 41.31 | 32.89 | 29.90 | 61.90 | 54.07 | 44.81 | 44.15
14 | Distill (Peng et al., 2022) | 41.28 | 33.83 | 30.17 | 62.53 | 54.84 | - | -
15 | VALHALLA (Li et al., 2022b) | 42.70 | 35.10 | 30.70 | 63.10 | 56.00 | 46.50 | 45.68
Our Image-free Systems
16 |  MultiTrans (S) (Yao and Wan, 2020) | 41.46 | 34.36 | 30.07 | 62.54 | 54.82 | 44.39 | 44.61
17 |  Integrated (S) | 42.21 | 33.94 | 31.05 | 62.76 | 54.73 | 45.14 | 44.97
18 |  Ours (S) | 42.50 | 36.04 | 31.95 | 63.71 | 56.17 | 46.43 | 46.13

Table 1:  BLEU scores on Multi30K Test2016, Test2017 and MSCOCO test sets of
En-De and En-Fr tasks. "(A)" indicates using authentic images during the
inference stage, while "(S)" indicates using synthetic images. † indicates
results under the Transformer-Tiny configuration, which are quoted from Wu et
al. (2021).

##  4 Experiments

###  4.1 Dataset

We conduct experiments on the Multi30K dataset (Elliott et al., 2016).
Multi30K contains bilingual parallel sentence pairs with image annotations,
where the English description of each image is manually translated into German
(De) and French (Fr). The training and validation sets consist of 29,000 and
1,014 instances, respectively. We report the results on the Test2016, Test2017
and ambiguous MSCOCO test sets (Elliott et al., 2017), which contain 1,000,
1,000 and 461 instances respectively. We apply the byte-pair encoding (BPE)
(Sennrich et al., 2016) algorithm with 10K merge operations to segment words
into subwords, resulting in a shared vocabulary of 9,712 and 9,544 entries for
the En-De and En-Fr translation tasks, respectively.

###  4.2 System Settings

Stable Diffusion We build our own inference pipeline with diffusers (von
Platen et al., 2022). We use the pre-trained VAE and U-Net
models111https://huggingface.co/CompVis/stable-diffusion-v1-4. The pre-trained
CLIP (Radford et al., 2021) text encoder222https://huggingface.co/openai/clip-
vit-large-patch14 is used to encode the input text. The number of denoising
steps is set to 50. The seed of the generator is set to 0. The scale of
classifier-free guidance is 7.5 and the batch size is 1.

Visual Features For both synthetic and authentic images, we use the pre-
trained ViT-B/32 CLIP model333https://github.com/openai/CLIP to extract the
visual features, which have a dimension of $[1\times 512]$.

Translation Model We follow Wu et al. (2021) to conduct experiments with the
Transformer-Tiny configuration, which proves to be more effective on the small
dataset like Multi30K. The translation model consists of 4 encoder and decoder
layers. The hidden size is 128 and the filter size of FFN is 256. There are 4
heads in the multi-head self-attention module. The dropout is set to 0.3 and
the label smoothing is 0.1. Our implementation is based on the open-source
framework _fairseq_ 444https://github.com/facebookresearch/fairseq (Ott et
al., 2019). Each training batch contains 2,048 tokens and the update frequency
is set to 4. For evaluation, we average the last 10 checkpoints following
previous works. The beam size is set to 5. We measure the results with BLEU
(Papineni et al., 2002) scores for all test sets. All models are trained and
evaluated using 1 Tesla V100 GPU. We set the KL loss weight $\lambda$ to 0.5.
The OT loss weight $\gamma$ is set to 0.1 for En-De and 0.9 for En-Fr.

# | $\mathcal{L}_{kl}$ | $\mathcal{L}_{ot}$ | En-De | Average
---|---|---|---|---
Test2016 | Test2017 | MSCOCO
1 | × | × | 42.21 | 33.94 | 31.05 | 35.73
2 | × | ✓ | 41.71 | 33.57 | 29.42 | 34.90
3 | ✓ | × | 42.31 | 35.76 | 31.04 | 36.37
4 | ✓ | ✓ | 42.50 | 36.04 | 31.95 | 36.83

Table 2:  BLEU scores on three test sets with different training objectives in
the image-free setting.

###  4.3 Basline Systems

We conduct experiments in two different settings: image-must and image-free.
In the image-must setting, authentic images from the test set are used to
provide the visual context. In contrast, the authentic images are not used in
the image-free setting. We implement the following three baseline systems for
comparision:

Text-only Text-only Transformer is implemented under the Transformer-Tiny
configuration. It follows an encoder-decoder paradigm (Vaswani et al., 2017),
taking only texts as input.

MultiTrans Multimodal Transformer (Yao and Wan, 2020) trained on the original
Multi30K dataset, as described in Section 2.2.

Integrated Multimodal Transformer trained on our augmented Multi30K dataset,
where each sentence is paired with an authentic image and a synthetic image,
as described in Section 3.1.

Besides, in the image-must setting, we include DCCN (Lin et al., 2020), RMMT
(Wu et al., 2021), Doubly-ATT (Calixto et al., 2017), Gated Fusion (Wu et al.,
2021), Selective Attention (Li et al., 2022a), Noise-robust (Ye et al., 2022),
and VALHALLA (Li et al., 2022b) for comparison. In the image-free setting, we
include UVR-NMT (Zhang et al., 2020), Imagination (Elliott and Kádár, 2017),
Distill (Peng et al., 2022), and VALHALLA (Li et al., 2022b) for comparison.

###  4.4 Main Results on the Multi30K Dataset

Table 1 summarizes the results in both image-must and image-free settings.
Each model is evaluated on three test sets for two language pairs.

Firstly, our method and the majority of the baseline systems demonstrate a
substantial performance advantage over the Text-only baseline, underscoring
the importance of the visual modality.

Secondly, for the MultiTrans baseline, it is evident that the synthetic and
authentic images yield different translation results. Overall, the authentic
images outperform the synthetic images. Directly substituting authentic images
with synthetic images results in a performance decline due to the distribution
shift in images.

Thirdly, for our Integrated baseline, we incorporate synthetic images into the
training process. Experimental results show some improvements compared to the
MultiTrans baseline. The utilization of synthetic or authentic images during
inference does not significantly affect the results, indicating the
effectiveness of this approach in addressing the distribution shift between
training and inference.

Finally, our method significantly outperforms all baseline systems in both
image-must and image-free settings. It achieves state-of-the-art performance
on all test sets, while remaining independent of the authentic images. The
superior performance demonstrates the effectiveness of encouraging the
representation and prediction consistency between synthetic and authentic
images in the MMT training.

Model | Similarity
---|---
MultiTrans | 39.43%
Integrated | 86.60%
Ours (S) | 100.00%
Table 3:  The average cosine similarity between visual representations of the
synthetic and authentic images on the Test2016 test set.

##  5 Analysis

###  5.1 Effects of Training Objectives

To investigate the impact of the KL divergence (KL) loss and the optimal
transport (OT) loss, we conduct ablation experiments on the training
objectives. The results are presented in Table 2. When only applying the OT
loss (#2), we observe a performance decline. When only using the KL loss (#3),
we observe an improvement of 0.64 BLEU, highlighting the benefits of promoting
prediction consistency. When both KL loss and OT loss are utilized (#4), we
achieve a more significant improvement of 1.10 BLEU, demonstrating the
benefits of simultaneously encouraging the representation and prediction
consistency.

Model | En-De | Average
---|---|---
Test2016 | Test2017 | MSCOCO
Selective (A) | 41.64 | 33.88 | 29.92 | 35.15
Integrated (A) | 41.21 | 34.31 | 30.10 | 35.21
Ours (A) | 42.42 | 35.44 | 31.88 | 36.58
Selective (S) | 41.64 | 33.88 | 29.92 | 35.15
Integrated (S) | 41.21 | 34.31 | 30.10 | 35.21
Ours (S) | 42.42 | 35.44 | 31.88 | 36.58

Table 4:  BLEU scores of three systems built upon the Selective Attention (Li
et al., 2022a) model.

Model | En-De | Average
---|---|---
Test2016 | Test2017 | MSCOCO
Ours (S) | 42.50 | 36.04 | 31.95 | 36.83
Random | 41.96 | 35.68 | 31.20 | 36.28
Noise | 41.86 | 35.72 | 30.77 | 36.12

Table 5:  BLEU scores of our approach and other two regularization methods.

###  5.2 Representation Consistency

To investigate the effectiveness of our method in aligning the representations
of synthetic and authentic images, we measure the cosine similarity between
their respective representations ($\mathbf{H}^{s}$ and $\mathbf{H}^{a}$) on
the test set. Specifically, we measure the cosine similarity of the visual
representations after the shared-weights FFN network on Test2016 of En-De
translation task during inference. Table 3 presents the results. For the
MultiTrans baseline, a significant disparity exists between the visual
representations of synthetic and authentic images. However, when the synthetic
images are integrated into the training process (Integrated), the similarity
between synthetic and authentic images increases to 86.60%. In contrast, our
method achieves an impressive similarity of 100.00%, highlighting the
effectiveness of our approach in bridging the representation gap between
synthetic and authentic images. The optimal transport loss plays a crucial
role in reducing the representation gap.

Model | En-De | Average
---|---|---
Test2016 | Test2017 | MSCOCO
Ours (S) | 42.50 | 36.04 | 31.95 | 36.83
Cosine | 42.46 | 35.53 | 32.10 | 36.70
L2 | 42.48 | 36.00 | 31.77 | 36.75

Table 6:  BLEU scores of our approach and other two loss functions.

###  5.3 Apply Our Method to Other Model Architecture

Our approach focuses on improving the training methodology for MMT, which is
not restricted to a specific model architecture. To validate the generality of
our method, we conduct experiments using an alternative model architecture:
Selective Attention (Selective) (Li et al., 2022a), which incorporating visual
representations with a selective attention module and a gated fusion module.
We apply the optimal transport loss before the selective attention module, and
apply the KL loss in the decoder. The model settings and hyperparameters are
the same as our experiments on the Multimodal Transformer.

The results are shown in Table 4. For the Integrated baseline, integrating
synthetic images into the training process can eliminate the gap and bring
slight improvements. When we add the KL loss and OT loss during training, we
observe a 1.43 BLEU improvements compared with the Selective baseline,
demonstrating the effectiveness and generalizability of our method across
different model architectures.

###  5.4 Comparison with Regularization Methods

We employ the Stable Diffusion model to generate images from textual
descriptions as visual contexts. To verify the necessity of the text-to-image
generation model, we compare its performance with other two regularization
methods: Random and Noise. Random means shuffling the correspondences between
synthetic images and textual descriptions in the training set. Noise means
using noise vectors as the visual representations. Results in Table 5 show
that both two strategies perform worse than our approach, demonstrating that
our superior performance is not solely attributed to regularization effects
but rather to the utilization of appropriate images and semantic information.

###  5.5 Comparison with Other Loss Functions

We measure the representation similarity between synthetic and authentic
images, and minimize their distance during training. To assess the
significance of the optimal transport loss, we conduct experiments using
various loss functions. Table 6 presents the results when substituting the
optimal transport loss with the Cosine embedding loss and L2 loss functions in
order to improve representation consistency. The hyperparameters are the same
as our experiments on the optimal transport loss. The drop of results
indicates the effectiveness of the optimal transport loss in mitigating the
representation disparity.

###  5.6 Results on Different Languages and Datasets

To verify that our method can be utilized in different language pairs and
datasets, we conduct experiments on the En-Cs task of the Multi30K dataset and
the En-De translation task of the MSCTD dataset (Liang et al., 2022). The
MSCTD dataset is a multimodal chat translation dataset, including 20,240,
5,063 and 5,047 instances for the training, validation and test sets
respectively. The results are shown in Table 7. Our method still achieves
significant improvements among all the baselines on the En-Cs task and the
MSCTD dataset, showing the generality of our method among different languages
and datasets.

Model | En-Cs | Average | MSCTD
---|---|---|---
Test2016 | Test2018
Text-only | 34.01 | 29.46 | 31.73 | 21.37
MultiTrans (S) | 34.18 | 30.03 | 32.11 | 21.48
Integrated (S) | 33.50 | 29.62 | 31.56 | 21.65
Ours (S) | 35.23 | 31.19 | 33.21 | 24.40

Table 7:  BLEU scores of our approach on different languages and datasets.

###  5.7 Incongruent Decoding

We follow method (Elliott, 2018) to use an adversarial evaluation method to
test if our method is more sensitive to the visual context. We set the
congruent image’s feature to all zeros. Then we observe the value $\Delta$BLEU
by calculating the difference between the congruent data and the incongruent
one. A larger value means the model is more sensitive to the image context. As
shown in Table 8, we conduct experiment on the three test sets of the En-De
translation task and calculate the average incongruent results. Notably, our
method achieves the highest score, providing substantial evidence of its
sensitivity to visual information.

###  5.8 Case Study

Table 9 presents two translation cases in the image-free scenario involving
the four systems. In the first case, our model accurately translates the
phrase "im maul" (in the mouth) and "rennt im schnee" (run in the snow),
whereas the other systems provide incomplete translations. In the second case,
the phrases "spielt" (play) and "auf einer roten gitarre" (on a red guitar)
are correctly translated by our proposed model while other systems translate
inaccurately. To verify that our system benefits from the incorporation of
visual information, we also compute the BLEU score for each sentence in the
En-De Test2016 test set and conduct a comparison between our model and the
Text-only baseline. Among 1,000 data instances, 34% achieve higher blue scores
upon incorporating visual information, whereas 24.9% exhibit lower blue
scores. We also manually annotated the En-De Test2016 test set, enlisting a
German language expert who is certified at the professional eighth level for
the annotation task. The annotation results reveal that 35.7% of the instances
benefit from the visual features, while 30.8% show no improvement from visual
features. These results demonstrate that our model enhances translation
performance when visual information is incorporated, further confirming the
improvement in translation results achieved by aligning synthetic and
authentic images.

Model |  $\Delta$BLEU
---|---
MultiTrans(S) | 0.23
Integrated(S) | 0.43
Ours(S) | 0.56
Table 8:  Results of the Incongruent Decoding on the Multi30K En-De test sets.

|  SRC : | a black dog with a pink toy in its mouth is running in the snow
---|---|---
REF: | ein schwarzer hund mit einem rosa spielzeug im maul rennt im schnee
Text-only: | ein schwarzer hund mit einem rosa spielzeug im schnee rennt
MultiTrans (S): | ein schwarzer hund mit einem rosa spielzeug im schnee
Integrated (S): | ein schwarzer hund mit einem rosa spielzeug im schnee läuft
|  Ours (S): | ein schwarzer hund mit einem rosa spielzeug im maul rennt im schnee
|  SRC : | guitar player performs at a nightclub red guitar
REF: | gitarristin spielt in einem nachtklub auf einer roten gitarre
Text-only: | ein gitarrespieler tritt in einem nachtclub auf
MultiTrans (S): | ein gitarrespieler tritt in einem nachtclub auf
Integrated (S): | ein gitarrespieler spielt in einem nachtclub
|  Ours (S): | ein gitarrespieler spielt in einem nachtclub auf einer roten gitarre

Table 9:  Two translation cases of four systems on the En-De task in the
image-free scenario. The red and blue tokens denote error and correct
translations respectively.

##  6 Related Work

###  6.1 Multimodal Machine Translation

Multimodal Machine translation (MMT) has drawn great attention in the research
community. Existing methods mainly focus on integrating visual information
into Neural Machine Translation (NMT) (Zhang et al., 2023). Multimodal
Transformer (Yao and Wan, 2020) employs a graph perspective to fuse visual
information into the translation model. Gated and RMMT, introduced by Wu et
al. (2021), are two widely-used architectures for integrating visual and
textual modalities. Selective Attention (Li et al., 2022a) is a method that
utilizes texts to select useful visual features. Additionally, Ji et al.
(2022) proposes additional training objectives to enhance the visual awareness
of the MMT model. Yang et al. (2022) introduces cross-modal contrastive
learning to enhance low-resource NMT.

In addition to methods that rely on authentic images, there is a growing
interest in developing MMT systems in image-free scenarios due to the
difficulty of obtaining paired images during inference. The image retrieval-
based approaches (Zhang et al., 2020; Fang and Feng, 2022) utilize images
retrieved based on textual information as visual representations. The proposed
method (Li et al., 2022b) generates discrete visual representations from texts
and incorporate them in the training and inference process. Peng et al. (2022)
proposes distillation techniques to transfer knowledge from image
representations to text representations, enabling the acquisition of useful
multimodal features during inference. In this paper, we focus on using
synthetic visual representations generated from texts and aim to bridge the
gap between synthetic and authentic images.

###  6.2 Text-to-image Generation

Text-to-image generation models have made significant progress in generating
highly realistic images from textual descriptions. Early approaches (Karras et
al., 2018; Xu et al., 2018) rely on training Generative Adversarial Networks
(GANs) (Goodfellow et al., 2014) to generate images. CLIP (Radford et al.,
2021), a powerful vision-and-language model, has also been utilized to guide
the image generation process in various methods (Ramesh et al., 2022; Abdal et
al., 2022). Recently, researchers have achieved impressive results in zero-
shot text-to-image generation scenarios, where images are generated based on
textual descriptions without specific training data (Ramesh et al., 2021; Yu
et al., 2022). The adoption of diffusion-based methods (Nichol et al., 2022;
Saharia et al., 2022) has further pushed the boundaries of text-to-image
generation, resulting in high-quality image synthesis. In this work, we employ
the Stable Diffusion (Rombach et al., 2022) model, a diffusion-based approach,
as our text-to-image generation model to generate content-rich images from
texts.

##  7 Conclusion

In this paper, we strive to address the disparity between synthetic and
authentic images in multimodal machine translation. Firstly, we feed synthetic
and authentic images to the MMT model, respectively. Secondly, we minimize the
gap between the synthetic and authentic images by drawing close the input
image representations of the Transformer Encoder and the output distributions
of the Transformer Decoder through two loss functions. As a result, we
mitigate the distribution disparity introduced by the synthetic images during
inference, thereby freeing the authentic images from the inference process.
Through extensive experiments, we demonstrate the effectiveness of our
proposed method in improving the quality and coherence of the translation
results.

## Limitations

The utilization of the text-to-image generation process may introduce
additional computational overhead. Moreover, our method is currently limited
to translation tasks from English to other languages, as the pre-trained text-
to-image model lacks support for languages other than English. The application
of our method to other tasks and languages remains unexplored, and we consider
these limitations as potential areas for future research.

## References

  * Abdal et al. (2022) Rameen Abdal, Peihao Zhu, John Femiani, Niloy J. Mitra, and Peter Wonka. 2022.  Clip2stylegan: Unsupervised extraction of stylegan edit directions.  In _SIGGRAPH ’22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 - 11, 2022_ , pages 48:1–48:9. ACM.
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-attentive decoder for multi-modal neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 1913–1924. Association for Computational Linguistics.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  UNITER: universal image-text representation learning.  In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX_ , volume 12375 of _Lecture Notes in Computer Science_ , pages 104–120. Springer.
  * Elliott (2018) Desmond Elliott. 2018.  Adversarial evaluation of multimodal machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 2974–2978. Association for Computational Linguistics.
  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017_ , pages 215–233. Association for Computational Linguistics.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language, hosted by the 54th Annual Meeting of the Association for Computational Linguistics, VL@ACL 2016, August 12, Berlin, Germany_. The Association for Computer Linguistics.
  * Elliott and Kádár (2017) Desmond Elliott and Ákos Kádár. 2017.  Imagination improves multimodal translation.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers_ , pages 130–141. Asian Federation of Natural Language Processing.
  * Fang and Feng (2022) Qingkai Fang and Yang Feng. 2022.  Neural machine translation with phrase-level universal visual representations.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 5687–5698. Association for Computational Linguistics.
  * Fang and Feng (2023) Qingkai Fang and Yang Feng. 2023.  Understanding and bridging the modality gap for speech translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_.
  * Fang et al. (2022) Qingkai Fang, Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022.  Stemm: Self-learning with speech-text manifold mixup for speech translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial networks.
  * Guo et al. (2022) Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, and Zheng Cui. 2022.  LVP-M3: Language-aware visual prompt for multilingual multimodal machine translation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2862–2872, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ji et al. (2022) Baijun Ji, Tong Zhang, Yicheng Zou, Bojie Hu, and Si Shen. 2022.  Increasing visual awareness in multimodal neural machine translation from an information theoretic perspective.  _CoRR_ , abs/2210.08478.
  * Karras et al. (2018) Tero Karras, Samuli Laine, and Timo Aila. 2018.  A style-based generator architecture for generative adversarial networks.  _CoRR_ , abs/1812.04948.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 5583–5594. PMLR.
  * Kusner et al. (2015) Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015.  From word embeddings to document distances.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 957–966. JMLR.org.
  * Li et al. (2022a) Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and Jingbo Zhu. 2022a.  On vision features in multimodal machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 6327–6337. Association for Computational Linguistics.
  * Li et al. (2022b) Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogério Feris, David D. Cox, and Nuno Vasconcelos. 2022b.  VALHALLA: visual hallucination for machine translation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 5206–5216. IEEE.
  * Liang et al. (2022) Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2022.  MSCTD: A multimodal sentiment chat translation dataset.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 2601–2613. Association for Computational Linguistics.
  * Lin et al. (2020) Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020.  Dynamic context-guided capsule network for multimodal machine translation.  In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_ , pages 1320–1329. ACM.
  * Long et al. (2021) Quanyu Long, Mingxuan Wang, and Lei Li. 2021.  Generative imagination elevates machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5738–5748, Online. Association for Computational Linguistics.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 16784–16804. PMLR.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations_ , pages 48–53. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Peng et al. (2022) Ru Peng, Yawen Zeng, and Junbo Zhao. 2022.  Distill the image to nowhere: Inversion knowledge distillation for multimodal machine translation.  _CoRR_ , abs/2210.04468.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with CLIP latents.  _CoRR_ , abs/2204.06125.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8821–8831. PMLR.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 10674–10685. IEEE.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Villani (2009) Cédric Villani. 2009.  Optimal transport: old and new.  volume 338. Springer.
  * von Platen et al. (2022) Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. 2022.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers.
  * Wu et al. (2021) Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021.  Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 6153–6166. Association for Computational Linguistics.
  * Xu et al. (2018) Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 1316–1324. Computer Vision Foundation / IEEE Computer Society.
  * Yang et al. (2022) Zhe Yang, Qingkai Fang, and Yang Feng. 2022.  Low-resource neural machine translation with cross-modal alignment.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10134–10146, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020.  Multimodal transformer for multimodal machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4346–4350. Association for Computational Linguistics.
  * Ye et al. (2022) Junjie Ye, Junjun Guo, Yan Xiang, Kaiwen Tan, and Zhengtao Yu. 2022.  Noise-robust cross-modal interactive learning with text2image mask for multi-modal neural machine translation.  In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022_ , pages 5098–5108. International Committee on Computational Linguistics.
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022.  Scaling autoregressive models for content-rich text-to-image generation.  _Trans. Mach. Learn. Res._ , 2022.
  * Zhang et al. (2023) Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023.  Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models.
  * Zhang et al. (2020) Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020.  Neural machine translation with universal visual representation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhou et al. (2023) Yan Zhou, Qingkai Fang, and Yang Feng. 2023.  CMOT: Cross-modal mixup via optimal transport for speech translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7873–7887, Toronto, Canada. Association for Computational Linguistics.
  * Zhou and Long (2023) Yucheng Zhou and Guodong Long. 2023.  Improving cross-modal alignment for text-guided image inpainting.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_ , pages 3437–3448. Association for Computational Linguistics.


# Milky Way as a Neutrino Desert Revealed by IceCube Galactic Plane
Observation

Ke Fang  Department of Physics, Wisconsin IceCube Particle Astrophysics
Center, University of Wisconsin, Madison, WI, 53706  John S. Gallagher
Department of Astronomy, University of Wisconsin, Madison, WI, 53706
Department of Physics and Astronomy, Macalester College, St. Paul., MN, 55105
Francis Halzen  Department of Physics, Wisconsin IceCube Particle Astrophysics
Center, University of Wisconsin, Madison, WI, 53706

###### Abstract

The Galactic diffuse emission (GDE) is formed when cosmic rays leave the
sources where they were accelerated, diffusively propagate in the Galactic
magnetic field, and interact with the interstellar medium and interstellar
radiation field. GDE in $\gamma$-ray (GDE-$\gamma$) has been observed up to
sub-PeV energies, though its origin may be explained by either cosmic-ray
nuclei or electrons. We show that the $\gamma$-rays accompanying the high-
energy neutrinos recently observed by the IceCube Observatory from the
Galactic plane have a flux that is consistent with the GDE-$\gamma$ observed
by the Fermi-LAT and Tibet AS$\gamma$ experiments around 1 TeV and 0.5 PeV,
respectively. The consistency suggests that the diffuse $\gamma$-ray emission
above $\sim$1 TeV could be dominated by hadronuclear interactions, though
partial leptonic contribution cannot be excluded. Moreover, by comparing the
fluxes of the Galactic and extragalactic diffuse emission backgrounds, we find
that the neutrino luminosity of the Milky Way is one to two orders of
magnitude lower than the average of distant galaxies. This implies that our
Galaxy has not hosted the type of neutrino emitters that dominates the
isotropic neutrino background at least in the past few tens of kiloyears.

High-energy neutrinos have been observed from the Milky Way by the IceCube
Observatory between 0.5 TeV and multi-PeV [1, 2]. The Galactic diffuse
emission (GDE) in neutrinos (GDE-$\nu$) is identified at a $4.5\,\sigma$
significance using cascade data and templates describing the diffuse flux of
photons. Unresolved individual sources also potentially contribute to the
observed events. Below $\sim 10$ TeV, the GDE flux measured using the
$\pi^{0}$ model, based on the MeV-to-GeV $\pi^{0}$ component measured by the
Fermi Large Area Telescope (LAT) [3], and the CRINGE model [4], based on a
global fit of cosmic rays, is higher than that using the KRA models [5], which
implement radially-dependent cosmic-ray diffusion. Although the $\pi^{0}$ and
CRINGE models are slightly favored by the data, the preference is not
statistically significant [2].

The Galactic diffuse emission in $\gamma$-rays (GDE-$\gamma$) has been
measured by Fermi-LAT from 100 MeV to 1 TeV [3]. Above $\sim$1 TeV,
GDE-$\gamma$ has been observed by ground-based $\gamma$-ray experiments from
the parts of the Galactic Plane that are accessible to the detectors [6, 7].
The GDE-$\gamma$ above 100 TeV has been detected by the Tibet AS$\gamma$ [8],
HAWC [7], and LHAASO-KM2A [9] observatories. Most of the photons observed by
Tibet AS$\gamma$ with energy above 398 TeV do not point to known $\gamma$-ray
sources, suggesting that the emission could be diffuse in nature. On the other
hand, LHAASO-KM2A finds a lower GDE intensity for a similar sky region when
masking known and new sources detected by LHAASO [9]. This hints at the
postulation that the Tibet AS$\gamma$ flux could partly be contributed by
unresolved sources [10].

GDE may come from protons and nuclei when they interact with gas in the
interstellar medium (ISM). Diffuse $\gamma$-ray emission may also be produced
by the inverse Compton radiation of relativistic electrons. The fraction of
leptonic contribution to the GDE-$\gamma$ is still under debate. Significant
inverse Compton radiation has been suggested to come from the Galactic bulge
and inner Galaxy, contributing to both around a few GeV and above 10 TeV [11,
12]. A hardening in the diffuse $\gamma$-ray spectrum at 0.1-1 TeV, sometimes
referred to as the “TeV excess”, has been interpreted as a signature of
unresolved TeV halos due to electrons trapped around middle-aged pulsars [13]
or pulsar winde nebulae [14], or as a result of a progressive hardening of
cosmic-ray nuclei spectra toward the Galactic center due to effects such as
the anisotropic cosmic-ray transport [15].

Cosmic-ray protons and nuclei interact with the ISM gas and produce neutrinos
and $\gamma$-rays simultaneously via $\pi^{\pm}\rightarrow
e^{\pm}\nu_{e}(\bar{\nu}_{e})\bar{\nu}_{\mu}\nu_{\mu}$ and $\pi^{0}\rightarrow
2\gamma$. The flux of the $\gamma$-rays that accompany the IceCube Galactic
diffuse neutrinos can be estimated by $E_{\gamma}^{2}({dN_{\rm
inj}}/{dE_{\gamma}})\approx{2}/{3}E_{\nu}^{2}({dN}/{dE_{\nu}})$, with
${E_{\gamma}\approx 2\,{E_{\nu}}}$ [16]. Here ${dN_{\rm inj}}/{dE_{\gamma}}$
denotes the injected $\gamma$-ray spectrum, which could be different from the
observed $\gamma$-ray spectrum due to the pair production of $\gamma$-rays on
low-energy photons. Photons above $\sim 100$ TeV may be absorbed by the
interstellar radiation field, while the attenuation effect is negligible for
lower-energy photons [17, 18]. As shown in Figure 1, the $\gamma$-ray flux
derived from the IceCube measurement is comparable to the Fermi-LAT Galactic
interstellar emission model around 1 TeV (note that the terms “interstellar
$\gamma$-ray emission” and “Galactic diffuse emission” both refer to the
diffuse emission made by energetic cosmic rays interacting with interstellar
nucleons and photons [19]). The model is obtained by fitting various templates
of the diffuse $\gamma$-ray emission and models of resolved and unresolved
sources to the Fermi-LAT data between 100 MeV and 1 TeV [20]. The systematic
error in the effective area of Fermi-LAT Pass 8 data is estimated to be 5%
between 0.1 and 100 GeV, and 15% at 1 TeV with a linear interpolation in
logarithm of energy between 100 GeV and 1 TeV. We used the systematic error to
estimate the uncertainty of the Galactic interstellar emission spectrum. The
actual measurement error may also arise from the model uncertainties and the
separation of the isotropic emission [19], and thus be higher than the
systematics of the detector.

The flux of GDE-$\nu$ observed by IceCube is also consistent with that of the
diffuse neutrinos expected to accompany the sub-PeV diffuse $\gamma$-rays
observed by Tibet AS$\gamma$. The shaded silver region in Figure 1, from
reference [18], is an estimation of the Galactic plane emission derived from
the Tibet AS$\gamma$ measurements in two sky regions, namely, region A:
$25^{\circ}<l<100^{\circ}$ and region B: $50^{\circ}<l<200^{\circ}$, both with
$|b|<5^{\circ}$. The width of the band accounts for the uncertainties due to
the spectra and spatial distribution of cosmic rays, gas density, and infrared
emission of the ISM. The IceCube observation between 10 and $\sim$60 TeV
agrees with this Tibet-converted neutrino flux within the uncertainties.

The consistency in the GDE measurements by Fermi-LAT, Tibet AS$\gamma$, and
IceCube at various energies suggests that the GDE-$\gamma$ could be dominantly
produced by hadronic interaction above $\sim$1 TeV. However, given the
uncertainty of the GDE-$\nu$ flux associated with the analysis templates and
the potential contribution from unresolved neutrino sources, leptonic
processes may still play a role in particular between 0.1 and a few TeV.

Below we consider a GDE model under the assumption that the IceCube flux based
on the $\pi^{0}$ template represents the diffuse neutrino flux and the source
contribution is negligible. The modeling of diffuse neutrino and $\gamma$-ray
emission is impacted by several poorly known factors, including 1) cosmic-ray
spectra above the rigidity $\sim$10 TV observed at the Earth, 2) the
difference in the cosmic-ray density at a location $\vec{x}$ in the Galaxy and
that at the observer point, $n(\vec{x},E)/n(r_{\odot},E)$, which is determined
by the source distribution in the Galaxy, timescales of the sources, and the
particle diffusion in the Galactic magnetic field, and 3) the density profile
of the neutral, ionized, and dark gas. The effects of these factors could be
coupled and are not constrained by current observations, as also noted by
e.g., references [12, 21, 22, 4].

To limit the degrees of freedom of our model, as in reference [23, 17, 24] we
consider a simplified model where the spatial and spectral components of the
nucleon flux are assumed to be independent,
$\Phi(\vec{x},E)=\Phi(\vec{x}_{\odot},E)\,\left[n(\vec{x})/n(\vec{x}_{\odot})\right]$.
We obtain the ratio of cosmic-ray density at position $\vec{x}$ to that at the
solar neighborhood, $n(\vec{x})/n(\vec{x}_{\odot})$, using a numerical
simulation that propagates cosmic rays from synthetic sources in the Galactic
magnetic field. We fix the spectra of protons and helium nuclei at the solar
neighborhood, $\Phi(\vec{x}_{\odot},E)$, to the best-fit model obtained by
fitting to the cosmic-ray measurements between $\sim 10$ GeV and $\sim$10 PeV.
More details about the simulation and the calculation of the intensity of
neutrino and $\gamma$-ray emission are explained in Supplemental Material’s
Section 1 and 2, respectively.

The dashed curves in Figure 1 from our model show that the hadronic
interaction may simultaneously explain the observed $\gamma$-ray spectra
between $100$ GeV to 100 TeV and the diffuse neutrino flux of the Galactic
Plane measured using the $\pi^{0}$ template. More complicated models including
multiple components of cosmic-ray sources and diffusion regions [21, 4] may
provide better fits to the data. Future measurements of longitudinal and
latitudinal profiles of neutrino emission above 1 TeV, identification of
individual Galactic neutrino sources, and observation of GDE-$\gamma$ from the
Southern sky at TeV-PeV energies are needed to break down the degeneracy of
the model parameters.

The GDE flux reflects the emissivity of our own Galaxy in high-energy
neutrinos while the extragalactic background (EB) reveals the contribution of
powerful sources in distant galaxies. Had the local and distant sources been
similarly luminous, the GDE would be brighter than the EB due to geometry.
Figure 1 contrasts the intensities of the GDE and EB in $\gamma$-ray and
neutrinos. Notably, the all-sky averaged GDE-$\gamma$ is brighter than the
extragalactic $\gamma$-ray background (EGB) between 1 GeV and 1 TeV, whereas
the GDE-$\nu$ is fainter than the extragalactic neutrino background (ENB)
between 1 and 100 TeV. The fact suggests that the Milky Way in its current
state is not a typical source of high-energy neutrinos.

The integrated differential flux at the neutrino energy $E_{\nu}$ observed
today from galaxies extending to cosmic “high noon” ($z\sim 2-3$) where the
star formation rate peaks can be calculated as

$\displaystyle E_{\nu}^{2}\Phi_{\nu}^{\rm EG}(E_{\nu})$ $\displaystyle=$
$\displaystyle\frac{c}{4\pi}\int_{0}^{z_{\rm
max}}dz\Big{|}\frac{dt}{dz}\Big{|}\frac{1}{1+z}\int_{M_{\rm min}}^{M_{\rm
max}}dM\frac{dn}{dM}(M,z)L_{\nu}^{\rm EG}(E^{\prime}_{\nu},M,z),$ (1)

here $E_{\nu}^{\prime}=E_{\nu}(1+z)$ is the energy of a neutrino at redshift
$z$, $|dt/dz|=(H_{0}(1+z)\sqrt{\Omega_{M}(1+z)^{3}+\Omega_{\Lambda}})^{-1}$
with $H_{0}=67.4\,\rm km\,s^{-1}\,Mpc^{-1}$, $\Omega_{M}=0.315$, and assuming
a flat universe [25], and $L_{\nu}^{\rm EG}(E^{\prime}_{\nu},M,z)\equiv
E_{\nu}^{{}^{\prime}2}d\dot{N}_{\nu}/dE^{\prime}_{\nu}(M,z)$ is the neutrino
luminosity of an external galaxy. The number density $n$ of galaxies with
stellar mass $M$ at redshift z is $dn/dM(M,z)$ given by the Schechter [26]
function, ${dn}/{d\log M}=\phi^{*}\,\ln(10)(10^{\log M-\log
M^{*}})^{\alpha+1}\exp(-10^{\log M-\log M^{*}})$. The normalization
$\phi^{*}$, slope $\alpha$, and characteristic mass $M^{*}$ are found by
fitting the mass function to galaxy distribution at different redshift bins up
to $z\sim 3$ [27, 28, 29].

We assume that the neutrino emissivity of a galaxy, including contributions
from both the individual sources hosted by the galaxy and the galactic diffuse
emission, is related to the stellar mass and redshift independently:

$\displaystyle L_{\nu}^{\rm EG}(E^{\prime}_{\nu},M,z)$ $\displaystyle=$
$\displaystyle L_{\nu}^{\rm EG}(E_{\nu},M_{\rm MW},0)\left(\frac{M}{M_{\rm
MW}}\right)^{\beta}g(z)(1+z)^{2-s}.$ (2)

In the above expression, we have parameterized the dependence of the stellar
mass as a power-law with an index $\beta$ depending on source models. In an
extreme scenario where all galaxies are equally luminous, $\beta=0$. In a more
realistic scenario where the neutrino luminosity scales to the optical, near
infrared, or X-ray luminosity of galaxies, one would expect $\beta\sim 1$ [30,
31]. The $\gamma$-ray luminosities of the star-forming galaxies detected by
Fermi-LAT present a relation of $L_{\gamma}\propto L_{\rm IR}^{1.35}$ [32]. If
the neutrino luminosity is proportional to the bolometric luminosity of the
AGN, then $\beta$ may reach as high as $1.47$ [33]. Without specifying the
neutrino source types, below we float $\beta$ from 0 to 2.0.

The function $g(z)$ in equation 2 describes the source evolution over
redshift. In a uniform evolution scenario, $g(z)=1$. If the sources follow a
star-formation history, $g(z)$ can be modelled as $g(z)\propto(1+z)^{3.4}$ at
$z<1$, $(1+z)^{-0.3}$ at $1<z<4$, and $(1+z)^{-3.5}$ at $z>4$ [34]. A star-
formation model with higher redshift contributions enhances the integrated
flux by a factor of order unity. Our calculation considers both the uniform
and star-formation models.

The last term in equation 2 arises from the fact that a neutrino observed at
$E_{\nu}$ today was at the energy $E_{\nu}(1+z)$ at the source. So
$E_{\nu}^{{}^{\prime}2}dN/dE^{\prime}_{\nu}\propto(1+z)^{2-s}$ when assuming
that the neutrino spectrum follows a non-broken power-law
$dN_{\nu}/dE_{\nu}\propto E_{\nu}^{-s}$. Measurements of the diffuse isotropic
neutrino flux find the index $s\sim 2.5$ between $\sim 1$ TeV and a few PeV
[35, 36, 37]. Without loss of generality, we take $s=2.0-3.0$.

As explained in the Methods, the all-sky-averaged intensity of the GDE
observed at the solar neighborhood may be related to the total neutrino power
of the Milky Way through $E_{\nu}^{2}\Phi_{\nu}^{\rm
MW}(E_{\nu})=F_{\epsilon}(E_{\nu})({3}/{4\pi)}\left({L_{\nu}^{\rm
MW}(E_{\nu})}/{4\pi r_{\odot}^{2}}\right)$ with $r_{\odot}\approx 8.5\,\rm
kpc$ being our distance to the Galactic center and $F_{\epsilon}$ a geometry
factor of the order unity that accounts for the profiles of gas and sources.

Equations 1 and 2 may also apply to $\gamma$-rays when the attenuation due to
$\gamma\gamma$ pair production with the cosmic microwave background and the
interstellar radiation field is negligible, which is the case below $\sim$100
GeV. Note that equation 2 makes no assumption on the production mechanism of
the $\gamma$-rays and thus holds true regardless of their hadronic or leptonic
origin. When deriving the total luminosity of the Milky Way using the GDE, we
have assumed that the contribution of resolved and unresolved sources in the
Galaxy is negligible. This is consistent with the observation of Fermi-LAT up
to $\sim 100$ GeV [3, 19].

The ratio of the luminosities of an external, Milky Way-like galaxy to the
Milky Way, $L^{\rm EG}/L^{\rm MW}\equiv L^{\rm EG}(M_{\rm MW},0)/L^{\rm MW}$
in neutrinos can be estimated as

$\displaystyle\frac{L_{\nu}^{\rm EG}}{L_{\nu}^{\rm MW}}$
$\displaystyle\approx$ $\displaystyle\frac{\Phi_{\nu}^{\rm
EG}}{\Phi_{\nu}^{\rm MW}}\frac{3F_{\epsilon}}{4\pi
r_{\odot}^{2}n_{0}ct_{H}\xi_{z}}=120\,\left(\frac{{\Phi_{\nu}^{\rm
EG}}/{\Phi_{\nu}^{\rm MW}}}{5}\right)\left(\frac{n_{0}}{0.01\,\rm
Mpc^{-3}}\right)^{-1}\left(\frac{\xi_{z}}{3}\right)^{-1}\left(\frac{F_{\epsilon}}{1}\right),$
(3)

where $n_{0}$ is the local density of galaxies with a similar stellar mass as
the Milky Way and the quantity $\xi_{z}$ accounts for the evolution of the
source emissivity over cosmic time [38]. $\xi_{z}$ is determined by $g(z)$ and
varies from $\sim 0.5$ for uniform evolution to $\sim 3$ for star-formation
evolution. While the analytical expression in equation 3 demonstrates the
dependence on various parameters, the extragalactic-Galactic ratio presented
below is numerically computed using equations 1, 2, 4 and 5.

Figure 2 presents $L^{\rm EG}/L^{\rm MW}$ derived from TeV-PeV neutrino and
GeV-TeV $\gamma$-ray observations. For the extragalactic diffuse background,
we take $\beta=1$, $s=2.5$, and $g(z)$ following the star-formation history as
the benchmark model. The dark shaded regions show the results obtained with
the extragalactic benchmark model and by assuming a uniform gamma-ray and
neutrino emissivity ($\epsilon_{\gamma,\nu}$ in equation 5) inside the
Galactic Plane. The width of the dark shaded region are propagated errors from
the observational uncertainties of the ENB measured with the IceCube 6-year
cascade events [36], the GDE-$\nu$ flux found by IceCube with the $\pi^{0}$
diffusion template [1], and the EGB measured by Fermi-LAT assuming foreground
model A [39]. For comparison, the dashed curve shows the ratio derived with
the same extragalactic model parameters but with a more realistic GDE model
from the simulation. The light shaded region additional accounts for the
uncertainties in the parameters $\beta$, $s$, and $g(z)$. In all cases, we
find that $L_{\gamma}^{\rm EG}$ is not significantly different from
$L_{\gamma}^{\rm MW}$ but $L_{\nu}^{\rm EG}/L_{\nu}^{\rm MW}\gg 1$. In other
words, the Milky Way at the present time is an atypical neutrino emitter. The
IceCube observation of the GDE confines this ratio to $\sim 30-10^{3}$
depending on the neutrino energy. Our finding also suggests that GeV
$\gamma$-rays are suppressed relative to the neutrinos observed by IceCube.
GeV $\gamma$-rays are either barely produced in the process where TeV
neutrinos are generated or, the accompanying gamma rays are attenuated by the
radiation field at the neutrino production site.

Our result suggests that the Galaxy has not hosted the type of emitters that
dominate the ENB in the past $D/c\sim 26\,{\rm kyr}\,(D/8\,\rm kpc)$, which is
the time taken by a neutrino to travel from a Galactic source at a kpc-scale
distance $D$ to the Earth. Cosmic rays at TeV-PeV energy are confined by the
Galactic magnetic field for million-year durations [40]. Had any major cosmic-
ray sources injected protons into the ISM within that time period, the diffuse
neutrino flux of the Galactic plane would be higher and the gap in the
neutrino luminosity of our Galaxy and an external Milky Way-like galaxy would
be smaller.

There is compelling evidence for a highly energetic Seyfert explosion from the
supermassive black hole at the Galactic center a few million years in the
past. Among that, the clearest indications are the Fermi and eROSITA and
bubbles [41, 42]. The time to the last burst/flare is constrained to 2-10 Myr
by both the the mechanical timescales needed to explain the morphology and
multi-wavelength spectra of the observed bubbles and haze [43] and kinematic
studies of halo gas [44, 45, 46]. In addition, elevated ionizing radiation
along the Magellanic Stream [47] independently constrains this nuclear
activity. These timescales are consistent with the scenario where most PeV
protons from the last jet activity have already left the Galaxy today and the
Milky Way is no longer an active neutrino emitter. Other sources or mechanisms
that are not present or extremely rare in the Milky Way over the past tens of
thousands of years, such as tidal disruption events, could also contribute to
the extragalactic neutrino background. These sources are likely $\gamma$-ray-
obscured as suggested by both the observations of the isotropic neutrino flux
[48, 49] and individual neutrino sources [50].

Neutrino Emissivity and Intensity of the Galaxy

The all-sky-averaged intensity of the Galactic Plane observed at the solar
neighborhood is

$\displaystyle E_{\nu}^{2}\Phi_{\nu}^{\rm MW}(E_{\nu})$ $\displaystyle=$
$\displaystyle\frac{1}{4\pi}\int_{-\pi/2}^{\pi/2}\cos(b)db\int_{0}^{2\pi}dlI_{\nu}(l,b,E_{\nu})$
(4)

where
$I_{\nu}(l,b,E_{\nu})=(1/4\pi)\int_{0}^{\infty}ds\,\epsilon_{\nu}(l,b,r,E_{\nu})$
is the intensity from the direction with Galactic longitude and latitude
$(l,b)$ along a line of sight $s$ of the observer, and
$\epsilon_{\nu}(l,b,r,E_{\nu})$ is the production rate of neutrinos per unit
volume in units of $\rm eV\,s^{-1}\,cm^{-3}$ at a distance $r$ from the
Galactic center, which is related to the total neutrino power emitted by the
Milky Way, $L_{\nu}^{\rm MV}$, through

$L_{\nu}^{\rm
MW}(E_{\nu})=\int_{-\pi/2}^{\pi/2}\cos(b)db\int_{0}^{2\pi}dl\int_{0}^{\infty}r^{2}dr\,\epsilon_{\nu}(l,b,r,E_{\nu}).$
(5)

Unresolved point-like and extended sources may contribute to both the Tibet
AS$\gamma$ and IceCube observations [1, 8]. Equations 4 and 5 still apply in
the presence of individual sources. Like equations 1 and 2, these two
equations may also apply to $\gamma$-rays when the $\gamma$-ray absorption by
the interstellar radiation field is negligible.

The intensity of the Galactic Plane depends on the emission profile. We can
rewrite equations 4 and 5 as $E_{\nu}^{2}\Phi_{\nu}^{\rm
MV}(E_{\nu})=F_{\epsilon}(E_{\nu})({3}/{4\pi)}\left({L_{\nu}^{\rm
MW}(E_{\nu})}/{4\pi r_{\odot}^{2}}\right)$ with $F_{\epsilon}$ denoting a
geometry factor of the order unity that accounts for the profiles of gas and
sources. It essentially says that the total flux of GDE observed at the solar
neighborhood is, to the first order, comparable to the flux from a point
source at the Galactic center that carries the power of the entire Galaxy.
When assuming that $\epsilon_{\nu}$ is independent on $E_{\nu}$ and uniform in
the Galactic disk as in the leaky box model, with the disk defined as $R_{\rm
d}<15$ kpc and $z_{\rm d}<0.2$ kpc, we obtain $F_{\epsilon}=1.16$.
Alternatively, we find $F_{\epsilon}=0.97$ when assuming that $\epsilon_{\nu}$
follows the spatial distribution of supernova remnants [18].

Source data for Figure 1 are attached. Any additional data are available from
the corresponding author upon reasonable request.

The calculation used publicly available software packages including HERMES
(https://github.com/cosmicrays/hermes) and CRPropa
(https://github.com/CRPropa).

The work of K.F and F.H is supported by the Office of the Vice Chancellor for
Research and Graduate Education at the University of Wisconsin-Madison with
funding from the Wisconsin Alumni Research Foundation. K.F. acknowledges
support from National Science Foundation (PHY-2110821, PHY-2238916) and NASA
(NMH211ZDA001N-Fermi). This work was supported by a grant from the Simons
Foundation (00001470, KF). J.S.G. thanks the University of Wisconsin College
of Letters and Science for partial support of his IceCube-related research.
The research of F.H was also supported in part by the U.S. National Science
Foundation under grants PHY-2209445 and OPP-2042807.

K.F. carried out the simulations and analyses and prepared the manuscript. All
authors participated in the interpretation of the results and edited the
manuscript.

The authors declare no competing interests.

Figure 1:  All-sky-averaged intensities of the Galactic diffuse emission (GDE)
and extragalactic background (EB) in $\gamma$-ray and neutrinos, scaled by
$E^{2.5}$. Gamma-ray is indicated in red and neutrino (per-flavor flux
including both neutrinos and antineutrinos) is in blue. Galactic components
(unhatched regions) include the diffuse neutrino emission from the Galactic
plane measured by IceCube using the $\pi^{0}$ template [1] (blue shaded region
indicating the $1\,\sigma$ uncertainties), the neutrino flux derived from the
GDE-$\gamma$ measured by Tibet AS$\gamma$ [8, 18] (silver shaded region), and
the Galactic interstellar emission model of Fermi-LAT [20] (red shaded
region). The dashed curves present a numerical simulation of the diffuse
emission that accounts for the spatial distribution of sources and gas in the
Milky Way. Extragalactic components (hatched regions) include the isotropic
diffuse neutrino background measured by IceCube [36] (blue hatched region) and
the extragalactic $\gamma$-ray background measured by Fermi-LAT [39] (red
hatched region).  Figure 2:  Derived ratio of the average luminosity of an
external, Milky Way-like galaxy and the Galactic luminosity in GeV-TeV
$\gamma$-ray and TeV-PeV neutrinos. The ratio is calculated using equations 1,
2, 4 and 5 with the Fermi-LAT and IceCube measurements of the GDE. The dark
shaded regions adopt the benchmark extragalactic model parameters and assume a
uniform emissivity within the Galactic disk. Their widths correspond to the
uncertainties in the GDE and EB observations. The dashed curves also use the
benchmark extragalactic model but a more realistic Galactic model that takes
into account the spatial distribution of sources and gas in the Milky Way. The
light shaded region further accounts for the uncertainties in the parameters
of the extragalactic model.

## References

  * [1] IceCube Collaboration.  Observation of high-energy neutrinos from the galactic plane.  _Science_ 380, 1338–1343 (2023).
  * [2] Fuerst, P. M. _et al._ Galactic and Extragalactic Analysis of the Astrophysical Muon Neutrino Flux with 12.3 years of IceCube Track Data.  _PoS_ ICRC2023, 1046 (2023).
  * [3] Ackermann, M. _et al._ Fermi-LAT Observations of the Diffuse Gamma-Ray Emission: Implications for Cosmic Rays and the Interstellar Medium.  _Astrophys. J._ 750, 3 (2012).
  * [4] Schwefer, G., Mertsch, P. & Wiebusch, C.  Diffuse Emission of Galactic High-energy Neutrinos from a Global Fit of Cosmic Rays.  _ApJ_ 949, 16 (2023).
  * [5] Gaggero, D., Grasso, D., Marinelli, A., Urbano, A. & Valli, M.  The Gamma-Ray and Neutrino Sky: A Consistent Picture of Fermi-LAT, Milagro, and IceCube Results.  _ApJ_ 815, L25 (2015).
  * [6] Abramowski, A. _et al._ Diffuse Galactic gamma-ray emission with H.E.S.S.  _Phys. Rev. D_ 90, 122007 (2014).
  * [7] Abeysekara, A. U. _et al._ Galactic Gamma-Ray Diffuse Emission at TeV energies with HAWC Data.  _PoS_ ICRC2021, 835 (2021).
  * [8] Tibet AS${\gamma}$ Collaboration _et al._ First Detection of sub-PeV Diffuse Gamma Rays from the Galactic Disk: Evidence for Ubiquitous Galactic Cosmic Rays beyond PeV Energies.  _Phys. Rev. Lett._ 126, 141101 (2021).
  * [9] Cao, Z. _et al._ Measurement of ultra-high-energy diffuse gamma-ray emission of the Galactic plane from 10 TeV to 1 PeV with LHAASO-KM2A.  _arXiv e-prints_ arXiv:2305.05372 (2023).
  * [10] Vecchiotti, V., Zuccarini, F., Villante, F. L. & Pagliaroli, G.  Unresolved Sources Naturally Contribute to PeV Gamma-Ray Diffuse Emission Observed by Tibet AS$\gamma$.  _ApJ_ 928, 19 (2022).
  * [11] Porter, T. A., Jóhannesson, G. & Moskalenko, I. V.  High-energy Gamma Rays from the Milky Way: Three-dimensional Spatial Models for the Cosmic-Ray and Radiation Field Densities in the Interstellar Medium.  _ApJ_ 846, 67 (2017).
  * [12] Porter, T. A., Jóhannesson, G. & Moskalenko, I. V.  The GALPROP Cosmic-ray Propagation and Nonthermal Emissions Framework: Release v57.  _ApJS_ 262, 30 (2022).
  * [13] Linden, T. & Buckman, B. J.  Pulsar TeV Halos Explain the Diffuse TeV Excess Observed by Milagro.  _Phys. Rev. Lett._ 120, 121101 (2018).
  * [14] Vecchiotti, V., Pagliaroli, G. & Villante, F. L.  The contribution of Galactic TeV pulsar wind nebulae to Fermi large area telescope diffuse emission.  _Communications Physics_ 5, 161 (2022).
  * [15] Cerri, S. S., Gaggero, D., Vittino, A., Evoli, C. & Grasso, D.  A signature of anisotropic cosmic-ray transport in the gamma-ray sky.  _J. Cosmology Astropart. Phys_ 2017, 019 (2017).
  * [16] Ahlers, M. & Murase, K.  Probing the Galactic Origin of the IceCube Excess with Gamma-Rays.  _Phys.Rev._ D90, 023010 (2014).
  * [17] Lipari, P. & Vernetto, S.  Diffuse Galactic gamma-ray flux at very high energy.  _Phys. Rev. D_ 98, 043003 (2018).
  * [18] Fang, K. & Murase, K.  Multimessenger Implications of Sub-PeV Diffuse Galactic Gamma-Ray Emission.  _ApJ_ 919, 93 (2021).
  * [19] Acero, F. _et al._ Development of the Model of Galactic Interstellar Emission for Standard Point-source Analysis of Fermi Large Area Telescope Data.  _ApJS_ 223, 26 (2016).
  * [20] Abdollahi, S. _et al._ Fermi Large Area Telescope Fourth Source Catalog.  _ApJS_ 247, 33 (2020).
  * [21] Evoli, C. _et al._ Cosmic-ray propagation with DRAGON2: I. numerical solver and astrophysical ingredients.  _J. Cosmology Astropart. Phys_ 2017, 015 (2017).
  * [22] De La Torre Luque, P. _et al._ Galactic diffuse gamma rays meet the PeV frontier.  _A &A_ 672, A58 (2023).
  * [23] Pagliaroli, G., Evoli, C. & Villante, F. L.  Expectations for high energy diffuse galactic neutrinos for different cosmic ray distributions.  _J. Cosmology Astropart. Phys_ 2016, 004 (2016).
  * [24] Cataldo, M., Pagliaroli, G., Vecchiotti, V. & Villante, F. L.  Probing galactic cosmic ray distribution with TeV gamma-ray sky.  _J. Cosmology Astropart. Phys_ 2019, 050 (2019).
  * [25] Planck Collaboration _et al._ Planck 2018 results. VI. Cosmological parameters.  _A &A_ 641, A6 (2020).
  * [26] Schechter, P.  An analytic expression for the luminosity function for galaxies.  _ApJ_ 203, 297–306 (1976).
  * [27] Fasano, G. _et al._ WINGS: a WIde-field Nearby Galaxy-cluster Survey. I. Optical imaging.  _A &A_ 445, 805–817 (2006).
  * [28] Tomczak, A. R. _et al._ Galaxy Stellar Mass Functions from ZFOURGE/CANDELS: An Excess of Low-mass Galaxies since z = 2 and the Rapid Buildup of Quiescent Galaxies.  _ApJ_ 783, 85 (2014).
  * [29] Conselice, C. J., Wilkinson, A., Duncan, K. & Mortlock, A.  The Evolution of Galaxy Number Density at z $<8$ and Its Implications.  _ApJ_ 830, 83 (2016).
  * [30] Bell, E. F., McIntosh, D. H., Katz, N. & Weinberg, M. D.  The Optical and Near-Infrared Properties of Galaxies. I. Luminosity and Stellar Mass Functions.  _ApJS_ 149, 289–312 (2003).
  * [31] Gilfanov, M.  Low-mass X-ray binaries as a stellar mass indicator for the host galaxy.  _MNRAS_ 349, 146–168 (2004).
  * [32] Kornecki, P. _et al._ $\gamma$-ray/infrared luminosity correlation of star-forming galaxies.  _A &A_ 641, A147 (2020).
  * [33] Suh, H. _et al._ No Significant Evolution of Relations between Black Hole Mass and Galaxy Total Stellar Mass Up to z $\sim$ 2.5.  _ApJ_ 889, 32 (2020).
  * [34] Hopkins, A. M. & Beacom, J. F.  On the Normalization of the Cosmic Star Formation History.  _ApJ_ 651, 142–154 (2006).
  * [35] IceCube Collaboration _et al._ IceCube high-energy starting event sample: Description and flux characterization with 7.5 years of data.  _Phys. Rev. D_ 104, 022002 (2021).
  * [36] Aartsen, M. G. _et al._ Characteristics of the diffuse astrophysical electron and tau neutrino flux with six years of IceCube high energy cascade data.  _Phys. Rev. Lett._ 125, 121104 (2020).
  * [37] Abbasi, R. _et al._ Improved Characterization of the Astrophysical Muon–neutrino Flux with 9.5 Years of IceCube Data.  _Astrophys. J._ 928, 50 (2022).
  * [38] Waxman, E. & Bahcall, J.  High energy neutrinos from astrophysical sources: An upper bound.  _Phys. Rev. D_ 59, 023002 (1998).
  * [39] Ackermann, M. _et al._ The Spectrum of Isotropic Diffuse Gamma-Ray Emission between 100 MeV and 820 GeV.  _ApJ_ 799, 86 (2015).
  * [40] Strong, A. W., Moskalenko, I. V. & Ptuskin, V. S.  Cosmic-Ray Propagation and Interactions in the Galaxy.  _Annual Review of Nuclear and Particle Science_ 57, 285–327 (2007).
  * [41] Ackermann, M. _et al._ The Spectrum and Morphology of the $Fermi$ Bubbles.  _Astrophys. J._ 793, 64 (2014).
  * [42] Predehl, P. _et al._ Detection of large-scale X-ray bubbles in the Milky Way halo.  _Nature_ 588, 227–231 (2020).
  * [43] Yang, H. Y. K., Ruszkowski, M. & Zweibel, E. G.  Fermi and eROSITA bubbles as relics of the past activity of the Galaxy’s central black hole.  _Nature Astron._ 6, 584–591 (2022).
  * [44] Fox, A. J. _et al._ Probing the Fermi Bubbles in Ultraviolet Absorption: A Spectroscopic Signature of the Milky Way’s Biconical Nuclear Outflow.  _ApJ_ 799, L7 (2015).
  * [45] Miller, M. J. & Bregman, J. N.  The Interaction of the Fermi Bubbles with the Milky Way’s Hot Gas Halo.  _ApJ_ 829, 9 (2016).
  * [46] Bordoloi, R. _et al._ Mapping the Nuclear Outflow of the Milky Way: Studying the Kinematics and Spatial Extent of the Northern Fermi Bubble.  _ApJ_ 834, 191 (2017).
  * [47] Bland-Hawthorn, J. _et al._ The Large-scale Ionization Cones in the Galaxy.  _ApJ_ 886, 45 (2019).
  * [48] Murase, K., Guetta, D. & Ahlers, M.  Hidden Cosmic-Ray Accelerators as an Origin of TeV-PeV Cosmic Neutrinos.  _Phys. Rev. Lett._ 116, 071101 (2016).
  * [49] Fang, K., Gallagher, J. S. & Halzen, F.  The TeV Diffuse Cosmic Neutrino Spectrum and the Nature of Astrophysical Neutrino Sources.  _ApJ_ 933, 190 (2022).
  * [50] IceCube Collaboration _et al._ Evidence for neutrino emission from the nearby active galaxy NGC 1068.  _Science_ 378, 538–543 (2022).

See pages - of SM.pdf