# Exploring the Necessity of Visual Modality
in Multimodal Machine Translation using Authentic Datasets

###### Abstract

Recent research in the field of multimodal machine translation (MMT) has
indicated that the visual modality is either dispensable or offers only
marginal advantages. However, most of these conclusions are drawn from the
analysis of experimental results based on a limited set of bilingual sentence-
image pairs, such as Multi30k. In these kinds of datasets, the content of one
bilingual parallel sentence pair must be well represented by a manually
annotated image, which is different from the real-world translation scenario.
In this work, we adhere to the universal multimodal machine translation
framework proposed by Tang et al. (2022). This approach allows us to delve
into the impact of the visual modality on translation efficacy by leveraging
real-world translation datasets. Through a comprehensive exploration via
probing tasks, we find that the visual modality proves advantageous for the
majority of authentic translation datasets. Notably, the translation
performance primarily hinges on the alignment and coherence between textual
and visual contents. Furthermore, our results suggest that visual information
serves a supplementary role in multimodal translation and can be substituted.

Keywords: MMT, image retrieval, visual noise filtering, supplementary text
retrieval

Exploring the Necessity of Visual Modality
in Multimodal Machine Translation using Authentic Datasets

Zi Long1, Zhenhao Tang2, Xianghua Fu1, Jian Chen2, Shilong Hou2, Jinze Lyu2
---
1College of Big Data and Internet, Shenzhen Technology University, Shenzhen,
China
2College of Application and Technology, Shenzhen University, Shenzhen, China

Abstract content

## 1\. Introduction

With the development of neural machine translation (NMT), the role of visual
information in machine translation has attracted researchers’ attention
(Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018). Different
from those text-only NMT (Bahdanau et al., 2014a; Gehring et al., 2016), a
bilingual parallel corpora with manual image annotations are used to train an
MMT model by an end-to-end framework, and therefore visual information can
assist NMT model to achieve better translation performance (Calixto and Liu,
2017; Calixto et al., 2017; Su et al., 2021).

Concurrently, researchers have also undertaken a diverse range of experiments
in an effort to validate the specific role of visual information in NMT. For
example, Grönroos et al. (2018a) and Lala et al. (2018) observed that the
robustness of MMT systems remains unaffected when the input image lacks direct
relevance to the accompanying text. Notably, the absence of visual features,
as highlighted by Elliott (2018), also does not yield detrimental effects. Wu
et al. (2021) underscores that the utilization of the visual modality serves
as a regularization mechanism during training rather than serving as a true
complement to the textual modality. Oppositely, Caglayan et al. (2019) delve
into the correlation between visual features and text. Their investigation
reveals that incorporating the input image aids translation, particularly when
certain input words are masked. Li et al. (2022) design more detailed probing
tasks and found that stronger vision features strengthen MMT systems.

Note that most of the previous conclusions are drawn from the analysis of
experimental results based on a restricted selection of manually annotated
bilingual sentence-image pairs, known as the Multi30k dataset (Elliott et al.,
2016). Within the Multi30k dataset, as depicted in Table 1, the sentences
primarily comprise common and straightforward vocabulary, with each bilingual
parallel sentence pair being effectively depicted by a single image. Table 1
also presents an illustration of a bilingual sentence-image pair extracted
from a genuine news report from the United Nations News111
https://news.un.org/en/ , alongside examples of sentence pairs derived from
other other authentic translation datasets. Evidently, a substantial disparity
exists between the Multi30k dataset and the authentic translation data. Hence,
the evidence and findings derived from Multi30k may potentially exhibit
inadequate generalizability and offer limited utility when attempting to
analyze the role of the visual modality in MMT within real-world translation
scenarios. In these scenarios, sentences often incorporate rare and uncommon
words and are only partially depicted by accompanying images.

Data source | Sentences | Image
---|---|---
Multi30k |  | EN:  |  A dog is running in the snow.
---|---
DE:  |  Ein Hund rennt im Schnee.
  |
---

UN News |  | EN:  |  Rescue workers look for survivors in a building in Samada, Syria destroyed by the February 6 earthquake.
---|---
DE:  |  Rettungskräfte suchen nach Überlebenden in einem Gebäude in Samada, Syrien, das durch das Erdbeben vom 6. Februar zerstört wurde.
  |
---

Bible |  | EN:  |  I saw, and behold, there was no man,and all the birds of the sky had fled.
---|---
DE:  |  Ich sah, und siehe, da war kein Mensch, und alle Vögel unter dem Himmel waren weggeflogen.
no image
MultiUN |  | EN:  |  Development assistance cannot by itself prevent or end conflict.
---|---
DE:  |  Entwicklungshilfe allein kann Konflikte weder verhüten noch beenden.
no image
Table 1: Comparison between Multi30k Dataset and Authentic Datasets

In a recent study, Tang et al. (2022) introduced a universal multimodal neural
machine translation model that integrates open-vocabulary image retrieval
techniques. In this work, inspired by Tang et al. (2022), we formulate a set
of comprehensive probing tasks aimed at assessing the extent to which the
visual modality enhances MMT within real-world translation scenarios. In
addition to commonly used Multi30k, we conduct an extensive set of experiments
across four authentic text-only translation datasets. We further evaluated two
visual noise filtering approaches based on the correlation between textual and
visual content. Furthermore, we investigate the necessity of visual modality
in the current multimodal translation process by substituting visual data with
closely equivalent textual content. To summarize, our findings are:

(1)

Visual modality is mostly beneficial for translation, but its effectiveness
wanes as text vocabulary becomes less image-friendly.

(2)

The MMT performance depends on the consistency between textual and visual
contents, and utilizing filters based on the textual-visual correlation can
enhance the performance.

(3)

Visual information plays a supplementary role in the multimodal translation
process and can be substituted by the incorporation of additional textual
information.

## 2\. Related Work

The integration of extra knowledge to build fine-grained representations is a
crucial aspect in language modeling (Li et al., 2020a, b; Zhang et al., 2020).
Incorporating the visual modality into language modeling has the potential to
enhance the machine’s understanding of the real world from a more
comprehensive perspective. Inspired by the studies on the image description
generation task (Elliott et al., 2015; Venugopalan et al., 2015; Xu et al.,
2015), MMT models have gradually become a hot topic in machine translation
research. In some cases, visual features are directly used as supplementary
information to the text presentation. For example, Huang et al. (2016) take
global visual features and local visual features as additional information for
sentences. Calixto and Liu (2017) initializes the encoder hidden states or
decoder hidden states through global visual features. Calixto et al. (2017)
use an independent attention mechanism to capture visual representations.
Caglayan et al. (2016) incorporate spatial visual features into the MMT model
via an independent attention mechanism. On this basis, Delbrouck and Dupont
(2017b) employs compact bilinear pooling to fuse two modalities. Lin et al.
(2020) attempt to introduce the capsule network into MMT, they use the
timestep-specific source-side context vector to guide the routing procedure.
Su et al. (2021) introduce image-text mutual interactions to refine their
semantic representations.

Figure 1: Frameworks of three probing methods

Researchers have also come to recognize the potential redundancy of the visual
modality. Inconsequential images exhibit minimal impact on translation
quality, and the absence of images does not yield a significant drop in BLEU
scores, as noted by Elliott (2018). Encouraging findings emerged in the study
by Caglayan et al. (2019). They highlighted the continuing utility of the
visual modality in scenarios where linguistic context is limited but noted its
diminished sensitivity when exposed to complete sentences. In a more recent
investigation, Wu et al. (2021) attributed the observed BLEU improvement in
MMT tasks to training regularization. They underscored the importance of
constructing appropriate probing tasks with inadequate textual input. It’s
important to highlight that the proposed probing task represents an enhanced
iteration building upon prior research (Caglayan et al., 2019; Wu et al.,
2021). Li et al. (2022) made a systematic study on whether stronger vision
features are helpful. All the preceding research has been conducted
exclusively on the Multi30k dataset, which has limitations in scale and
considerably differs from real-world translation scenarios. In this study, we
employ the framework introduced by Tang et al. (2022) to systematically
examine the influence of visual information across various authentic
translation datasets, extending our analysis beyond the limitations of the
small and specialized Multi30k dataset.

## 3\. Preliminary

We start with a description of three probing methods employed in this work,
which encompass the approach introduced by Tang et al. (2022) and two
additional methods derived from it. Figure 1 shows frameworks of these three
methods.

### 3.1. MMT with Search Engine Based Image Retrieval

As depicted in the top section of Figure 1, Tang et al. (2022) introduced a
search engine-based image retrieval technique and a text-aware attention image
encoder. This innovation enables the handling of authentic text-only
translation data within MMT systems. We implement this approach across
multiple authentic translation datasets to examine the influence of visual
information across datasets with varying styles. To ensure the
comprehensiveness of this paper, this section will provide a brief overview of
the approach proposed by Tang et al. (2022).

#### Text Encoder

In this work, we employ a commonly utilized bi-directional LSTM as the RNN
text encoder. For a given sentence denoted as $X$, the output of the text
encoder is represented as
$C=(\boldsymbol{h}_{1},\boldsymbol{h}_{2},\ldots,\boldsymbol{h}_{N})$, where
$N$ denotes the length of the sentence $X$.

#### Image Retrieval

To emphasize the core components of the sentence and mitigate the impact of
noise, including stopwords and infrequent words, Tang et al. (2022) utilized
the TF-IDF method (Witten et al., 2005) to generate search queries for image
search engines. Subsequently, the generated search queries are utilized in
image search engines to retrieve the first available image associated with
each query. For each given sentence $X$, $M$ search queries denoted as
($q_{1},q_{2},\ldots,q_{M}$) are generated, and subsequently $M$ images
represented as ($A_{1},A_{2},\ldots,A_{M}$) are retrieved from search engines.

#### Text-Aware Attentive Visual Encoder

Each image $A_{m}$ ($m=1,\dots,M$) is transformed into a $196\times 1024$
dimensional feature vector using ResNet-50 (He et al., 2016) . A simple but
effective scaled dot-product attention in visual encoder is subsequently
employed in the visual encoder to derive a resultant visual representation.
Here, we utilize the average pooling $C_{{}^{\prime}}$ of the text
representation
$C=(\boldsymbol{h}_{1},\boldsymbol{h}_{2},\ldots,\boldsymbol{h}_{N})$ as the
query, while the visual feature vectors $A_{1},A_{2},\ldots,A_{M}$ serve as
the keys and values in this attention mechanism. The resultant visual
representation $A$ is also expressed as a $196\times 1024$ dimensional feature
vector, which can be regarded as a matrix
$A=(\boldsymbol{a}_{1},\boldsymbol{a}_{2},\ldots,\boldsymbol{a}_{L})$, where
$L=196$ and each $\boldsymbol{a}_{l}\in R^{1024}$ ($l=1,\dots,L$). Visual
representation
$A=(\boldsymbol{a}_{1},\boldsymbol{a}_{2},\ldots,\boldsymbol{a}_{L})$ and text
representation
$C=(\boldsymbol{h}_{1},\boldsymbol{h}_{2},\ldots,\boldsymbol{h}_{N})$ are then
used as the inputs of translation decoder.

#### Translation Decoder

For the decoder, we adopt the approach introduced by Su et al. (2021),
implementing both a bidirectional attention network and a co-attention network
to effectively capture the underlying semantic interactions between textual
and visual elements. Based on the results of the preliminary experiment, it
was evident that transformer-based models did not confer a performance
advantage on datasets like Global Voices and other smaller ones. Consequently,
we followed the approach of Tang et al. (2022) and selected LSTM as our
foundational model. The bidirectional attention network enhances the
representations of both text and image. These enhanced representations are
subsequently input into the co-attention network to obtain the time-dependent
context vector $c_{t}$ and the visual vector $v_{t}$. Within the co-attention
network, we calculate the probability distribution for the next target word
$y_{t}$ using the previous hidden state $s_{t-1}$, the previously generated
target word $y_{t-1}$, the time-dependent context vector $c_{t}$, and the
time-dependent visual vector $v_{t}$.

### 3.2. MMT with Visual Noise Filtering

Considering that the noise images obtained from search engines could have a
substantial impact on the performance of the MMT system, we further evaluated
two visual noise filtering approaches based on the correlation between textual
and visual content, as depicted in the central part of Figure 1. One approach
utilizes the pretrained CLIP model to filter out noise images, while the other
employs a region-level image-text attentive filter module to filter out noisy
image regions.

#### Noise Image Filter

In the CLIP-based noise image filtering approach, we begin by retrieving
$M^{{}^{\prime}}$ ($M^{{}^{\prime}}>M$) images from search engines for each
input sentence. Following this, we calculate the correlation between the input
text and the retrieved images using a pretrained CLIP model (Radford et al.,
2021). Subsequently, we select only the top-$M$ images with the highest
correlation to the input source text as the output of the image retrieval
process.

#### Noise Region Filter

In the noise image region filtering approach, we begin by extracting
convolutional feature maps from the top-$O$ most confident regions denoted as
($r_{1},\dots,r_{O}$) in each collected image. This is achieved using a
pretrained Faster R-CNN model (Ren et al., 2015), aiding in the initial
filtration of visual information that may be challenging to distinguish as
distinct regions in the images. The image region of each collected image is
then represented as a $1024$ dimensional feature vector using ResNet-50. For
all the retrieved $M$ images, we extract a total of $M\times O$ regions
($r_{1},\dots,r_{M\times O}$), resulting in $M\times O$ feature vectors
($\boldsymbol{a}_{1},\dots,\boldsymbol{a}_{M\times O},\boldsymbol{a}_{o}\in
R^{1024}$). Subsequently, we compute the correlation score between each image
region and the input text using the following equation:

$\displaystyle S(\boldsymbol{a}_{o},C^{{}^{\prime}})$ $\displaystyle=$
$\displaystyle V_{a}{\rm tanh}(W_{a}\boldsymbol{a}_{o}+U_{a}C^{{}^{\prime}})$

Here, $C^{{}^{\prime}}$ represents the average pooling of the text
representation
$C=(\boldsymbol{h}_{1},\boldsymbol{h}_{2},\ldots,\boldsymbol{h}_{N})$. We
retain only the visual information from the top-$O$ most relevant regions out
of the initially extracted $M\times O$ regions. This preserved visual
information serves as the visual representation for the given input sentence,
denoted as $A=\\{\boldsymbol{a}_{o}|{S}(\boldsymbol{a}_{o},C^{{}^{\prime}})$
ranks in the top-$O,1\leq o\leq M\times O\\}$ , and it is subsequently fed
into the translation decoder module. Less relevant regions are discarded
during this process.

### 3.3. Supplementary Text Enhanced NMT

As discussed by Caglayan et al. (2019), multimodal translation models
typically view visual information as a complementary component to textual
information. However, we raise the question of whether this complementary role
can also be achieved by incorporating additional textual information,
potentially obviating the need for images in the process. Hence, our
investigation aims to assess the necessity of visual information in the
existing multimodal translation process by substituting visual data with
nearly equivalent textual information. As illustrated in the lower section of
Figure 1, we replace the image retrieval module with a supplementary text
retrieval module and substitute the text-aware attentive visual encoder with a
similar text-aware attentive supplementary text encoder.

#### Supplementary Text Retrieval

Similar to the process of retrieving images from search engines, we collected
supplementary textual data from search engines. For every input source
sentence $X$, we follow the same approach as outlined in Section 3.1 to
generate $M$ search queries ($q_{1},\dots,q_{M}$). Subsequently, we collect
$M$ sentences ($T_{1},\dots,T_{M}$) that contains all the terms present in the
respective search queries ($q_{i}\subseteq T_{i},1\leq i\leq M$).

#### Text-Aware Attentive Supplementary Text Encoder

Each gathered supplementary text $T_{m}$ ($m=1,\dots,M$) is transformed into a
$N\times 1024$ dimensional feature vector using BERT (Devlin et al., 2018),
where $N$ denotes the length of the gathered text data. To ensure consistency,
these textual feature vectors are subsequently padded to match the dimensions
of $L\times 1024$ ($L=196$), aligning them with the visual feature vectors.
These feature vectors are then integrated into the scaled dot-product
attention module as keys and values, with the average pooling
$C^{{}^{\prime}}$ representing the input text serving as the query. The
resultant supplementary text representation is then passed to the translation
decoder.

## 4\. Experiment Setup

### 4.1. Dataset

We conducted experiments on five commonly used machine translation datasets,
including multimodal machine translation dataset Multi30k (Elliott et al.,
2016) English-to-German, Global Voices (Tiedemann, 2012) English-to-German ,
and WMT’ 16 (100k) English-to-German (Newstest2016 as the test set)222 To
ensure a focused evaluation of the retrieved visual information’s
effectiveness, we intentionally sought to minimize the impact of data size.
Consequently, we opted to construct our training set by randomly sampling
100,000 sentence pairs from the total pool of 4.5 million sentence pairs. This
sampling approach aligns our dataset size more closely with that of other
datasets for a fairer assessment.  , Bible (Christodouloupoulos and Steedman,
2015) English-to-German, and MultiUN (Eisele and Chen, 2010) English-to-
German. The statistics for each dataset are presented in Table 2.

dataset | training set | dev set | test set
---|---|---|---
Multi30k | 29,000 | 1,014 | 1,000
Global Voices | 69,227 | 2,000 | 2,000
WMT’16 (100k) | 100,000 | 2,000 | 3,000
Bible | 56,734 | 1,953 | 1,821
MultiUN | 56,235 | 4,000 | 4,000
Table 2: Statistics of datasets Method | BLEU Score
---|---
Text-only | Bi-LSTM (Calixto et al., 2017) | 33.70
NMT | Transformer (Zhang et al., 2019) | 36.86
| Zhang et al. (2019) | 36.86
MMT with | Zhao et al. (2021) | 38.40
Original Images | Su et al. (2021) | 39.20
|  Tang et al. (2022) (Section 3.1) | 38.14
MMT with | Zhang et al. (2019) | 36.94
Retrieved Images |  Tang et al. (2022) (Section 3.1) | 38.43
| MMT with Visual Noise Filtering (Section 3.2) | 38.51
NMT with Retrieved Supplementary Text (Section 3.3) | 39.13
Table 3: Results on Multi30K Method | Dataset
---|---
| Multi30k | Global Voices | WMT‘16 (100k) | Bible | MultiUN
Text-only NMT | 33.70 | 9.22 | 7.99 | 35.23 | 39.49
MMT with Random Images | 37.65 | 9.29 | 8.11 | 35.31 | 39.48
MMT with Blank Images | 37.79 | 9.46 | 8.31 | 35.39 | 39.52
MMT with Retrieved Images | 38.43 | 9.81 | 8.41 | 35.42 | 39.53
Table 4: Translation performance across diverse datasets under varied image
conditions (BLEU score)

### 4.2. Model Implementation

For image retrieval, we used the Microsoft Bing333
https://global.bing.com/images as the image search engine. In contrast, for
supplementary text retrieval, we gathered sample sentences that included all
the terms found in the respective search queries by referencing the Microsoft
Bing Dictionary444 https://www.bing.com/dict . As described in Section 3.1 and
Section 3.3, we set $M$ to 5. This choice signifies that we formulated 5
search queries and procured 5 images or supplementary text instances555 When
an insufficient number of sample sentences can be collected, we resort to
large pretrained models like ChatGPT to generate sentences that meet the
search query.  for every source language sentence.

Regarding the text encoder, we used a bi-directional RNN with GRU to extract
text features. Specifically, we used a 256 dimensional single-layer forward
RNN and a 256 dimensional single-layer backward RNN. For the translation
decoder, we adhered to the approach proposed by Su et al. (2021) and utilized
a modified cGRU with hidden states of 256 dimensions. Furthermore, we
configured the embedding sizes for both source and target words to be 128.

As described in Section 3.1, the visual encoder we employed leveraged the
$\mathrm{res4f}$ layer of a pretrained ResNet-50(He et al., 2016) model to
extract visual features of dimensions $196\times 1024$. Furthermore, as
described in Section 3.3, the supplementary text encoder utilized a BERT model
pretrained on the BooksCorpus(Zhu et al., 2015) and English Wikipedia666
https://en.wikipedia.org/wiki/English_Wikipedia . This model was employed to
extract textual features of dimensions $N\times 1024$, where $N$ represents
the length of the retrieved supplementary text.

Regarding the noise image filter, we set $M^{{}^{\prime}}=10$ and used a CLIP
model (Radford et al., 2021) pretrained on the YFCC100M dataset (Thomee et
al., 2016) to filter out noisy images. For the noise region fitler, we
configured it with $O=128$. Here, we utilized a pretrained Faster R-CNN model
(Ren et al., 2015) that had been trained on the Open Images dataset
(Kuznetsova et al., 2020). This model was employed to identify and filter
noisy regions in images effectively.

### 4.3. Training Parameters

We initiated the word embeddings and other associated model parameters
randomly, following a uniform distribution with a range of $-0.1$ to $0.1$.
During training, we employed the Adam optimizer with a mini-batch size of 32
and set the learning rate to 0.001. Additionally, a dropout strategy with a
rate of 0.3 was applied to further enhance the models. The training process
continued for up to 15 epochs, with early stopping activated if the BLEU
(Papineni et al., 2002) score on the development set did not exhibit
improvement for 3 consecutive epochs. The model with the highest BLEU score on
the dev set was selected for evaluation on the test set. To minimize the
impact of random seeds on experimental results and ensure result stability, we
conducted the experiment 5 times with fixed random seeds and reported the
macro-average of BLEU scores as the final result.

### 4.4. Baselines

In the case of the Multi30k dataset, we conducted a quantitative comparison of
the probing methods with several recent MMT models (Zhang et al., 2019; Zhao
et al., 2021; Su et al., 2021; Tang et al., 2022). However, the main focus of
this research is to evaluate the necessity of visual information within real-
world translation scenarios. Four out of the five datasets utilized in our
evaluation experiments are authentic text-only translation datasets without
any visual annotation. Consequently, for each dataset, we exclusively employed
the text-only Bi-LSTM (Calixto et al., 2017) as a baseline.

The baseline model and the models detailed in Section 3 were all trained using
the same training set and identical training parameters. For all these models,
we present the 4-gram BLEU score (Papineni et al., 2002) as the primary
evaluation metric.

## 5\. Results and Analysis

Table 3 presents the experimental results of the Multi30k dataset. Compared to
various baseline models, all three probing methods mentioned in Section 3 have
achieved promising results. Notably, the MMT model with visual noise filtering
(Section 3.2) achieved a BLEU score of 38.51, while the NMT model with
retrieved supplementary text (Section 3.3) achieved an impressive BLEU score
of 39.13. In comparison to text-only NMT models (Calixto et al., 2017; Vaswani
et al., 2017), the NMT model with retrieved supplementary text significantly
outperforms them, showcasing a substantial increase in BLEU score. When
compared to existing MMT methods that utilize original images (Zhang et al.,
2019; Zhao et al., 2021; Su et al., 2021), the NMT model with retrieved
supplementary text obtains a comparable BLEU score. Furthermore, in contrast
to the MMT methods with retrieved images (Zhang et al., 2019; Tang et al.,
2022), the NMT model with retrieved supplementary text demonstrates
performance gains of approximately 2.2 and 0.7 BLEU points, respectively.

Further experimental results and analysis will be presented in the following
sections.

| Multi30k | Global Voices | WMT’16 (100k) | Bible | MultiUN
---|---|---|---|---|---
Number of sentences with half or more non-entity keywords | 27 | 94 | 796 | 398 | 818
Number of sentences with half of more noise images | 61 | 228 | 685 | 761 | 663
Table 5: Summary of manual analysis of image retrieval outcomes for each
dataset

### 5.1. Translation Performances across Varied Datasets

We firstly quantitatively compared text-only NMT (Calixto et al., 2017) with
MMT utilizing retrieved images (Section 3.1) across five diverse datasets
mentioned in Section 4.1. As demonstrated in Table 4, MMT achieved
significantly higher BLEU scores on Multi30k, higher BLEU scores on Global
Voices and WMT’16 (100k), and slightly higher BLEU scores on Bible and
MultiUN. It is intriguing to note that the improvement in translation
performance is substantial on Multi30k, with an increase of approximately 4.7,
whereas the gain on MultiUN is relatively modest, at approximately 0.04.

We speculate that the variations in results among the aforementioned
translation datasets, such as Multi30k and other datasets, may be attributed
to the differing qualities of images collected through the search engine. To
evaluate the influence of the quality of collected images, we train the MMT
model with randomly retrieved unrelated images, blank images, and retrieved
images from image search engines, respectively.

The evaluation results are shown in table 4. It is obvious that MMT models
with retrieved images achieves the highest BLEU score on all Multi30k and
other four datasets, demonstrating the effectiveness of visual information
from retrieved images. Compared with the model with random images and blank
images, the performance gain of collected images is approximately 0.7 & 0.6
BLEU score on Multi30k, and 0.5 & 0.3 BLUE score on Global Voices. However, on
WMT’16 (100k), Bible, and MultiUN datasets, models with retrieved images
achieve almost the same BLEU score as the model with blank images.

One of the possible reason is that sentences from those three datasets
contains fewer entity words that can be represented by images, and therefore,
the search engine based image retrieval method collects numbers of noise
images. Sentences from WMT’16 (100k), Bible, and MultiUN datasets describe
abstract concepts and complex events, while sentences from Multi30k and Global
Voices describe real objects and people, which is more reliable for image
retrieval. 777 Examples of retrieved images from various datasets are
presented in Table 8.

To validate the hypotheses, we manually analyzed the image retrieval outcomes
of each dataset. In detail, we initially conducted a random sampling of 1,000
sentences and employed the image retrieval methods outlined in Section 3.1 to
gather keywords and images for each sentence. Regarding the extracted
keywords, we conducted manual assessments to identify whether each keyword
qualifies as an entity word. Regarding the collected images, we carried out
manual evaluations to determine if an image could offer pertinent visual
information for the search query, and those that could not. Images in the
latter category were categorized as noise images. Lastly, we tallied the
quantity of sentences containing at least half of non-entity keywords and the
quantity of sentences harboring at least half of noise images among the
collected images.

As presented in Table 5, for the Multi30k dataset, out of 1000 sentences, only
27 sentences contained half or more non-entity keywords, and 61 sentences
gathered half or more noise images from search engines. However, in the WMT’16
(100k) dataset, there are 796 sentences with half or more non-entity keywords
and 685 sentences with half or more noise images, accounting for more than
half of the sampled sentences. Consequently, our method shows poor performance
on the WMT’16 (100k) dataset. The Bible dataset and MultiUN dataset exhibit a
similar situation. For the Global Voices dataset, there are 94 sentences with
half or more non-entity keywords and 228 sentences with half or more noise
images. This falls between the Multi30K and WMT’16 (100k) datasets. It is
interesting to note that the Multi30k dataset, which has the smallest
proportions of non-entity keywords and noise images, achieves the most
significant gain in translation performance. On the other hand, datasets with
the largest proportions of non-entity keywords and noise images show the
smallest gain in translation performance.

Method | Dataset
---|---
| Multi30k | Global Voices | WMT’16 (100k) | Bible | MultiUN
MMT with retrieved images | 38.43 | 9.81 | 8.41 | 35.42 | 39.53
(Tang et al., 2022) |  |  |  |  |
$+$ noise image filter | 38.50 | 10.12 | 8.89 | 36.12 | 39.91
$+$ noise region filter | 38.46 | 9.95 | 8.78 | 35.84 | 39.72
$+$ noise image & region filter | 38.51 | 10.23 | 8.93 | 36.38 | 39.95
Table 6: Results of image and region filtering method across diverse datasets
(BLEU score)

### 5.2. Influence of the Correlation between Text and Images

Table 6 shows the evaluation results of applying two filtering approaches
described in Section 3.2 in MMT. It is obvious that MMT models with both noise
image filter and noise region filter achieves the highest BLEU score across
all datasets, including Multi30k and the other four,underscoring the
effectiveness of these two filtering approaches. 888 A correct example
generated by MMT with visual noise filtering is presented in Table 9.

Notably, it is intriguing to note that the noise filtering techniques
exhibited more substantial enhancements in translation performance for the
WMT’16 (100k), Bible, and MultiUN datasets, in contrast to the improvements
observed in the Multi30k and Global Voices datasets. This further underscores
the significant impact of the correspondence between image and text content on
the translation performance the alignment and coherence between image and text
content on the translation performance of the MMT system. It also elucidates
why noise filtering methods yield marginal improvements on the Multi30K
dataset.

In conclusion, the translation performance of the multimodal model primarily
hinges on the consistency between textual and visual content. In other words,
the more alignment exists between textual and visual content, the greater
enhancement in translation performance with multimodal translation compared to
text-only translation. Hence, we arrive at a conclusion that aligns closely
with Caglayan et al. (2019), which suggest that multimodal translation models
predominantly treat visual information as a complement to textual information.

### 5.3. Exploring the Necessity of Visual Modality

Method | BLEU score
---|---
text-only NMT | 33.70
$+$ visual information | 38.43
(MMT with retrieved images) |
(Tang et al., 2022) |
$+$ textual information | 39.13
$+$ visual & textual information | 38.55
Table 7: Results on Multi30k using visual information or textual information
enhanced NMT

We conducted a quantitative comparison between MMT with retrieved images
(Section 3.1) and NMT with retrieved supplementary texts on the Multi30k
dataset. Table 7 shows the experimental results. In comparison to MMT model
employing images for translation enhancement, the approach integrating
supplementary textual data for translation enhancement demonstrated a
significantly higher BLEU score of 39.13. Remarkably, the combined utilization
of both images and supplementary texts for translation enhancement yielded a
BLEU score of 38.55, positioning itself between image-enhanced NMT and text-
enhanced NMT.

This demonstrates that both additional visual and supplementary textual
information play an entirely equivalent supplementary role in the translation
process. Moreover, in most cases, the utilization of supplementary textual
information assists the translation process more effectively. 999 A correct
example comparing NMT with retrieved supplementary texts to MMT with retrieved
images is presented in Table 10.

Therefore, we speculate that multimodal translation models trained on a large
volume of data might face challenges in outperforming text-only translation
models trained on comparable data volumes. This is because as the volume of
data used in multimodal model training increases, the potential impact of
visual information could diminish. We will verify this in future work.

## 6\. Conclusions

In this paper, we conduct an in-depth exploration into the role of visual
information within the multimodal translation process on Multi30k and other
four authentic translation datasets. Our findings emphasize that the
substantial correlation between visual and textual content significantly
impacts the efficacy of multimodal translation, while employing filtering
mechanisms based on the textual-visual correlation can enhance translation
performance. Additionally, experimental results reveal that visual information
plays a supplementary role in the multimodal translation process. This
supplementary function of visual information can be substituted by the
incorporation of supplementary textual information. As one of our future work,
we plan to assess the impact of the visual modality on more extensive
translation datasets, including the complete WMT’16 dataset. We speculate that
as multimodal translation models are trained using larger datasets, the impact
of visual information is likely to diminish.

## Acknowledgements

This research is supported by the Research Promotion Project of Key
Construction Discipline in Guangdong Province (2022ZDJS112).

## References

##

  * Aho and Ullman (1972) Alfred V. Aho and Jeffrey D. Ullman. 1972.  _The Theory of Parsing, Translation and Compiling_ , volume 1.  Prentice-Hall, Englewood Cliffs, NJ.
  * American Psychological Association (1983) American Psychological Association. 1983.  _Publications Manual_.  American Psychological Association, Washington, DC.
  * Ando and Zhang (2005) Rie Kubota Ando and Tong Zhang. 2005.  A framework for learning predictive structures from multiple tasks and unlabeled data.  _Journal of Machine Learning Research_ , 6:1817–1853.
  * Andrew and Gao (2007) Galen Andrew and Jianfeng Gao. 2007.  Scalable training of $L_{1}$-regularized log-linear models.  In _Proceedings of the 24th International Conference on Machine Learning_ , pages 33–40.
  * Bahdanau et al. (2014a) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014a.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Bahdanau et al. (2014b) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014b.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Barrault et al. (2018) Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , volume 2, pages 308–327.
  * Caglayan et al. (2016) Ozan Caglayan, Loïc Barrault, and Fethi Bougares. 2016.  Multimodal attention for neural machine translation.  _arXiv preprint arXiv:1609.03976_.
  * Caglayan et al. (2019) Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019.  Probing the need for visual context in multimodal machine translation.  In _Proceedings of the 2019 Conference of the North_ , pages 4159–4170. Association for Computational Linguistics.
  * Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017.  Incorporating global visual features into attention-based neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1003.
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-attentive decoder for multi-modal neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1913–1924.
  * Chandra et al. (1981) Ashok K. Chandra, Dexter C. Kozen, and Larry J. Stockmeyer. 1981.  Alternation.  _Journal of the Association for Computing Machinery_ , 28(1):114–133.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder-decoder approaches.  _arXiv preprint arXiv:1409.1259_.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: The Bible in 100 languages.  _Language Resources and Evaluation_ , 49(2):375–395.
  * Clark et al. (2011) Jonathan H Clark, Chris Dyer, Alon Lavie, and Noah A Smith. 2011.  Better hypothesis testing for statistical machine translation: Controlling for optimizer instability.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 176–181.
  * Cooley and Tukey (1965) James W. Cooley and John W. Tukey. 1965.  An algorithm for the machine calculation of complex Fourier series.  _Mathematics of Computation_ , 19(90):297–301.
  * Delbrouck and Dupont (2017a) Jean-Benoit Delbrouck and Stéphane Dupont. 2017a.  An empirical study on the effectiveness of images in multimodal neural machine translation.  _arXiv preprint arXiv:1707.00995_.
  * Delbrouck and Dupont (2017b) Jean-Benoit Delbrouck and Stephane Dupont. 2017b.  Multimodal compact bilinear pooling for multimodal neural machine translation.  _arXiv preprint arXiv:1703.08084_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Eisele and Chen (2010) Andreas Eisele and Yu Chen. 2010.  MultiUN: A multilingual corpus from united nation documents.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_.
  * Elliott (2018) Desmond Elliott. 2018.  Adversarial evaluation of multimodal machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2974–2978.
  * Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation_ , pages 215–233.
  * Elliott et al. (2015) Desmond Elliott, Stella Frank, and Eva Hasler. 2015.  Multilingual image description with neural sequence models.  _arXiv preprint arXiv:1510.04709_.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30k: Multilingual english-german image descriptions.  _arXiv preprint arXiv:1605.00459_.
  * Gehring et al. (2016) Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. 2016.  A convolutional encoder model for neural machine translation.  _arXiv preprint arXiv:1611.02344_.
  * Graves (2012) Alex Graves. 2012.  Long short-term memory.  _Supervised sequence labelling with recurrent neural networks_ , pages 37–45.
  * Grönroos et al. (2018a) Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018a.  The MeMAD submission to the WMT18 multimodal translation task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 603–611, Belgium, Brussels. Association for Computational Linguistics.
  * Grönroos et al. (2018b) Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, et al. 2018b.  The memad submission to the wmt18 multimodal translation task.  _arXiv preprint arXiv:1808.10802_.
  * Gusfield (1997) Dan Gusfield. 1997.  _Algorithms on Strings, Trees and Sequences_.  Cambridge University Press, Cambridge, UK.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Huang et al. (2016) Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016.  Attention-based multimodal neural machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 639–645.
  * Ive et al. (2019) Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019.  Distilling translations with visual awareness.  _arXiv preprint arXiv:1906.07701_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017.  Opennmt: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72.
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision_ , 128(7):1956–1981.
  * Lala et al. (2018) Chiraag Lala, Pranava Swaroop Madhyastha, Carolina Scarton, and Lucia Specia. 2018.  Sheffield submissions for WMT18 multimodal translation shared task.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 624–631, Belgium, Brussels. Association for Computational Linguistics.
  * Li et al. (2022) Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022.  On vision features in multimodal machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6327–6337, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2020a) Zuchao Li, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020a.  Explicit sentence compression for neural machine translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8311–8318.
  * Li et al. (2020b) Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020b.  Data-dependent gaussian prior objective for language generation.  In _Eighth International Conference on Learning Representations_.
  * Lin et al. (2020) Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020.  Dynamic context-guided capsule network for multimodal machine translation.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 1320–1329.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Rasooli and Tetreault (2015) Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015.  Yara parser: A fast and accurate dependency parser.  _Computing Research Repository_ , arXiv:1503.06733.  Version 2.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Specia et al. (2016) Lucia Specia, Stella Frank, Khalil Sima’An, and Desmond Elliott. 2016.  A shared task on multimodal machine translation and crosslingual image description.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 543–553.
  * Su et al. (2021) Jinsong Su, Jinchang Chen, Hui Jiang, Chulun Zhou, Huan Lin, Yubin Ge, Qingqiang Wu, and Yongxuan Lai. 2021.  Multi-modal neural machine translation with deep semantic interactions.  _Information Sciences_ , 554:47–60.
  * Tang et al. (2022) ZhenHao Tang, XiaoBing Zhang, Zi Long, and XiangHua Fu. 2022.  Multimodal neural machine translation with search engine based image retrieval.  In _Proceedings of the 9th Workshop on Asian Translation_ , pages 89–98.
  * Thomee et al. (2016) Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12)_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Turney (2000) Peter D Turney. 2000.  Learning algorithms for keyphrase extraction.  _Information retrieval_ , 2(4):303–336.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Venugopalan et al. (2015) Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2015.  Sequence to sequence-video to text.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4534–4542.
  * Witten et al. (2005) Ian H Witten, Gordon W Paynter, Eibe Frank, Carl Gutwin, and Craig G Nevill-Manning. 2005.  Kea: Practical automated keyphrase extraction.  In _Design and Usability of Digital Libraries: Case Studies in the Asia Pacific_ , pages 129–152. IGI global.
  * Wu et al. (2021) Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021.  Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6153–6166, Online. Association for Computational Linguistics.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015.  Show, attend and tell: Neural image caption generation with visual attention.  In _International conference on machine learning_ , pages 2048–2057. PMLR.
  * Zhang et al. (2017) Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Graham Neubig, and Satoshi Nakamura. 2017.  Nict-naist system for wmt17 multimodal translation task.  In _Proceedings of the Second Conference on Machine Translation_ , pages 477–482.
  * Zhang et al. (2019) Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utiyama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2019.  Neural machine translation with universal visual representation.  In _International Conference on Learning Representations_.
  * Zhang et al. (2020) Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020.  Semantics-aware bert for language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 9628–9635.
  * Zhao et al. (2021) Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. 2021.  Word-region alignment-guided multimodal neural machine translation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:244–259.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_ , pages 19–27.

##  Appendix A Qualitative Examples

In this appendix, we provide examples of retrieved images (Table 8), as well
as translation examples for MMT with visual noise filtering (Table 9) and NMT
with retrieved supplementary texts (Table 10).

Dataset | English Sentence | One of five retrieved images
---|---|---
Multi30k |  |  The person in the striped shirt is mountain climbing.
---
|
---
Global Voices |  |  Now the city is under a siege from the security forces.
---
|
---
WMT’16 (100k) |  |  In the future, integration will be a topic for the whole of society even more than it is today.
---
|
---
Bible |  |  You are Yahweh, even you alone. You have made heaven. the heaven of heavens, with all their army, the earth and all things that are on it, the seas and all that is in them and you preserve them all.
---
|
---
MultiUN |  |  Development assistance cannot by itself prevent or end conflict.
---
|
---
Table 8: Examples of retrieved image from different datasets. For the sentence from Multi30k dataset, our method efficiently retrieves an image that accurately represents the sentence’s content “A man is rock climbing”. For the sentence from Global Voice dataset, the retrieved image exhibits a degree of alignment with the source sentences, encompassing elements like “city”,“siege” and“forces”. However, for the sentence from WMT’16 (100k), Bible and MultiUN datasets, it becomes evident that the retrieved images offer limited relevant visual information and thus provide little assistance for translation. Source (En) |  But he answered and said, "Every plant which my heavenly Father didn’t plant will be uprooted.
---|---
Target (De) |  Aber er antwortete und sprach: Alle Pflanzen, die mein himmlischer Vater nicht pflanzte, die werden ausgereutet.
Retrieved images |
MMT with retrieved images |  Er antwortete aber und sprach: Alle Pflanzen, die mein himmlischer Vater nicht verderbte Quelle.
Retrieved images with noise image filter |
MMT with noise image filter |  Er antwortete aber und sprach: Alle Pflanzen, die mein himmlischer Vater nicht pflanzte.
MMT with both noise image and region filter |  Er antwortete aber und sprach: Alle Pflanzen, die mein himmlischer Vater nicht pflanzte, wird entwurzelt werden.
Table 9: A correct example generated by MMT with visual noise filtering. Due to its unique characteristics, the Bible dataset contains numerous entity words but is challenging to obtain images that effectively represent the textual content. However, visual noise filtering based on visual-text correlation can partially alleviate this situation. In this example, the filtered visual information has enabled the translation of “uprooted” to be correct. Source (En) |  Group of Asian boys wait for meat to cook over barbecue.
---|---
Target (De) |  Eine Gruppe asiatischer Jungen wartet am Grill darauf, dass Fleisch gar wird.
Text-only NMT |  Eine asiatische Jungen warten auf dem Fleisch, um den Grill zu kochen.
Retrieved images |
MMT with retrieved images |  Eine Gruppe von asiatischen Jungen wartet darauf, um Fleisch zu grillen.
Retrieved supplementary texts |  (1) Delivery is hardly limited to pizza at this point; everything from sushi to barbecue seems available as a to-go order. (2) While the savory aroma of barbecue filled the air, friends and family gathered around the grill, eagerly sharing stories and laughter as they waited for the delicious meal to be ready. (3) As the sun dipped below the horizon, our group of friends decided to have a barbecue in the backyard, lighting up the grill and eagerly waiting for the charcoal to heat up so that we could start cooking our favorite dishes. (4) At the lively outdoor barbecue gathering, a diverse group of friends, including a talented Asian chef, couldn’t wait to cook up a mouthwatering feast. (5) While the enthusiastic Asian group gathered around the barbecue, they took turns to cook their favorite dishes, making everyone else eagerly wait in anticipation of the delicious meal.
MMT with retrieved supplementary texts |  Eine Gruppe von asiatischen Jungen wartet darauf, dass Fleisch über Grill zukochen.
Table 10: A correct example generated by NMT with retrieved supplementary
texts. In this example, in contrast to text-only NMT without any supplementary
information, visual information and supplementary text information play an
equivalent role, correctly translating “Group” to “Gruppe”. Benefiting from
the rich information in the supplementary text, the NMT with retrieved
supplementary text achieves more accurate translations compared to MMT with
retrieved images.


Some Recent Results on Renormalization-Group Properties of Quantum Field
Theories

Robert Shrock

C. N. Yang Institute for Theoretical Physics and Department of Physics and
Astronomy, Stony Brook University, Stony Brook NY 11794, USA,
<EMAIL_ADDRESS>
XXXIII International (ONLINE) Workshop on High Energy Physics
“Hard Problems of Hadron Physics: Non-Perturbative QCD & Related Quests”
November 8-12, 2021 10.21468/SciPostPhysProc.?

## Abstract

We discuss some higher-loop studies of renormalization-group flows and fixed
points in various quantum field theories.

###### Contents

  1. 1 Introduction
  2. 2 Asymptotically Free Nonabelian Gauge Theories
  3. 3 RG Studies of Other Theories
    1. 3.1 Finite-$N$ Gross-Neveu Model
    2. 3.2 6D $\phi^{3}$ Theories
    3. 3.3 Studies of IR-free Theories, Including 4D U(1) and O($N$) $\lambda|{\vec{\phi}}|^{4}$
  4. 4 Asymptotically Free Chiral Gauge Theories
  5. 5 Conclusion

##  1 Introduction

A fundamental question in quantum field theory (QFT) concerns how the running
coupling of a theory changes as a function of the reference Euclidean
energy/momentum scale $\mu$ where it is measured. The variation of this
coupling with $\mu$ is described by the renormalization group (RG) beta
function of the theory. Here we will discuss some results that we have
obtained in this area. Much of this work was with T. A. Ryttov. We will focus
mainly on vectorial asymptotically free nonabelian gauge theories in $d=4$
dimensions, but also discuss some other asymptotically free theories, namely
the 2D finite-$N$ Gross-Neveu model and 6D $\phi^{3}$ theories, as well as
some infrared-free theories, including U(1) gauge theory, O($N$) $\phi^{4}$
theory, and chiral gauge theories..

##  2 Asymptotically Free Nonabelian Gauge Theories

Let us consider an asymptotically free (AF) vectorial nonabelian gauge theory
(in $d=4$ dimensions) with gauge group $G$ and $N_{f}$ massless fermions
$\psi_{j}$, $j=1,...,N_{f}$, transforming according to a representation $R$ of
$G$. We denote the running gauge coupling as $g(\mu)$ and define
$\alpha(\mu)\equiv g(\mu)^{2}/(4\pi)$ and $a(\mu)\equiv
g(\mu)^{2}/(16\pi^{2})$. The dependence of $\alpha(\mu)$ on $\mu$ is described
by the RG beta function, $\beta=d\alpha(\mu)/dt$, where $dt=d\ln\mu$. This has
the series expansion

$\beta=-2\alpha\sum_{\ell=1}^{\infty}b_{\ell}\,a^{\ell}\ ,$ (1)

where $b_{\ell}$ is the $\ell$-loop coefficient. For a general operator ${\cal
O}$, we denote the full scaling dimension as $D_{\cal O}$ and its free-field
value as $D_{{\cal O},free}$. The anomalous dimension of this operator,
denoted $\gamma_{\cal O}$, is defined via $D_{\cal O}=D_{{\cal
O},free}-\gamma_{\cal O}$. The coefficients $b_{1}$ and $b_{2}$ are
independent of the scheme used for regularization and renormalization and are
$b_{1}=(1/3)[11C_{A}-4T_{f}N_{f}]$ [1, 2] and
$b_{2}=(1/3)[34C_{A}^{2}-4(5C_{A}+3C_{f})N_{f}T_{f}]$ [3, 4], where $C_{2}(R)$
is the quadratic Casimir invariant, and $T(R)$ is the trace invariant, for the
representation $R$, and we use the notation $C_{2}(adj)\equiv C_{A}$,
$T(R)\equiv T_{f}$, and $C_{2}(R)\equiv C_{f}$. The AF condition means that
$b_{1}>0$, i.e., $N_{f}<N_{u}$, where $N_{u}=11C_{A}/(4T_{f})$. Since
$\alpha(\mu)$ is small at large $\mu$, one can self-consistently calculate
$\beta$ as a power series in $\alpha(\mu)$. As $\mu$ decreases from large
values in the ultraviolet (UV) to small values in the infrared (IR),
$\alpha(\mu)$ increases.

A situation of special interest occurs if $\beta$ has a zero at a nonzero
(physical) value $\alpha=\alpha_{IR}$. In the asymptotically free regime, this
happens if the condition $N_{u}>N_{f}>17C_{A}^{2}/[2(5C_{A}+3C_{f})T_{f}]$
holds, so that $b_{2}<0$. At the two-loop ($2\ell$) level, the zero in $\beta$
occurs at $\alpha_{IR,2\ell}=-4\pi b_{1}/b_{2}$. If $N_{f}$ is close enough to
$N_{u}$ that this IR zero of $\beta$ occurs at small enough coupling so that
the gauge interaction does not produce any spontaneous chiral symmetry
breaking (S$\chi$SB), then it is an exact IR fixed point (IRFP) of the RG. The
theory at this IRFP exhibits scale invariance and is inferred to exhibit
conformal invariance, whence the term “conformal window” for this regime. In
this IR limit, the theory is in a chirally symmetric, deconfined, nonabelian
Coulomb phase (NACP). If, on the other hand, as $\mu$ decreases and
$\alpha(\mu)$ increases toward $\alpha_{IR}$, there is a scale $\mu=\Lambda$
at which $\alpha(\mu)$ exceeds a critical value, $\alpha_{cr}$, for the
formation of a fermion condensate $\langle\bar{\psi}\psi\rangle$ with
associated S$\chi$SB, then the fermions gain dynamical masses of order
$\Lambda$. These fermions are then integrated out of the low-energy effective
field theory operative for $\mu<\Lambda$. In this case, $\alpha_{IR}$ is only
an approximate IRFP. We define $N_{f,cr}$ to be the critical value of $N_{f}$
such that as $N_{f}$ decreases below $N_{f,cr}$, there is S$\chi$SB. If
$N_{f}$ is only slightly less than $N_{u}$, so that $\alpha_{IR}$ is small,
then the theory at the IRFP is weakly coupled and is amenable to perturbative
analysis [5]. A case of interest for studies of physics beyond the Standard
Model (BSM) is $N_{f}$ slightly less than $N_{f,cr}$. In this case, there is
slow-running, quasi-conformal behavior of $\alpha(\mu)$ over an extended
interval of $\mu$. The dynamical breaking of the approximate scale
(dilatation) symmetry then leads to a light pseudo-Nambu-Goldstone boson, the
dilaton. In a BSM application, with the Higgs boson being at least partially a
dilaton, this might help to solve the fine-tuning problem of why the Higgs
mass is protected against large radiative corrections.

It is of interest to investigate the properties of IRFPs in these vectorial AF
gauge theories. Among these properies are the anomalous dimensions of (gauge-
invariant) operators, such as
$\bar{\psi}\psi=\sum_{i=1}^{N_{f}}\bar{\psi}_{i}\psi_{i}$, denoted
$\gamma_{\bar{\psi}\psi,IR}$. In general, one can express the anomalous
dimension $\gamma_{\bar{\psi}/\psi}$ as the series expansion

$\gamma_{\bar{\psi}\psi}=\sum_{\ell=1}^{\infty}c_{\ell}\,a^{\ell}\ ,$ (2)

where $c_{\ell}$ is the $\ell$-loop coefficient. Evaluating this with $\alpha$
set equal to the IRFP value, calculated to a given $n$-loop ($n\ell$) order
then yields $\gamma_{\bar{\psi}\psi,IR}$ to this order, denoted as
$\gamma_{\bar{\psi}\psi,IR,n\ell}$. Another operator of interest is ${\rm
Tr}(F_{\lambda\rho}F^{\lambda\rho})$, where $F^{b}_{\lambda\rho}$ is the
field-strength tensor (with $b$ a group index). The anomalous dimension
$\gamma_{F^{2},IR}$ of this operator at the IRFP satisfies
$\gamma_{F^{2}}=-\beta^{\prime}_{IR}$, where $\beta^{\prime}=d\beta/d\alpha$.

As $N_{f}$ decreases through the conformal regime, $\alpha_{IR}$ increases,
motivating higher-loop calculations of anomalous dimensions. We have carried
out this program of calculating the UV to IR renormalization-group evolution
and anomalous dimensions at an IRFP to higher-loop order in a series of
papers, many with T. A. Ryttov, including [6]-[23]. Our first calculations
were at the 4-loop level [6], and subsequently, we have extended these to the
5-loop level, with inputs (in the $\overline{\rm MS}$ scheme) up to the 5-loop
level from [24, 25]. (At the 4-loop level, see also [26]). Our calculations to
higher-loop order enable us to describe the IR properties of the theory
throughout a larger portion of the conformal window than would be possible
with the lowest-order (two-loop) results. As $N_{f}$ decreases below
$N_{f,cr}$, the properties of the IR theory change qualitatively, and the
perturbative calculations do not apply. A unitarity upper bound in the
conformal regime is $\gamma_{\bar{\psi}\psi,IR}<2$ (reviewed in [27]), and
studies of Schwinger-Dyson equations [28] suggest that the onset of S$\chi$SB
occurs if $\gamma_{\bar{\psi}\psi,IR}>1$. Thus, for a given $G$ and $R$, our
higher-loop calculations of $\gamma_{\bar{\psi}\psi,IR}$ yield estimates for
$N_{f,cr}$; in turn, this information is relevant for the above-mentioned BSM
theories.

There is an intensive ongoing program of research in the lattice gauge theory
community to study this physics. Much work has been done for $G={\rm SU}(3)$
with $R$ equal to the fundamental representation. For this theory,
$N_{u}=16.5$ (where a formal continuation from physical integer $N_{f}$ to
real $N_{f}$ is understood). There is not yet a consensus among lattice groups
concerning the value of $N_{f,cr}$ (i.e., the lower end of the conformal
window as a function of $N_{f}$) for this theory. As an example, we consider
the case $N_{f}=12$. Several lattice groups [29, 30, 31, 32, 33, 34] have
found that this theory is IR-conformal, while Ref. [35] has argued that it is
chirally broken and hence not IR-conformal. For our 5-loop analysis, we have
made use of Padé resummation methods in addition to direct analysis of series
expansions. As above, we denote our $n$-loop value of
$\gamma_{\bar{\psi}\psi,IR}$ as $\gamma_{\bar{\psi}\psi,IR,n\ell}$. We
calculate $\gamma_{\bar{\psi}\psi,IR,2\ell}=0.773$,
$\gamma_{\bar{\psi}\psi,IR,3\ell}=0.312$,
$\gamma_{\bar{\psi}\psi,IR,4\ell}=0.253$, and
$\gamma_{\bar{\psi}\psi,IR,5\ell}=0.255$. These results show reasonable
convergence at the 4-loop and 5-loop levels, and our values of
$\gamma_{\bar{\psi}\psi,IR,4\ell}$ and $\gamma_{\bar{\psi}\psi,IR,5\ell}$ are
in very good agreement with the values $\gamma_{\bar{\psi}\psi,IR}=0.23(6)$
[33] (in accord with [31, 32]) and $\gamma_{\bar{\psi}\psi,IR}=0.235(46)$ [34]
measured in lattice simulations. Our values are also in agreement with the
range of effective values reported in [35]. For $\beta^{\prime}_{IR}$ in this
$N_{f}=12$ theory, as calculated via power series in the IR coupling, we find
$\beta^{\prime}_{IR,2\ell}=0.360$, $\beta^{\prime}_{IR,3\ell}=0.295$, and
$\beta^{\prime}_{IR,4\ell}=0.282$. Again, these values show good convergence,
and the 4-loop value is in very good agreement with the value
$\beta^{\prime}_{IR}=0.26(2)$ obtained from lattice measurements [32]. In our
papers we have discussed corresponding comparisons with lattice results for
other gauge groups $G$, fermion representations $R$, and flavor numbers
$N_{f}$. We have also studied theories with fermions in multiple different
representations [23].

Since the $b_{\ell}$ for $\ell\geq 3$ and the $c_{\ell}$ for $\ell\geq 2$
depend on the scheme used for regularization and renormalization, it is
important to assess the effects of this scheme dependence. We have done this
in [10, 11, 12, 13, 14]. This scheme dependence is a generic feature of
higher-order perturbative calculations, e.g., in QCD. A scheme transformation
can be expressed as a mapping between $\alpha$ and $\alpha^{\prime}$, or
equivalently, $a$ and $a^{\prime}$, which we write as
$a=a^{\prime}f(a^{\prime})$, where $f(a^{\prime})$ is the scheme
transformation function. We can write $f(a^{\prime})$ as a series expansion

$f(a^{\prime})=1+\sum_{s=1}^{s_{max}}k_{s}(a^{\prime})^{s}\ ,$ (3)

where $s_{max}$ may be finite or infinite. In the new scheme, the beta
function is
$\beta_{\alpha^{\prime}}=-2\alpha^{\prime}\sum_{\ell=1}^{\infty}b^{\prime}_{\ell}\,(a^{\prime})^{\ell}$.
We have calculated the $b^{\prime}_{\ell}$ in terms of the $b_{\ell}$ and
$k_{s}$. In addition to the results $b_{1}^{\prime}=b_{1}$ and
$b_{2}^{\prime}=b_{2}$, we find

$b_{3}^{\prime}=b_{3}+k_{1}b_{2}+(k_{1}^{2}-k_{2})b_{1}\ ,$ (4)
$b_{4}^{\prime}=b_{4}+2k_{1}b_{3}+k_{1}^{2}b_{2}+(-2k_{1}^{3}+4k_{1}k_{2}-2k_{3})b_{1}\
,$ (5)

and so forth for higher $\ell$. We have specified a set of conditions that a
physically acceptable scheme transformation must satisfy and have shown that
although these can easily be satisfied in the vicinity of zero coupling, they
are not automatic, and can be quite restrictive, at a nonzero coupling, as is
relevant for an IRFP in an UV-free (AF) theory, or a UVFP in an IR-free
theory. As part of this work, we have constructed scheme transformations that
can map to a scheme with vanishing coefficients at loop level $\ell\geq 3$ in
the vicinity of the origin, but we have also shown that it is more difficult
to try to do this at a zero of $\beta$ away from the origin. We have applied
these results to assess the degree of scheme dependence in our higher-loop
calculations of anomalous dimensions at IRFPs in AF gauge theories and have
shown that this dependence is small. This is similar to the experience in QCD,
where calculations performed to higher order exhibited reduced scheme
dependence (e.g. [36] and references therein).

The anomalous dimensions of gauge-invariant operators at the IRFP are physical
and hence cannot depend on the scheme used for regularization and
renormalization. However, this property is not maintained by finite-order
perturbative series expansions beyond the lowest orders. It is therefore
useful to calculate these anomalous dimensions in a scheme-independent (SI)
manner [5, 37, 38]. To do this, one utilizes the fact that $\alpha_{IR}\to 0$
as $N_{f}\to N_{u}$. Hence, one can reexpress the anomalous dimensions as
series expansions in the manifestly scheme-independent variable
$\Delta_{f}=N_{u}-N_{f}$, rather than as power series in the IR coupling:

$\gamma_{\bar{\psi}\psi,IR}=\sum_{j=1}^{\infty}\kappa_{j}\,\Delta_{f}^{j}$ (6)

and

$\beta^{\prime}_{IR}=\sum_{j=1}^{\infty}d_{j}\,\Delta_{f}^{j}\ ,$ (7)

where $d_{1}=0$. In general, the calculation of the coefficient $\kappa_{j}$
in Eq. (6) requires, as inputs, the values of the $b_{\ell}$ for
$1\leq\ell\leq j+1$ and the $c_{\ell}$ for $1\leq\ell\leq j$. The calculation
of the coefficient $d_{j}$ in Eq. (7) requires, as inputs, the values of the
$b_{\ell}$ for $1\leq\ell\leq j$. We denote the truncation of these series to
maximal power $j=p$ as $\gamma_{\bar{\psi}\psi,IR,\Delta_{f}^{p}}$ and
$\beta^{\prime}_{IR,\Delta_{f}^{p}}$, respectively. With Ryttov we have
calculated (i) the $\kappa_{j}$ up to $j=4$, and thus the series expansion for
$\gamma_{\bar{\psi}\psi,IR}$ to $O(\Delta_{f}^{4})$, and (ii) the $d_{j}$ up
to $j=5$ and hence $\beta^{\prime}_{IR}$ to $O(\Delta_{f}^{5})$ for general
$G$ and $R$. We have studied a number of specific theories in detail,
including the gauge groups SU($N_{c}$) with $R$ equal to the fundamental,
adjoint, and rank-2 symmetric and antisymmetric tensor representations, and
similarly for SO($N_{c}$) and Sp($N_{c}$) for various $N_{c}$. For the
illustrative theory discussed above, namely SU(3) with $N_{f}=12$ fermions in
the fundamental representation, our calculations of
$\gamma_{\bar{\psi}\psi,IR}$ via Eq. (6) yield slightly larger values than our
calculations via Eq. (2), and our computations of $\beta^{\prime}_{IR}$ yield
slightly smaller values than those that we obtained via series expansions in
the IR coupling.

An interesting feature of our scheme-independent results is that $\kappa_{1}$
and $\kappa_{2}$ are manifestly positive, and this positivity also holds for
$\kappa_{3}$ and $\kappa_{4}$ for a general $G$ and all of the representations
$R$ that we have studied. This leads to two monotonicity properties in the
conformal regime: (i) for a fixed $p$ with $1\leq p\leq 4$, the anomalous
dimension $\gamma_{\bar{\psi}\psi,IR,\Delta_{f}^{p}}$ is a monotonically
increasing function of $\Delta_{f}$, i.e., increases monotonically with
decreasing $N_{f}$; (ii) for a fixed $N_{f}$,
$\gamma_{\bar{\psi}\psi,IR,\Delta_{f}^{p}}$ is a monotonically increasing
function of $p$ in the range $1\leq p\leq 4$. From our analyis of a ${\cal
N}=1$ supersymmetric SU($N_{c}$) gauge theory with $N_{f}$ conjugate pairs of
chiral superfields [19], we have found that this positivity property of the
$\kappa_{j}$ is true for all $j$

##  3 RG Studies of Other Theories

We have also performed higher-loop studies of RG flows and possible zeros of
beta functions for other theories, including (i) the 2D finite-$N$ Gross-Neveu
model [39], (ii) various $\phi^{3}$ theories in 6D [40, 41], (iii) 4D U(1)
gauge theory [42], (iv) 4D nonabelian gauge theories with $N_{f}>N_{u}$ [42],
and (v) 4D O($N$) $\lambda|\vec{\phi}|^{4}$ theory [43, 44, 45]. The theories
(i) and (ii) are UV-free (i.e., AF), while the theories (iii)-(v) are IR-free.
In these studies, we combined direct analyses of higher-loop beta functions
with Padé approximants and scheme transformations to derive results.

###  3.1 Finite-$N$ Gross-Neveu Model

The Gross-Neveu (GN) model [46] is a 2D QFT with an $N$-component massless
fermion, $\psi_{j}$, $j=1,...,N$ and a four-fermion interaction. This model
has been of interest because it exhibits some properties similar to QCD,
namely asymptotic freedom and formation of massive bound states of fermions.
The model was solved exactly in the $N\to\infty$ limit in [46]. In this limit,
the beta function has no IR zero. This leaves open the question of whether the
beta function has an IR zero for finite $N$. We investigated this in [39],
using the beta function up to the 4-loop level from [47]. We found that, where
the perturbative calculation of the beta function is reliable, it does not
exhibit robust evidence for an IR zero.

###  3.2 6D $\phi^{3}$ Theories

$\phi^{3}$ theories in $d=6$ dimensions are asymptotically free, and it is of
interest to investigate whether they exhibit IRFPs. We have done this in [40]
with Gracey and Ryttov, using beta functions calculated up to the 4-loop
order. As before, without loss of generality, we take the matter field to be
massless, since a $\phi$ field with nonzero mass $m_{\phi}$ would be
integrated out of the low-energy effective theory for momentum scales
$\mu<m_{\phi}$ and hence is not relevant for the IR limit $\mu\to 0$. We have
studied $\phi^{3}$ theories with a real 1-component $\phi$ field and also with
an $N$-component field $\phi_{i}$ transforming according to the fundamental
representation of a global SU($N$) symmetry, with a self-interaction $\propto
d_{ijk}\phi^{i}\phi^{j}\phi^{k}+h.c.$. For both of these theories, we find
evidence against an IRFP. An interesting study of $\phi^{3}$ theory in a 6D
spacetime with two compact dimensions by Kisselev and Petrov is [48]. In [41],
we show that if a beta function is not identically zero but has a vanishing
one-loop term, then it is not, in general, possible to use scheme
transformations to eliminate $\ell$-loop terms with $\ell\geq 3$ in the beta
function, even in the vicinity of the origin in coupling constant space.

###  3.3 Studies of IR-free Theories, Including 4D U(1) and O($N$)
$\lambda|{\vec{\phi}}|^{4}$

If the $\beta$ function of a theory is positive near zero coupling, then this
theory is IR-free; as the reference scale $\mu$ decreases, the coupling
decreases toward 0. As $\mu$ increases from the IR, the coupling increases,
and a basic question is whether the beta function has a UV zero (in the
perturbatively calculable range), which would be a UV fixed point (UVFP) of
the RG.

An explicit example of a UVFP in an IR-free theory occurs in the O($N$)
nonlinear $\sigma$ model in $d=2+\epsilon$ dimensions. From a solution of this
model in the $N\to\infty$ limit, one finds, for small $\epsilon$ [49, 50],

$\beta(\xi)=\epsilon\xi\Big{(}1-\frac{\xi}{\xi_{c}}\Big{)}\ ,$ (8)

where $\xi$ is the effective coupling and $\xi_{c}=2\pi\epsilon$. Hence,
assuming that $\xi$ is small for small $\mu$, it follows that
$\lim_{\mu\to\infty}\xi(\mu)=\xi_{c}$, so the theory has a UVFP at $\xi_{c}$.

Let us consider a 4D U(1) gauge theory with $N_{f}$ fermions with a charge
$q$. This theory is IR-free, and the 1-loop and 2-loop coefficients in $\beta$
have the same sign, so there is no UV zero in $\beta$ at the maximal scheme-
independent order. In [42] we investigated a possible UVFP at higher-loop
order. One part of our work in [42] was an analysis of the beta function using
the 5-loop coefficient [51, 52]. Another part made use of exact closed-form
results for $N_{f}\to\infty$ [53]. In [42] we also performed a corresponding
investigation of possible UVFP for a nonabelian gauge theory with
$N_{f}>N_{u}$. In both the U(1) and nonabelian case, we found evidence against
a UVFP. Of course, in neither case does this imply that the theory has a
Landau pole, because the running gauge coupling gets too large for
perturbative calculations to be reliable before one actually reaches this
would-be pole.

In [43, 44, 45] we investigated the RG behavior of 4D O($N$)
$\lambda|{\vec{\phi}}|^{4}$ theory to six-loop order, using $b_{5}$ from [54]
and $b_{6}$ from [55] (in the $\overline{\rm MS}$ scheme). Again, for values
of the interaction coupling where the perturbative (and Padé resummation)
methods were applicable, we did not find robust evidence for a UVFP.

##  4 Asymptotically Free Chiral Gauge Theories

The analysis of asymptotically free chiral gauge theories is also of
considerable interest. The (massless) fermion content is chosen so as to avoid
any gauge anomalies, mixed gauge-gravitational anomalies, and global anomaly.
As the theory flows from the UV to the IR and the coupling grows, several
possible types of behavior can occur, including (i) an exact IRFP in a
conformal phase; (ii) bilinear fermion condensate formation with dynamical
breaking of gauge and global symmetries; or (iii) confinement with formation
of massless composite fermions. These theories have been of interest for BSM
physics (e.g, [56]). Our works in this area include [57]-[62], which contain
references to the extensive literature.

##  5 Conclusion

Studies of RG flows and possible RG fixed points in quantum field theories
continue to be of great interest, both from the point of view of formal theory
and for applications to BSM physics. Here we have briefly discussed some of
our results on higher-loop perturbative calculations with inputs up to the
five-loop level for anomalous dimensions at IR fixed points in asymptotically
free nonabelian gauge theories and comparisons of these results with lattice
measurements. We have also discussed our results on RG flows and investigation
of possible RG fixed points for several other UV-free theories and for several
IR-free theories. There are many opportunities for further work in this area.

## Acknowledgements

I am grateful to T. A. Ryttov and my other coauthors for fruitful
collaborations. I would also like to thank V. A. Petrov and the organizers for
kindly inviting me to give this talk at the XXXIII International Workshop on
High-Energy Physics at the Logunov IHEP.

#### Funding information

The research of R.S. was supported in part by the U.S. National Science
Foundation under the grants NSF-16-20628 and NSF-19-15093.

## References

  * [1] D. J. Gross and F. Wilczek, Ultraviolet Behavior of Nonabelian Gauge Theories, Phys. Rev. Lett. 30, 1343 (1973), 10.1103/PhysRevLett.30.1343.
  * [2] H. D. Politzer, Reliable Perturbative Results for Strong Interactions?, Phys. Rev. Lett. 30, 1346 (1973), 10.1103/PhysRevLett.30.1346.
  * [3] W. E. Caswell, Asymptotic Behavior of Nonabelian Gauge Theories to Two-Loop Order, Phys. Rev. Lett. 33, 244 (1974), 10.1103/PhysRevLett.33.244.
  * [4] D. R. T. Jones, Two-Loop Diagrams in Yang-Mills Theory, Nucl. Phys. B 75, 531 (1974), 10.1016/0550-3213(74)90093-5.
  * [5] T. Banks and A. Zaks, On the Phase Structure of Vector-Like Gauge Theories with Massless Fermions, Nucl. Phys. B 196, 189 (1982), 10.1016/0550-3213(82)90035-9.
  * [6] T. A. Ryttov and R. Shrock, Higher-Loop Corrections to the Infrared Evolution of a Gauge Theory with Fermions, Phys. Rev. D 83, 056011 (2011), 10.1103/PhysRevD.83.056011.
  * [7] R. Shrock, Higher-Loop Structural Properties of the $\beta$ Function in Asymptotically Free Vectorial Gauge Theories, Phys. Rev. D 87, 105005 (2013), 10.1103/PhysRevD.87.105005.
  * [8] R. Shrock, Higher-Loop Calculations of the Ultraviolet to Infrared Evolution of a Vectorial Gauge Theory in the Limit $N_{c}\to\infty$, $N_{f}\to\infty$ with $N_{f}/N_{c}$ Fixed, Phys. Rev. D 87, 116007 (2013), 10.1103/PhysRevD.87.116007.
  * [9] T. A. Ryttov and R. Shrock, Infrared Zero of $\beta$ and Value of $\gamma_{m}$ for an SU(3) Gauge Theory at the Five-Loop Level, Phys. Rev. D 94, 105015 (2016), 10.1103/PhysRevD.94.105015.
  * [10] T. A. Ryttov and R. Shrock, An Analysis of Scheme Transformations in the Vicinity of an Infrared Fixed Point, Phys. Rev. D 86, 085005 (2012), 10.1103/PhysRevD.86.085005.
  * [11] R. Shrock, Study of Scheme Transformations to Remove Higher-Loop Terms in the $\beta$ Function of a Gauge Theory, Phys. Rev. D 88, 036003 (2013), 10.1103/PhysRevD.88.036003.
  * [12] R. Shrock, Generalized Scheme Transformations for the Elimination of Higher-Loop Terms in the Beta Function of a Gauge Theory, Phys. Rev. D 90, 045011 (2014), 10.1103/PhysRevD.90.045011.
  * [13] G. Choi and R. Shrock, New Scheme Transformations and Application to Study Scheme Dependence of an Infrared Zero of the Beta Function in Gauge Theories, Phys. Rev. D 90, 125029 (2014), 10.1103/PhysRevD.90.125029.
  * [14] G. Choi and R. Shrock, An Integral Formalism for the Construction of Scheme Transformations in Quantum Field Theory, Phys. Rev. D 94, 065038 (2016), 10.1103/PhysRevD.94.065038.
  * [15] T. A. Ryttov and R. Shrock, Scheme-Independent Calculation of $\gamma_{\bar{\psi}\psi}$ for an SU(3) Gauge Theory, Phys. Rev. D 94, 105014 (2016), 10.1103/PhysRevD.94.105014.
  * [16] T. A. Ryttov and R. Shrock, Scheme-Independent Series Expansions at an Infrared Zero of the Beta Function in Asymptotically Free Gauge Theories, Phys. Rev. D 94 125005 (2016), 10.1103/PhysRevD.94.125005.
  * [17] T. A. Ryttov and R. Shrock, Higher-Order Scheme-Independent Series Expansions of $\gamma_{\bar{\psi}\psi,IR}$ and $\beta^{\prime}_{IR}$ in Conformal Field Theories, Phys. Rev. D 95, 105004 (2017), 10.1103/PhysRevD.95.105004.
  * [18] T. A. Ryttov and R. Shrock, Infrared Fixed Point Physics in ${\rm SO}(N_{c})$ and ${\rm Sp}(N_{c})$ Gauge Theories, Phys. Rev. D 96, 105015 (2017), 10.1103/PhysRevD.96.105015.
  * [19] T. A. Ryttov and R. Shrock, Scheme-Independent Calculations of Physical Quantities in an ${\cal N}=1$ Supersymmetric Gauge Theory, Phys. Rev. D 96, 105018 (2017), 10.1103/PhysRevD.96.105018.
  * [20] T. A. Ryttov and R. Shrock, Physics of the Non-Abelian Coulomb Phase: Insights from Padé Approximants, Phys. Rev. D 97, 025004 (2018), 10.1103/PhysRevD.97.025004.
  * [21] T. A. Ryttov and R. Shrock, Scheme-Independent Series for Anomalous Dimensions of Higher-Spin Operators at an Infrared Fixed Point in a Gauge Theory, Phys. Rev. D 101, 076018 (2020), 10.1103/PhysRevD.101.076018.
  * [22] J. A. Gracey, T. A. Ryttov, and R. Shrock, Scheme-Independent Calculations of Anomalous Dimensions of Baryon Operators in Conformal Field Theories, Phys. Rev. D 97, 116018 (2018), 10.1103/PhysRevD.97.116018.
  * [23] T. A. Ryttov and R. Shrock, Scheme-Independent Series Calculations of Properties at a Conformal Infrared Fixed Point in Gauge Theories with Multiple Fermion Representations, Phys. Rev. D 98, 096003 (2018), 10.1103/PhysRevD.98.096003.
  * [24] P. A. Baikov, K. G. Chetyrkin, and J. H. Kühn, Five-Loop Running of the QCD Coupling Constant, Phys. Rev. Lett. 118, 082002 (2017), 10.1103/PhysRevLett.118.082002.
  * [25] F. Herzog, B. Ruijl, T. Ueda, J. A. M. Vermaseren, and A. Vogt, The Five-loop Beta Function of Yang-Mills Theory with Fermions, JHEP 02(2017)090, 10.1007/JHEP02(2017)090.
  * [26] C. Pica and F. Sannino, Ultraviolet and Infrared Zeros of Gauge Theories at the Four-loop Order and Beyond, Phys. Rev. D 83, 035013 (2011), 10.1103/PhysRevD.83.035013.
  * [27] Y. Nakayama, Scale Invariance vs. Conformal Invariance, Phys. Rept. 569, 1 (2015), 10.1016/j.physrep.2014.12.003.
  * [28] T. Appelquist, K. D. Lane, and U. Mahanta, On the Ladder Approximation for Spontaneous Chiral Symmetry Breaking, Phys. Rev. Lett. 61, 1553 (1988), 10.1103/PhysRevLett.61.1553.
  * [29] T. Appelquist, G. Fleming, and E. Neil, Lattice Study of the Conformal Window in QCD-like Theories, Phys. Rev. Lett. 100, 171607 (2008), 10.1103/PhysRevLett.100.171607.
  * [30] T. Appelquist, G. Fleming, and E. Neil, Lattice Study of Conformal Behavior in SU(3) Yang-Mills Theories, Phys. Rev. D 79, 076010 (2009), 10.1103/PhysRevD.79.076010.
  * [31] A. Cheng, A. Hasenfratz, G. Petropoulos, and D. Schaich, Scale-Dependent Mass Anomalous Dimension from Dirac Eigenmodes, JHEP 07 (2013) 061, 10.1007/JHEP07(2013)061.
  * [32] A. Hasenfratz and D. Schaich, Nonperturbative $\beta$ Function of Twelve-flavor SU(3) Gauge Theory, JHEP 02 (2018)132, 10.1007/JHEP02(2018)132.
  * [33] A. Carosso, A. Hasenfratz, and E. T. Neil, Nonperturbative Renormalization of Operators in Near-Conformal Systems Using Gradient Flows, Phys. Rev. Lett. 121, 201601 (2018), doi10.1103/PhysRevLett.121.201601.
  * [34] M. P. Lombardo, K. Miura, T. J. Nunes da Silva, and E. Pallante, On the Particle Spectrum and the Conformal Window, JHEP 12(2014)183, 10.1007/JHEP12(2014)183.
  * [35] Z. Fodor, K. Holland, J. Kuti, D. Nogradi, and C.-H. Wong, Fate of the Conformal Fixed Point with Twelve Massless Fermions and SU(3) Gauge Group, Phys. Rev. D 94, 091501 (2016), 10.1103/PhysRevD.94.091501.
  * [36] S. J. Brodsky, M. Mojaza, and X.-G. Wu, Systematic Scale-Setting to All Orders: The Principle of Maximum Conformality and Commensurate Scale Relations, Phys. Rev. D 89, 014027 (2014), 10.1103/PhysRevD.89.014027.
  * [37] G. Grunberg, Method of Effective Charges and Brodsky-Lepage-Mackenzie Criterion, Phys. Rev. D 46, 2228 (1992), 10.1103/PhysRevD.46.2228.
  * [38] T. A. Ryttov, Consistent Perturbative Fixed Point Calculations in QCD and Supersymmetric QCD, Phys. Rev. Lett. 117, 071601 (2016), 10.1103/PhysRevD.94.105014.
  * [39] G. Choi, T. A. Ryttov, and R. Shrock, On the Question of a Possible Infrared Zero in the Beta Function of the Finite-N Gross-Neveu Model, Phys. Rev. D 95, 025012 (2017), 10.1103/PhysRevD.95.025012.
  * [40] J. A. Gracey, T. A. Ryttov, and R. Shrock, Renormalization-Group Behavior of $\phi^{3}$ Theories in $d=6$ Dimensions, Phys. Rev. D 102, 045016 (2020), 10.1103/PhysRevD.102.045016.
  * [41] T. A. Ryttov and R. Shrock, Effect of Scheme Transformations on a Beta Function with Vanishing One-Loop Term, Phys. Rev. D 102, 056016 (2020), 10.1103/PhysRevD.102.056016.
  * [42] R. Shrock, Study of UV Zero of the Beta Function in Gauge Theories with Many Fermions, Phys. Rev. D 89, 045019 (2014), 10.1103/PhysRevD.89.045019.
  * [43] R. Shrock, On the Question of a Zero in the Beta Function of the $\lambda(\vec{\phi}^{\ 2})^{2}_{4}$ Theory, Phys. Rev. D 90, 065023 (2014) 10.1103/PhysRevD.90.065023.
  * [44] R. Shrock, Study of the Six-Loop Beta Function of the $\lambda\phi_{4}^{4}$ Theory, Phys. Rev. D 94, 125026 (2016), 10.1103/PhysRevD.94.125026.
  * [45] R. Shrock, Study of the Question of an Ultraviolet Zero in the Six-Loop Beta Function of the O($N$) $\lambda|\vec{\phi}|^{4}$ Theory, Phys. Rev. D 96, 056010 (2017), 10.1103/PhysRevD.96.056010.
  * [46] D. J. Gross and A. Neveu, Dynamical Symmetry Breaking in Asymptotically Free Field Theories, Phys. Rev. D 10, 3235 (1974), 10.1103/PhysRevD.10.3235.
  * [47] J. A. Gracey, T. Luthe, and Y. Schröder, Four Loop Renormalization of the Gross-Neveu Model, Phys. Rev. D 94, 125028 (2016), 10.1103/PhysRevD.94.125028.
  * [48] A. V. Kisselev and V. A. Petrov, Can Effective Four-Dimensional Scalar Theory be Asymptotically Free in a Spacetime with Higher Dimensions?, Phys. Rev. D 103, 085012 (2021), 10.1103/PhysRevD.103.085012.
  * [49] E. Brézin and J. Zinn-Justin, Spontaneous Breakdown of Continuous Symmetries Near Two Dimensions, Phys. Rev. B 14, 3110 (1976), 10.1103/PhysRevB.14.3110.
  * [50] W. A. Bardeen, B. W. and R. E. Shrock, Phase Transition in the Nonlinear $\sigma$ Model in a $2+\epsilon$ Dimensional Continuum, Phys. Rev. D 14, 985 (1976), 10.1103/PhysRevD.14.985.
  * [51] A. L. Kataev and S. A. Larin (2012) Analytical Five-loop Expressions for the Renormalization Group QED $\beta$-Function in Different Renormalization Schemes, JETP Lett. 96, 64 (2012), 10.1134/S0021364012130073.
  * [52] P. A. Baikov, K. G. Chetyrkin, J. H. Kühn, and J. Rittinger, Vector Correlator in Massless QCD at Order $O(\alpha_{s}^{4})$ and the QED Beta-Function at Five Loops, JHEP 07(2012)017, 10.1007/JHEP07(2012)017z.
  * [53] A. Palanques-Mestre and P. Pascual, The $1/N_{f}$ Expansion of the Gamma and Beta Functions in QED Commun. Math. Phys. 95, 277 (1984), 10.1007/BF01212398.
  * [54] H. Kleinert, J. Neu, V. Schulte-Frohlinde, K. G. Chetyrkin, and S. A. Larin, Five Loop Renormalization Group Functions of $O(n)$ Symmetric $\phi^{4}$ Theory and Epsilon Expansions of Critical Exponents up to $\epsilon^{5}$, Phys. Lett. B 272, 39 (1991), 10.1016/0370-2693(91)91009-K.
  * [55] V. Kompaniets and E. Panzer, Minimally Subtracted Six-Loop Renormalization of $O(n)$-Symmetric $\phi^{4}$ Theory and Critical Exponents, Phys. Rev. D 96, 036016 (2017), 10.1103/PhysRevD.96.036016.
  * [56] S. Raby, S. Dimopoulos, and L. Susskind, Tumbling Gauge Theories, Nucl. Phys. B 169, 373 (1980), 10.1016/0550-3213(80)90093-0.
  * [57] T. Appelquist, A. Cohen, M. Schmaltz, and R. Shrock, New Constraints on Chiral Gauge Theories, Phys. Lett. B 459, 235 (1999), 10.1016/S0370-2693(99)00616-4.
  * [58] T. Appelquist and R. Shrock, Neutrino Masses in Theories with Dynamical Electroweak Symmetry Breaking, Phys. Lett. B 548, 204 (2002), 10.1016/S0370-2693(02)02854-X.
  * [59] T. Appelquist and R. Shrock, Dynamical Symmetry Breaking of Extended Gauge Symmetries, Phys. Rev. Lett. 90, 201801 (2003), 10.1103/PhysRevLett.90.201801.
  * [60] T. Appelquist and R. Shrock, On the Ultraviolet to Infrared Evolution of Chiral Gauge Theories, Phys. Rev. D 88, 105012 (2013), 10.1103/PhysRevD.88.105012.
  * [61] Y. Shi and R. Shrock, Renormalization-Group Evolution and Nonperturbative Behavior of Chiral Gauge Theories with Fermions in Higher-Dimensional Representations, Phys. Rev. D 92, 125009 (2015), 10.1103/PhysRevD.92.125009.
  * [62] Y. Shi and R. Shrock, $A_{k}\,\bar{F}$ Chiral Gauge Theories, Phys. Rev. D 92, 105032 (2015), 10.1103/PhysRevD.92.105032.