# Average Token Delay: A Duration-aware Latency Metric for Simultaneous
Translation

Yasumasa Kano⋄ Katsuhito Sudoh⋄ Satoshi Nakamura⋄

⋄Nara Institute of Science and Technology / 8916-5 Takayama-cho, Ikoma, Nara,
Japan
<EMAIL_ADDRESS>
###### Abstract

Simultaneous translation is a task in which the translation begins before the
end of an input speech segment. Its evaluation should be conducted based on
latency in addition to quality, and for users, the smallest possible amount of
latency is preferable. Most existing metrics measure latency based on the
start timings of partial translations and ignore their duration. This means
such metrics do not penalize the latency caused by long translation output,
which delays the comprehension of users and subsequent translations. In this
work, we propose a novel latency evaluation metric for simultaneous
translation called _Average Token Delay_ (ATD) that focuses on the duration of
partial translations. We demonstrate its effectiveness through analyses
simulating user-side latency based on Ear-Voice Span (EVS). In our experiment,
ATD had the highest correlation with EVS among baseline latency metrics under
most conditions.

##  1 Introduction

Machine translation (MT), which has evolved rapidly due to recent neural
network techniques, is now widely used for both written and spoken languages.
MT for speech attracts the translation of various conversations, lecture,
talks, etc. In situations requiring real-time communication, MT should
function simultaneously without waiting for the speech’s conclusion. Such a
task is often called _simultaneous machine translation_ (SimulMT).
Hereinafter, the term SimulMT covers both text and speech inputs. One crucial
challenge in SimulMT is the quality-latency trade-off. Although taking a
longer input with a longer wait can improve the translation quality,
unfortunately it also creates more latency, and vice versa.

Figure 1: Example calculation of mean Ear-Voice Span (EVS): Black lines
represent word alignment, and time distance of the lines is EVS. This example
excludes word alignment for stop words and function words. The mean EVS is
smaller in case 1; AL is smaller in case 2.

In SimulMT studies, we compare models in such a quality-latency trade-off. In
the research of human interpretation, Ear-Voice Span (EVS) has been used to
evaluate latency. Basically, EVS is the time difference between the source
word and the corresponding target word, although variations can be found Robbe
(2019).

Figure 1 illustrates an example calculation of a mean EVS. For both cases, the
first input chunk is translated into the first output chunk, and the second
one is translated next. In this example, EVS is represented as the time
distance of the black lines connecting the source and the semantically
corresponding target word. When we calculate the mean value of these EVSs,
case 1 has a smaller value. Therefore, the latency of case 1 is smaller than
that of case 2.

Although EVS is a persuasive latency metric, it has not been used in SimulMT
research. One reason is that SimulMT research emerged from text-to-text
simultaneous translation while EVS is a latency metric for speech-to speech
simultaneous translation by human interpreters. Therefore, latency metrics
have been proposed that can calculate the latency of text-to-text simultaneous
machine translation.

One example is Average Lagging (AL; Ma et al., 2019), which is the most
commonly used latency metric. AL, which is based on the number of input words
that are available when starting a translation, measures the average amount of
input words over the whole translation process. It matches wait-$k$ SimulMT
(Ma et al., 2019) that waits for the $k$ input tokens before starting the
translation and alternately repeats reading and writing one token. In Figure
1, AL is smaller in case 2, which does not agree with its mean EVS. This
difference comes from the definition of AL, which counterintuitively gives a
smaller latency value to a long chunk output at a time step. Such a long chunk
output causes inevitable delays for the translation of subsequent parts in
speech-to-speech SimulST because the speech synthesis cannot start a new
speech output while _speaking_. Even for closed caption output, long outputs
in a short time period cause a high cognitive load for users. Therefore, since
a translation output’s length affects user experiences, this length should be
considered in latency measurements. This observation suggests the need for
another latency metric to cope with such situations.

In this work, we propose a novel latency metric called _Average Token Delay_
(ATD)111ATD is implemented in https://github.com/facebookresearch/SimulEval.
that focuses on the end timings of partial translations in SimulMT. ATD
generalizes latency measurements for both speech and text outputs and works
intuitively for chunk-based outputs that are not properly handled by AL, as
discussed above. ATD is much simpler than EVS, which requires a long process
like transcribing input and output speech, getting word timestamps, and
obtaining word alignment by annotators. We present simulation results that
clarify ATD’s characteristics and demonstrate its effectiveness through
SimulMT experiments and comparisons with ATD with baseline latency metrics.
Since no experiments have compared latency metrics, no evaluation dataset
exists. Therefore, we created a small dataset and used it to compare the
correlation between each latency metric and the mean EVS. In our experiments,
ATD had the highest correlation among the baseline latency metrics in most
conditions.

##  2 Simultaneous Machine Translation

First, we review the SimulMT formulation to share the notation used in this
paper.

In standard sentence-level NMT, let $\bm{x}=x_{1},x_{2},...,x_{m}$ be an input
sentence and $\bm{y}=y_{1},y_{2},...,y_{n}$ be its translation, the output
probability is denoted as

$p(\bm{y}|\bm{x})=\prod_{t=1}^{n}P(y_{t}|\bm{x},\bm{y}_{<t}).$ (1)

SimulMT takes a prefix of the input for its incremental decoding:

$p(\bm{y}|\bm{x})=\prod_{t=1}^{n}P(y_{t}|\bm{x}_{\leq g(t)},\bm{y}_{<t}),$ (2)

where $g(t)$ is a monotonic non-decreasing function that represents the number
of input tokens read until the prediction of $t$-th output token $y_{t}$, so
that $\bm{x}_{\leq g(t)}$ means an input prefix, and $\bm{y}_{<t}$ is the
prefix translation predicted so far. This means that we can obtain a pair of
an input prefix and a corresponding output prefix $(\bm{x}_{\leq
g(t)},\bm{y}_{\leq t})$ by that time.

The incremental decoding can be represented by a sequence of actions, READ and
WRITE. READ is an action that takes a new input, typically one token for the
text input or a fixed number of frames for the speech input. WRITE is an
action that predicts an output, typically one token for a text output or a
corresponding speech signal for the speech output.

##  3 Existing Latency Metrics for Simultaneous Translation

Several latency metrics have been proposed in the SimulMT field. In this
section, we review them before we propose ATD.

Gu et al. (2017) proposed a latency metric called the Consecutive Wait length
(CW), which counts the number of consecutive waited input tokens between two
output tokens and measures the local delays in a sentence. Cho and Esipova
(2016) proposed an AP that measures the average latency for an entire
sentence. However, AP suffers from the following problem: the latency value
differs depending on the input and the output sequence lengths even for the
same READ-WRITE strategy.

Ma et al. (2019) proposed Average Lagging (AL), which has recently become the
most commonly used method, denoted as follows:

$AL_{g}(\bm{x},\bm{y})=\frac{1}{\tau_{g}(|\bm{x}|)}\sum_{t=1}^{\tau_{g}(|\bm{x}|)}\left(g(t)-\frac{t-1}{r}\right),$
(3)

where $r$ is the length ratio defined as $|\bm{y}|/|\bm{x}|$ and
$\tau_{g}(|\bm{x}|)$ is the cut-off step:

$\tau_{g}(|\bm{x}|)=\min\\{t\mid g(t)=|\bm{x}|\\},$ (4)

denoting the index of the output token predicted right after the observation
of the entire source sentence.

However, AL still suffers from unintuitive latency measurement because it can
be negative when the model finishes the translation before reading the entire
input. This is because if $|\bm{y}|<<|\bm{x}|$, then $r<<1$, and so the second
term $(t-1)/r$ in the subtraction becomes too large. To mitigate this problem,
Ma et al. (2020) modified AL by changing the calculation of length ratio $r$
to $|\bm{y^{*}}|/|\bm{x}|$ based on the length of reference translation
$\bm{y^{*}}$. Papi et al. (2022) proposed Length-Adaptive Average Lagging
(LAAL), which modified $r$ to $max(|\bm{y}|,|\bm{y^{*}}|)/|\bm{x}|$ to
appropriately evaluate the latency of a translation that is longer than the
reference. However, these modifications are not enough to deal with the
problem of AL in which the latency becomes smaller for longer partial output.
When the partial translation output becomes longer, $t$ increases and
$(t-1)/r$ becomes larger although $g(t)$ does not change. As a consequence,
the result of subtraction in Equation 3 is reduced.

Arivazhagan et al. (2019) proposed another AL variant called Differentiable
Average Lagging (DAL), which can optimize a simultaneous translation model:

$DAL_{g}(\bm{x},\bm{y})=\frac{1}{|\bm{y}|}\sum_{t=1}^{|\bm{y}|}\left(g^{\prime}(t)-\frac{t-1}{r}\right),$
(5)

$g^{\prime}(t)=\begin{cases}g(t)&t=1\\\
\max(g(t),g^{\prime}(t-1)+r)&t>1\end{cases}.$ (6)

DAL replaces the $g(t)$ of AL with $g^{\prime}(t)$ and does not use cut-off
step $\tau_{g}(|\bm{x}|)$. DAL mitigates the AL problem that long chunk output
reduces the delay, although it remains insufficient. Suppose
$|\bm{x}|=|\bm{y}|$ ($r=1$). When the partial translation output becomes
longer than the partial input, $g^{\prime}(t-1)+r$ exceeds $g(t)$ in Equation
6. In this situation, every time a new target token is output, $g^{\prime}(t)$
increases by one, as does $(t-1)/r$. Therefore, the difference between the two
terms in Equation 5 is not changed by long output; nor does the delay
increase. Even though a long output should delay the start of the next chunk
translation, such delay is not counted in DAL.

The above latency metrics are proposed to evaluate sentence-level SimulMT, and
so Iranzo-Sánchez et al. (2021) proposed a method to calculate them for
streaming input by converting the global delay in the streaming input to the
local delay of each sentence.

##  4 Proposed Metric: Average Token Delay

We propose a novel latency metric ATD to include the delay caused by a long
chunk output in latency measurement. We start from the ATD calculation in the
case of speech-to-speech SimulMT and generalize it for speech-to-text and
text-to-text cases.

In the following explanation, we first suppose the computation time is
included in ATD’s latency calculation. Then we explain the non-computation-
aware version of ATD which excludes the computation time.

Figure 2: Step-by-step example of simultaneous speech-to-speech MT: Time
passes from left to right.

###  4.1 ATD for simultaneous speech-to-speech translation

Figure 2 illustrates a step-by-step workflow of a speech-to-speech SimulMT. In
this figure, the white boxes with an order (1st, 2nd) represent the duration
of the speech segments, the orange ones represent the processing time to
encode the input prefixes and to judge whether we should _READ_ or _WRITE_ ,
and the blue ones represent the decoding time.

To calculate ATD, we divide each speech segment into sub-segments of length
$\tau$ from the beginning of the segment, assuming one word is uttered in
duration $\tau$. Intuitively, ATD is the average of the time differences
between the points of the same color (black, white, red, and blue) at the
ending time of sub-segments (Step 5).

ATD is defined as follows:

$\textrm{ATD}(\bm{x},\bm{y})=\frac{1}{|\bm{y}|}\sum_{t=1}^{|\bm{y}|}\left(T(y_{t})-T(x_{a(t)})\right),$
(7)

where

$a(t)=\min(t-d(t),g(t))$ (8) $d(t)=(t-1)-a(t-1).$ (9)

$T(\cdot)$ in Equation 7 represents the ending time of each input or output
token, shown as colored points in Figure 2. The token is a sub-segment in
speech; it is a character or a word in text. $a(t)$ represents the index of
the input token corresponding to $y_{t}$ in the time difference calculation
and $a(0)$ = 0. $d(t)$ in Equation 9 represents how much longer the duration
of the previous translation prefix is than that of the previous input prefix.
As shown in Equation 8, if $d(t)$ > $0$, $a(t)$ becomes smaller than output
index $t$. This means the previous long output increases the time difference
between the input and corresponding output tokens.

ATD is the average delay of the output tokens against their corresponding
input tokens, considering the latency required for inputs and outputs.
Although the input-output correspondence does not necessarily denote semantic
equivalence, especially for language pairs with large differences in their
word order and numbers of tokens, we use this simplified formulation for
latency measurements like in AL.

1st |  |  |
---|---|---|---
$x_{1}$ | $x_{2}$ | $x_{3}$ |  |  |
|  |  | $y_{1}$ | $y_{2}$ | $y_{3}$
|  |  | 1st
(a) 1st translation has identical length as 1st input

1st |  |  |  |
---|---|---|---|---
$x_{1}$ | $x_{2}$ | $x_{3}$ |  |  |  |
|  |  | $y_{1}$ | $y_{2}$ | $y_{3}$ | $y_{4}$
|  |  | 1st
(b) 1st translation is longer than 1st input

Figure 3: Examples of 1st chunk translation

1st | 2nd |  |  |
---|---|---|---|---
$x_{1}$ | $x_{2}$ | $x_{3}$ | $x_{4}$ | $x_{5}$ |  |  |
|  |  | $y_{1}$ | $y_{2}$ | $y_{3}$ | $y_{4}$ | $y_{5}$
|  |  | 1st | 2nd
(a) 1st translation is same length as 1st input.

1st | 2nd |  |  |  |
---|---|---|---|---|---
$x_{1}$ | $x_{2}$ | $x_{3}$ | $x_{4}$ | $x_{5}$ |  |  |  |
|  |  | $y_{1}$ |  | $y_{2}$ | $y_{3}$ | $y_{4}$ | $y_{5}$
|  |  | 1st |  | 2nd
(b) 1st translation is shorter than 1st input.

1st | 2nd |  |  |  |
---|---|---|---|---|---
$x_{1}$ | $x_{2}$ | $x_{3}$ | $x_{4}$ | $x_{5}$ |  |  |  |
|  |  | $y_{1}$ | $y_{2}$ | $y_{3}$ | $y_{4}$ | $y_{5}$ | $y_{6}$
|  |  | 1st | 2nd
(c) 1st translation is longer than 1st input.

Figure 4: Examples of 2nd chunk translation

Figure 3 shows examples to explain Equation 8 for the first chunk translation.
Here we simplify the duration of the input and output tokens to the same
length. In Figure 3, we measured the token delay on $y_{3}$. Here $d(3)=0$,
and then we obtained $a(3)=t-0=3$, and so $y_{3}$ corresponds to $x_{3}$. In
Figure 3, we measured the token delay on $y_{4}$, obtained $d(4)=0$, and then
obtained $a(4)=g(4)=3$, and so $y_{4}$ corresponds to $x_{3}$. In Figure 3,
the input and output lengths are identical, as are the corresponding indexes
of the input and output tokens. However, in Figure 3, since the output is
longer than the input, we corresponded the $y_{3}$ and latter tokens like
$y_{4}$ to identical input token $x_{3}$.

Figure 4 shows examples to explain Equation 8 for the second chunk
translation. Suppose we measure the token delay on $y_{5}$. In Figure 4 and
Figure 4, $d(5)=0$, and so $a(5)=t-0=5$. Therefore, $y_{5}$ corresponds to
$x_{5}$. In Figure 4, $d(5)=4-a(4)=1$, according to the the result of Figure
3. Therefore, $a(5)=t-1=4$ and $y_{5}$ corresponds to $x_{4}$. Unlike Figure 4
and Figure 4, the first chunk output, which is longer than the first chunk
input in Figure 4, produces the difference of the corresponding indexes of the
input and output tokens (Figure 3). The difference of the indexes is
accumulated and passed to the latter tokens (Figure 4).

Figure 5: Summary view for latency measurement for simultaneous speech-to-text
translation

Figure 6: Summary view for latency measurement for simultaneous text-to-text
translation

Figure 7: Summary view for non-computation-aware latency measurement for
simultaneous text-to-text translation

###  4.2 ATD for simultaneous {speech,text}-to-text translation

Figure 7 illustrates the latency measurement for speech-to-text SimulMT,
assuming the text output duration is negligible. Figure 7 is the latency
measurement for the text-to-text SimulMT. We reserve the input duration here
(shown by yellow rectangles) as the computation time of the automatic speech
recognition (ASR) because the input for text-to-text SimulMT should come from
speech.

###  4.3 Non-computation-aware ATD

We sometimes use a latency measurement independent from the computation time
for theoretical analyses. In Figures 2, 7, and 7, we remove the orange, blue,
and yellow parts and only include the duration of the speech segments to
calculate the delay. However, this means all the terms in the text-to-text
SimulMT become 0. We follow the conventional step-wise latency measurement as
AP and AL by letting each input and output word spend one step (Figure 7).
Here we assume the model can read the next input and output a partial
translation in parallel.

##  5 Simulation

Before experiments using real data, we show simulations that compare AL, DAL
and ATD in different conditions in a text-to-text SimulMT.

###  5.1 Cases 1 and 2

| AL | DAL | ATD | EVS
---|---|---|---|---
Case 1 | 1.2 | 1.84 | 2.4 | 2.3
Case 2 | 0.25 | 1.19 | 3.75 | 3.7
Increase rate | -79% | -0.35% | +36% | +38%
Table 1: Latency values for cases in Figure 1

First, we calculated the latency metrics for the two cases in Figure 1 and
show the calculated values in Table 1. AL and DAL are larger in case 1, while
ATD is larger in case 2 as is EVS. This example shows that ATD sufficiently
considers the delay caused by the long outputs.

###  5.2 Case 3: 20-20

Figure 8: Case 3 (20 input and 20 output tokens)

We conducted another type of simulation by supposing we have input of 20
tokens as well as an identical amount of output. We simulated two simultaneous
translation strategies, wait-k and fixed-size segmentation. In the fixed-size
segmentation, we assume for simplicity the length of the input and output
chunks to be $k$ until the end-of-sentence token is predicted. We call this
simple strategy chunk-$k$ in this simulation. Hyperparameter $k$ for wait-$k$
and chunk-$k$ varies from 1 to 20.

Figure 8 shows the results. Except for AL evaluating chunk-$k$, since all the
metrics have identical values, they are plotted using the same color. There is
a gap between wait-$k$ and chunk-$k$ when using AL, although ATD and DAL
result in the same values for them. One serious problem raised here is the
large jump in AL at $k=20$. For chunk-$19$, AL uses $r=1$,
$\tau_{g_{\textrm{chunk-}19}}(|\bm{x}|)=20$, $g_{\textrm{chunk-}19}(\tau)=19$
($1\leq\tau\leq 19$), and $g_{\textrm{chunk-}19}(\tau)=20$ ($\tau=20$). The AL
value becomes
$\frac{1}{20}\left\\{\left(\sum_{\tau=1}^{19}\tau\right)+1\right\\}=\frac{191}{20}=9.55$.
However, in the case of chunk-$20$, $g_{\textrm{chunk-}20}(\tau)=20$ for all
$\tau$ and $\tau_{g_{\textrm{chunk-}20}}(|\bm{x}|)=1$, according to Equation
4. As a result, the AL value becomes 20. This phenomenon shows the problem of
AL in which a long partial translation output reduces the latency.

###  5.3 Case 4: (10+10)-($L_{1}$+10)

Figure 9: Case 4 (10+10 input and $L_{1}$+10 output tokens with varying
$L_{1}$)

In another case, we assume the input is divided into two segments, each of
which has 10 tokens and each corresponding output consists of $L_{1}$ and 10
tokens with varying $L_{1}$. Figure 9 shows the results.

If $L_{1}<10$, some target tokens corresponding to the first input chunk come
after the start of reading the second chunk. Figure 4 illustrates such a
situation, in which $y_{2}$ is in the second chunk output although the
corresponding $x_{2}$ is in the first chunk input. If $y_{2}$ is in the first
chunk output, the time difference between $x_{2}$ and $y_{2}$ becomes smaller.
It also reduces the time difference of the latter token pairs, such as $x_{3}$
and $y_{3}$. Therefore, the shorter $L_{1}$ becomes, the larger the delay
grows in this range.

If $L_{1}>10$, the translation of the second chunk is delayed due to the long
translation outputs for the first chunk (Figure 4). Therefore, the longer
$L_{1}$ becomes, the larger the delay grows in this range. ATD reflects this
phenomenon, but AL decreases monotonically with $L_{1}$; DAL does not
increase.

##  6  SimulMT Analyses

We conducted analyses on actual SimulMT results to investigate ATD’s
effectiveness. Since most existing latency metrics were originally proposed
for text-to-text SimulMT, we compared text-to-text models in this experiment.

###  6.1 Data

We used the data from the shared task of English-to-German simultaneous
translation in the IWSLT 2022 evaluation campaign Anastasopoulos et al.
(2022). We used the WMT 2014 training set (4.5 M sentence pairs) for pre-
training and the IWSLT 2017 training set (206 K sentence pairs) for fine-
tuning. The development set consisted of dev2010, tst2010, tst2011, and
tst2012 (5,589 sentence pairs in total), and the evaluation set was tst2015
(1,080 sentence pairs).

Following the experimental settings in the literature (Kano et al., 2022), we
compared several SimulMT methods: wait-$k$ (Ma et al., 2019), Meaningful Unit
(MU; Zhang et al., 2020), Incremental Constitute Label Prediction (ICLP; Kano
et al., 2021), and Prefix Alignment (PA; Kano et al., 2022).

###  6.2 Results

(a) Latency measurement by AL

(b) Latency measurement by ATD

Figure 10: Latency and BLEU

Figure 11: Length ratio and AL

Figure 11 shows the results. Compared to the AL shown in Figure 11, the ATD in
Figure 11 demonstrated clear differences in delay among models. MU and ICLP
were affected by the change in the latency metric. We scrutinized their
results and found this degradation was caused by over-translation, suggested
by observations of the length ratio results shown in Figure 11. MU and ICLP
generated long translations exceeding a length ratio of 1.0 when they worked
with small latency. One interesting finding here is the correlation between
BLEU and ATD by MU; a larger latency did not always increase BLEU. Over-
translation increased ATD, but it simultaneously decreased BLEU. In contrast,
the wait-$k$ strategy just generates one output token at a time and does not
suffer from this issue. PA also worked well with the latency measurement by
ATD because it fine-tunes the translation model to prevent over-translation.

##  7 Using EVS as reference

Since the above analyses were not enough to verify ATD’s effectiveness, we
conducted the following experiments to compare different latency metrics. To
verify which was superior, we compared the correlation between each latency
metric and the mean EVS, as calculated from the outputs of real speech-to-
speech SimulMT.

###  7.1 Methodology to calculate EVS

EVS can be measured in several ways Robbe (2019). We used an automatic word
alignment tool to assist humans to choose the correct alignment due to the
high cost of word alignment from scratch by humans. EVS is calculated by the
following process:

##### 1.

Run speech-to-speech SimulMT models for the test dataset and output target
speech.

##### 2.

Use awesome-align Dou and Neubig (2021) to obtain automatic word alignment in
each sentence transcription of the source and target speech. Transcription is
obtained by Automatic Speech Recognition (ASR) using Whisper Radford et al.
(2022).

##### 3.

A human annotator chooses the correct alignment from the awesome-align output.

##### 4.

Obtain the timestamps for the source and target words from the speech and its
transcription by WhisperX Bain et al. (2023).

##### 5.

Take the time difference between the start time of the source and target words
with a correct alignment as EVS, and calculate the mean value of EVSs in a
sentence as the mean EVS.

For the analysis, we also calculated the mean value of all the EVSs, including
the wrong word alignments and call this the mean automatic EVS.

###  7.2 Settings

We used the following models and evaluation dataset in our experiment.

####  7.2.1 Models

We used a cascaded speech-to-speech SimulMT model by connecting the speech-to-
text SimulMT and incremental Text-To-Speech (TTS) models following Fukuda et
al. (2023). We used their TTS model (Fukuda et al., 2023) for the following
three types of speech-to-text SimulMT models and their variations:

##### Test-time wait-k (Ma et al., 2019):

This method first reads k input speech segments and alternately repeats
outputting one word and reading one input speech segment. Hyperparameter k of
test-time wait-k was set to 3 and 5. The source speech is input to the
translation model in 250-ms segments.

##### Fixed-size segmentation:

This method immediately starts a translation every time a new input speech
segment comes. We compared segment sizes of 750 and 2500 ms.

##### System by Fukuda et al. (2023):

This scheme uses Prefix Alignment (Kano et al., 2022) and Local Agreement (Liu
et al., 2020) to mitigate over-translation with fixed-size segmentation. We
compared segment sizes of 250 and 1250 ms.

The settings of the training NMT models followed published research (Fukuda et
al., 2023). For fixed-size segmentation and test-time wait-k, we shared a
single NMT model, which was not fine-tuned with prefix alignment pairs.

####  7.2.2 Evaluation data

We used the first 30 sentences of the English-Japanese tst-COMMON in MuST-C.
We used three types of SimulMT methods, each of which has two variations as
described above. 180 sentences are evaluated.

In the streaming SimulMT, since there are no obvious sentence boundaries, the
input sequence sometimes becomes longer than a sentence. Such a long
translation can delay the translation of subsequent inputs. Therefore, we also
evaluated the latency of the concatenation of two sentences.

We used Spearman’s $\rho$ to measure the correlation between EVS and each
latency metric.

###  7.3 Compared latency metrics

We compared ATD with the baselines of the speech-to-speech and text-to-text
latency metrics.

####  7.3.1 Speech-to-speech

Since the existing metrics described in Section 3 cannot be applied to speech-
to-speech SimulMT, we used two baseline latency metrics for this experiment:
Start Offset and End Offset, both of which were officially used in the
Simultaneous Translation Task of IWSLT 2023 Evaluation Campaign (Agarwal et
al., 2023). Start Offset is literally the time difference between the starts
of the source speech input and the target speech output. End Offset uses the
ends instead of the starts. For ATD, we set $\tau$ to 300 ms in the
experiment.

We evaluated the latency metrics using the speech-to-speech SimulMT outputs
and conducted an experiment on both computation-aware (CA) and non-
computation-aware (NCA) conditions. In the former, the output speech reflected
the actual computation time as silence. Therefore, the target token timestamps
in the CA condition should be different from those in the NCA condition. We
used one NVIDIA RTX TITAN throughout the experiments.

####  7.3.2 Text-to-text

We compared AL Ma et al. (2019), AL with reference Ma et al. (2020), LAAL Papi
et al. (2022), DAL Arivazhagan et al. (2019), and ATD on the text-to-text
condition. By using the word timestamps obtained by WhisperX, we corresponded
the chunk boundaries to the text boundaries.

###  7.4 Result: speech-to-speech

| 1 sentence | 2 sentences
---|---|---
Metris | NCA | CA | NCA | CA
Start Offset | 0.632 (0.000) | 0.374 (0.000) | 0.368 (0.000) | 0.103 (0.225)
End Offset | 0.611 (0.000) | 0.755 (0.000) | 0.541 (0.000) | 0.755 (0.000)
ATD |  0.836 (0.000) |  0.859 (0.000) |  0.779 (0.000) |  0.832 (0.000)
Mean auto. EVS | 0.869 (0.000) | 0.897 (0.000) | 0.839 (0.000) | 0.887 (0.000)
#samples | 149 | 148 | 143 | 142
Table 2: Speech-to-speech: Spearman’s correlation between mean EVS and latency metrics: Value in parentheses is p-value. Metrics | 1 sentence | 2 sentences
---|---|---
AL | 0.568 (0.000) | 0.246 (0.003)
AL-ref | 0.602 (0.000) | 0.257 (0.002)
LAAL | 0.624 (0.000) | 0.270 (0.001)
DAL | 0.646 (0.000) | 0.381 (0.000)
ATD |  0.651 (0.000) |  0.461 (0.000)
#samples | 149 | 143
Table 3: Text-to-text: Spearman’s correlation between mean EVS and latency metrics # Sentence pairs | 223,108
---|---
# English words | 4,593,204
# Japanese words | 4,794,912
# Japanese characters | 8,838,777
Table 4: Statistics of IWSLT2017 English-Japanese training dataset | 1 sentence | 2 sentences
---|---|---
Metric | 1 char | 2 char | 1 char | 2 char
AL | 0.567 (0.000) | 0.566 (0.000) | 0.149 (0.075) | 0.147 (0.080)
AL-ref | 0.575 (0.000) | 0.613 (0.000) | 0.193 (0.021) | 0.264 (0.001)
LAAL | 0.613 (0.000) | 0.614 (0.000) | 0.198 (0.018) | 0.264 (0.001)
DAL |  0.634 (0.000) | 0.630 (0.000) | 0.302 (0.000) | 0.313 (0.000)
ATD | 0.601 (0.000) |  0.641 (0.000) |  0.498 (0.000) |  0.506 (0.000)
#samples | 149 | 149 | 143 | 143
Table 5: Text-to-text (character-level output): Spearman’s correlation between
mean EVS and latency metrics

Table 2 shows the correlation between the mean EVS and latency metrics for the
speech-to-speech SimulMT results. #samples represent the number of evaluated
sentences after removing the sentences with no correct word alignments.
Regardless of the input sequence length and the computation-awareness, ATD has
the highest correlation in all the latency metrics that do not use word
alignment. Mean automatic EVS is the best because the mean EVS is calculated
based on word alignments, which are part of the alignments used to calculate
the mean automatic EVS.

For the two sentences with both CA and NCA, the correlation of Start Offset
largely decreased from the result for one sentence. When the input increased,
the impact of the delay at the first start of the translation fell for the
entire delay. Therefore, Start Offset has lower correlation, and in the CA
condition, it has no significant correlation. ATD maintains a relatively high
correlation, as does the mean automatic EVS. When the input sequence is
lengthened, the accumulated delay caused by the previous long chunk
translation output increases. Since ATD adequately considers the output
length, it still has high correlation.

###  7.5 Result: text-to-text

Table 3 shows the correlation between the mean EVS and text-to-text latency
metrics. ATD has the highest correlation among all the text-to-text latency
metrics. AL is the most common latency metric, but its correlation was worse
than ATD and its variants (AL-ref, LAAL, and DAL). DAL has the best
correlation among the AL variants.

For two sentences, the correlations of all the latency metrics greatly
decreased, but the differences between ATD and the other latency metrics are
larger than for one sentence. This is because ATD includes enough delay caused
by the end time of the previous sentence translation, as explained in the
speech-to-speech result.

###  7.6 Result: text-to-text (character-level)

Text-to-text latency metrics basically assume each text token is a word.
However, languages like Japanese and Chinese have no white spaces between
words. Therefore, SimulEval Ma et al. (2020), which is commonly used to
evaluate simultaneous translation systems, regards characters as tokens for
Japanese for simplicity without tokenizing them into words. We additionally
conducted an evaluation with character-based text tokens.

Table 4 shows the statistics of the IWSLT 2017 English-Japanese training
dataset. The number of English and Japanese words is close, although the
number of Japanese characters is twice as large. Therefore, to calculate the
latency metrics, we compared one character as one token and two characters as
one token 222For two characters as one token, we regard the last one character
in the output chunk as one token if the number of characters in the chunk is
odd..

Table 5 shows the result. Basically, the word-level latency calculation in
Table 3 has higher correlation than the character-level, and two characters as
one token is better than one character as one toke, especially for two
sentences. This is because the amount of information conveyed by one Japanese
word is closer to that of one English word than one or two Japanese
characters. For the same reason, two Japanese characters as one token
outperformed one Japanese character as one token, according to Table 4.

ATD is the best except for a condition of one character as one token for one
sentence. ATD is the latency metric most affected by the output length. An
output of one Japanese character has less correspondence with the input of one
English word than other output-levels, and the output length by character is
much longer than the input length by word. As a result, ATD added excessive
delay caused by this long output, and its correlation decreased.

According to the character-level result, we must carefully address
tokenization, not only for ATD, but also for other latency metrics.

##  8 Conclusion

We proposed a novel latency metric ATD for SimulMT, which addresses the
problem of latency evaluations for long chunk outputs by taking the output
length into account. ATD gives a large latency value to a long output based on
the assumption that the output also causes a delay, unlike in AL. We
identified ATD’s effectiveness by analyzing the simulations and it had the
highest correlation with EVS among the token-based latency metrics. Despite
being much simpler than mean automatic EVS and without requiring a long
process like ASR, word alignment, or getting timestamps, ATD correlates mean
EVS very well. Therefore, it can be easily used to evaluate the latency of
simultaneous translation. In future work, we will investigate the correlation
between latency metrics and the delay experienced by listeners.

##  9 Acknowledgement

This paper is an extended version of our conference paper Kano et al. (2023)
with additional experiments and analyses. Part of this work was supported by
JSPS KAKENHI Grant Numbers JP21H05054 and JP21H03500.

## References

  * Agarwal et al. (2023) Milind Agarwal, Sweta Agrawal, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Mingda Chen, William Chen, Khalid Choukri, Alexandra Chronopoulou, Anna Currey, Thierry Declerck, Qianqian Dong, Kevin Duh, Yannick Estève, Marcello Federico, Souhir Gahbiche, Barry Haddow, Benjamin Hsu, Phu Mon Htut, Hirofumi Inaguma, Dávid Javorský, John Judge, Yasumasa Kano, Tom Ko, Rishu Kumar, Pengwei Li, Xutai Ma, Prashant Mathur, Evgeny Matusov, Paul McNamee, John P. McCrae, Kenton Murray, Maria Nadejde, Satoshi Nakamura, Matteo Negri, Ha Nguyen, Jan Niehues, Xing Niu, Atul Kr. Ojha, John E. Ortega, Proyag Pal, Juan Pino, Lonneke van der Plas, Peter Polák, Elijah Rippeth, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Yun Tang, Brian Thompson, Kevin Tran, Marco Turchi, Alex Waibel, Mingxuan Wang, Shinji Watanabe, and Rodolfo Zevallos. 2023.  FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 1–61, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Anastasopoulos et al. (2022) Antonios Anastasopoulos, Loïc Barrault, Luisa Bentivogli, Marcely Zanon Boito, Ondřej Bojar, Roldano Cattoni, Anna Currey, Georgiana Dinu, Kevin Duh, Maha Elbayad, Clara Emmanuel, Yannick Estève, Marcello Federico, Christian Federmann, Souhir Gahbiche, Hongyu Gong, Roman Grundkiewicz, Barry Haddow, Benjamin Hsu, Dávid Javorský, Vĕra Kloudová, Surafel Lakew, Xutai Ma, Prashant Mathur, Paul McNamee, Kenton Murray, Maria Nǎdejde, Satoshi Nakamura, Matteo Negri, Jan Niehues, Xing Niu, John Ortega, Juan Pino, Elizabeth Salesky, Jiatong Shi, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Marco Turchi, Yogesh Virkar, Alexander Waibel, Changhan Wang, and Shinji Watanabe. 2022.  Findings of the IWSLT 2022 evaluation campaign.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 98–157, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. 2019.  Monotonic infinite lookback attention for simultaneous machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1313–1323, Florence, Italy. Association for Computational Linguistics.
  * Bain et al. (2023) Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023.  WhisperX: Time-Accurate Speech Transcription of Long-Form Audio.  _INTERSPEECH 2023_.
  * Cho and Esipova (2016) Kyunghyun Cho and Masha Esipova. 2016.  Can neural machine translation do simultaneous translation?  _arXiv preprint arXiv:1606.02012._
  * Dou and Neubig (2021) Zi-Yi Dou and Graham Neubig. 2021.  Word alignment by fine-tuning embeddings on parallel corpora.  In _Conference of the European Chapter of the Association for Computational Linguistics (EACL)_.
  * Fukuda et al. (2023) Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Yuka Ko, Tomoya Yanagita, Kosuke Doi, Mana Makinae, Sakriani Sakti, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  NAIST simultaneous speech-to-speech translation system for IWSLT 2023.  In _Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)_ , pages 330–340, Toronto, Canada (in-person and online). Association for Computational Linguistics.
  * Gu et al. (2017) Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. 2017.  Learning to translate in real-time with neural machine translation.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1053–1062, Valencia, Spain. Association for Computational Linguistics.
  * Iranzo-Sánchez et al. (2021) Javier Iranzo-Sánchez, Jorge Civera Saiz, and Alfons Juan. 2021.  Stream-level latency evaluation for simultaneous machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 664–670, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kano et al. (2021) Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2021.  Simultaneous neural machine translation with constituent label prediction.  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1124–1134, Online. Association for Computational Linguistics.
  * Kano et al. (2022) Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2022.  Simultaneous neural machine translation with prefix alignment.  In _Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)_ , pages 22–31, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
  * Kano et al. (2023) Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2023.  Average Token Delay: A Latency Metric for Simultaneous Translation.  In _Proc. INTERSPEECH 2023_ , pages 4469–4473.
  * Liu et al. (2020) Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020.  Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection.  In _Proc. Interspeech 2020_ , pages 3620–3624.
  * Ma et al. (2019) Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019.  STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3025–3036, Florence, Italy. Association for Computational Linguistics.
  * Ma et al. (2020) Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino. 2020.  SIMULEVAL: An evaluation toolkit for simultaneous translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 144–150, Online. Association for Computational Linguistics.
  * Papi et al. (2022) Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022.  Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation.  In _Proceedings of the Third Workshop on Automatic Simultaneous Translation_ , pages 12–17, Online. Association for Computational Linguistics.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.  _arXiv preprint arXiv:2212.04356_.
  * Robbe (2019) Elisa Robbe. 2019.  Ear-voice span in simultaneous conference interpreting en-es and en-nl: A case study.  In _Doctoral dissertation_. Ghent University.
  * Zhang et al. (2020) Ruiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2020.  Learning adaptive segmentation policy for simultaneous translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2280–2289, Online. Association for Computational Linguistics.


#  FeAmGen.jl: A Julia Program for Feynman Amplitude Generation

Quan-feng Wu<EMAIL_ADDRESS>Zhao Li<EMAIL_ADDRESS>Institute of
High Energy Physics, Chinese Academy of Sciences, Beijing 100049, China
University of Chinese Academy of Sciences, Beijing 100049, China  Center for
High Energy Physics, Peking University, Beijing 100871, China

###### Abstract

FeAmGen.jl is a Julia package designed to generate Feynman diagrams and their
corresponding amplitudes for various processes in particle physics. Utilizing
the models in the Universal Feynman Output (UFO) format and Qgraf for diagram
generation, it also employs SymEngine.jl and Form for amplitude generation.
Additionally, the package offers functions for constructing Feynman integral
topologies. In conclusion, FeAmGen.jl provides usability and versatility for
the high precision calculations of the perturbative quantum field theory in
the Standard Model or in extensions beyond it.

###### keywords:

Feynman diagrams , Feynman amplitudes , Perturbation theory , Quantum field
theory

††journal: Computer Physics Communications

###### Contents

  1. 1 Introduction
  2. 2 Using FeAmGen.jl
    1. 2.1 Main functions
    2. 2.2 Configuration files
    3. 2.3 Generate amplitudes
    4. 2.4 Construct complete topologies
    5. 2.5 Model files
    6. 2.6 Run tests
      1. 2.6.1 Basic tests
      2. 2.6.2 Generic process generation
      3. 2.6.3 Construct complete top topologies
    7. 2.7 Results
      1. 2.7.1 Directory tree structures
      2. 2.7.2 Amplitudes
      3. 2.7.3 Visuals
      4. 2.7.4 Topologies
  3. 3 Installation
    1. 3.1 Install Julia
    2. 3.2 Install External Programs
    3. 3.3 Install Julia Packages
      1. 3.3.1 Add Registry IHEP-Multiloop
      2. 3.3.2 Addtional information about PyCall.jl
    4. 3.4 Install FeAmGen.jl
  4. 4 Conclusion
  5. Appendices
  6. A Canonicalization of Feynman Integrals
    1. A.1 Canonical Form of Feynman Integrals
    2. A.2 Canonicalization Algorithm
  7. B Feynman Integral Topology

PROGRAM SUMMARY

_Program Title:_ FeAmGen.jl
_CPC Library link to program files:_ (to be added by Technical Editor)
_Developer’s repository link:_ https://code.ihep.ac.cn/IHEP-
Multiloop/FeAmGen.jl
_Code Ocean capsule:_ (to be added by Technical Editor)
_Licensing provisions:_ MIT License
_Programming language:_ The Julia Programming Language
_External routines/libraries used:_

  1. 1.

Julia Standard Libraries: Dates [1], Pkg [2], SHA [3];

  2. 2.

Julia Packages: AbstractAlgebra.jl [4], Combinatorics.jl [5], FORM_jll.jl [6],
JLD2.jl [7], nauty_jll.jl [8], OrderedCollections.jl [9], Pipe.jl [10],
PyCall.jl [11], SymEngine.jl [12], YAML.jl [13];

  3. 3.

Non-Julia Programs/Packages: Form [14], nauty [15], Qgraf [16], SymEngine
[17].

_Nature of problem:_ The generation of the Feynman diagrams and the
corresponding amplitudes in the quantum field theory.
_Solution method:_ We present the Julia package FeAmGen.jl, which allows users
to define arbitrary models and couplings conveniently in the Universal Feynman
Output (UFO) format and to call Qgraf directly to generate Feynman diagrams
and amplitudes for arbitrary processes. Some symbolic simplifications will be
applied to the amplitudes automatically by calling Form, the well-known
computer algebra system for big expressions. The diagrams are written in the
LaTeX files that utilize TikZ-Feynman for creating beautiful and clear
figures. Finally, we also provide the algorithm to construct Feynman integral
topology.
_Additional comments:_ Thanks to the Julia Yggdrasil Project, this package can
run on almost all operating systems except Windows, where WSL2 is recommended.
Other restrictions are determined by the CPU time and the available RAM of the
computer.
_References:_

  * [1] Dates $\cdot$ The Julia Language. URL: https://docs.julialang.org/en/v1/stdlib/Dates.
  * [2] Pkg $\cdot$ The Julia Language. URL: https://docs.julialang.org/en/v1/stdlib/Pkg.
  * [3] SHA $\cdot$ The Julia Language. URL: https://docs.julialang.org/en/v1/stdlib/SHA.
  * [4] AbstractAlgebra.jl. URL: https://nemocas.github.io/AbstractAlgebra.jl.
  * [5] Combinatorics.jl. URL: https://juliamath.github.io/Combinatorics.jl.
  * [6] JuliaBinaryWrappers/FORM_jll.jl. URL: https://github.com/JuliaBinaryWrappers/FORM_jll.jl.
  * [7] Julia Data Format - JLD2. URL: https://juliaio.github.io/JLD2.jl/stable.
  * [8] JuliaBinaryWrappers/nauty_jll.jl. URL: https://github.com/JuliaBinaryWrappers/nauty_jll.jl.
  * [9] OrderedCollections.jl. URL: https://juliacollections.github.io/OrderedCollections.jl/latest.
  * [10] oxinabox/Pipe.jl: An enhancement to julia piping syntax. URL: https://github.com/oxinabox/Pipe.jl.
  * [11] JuliaPy/PyCall.jl: Package to call Python functions from the Julia language. URL: https://github.com/JuliaPy/PyCall.jl.
  * [12] SymEngine.jl: Julia wrappers of SymEngine. URL: https://symengine.org/SymEngine.jl.
  * [13] JuliaData/YAML.jl: Parse yer YAMLs. URL: https://github.com/JuliaData/YAML.jl.
  * [14] The FORM project for symbolic manipulation of very big expressions. URL: https://www.nikhef.nl/~form/.
  * [15] nauty and Traces: graph canonical labeling and automorphism group computation. URL: https://pallini.di.uniroma1.it.
  * [16] Qgraf — a computer program that generates Feynman diagrams for various types of QFT models. URL: http://cfif.ist.utl.pt/~paulo/qgraf.html.
  * [17] SymEngine: a fast symbolic manipulation library written in C++ https://github.com/symengine/symengine

##  1 Introduction

The quantum field theory for particle physics is very successful because of
the stunning agreements between the predictions of theory and the data from
experiments[1, 2, 3, 4]. In particular, the discovery of the Higgs boson at
the CERN Large Hadron Collider (LHC) in 2012 provided the final puzzle piece
to the Standard Model [5, 6]. Since then, particle physics has entered a new
era. The High-Luminosity LHC [7] was proposed to increase the integrated
luminosity by a factor of 10 beyond the design value of the present LHC
project so that the statistical error can be reduced significantly.
Furthermore, in order to achieve the most precise investigation on the Higgs
boson, several designs of the Higgs factory were proposed, including the
International Linear Collider (ILC) [8, 9, 10], the Circular Electron Positron
Collider (CEPC) [11, 12], and the Future Circular Collider (FCC-ee) [13, 14,
15], where the experiment error is expected to reach $\mathcal{O}(0.1\%)$.
With the steady improvement of experiment accuracies, the particle physics
community urgently needs the matching theoretical predictions [16, 17, 18, 19,
20, 21, 3, 4, 22, 23].

The perturbative quantum field theory framework is the most important approach
to implementing high-precision theoretical predictions. Furthermore, the core
of the perturbative quantum field theory framework is the Feynman diagram
technique. In this technique, the theoretical calculation begins with
generating the Feynman diagrams and the relevant amplitude expressions [1, 2].
However, the number of corresponding Feynman diagrams could increase
exponentially for high-order scattering processes, especially including loop
corrections [24, 1, 25, 26, 2, 27, 28, 29]. It is almost impossible to
enumerate the complete set of diagrams and amplitudes manually in the
calculation of theoretical predictions with high precision. Therefore, many
attempts have been made to systematically mechanize this process with
computers, e.g., FeynArts [27, 29], FeynCalc [30, 31, 32], Qgraf [25], and
etc.

In this paper we present the computer program FeAmGen.jl, written in the Julia
programming language111https://julialang.org, to generate the Feynman diagrams
and the corresponding amplitudes for the given process. A user-friendly
interface between Qgraf and the Universal Feynman Output format (UFO) [33, 34]
is provided. The program Qgraf is widely used to generate the multi-loop
Feynman diagrams for the given process automatically. The corresponding
amplitudes are then generated and simplified via the computer algebra systems
(CAS) of SymEngine.jl and Form. Finally, FeAmGen.jl also uses the LaTeX
package TikZ-Feynman [35] to generate the PDF files for visualizing Feynman
diagrams.

This paper is organized as follows: In Sec. 2, we briefly introduce the main
functions and demonstrate some examples. Sec. 3 gives the instructions for
installing FeAmGen.jl in detail. Finally, we make a conclusion in Sec. 4.
Appendices contain the details of the algorithms for canonicalizing the
Feynman integrals and constructing the complete Feynman integral topologies.

##  2 Using FeAmGen.jl

FeAmGen.jl works in two basic steps that are sketched in Fig. 1. Users should
prepare the UFO model files, which can be generated by the package FeynRules
[36, 37], and edit the seed YAML file according to the model and the process.
Details of configuring in YAML format can be found at Sec. 2.2. Then users
could use the function digest_seed_proc to process the seed YAML file to
obtain the process YAML files for the specified processes, where the function
digest_seed_proc will check the lepton number violation and the quark
generation violation and expand all partons according to the information in
the user-defined seed YAML file. Finally, the function generate_amp is called
to read the process YAML files generated in the previous step, then the
Feynman diagrams and the corresponding amplitudes will be generated. Please
refer to Sec. 2.3 for more information about the generation of amplitudes.
There are two directories that will be generated for archiving the amplitude
files (in the JLD2 file format) and the visual files (in the LaTeX file format
with the package TikZ-Feynman [35], respectively. One could refer to Sec. 2.7
for more details of the results.

Figure 1: Workflow of FeAmGen.jl

###  2.1 Main functions

There are three functions exported for users:

  1. 1.

digest_seed_proc( "/path/to/seed.yaml" ):
The options defined in the seed YAML file will be checked, e.g., the lepton
number violation and the quark generation violation. The partons will be
expanded explicitly into several sub-processes, and the corresponding sub-
process YAML files will be generated automatically, which contain all the
information for generating the Feynman diagrams and the corresponding
amplitudes.

  2. 2.

generate_amp( "/path/to/process.yaml" ):
This function will read in the process YAML file and the specified UFO model.
Then it will generate all amplitudes files in the JLD2 format and their
corresponding visual files as LaTeX files (TikZ-Feynman package is required
for generating the corresponding PDF files). The human-readable text files
(with the suffix out) will also be generated.

  3. 3.

construct_den_topology( "/path/to/amplitude/directory/" ):
Based on the amplitudes generated by the function generate_amp, users can
utilize this function to construct the topologies that cover all Feynman
amplitudes/integrals of the specified process.

In the following section, we will go into more detail about the usages of
these functions. In particular, we have provided the sample scripts archived
in test/ to demonstrate the usages of these functions as shown in Sec. 2.6.
All the following sample codes have the assumption of using the package
FeAmGen.jl via

⬇

using FeAmGen

###  2.2 Configuration files

Users need to prepare the UFO model and provide the information about the
process to be generated. This information should be written in the YAML
format222https://yaml.org, a human-friendly data serialization language. The
seed YAML file will be detailed in the example gbtw_Test.jl for the process
$gb\to tW^{-}$. The users are encouraged to copy it and make modifications as
needed.

1 is the content of the seed YAML file for the process $gb\to tW^{-}$, which
sets up 18 options in total. This configuration file in YAML format could be
read by the functions digest_seed_proc and generate_amp. However, there are
three options partons, AllowLeptonNumberViolation and
AllowQuarkGenerationViolation that are only recognized by the function
digest_seed_proc. Notice that the function generate_amp cannot resolve the
"parton" in the incoming/outgoing particle list.

Listing 1: Simple YAML example of the process $gb\to tW^{-}$

⬇

model_name: "sm_CKMdiag_Haa"

unitary_gauge: true

# only for ‘digest_seed_proc‘

partons: ["g", "u", "ubar", "d", "dbar", "b", "bbar"]

AllowLeptonNumberViolation: false

AllowQuarkGenerationViolation: false

# only for ‘digest_seed_proc‘

DropTadpole: true

DropWFcorrection: true

n_loop: 2 # nloop

QCDCT_order: 0

Amp_QCD_order: 5 # 1 + 2 * nloop

Amp_QED_order: 1

Amp_SPC_order: 0

Amp_Min_Ep_Xpt: -4 # -2 * nloop

Amp_Max_Ep_Xpt: 0

incoming: ["b", "g"]

outgoing: ["Wminus", "t"]

momentum_symmetry: []

color_symmetry: []

All options in the YAML file are:

  1. 1.

model_name: The name of the UFO model to invoke. It is worth mentioning that
the function digest_seed_proc will search for the model files from the working
directory if the location of the model file is missing.

  2. 2.

unitary_gauge: (true/false) The option to choose the unitary gauge or not for
the internal massive gauge bosons.

  3. 3.

partons: (for digest_seed_proc only) The set of particles defined by the user
can be used to generate the sub-processes. The users could use "parton" in the
incoming/outgoing particle list instead of the specific particle name. Then
the function digest_seed_proc will check this option and expand the "parton"’s
into several sub-processes according to the particle list here.

  4. 4.

AllowLeptonNumberViolation:(true/false, for digest_seed_proc only) The option
for allowing/forbidding the violation of the lepton number.

  5. 5.

AllowQuarkGenerationViolation: (true/false, for digest_seed_proc only) The
option for allowing/forbidding the violation of the quark generation.

  6. 6.

DropTadpole: (true/false) The option for dropping the diagrams with tadpoles
or not (also known as the diagrams amputated or not).

  7. 7.

DropWFcorrection: (true/false) The option for dropping the diagrams with
corrections of wave functions or not.

  8. 8.

n_loop: Number of loops.

  9. 9.

QCDCT_order: Order of QCD counter terms. If the users want to generate
order-$n_{\text{order}}$ with order-$n_{\text{CT}}$ Feynman
diagrams/amplitudes, the entry n_loop should be set to
$n_{\text{order}}-n_{\text{CT}}$ and the QCDCT_order should be set to
$n_{\text{CT}}$.

  10. 10.

Amp_QCD_order: Order of the QCD coupling in the amplitudes.

  11. 11.

Amp_QED_order: Order of the QED coupling in the amplitudes.

  12. 12.

Amp_SPC_order: Order of the special coupling in the amplitudes.
The users could mark the special coupling order (SPC order) in the UFO model
file for the vertices they are interested in. Then the users could specify the
SPC order here for the diagram generation.

  13. 13.

Amp_Min_Ep_Xpt: The minimal order of $\varepsilon$ in the series expansion of
$\varepsilon$ in the amplitudes.

  14. 14.

Amp_Max_Ep_Xpt: The maximal order of $\varepsilon$ in the series expansion of
$\varepsilon$ in the amplitudes.

  15. 15.

incoming: The set of incoming particles ("parton" is acceptable and it will be
expanded by digest_seed_proc).

  16. 16.

outgoing: The set of outgoing particles ("parton" is acceptable and it will be
expanded by digest_seed_proc).

  17. 17.

momentum_symmetry: Symmetries between external momenta. This option could be
needed in the evaluation of cut diagrams. For instance, [K1, K3] means
$K_{1}=K_{3}$, i.e., the replace $K_{3}\to K_{1}$ will be applied in the final
expressions of the amplitudes.

  18. 18.

color_symmetry: Symmetries between the colors carried by external particles.
This option could be needed in the decay or scattering process of the bound
state. For example, [1, 3] means the colors of particle 1 and particle 3 are
the same, i.e. $\delta_{c_{1}c_{3}}$ will be inserted into the color factors
and then calculated automatically.

With this seed YAML file, users could run the following code in the Julia REPL
or script.

⬇

digest_seed_proc( "/path/to/seed.yaml" )

It will generate several process YAML files that archive the information of
relevant sub-processes. The options partons, AllowLeptonNumberViolation and
AllowQuarkGenerationViolation will be checked. If the users want to generate
diagrams and amplitudes for the process including "parton", e.g. $p+p\to
t+\bar{t}$, the function digest_seed_proc will expand the partons and filter
the sub-processes according the values of options AllowLeptonNumberViolation
and AllowQuarkGenerationViolation. Please refer to the example of
ppttbar_Test.jl for expanding partons in detail.

The process YAML file generated by digest_seed_proc with 1 as input will be
saved in b_g_TO_Wminus_t_2Loop/b_g_TO_Wminus_t.yaml, where
"b_g_TO_Wminus_t_2Loop" represents the process $gb\to tW^{-}$ at 2-loop
obviously.

###  2.3 Generate amplitudes

There are several process YAML files generated by the function
digest_seed_proc, which are almost structurally identical to the seed YAML
file, except that they do not have the three options that will only be handled
by the function digest_seed_proc and the entries "parton" in the
incoming/outgoing particle list are expanded into partons explicitly. The
users could further call the function generate_amp to generate all Feynman
diagrams and the corresponding Feynman amplitudes as

⬇

generate_amp( "b_g_TO_Wminus_t_2Loop/b_g_TO_Wminus_t.yaml" )

Then two directories will be created: b_g_TO_Wminus_t_2Loop_amplitudes storing
the amplitude files and b_g_TO_Wminus_t_2Loop_visuals storing the
visualization files, respectively. Please refer to Sec. 2.7 for these two
directories in detail.

The function generate_amp calls the external program Qgraf for generating the
Feynman diagrams. The corresponding amplitudes will be constructed according
to the generated diagrams and the information from the UFO model. The
canonicalization will be applied to the amplitudes, which is explained in A.
Then the amplitudes will be simplified by the computer algebra systems of
SymEngine.jl and Form.

###  2.4 Construct complete topologies

FeAmGen.jl offers the function construct_den_topology for constructing the
complete topologies. This function works in two modes:

  1. 1.

Without momentum shift:

⬇

construct_den_topology(

"b_g_TO_Wminus_t_3Loop_amplitudes/";

mom_shift_opt=false

)

  2. 2.

With momentum shift:

     * (a)

Without reference denominator topologies:

⬇

construct_den_topology(

"b_g_TO_Wminus_t_3Loop_amplitudes/";

mom_shift_opt=true

)

     * (b)

With reference denominator topologies:

⬇

construct_den_topology(

"b_g_TO_Wminus_t_3Loop_amplitudes/";

mom_shift_opt=true,

ref_dentop_collect=<base_top_dir>

)

where <base_top_dir> is the directory that archives the generated topologies
for references. In this approach, the function construct_den_topology will
read the topologies from the directory <base_top_dir> and then construct the
topologies according to these topologies.

Then the complete set of topologies will be generated and saved in the
topology directory (See Sec. 2.7.1 for more details). The algorithm for
constructing Feynman integral topology is explained in B.

###  2.5 Model files

The Universal Feynman Output format [33], also known as Universal FeynRules
Output format in its first version [34], is a model file format for several
automatic matrix element generation software in high energy physics. The users
could use the package FeynRules333http://feynrules.irmp.ucl.ac.be [36, 37] to
generate the UFO models from the Lagrangian. The model files used in
FeAmGen.jl are written in the UFO format, since it provides a flexible format
archiving all the information about a theoretical model in an abstract form.
Users can also specify the directory to search for the model files in the
function digest_seed_proc as

⬇

digest_seed_proc( "/path/to/seed.yaml"; model_paths=[

"/path/to/the/first/model/directory",

"/path/to/the/second/model/directory"

] )

and in the function generate_amp as

⬇

generate_amp( "/path/to/seed.yaml"; model_paths=[

"/path/to/the/first/model/directory",

"/path/to/the/second/model/directory"

] )

In the UFO format, the information on the particles, parameters and vertices
of the model are stored in a set of Python2 objects. For accessing the model
files in FeAmGen.jl, PyCall.jl, the Julia interface to Python, is needed.
Please refer to Sec. 3.3.2 for more details.

###  2.6 Run tests

We provide several tests in the directory FeAmGen.jl/test, which can be simply
run by Pkg.jl as

⬇

using Pkg

Pkg.test("FeAmGen")

Details of every test are presented as follows.

####  2.6.1 Basic tests

The basic test file is basicTest.jl. Some key functions will be tested in this
script. And users could report any bugs and make suggestions as Git
issues444https://code.ihep.ac.cn/IHEP-
Multiloop/FeAmGen.jl.git/-/issues${}^{\text{ or
}}$555https://github.com/zhaoli-IHEP/FeAmGen.jl.git/issues.

####  2.6.2 Generic process generation

Generating all the Feynman diagrams and the corresponding amplitudes for the
specific process is the most important feature of this package, which is
covered in detail in the following test scripts.

  1. 1.

ppttbar_Test.jl for $\text{p + p}\to t+\bar{t}$ at the tree and 1-loop levels;
The incoming particles in this test are set to be "parton", which will be
expanded by the function digest_seed_proc. In subsequent steps of this test,
we will only generate the amplitudes and the visuals of the sub-process $gg\to
t\bar{t}$ for simplicity.

  2. 2.

DrellYan_Test.jl for $u\bar{d}\to W^{+}$ at tree, 1-loop and 2-loop level;

  3. 3.

eeHZ_Test.jl for $e^{+}e^{-}\to HZ$ at tree and 1-loop level;

  4. 4.

gbtw_Test.jl for $gb\to tW^{-}$ at tree, 1-loop and 2-loop level;

  5. 5.

tWb_Test.jl for $t\to bW^{-}$ at 3-loop level;

  6. 6.

tWtW_Test.jl for $tW^{+}\to tW^{+}$ at 3-loop level.

These tests will generate corresponding seed YAML files in the directory of
test/ automatically. Then, FeAmGen.jl will read the seed YAML files and
generate amplitudes and visuals for each process.

####  2.6.3 Construct complete top topologies

In this test, the complete topologies for the process $tW^{+}\to tW^{+}$ (at
2-loop and 3-loop) and the process $t\to t$ (at 4-loop) will be constructed.
Before running this test, the results produced by tWb_Test.jl and tWtW_Test.jl
must be prepared, which include the directories archiving the corresponding
amplitudes in the JLD2 format. Please see Sec. 2.7.4 for more information
about the constructed topologies.

###  2.7 Results

The results generated by FeAmGen.jl can be directly read by the subsequent
programs we are developing. However, the users are also encouraged to utilize
the results in their own way. We will demonstrate the details of the results
generated by FeAmGen.jl in the process $tW^{+}\to tW^{+}$ at 3-loop as an
example.

####  2.7.1 Directory tree structures

The generated files for $tW^{+}\to tW^{+}$ at 3-loop will be stored in the
following structure.

  * 1.

Wplus_t_TO_Wplus_t_2Loop/
Main directory of $tW^{+}\to tW^{+}$ at 2-loop.

    * (a)

Wplus_t_TO_Wplus_t.yaml
Process YAML file for $tW^{+}\to tW^{+}$ at 2-loop.

    * (b)

Wplus_t_TO_Wplus_t_2Loop_amplitudes/
Amplitude directory.

      * i.

amp1.jld2
JLD2 file for amplitude.

      * ii.

…

      * iii.

amp1.out
Human-readable file for amplitude.

      * iv.

…

    * (c)

Wplus_t_TO_Wplus_t_2Loop_shifted_amplitudes/
Amplitude directory with momentum shift, which will be generated only if the
option mom_shift_opt=true.

      * i.

shifted_amp1.jld2
JLD2 file for shifted amplitude.

      * ii.

…

    * (d)

Wplus_t_TO_Wplus_t_2Loop_shifted_topologies/
Topology directory with momentum shift, which will be generated only if the
option mom_shift_opt=true.

      * i.

shifted_topology.out
Human-readable file for topology.

      * ii.

topology1.jld2
JLD2 file for topology.

      * iii.

…

    * (e)

Wplus_t_TO_Wplus_t_2Loop_topologies/
Topology directory without momentum shift.

      * i.

topology.out
Human-readable file for topology.

      * ii.

topology1.jld2
JLD2 file for topology.

      * iii.

…

    * (f)

Wplus_t_TO_Wplus_t_2Loop_visuals/
Visual directory.

      * i.

visual_diagram1.tex
LaTeX file for plotting the Feynman diagram

      * ii.

…

      * iii.

expression_diagram1.out
Human-readable file for amplitude

      * iv.

…

      * v.

generate_diagram_pdf.jl
Julia script for building all LaTeX files

The details of amplitudes, visuals, and topologies will be explained in the
following sections.

####  2.7.2 Amplitudes

The most important part of the results generated by FeAmGen.jl is the Feynman
amplitudes, which are archived in the directory <process>/<amplitude>, e.g.
Wplus_t_TO_Wplus_t_2Loop/Wplus_t_TO_Wplus_t_2Loop_amplitudes. There are 2
types of files in the amplitude directory, the JLD2 files with suffix jld2 and
the UTF-8 format text files with suffix out. The files with suffix out are the
human-readable text files, which have the same information as the JLD2 files.
FeAmGen.jl uses the JLD2 file format to archive the generated amplitudes. For
reading it, JLD2.jl package is required, which is shown as

⬇

using JLD2

amp1 = load( "Wplus_t_TO_Wplus_t_2Loop/\

Wplus_t_TO_Wplus_t_2Loop_amplitudes/amp1.jld2" )

or

⬇

using JLD2

amp1 = jldopen( "Wplus_t_TO_Wplus_t_2Loop/\

Wplus_t_TO_Wplus_t_2Loop_amplitudes/amp1.jld2" )

<... The codes processing the amplitudes ...>

# One may need close(amp1) after all.

Then the variable amp1 will have type Dict (or type JLD2.JLDFile in the second
case) in the Julia programming language. Users could access the amplitude
information by amp1[<key>], where the <key> can be chosen as follows in this
example.

  * 1.

"Generator": Name of the generator.
amp1["Generator"] == "FeAmGen.jl"

  * 2.

"n_inc": Number of incoming particles.
amp1["n_inc"] == 2

  * 3.

"n_loop": Number of loops.
amp1["n_loop"] == 2

  * 4.

"min_ep_xpt": Minimal order of $\varepsilon$ in the series expansion of
$\varepsilon$.
amp1["min_ep_exp"] == -4

  * 5.

"max_ep_xpt": Maximal order of $\varepsilon$ in the series expansion of
$\varepsilon$.
amp1["max_ep_exp"] == 0

  * 6.

"couplingfactor": Coupling factor.
amp1["couplingfactor"] == "1"

  * 7.

"ext_mom_list": External momenta list.
amp1["ext_mom_list"] == ["K1", "K2", "K3", "K4"]

  * 8.

"scale2_list": List of mass squared scales.
amp1["scale2_list"] == ["shat", "mw^2", "ver1", "mt^2"]

  * 9.

"loop_den_list": List of denominators of the amplitudes.
amp1["loop_den_list"] == String[...] with

    * (a)

"Den(q2, 0, 0)";

    * (b)

"Den(K3 - K1 - K2 + q1 + q2, 0, 0)";

    * (c)

"Den(K3 - K1 + q1, 0, 0,)";

    * (d)

"Den(q1, mt, 0)";

    * (e)

"Den(q1 + q2, mt, 0)",

where "Den(P, m, ieta)" means $\quantity(P^{2}-m^{2}+\mathrm{i}\eta)^{-1}$.
Notice that we adopt the conversions that K<i>’s(k<i>’s) and q<i>’s represent
massive(massless) external momenta and loop momenta, respectively, where <i>
is the integer index of the momenta.

  * 10.

"loop_den_xpt_list": List of the exponents for the corresponding denominators.
amp1["loop_den_xpt_list"] == [1, 1, 1, 1, 1]
According to the information we have, the scalar integral should be

$I[1,1,1,1,1]=\int\quantity[\prod_{r=1}^{2}\frac{\differential^{D}q_{r}}{\mathrm{i}\pi^{D/2}}]\frac{1}{q_{2}^{2}\quantity(q_{12}+k_{123})^{2}\quantity(q_{1}+k_{13})^{2}\quantity(q_{1}^{2}-m_{t}^{2})\quantity(q_{12}^{2}-m_{t}^{2})},$

where $k_{123}:=-K_{1}-K_{2}+K_{3}$, $k_{13}:=-K_{1}+K_{3}$ and
$q_{12}:=q_{1}+q_{2}$.

  * 11.

"kin_relation": Dictionary of the kinematic relations.
amp1["kin_relation"] == Dict{String, String}(...) with

    * (a)

"Den(K3 - K1, mt, 0)" => "ver2";

    * (b)

…

  * 12.

"mom_symmetry": Symmetries between the external momenta.
amp1["symmetry_map"] == Dict{String, String}().
If not empty, e.g., Dict{"K1" => "K3"} means $K_{1}=K_{3}$.

  * 13.

"color_symmetry": Symmetries between the external momenta.
amp1["symmetry_map"] == Dict{Int, Int}().
If not empty, e.g., Dict{1 => 3} means the colors of particle 1 and particle 3
are the same, i.e. $\delta_{c_{1}c_{3}}$ will be inserted into the color
factors and then calculated automatically.

  * 14.

"model_parameter_dict": Dictionary of model parameters.
amp1["model_parameter_dict"] == Dict{String, String(…)} with

    * (a)

"sw" => "sqrt(sw2)";

    * (b)

…

  * 15.

"model_coupling_dict": Dictionary of model couplings.
amp1["model_coupling_dict"] == Dict{String, String(…)} with

    * (a)

"gcsub33" => "-6*im*lamh";

    * (b)

…

  * 16.

"signed_symmetry_factor": Signed symmetry factor, the product relative sign
from fermion line or loop and the symmetry factor of the Feynman diagram.
amp1["signed_symmetry_factor"] == "1"

  * 17.

"amp_color_list": List of color part expressions of the amplitude.
amp1["amp_color_list] == String["..."]

  * 18.

"amp_lorentz_list": List of Lorentz part expressions of the amplitude.
amp1["amp_lorentz_list] == String["..."]

The users could read this information in the human-readable files with suffix
out directly for convenience, and it is preferred to read JLD2 files in the
programs.

####  2.7.3 Visuals

The visualization of Feynman diagrams could be helpful for the users to
understand the interaction mechanism. Therefore, all visual files are provided
in the directory <process>/<visual>, e.g.
Wplus_t_TO_Wplus_t_2Loop/Wplus_t_TO_Wplus_t_2Loop_visuals for $tW^{+}\to
tW^{+}$ at 2-loop. There are also two types of files in the visual directory,
which are LaTeX files with suffix tex and the UTF-8 text files with suffix
out. The files with suffix out are also the human-readable text files, which
archive the information of

  * 1.

Loop denominators;

  * 2.

Loop denominator powers;

  * 3.

Coupling factor;

  * 4.

Symmetry of external momenta;

  * 5.

List of color factors;

  * 6.

List of Lorentz part factors of the amplitude.

Please refer to Sec. 2.7.2 for more information about the above entries.

The LaTeX files with suffix tex should be compiled via lualatex for the TikZ-
Feynman LaTeX package. We also provide the Julia script
generate_diagram_pdf.jl in every visual directory to compile these files in
batch (see Sec. 2.7.1 for details). The first Feynman diagram generated by
FeAmGen.jl for $tW^{+}\to tW^{+}$ at 2-loop is shown as Fig. 2.

Figure 2: Diagram 1 (Sign: 1, Symmetry factor: 1) for $tW^{+}\to tW^{+}$ at 2
loop

####  2.7.4 Topologies

The topology information is archived in the directory <process>/<topology>,
which archives a human-readable file named (shifted_)topology.out in the UTF-8
format. Also, the separate JLD2 files archive relevant single topology. As an
example, the first topology constructed for $tW^{+}\to tW^{+}$ at 2-loop with
momentum shift contains the following information.

  * 1.

The amplitude files covered by this topology:

    * (a)

/path/to/amp11.jld2

    * (b)

/path/to/amp7.jld2 with momentum shift:

      * i.

$q_{1}\to-q_{1}-K_{3}$

      * ii.

$q_{2}\to-q_{2}$

    * (c)

/path/to/amp29.jld2

    * (d)

/path/to/amp157.jld2

  * 2.

Denominators of this topology

    * (a)

Den(K3 - K1 + q1, 0, 0)

    * (b)

Den(K3 - K1 - K2 + q1 + q2, 0, 0)

    * (c)

Den(q1, mt, 0)

    * (d)

Den(K3 + q1, 0, 0)

    * (e)

Den(q2, 0, 0)

    * (f)

Den(q1 + q2, mt, 0)

    * (g)

Den(K1 + q2, mt, 0)

    * (h)

Den(K2 + q2, 0, 0)

By including the momentum shifts, we can construct 45 topologies to cover the
281 amplitudes of $tW^{+}\to tW^{+}$ at 2-loop.

##  3 Installation

Within the Julia ecosystem, the users could just use the package manager
Pkg.jl to install the FeAmGen.jl. Julia v1.6 or later is required. We
recommend the Linux platform for FeAmGen.jl. However, MacOS is also supported
by the Julia ecosystem.

###  3.1 Install Julia

The ecosystem Julia can be installed whether using precompiled binaries or
compiling from source by following the instructions at
https://julialang.org/downloads directly.

###  3.2 Install External Programs

FeAmGen.jl needs external programs of Form and Qgraf as back-ends. Please
refer to the following paragraphs for their installations.

##### Install QGRAF (v3.6.5)

Qgraf666http://cfif.ist.utl.pt/~paulo/qgraf.html is a computer program written
in Fortran, which could generate Feynman diagrams for various types of QFT
models. FeAmGen.jl will download Qgraf from
http://cfif.ist.utl.pt/~paulo/d.html and build it automatically if the Fortran
compiler is provided by ENV["FC"] = "/path/to/fortran. The users should agree
with the license of Qgraf before using this package. The users can set this
variable in the Julia startup
file777https://docs.julialang.org/en/v1/manual/command-line-
interface/#Startup-file, which archives the codes that you want executed
whenever Julia is run. FeAmGen.jl will also check the variable ENV["QGRAF"]
for the path to the Qgraf installed in your computer. Therefore, the user
should add ENV["QGRAF"] = "/path/to/qgraf" into the Julia startup file.

##### Install FORM (v4.3.1)

Form is designed for the symbolic manipulation of big expressions. Please
visit the home page of Form888https://www.nikhef.nl/~form/ for more
information and installation instructions. Also FeAmGen.jl can automatically
invoke the binary interface for Form provided by the Julia package FORM_jll.jl
on platforms other than Windows. For Windows users, we recommend using the
Windows Subsystem for Linux 2 (WSL2)999Please refer to
https://learn.microsoft.com/en-us/windows/wsl/install for WSL2 installation.,
which provides a Linux virtual machine but with the performance comparable to
the physical machine.

###  3.3 Install Julia Packages

Most of Julia packages required by FeAmGen.jl are managed by the Pkg.jl, which
is the package management for Julia programming language. Therefore, they
could be installed automatically when you install FeAmGen.jl.

####  3.3.1 Add Registry IHEP-Multiloop

Julia registries contain information about packages, such as available
releases and dependencies, and where they can be downloaded. The General
registry101010https://github.com/JuliaRegistries/General is the default
registry. The IHEP-Multiloop registry111111https://code.ihep.ac.cn/IHEP-
Multiloop/JuliaRegistry is the Julia registry maintained by the authors, which
provides the information about the package FeAmGen.jl and its dependencies
developed and maintained by the authors.

⬇

using Pkg

Pkg.Registry.add("General")

Pkg.Registry.add( Pkg.Registry.RegistrySpec( url=

"https://code.ihep.ac.cn/IHEP-Multiloop/JuliaRegistry")

)

####  3.3.2 Addtional information about PyCall.jl

The UFO format is written in the Python programming language. Therefore, the
Julia package PyCall.jl is used in FeAmGen.jl to read the model files in the
UFO format. However, the UFO formats exported from the latest version of
FeynRules are still written in Python2, which had been stopped by Python
developers in 2020.121212https://peps.python.org/pep-0373/ Fortunately,
PyCall.jl supports calling Python2 directly right now. Hence, there are two
approaches to reading UFO format in FeAmGen.jl:

  1. 1.

Build the PyCall.jl with Python2.7;

  2. 2.

Convert the UFO formats from Python2 to Python3.

The first is more cumbersome and it will not be presented here. Users can
consult the documentation of PyCall.jl for more details. We recommend the
second approach. The conversion can be done directly by the tool script
2to3131313https://docs.python.org/3/library/2to3.html provided by Python3.

###  3.4 Install FeAmGen.jl

After adding the IHEP-Multiloop registry, users may install FeAmGen.jl by
running

⬇

using Pkg

Pkg.add("FeAmGen")

All other Julia dependencies are installed automatically during the
installation procedure.

##  4 Conclusion

In this paper, we present the Julia package FeAmGen.jl, a powerful and
versatile Julia program that can handle the challenging task of generating and
calculating Feynman diagrams and amplitudes for various quantum field theory
processes. The program is designed to meet the increasing demand for high
precision theoretical predictions in particle physics, especially in the era
of precise tests of the Standard Model and beyond. The program takes advantage
of the UFO model format, which allows users to define arbitrary models and
couplings conveniently. The program also relies on Qgraf and Form, the well-
established external tools that can efficiently generate Feynman diagrams and
simplify amplitude expressions. Moreover, the program provides useful
functions to construct topologies that cover all possible Feynman integrals of
a given process at any loop level, and to produce visual files for the
diagrams via TikZ-Feynman, a LaTeX package that can create beautiful and clear
diagrams.

In this paper, we give a detailed description of the main functions and
workflow of the program, as well as the installation instructions and test
examples. The paper also showcases some applications of the program for
generating process amplitudes at different loop levels, with examples ranging
from Drell-Yan process to top quark decay.

Note Added: This package has recently had some applications for SMEFT one-loop
matching, for which we are working on a tiny derivative of this package.

## Acknowledgements

The authors want to thank Long-Bin Chen, Haitao Li, Yan-Qing Ma, Jian Wang,
Yefan Wang and Di Wu for useful discussions and valuable suggestions. This
work was supported by the National Natural Science Foundation of China under
grant No. 12075251.

## Appendices

##  Appendix A Canonicalization of Feynman Integrals

###  A.1 Canonical Form of Feynman Integrals

The general form of the Feynman integral is given as (for $n_{\text{ext}}$
external legs, $n_{\text{int}}$ propagators, $\ell$ loops and $D$ space-time
dimensions)

$I=\int\quantity[\prod_{r=1}^{\ell}\frac{\differential^{D}q_{r}}{\mathrm{i}\pi^{D/2}}]\prod_{j=1}^{n_{\text{int}}}\frac{1}{\quantity(P_{j}^{2}-m_{j}^{2}+\mathrm{i}\eta)^{\nu_{j}}},$
(1)

where

$P_{j}=\sum_{r=1}^{\ell}\lambda_{jr}q_{r}+\sum_{r=1}^{n_{\mathrm{ext}}-1}\sigma_{jr}k_{r}.$
(2)

Here $\quantity{q_{r}}_{r=1}^{\ell}$ is the set of loop momenta and
$\quantity{k_{r}}_{r=1}^{n_{\mathrm{ext}}}$ is the set of external momenta,
which is assumed to be linearly independent. There are $n_{\text{int}}$
propagators with momenta $\quantity{P_{j}}_{j=1}^{\ell}$ and masses
$\quantity{m_{j}}_{j=1}^{n_{\text{int}}}$. The coefficients $\lambda_{jr}$’s
and $\sigma_{jr}$’s are determined by the momentum conservation and the
momentum symmetry.

Figure 1:  A simple example of 2-loop Feynman diagram. $K_{j}$’s for massive
external momenta and $k_{j}$’s for massless external momenta. The thick lines
represent the massive propagators and massive external particles with mass
$m$, i.e., $K_{j}^{2}=m^{2}$ here.

For example, the 2-loop Feynman integral of Fig. 1 is given as

$I_{2}=\int\frac{\differential[D]{q_{1}}}{\mathrm{i}\pi^{D/2}}\frac{\differential[D]{q_{2}}}{\mathrm{i}\pi^{D/2}}\frac{1}{\mathcal{D}_{1}\mathcal{D}_{2}\cdots\mathcal{D}_{7}},$
(3)

where

$\displaystyle\mathcal{D}_{1}$ $\displaystyle=q_{1}^{2}+\mathrm{i}\eta,$ (4)
$\displaystyle\mathcal{D}_{2}$ $\displaystyle=q_{2}^{2}+\mathrm{i}\eta,$ (5)
$\displaystyle\mathcal{D}_{3}$
$\displaystyle=(q_{1}+K_{2})^{2}-m^{2}+\mathrm{i}\eta,$ (6)
$\displaystyle\mathcal{D}_{4}$
$\displaystyle=(q_{1}-k_{1})^{2}+\mathrm{i}\eta,$ (7)
$\displaystyle\mathcal{D}_{5}$
$\displaystyle=(q_{2}+K_{3})^{2}-m^{2}+\mathrm{i}\eta,$ (8)
$\displaystyle\mathcal{D}_{6}$
$\displaystyle=(q_{1}-q_{2}-k_{1})^{2}+\mathrm{i}\eta,$ (9)
$\displaystyle\mathcal{D}_{7}$
$\displaystyle=(q_{1}-q_{2}+K_{2}-K_{3})^{2}+\mathrm{i}\eta.$ (10)

However, this form is not unique. The momentum shift

$\left\\{\begin{aligned} q_{1}&\to q_{1}^{\prime}+k_{1}\\\ q_{2}&\to
q_{1}^{\prime}+q_{2}^{\prime}-k_{1}\end{aligned}\right.$ (11)

could be applied to the above integral, and the new integral is given as

$I_{2}^{\prime}=\int\frac{\differential[D]{q_{1}^{\prime}}}{\mathrm{i}\pi^{D/2}}\frac{\differential[D]{q_{2}^{\prime}}}{\mathrm{i}\pi^{D/2}}\frac{1}{\mathcal{D}_{1}^{\prime}\mathcal{D}_{2}^{\prime}\cdots\mathcal{D}_{7}^{\prime}},$
(12)

where

$\displaystyle\mathcal{D}_{1}^{\prime}$
$\displaystyle=(q_{1}^{\prime}+k_{1})^{2}+\mathrm{i}\eta,$ (13)
$\displaystyle\mathcal{D}_{2}^{\prime}$
$\displaystyle=(q_{1}^{\prime}+q_{2}^{\prime}-k_{1})^{2}+\mathrm{i}\eta,$ (14)
$\displaystyle\mathcal{D}_{3}^{\prime}$
$\displaystyle=(q_{1}^{\prime}+k_{1}+K_{2})^{2}-m^{2}+\mathrm{i}\eta,$ (15)
$\displaystyle\mathcal{D}_{4}^{\prime}$ $\displaystyle=q_{1}^{\prime
2}+\mathrm{i}\eta,$ (16) $\displaystyle\mathcal{D}_{5}^{\prime}$
$\displaystyle=(q_{1}^{\prime}+q_{2}^{\prime}-k_{1}+K_{3})^{2}-m^{2}+\mathrm{i}\eta,$
(17) $\displaystyle\mathcal{D}_{6}^{\prime}$
$\displaystyle=(q_{2}^{\prime}-k_{1})^{2}+\mathrm{i}\eta,$ (18)
$\displaystyle\mathcal{D}_{7}^{\prime}$
$\displaystyle=(q_{1}^{\prime}-2k_{1}+K_{2}-K_{3})^{2}+\mathrm{i}\eta.$ (19)

Obviously, $I_{2}\equiv I_{2}^{\prime}$. The momentum shift, however,
introduces redundancy in the description of the Feynman Integrals, which
hinders our handling of the Feynman integrals, e.g. the integral-by-part (IBP)
reduction.

The redundancy can be removed by the canonicalization of the Feynman
integrals, i.e., we choose the unique form from the equivalent Feynman
integrals to represent all of them. The unique form is called the canonical
form in the following discussions. The requirements that the canonical form of
the Feynman integral in the FeAmGen.jl should meet are shown as follows:

  1. 1.

The coefficients in Eq. 2 should satisfy

$\lambda_{jr}=\left\\{\begin{matrix}\delta_{jr}&j\leq\ell,\\\ 0\text{ or
}1&j>\ell,\end{matrix}\right.\mbox{\quad
and\quad}\sigma_{jr}=\left\\{\begin{matrix}0&j\leq\ell,\\\ 0\text{ or }\pm
1&j>\ell.\end{matrix}\right.$ (20)

  2. 2.

The relevant vacuum integral for the Feynman integral Eq. 1 could be expressed
as

$I_{\text{vac}}\sim\int\quantity[\prod_{r=1}^{\ell}\frac{\differential^{D}q_{r}}{\mathrm{i}\pi^{D/2}}]\prod_{j=1}^{n_{\text{int}}}\frac{1}{\quantity(P_{j,\mathrm{vac}}^{2}-m_{j}^{2}+\mathrm{i}\eta)^{\nu_{j}}},$
(21)

where

$P_{j,\mathrm{vac}}=\sum_{r=1}^{\ell}\lambda_{jr}q_{r}.$ (22)

In the cases of up to 4 loops, the vacuum momenta
$\quantity{P_{j,\mathrm{vac}}}_{j=1}^{n_{\mathrm{int}}}$ of the canonical form
of the Feynman integral should be the subset of any of the following sets:

     * (a)

1-loop: $\quantity{q_{1}}$;

     * (b)

2-loop: $\quantity{q_{1},q_{2},q_{1}+q_{2}}$;

     * (c)

3-loop:

       * i.

$\quantity{q_{1},q_{2},q_{3},q_{1}+q_{3},q_{2}+q_{3},q_{1}+q_{2}+q_{3}}$;

       * ii.

$\quantity{q_{1},q_{2},q_{3},q_{1}+q_{2},q_{1}+q_{3},q_{2}+q_{3}}$;

     * (d)

4-loop:

       * i.

$\\{q_{1},q_{2},q_{3},q_{4},q_{1}+q_{2},q_{1}+q_{3},q_{2}+q_{4},q_{1}+q_{2}+q_{3},q_{1}+q_{2}+q_{4},q_{1}+q_{2}+q_{3}+q_{4}\\}$;

       * ii.

$\\{q_{1},q_{2},q_{3},q_{4},q_{1}+q_{2},q_{1}+q_{3},q_{2}+q_{4},q_{3}+q_{4},q_{2}+q_{3}+q_{4},q_{1}+q_{2}+q_{3}+q_{4}\\}$;

       * iii.

$\\{q_{1},q_{2},q_{3},q_{4},q_{1}+q_{2},q_{1}+q_{3},q_{2}+q_{3},q_{2}+q_{4},q_{1}+q_{2}+q_{4},q_{2}+q_{3}+q_{4}\\}$.

  3. 3.

In the propagator momenta $P_{j}$’s, the coefficients of the loop momenta
$q$’s and external momenta $k$’s (for massless particles and $K$’s for massive
particles) should be $0$ or $\pm 1$.

Although these three requirements can filter out most of the momentum shifts,
there are still a large number of possibilities that will not be removed. We
will define an order to sort all remaining possible forms of the Feynman
integrals and define the one with the smallest order as the canonical form. In
the next sub-section, we will describe the algorithm of canonicalization in
detail.

###  A.2 Canonicalization Algorithm

The first part of the canonicalization algorithm is to find the possible
momentum shifts that can be applied to the Feynman integral. It is important
to note that the momentum shift is a transformation of the loop momenta, which
are the direct integration variables of the Feynman integral. After such a
momentum shift, we hope that the general form of Feynman integral of Eq. 1
should be preserved, which requires

$\absolutevalue{\det T}\equiv 1,$ (23)

where $T$ is the transformation matrix of the momentum shift, i.e., $q\to
q^{\prime}=Tq$. For simplicity, we only consider the momentum shift that could
be derived from the original Feynman diagrams.

Figure 2: Internal graph of Fig. 1, where the number $i$ label the vertex
connecting the $i$-th external leg that has been removed.

Firstly, we consider the momenta shift without external momenta. We introduce
the internal graph $G_{\text{int}}$ associated to the Feynman diagram, which
is obtained by removing all external legs but preserving the vertices
connecting external legs from the Feynman diagram. For example, the internal
graph of Fig. 1 is shown in Fig. 2. If the external momenta are set to 0,
there are only three unique propagator momenta in this internal graph, e.g.,
$\quantity{q_{1},q_{2},q_{1}\pm q_{2}}$. One could choose any two propagators
from the internal graph, and make their propagator momenta as
$\pm\tilde{q}_{1}$ or $\pm\tilde{q}_{2}$. If the chosen propagators are valid,
i.e., the corresponding propagator momenta are linearly independent, then the
momentum shift could be calculated by solving the linear equations
$\tilde{q}=\tilde{T}q$, where $\tilde{T}_{im}=\pm\lambda_{jm}$. Here $j$ is
the chosen propagator index, and $i$ is the shifted loop momentum index. The
momentum shift without external momenta is simply given as
$q\to\tilde{T}^{-1}\tilde{q}$.

The external momenta are then considered. Following the same procedure, we
could choose any two propagators from the original graph, and make their
propagator momenta as $\pm q_{1}^{\prime}$ or $\pm q_{2}^{\prime}$. Obviously,

$q_{i}^{\prime}=\tilde{q}_{i}\pm\sum_{m=1}^{n_{\mathrm{ext}}-1}\sigma_{jm}k_{m},$
(24)

where $\tilde{q}_{i}$’s are the corresponding shifted loop momenta without the
consideration of external momenta. Then,

$q_{i}^{\prime}\mp\sum_{m=1}^{n_{\mathrm{ext}}-1}\sigma_{jm}k_{m}=\quantity(\tilde{T}q)_{i}.$
(25)

Finally, we have the momentum shift with external momenta as

$q\to\tilde{T}^{-1}\quantity(q^{\prime}\mp\sigma\cdot k).$ (26)

The algorithm will search all possible momentum shifts that could be derived
from the original Feynman diagrams as described above. Then, the requirements
of the canonical form of the Feynman integrals will be checked. However, an
enormous number of possible momentum shifts are still allowed. We propose an
order to sort all possible shifted propagator momenta, and the leading one
will be chosen as the canonical form. The order is defined as a two-entry
tuple of integers. Considering the case with 3 loops and 4 external legs (3
independent external momenta), the first entry is calculated by the following
steps

  1. 1.

For every propagator momentum, the normalization should be applied. This means
that the coefficient of the first loop momentum $q_{i}$ with a non-zero
coefficient in the propagator momenta should be normalized to $+1$, where the
first means the smallest index of the loop momenta. For instance,
$-q_{2}+q_{3}-K_{1}+k_{2}$ should be normalized to $q_{2}-q_{3}+K_{1}-k_{2}$.

  2. 2.

Because of the second and the third requirements of the canonical form, the
coefficients of all loop momenta and external momenta are either $0$ or $\pm
1$. For every propagator momentum, we could calculate the coefficients of all
loop momenta and external momenta. For instance, the coefficients of
normalized propagator momentum $q_{2}-q_{3}+K_{1}-k_{2}$ are

$\begin{matrix}k_{3}&k_{2}&K_{1}&q_{3}&q_{2}&q_{1}\\\ 0&-1&+1&-1&+1&0\\\
0&2&1&2&1&0\end{matrix}$ (27)

where we substitute $-1$ with $2$ in the last line. Now the the digit sequence
of the coefficients of all loop momenta and external momenta is obtained as
$021210$, which could be converted to a ternary number
$(21210)_{3}=(210)_{10}$, which is called propagator momentum number here.

  3. 3.

Finally, the first entry of the order is the product of the propagator
momentum numbers of all propagator momenta.

The second entry is the SHA-256 number calculated from the string of the list
of the propagator momenta where the propagator momenta is sorted by
lexicographical order.141414See https://docs.julialang.org/en/v1/stdlib/SHA
for more information.

Thus, the canonical form of the Feynman integral is obtained.

##  Appendix B Feynman Integral Topology

For a general form of scalar Feynman integral of

$I[\nu_{1}\nu_{2}\cdots\nu_{\text{int}}]=\int\quantity[\prod_{r=1}^{\ell}\frac{\differential^{D}q_{r}}{\mathrm{i}\pi^{D/2}}]\prod_{j=1}^{n_{\text{int}}}\frac{1}{\quantity(P_{j}^{2}-m_{j}^{2}+\mathrm{i}\eta)^{\nu_{j}}},$
(28)

where $\nu_{i}\in\mathbb{Z}$. We call the set of $n_{\text{int}}$ denominators
$\\{P_{j}^{2}-m_{j}^{2}+\mathrm{i}\eta\\}_{j=1}^{n_{\text{int}}}$ as the
topology of this scalar Feynman integrals. There are, however,
$N_{V}$($\equiv(\ell+1)\ell/2+\ell(n_{\text{ext}}-1)$) independent scalar
products (ISP) of loop momenta and external momenta for $\ell$-loop
$n_{\text{ext}}$-leg Feynman diagrams. If $n_{\text{int}}<N_{V}$ or not all
ISPs can be represented as the linear combination of all the inverses of
propagators, some additional propagators should be added to make the topology
complete. [28]

The well-known algorithm to solve this problem is the Pak’s algorithm [38] and
its derivations [39, 40]. Instead of Pak’s algorithm, FeAmGen.jl apply another
algorithm similar to the algorithm of canonicalization of Feynman integrals
which is described in A for constructing the topology of Feynman integrals.
The algorithm is described as follows.

  1. 1.

Without momentum shift: In this approach, FeAmGe.jl do not consider the
momentum shift. Then the function construct_den_topology will construct the
minimum topology that can cover as many as Feynman diagrams as possible.
Finally, the generated topologies will be completed.

  2. 2.

With mometnum shift:

     * (a)

Without reference denominator topologies: In this way, the reference
denominator topologies is set to be empty. Then the first denominator topology
will be added into the reference denominator topologies. For the rest
denominator topologies, we will search the momentum shift that can make the
denominator topology covered by the reference denominator topologies or
covering one of the reference denominator topologies. If the momentum shift is
found, the shifted denominator topology will be chosen and added into the
reference denominator topologies. Otherwise, the original denominator topology
will be added into the reference denominator topologies directly. Finally, all
of chosen denominator topologies will be completed.

     * (b)

With reference denominator topologies: In this approach, the reference
denominator topologies is given. The rest is the same as the previous
approach.

To complete the topology, some additional propagators should be added to the
denominator topologies. The additional propagators are constructed from the
linear combination of loop mometna in the second requirement for canonical
form of Feynman integrals, which is described in A. For example, we consider
the independent external momenta
$K_{\mathrm{ext}}=\quantity{K_{1},k_{2},k_{3}}$ and the propagator momenta is
a subset of the first case in the 3-loop, i.e. a subset of
$Q_{\mathrm{int}}=\quantity{q_{1},q_{2},q_{3},q_{1}+q_{3},q_{2}+q_{3},q_{1}+q_{2}+q_{3}}$.
Then, the ISPs in $\quantity{\quantity(Q\pm K)^{2}\middle|Q\in
Q_{\mathrm{int}}\wedge K\in K_{\mathrm{ext}}}$ will be checked whether they
can be represented as the linear combination of all the inverses of
propagators. If not, the additional propagators will be added to the
denominator topologies.

Finally, the complete topologies are constructed.

## References

  * [1] M. E. Peskin, D. V. Schroeder, An Introduction to Quantum Field Theory, Addison-Wesley Pub. Co., Reading, USA, 1995.
  * [2] M. Veltman, Diagrammatica, 1st Edition, Cambridge University Press, 1994.  doi:10.1017/CBO9780511564079.
  * [3] J. Butler, R. Chivukula, A. de Gouvea, et al., Report of the 2021 U.S. Community Study on the Future of Particle Physics Snowmass 2021, in: Snowmass, Seattle, WA (United States), September 2021, US DOE, 2023.  doi:10.2172/1922503.
  * [4] Particle Data Group, Review of Particle Physics, Prog. Theo. and Exp. Phys. 2022 (8) (aug 2022).  doi:10.1093/ptep/ptac097.
  * [5] The CMS Collaboration, Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC, Phys. Lett. B 716 (1) (2012) 30–61.  arXiv:1207.7235, doi:10.1016/j.physletb.2012.08.021.
  * [6] The ATLAS Collaboration, Observation of a New Particle in the Search for the Standard Model Higgs Boson with the ATLAS Detector at the LHC, Phys. Lett. B 716 (1) (2012) 1–29.  doi:10.1016/j.physletb.2012.08.020.
  * [7] G. Apollinari, O. Brüning, T. Nakamoto, L. Rossi, High Luminosity Large Hadron Collider HL-LHC, CERN Yellow Rep. (5) (2015) 1–19.  arXiv:1705.08830, doi:10.5170/CERN-2015-005.1.
  * [8] The International Linear Collider Technical Design Report - Volume 2: Physics (6 2013).  arXiv:1306.6352.
  * [9] The International Linear Collider Technical Design Report - Volume 1: Executive Summary (6 2013).  arXiv:1306.6327.
  * [10] P. Bambade, et al., The International Linear Collider: A Global Project (3 2019).  arXiv:1903.01629.
  * [11] M. Dong, et al., CEPC Conceptual Design Report: Volume 2 - Physics & Detector (11 2018).  arXiv:1811.10545.
  * [12] CEPC Conceptual Design Report: Volume 1 - Accelerator (9 2018).  arXiv:1809.00285.
  * [13] M. Bicer, et al., First Look at the Physics Case of TLEP, JHEP 01 (2014) 164.  arXiv:1308.6176, doi:10.1007/JHEP01(2014)164.
  * [14] A. Abada, et al., FCC Physics Opportunities: Future Circular Collider Conceptual Design Report Volume 1, Eur. Phys. J. C 79 (6) (2019) 474.  doi:10.1140/epjc/s10052-019-6904-3.
  * [15] A. Abada, et al., FCC-ee: The Lepton Collider: Future Circular Collider Conceptual Design Report Volume 2, Eur. Phys. J. ST 228 (2) (2019) 261–623.  doi:10.1140/epjst/e2019-900045-4.
  * [16] The CEPC Study Group, CEPC Conceptual Design Report: Volume 1 – Accelerator (Sep. 2018).  arXiv:1809.00285.
  * [17] The CEPC Study Group, CEPC Conceptual Design Report: Volume 2 – Physics & Detector (Nov. 2018).  arXiv:1811.10545.
  * [18] The FCC Collaboration, FCC-ee: The Lepton Collider: Future Circular Collider Conceptual Design Report Volume 2, Eur. Phys. J. Special Topics 228 (2) (2019) 261–623.  doi:10.1140/epjst/e2019-900045-4.
  * [19] The FCC Collaboration, FCC-hh: The Hadron Collider: Future Circular Collider Conceptual Design Report Volume 3, Eur. Phys. J. Special Topics 228 (4) (2019) 755–1107.  doi:10.1140/epjst/e2019-900087-0.
  * [20] The FCC Collaboration, FCC Physics Opportunities: Future Circular Collider Conceptual Design Report Volume 1, Eur. Phys. J. C 79 (6) (jun 2019).  doi:10.1140/epjc/s10052-019-6904-3.
  * [21] E. Ploerer, QCD at the Future Circular ${e^{+}e^{-}}$ Collider (Aug. 2022).  arXiv:2208.08250.
  * [22] The ILC Collaboration, The International Linear Collider: Report to Snowmass 2021 (Mar. 2022).  arXiv:2203.07622.
  * [23] The ILC Collaboration, The International Linear Collider Technical Design Report – Volume 2: Physics (Jun. 2013).  arXiv:1306.6352.
  * [24] X. Liu, Y.-Q. Ma, AMFlow: A Mathematica Package for Feynman Integrals Computation via Auxiliary Mass Flow, Comput. Phys. Commun. 283 (2023) 108565.  arXiv:2201.11669, doi:10.1016/j.cpc.2022.108565.
  * [25] P. Nogueira, Automatic Feynman Graph Generation, J. Comput. Phys. 105 (2) (1993) 279–289.  doi:10.1006/jcph.1993.1074.
  * [26] Z. Li, Y. Wang, Q.-f. Wu, Categorization of Two-loop Feynman Diagrams in the $\mathcal{O}(\alpha^{2})$ Correction to $e^{+}e^{-}\to ZH$, Chin. Phys. C 45 (5) (2021) 053102.  arXiv:2012.12513, doi:10.1088/1674-1137/abe84d.
  * [27] J. Küblbeck, M. Böhm, A. Denner, Feyn arts – Computer-algebraic Generation of Feynman Graphs and Amplitudes, Comput. Phys. Commun. 60 (2) (1990) 165–180.  doi:10.1016/0010-4655(90)90001-H.
  * [28] S. Weinzierl, Feynman Integrals, Springer International Publishing, 2022.  arXiv:2201.03593, doi:10.1007/978-3-030-99558-4.
  * [29] T. Hahn, Generating Feynman Diagrams and Amplitudes with FeynArts 3, Comput. Phys. Commun. 140 (3) (2001) 418–431.  arXiv:hep-ph/0012260, doi:10.1016/S0010-4655(01)00290-9.
  * [30] R. Mertig, M. Böhm, A. Denner, Feyn Calc – Computer-Algebraic Calculation of Feynman Amplitudes, Comput. Phys. Commun. 64 (3) (1991) 345–359.  doi:10.1016/0010-4655(91)90130-D.
  * [31] V. Shtabovenko, R. Mertig, F. Orellana, New Developments in FeynCalc 9.0, Comput. Phys. Commun. 207 (2016) 432–444.  arXiv:1601.01167, doi:10.1016/j.cpc.2016.06.008.
  * [32] V. Shtabovenko, R. Mertig, F. Orellana, FeynCalc 9.3: New features and improvements, Comput. Phys. Commun. 256 (2020) 107478.  arXiv:2001.04407, doi:10.1016/j.cpc.2020.107478.
  * [33] L. Darmé, C. Degrande, C. Duhr, et al., UFO 2.0 – The Universal Feynman Output format (Apr. 2023).  arXiv:2304.09883.
  * [34] C. Degrande, C. Duhr, B. Fuks, et al., UFO –The Universal FeynRules Output, Comput. Phys. Commun. 183 (6) (2012) 1201–1214.  doi:10.1016/j.cpc.2012.01.022.
  * [35] J. P. Ellis, TikZ-Feynman: Feynman diagrams with TikZ, Comput. Phys. Commun. 210 (2017) 103–123.  doi:10.1016/j.cpc.2016.08.019.
  * [36] A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, B. Fuks, FeynRules 2.0 – A Complete Toolbox for Tree-level Phenomenology, Comput. Phys. Commun. 185 (8) (2013) 2250–2300.  arXiv:1310.1921, doi:10.1016/j.cpc.2014.04.012.
  * [37] N. D. Christensen, C. Duhr, FeynRules – Feynman Rules Made Easy, Comput. Phys. Commun. 180 (9) (2008) 1614–1641.  arXiv:0806.4194, doi:10.1016/j.cpc.2009.02.018.
  * [38] A. Pak, The Toolbox of modern multi-loop calculations: novel analytic and semi-analytic techniques, J. Phys. Conf. Ser. 368 (2012) 012049.  arXiv:1111.0868, doi:10.1088/1742-6596/368/1/012049.
  * [39] V. Shtabovenko, FeynCalc goes multiloop, J. Phys. Conf. Ser. 2438 (1) (2023) 012140.  arXiv:2112.14132, doi:10.1088/1742-6596/2438/1/012140.
  * [40] Z. Wu, J. Boehm, R. Ma, H. Xu, Y. Zhang, NeatIBP 1.0, A package generating small-size integration-by-parts relations for Feynman integrals (5 2023).  arXiv:2305.08783.