aainstitutetext: Beijing Computational Science Research Center, Beijing
100084, Chinabbinstitutetext: Zhejiang Institute of Modern Physics, Zhejiang
University, Hangzhou, 310027, P. R. Chinaccinstitutetext:  Center of
Mathematical Science, Zhejiang University, Hangzhou, 310027, P. R.
Chinaddinstitutetext: Peng Huanwu Center for Fundamental Theory, Hefei, Anhui
230026, China

#  Module Intersection and Uniform Formula for Iterative Reduction of One-loop
Integrals

Jiaqi Chen a,b,c,d Bo Feng<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we develop an iterative sector-level reduction strategy for
Feynman integrals, which bases on module intersection in the Baikov
representation and auxiliary vector for tensor structure. Using this strategy
we have studied the reduction of general one-loop integrals, i.e., integrals
having arbitrary tensor structures and arbitrary power for propagators.
Inspired by these studies, a uniform and compact formula that iteratively
reduces all one-loop integrals has been written down, where messy polynomials
in integration-by-parts (IBP) relations have organized themselves to Gram
determinants.

##  1 Introduction

Particle physics is highly developed and comes to the era of precise
measurement, which calls for high precision theoretical predictions. One of
the most important parts of theoretical predictions is perturbation
computations in quantum field theory. For such computations, reducing general
Feynman integrals into master integrals plays an important role
Chetyrkin:1981qh . It not only greatly reduces the number of integrals to be
calculated, but also produces the differential equations Kotikov:1990kg ;
Gehrmann:1999as ; Argeri:2007up ; Henn:2014qga  for solving the master
integrals. Based on differential equations, many analytical or numerical
methods and packages for solving master integrals have been developed
Henn:2013pwa ; Moriello:2019yhu ; Bonciani:2019jyb ; Frellesvig:2019byn ;
Hidding:2020ytt ; Liu:2017jxz ; Liu:2022tji ; Liu:2022mfb ; Liu:2022chg ;
Liu:2021wks ; Liu:2020kpc ; Armadillo:2022ugh , which can be widely applied to
multi-loop and multi-scale processes.

The reduction problem can be roughly divided into the reduction of loop
momenta in the numerator (called ”tensor reduction” or TR) and the reduction
of propagators in the denominator with general powers (called ”denominator
reduction” or DR). These two kinds of reductions (i.e., TR and DR) are tightly
connected111For example, in Feng:2022uqp  one has used the tensor reduction to
solve reduction with general denominators for one-loop integrals.. The well-
established method to do reduction is the integration-by-parts (IBP) method
introduced in Chetyrkin:1981qh . When combining with the Laporta algorithm
Laporta:2000dsw , several widely used packages have been developed such as
Smirnov:2019qkx ; vonManteuffel:2012np ; Lee:2012cn ; Klappert:2020nbg . IBP
relations give many linear relations between various scalar integrals and
solving them gives relations between a given integral and the master integrals
(i.e., the reduction). When it comes to multi-loop and multi-scale integrals
required by higher precision theoretical predictionw, the size of linear
equations generated by IBP relations increases dramatically. Solving them
becomes more and more difficult. Motivated by this, many developments of
reduction methods have been explored for higher loops222For one-loop
integrals, two very efficient methods are the unitarity cut method Bern:1994zx
; Bern:1994cg ; Britto:2004nc ; Britto:2005ha  for reduction at the integral
level and OPP method Ossola:2006us  for reduction at the integrand level.
Gluza:2010ws ; Peraro:2019svx ; Chestnov:2022alh ; Mastrolia:2018uzb ;
Frellesvig:2019kgj ; Frellesvig:2019uqt ; Weinzierl:2020xyy ; Mizera:2019ose ;
Frellesvig:2020qot ; Liu:2018dmc ; Guan:2019bcx ; Larsen:2015ped ;
Larsen:2016tdk ; Zhang:2016kfo ; Georgoudis:2016wff ; Georgoudis:2017iza ;
Bohm:2017qme ; Bohm:2018bdy ; Bendle:2019csk ; Boehm:2020zig ; Bendle:2021ueg
; Feng:2022iuc .

Most of the above methods solve a hugely redundant linear system with many
equations using programs. On one side, it makes the computation automatically.
On the other side, it works likes in a black box. Although we have obtained
the wanted results, we lost a full understanding of these equations. Knowing
these details not only helps overcome difficulties faced by these methods but
also gives a better understanding of the mathematical structures of general
Feynman integrals.

For example, there are some iterative structures in reduction. Let us consider
the bubble topology with propagators:

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \
z_{2}=(l-p)^{2}-m_{2}^{2}.~{}~{}~{}$ (1)

There are three master integrals defined by

$\displaystyle\vec{f}=\\{\text{I}_{1,0},\text{I}_{0,1},\text{I}_{1,1}\\},~{}~{}~{}~{}\text{I}_{a_{1},a_{2}}\equiv\int\frac{d^{d}l}{i(\pi)^{d/2}}\frac{1}{z_{1}^{a_{1}}z_{2}^{a_{2}}}~{}.~{}~{}~{}$
(2)

One kind of differential equations of master integrals is given by

$\displaystyle\partial_{m_{1}}\vec{f}=A_{m_{1}}\vec{f}~{}.~{}~{}$ (3)

It is easy to see when taking $\partial_{m_{1}}$ on $\vec{f}$ iteratively we
have

$\displaystyle\partial_{m_{1}}^{n}\vec{f}=\\{(n-1)!\ \text{I}_{n+1,0},\ 0,\
(n-1)!\ \text{I}_{n+1,1}\\}=A_{m_{1}}^{(n)}\vec{f}~{}~{}~{}$ (4)

where

$\displaystyle
A_{m_{1}}^{(n)}=\partial_{m_{1}}A_{m_{1}}^{(n-1)}+A_{m_{1}}^{(n-1)}.\
A_{m_{1}}~{}~{}~{}$ (5)

The iterative structure of $A_{m_{1}}^{(n)}$ in (5) gives a very simple way to
find the reduction of $\text{I}_{n+1,0}$ and $\text{I}_{n+1,1}$ (DR type)
using (4). This is an iterative reduction relation at topology-level333When we
say a reduction is at the topology-level, we mean that the integral is written
as the combination of master integrals of the same topology and master
integrals of the sub-topologies. . While the iterative relation at the
topology-level is powerful, it is also not easy to get. As in this example,
the $A_{m_{1}}$ is usually calculated by the traditional IBP method, which is
difficult to get for more complicated cases.

Another well-known phenomenon of reduction is that the reduction can be
organized into different sectors. Usually, the reduction in the top sector can
be done easily using some tricks, for example, the maximum cut for the one-
loop box in Britto:2004nc . However, such a trick lost the information of sub-
sectors. Thus if there is a way such that the reduction to its top sector is
not much harder than the maximal cut, but the complete information of sub-
sectors has been kept, we can carry out the whole reduction top-down (or
triangulated) by treating each sub-sector as another ”top sector”. This
motivates us to consider the iterative structure at the sector-level but not
the topology-level444When we say the reduction is at the sector-level, we mean
that the integral is written as the combination of master integrals of the
same topology and integrals (not need to be master integrals) of the sub-
topologies. .

In this paper, we will explore more information hidden inside the IBP
relations. The method we will introduce mainly bases on syzygy and module
intersection Gluza:2010ws ; Schabinger:2011dz ; Larsen:2015ped ;
Larsen:2016tdk ; Zhang:2016kfo ; Georgoudis:2016wff ; Georgoudis:2017iza ;
Bohm:2017qme ; Bohm:2018bdy ; Bendle:2019csk ; Boehm:2020zig ; Bendle:2021ueg
in Baikov representation Baikov:1996iu . We will give a quick review in Sec.2
for Baikov representation and module intersection. The main point of our
method comparing to the module intersection in Bendle:2019csk  is that we will
not generate all IBP relations with basis obtained by the module intersection
and solve these redundant linear equations. On the contrary, we pick some
elements in the module intersection. These elements generate iterative
reduction relations for corresponding sector. Using just a few relations, one
can iteratively reduce any integral in this sector to master integrals of this
sector, and keep the complete information of sub-sectors. In such a strategy
of iteration at the sector-level, iterative reduction relations are easier to
obtain and they avoid the huge redundancy in linear system of traditional IBP
methods. To demonstrate our method, we will present one TR example and one DR
example in Sec.4 and discuss the similarity of these two examples. Inspired by
examples in the previous section, we construct an uniform formula in Sec.5. It
solves both TR and DR for general one-loop integrals. In Sec.6, we give a
brief discussion for degenerated situations, i.e., kinematics and masses take
some specific values, such as null momenta or on-shell momenta. We will find
that naively applying the uniform formula laid out in Sec.5 sometimes does not
lead to the simplest iterative relation and more careful treatment is needed.
However the method introduced in Sec.3 still works well by giving the simplest
iterative relation. Finally, we give a brief discussion in the Sec.7.

##  2 Baikov representation and module intersection

In this section, we will review the Baikov representation of integrals
Baikov:1996iu . In this frame, it is easier to implement the module
intersection as will be discussed shortly. The Baikov representation
transforms integrals in the standard form obtained from Feynman rules by
changing integral variables from $\prod_{i}d^{d}l_{i}$ to $\prod_{j}dz_{j}$,
where each $z_{i}$ represents a propagator (or related Lorentz invariant
scalar product involving loop momenta). For one-loop integrals, which are the
focus of our current paper, we denote propagators and integrals as

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \ z_{2}=(l+p_{1})^{2}-m_{2}^{2},\ \ \
z_{3}=(l+p_{1}+p_{2})^{2}-m_{3}^{2},\cdots\ $ $\displaystyle
z_{n}=(l+p_{1}+\dots+p_{E})^{2}-m_{n}^{2}.$
$\displaystyle\text{I}_{\\{a_{i}\\}}\equiv\text{I}_{a_{1},a_{2},\cdots,a_{n}}\equiv\int\frac{d^{d}l}{i(\pi)^{d/2}}\frac{1}{\prod_{i=1}^{n}z_{i}^{a_{i}}},~{}~{}~{}$
(6)

where E is the number of independent external momenta and $n=E+1$. The Baikov
representation of integrals is

$\displaystyle\text{I}_{a_{1},a_{2},\cdots,a_{E+1}}$ $\displaystyle=\int
C_{n}(d)\mathcal{K}^{-(d-n)/2}\mathcal{G}(\bm{z})^{(d-n-1)/2}\frac{dz_{i}}{\prod_{i=1}^{n}z_{i}^{a_{i}}}~{}~{}~{}$
(7)

where the constant-coefficient $C_{n}(d)$ and the Gram determinant
$\mathcal{K}$ of external momenta do not involve $z_{i}$, so they do not
affect our later discussions and can be ignored. The $\mathcal{G}(\bm{z})$ is
another Gram determinant depending on both loop momentum and external momenta,
i.e.,

$\displaystyle\mathcal{G}(\bm{z})=G(l,p_{1},\cdots,p_{E})~{}~{}~{}$ (8)

with $G$ defined as

$G(q_{1},\ldots,q_{n})\equiv\det(q_{i}\cdot
q_{j})\equiv\det\begin{pmatrix}q_{1}\cdot q_{1}&q_{1}\cdot
q_{2}&\cdots&q_{1}\cdot q_{n}\\\ q_{2}\cdot q_{1}&q_{2}\cdot q_{2}&&\vdots\\\
\vdots&&\ddots&\vdots\\\ q_{n}\cdot q_{1}&\cdots&\cdots&q_{n}\cdot
q_{n}\end{pmatrix}\,.~{}~{}~{}$ (9)

The well-established IBP relations can also be easily implemented in the
Baikov representation. However, the differentiation on $\mathcal{G}$ will
change its power, which is equivalent to shifting the space-time dimension $d$
to different values.

To avoid such a situation, the syzygy module is introduced Larsen:2015ped .
Let us consider the IBP relation

$\displaystyle
C\int\sum_{i=1}^{n}\left[\partial_{z_{i}}\left(P_{i}\frac{1}{\prod_{i=1}^{n}z_{i}^{a_{i}}}\mathcal{G}(\bm{z})^{(d-n-1)/2}\right)\right]\prod_{i=1}^{n}dz_{i}$
(10)

with $P_{j}$s being polynomials of $z_{i}$, one can see that if these
$P_{i}$’s are properly chosen, i.e., they satisfy

$\displaystyle\sum_{i}^{n}(P_{i}\partial_{z_{i}}\mathcal{G})+P_{0}\mathcal{G}=0,$
(11)

the power of $\mathcal{G}$ will not be shifted. The relation (11) is a syzygy
equation for the set of $(n+1)$ polynomials

$\displaystyle\left\langle\partial_{z_{1}}\mathcal{G},\cdots,\partial_{z_{n}}\mathcal{G},\mathcal{G}\right\rangle~{}~{}~{}$
(12)

All solutions of (11) give the syzygy module of the set (12). Putting every
solution back to (10) we get an IBP relation with a given $\\{a_{i}\\}$ set,
which does not involve dimension shift.

For later convenience we define the following notations:

$\displaystyle\left\langle P\right\rangle=\left\langle
P_{1},P_{2},\cdots,P_{n},P_{0}\right\rangle$ $\displaystyle D_{\left\langle
P\right\rangle}\equiv\left\\{D_{P_{1}},\cdots,D_{P_{n}},D_{P_{0}}\right\\}\equiv\left\\{\partial_{z_{1}}\left(P_{1}\bm{\cdot}\right),\cdots,\partial_{z_{n}}\left(P_{n}\bm{\cdot}\right),\frac{d-n-1}{2}P_{0}\bm{\cdot}\right\\}$
$\displaystyle D_{\left\langle
P\right\rangle}\bm{\cdot}Q\equiv-\sum_{i=1}^{n}\left[\partial_{z_{i}}\left(P_{i}\bm{\cdot}Q\right)\right]+\frac{d-n-1}{2}P_{0}\bm{\cdot}Q.~{}~{}~{}~{}$
(13)

Then the IBP relation (10) can be written in a more compact form

$\displaystyle C\int\left\\{D_{\left\langle
P\right\rangle}\bm{\cdot}\frac{1}{\prod_{i=1}^{n}z_{i}^{a_{i}}}\right\\}\mathcal{G}(\bm{z})^{(d-n-1)/2}\prod_{i=1}^{n}dz_{i}.$
(14)

The syzygy module is a linear space with a basis of generators555For one-loop
integrals, it has been proved that the number of generators is exactly the
number of propagators.

$\displaystyle\\{\bm{e}_{1},\cdots,\bm{e}_{n}\\}~{}~{}~{}~{}$ (15)

and the general solution of (11) can be written as $\left\langle
P\right\rangle=\sum_{i=1}^{n}f_{i}\bm{e}_{i}$ with $f_{i}$s being arbitrary
polynomials of $z_{i}$.

Another well-known phenomenon in IBP relation is the changing of power of
propagators in (14). The power can be increased or decreased. Among them, only
$\partial_{z_{i}}z_{i}^{-a_{i}}$ increases the power. To avoid the increase,
we can do a similar thing by requiring $\left\langle P\right\rangle$ in (13)
to be the module generated by the following basis

$\displaystyle\bm{d}_{1}=\\{z_{1},0,\cdots,0,0\\}$
$\displaystyle\bm{d}_{2}=\\{0,z_{2},\cdots,0,0\\}$ $\displaystyle\cdots$
$\displaystyle\bm{d}_{n}=\\{0,0,\cdots,z_{n},0\\}$
$\displaystyle\bm{d}_{n+1}=\\{0,0,\cdots,0,1\\}.~{}~{}~{}~{}$ (16)

Up to now, we have two modules: one is given by (11) and avoids the dimension
shift of space-time, while another is given by (16) and avoids the increase of
power of propagators. If we want to avoid both things, we just take the
intersection of the above two modules, i.e.,
$\\{\bm{h}_{i}\\}\equiv\\{\bm{e}_{i}\\}\cap\\{\bm{d}_{i}\\}$. Notice that the
syzygy module (11) and the module intersection $\bm{h}_{i}$ can be solved by
computational algebraic geometry Zhang:2016kfo , and in this work, we use the
package Singular DGPS  to do this. In the examples given in this paper, it
takes only seconds or even less to finish the computation. The syzygy of Gram
determinant can also easily be obtained by Laplace expansion of the
determinant Bohm:2017qme .

##  3 The method

In this paper, we will re-investigate the reduction problem for general one-
loop integrals, i.e., with arbitrary tensor structure and arbitrary power of
propagators. As one can see, these two different reductions, i.e., the tensor
reduction (TR) and denominator reduction (DR) can be treated uniformly by
module intersection method Larsen:2015ped ; Boehm:2020zig .

As pointed out in several papers Feng:2021enk ; Hu:2021nia ; Feng:2022uqp ;
Feng:2022iuc ; Feng:2022rwj ; Feng:2022rfz , arbitrary tensor structure can be
compactly organized using an auxiliary vector $R$. Thus for TR of one-loop
n-point integrals, we enlarge the set of propagators given in (6) by adding
one new propagator, i.e.,

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \ z_{2}=(l+p_{1})^{2}-m_{2}^{2},\ \ \
z_{3}=(l+p_{1}+p_{2})^{2}-m_{3}^{2},\cdots\ $ $\displaystyle
z_{n}=(l+p_{1}+\dots+p_{n-1})^{2}-m_{n}^{2},\ \ \ z_{n+1}=l\cdot R.~{}~{}~{}$
(17)

with the power of $z_{n+1}$ to be non-positive integer. Now the Baikov
representation becomes666By (7), the $C$ of (18) will depend on $R$ also, but
it will not influnce IBP relations derived later.

$\displaystyle\text{I}_{a_{1},a_{2},\cdots,a_{n},a_{n+1}}$
$\displaystyle=C\int\mathcal{G}(\bm{z})^{(d-n-2)/2}\frac{dz_{i}}{\prod_{i=1}^{n+1}z_{i}^{a_{i}}}$
$\displaystyle\mathcal{G}(\bm{z})$
$\displaystyle=G(l,p_{1},\cdots,p_{n-1},R)~{}~{}~{}$ (18)

Let us denote the basis of syzygy module corresponding to

$\displaystyle\left\langle\partial_{z_{1}}\mathcal{G},\cdots,\partial_{z_{n+1}}\mathcal{G},\mathcal{G}\right\rangle~{}~{}~{}$
(19)

as $\\{\bm{e}_{i}\\}$, while the basis of another module is $\\{\bm{d}_{i}\\}$
with

$\displaystyle\bm{d}_{1}=\\{z_{1},0,\cdots,0,0,0\\}$ $\displaystyle\cdots$
$\displaystyle\bm{d}_{n}=\\{0,0,\cdots,z_{n},0,0\\}$
$\displaystyle\bm{d}_{n+1}=\\{0,0,\cdots,0,1,0\\}$
$\displaystyle\bm{d}_{n+2}=\\{0,0,\cdots,0,0,1\\}~{}~{}~{}$ (20)

After obtained the module intersection
$\\{\bm{h}_{i}\\}\equiv\\{\bm{e}_{i}\\}\cap\\{\bm{d}_{i}\\}$, we use elements
in $\\{\bm{h}_{i}\\}$ to generate differential operators as in (13) and
produce corresponding IBP relations like this777It is possible that the IBP
relation can not be written to the form (21). For such a situation, we just
throw away this IBP relation. :

$\displaystyle\text{I}_{\bm{a},-r_{max}}=\sum_{j=1}^{m}c_{j}\text{I}_{\bm{a},-r_{max}+j}+l.p.p.t.,$
(21)

with $r_{max}>0$ and $a_{i}>0$ for $i<n$, where the l.p.p.t. denote terms with
lower power of propagators. More explicitly, we say $\text{I}_{\bm{b},-r_{b}}$
is a l.p.p.t. corresponding to $\text{I}_{\bm{a},-r_{a}}$, if it satisfies
$b_{i}\leq a_{i}$ for all $i\leq n$, and
$\sum_{i}^{n}b_{i}<\sum_{i}^{n}a_{i}$. Notice that when propagators’ power is

$\displaystyle\\{\bm{a},-r\\}=\\{1,\cdots,1,-r\\},~{}~{}~{}$ (22)

the l.p.p.t. are all terms of sub-sectors.

To have a nice tensor reduction relation, there are some requirements for the
(21). Firstly, the sign of power $a_{n+1}$ of $z_{n+1}$ indicates it is a
numerator or a denominator. Since we want to discuss the tensor reduction,
$a_{n+1}$ should be a non-positive integer and relation (21) should not
include any term with $a_{n+1}$ positive. Thus for any $r_{max}>0$, we should
require

$\displaystyle c_{j}=0\ \ \ \text{when}\ \ j>r_{max},$ (23)

Secondly, no $c_{j}$ becomes infinity for any $r_{max}>0$ for (21) to be well
defined.

For the reduction of propagators with arbitrary powers (i.e., the DR), the
idea is similar. Without loss of generality, let us consider how to reduce the
general power $a_{n}$ of the $n$-th propagator to one. With propagators given
by

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \ z_{2}=(l+p_{1})^{2}-m_{2}^{2},\ \ \
z_{3}=(l+p_{1}+p_{2})^{2}-m_{3}^{2},\cdots\ $ $\displaystyle
z_{n}=(l+p_{1}+\dots+p_{n-1})^{2}-m_{n}^{2}.~{}~{}~{}~{}$ (24)

the Baikov representation is given by

$\displaystyle\text{I}_{a_{1},a_{2},\cdots,a_{n}}$
$\displaystyle=C\int\mathcal{G}(\bm{z})^{(d-n-1)/2}\frac{dz_{i}}{\prod_{i=1}^{n}z_{i}^{a_{i}}}$
$\displaystyle\mathcal{G}(\bm{z})$
$\displaystyle=G(l,p_{1},\cdots,p_{n-1}).~{}~{}~{}~{}$ (25)

Noting that for $\mathcal{G}$, when writing using momentum variables, it is
the same as the one given in (18). However, when writing using the $z$
variables, the linear form in (17) and the quadratic form in (24) do make some
differences. Again first we find the syzygy module for relation (11) to avoid
the shift of the space-time dimension. However, the second module will be a
little different from the one given in (16). More explicitly, generators
become

$\displaystyle\bm{d}_{1}=\\{z_{1},0,\cdots,0,0,0\\}$ $\displaystyle\cdots$
$\displaystyle\bm{d}_{n-1}=\\{0,0,\cdots,z_{n-1},0,0\\}$
$\displaystyle\bm{d}_{n}=\\{0,0,\cdots,0,1,0\\}$
$\displaystyle\bm{d}_{n+1}=\\{0,0,\cdots,0,0,1\\},~{}~{}~{}$ (26)

where the $\bm{d}_{n}$ is different. The reason is that now we do not ask to
avoid the increase of the power of the $n$-th propagator. Finding the module
intersection $\\{\bm{h}_{i}\\}\equiv\\{\bm{e}_{i}\\}\cap\\{\bm{d}_{i}\\}$ we
will get IBP relations of the form

$\displaystyle\text{I}_{\bm{a},a_{n,max}}=\sum_{j=1}^{m}c_{j}\text{I}_{\bm{a},a_{n,max}-j}+l.p.p.t.,$
(27)

where no $c_{j}$ becomes infinity for any $a_{n,max}>1$. This equation is
similar to (21), but with the following difference: in (21) it is the smallest
power $-r_{max}$ at the left-hand side while in (27) it is the maximum power
$a_{n,max}$ at the left-hand side. Another difference is that in (27) we do
not need to require $c_{j}=0$ when $j>a_{n,max}$ since now it becomes the
tensor of the sub-sector.

Before ending this section, let us emphasize that the reason we are able to
treat the TR and DR uniformly using module intersection is following two key
points. First, we have introduced the auxiliary vector $R$ to represent all
tensor structures888We want to emphasize that introducing $R$ is different
from introducing irreducible scalar products in usual IBP method. For later,
if there are $m$ ISP’s we need to introduce $m$ factors, but for $R$, we need
to just introduce one for each loop momentum.. Secondly, we are not write down
all relations coming from module intersection, but select minimum ones, which
have nice property, i.e., giving iteration at the sector-level. The meaning of
this point will be clear by examples in the Sec.4.

##  4 Pedagogical examples

Having discussed the method in the previous section, we will present two
examples to demonstrate our method.

###  4.1 tensor reduction of bubbles

In this subsection, we consider the tensor reduction of one-loop bubble
integrals. The propagators are

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \ z_{2}=(l+p_{1})^{2}-m_{2}^{2},\ \ \
z_{3}=l\cdot R.$ (28)

and the Gram determinant in Baikov representation is

$\displaystyle\mathcal{G}=\det\left(\begin{array}[]{ccc}m_{1}^{2}+z_{1}&\frac{1}{2}\left(-m_{1}^{2}+m_{2}^{2}-p_{1}^{2}-z_{1}+z_{2}\right)&z_{3}\\\
\frac{1}{2}\left(-m_{1}^{2}+m_{2}^{2}-p_{1}^{2}-z_{1}+z_{2}\right)&p_{1}^{2}&R\cdot
p_{1}\\\ z_{3}&R\cdot p_{1}&R^{2}\\\ \end{array}\right).~{}~{}~{}$ (32)

Using the expression in (32) we can find

$\displaystyle\partial_{z_{1}}\mathcal{G}=\frac{1}{2}R^{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}-z_{1}+z_{2}\right)-z_{3}R\cdot
p_{1}-\left(R\cdot p_{1}\right)^{2}$
$\displaystyle\partial_{z_{2}}\mathcal{G}=\frac{1}{2}R^{2}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}+z_{1}-z_{2}\right)+z_{3}R\cdot
p_{1}$ $\displaystyle\partial_{z_{3}}\mathcal{G}=-R\cdot
p_{1}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}+z_{1}-z_{2}\right)-2p_{1}^{2}z_{3}~{}~{}~{}$
(33)

and the solutions of equation

$\displaystyle\sum_{i}^{3}(P_{i}\partial_{z_{i}}\mathcal{G})+P_{0}\mathcal{G}=0~{}~{}~{}$
(34)

can be solved by the syzygy module with three generators

$\displaystyle\\{\bm{e}_{i}\\}=\left(\begin{array}[]{cccc}2z_{3}&2\left(R\cdot
p_{1}+z_{3}\right)&R^{2}&0\\\
m_{1}^{2}+m_{2}^{2}-p_{1}^{2}+z_{1}+z_{2}&2\left(m_{2}^{2}+z_{2}\right)&R\cdot
p_{1}+z_{3}&-2\\\
-2\left(m_{2}^{2}-p_{1}^{2}+z_{2}\right)&m_{1}^{2}-3m_{2}^{2}-p_{1}^{2}+z_{1}-3z_{2}&-2R\cdot
p_{1}-z_{3}&2\\\ \end{array}\right).~{}~{}~{}$ (38)

Meanwhile, the module $\\{\bm{d}_{i}\\}$ is generated by

$\displaystyle\text{DM}[z_{1},z_{2},1,1],~{}~{}~{}$ (39)

where the DM denotes the diagonal matrix. The module intersection of them is
given by $\\{\bm{h}_{i}\\}$ with $10$ basis as polynomials of variables
$\left\\{z_{1},z_{2},z_{3},m_{1}^{2},m_{2}^{2},p_{1}^{2},R\cdot
p_{1},R^{2}\right\\}$999The reason not using
$\left\\{z_{1},z_{2},z_{3}\right\\}$ is explained in Bendle:2019csk . when
using Singular DGPS . Among them, the one with the lowest total power of
$z_{i}$ is given by

$\displaystyle\bm{h}_{1,1}=2z_{1}\left(R\cdot
p_{1}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}+z_{1}-z_{2}\right)+2p_{1}^{2}z_{3}\right)$
$\displaystyle\bm{h}_{1,2}=2z_{2}\left(R\cdot
p_{1}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}+z_{1}-z_{2}\right)+2p_{1}^{2}z_{3}\right)$
$\displaystyle\bm{h}_{1,3}=R^{2}\left(-p_{1}^{2}\left(2m_{1}^{2}+2m_{2}^{2}+z_{1}+z_{2}\right)+\left(m_{1}^{2}-m_{2}^{2}\right)\left(m_{1}^{2}-m_{2}^{2}+z_{1}-z_{2}\right)+\left(p_{1}^{2}\right)^{2}\right)$
$\displaystyle+2\left(\left(2m_{1}^{2}+z_{1}\right)\left(R\cdot
p_{1}\right)^{2}+z_{3}R\cdot
p_{1}\left(2m_{1}^{2}-2m_{2}^{2}+2p_{1}^{2}+z_{1}-z_{2}\right)+2p_{1}^{2}z_{3}^{2}\right)$
$\displaystyle\bm{h}_{1,4}=-4\left(R\cdot
p_{1}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}+z_{1}-z_{2}\right)+2p_{1}^{2}z_{3}\right).$
(40)

Using (40), one can check that the IBP relation generated by
$D_{\left\langle\bm{h}_{1}\right\rangle}$ acting on
$\text{I}_{a_{1},a_{2},-r-1}$ can be rewritten as

$\displaystyle\text{I}_{a_{1},a_{2},-r}=\frac{1}{4p_{1}^{2}\left(a_{1}+a_{2}-r-d+1\right)}\times$
$\displaystyle\left[-2\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}\right)R\cdot
p_{1}\left(a_{1}+a_{2}-d-2r+2\right)\text{I}_{a_{1},a_{2},-(r-1)}\right.$
$\displaystyle+(r-1)\left(4m_{1}^{2}\left(R\cdot
p_{1}\right)^{2}-2m_{1}^{2}R^{2}\left(m_{2}^{2}+p_{1}^{2}\right)+R^{2}\left(m_{2}^{2}-p_{1}^{2}\right)^{2}+m_{1}^{4}R^{2}\right)\text{I}_{a_{1},a_{2},-(r-2)}$
$\displaystyle\left.+l.p.p.t.\right],$ (41)

where

$\displaystyle l.p.p.t.=$ $\displaystyle-2R\cdot
p_{1}\left(a_{1}+a_{2}-r-d+1\right)\text{I}_{a_{1}-1,a_{2},-(r-1)}$
$\displaystyle+2R\cdot
p_{1}\left(a_{1}+a_{2}-r-d+1\right)\text{I}_{a_{1},a_{2}-1,-(r-1)}$
$\displaystyle-(r-1)R^{2}\left(m_{1}^{2}-m_{2}^{2}+p_{1}^{2}\right)\text{I}_{a_{1},a_{2}-1,-(r-2)}$
$\displaystyle+(r-1)\left(R^{2}\left(m_{1}^{2}-m_{2}^{2}-p_{1}^{2}\right)+2\left(R\cdot
p_{1}\right)^{2}\right)\text{I}_{a_{1}-1,a_{2},-(r-2)}.~{}~{}~{}$ (42)

Result (41) is the recursive relation we are looking for. Notice that when
$r=1$, the coefficient of
$\text{I}_{a_{1},a_{2},-(r-2)}=\text{I}_{a_{1},a_{2},1}$ is zero by the $r-1$
factor, which satisfies the condition (21). Since $r$ is the rank of tensor in
the numerator, the relation (41) tells us that the integrals of tensor rank
$r$ can be written as the sum of integrals of tensor rank $(r-1)$ and $(r-2)$
with proper rational coefficients. This kind of relations has been firstly
observed in Feng:2022iuc  for the case $a_{1}=a_{2}=1$ and then has been
proved in Feng:2022rfz . Here we give a simple derivation using module
intersection with generalization to arbitrary power of propagators, as well as
given the analytic l.p.p.t part missed in Feng:2022iuc ; Feng:2022rfz . It is
obvious that applying this kind of second-order iterative relation, one can
immediately reduce any tensor integrals of this sector to scalar integrals of
the same sector and tensor integrals of sub-sectors.

Another interesting property of relation (41) is that in (41) the
$a_{1},a_{2}$ of bubbles are invariant, while for l.p.p.t. in (42),
$(a_{1}+a_{2})$ has changed only by minus one. This observation will be
explained in Sec.5.

Before ending this subsection, let us emphasize that to deal with tensor
reduction of bubble topology, we have required $a_{3}=-r<0$. If we consider
the case $a_{3}>0$, we will get the triangle topology, and the relation
becomes IBP relation for triangles, but with the third propagator is not the
standard quadratic one. In next subsection, we will consider triangle topology
with the standard Feynman propagators, thus it will be useful to compare
results in these two subsections.

###  4.2 DR reduction of triangle

In this subsection, we discuss the reduction of triangles with arbitrary
powers for propagators. The propagators are

$\displaystyle z_{1}=l^{2}-m_{1}^{2},\ \ \ z_{2}=(l+p_{1})^{2}-m_{2}^{2},\ \ \
z_{3}=(l+p_{1}+p_{2})^{2}-m_{3}^{2}.$ (43)

and the corresponding Gram determinant $\mathcal{G}$ in Baikov representation
is

$\displaystyle\det\left(\begin{array}[]{ccc}m_{1}^{2}+z_{1}&\cdots&\cdots\\\
\frac{1}{2}\left(-m_{1}^{2}+m_{2}^{2}-p_{1}^{2}-z_{1}+z_{2}\right)&p_{1}^{2}&\cdots\\\
\frac{1}{2}\left(-m_{2}^{2}+m_{3}^{2}-p_{2}^{2}-2p_{1}\cdot
p_{2}-z_{2}+z_{3}\right)&p_{1}\cdot p_{2}&p_{2}^{2}\\\ \end{array}\right)$
(47)

where the $\cdots$ denote the terms, which can be obtained by symmetry. With

$\displaystyle\partial_{z_{1}}\mathcal{G}=\frac{1}{2}\left(p_{2}^{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}-z_{1}+z_{2}\right)+p_{1}\cdot
p_{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}+z_{2}-z_{3}\right)\right),$
$\displaystyle\partial_{z_{2}}\mathcal{G}=\frac{1}{2}\left(p_{1}\cdot
p_{2}\left(m_{1}^{2}-2m_{2}^{2}+m_{3}^{2}-p_{1}^{2}-p_{2}^{2}+z_{1}-2z_{2}+z_{3}\right)+m_{3}^{2}p_{1}^{2}+m_{1}^{2}p_{2}^{2}\right.$
$\displaystyle\left.-m_{2}^{2}\left(p_{1}^{2}+p_{2}^{2}\right)+p_{2}^{2}z_{1}-p_{1}^{2}z_{2}-p_{2}^{2}z_{2}+p_{1}^{2}z_{3}-2\left(p_{1}\cdot
p_{2}\right)^{2}\right),$
$\displaystyle\partial_{z_{3}}\mathcal{G}=\frac{1}{2}\left(p_{1}^{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}+z_{2}-z_{3}\right)+p_{1}\cdot
p_{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}-z_{1}+z_{2}\right)\right).$ (48)

one can solve

$\displaystyle\sum_{i}^{3}(P_{i}\partial_{z_{i}}\mathcal{G})+P_{0}\mathcal{G}=0,$
(49)

with the basis $\\{\bm{e}_{i}\\}$ of the syzygy module is

$\displaystyle\left(\begin{array}[]{cccc}m_{1}^{2}+m_{3}^{2}-s+z_{1}+z_{3}&m_{2}^{2}+m_{3}^{2}-p_{2}^{2}+z_{2}+z_{3}&2\left(m_{3}^{2}+z_{3}\right)&-2\\\
m_{1}^{2}+m_{2}^{2}-p_{1}^{2}+z_{1}+z_{2}&2\left(m_{2}^{2}+z_{2}\right)&m_{2}^{2}+m_{3}^{2}-p_{2}^{2}+z_{2}+z_{3}&-2\\\
2\left(m_{1}^{2}+z_{1}\right)&m_{1}^{2}+m_{2}^{2}-p_{1}^{2}+z_{1}+z_{2}&m_{1}^{2}+m_{3}^{2}-s+z_{1}+z_{3}&-2\\\
\end{array}\right)$ (53)

where $s=(p_{1}+p_{2})^{2}$. Another module is generated by (see (26))

$\displaystyle\\{\bm{d}_{i}\\}=\text{DM}[z_{1},z_{2},1,1],$ (54)

From them, we can compute $\\{\bm{h}_{i}\\}$ as the module intersection of
them. Among them, the one with the lowest total power of $z_{i}$ is given by

$\displaystyle\bm{h}_{1,1}=2z_{1}\left(p_{1}^{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}+z_{2}-z_{3}\right)+p_{1}\cdot
p_{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}-z_{1}+z_{2}\right)\right),$
$\displaystyle\bm{h}_{1,2}=2z_{2}\left(p_{1}^{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}+z_{2}-z_{3}\right)+p_{1}\cdot
p_{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}-z_{1}+z_{2}\right)\right),$
$\displaystyle\bm{h}_{1,3}=-2\left(m_{1}^{2}p_{2}^{2}z_{1}-m_{1}^{2}p_{2}^{2}z_{2}-m_{3}^{2}p_{1}^{2}z_{2}+2\left(p_{1}\cdot
p_{2}\right)^{2}\left(2m_{2}^{2}+z_{2}\right)+2m_{3}^{2}p_{1}^{2}z_{3}\right.$
$\displaystyle+m_{2}^{2}\left(-2m_{3}^{2}p_{1}^{2}-2m_{1}^{2}p_{2}^{2}+p_{1}^{2}z_{2}-2p_{1}^{2}z_{3}-p_{2}^{2}z_{1}+p_{2}^{2}z_{2}\right)$
$\displaystyle+p_{1}\cdot
p_{2}\left(-m_{2}^{2}\left(2m_{3}^{2}-2p_{1}^{2}-2p_{2}^{2}+z_{1}-2z_{2}+2z_{3}\right)-m_{1}^{2}\left(2m_{2}^{2}-2m_{3}^{2}+2p_{2}^{2}+z_{2}-2z_{3}\right)\right.$
$\displaystyle\left.-2m_{3}^{2}p_{1}^{2}+m_{3}^{2}z_{1}-m_{3}^{2}z_{2}+2m_{2}^{4}-p_{2}^{2}z_{1}+p_{1}^{2}z_{2}+p_{2}^{2}z_{2}-2p_{1}^{2}z_{3}+2p_{1}^{2}p_{2}^{2}+z_{1}z_{3}-z_{2}z_{3}\right)$
$\displaystyle+m_{1}^{4}p_{2}^{2}-2m_{1}^{2}p_{1}^{2}p_{2}^{2}+m_{3}^{4}p_{1}^{2}-2m_{3}^{2}p_{1}^{2}p_{2}^{2}+m_{2}^{4}\left(p_{1}^{2}+p_{2}^{2}\right)+p_{1}^{2}z_{3}^{2}-p_{1}^{2}p_{2}^{2}z_{1}-2p_{1}^{2}p_{2}^{2}z_{3}$
$\displaystyle\left.-p_{1}^{2}z_{2}z_{3}+p_{1}^{2}p_{2}^{4}+p_{1}^{4}p_{2}^{2}\right),$
$\displaystyle\bm{h}_{1,4}=4p_{1}\cdot
p_{2}\left(m_{1}^{2}-m_{2}^{2}-p_{1}^{2}+z_{1}-z_{2}\right)-4p_{1}^{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}+z_{2}-z_{3}\right).$
(55)

Using (55) the action of $D_{\left\langle\bm{h}_{1}\right\rangle}$ on
$\text{I}_{a_{1},a_{2},a_{3}-1}$ gives the wanted IBP relation

$\displaystyle\text{I}_{a_{1},a_{2},a_{3}}=\frac{1}{Q_{1}}\times\left[-p_{1}^{2}\left(a_{1}+a_{2}+a_{3}-d-1\right)\text{I}_{a_{1},a_{2},a_{3}-2}\right.$
$\displaystyle\left(2a_{3}+a_{1}+a_{2}-d-2\right)\left(p_{1}^{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}\right)+p_{1}\cdot
p_{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}\right)\right)\text{I}_{a_{1},a_{2},a_{3}-1}$
$\displaystyle\left.+l.p.p.t.\right]$ (56)

where

$\displaystyle l.p.p.t.$
$\displaystyle=\left(a_{3}-1\right)\left(p_{2}^{2}\left(-m_{1}^{2}+m_{2}^{2}+p_{1}^{2}\right)+p_{1}\cdot
p_{2}\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}\right)\right)\text{I}_{a_{1}-1,a_{2},a_{3}}$
$\displaystyle-\left(a_{3}-1\right)\left(p_{1}\cdot
p_{2}\left(-m_{1}^{2}+2m_{2}^{2}-m_{3}^{2}+p_{1}^{2}+p_{2}^{2}\right)-m_{3}^{2}p_{1}^{2}-m_{1}^{2}p_{2}^{2}+m_{2}^{2}\left(p_{1}^{2}+p_{2}^{2}\right)\right.$
$\displaystyle\left.+2\left(p_{1}\cdot
p_{2}\right)^{2}\right)\text{I}_{a_{1},a_{2}-1,a_{3}}$ $\displaystyle-
p_{1}\cdot
p_{2}\left(a_{1}+a_{2}+a_{3}-d-1\right)\text{I}_{a_{1}-1,a_{2},a_{3}-1}$
$\displaystyle+\left(p_{1}^{2}+p_{1}\cdot
p_{2}\right)\left(a_{1}+a_{2}+a_{3}-d-1\right)\text{I}_{a_{1},a_{2}-1,a_{3}-1}$
$\displaystyle Q_{1}$
$\displaystyle=\left(a_{3}-1\right)\left(m_{1}^{4}p_{2}^{2}-2m_{1}^{2}p_{1}^{2}p_{2}^{2}+4m_{2}^{2}\left(p_{1}\cdot
p_{2}\right)^{2}+m_{3}^{4}p_{1}^{2}-2m_{3}^{2}p_{1}^{2}p_{2}^{2}+m_{2}^{4}\left(p_{1}^{2}+p_{2}^{2}\right)\right.$
$\displaystyle\left.-2p_{1}\cdot
p_{2}\left(m_{1}^{2}-m_{2}^{2}-p_{1}^{2}\right)\left(m_{2}^{2}-m_{3}^{2}+p_{2}^{2}\right)-2m_{2}^{2}\left(m_{3}^{2}p_{1}^{2}+m_{1}^{2}p_{2}^{2}\right)+p_{1}^{2}p_{2}^{4}+p_{1}^{4}p_{2}^{2}\right).~{}~{}~{}$
(57)

Notice that when $a_{3}=2$, the coefficient of
$\text{I}_{a_{1},a_{2},a_{3}-2}=\text{I}_{a_{1},a_{2},0}$ is a term of sub-
sector (here is the bubble topology). Also for $a_{3}=1$ we don’t need to
apply this relation for DR since it is already the final goal we want to
achieve. The relation (56) is also a second-order iterative relation relating
$\text{I}_{a_{1},a_{2},a_{3}}$ to $\text{I}_{a_{1},a_{2},a_{3}-1}$,
$\text{I}_{a_{1},a_{2},a_{3}-2}$ and l.p.p.t.. Similar to (41), the
$a_{1},a_{2}$ of triangles in (56) are invariant, while for l.p.p.t. in (57),
$(a_{1}+a_{2})$ has changed only by minus one.

Thus applying the relation (56) iteratively one can immediately reduce any
higher power of $z_{3}$ to one. Similar iterative relations for reducing the
power of other propagators can be obtained. Combining them, we can reduce
integrals with arbitrary high power of propagators in this sector.

##  5 Uniform formula for general one-loop reduction

The method laid out in section 3 and related computations done in section 4
look fresh, but maybe not so surprising. However, as we will show in this
section, for one-loop integrals, we can write down explicit recursive
relations for both TR and DR uniformly for any general $n$-point one-loop
integrals. In other words, we have solved IBP relations analytically.

The key of our method is to select particular elements in module intersection
$\\{\bm{h}_{i}\\}$. Let’s make an observation for two examples in section 4,
i.e., both results (40) and (55), we find that

$\displaystyle\\{\bm{h}_{1,1},\bm{h}_{1,2},\bm{h}_{1,4}\\}=C\times\\{z_{1}\partial_{z_{3}}\mathcal{G},z_{2}\partial_{z_{3}}\mathcal{G},-2\partial_{z_{3}}\mathcal{G}\\}$
(58)

and then by (49), we have

$\displaystyle\bm{h}_{1,3}=C\times\left(2\mathcal{G}-z_{1}\partial_{z_{1}}\mathcal{G}-z_{2}\partial_{z_{2}}\mathcal{G}\right).$
(59)

This pattern indicates that for DR of the $N$-th propagator of the $N$-point
one-loop integrals or the TR of $(N-1)$-point one-loop integrals (where the
$(R\cdot\ell)$ has been considered as the $N$-th propagator), the wanted
element in the intersection module is given by

$\displaystyle P_{ui}=z_{i}\partial_{z_{N}}\mathcal{G}~{}~{}~{}\text{for
}~{}1\leq i\leq N-1,$ $\displaystyle
P_{uN}=2\mathcal{G}-\sum_{i=1}^{N-1}z_{i}\partial_{z_{i}}\mathcal{G},~{}~{}~{}~{}~{}~{}P_{u0}=-2\partial_{z_{N}}\mathcal{G}.~{}~{}~{}$
(60)

It is easy to check that (60) satisfies (11) and belongs to the module

$\displaystyle\\{\bm{d}_{i}\\}=\text{DM}[z_{1},z_{2},\cdots,z_{N-1},1,1],$
(61)

then the differential operator $D_{\langle P_{u}\rangle}$ gives the generic
iterative relation for both TR and DR of general one-loop integrals.

To write down explicit relation, let us compute $P_{u}$ in (60). For one-loop
integrals, Gram determinant $\mathcal{G}$ is always quadratic polynomial of
$z_{i}$s

$\displaystyle\mathcal{G}(\bm{z})=\sum_{i,j\leq
i}C_{2}^{(ij)}z_{i}z_{j}+\sum_{i}C_{1}^{(i)}z_{i}+C_{0}~{}~{}~{}$ (62)

where

$\displaystyle
C_{2}^{(ii)}=\partial_{z_{i}}^{2}\mathcal{G}/2,~{}~{}~{}C_{1}^{(i)}=\left(\partial_{z_{i}}\mathcal{G}\right)|_{\bm{z}=0},~{}~{}~{}C_{0}=\mathcal{G}|_{\bm{z}=0},$
$\displaystyle
C_{2}^{(ij)}=C_{2}^{(ji)}=\partial_{z_{i}}\partial_{z_{j}}\mathcal{G}~{}~{}~{}~{}~{}\text{for}~{}~{}j\neq
i.~{}~{}~{}$ (63)

This leads to

$\displaystyle\partial_{z_{i}}\mathcal{G}=2C_{2}^{(ii)}z_{i}+\sum_{j\neq
i}C_{2}^{(ij)}z_{j}+C_{1}^{(i)},$
$\displaystyle\sum_{i=1}^{N}z_{i}\partial_{z_{i}}\mathcal{G}=2\sum_{i,j\leq
i}C_{2}^{(ij)}z_{i}z_{j}+\sum_{i}C_{1}^{(i)}z_{i},$ $\displaystyle
P_{uN}=z_{N}\partial_{z_{N}}\mathcal{G}+\sum_{i}C_{1}^{(i)}z_{i}+2C_{0}.~{}~{}~{}$
(64)

The rewriting of $P_{uN}$ in (64) tells us that $P_{uN}$ depends on
$z_{i},i=1,...,N-1$ only linearly.

Combining $P_{ui}$ in (60) and $\partial_{z_{i}}\mathcal{G}$ in (64), one can
see that the power of $z_{i}$s will lead all
$\text{I}_{a^{\prime}_{1},\cdots,a^{\prime}_{N-1},a^{\prime}_{N}}$ to satisfy
$0\leq\sum_{i=1}^{N-1}a_{i}-\sum_{i=1}^{N-1}a^{\prime}_{i}\leq 1$. To show
that, let us do the following explicit computations.

For $i\leq N-1$, carrying out

$\displaystyle-
D_{P_{ui}}\bm{\cdot}\frac{1}{\prod_{i=1}^{N}z_{i}^{a_{i}}}=-\partial_{z_{i}}\left(\frac{z_{i}\partial_{z_{N}}\mathcal{G}}{\prod_{i=1}^{N}z_{i}^{a_{i}}}\right),~{}~{}~{}$
(65)

we find

$\displaystyle(a_{i}-1)\left(2C_{2}^{(NN)}\text{I}_{\cdots,a_{N}-1}+C_{1}^{(N)}\text{I}_{\cdots,a_{N}}\right)$
$\displaystyle+$
$\displaystyle(a_{i}-2)C_{2}^{(Ni)}\text{I}_{\cdots,a_{i}-1,\cdots,a_{N}}+(a_{i}-1)\sum_{j=1,j\neq
i}^{N-1}C_{2}^{(Nj)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}}.~{}~{}~{}$ (66)

For $i=N$, action of $D_{P_{N}}$ gives

$\displaystyle
2\left[(a_{N}-2)C_{2}^{(NN)}\text{I}_{\cdots,a_{N}-1}+(a_{N}-1)C_{1}^{(N)}\text{I}_{\cdots,a_{N}}+a_{N}C_{0}\text{I}_{\cdots,a_{N}+1}\right]$
$\displaystyle+\left[(a_{N}-1)\sum_{j\neq
N}C_{2}^{(Nj)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}}+a_{N}\sum_{j\neq
N}C_{1}^{(j)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}-1}\right].~{}~{}~{}$ (67)

Finally action of $D_{P_{0}}$ gives

$\displaystyle-\left(d-N-1\right)\left(C_{1}^{(N)}\text{I}_{\cdots,a_{N}}+2C_{2}^{(NN)}\text{I}_{\cdots,a_{N}-1}+\sum_{j=1}^{N-1}C_{2}^{(Nj)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}}\right).~{}~{}~{}$
(68)

Combining all together, we have

$\displaystyle
2a_{N}C_{0}\text{I}_{\cdots,a_{N}+1}+\left(\sum_{i=1}^{N-1}a_{i}+2a_{N}-d\right)C_{1}^{(N)}\text{I}_{\cdots,a_{N}}$
$\displaystyle+$
$\displaystyle\left(\sum_{i=1}^{N}a_{i}-d\right)2C_{2}^{(NN)}\text{I}_{\cdots,a_{N}-1}+l.p.p.t.=0~{}~{}~{}$
(69)

where

$\displaystyle l.p.p.t.=$
$\displaystyle\left(\sum_{i=1}^{N}a_{i}-d\right)\sum_{j=1}^{N-1}C_{2}^{(Nj)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}}+a_{N}\sum_{j=1}^{N-1}C_{1}^{(j)}\text{I}_{\cdots,a_{j}-1,\cdots,a_{N}-1}~{}.~{}~{}$
(70)

Expressions (69) and (70) are our main results for this paper.

When $a_{N}<0$, it corresponds to the numerator and we should use (69) to
express $\text{I}_{\cdots,a_{N}-1}$ by others for the TR, while when $a_{N}>0$
it corresponds to the denominator with a higher power and we should use (69)
to express $\text{I}_{\cdots,a_{N}+1}$ by others for the DR. When $a_{N}=0$,
the (69) will give the relation between $\text{I}_{\cdots,-1}$ and
$\text{I}_{\cdots,0}$, which is just the reduction of tensor with rank one.

##  6 Example of degenerated case

In section 5, we have assumed that the kinematics and masses are general. But
when kinematics and masses take some special values, the Gram determinant may
be zero and we meet the degenerated situations. For these cases, directly
applying (69) and (70) still work, but why it works should be carefully
explained. We will show that there are three different possibilities. The
first one is that while it works, it may not give the simplest iterative
relation. The second one is that while it does not work for some situations,
there is at least one situation it works. The third one is that it does not
work at all, but for this case, there are no master integrals, so it does not
matter. No matter which situation one meets, directly applying the method of
module intersection presented in the Sec.3 gives an alternative way to do
reduction than using (69) and (70).

Now we give an example to elaborate on the above claim. Let us consider the
reduction of triangles

$\displaystyle z_{1}=l^{2},\ \ \ z_{2}=(l+p_{1})^{2},\ \ \
z_{3}=(l+p_{1}+p_{2})^{2},$ $\displaystyle z_{4}=l\cdot R,\ \ \
p_{1}^{2}=p_{2}^{2}=0.~{}~{}~{}~{}$ (71)

with specific kinematics. As shown later, by IBP relations one can see that
all integrals in this sector can be reduced to sub-sectors.

For TR of (71), $\mathcal{G}=G(l,p_{1},p_{2},R)$. Directly applying the
formula (69) and (70), we will get101010$R^{2}$ does not appear in (72)
because $G(l,p_{1},p_{2})|_{\bm{z}=0}=0$ when using (69) and (70).

$\displaystyle\text{I}_{a_{1},a_{2},a_{3},-r-1}=\frac{R\cdot p_{1}\left(r\
R\cdot p_{1}\
\text{I}_{a_{1},a_{2},a_{3},1-r}-\left(\sum_{i}^{3}a_{i}-d-2r\right)\text{I}_{a_{1},a_{2},a_{3},-r}\right)}{\sum_{i}^{3}a_{i}-d-r}+l.p.p.t.\
,$ (72)

which is a second-order iterative relation. Using (72), one can still reduce
all $\text{I}_{1,1,1,-r}$ to $\text{I}_{1,1,1,0}$. However,
$\text{I}_{1,1,1,0}$ is not a master integral in this example. To reduce
$\text{I}_{1,1,1,0}$ we should use (69) and (70) for DR, which will be
explained shortly.

If we calculate the module intersection follow the method in Sec.3, one can
find another element $\bm{h}_{i}$ different from the one given in (60), which
will give us the following iterative reduction relation

$\displaystyle\text{I}_{a_{1},a_{2},a_{3},-r}=\frac{rR\cdot
p_{1}\text{I}_{a_{1},a_{2},a_{3},1-r}}{2a_{2}+2a_{3}-d-r}+l.p.p.t.\
.~{}~{}~{}~{}$ (73)

Obviously, as a first-order iterative relation, (73) is simpler than (72),
which supports the claim of the first possibility.

For DR111111Now there is no $z_{4}$ in (71)., the Gram determinant in Baikov
representation is

$\displaystyle\mathcal{G}=G(l,p_{1},p_{2})=\frac{1}{2}p_{1}\cdot
p_{2}\left(\left(z_{1}-z_{2}\right)\left(z_{2}-z_{3}\right)-2p_{1}\cdot
p_{2}z_{2}\right).~{}~{}~{}$ (74)

To determine the number of master integrals, we can consider the maximum cut
of this sector in Baikov representation, i.e., setting $z_{i}=0$ in
$\mathcal{G}$. If $\mathcal{G}|_{\bm{z}=0}$ is a nonzero constant, the number
of master integrals is just one by counting critical points Lee:2013hzt ;
Frellesvig:2019kgj ; Chen:2022lzr . But if $\mathcal{G}|_{\bm{z}=0}=0$, there
is no master integral in this sector. It is easy to see that
$\mathcal{G}|_{\bm{z}=0}=0$ in (74), thus $\text{I}_{1,1,1}$ can be reduced to
sub-sectors.

One can check that if we regard $z_{1}$ or $z_{3}$ as the $z_{N}$ and apply
(69), the $C_{0},C_{1}^{(N)},C_{2}^{(NN)}$ are all zero and only l.p.p.t. is
left. In other words, for these cases, (69) does not produce the wanted
relation for reduction purpose. However, if we regard $z_{2}$ as the $z_{N}$,
the iterative relation reduce to first-order due to $C_{0}=0$,
$C_{1}^{(2)}\neq 0$ and $C_{2}^{(22)}\neq 0$. Using it, we can reduce
$I_{a_{1},a_{2},a_{3},0}$ (including the $I_{1,1,1,0}$ discussed in previous
paragraph) to sub-sectors $I_{a_{1},0,a_{3},0}$ and others. The $a_{1}$ and
$a_{3}$ are left to be reduced by DR of (69) in the sub-sector. So, as pointed
out for the second possibility, although using (69) for the DR does not work
for $z_{1}$ and $z_{3}$, there is $z_{2}$ it works.

Nevertheless, even if we regard $z_{3}$ as the $z_{N}$, we can do similar
module computation proposed in Sec.3. The syzygy module is generated by

$\displaystyle\\{\bm{e_{i}}\\}=\left(\begin{array}[]{cccc}z_{2}&z_{2}&z_{3}&-1\\\
2p_{1}\cdot p_{2}-z_{3}&-z_{3}&z_{2}-2z_{3}&1\\\ -2p_{1}\cdot
p_{2}+z_{1}-z_{2}+z_{3}&z_{3}&z_{3}&-1\\\ 4p_{1}\cdot
p_{2}+z_{2}-2z_{3}&z_{1}-2z_{3}&-2p_{1}\cdot p_{2}+z_{1}-2z_{3}&1\\\
\end{array}\right),$ (79)

while the element in the intersection module is taken to be

$\displaystyle\left\langle P\right\rangle=\langle z_{1},z_{2},z_{2},1\rangle.$
(80)

Using (80), the iterative relation is

$\displaystyle\text{I}_{a_{1},a_{2},a_{3},0}=-\frac{2a_{3}\text{I}_{a_{1},a_{2}-1,a_{3}+1,0}}{2a_{1}+2a_{2}-d}~{}~{}~{}~{}$
(81)

where the $(a_{1}+a_{2})$ has been reduced by one at the right-hand side,
which is just a l.p.p.t.. Although this relation raises the power of $z_{3}$
in denominator, it will lower $a_{2}$ to $0$ finally (i.e., reduce to sub-
sector), so it shows that method of module intersection still works.

From this example, it is easy to see that only for more degenerated case,
i.e., all $C_{2}^{(ii)}$, $C_{1}^{(i)}$ and $C_{0}$ in $\mathcal{G}$ equal
zero, (69) can not reduce integrals in this sector to sub-sectors. But for
this situation, all sub-sectors have no master integrals by counting the
critical points. To be more explicitly, the number of master integrals in the
corresponding sector is equal to the number of solutions to the equations

$\displaystyle
0=\partial_{z_{i}}\log\left(\mathcal{G}^{(d-n-1)/2}\right)|_{\bm{z^{\prime}}=0},\
\ \ \text{for all}~{}z_{i}\not\in\\{\bm{z}^{\prime}\\},$ (82)

Now the $\mathcal{G}$ takes the form

$\displaystyle\mathcal{G}=\sum_{i,j\neq i}c_{ij}z_{i}z_{j}$ (83)

and the equations (82) take the form

$\displaystyle\frac{C}{z_{i}-C_{i}}=0,$ (84)

which obviously has no solution. It means that integrals in this topology are
all scaleless integrals, and this topology has no master integral. People can
test this conclusion by taking $p_{1}\cdot p_{2}$ to zero in (74) and
immediately find this topology to be scaleless. This is the third possibility
we have mentioned.

##  7 Summary and Outlook

In this paper, a natural extension of syzygy and module intersection has been
shown that it can uniformly reduce one-loop integrals with arbitrary tensor
(by using auxiliary vector) and propagators with high powers. With a nice
observation, powerful iterative relation can be written down even without
doing explicit computations using computational algebraic geometry and module
intersection. Furthermore, in this formula, the polynomials that look messy in
the traditional reduction methods are arranged themselves to Gram
determinants. Such a property will not only speed up the analytic reduction of
Feynman integrals, but also help to investigate mathematical structures of
Feynman integrals and amplitudes in the future.

It is obvious that the motivation of our study is the reduction for high
loops. If the method is good, it must work well at the one-loop level. Results
in this paper have demonstrate this point. To generalize this method to high
loops, although steps of computational algebraic geometry can be easily
applied, some nontrivial problems will arise. There are irreducible scalar
products of loop momenta and more than one master integrals for a given sector
in most multi-loop integrals. This suggests that for multi-loop integrals,
elements of module intersection needed may also be more than one. More
importantly, could we obtain the general reduction formulas like (69) for
multi-loops? Are there also some hidden informations or structures in the
messy polynomials in the IBP relation? These problems are definitely
interesting for future exploration.

## Acknowledgments

This work is supported by Chinese NSF funding under Grant No.11935013,
No.11947301, No.12047502 (Peng Huanwu Center).

## References

  * (1) K.G. Chetyrkin and F.V. Tkachov, _Integration by Parts: The Algorithm to Calculate beta Functions in 4 Loops_ , _Nucl. Phys. B_ 192 (1981) 159.
  * (2) A.V. Kotikov, _Differential equations method: New technique for massive Feynman diagrams calculation_ , _Phys. Lett. B_ 254 (1991) 158.
  * (3) T. Gehrmann and E. Remiddi, _Differential equations for two loop four point functions_ , _Nucl. Phys. B_ 580 (2000) 485 [hep-ph/9912329].
  * (4) M. Argeri and P. Mastrolia, _Feynman Diagrams and Differential Equations_ , _Int. J. Mod. Phys. A_ 22 (2007) 4375 [0707.4037].
  * (5) J.M. Henn, _Lectures on differential equations for Feynman integrals_ , _J. Phys. A_ 48 (2015) 153001 [1412.2296].
  * (6) J.M. Henn, _Multiloop integrals in dimensional regularization made simple_ , _Phys. Rev. Lett._ 110 (2013) 251601 [1304.1806].
  * (7) F. Moriello, _Generalised power series expansions for the elliptic planar families of Higgs + jet production at two loops_ , _JHEP_ 01 (2020) 150 [1907.13234].
  * (8) R. Bonciani, V. Del Duca, H. Frellesvig, J.M. Henn, M. Hidding, L. Maestri et al., _Evaluating a family of two-loop non-planar master integrals for Higgs + jet production with full heavy-quark mass dependence_ , _JHEP_ 01 (2020) 132 [1907.13156].
  * (9) H. Frellesvig, M. Hidding, L. Maestri, F. Moriello and G. Salvatori, _The complete set of two-loop master integrals for Higgs + jet production in QCD_ , _JHEP_ 06 (2020) 093 [1911.06308].
  * (10) M. Hidding, _DiffExp, a Mathematica package for computing Feynman integrals in terms of one-dimensional series expansions_ , _Comput. Phys. Commun._ 269 (2021) 108125 [2006.05510].
  * (11) X. Liu, Y.-Q. Ma and C.-Y. Wang, _A Systematic and Efficient Method to Compute Multi-loop Master Integrals_ , _Phys. Lett. B_ 779 (2018) 353 [1711.09572].
  * (12) Z.-F. Liu and Y.-Q. Ma, _Automatic computation of Feynman integrals containing linear propagators via auxiliary mass flow_ , _Phys. Rev. D_ 105 (2022) 074003 [2201.11636].
  * (13) Z.-F. Liu and Y.-Q. Ma, _Feynman integrals are completely determined by linear algebra_ , 2201.11637.
  * (14) X. Liu and Y.-Q. Ma, _AMFlow: a Mathematica package for Feynman integrals computation via Auxiliary Mass Flow_ , 2201.11669.
  * (15) X. Liu and Y.-Q. Ma, _Multiloop corrections for collider processes using auxiliary mass flow_ , _Phys. Rev. D_ 105 (2022) L051503 [2107.01864].
  * (16) X. Liu, Y.-Q. Ma, W. Tao and P. Zhang, _Calculation of Feynman loop integration and phase-space integration via auxiliary mass flow_ , _Chin. Phys. C_ 45 (2021) 013115 [2009.07987].
  * (17) T. Armadillo, R. Bonciani, S. Devoto, N. Rana and A. Vicini, _Evaluation of Feynman integrals with arbitrary complex masses via series expansions_ , 2205.03345.
  * (18) B. Feng, T. Li, H. Wang and Y. Zhang, _Reduction of general one-loop integrals using auxiliary vector_ , _JHEP_ 05 (2022) 065 [2203.14449].
  * (19) S. Laporta, _High precision calculation of multiloop Feynman integrals by difference equations_ , _Int. J. Mod. Phys. A_ 15 (2000) 5087 [hep-ph/0102033].
  * (20) A.V. Smirnov and F.S. Chuharev, _FIRE6: Feynman Integral REduction with Modular Arithmetic_ , _Comput. Phys. Commun._ 247 (2020) 106877 [1901.07808].
  * (21) A. von Manteuffel and C. Studerus, _Reduze 2 - Distributed Feynman Integral Reduction_ , 1201.4330.
  * (22) R.N. Lee, _Presenting LiteRed: a tool for the Loop InTEgrals REDuction_ , 1212.2685.
  * (23) J. Klappert, F. Lange, P. Maierhöfer and J. Usovitsch, _Integral reduction with Kira 2.0 and finite field methods_ , _Comput. Phys. Commun._ 266 (2021) 108024 [2008.06494].
  * (24) Z. Bern, L.J. Dixon, D.C. Dunbar and D.A. Kosower, _One loop n point gauge theory amplitudes, unitarity and collinear limits_ , _Nucl. Phys. B_ 425 (1994) 217 [hep-ph/9403226].
  * (25) Z. Bern, L.J. Dixon, D.C. Dunbar and D.A. Kosower, _Fusing gauge theory tree amplitudes into loop amplitudes_ , _Nucl. Phys. B_ 435 (1995) 59 [hep-ph/9409265].
  * (26) R. Britto, F. Cachazo and B. Feng, _Generalized unitarity and one-loop amplitudes in N=4 super-Yang-Mills_ , _Nucl. Phys. B_ 725 (2005) 275 [hep-th/0412103].
  * (27) R. Britto, E. Buchbinder, F. Cachazo and B. Feng, _One-loop amplitudes of gluons in SQCD_ , _Phys. Rev. D_ 72 (2005) 065012 [hep-ph/0503132].
  * (28) G. Ossola, C.G. Papadopoulos and R. Pittau, _Reducing full one-loop amplitudes to scalar integrals at the integrand level_ , _Nucl. Phys. B_ 763 (2007) 147 [hep-ph/0609007].
  * (29) J. Gluza, K. Kajda and D.A. Kosower, _Towards a Basis for Planar Two-Loop Integrals_ , _Phys. Rev. D_ 83 (2011) 045012 [1009.0472].
  * (30) T. Peraro, _FiniteFlow: multivariate functional reconstruction using finite fields and dataflow graphs_ , _JHEP_ 07 (2019) 031 [1905.08019].
  * (31) V. Chestnov, F. Gasparotto, M.K. Mandal, P. Mastrolia, S.J. Matsubara-Heo, H.J. Munch et al., _Macaulay Matrix for Feynman Integrals: Linear Relations and Intersection Numbers_ , 2204.12983.
  * (32) P. Mastrolia and S. Mizera, _Feynman Integrals and Intersection Theory_ , _JHEP_ 02 (2019) 139 [1810.03818].
  * (33) H. Frellesvig, F. Gasparotto, S. Laporta, M.K. Mandal, P. Mastrolia, L. Mattiazzi et al., _Decomposition of Feynman Integrals on the Maximal Cut by Intersection Numbers_ , _JHEP_ 05 (2019) 153 [1901.11510].
  * (34) H. Frellesvig, F. Gasparotto, M.K. Mandal, P. Mastrolia, L. Mattiazzi and S. Mizera, _Vector Space of Feynman Integrals and Multivariate Intersection Numbers_ , _Phys. Rev. Lett._ 123 (2019) 201602 [1907.02000].
  * (35) S. Weinzierl, _On the computation of intersection numbers for twisted cocycles_ , _J. Math. Phys._ 62 (2021) 072301 [2002.01930].
  * (36) S. Mizera, _Status of Intersection Theory and Feynman Integrals_ , _PoS_ MA2019 (2019) 016 [2002.10476].
  * (37) H. Frellesvig, F. Gasparotto, S. Laporta, M.K. Mandal, P. Mastrolia, L. Mattiazzi et al., _Decomposition of Feynman Integrals by Multivariate Intersection Numbers_ , _JHEP_ 03 (2021) 027 [2008.04823].
  * (38) X. Liu and Y.-Q. Ma, _Determining arbitrary Feynman integrals by vacuum integrals_ , _Phys. Rev. D_ 99 (2019) 071501 [1801.10523].
  * (39) X. Guan, X. Liu and Y.-Q. Ma, _Complete reduction of integrals in two-loop five-light-parton scattering amplitudes_ , _Chin. Phys. C_ 44 (2020) 093106 [1912.09294].
  * (40) K.J. Larsen and Y. Zhang, _Integration-by-parts reductions from unitarity cuts and algebraic geometry_ , _Phys. Rev. D_ 93 (2016) 041701 [1511.01071].
  * (41) K.J. Larsen and Y. Zhang, _Integration-by-parts reductions from the viewpoint of computational algebraic geometry_ , _PoS_ LL2016 (2016) 029 [1606.09447].
  * (42) Y. Zhang, _Lecture Notes on Multi-loop Integral Reduction and Applied Algebraic Geometry_ , 12, 2016 [1612.02249].
  * (43) A. Georgoudis, K.J. Larsen and Y. Zhang, _Azurite: An algebraic geometry based package for finding bases of loop integrals_ , _Comput. Phys. Commun._ 221 (2017) 203 [1612.04252].
  * (44) A. Georgoudis, K.J. Larsen and Y. Zhang, _Cristal and Azurite: new tools for integration-by-parts reductions_ , _PoS_ RADCOR2017 (2017) 020 [1712.07510].
  * (45) J. Böhm, A. Georgoudis, K.J. Larsen, M. Schulze and Y. Zhang, _Complete sets of logarithmic vector fields for integration-by-parts identities of Feynman integrals_ , _Phys. Rev. D_ 98 (2018) 025023 [1712.09737].
  * (46) J. Böhm, A. Georgoudis, K.J. Larsen, H. Schönemann and Y. Zhang, _Complete integration-by-parts reductions of the non-planar hexagon-box via module intersections_ , _JHEP_ 09 (2018) 024 [1805.01873].
  * (47) D. Bendle, J. Böhm, W. Decker, A. Georgoudis, F.-J. Pfreundt, M. Rahn et al., _Integration-by-parts reductions of Feynman integrals using Singular and GPI-Space_ , _JHEP_ 02 (2020) 079 [1908.04301].
  * (48) J. Boehm, D. Bendle, W. Decker, A. Georgoudis, F.-J. Pfreundt, M. Rahn et al., _Module Intersection for the Integration-by-Parts Reduction of Multi-Loop Feynman Integrals_ , _PoS_ MA2019 (2022) 004 [2010.06895].
  * (49) D. Bendle, J. Boehm, M. Heymann, R. Ma, M. Rahn, L. Ristau et al., _Two-loop five-point integration-by-parts relations in a usable form_ , 2104.06866.
  * (50) B. Feng and T. Li, _PV-Reduction of Sunset Topology with Auxiliary Vector_ , 2203.16881.
  * (51) R.M. Schabinger, _A New Algorithm For The Generation Of Unitarity-Compatible Integration By Parts Relations_ , _JHEP_ 01 (2012) 077 [1111.4220].
  * (52) P.A. Baikov, _Explicit solutions of the multiloop integral recurrence relations and its application_ , _Nucl. Instrum. Meth. A_ 389 (1997) 347 [hep-ph/9611449].
  * (53) W. Decker, G.-M. Greuel, G. Pfister and H. Schönemann, “Singular 4-3-0 — A computer algebra system for polynomial computations.” http://www.singular.uni-kl.de, 2022.
  * (54) B. Feng, T. Li and X. Li, _Analytic tadpole coefficients of one-loop integrals_ , _JHEP_ 09 (2021) 081 [2107.03744].
  * (55) C. Hu, T. Li and X. Li, _One-loop Feynman integral reduction by differential operators_ , _Phys. Rev. D_ 104 (2021) 116014 [2108.00772].
  * (56) B. Feng, J. Gong and T. Li, _Universal Treatment of Reduction for One-Loop Integrals in Projective Space_ , 2204.03190.
  * (57) B. Feng, C. Hu, T. Li and Y. Song, _Reduction with Degenerate Gram matrix for One-loop Integrals_ , 2205.03000.
  * (58) R.N. Lee and A.A. Pomeransky, _Critical points and number of master integrals_ , _JHEP_ 11 (2013) 165 [1308.6676].
  * (59) J. Chen, X. Jiang, C. Ma, X. Xu and L.L. Yang, _Baikov representations, intersection theory, and canonical Feynman integrals_ , 2202.08127.


[acronym]long-short

# A Mini Review on the utilization of Reinforcement Learning with OPC UA

Simon Schindler, Martin Uray, Stefan Huber  Josef Ressel Centre for
Intelligent and Secure Industrial Automation
Salzburg University of Applied Sciences
5412 Puch/Hallein, Austria
{ simon.schindler, martin.uray, stefan.huber<EMAIL_ADDRESS>
###### Abstract

Reinforcement Learning (RL) is a powerful machine learning paradigm that has
been applied in various fields such as robotics, natural language processing
and game playing achieving state-of-the-art results. Targeted to solve
sequential decision making problems, it is by design able to learn from
experience and therefore adapt to changing dynamic environments. These
capabilities make it a prime candidate for controlling and optimizing complex
processes in industry. The key to fully exploiting this potential is the
seamless integration of RL into existing industrial systems. The industrial
communication standard Open Platform Communications Unified Architecture (OPC
UA) could bridge this gap.

However, since RL and OPC UA are from different fields, there is a need for
researchers to bridge the gap between the two technologies. This work serves
to bridge this gap by providing a brief technical overview of both
technologies and carrying out a semi-exhaustive literature review to gain
insights on how RL and OPC UA are applied in combination.

With this survey, three main research topics have been identified, following
the intersection of RL with OPC UA. The results of the literature review show
that RL is a promising technology for the control and optimization of
industrial processes, but does not yet have the necessary standardized
interfaces to be deployed in real-world scenarios with reasonably low effort.

###### Index Terms:

OPC UA, Reinforcement Learning, Survey

##  I Introduction

The fourth industrial revolution aims to improve efficiency and productivity,
while reducing costs and downtime of production systems, by establishing
interconnectivity and intercommunication of all relevant participants in the
production processes. Successful and long-term sustainable integration of
participants like Internet of Things (IoT) devices, Machine Learning (ML)
algorithms and Digital Twins (DTs) into production processes presents a
significant challenge for the manufacturing industry. To enable systematic
research and implementation strategies, the four design principles for the
Industry 4.0 (I4.0) of Hermann et al. [13] should be adhered to:

  1. 1.

Interconnection: Secure communication of all participants relevant to the
production process, such as machines, devices, sensors and people.

  2. 2.

Information transparency: Context-aware information presentation assures
interpretability.

  3. 3.

Decentralized decisions: Enabled by the interconnection of all devices and a
given semantic for the provided information participants can make
decentralized decisions.

  4. 4.

Technical assistance: Technical assistance aggregating and visualizing
relevant information is substantial for the ongoing shift of the role of
humans from machine operators to strategic decision-makers in manufacturing.

OPC UA is a common industrial communication standard that provides reliable
and secure end-to-end communication of data and events between cross-platform
devices and software applications [2]. It is used in a variety of industries
and acts as a de facto standard communication and information modeling
protocol within Operational Technology (OT). It is an enabling technology for
flexible, adaptive and transparent production systems and has a wide range of
use cases of which an overview is given in [22]. With its standardized data
model [1], it does not only provide a framework for the communication of data
and events, but also enables the modeling of the entire industrial information
network in an object-oriented way. Therefore, OPC UA satisfies the first two
design principles of I4.0 by providing a secure intercommunication platform
(1. interconnection) and by giving communicated information a semantic that
can be interpreted by all participants (2. information transparency).

RL is a popular ML technique that has been widely applied in various fields,
including robotics [15], natural language processing [28, 16], and game
playing at even superhuman levels [17, 25]. It is able to learn on the fly,
adapt to ever-changing environments and is specifically well suited for
solving sequential decision-making problems, such as the control of a
production system. Its possible applications in an industrial environment are
manifold as outlined by the extensive review on RL in industrial process
control provided by Nian in [18]. In combination with OPC UA, it could
potentially suffice the third and fourth design needs of Herrman et al. by
acting as a technical assistant for human decision makers or even as a
decision maker itself. OPC UA is specifically well suited to be used with RL
as it fulfills two essential needs of a RL environment: It provides access to
information relevant to the decision-making process and it enables the agent
to act on the environment.

The combination of both technologies, OPC UA and RL, could in theory enable
continuously self-optimizing control and operation of industrial systems and
systems-of-systems of large scale resulting in a significant increase in
efficiency and productivity.

To the best of our knowledge, there exists no overview of scientific
literature that utilizes OPC UA in combination with RL. Hence, this work would
be the first contribution in this area. Through that, we aim to open up this
field of research to both researchers and practitioners of the manufacturing
and ML domain.

To achieve that we first provide a short introduction to the technical
terminology of both OPC UA and RL and then present a selection of research
articles that utilize OPC UA in combination with RL. While this field is not
yet to be recognized as a much successful application domain, we show that
there are various research articles around, that demonstrate the rich
interplay of both, RL and OPC UA.

##  II Terminology and technical background

OPC UA and RL are nested within different domains of computer science. In
order to make the discussion of both topics more accessible to the reader of
this article, who may belong to just one of these fields, this section briefly
provides some basic information and terminology on both backgrounds.

###  II-A Reinforcement Learning

RL is a ML paradigm, that explicitly does not need any supervisory data. Its
approach of learning towards optimal decisions differs from supervised
learning in that sense that RL agents learn from experiences through
interactions with an environment rather than from labeled data.

RL is specifically designed to solve sequential decision making problems and
can deal with high-dimensional state and action spaces. Therefore it is very
well suited for the control of industrial processes, which often provide a
vast amount of data to consider (the state space) and have a high number of
degrees of freedom which actions to take (the action space).

The RL concept, as illustrated in Figure 1, is systematically very similar to
control theory. The analog to the controller would be the agent, the
environment would be the plant and the reward would be the control error. But
in contrast to classical control theory, the agent is not given a model (with
some exceptions) of the plant, but has to learn it from experience. Since its
model of the environment changes with learning iterations, its performance,
dependent on the quality of the model, is not static as in classic control
theory, but dynamic.

This type of learning is very similar to the learning process of humans, where
one is rewarded or punished for decisions taken. Further decisions are
impacted by the consequences in the past.

[width=.8]figures/rl_concept

Figure 1: The RL agent-environment interaction according Sutton and Barto
[26].

Many RL problems can be formulated as Markov Decision Processs (MDPs). Let $S$
be the set of all possible states of the environment and $A$ the set of all
possible actions. In a MDP the transitions from one state $s_{i}$ to the next
$s_{i+1}$ is stochastic, controlled by the taken action $a_{i}$. During
training of an agent, given a state $s_{i}\in S$, at each time step $i$, the
agent picks an action $a_{i}\in A$ that leads to a reward
$r_{i+1}\in\mathbb{R}$ and a successor state $s_{i+1}\in S$. The goal of the
agent is to max. the (discounted) cumulative reward
$G_{t}=\sum_{i=t+1}^{T}\gamma^{i-t-1}\cdot r_{i}$, where $t$ is the current
time step, $T$ the optimization horizon and $\gamma\in(0,1)$ is a discount
factor. This is achieved by the adaption of the agents’ policy
$\pi:S\rightarrow A$, a mapping from states to actions.

For a more detailed introduction to RL, the reader may be referred to Sutton
and Barto [26].

###  II-B OPC UA

The OPC UA is a standard by the IEC [1] released in 2008 that specifies a
cross-platform data exchange format for sensory- and machine data. In
comparison to other standards, with OPC UA not only the data is exchanged,
also the semantics of the data and relationships between network participants
are included. This makes OPC UA as a communication architecture specifically
well suited for the use of ML methods such as RL in a manufacturing
environment. This extra information can be at least beneficial but even
substantial for successfully applying ML.

For the purpose of semantically describing the data and the participants
interrelations, so-called Information Models (IMs) are utilized. Within such a
Information Model (IM), the definition of the structure, data types, and
relationships between the various information resources that are available for
communication are modeled. This framework utilizes flexibility and
extensibility, by enabling dynamic adaptions, e.g. adding new nodes. The nodes
represent, among other things, objects, variables or methods that are
accessible through the network. Each node is uniquely identifiable by its
NodeID and described by its NodeClass defining its properties and
capabilities. Furthermore, nodes can have references to one another,
representing their relationships and the ways in which they can be accessed.
For example, object nodes are physical or abstract objects within the system
and can have properties in the form of variable nodes and callable software
functions accessible through method nodes.

Communication in OPC UA is always established between a client and a server,
both of which a system can possess multiple instances. The flexibility of the
architecture allows for each client to be concurrently connected to multiple
servers and vice versa. The server provides the client with an interface to
information by granting access to nodes through the use of its so-called
address space. This information, once obtained, can be interpreted by the
client through the utilization of the corresponding IM.

In short OPC UA models the information and the relationships of participants
in an industrial environment and provides the possibility to interact with the
participants. This makes it a suited candidate to enable RL in an industrial
environment, since RL needs both information about the environment and a way
of interacting with it.

##  III Methodology

The research methodology used for this semi-exhaustive literature review is a
reproducible four-step procedure. This method is motivated by Tschuchnig et
al. [27] and based on Randolph [19]. The problem defined for this review is
the evaluation of the application of RL with OPC UA.

For proper search queries, the terms “reinforcement learning” and “opc ua”
were identified. No further keywords were considered to be included.

For data collection, the search engine Google Scholar was chosen. This search
engine enables to specifically formulate a search query using boolean
operators. Additionally, quotation marks can be used to indicate, that the
containing keywords must be included within the results. Using these
properties, the search query “reinforcement learning” AND “opc ua” was used.
Giving $10$ results per page, we limited the data collection to the results of
the $6$ first pages, i.e. resulting in $60$ references of different kind.

For further processing, a set of criteria was defined: In order to deliver a
qualitative review, for this work only conference- and journal publications
were taken into account.111We are aware that not all conference- and journal
publications may be peer-reviewed. Preprints, presentations, reports, or
commercial information were excluded. Furthermore, this review takes all
publications into account, that were published before 15th of February 2023,
with no restrictions on a lower bound. The language of the publication had to
be English and a full-text version had to be available for further steps.

The last filtering step was to read the full-text, to find out if the work is
in the context of our review’s research interest. Publication just naming the
methodologies, e.g. in a related work section or discussion, were excluded.

As a final step, the remaining literature was analyzed and were grouped into
manually identified clusters.

##  IV Results

The result of the semi-exhaustive literature review exposed a number of $17$
conference- and journal papers. In the following, the three identified
categories are listed and elaborated.

###  IV-A Cluster 1: Industrial Applications of RL utilizing OPC UA for
communication

The first cluster consists of ten papers. All of them applied RL in an
industrial setting and relied on OPC UA just for communication. The type of
applications is diverse. These range from an agent learning to play table
football using industrial drives [6, 20] to optimizing energy costs of a
production plant [4, 24, 23]. They all did not directly address the
combination of RL and OPC UA, but rather focused on the application of RL in
an industrial setting, while applying OPC UA just for communication. Hence,
they are not described in further detail in this work.

An interesting observation in this cluster is the predominant utilization of
virtual simulations for pre-training. The papers gathered in cluster 1 are
listed in table I. There, the results are summarized highlighting their
learning methods, the use of virtual simulations for pre-deployment training,
and the specific applications applied on.

Ref. |  RL-method | use simulation |  summary
---|---|---|---
[24, 23] |  Q-Learning (QL) | $\checkmark$ |  energy reduction of production plants by dynamic hibernation of sub-systems depending on current and future estimated production load
[7] |  QL | $\times$ |  process time optimization in an assembly process
[29] |  Deep Q-Learning (D-QL) | $\checkmark$ |  evaluation of DTs for industrial RL
[10] |  On- and Offline Policy Gradient Methods with DNNs | $\times$ |  automated PID controller tuning in an industrial environment
[4] |  D-QL | $\times$ |  energy cost optimization through adaptive utilization of production systems depending on energy prices
[6] |  Actor-Critic Policy Gradient Method with DNNs | $\checkmark$ |  simulation based deep-RL to play the attacker at table football using industrial drives
[20] |  D-QL | $\checkmark$ |  simulation based deep-RL to play the goalkeeper at table football using industrial drives
[9] |  Deep-RL (no further details provided) | $\checkmark$ |  hard- and software in the loop assessment of an industrial control architecture
[3] |  Proximal Policy Optimization (PPO) | $\checkmark$ |  case study of the integration of RL into the life cycle of industrial DTs
TABLE I: Cluster 1: industrial applications of RL just relying on OPC UA for
communication

###  IV-B Cluster 2: Architectures for the integration of RL into industrial
environments utilizing OPC UA

To enable the switch from the simulation environment to the actual industrial
environment, the interface between the agent and the affected systems should
not differ too much from the simulation. Using OPC UA for communication
between the agent and the environment already in simulation facilitates the
switch, but still requires some implementation effort, caused by a lack of a
standardized interface for RL within OPC UA. This lack of architectural
standardization is addressed by the second cluster of papers analyzed, listed
in table II.

Grothoff et al. [12] propose a mapping of states of standardized state
machines to generic interactions of RL algorithms for learning and inference
phases. They showcase a proof of concept in the form of a simulation of a coil
transport system for cold rolling steel mills.

An architecture for a modular RL environment generator is proposed by Csiszar
et al. [8]. This generator operates as a configurable adapter between the RL
agent and industrial communication protocols such as OPC UA. Their proposed
method especially targets practitioners of the domain of production
engineering, with no prerequisites in software engineering, to enable easy
integration of RL algorithms into industrial environments for non domain
experts. However, this work is only of conceptual nature and no implementation
of such a generator has been published.

In [21], the authors propose an OPC UA IM to deploy RL algorithms to
industrial environments. Therefore, OPC UA nodes are extended by sensor and
action properties. This enables automatic generation of the observation- and
action space and the deployment of RL agents without an engineering effort.
Using this architecture, a RL agent has been successfully deployed to a model
example using a Hardware-in-the-Loop simulation.

Gracia et al. [11] created a framework for the integration of robotics, in
terms of hardware and control software into industrial environments. The
communication is based on OPC UA. A configurable OPC UA server hosts the
control logic, such as RL algorithms for robotic components. In turn, these
are accessible as OPC UA clients as well. The attachment of a robotic
component to the framework is enabled by template-based plugins.

In [14], the authors developed a framework based on OPC UA for communication
which can be used to integrate RL algorithms into industrial environments. In
contrast to [11], this is rather focused on ML methods than on robotics. Their
approach enables a native integration via the Python programming language by
means of common ML libraries such as TensorFlow, PyTorch and others.
Furthermore, their work requires all used components to be compatible with the
Module Type Package (MTP), a standardized description of the automation
interface for self-contained production units.

Ref. | open-source |  summary
---|---|---
[8] | $\times$ |  environment generator as adapter between interface of agent and industrial communication protocols
[12] | $\times$ |  mapping of standardized state machine states to generic interactions of RL algorithms
[21] | $\times$ |  OPC UA IM for deploying and exchanging the RL agent
[11] | $\checkmark$ |  framework allowing integration of external hardware and software (s.a. a RL agent) based on OPC UA
[14] | $\checkmark$ |  framework allowing integration of external hardware and software (s.a. a RL agent) based on OPC UA and MTP
TABLE II: Cluster 2: architectures for the integration of RL into industrial
environments utilizing OPC UA

###  IV-C Cluster 3: RL applied for information inferences from OPC UA IMs

The third cluster contains work on information inferences about OPC UA IMs by
embedding them in knowledge graphs and applying RL with Graph Neural Networks
(GNNs). Knowledge graphs are semantic networks that can be represented in OPC
UA through information modeling. The cluster is listed in table III.

Bakakeu et al.[5] train a RL agent to be capable of inferring information from
semantically incomplete IMs. Here, the aim is to discover missing
relationships between nodes and the application of consistency checks. This
was achieved by constructing multi-hop relationship paths along the embedding
vector space of the knowledge graph. The knowledge graph represents the IM by
training a RL agent, based on Long short-term Memories (LSTMs) with PPO, with
the aim to predict the relationships between the given entities.

A similar approach is taken by Zheng et al. in [30]. They establish a
manufacturing system capable of a semantics-based solution search on the
established knowledge graph, that enables the organization of available on-
site manufacturing resources. To this end, they train multiple agents on an
embedding of a knowledge graph. In this case, this graph is not constructed
from the OPC UA IMs only but on information on the relationships from multiple
sources. Since their method builds on the initially constructed knowledge
graph, they highlight the challenge of dealing with ever-changing dynamic
environments as an open research question in this area.

Ref. |  RL-method |  summary
---|---|---
[5] |  PPO |  RL for reasoning on semantically incomplete OPC UA IMs for the discovery of missing relations between the entities
[30] |  multi-agent independent learning with a soft actor critic policy gradient method |  knowledge graph based multi-agent RL for self-(configuration, optimization, adjustment) of a manufacturing network
TABLE III: Cluster 3: RL applied for information inferences from OPC UA IMs

##  V Discussion

The first cluster exposes the variety of opportunities for the application of
RL in industrial environments, while building on OPC UA as a communication
platform. In eight out of these ten papers, virtual simulations of the
environment for parts of the training process were used. The utilization of
simulations enables faster training due to a separation from the production
environment during the initial training phase and the possibility of
parallelization. This separation from real machines also prevents the system
from causing costly downtimes, as well as the introduction of possible
hazardous situations during random exploration. Furthermore, this approach
enables a much more convenient form of experimentation, since different
algorithms and their hyperparameters can be tested without interaction with
the real-world. The resulting number of publications in this cluster shows,
that this topic is of general interest and of importance. In many cases, the
simulations of the real-world environment must be enabled and communication
interfaces in the simulation must be consistent with reality. This seems to be
a key factor for successful deployment, which strongly points in the direction
of DTs and OPC UA.

The great potential of utilizing RL in industrial environments is limited by
the lack of a standardized integration within OPC UA. In order to facilitate
the application of RL in industrial environments, this bottleneck needs to be
addressed. Still, only five publications have been found that address this
issue to at least some extent. Here, much more research and actual practical
considerations are needed in the future. The concept of Csiszar et al. for a
RL-environment generator for integrating RL into industrial networks [8] would
be a major contribution to this field, if it did not lack an implementation.
The work of Schäfer et al., which proposed a OPC UA IM for RL [21] could be
further developed to provide a solid basis for future official
standardization. The two implemented frameworks for integrating RL with OPC UA
[11, 14] are promising to significantly reduce the programming effort for
successful integration of an already pretrained agent, even though they are
not fully tailored to RL.

The work in Cluster 3 shows that RL is a suitable methodology among other
approaches for working with knowledge graphs created from OPC UA IMs. But
still, they are not yet able to adapt to changing environments. Extending
these systems to provide adaptivity, would be a great improvement, since this
could pave the way for self-optimizing and self-configuring industrial
networks.

This work is the first survey to highlight aspects of ML in the context OPC
UA. For further work, this survey could be extended to include other ML
methods for a more holistic overview.

##  VI Conclusion

OPC UA models the information and relationships of participants in an
industrial environment and provides a way to interact with the participants.
This makes it a perfect candidate to enable RL in an industrial environment,
since RL requires both information about the environment and a way to interact
with it.

This work provides an overview of the current state of research on the
application of RL with OPC UA by means of a systematic semi-exhaustive
literature review.

There have been found three main clusters in which the papers can be grouped.
Papers of the first cluster focus on using RL to control and optimize
industrial processes, including production lines and energy systems. A trend
that can be observed in this cluster is the use of simulation environments and
DTs, since training RL agents in cost and safety-sensitive environments can be
unsuitable. OPC UA can be even used in the simulations to closely resemble the
real-world industrial environment, in terms of behavior on interaction with
the agent and interfacing by means of communication protocols and standards.

The integration of RL algorithms with OPC UA is a significant challenge
addressed by the papers in the second cluster found, since there does not
exist a standardized framework for installing a RL environment in an
industrial setting. Worked on by only a few authors there is much room for
research left in this area to provide a solid base for future standardization
processes. The lack of open-source implementations of the proposed frameworks
is a major drawback of the papers in this cluster. Research in this
subcategory also publishing implementations would be most beneficial to the
whole field, since standardized and therefore faster integration of RL into
industrial environments would enable and encourage research in the other
subcategories.

The last group of papers considered the use of RL for information inference
from OPC UA IMs by embedding them in knowledge graphs. This has been explored
by some authors [5, 30]. They conclude that these techniques could be useful
to enable self-organizing manufacturing systems in the future, while still
addressing fundamental problems when dealing with dynamically changing
industrial environments opening the field up for further research.

In conclusion, the combination of RL and OPC UA is a promising area of
research and holds significant potential for improving industrial processes.
However, the beforehand outlined challenges need to be addressed.

## References

  * [1] “OPC UA Online Reference - Released Specifications.” [Online]. Available: https://reference.opcfoundation.org/
  * [2] “Unified Architecture - OPC Foundation.” [Online]. Available: https://opcfoundation.org/about/opc-technologies/opc-ua/
  * [3] F. Abdoune, M. Nouiri, O. Cardin, and P. Castagna, “Integration of artificial intelligence in the life cycle of industrial digital twins,” _IFAC-PapersOnLine_ , vol. 55, pp. 2545–2550, 1 2022.
  * [4] J. Bakakeu and J. Bauer, “An artificial intelligence approach for online energy optimization of flexible manufacturing systems,” _Article in Applied Mechanics and Materials_ , 2018.
  * [5] J. Bakakeu, F. Schafer, J. Franke, S. Baer, H. H. Klos, and J. Peschke, “Reasoning over opc ua information models using graph embedding and reinforcement learning,” _Proceedings - 2020 3rd International Conference on Artificial Intelligence for Industries, AI4I 2020_ , pp. 40–47, 9 2020.
  * [6] S. D. Blasi, S. Klöser, A. Müller, R. Reuben, F. Sturm, and T. Zerrer, “Kicker: An industrial drive and control foosball system automated with deep reinforcement learning,” _Journal of Intelligent and Robotic Systems: Theory and Applications_ , vol. 102, pp. 1–18, 5 2021.
  * [7] P. Burggräf, F. Steinberg, B. Heinbach, and M. Bamberg, “Reinforcement learning for process time optimization in an assembly process utilizing an industry 4.0 demonstration cell,” _Procedia CIRP_ , vol. 107, pp. 1095–1100, 1 2022.
  * [8] A. Csiszar, V. Krimstein, J. Bogner, and A. Verl, “Generating reinforcement learning environments for industrial communication protocols,” _Proceedings - 2021 4th International Conference on Artificial Intelligence for Industries, AI4I 2021_ , pp. 57–60, 2021.
  * [9] R. Dobrescu, O. Chenaru, G. Florea, G. Geampalia, and S. Mocanu, “Hardware-in-loop assessment of control architectures,” _2020 24th International Conference on System Theory, Control and Computing, ICSTCC 2020 \- Proceedings_ , pp. 880–885, 10 2020.
  * [10] O. Dogru, K. Velswamy, F. Ibrahim, Y. Wu, A. S. Sundaramoorthy, B. Huang, S. Xu, M. Nixon, and N. Bell, “Reinforcement learning approach to autonomous pid tuning,” _Computers and Chemical Engineering_ , vol. 161, p. 107760, 5 2022.
  * [11] J. B. Gracia, F. Leber, M. Aburaia, and W. Wöber, “A configurable skill oriented architecture based on opc ua,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 7317–7322.
  * [12] J. Grothoff and T. Kleinert, “Mapping of standardized state machines to utilize machine learning models in process control environments,” pp. 39–53, 2021.
  * [13] M. Hermann, T. Pentek, and B. Otto, “Design principles for industrie 4.0 scenarios,” _Proceedings of the Annual Hawaii International Conference on System Sciences_ , vol. 2016-March, pp. 3928–3937, 3 2016.
  * [14] V. Khaydarov, L. Neuendorf, T. Kock, N. Kockmann, and L. Urbas, “Mtppy: Open-source ai-friendly modular automation,” in _2022 IEEE 27th International Conference on Emerging Technologies and Factory Automation (ETFA)_ , 2022, pp. 1–7.
  * [15] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning in robotics: A survey,” _International Journal of Robotics Research_ , vol. 32, 2013.
  * [16] J. Luketina, N. Nardelli, G. Farquhar, J. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel, “A Survey of Reinforcement Learning Informed by Natural Language,” _IJCAI International Joint Conference on Artificial Intelligence_ , vol. 2019-August, pp. 6309–6317, 6 2019.
  * [17] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” 12 2013.
  * [18] R. Nian, J. Liu, and B. Huang, “A review on reinforcement learning: Introduction and applications in industrial process control,” _Computers and Chemical Engineering_ , vol. 139, p. 106886, 2020.
  * [19] J. Randolph, “A guide to writing the dissertation literature review,” _Practical Assessment, Research, and Evaluation_ , vol. 14, p. 13, 11 2019\.
  * [20] T. Rohrer, L. Samuel, A. Gashi, G. Grieser, and E. Hergenröther, “Foosball table goalkeeper automation using reinforcement learning.” in _LWDA_ , 2021, pp. 173–182.
  * [21] G. Schäfer, R. Kozlica, S. Wegenkittl, and S. Huber, “An architecture for deploying reinforcement learning in industrial environments,” in _Computer Aided Systems Theory – EUROCAST 2022_ , 2022, pp. 569–576.
  * [22] M. Schleipen, S.-S. Gilani, T. Bischoff, and J. Pfrommer, “Opc ua and industrie 4.0 - enabling technology with high diversity and variability,” _Procedia CIRP_ , vol. 57, pp. 315–320, 2016, factories of the Future in the digital environment - Proceedings of the 49th CIRP Conference on Manufacturing Systems.
  * [23] E. Schmidl, E. Fischer, J. Steindl, M. Wenk, and J. Franke, “Reinforcement learning for energy reduction of conveying and handling systems,” _Procedia CIRP_ , vol. 97, pp. 290–295, 1 2021.
  * [24] E. Schmidl, E. Fischer, M. Wenk, and J. Franke, “Knowledge-based generation of a plant-specific reinforcement learning framework for energy reduction of production plants,” in _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2020, pp. 1–4.
  * [25] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, and D. Hassabis, “Mastering the game of go without human knowledge,” _Nature 2017 550:7676_ , vol. 550, pp. 354–359, 10 2017.
  * [26] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT press, 2018.
  * [27] M. E. Tschuchnig and M. Gadermayr, “Anomaly detection in medical imaging - a mini review,” _Data Science – Analytics and Applications_ , pp. 33–38, 8 2022.
  * [28] V. Uc-Cetina, N. Navarro-Guerrero, A. Martin-Gonzalez, C. Weber, and S. Wermter, “Survey on reinforcement learning for language processing,” _Artificial Intelligence Review_ , pp. 1–33, 6 2022.
  * [29] K. Xia, C. Sacco, M. Kirkpatrick, C. Saidy, L. Nguyen, A. Kircaliali, and R. Harik, “A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence,” _Journal of Manufacturing Systems_ , vol. 58, pp. 210–230, 1 2021.
  * [30] P. Zheng, L. Xia, C. Li, X. Li, and B. Liu, “Towards self-x cognitive manufacturing network: An industrial knowledge graph-based multi-agent reinforcement learning approach,” _Journal of Manufacturing Systems_ , vol. 61, pp. 16–26, 10 2021.