# Cohomology and Relative Rota-Baxter-Nijenhuis structures on
$\mathsf{LieYRep}$ pairs

Jia Zhao  Jia Zhao, School of Sciences, Nantong University, Nantong 226019,
Jiangsu, China<EMAIL_ADDRESS>and  Yu Qiao  Yu Qiao (corresponding
author), School of Mathematics and Statistics, Shaanxi Normal University,
Xi’an 710119, Shaanxi, China<EMAIL_ADDRESS>
###### Abstract.

A $\mathsf{LieYRep}$ pair consists of a Lie-Yamaguti algebra and its
representation. In this paper, we establish the cohomology theory of
$\mathsf{LieYRep}$ pairs and characterize their linear deformations by the
second cohomology group. Then we introduce the notion of relative Rota-Baxter-
Nijenhuis structures on $\mathsf{LieYRep}$ pairs, investigate their
properties, and prove that a relative Rota-Baxter-Nijenhuis structure gives
rise to a pair of compatible relative Rota-Baxter operators under a certain
condition. Finally, we show the equivalence between $r$-matrix-Nijenhuis
structures and Rota-Baxter-Nijenhuis structures on Lie-Yamaguti algebras.

###### Key words and phrases:

Lie-Yamaguti algebra, cohomology, relative Rota-Baxter-Nijenhuis structure,
$r$-matrix-Nijenhuis structure

Mathematics Subject Classification (2020): Primary 17B38; Secondary
17B60,17A99

###### Contents

  1. 1 Introduction
  2. 2 Preliminaries
  3. 3 Cohomology and Linear deformations of $\mathsf{LieYRep}$ pairs
    1. 3.1 Cohomology of $\mathsf{LieYRep}$ pairs
    2. 3.2 Linear deformations of $\mathsf{LieYRep}$ pairs
  4. 4 Relative Rota-Baxter-Nijenhuis structures and compatible relative Rota-Baxter operators
    1. 4.1 Relative Rota-Baxter-Nijenhuis structures on $\mathsf{LieYRep}$ pairs
    2. 4.2 Compatible relative Rota-Baxter operators on Lie-Yamaguti algebras
  5. 5 $r$-matrix-Nijenhuis structures and Rota-Baxter-Nijenhuis structures

##  1\. Introduction

Poisson-Nijenhuis structures appeared in the work of Magri and Morosi [22] in
studying completely integrable systems. Such objects can also be found in [18,
19]. Later, Ravanpak, Rezaei-Aghdam, and Haghighatdoost exploited an $r-n$
structure on a Lie algebra, which is the infinitesimal right-invariant
Poisson-Nijenhuis structure on the Lie group $G$ integrating the Lie algebra
$\mathfrak{g}$ [25]. Namely, an $r-n$ structure on a Lie algebra
$\mathfrak{g}$ is a pair $(\pi,N)$, where $\pi\in\wedge^{2}\mathfrak{g}$ is a
classical $r$-matrix, and $N$ is a Nijenhuis operator on $\mathfrak{g}$ such
that some conditions are satisfied. Motivated by their works, Liu, Sheng and
their collaborators studied Poisson-Nijenhuis structures on Lie algebras,
associative algebras and pre-Lie algebras in [13, 15, 21], where they call
this object the Kupershmidt-Nijenhuis structures or
${{\mathcal{O}}}N$-structures. In particular, they named an $r-n$ structure as
an $r$-matrix-Nijenhuis structure and investigated its relation with the Rota-
Baxter-Nijenhuis structures on a $\mathsf{LieRep}$ pair in [13]. Based on
their works, the first author, Liu, and Sheng explored the Poisson-Nijenhuis
structures on $3$-$\mathsf{LieRep}$ pairs, in which Poisson-Nijenhuis
structures were called relative Rota-Baxter-Nijenhuis structures [35].

The notion of Lie-Yamaguti algebras, a generalization of Lie algebras and Lie
triple systems, can be dated back to Nomizu’s work on invariant affine
connections on homogeneous spaces in 1950’s ([24]) and Yamaguti’s work on
general Lie triple systems and Lie triple algebras ([31]). Then Yamaguti
introduced its representation and established its cohomology theory in [32,
33] during 1950’s to 1960’s. Later until 21st century, Kinyon and Weinstein
named this object as a Lie-Yamaguti algebra when studying Courant algebroids
in [17]. Lie-Yamaguti algebras have attracted much attention in recent years.
For instance, Benito, Draper, and Elduque investigated Lie-Yamaguti algebras
related to simple Lie algebras of type $G_{2}$ [7]. Afterwards, Benito,
Elduque, and Mart$\acute{i}$n-Herce explored irreducible Lie-Yamaguti algebras
in [8, 9]. More recently, Benito, Bremmer, and Madariaga examined orthogonal
Lie-Yamaguti algebras in [6].

Baxter first introduced the notion of Rota-Baxter operators on associative
algebras when studying fluctuation theory in 1960’s [5]. A more generalized
concept – relative Rota-Baxter operators (also called
${\mathcal{O}}$-operators or Kupershmidt operators) on Lie algebras, was
introduced by Kupershmidt in the study of classical Yang-Baxter equation,
which has lots of applications in physics and mathematical physics [20].
Kupershmidt found that a relative Rota-Baxter operator with respect to the
coadjoint representation is a solution to the classical Yang-Baxter equation.
Note that a relative Rota-Baxter operator is associated to an arbitrary
representation, whereas a Rota-Baxter operator is related with the adjoint
representation. One can see [1, 2, 3, 4] for Rota-Baxter operators on other
type of algebras and see [12] for more details about Rota-Baxter algebras.
Note that a relative Rota-Baxter operator on a Lie algebra (an associative
algebra) is a solution to the Maurer-Cartan equation of a certain graded Lie
algebra ([29, 30]), whereas a Poisson structure is also a solution to a
Maurer-Cartan equation of a graded Lie algebra whose structure are precisely
the Schouten-Nijenhuis bracket of multi-vector fields. Thus a relative Rota-
Baxter operator on a Lie algebra (an associative algebra) are analogues of a
Poisson structure. Another kind of crucial operators–a Nijenhuis operator on a
Lie algebra appeared in the study of integrable systems by Magri, Gelfand, and
Dorfman [10], and then Fuchssteiner and Fokas rediscovered it and named this
kind of operators as hereditary operators in [11]. In deformation theory of
Lie algebras, a Nijenhuis operator can be obtained via a trivial deformation.
One can see [10, 14, 23] for more details about Nijenhuis operators on Lie
algebras or even $n$-Lie algebras.

Recently, the first author and Sheng explored linear deformations, Nijenhuis
operators, and relative Rota-Baxter operators on Lie-Yamaguti algebras in [26]
and [27] respectively. We studied cohomology and deformations of relative
Rota-Baxter operators on Lie-Yamaguti algebras in [36], and established
bialgebra theory of Lie-Yamaguti algebras in [37]. Motivated by the virtual of
Poisson-Nijenhuis structures and our former works on Lie-Yamaguti algebras, we
shall investigate relative Rota-Baxter-Nijenhuis structures on a Lie-Yamaguti
algebra with a representation. By a $\mathsf{LieYRep}$ pair, denoted by
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$,
we mean a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with a representation $(V;\rho,\mu)$. Note that we still call a Poisson-
Nijenhuis structure on a $\mathsf{LieYRep}$ pair a relative Rota-Baxter-
Nijenhuis structure in the present paper.

Outline of the paper. After recalling some basic definitions in Section 2, we
establish the cohomology theory of $\mathsf{LieYRep}$ pairs and utilize it to
characterize linear deformations in Section 3. The corresponding complex is
exactly a subcomplex of that of the associated semidirect product Lie-Yamaguti
algebra with coefficients in the adjoint representation. Then we investigate
the linear deformations of $\mathsf{LieYRep}$ pairs and introduce the notion
of Nijenhuis structures $(N,S)$, where $N\in\mathfrak{gl}(\mathfrak{g})$ and
$S\in\mathfrak{gl}(V)$. A Nijenhuis structure gives rise to a Nijenhuis
operator $N+S$ on the semidirect Lie-Yamaguti algebra $(\mathfrak{g}\oplus
V,[\cdot,\cdot]{{}_{\ltimes}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\ltimes})$
and a trivial deformation of $(\mathfrak{g},\rho,\mu)$.

In Section 3, the notion of relative Rota-Baxter-Nijenhuis structures on
$\mathsf{LieYRep}$ pairs is introduced and some properties are investigated.
Given linear maps $N\in\mathfrak{gl}(\mathfrak{g})$ and
$S\in\mathfrak{gl}(V)$, and let $T:V\longrightarrow\mathfrak{g}$ be a relative
Rota-Baxter operator on a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to $(V;\rho,\mu)$. Then there exists a Lie-Yamaguti algebra
structure
$([\cdot,\cdot]{{}^{T}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$
on $V$, which is called the sub-adjacent Lie-Yamaguti algebra, and is denoted
by $V^{T}$. Consequently, we obtain the deformed brackets
$([\cdot,\cdot]{{}^{T}_{S}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{S})$;
meanwhile, we introduce linear maps
$\hat{\rho}:\mathfrak{g}\to\mathfrak{gl}(V)$ and
$\hat{\mu}:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(V)$ associated with linear
maps $N$ and $S$ to obtain the brackets
$([\cdot,\cdot]{{}^{T}_{\hat{\rho}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{\hat{\mu}})$.
It is not true that
$([\cdot,\cdot]{{}^{T}_{S}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{S})$
or
$([\cdot,\cdot]{{}^{T}_{\hat{\rho}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{\hat{\mu}})$
forms a Lie-Yamaguti algebra structure on $V$. In order to tackle this
problem, we introduce the main object of this paper: relative Rota-Baxter-
Nijenhuis structures. We find that deformed brackets
$([\cdot,\cdot]{{}^{T}_{S}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{S})$
and
$([\cdot,\cdot]{{}^{T}_{\hat{\rho}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{\hat{\mu}})$
are equal under the condition that the triple $(T,S,N)$ is a relative Rota-
Baxter-Nijenhuis structure. Besides, the linear map $S$ is also a Nijenhuis
operator on the sub-adjacent Lie-Yamaguti algebra
$(V,[\cdot,\cdot]{{}^{T}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$,
hence both
$([\cdot,\cdot]{{}^{T}_{S}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{S})$
and
$([\cdot,\cdot]{{}^{T}_{\hat{\rho}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T}_{\hat{\mu}})$
are Lie-Yamaguti algebra structures on $V$. Furthermore, we show that the
composition $N\circ T$ is a relative Rota-Baxter operator on the Lie-Yamaguti
algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to $(V;\rho,\mu)$. These results are summarized in the following
commutative diagram:

$\textstyle{(V,[\cdot,\cdot]_{S}^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{S}^{T})\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{S}$$\scriptstyle{N\circ
T}$$\scriptstyle{T}$$\textstyle{(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{N}$$\textstyle{(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{T}$$\textstyle{(\mathfrak{g},[\cdot,\cdot]\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),}$

where each arrows is a Lie-Yamaguti algebra homomorphism, i.e., each $T$ is a
relative Rota-Baxter operator, and both $S$ and $N$ are Nijenhuis operators.

Unlike those on Lie algebras or associative algebras, a relative Rota-Baxter-
Nijenhuis structure on a $\mathsf{LieYRep}$ pair does not give rise to a
hierarchy of relative Rota-Baxter operators and what is worse, it does not
guarantee that $T$ and $N\circ T$ are compatible even though $N\circ T$ is a
relative Rota-Baxter operator. They, however, are compatible if we impose an
extra condition on a relative Rota-Baxter-Nijenhuis structure. This phenomenon
is similar to the context of $3$-$\mathsf{LieRep}$ pairs ([35]) and
demonstrates that properties of a relative Rota-Baxter-Nijenhuis structure on
ternary operations are distinguished from those on binary operations.

Finally in Section 5, we introduce the notion of relative Rota-Baxter-dual
Nijenhuis structures by using dual-Nijenhuis structures, which is dual to that
of Nijenhuis structures. In fact, an $r$-matrix-Nijenhuis structure is indeed
a special relative Rota-Baxter-dual Nijenhuis structure with respect to the
coadjoint representation and we show that $r$-matrix-Nijenhuis structures and
Rota-Baxter-Nijenhuis structures are equivalent under certain conditions.

Acknowledgements: Qiao was partially supported by NSFC grant 11971282.

##  2\. Preliminaries

All vector spaces in the article are assumed to be over a field $\mathbb{K}$
of characteristic $0$ and finite-dimensional. In this section, we recall some
basic notions such as Lie-Yamaguti algebras, representations and their
cohomology theory.

######  Definition 2.1.

([17]) A Lie-Yamaguti algebra is a vector space $\mathfrak{g}$ equipped with a
bilinear operation $[\cdot,\cdot]:\wedge^{2}\mathfrak{g}\to\mathfrak{g}$ and a
trilinear operation
$\left\llbracket\cdot,\cdot,\cdot\right\rrbracket:\wedge^{2}\mathfrak{g}\otimes\mathfrak{g}\to\mathfrak{g}$,
such that the following conditions hold

$\displaystyle[[x,y],z]+[[y,z],x]+[[z,x],y]+\left\llbracket
x,y,z\right\rrbracket+\left\llbracket y,z,x\right\rrbracket+\left\llbracket
z,x,y\right\rrbracket=0,$
$\displaystyle\left\llbracket[x,y],z,w\right\rrbracket+\left\llbracket[y,z],x,w\right\rrbracket+\left\llbracket[z,x],y,w\right\rrbracket=0,$
$\displaystyle\left\llbracket x,y,[z,w]\right\rrbracket=[\left\llbracket
x,y,z\right\rrbracket,w]+[z,\left\llbracket x,y,w\right\rrbracket],$
$\displaystyle\left\llbracket x,y,\left\llbracket
z,w,t\right\rrbracket\right\rrbracket=\left\llbracket\left\llbracket
x,y,z\right\rrbracket,w,t\right\rrbracket+\left\llbracket z,\left\llbracket
x,y,w\right\rrbracket,t\right\rrbracket+\left\llbracket z,w,\left\llbracket
x,y,t\right\rrbracket\right\rrbracket,$

for all $x,y,z,w,t\in\mathfrak{g}$. We denote a Lie-Yamaguti algebra by
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.

Note that a Lie-Yamaguti algebra with $[x,y]=0$ for all $x,y\in\mathfrak{g}$
reduces to a Lie triple system, while a Lie-Yamaguti algebra with
$\left\llbracket x,y,z\right\rrbracket=0$ for all $x,y,z\in\mathfrak{g}$
reduces to a Lie algebra. The following example is taken from [24].

######  Example 2.2.

Let $M$ be a closed manifold 111a smooth compact manifold without boundary
with an affine connection, and denote by $\mathfrak{X}(M)$ the set of vector
fields on $M$. For all $x,y,z\in\mathfrak{X}(M)$, set

$\displaystyle[x,y]$ $\displaystyle=$ $\displaystyle-T(x,y),$
$\displaystyle\left\llbracket x,y,z\right\rrbracket$ $\displaystyle=$
$\displaystyle-R(x,y)z,$

where $T$ and $R$ are torsion tensor and curvature tensor respectively. It
turns out that the triple
$(\mathfrak{X}(M),[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
forms an (infinite-dimensional) Lie-Yamaguti algebra.

######  Definition 2.3.

[28] Let
$(\mathfrak{g},[\cdot,\cdot]_{\mathfrak{g}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{g}})$
and
$(\mathfrak{h},[\cdot,\cdot]_{\mathfrak{h}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{h}})$
be two Lie-Yamaguti algebras. A homomorphism from
$(\mathfrak{g},[\cdot,\cdot]_{\mathfrak{g}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{g}})$
to
$(\mathfrak{h},[\cdot,\cdot]_{\mathfrak{h}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{h}})$
is a linear map $\phi:\mathfrak{g}\to\mathfrak{h}$ preserving the Lie-Yamaguti
algebra structure, i.e., for all $x,y,z\in\mathfrak{g},$ the following
equalities hold

$\displaystyle\phi([x,y]_{\mathfrak{g}})$ $\displaystyle=$
$\displaystyle[\phi(x),\phi(y)]_{\mathfrak{h}},$
$\displaystyle~{}\phi(\left\llbracket x,y,z\right\rrbracket_{\mathfrak{g}})$
$\displaystyle=$
$\displaystyle\left\llbracket\phi(x),\phi(y),\phi(z)\right\rrbracket_{\mathfrak{h}}.$

The notion of representations of Lie-Yamaguti algebras was given in [32].

######  Definition 2.4.

Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra. A representation of $\mathfrak{g}$ is a vector
space $V$ endowed with a linear map $\rho:\mathfrak{g}\to\mathfrak{gl}(V)$ and
a bilinear map $\mu:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(V)$, which
satisfies the following conditions for all $x,y,z,w\in\mathfrak{g}$,

$\displaystyle\mu([x,y],z)-\mu(x,z)\rho(y)+\mu(y,z)\rho(x)=0,$
$\displaystyle\mu(x,[y,z])-\rho(y)\mu(x,z)+\rho(z)\mu(x,y)=0,$
$\displaystyle\rho(\left\llbracket
x,y,z\right\rrbracket)=[D_{\rho,\mu}(x,y),\rho(z)],$
$\displaystyle\mu(z,w)\mu(x,y)-\mu(y,w)\mu(x,z)-\mu(x,\left\llbracket
y,z,w\right\rrbracket)+D_{\rho,\mu}(y,z)\mu(x,w)=0,$
$\displaystyle\mu(\left\llbracket
x,y,z\right\rrbracket,w)+\mu(z,\left\llbracket
x,y,w\right\rrbracket)=[D_{\rho,\mu}(x,y),\mu(z,w)],$

where $D_{\rho,\mu}$ is given by

(1) $\displaystyle
D_{\rho,\mu}(x,y)=\mu(y,x)-\mu(x,y)+[\rho(x),\rho(y)]-\rho([x,y]),\quad\forall
x,y\in\mathfrak{g}.$

It is easy to see that $D_{\rho,\mu}$ is skew-symmetric. We denote a
representation of $\mathfrak{g}$ by $(V;\rho,\mu)$. In the sequel, we write
$D_{\rho,\mu}$ as $D$ for short without confusion.

The notion of representations of Lie-Yamaguti algebras is also a natural
generalization of Lie algebras and Lie triple systems. By a direct
computation, we have the following proposition.

######  Proposition 2.5.

If $(V;\rho,\mu)$ is a representation of a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
Then we have the following equalities

$\displaystyle D([x,y],z)+D([y,z],x)+D([z,x],y)=0;$ $\displaystyle
D(\left\llbracket x,y,z\right\rrbracket,w)+D(z,\left\llbracket
x,y,w\right\rrbracket)=[D(x,y),D(z,w)];$ $\displaystyle\mu(\left\llbracket
x,y,z\right\rrbracket,w)=\mu(x,w)\mu(z,y)-\mu(y,w)\mu(z,x)-\mu(z,w)D(x,y),$

for all $x,y,z,w\in\mathfrak{g}$.

######  Example 2.6.

Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra. We define linear maps
$\mathrm{ad}:\mathfrak{g}\to\mathfrak{gl}(\mathfrak{g})$ and
$\mathfrak{R}:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(\mathfrak{g})$ to be
$x\mapsto\mathrm{ad}_{x}$ and $(x,y)\mapsto\mathfrak{R}_{x,y}$ respectively,
where $\mathrm{ad}_{x}z=[x,z]$ and $\mathfrak{R}_{x,y}z=\left\llbracket
z,x,y\right\rrbracket$ for all $z\in\mathfrak{g}$. Then
$(\mathrm{ad},\mathfrak{R})$ forms a representation of $\mathfrak{g}$ on
itself, where $\mathfrak{L}:=D_{\mathrm{ad},\mathfrak{R}}$ is given by

$\mathfrak{L}_{x,y}z=\left\llbracket x,y,z\right\rrbracket,\quad\forall
z\in\mathfrak{g}.$

The representation $(\mathfrak{g};\mathrm{ad},\mathfrak{R})$ is called the
adjoint representation.

Representations of a Lie-Yamaguti algebra can be characterized by the
semidirect product Lie-Yamaguti algebras.

######  Proposition 2.7.

([34]) Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra and $V$ a vector space. Suppose that
$\rho:\mathfrak{g}\to\mathfrak{gl}(V)$ and
$\mu:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(V)$ are linear maps. Then
$(V;\rho,\mu)$ is a representation of
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
if and only if there is a Lie-Yamaguti algebra structure
$([\cdot,\cdot]_{\ltimes},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\ltimes})$
on the direct sum $\mathfrak{g}\oplus V$ which is defined to be

$\displaystyle[x+u,y+v]_{\ltimes}$ $\displaystyle=$
$\displaystyle[x,y]+\rho(x)v-\rho(y)u,$ $\displaystyle~{}\left\llbracket
x+u,y+v,z+w\right\rrbracket_{\ltimes}$ $\displaystyle=$
$\displaystyle\left\llbracket
x,y,z\right\rrbracket+D(x,y)w+\mu(y,z)u-\mu(x,z)v,$

for all $x,y,z\in\mathfrak{g},~{}u,v,w\in V$. This Lie-Yamaguti algebra
$(\mathfrak{g}\oplus
V,[\cdot,\cdot]_{\ltimes},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\ltimes})$
is called the semidirect product Lie-Yamaguti algebra, and is denoted by
$\mathfrak{g}\ltimes V$.

Cohomology theory on Lie-Yamaguti algebras was founded in [32]. Let
$(V;\rho,\mu)$ be a representation of a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$,
and denote the set of $p$-cochains by $C^{p}_{\rm
LieY}(\mathfrak{g},V)~{}(p\geqslant 1)$, where

$\displaystyle C^{n+1}_{\rm
LieY}(\mathfrak{g},V)\triangleq\begin{cases}\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n},V)\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n}\otimes\mathfrak{g},V),&\forall
n\geqslant 1,\\\ \mathrm{Hom}(\mathfrak{g},V),&n=0.\end{cases}$

The coboundary map $\delta:C^{p}_{\rm LieY}(\mathfrak{g},V)\longrightarrow
C^{p+1}_{\rm LieY}(\mathfrak{g},V)$ is given as follows:

  * •

If $n\geqslant 1$, for any $(f,g)\in C^{n+1}_{\rm LieY}(\mathfrak{g},V)$, the
coboundary map

$\delta=(\delta_{\rm I},\delta_{\rm II}):C^{n+1}_{\rm LieY}(\mathfrak{g},V)\to
C^{n+2}_{\rm LieY}(\mathfrak{g},V),$
$\qquad\qquad\qquad\qquad\qquad\quad(f,g)\mapsto\Big{(}\delta_{\rm
I}(f,g),\delta_{\rm II}(f,g)\Big{)},$

is given by

$\displaystyle\Big{(}\delta_{\rm
I}(f,g)\Big{)}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n+1})$ (2)
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}\rho(x_{n+1})g(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},y_{n+1})-\rho(y_{n+1})g(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},x_{n+1})$
$\displaystyle-g(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},[x_{n+1},y_{n+1}])\Big{)}$
$\displaystyle+\sum_{k=1}^{n}(-1)^{k+1}D_{\rho,\mu}(\mathfrak{X}_{k})f(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}_{k}},\cdots,\mathfrak{X}_{n+1})$
$\displaystyle+\sum_{1\leqslant k<l\leqslant
n+1}(-1)^{k}f(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}_{k}},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n+1}),$
$\displaystyle\Big{(}\delta_{\rm
II}(f,g)\Big{)}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n+1},z)$ (3)
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}\mu(y_{n+1},z)g(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},x_{n+1})-\mu(x_{n+1},z)g(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},y_{n+1})\Big{)}$
$\displaystyle+\sum_{k=1}^{n+1}(-1)^{k+1}D_{\rho,\mu}(\mathfrak{X}_{k})g(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}_{k}},\cdots,\mathfrak{X}_{n+1},z)$
$\displaystyle+\sum_{1\leqslant k<l\leqslant
n+1}(-1)^{k}g(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}_{k}},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n+1},z)$
$\displaystyle+\sum_{k=1}^{n+1}(-1)^{k}g(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}_{k}},\cdots,\mathfrak{X}_{n+1},\left\llbracket
x_{k},y_{k},z\right\rrbracket),$

where $\mathfrak{X}_{i}=x_{i}\wedge
y_{i}\in\wedge^{2}\mathfrak{g}~{}(i=1,\cdots,n+1),~{}z\in\mathfrak{g}$, and
the notation $\mathfrak{X}_{k}\circ\mathfrak{X}_{l}$ means that

$\mathfrak{X}_{k}\circ\mathfrak{X}_{l}:=\left\llbracket
x_{k},y_{k},x_{l}\right\rrbracket\wedge y_{l}+x_{l}\wedge\left\llbracket
x_{k},y_{k},y_{l}\right\rrbracket.$

  * •

If $n=0$, for any element $f\in C^{1}_{\rm LieY}(\mathfrak{g},V)$, the
coboundary map

$\delta:C^{1}_{\rm LieY}(\mathfrak{g},V)\to C^{2}_{\rm LieY}(\mathfrak{g},V),$
$\qquad\qquad\qquad f\mapsto\Big{(}\delta_{I}(f),\delta_{II}(f)\Big{)},$

is given by

(4) $\displaystyle\Big{(}\delta_{\rm I}(f)\Big{)}(x,y)$ $\displaystyle=$
$\displaystyle\rho(x)f(y)-\rho(y)f(x)-f([x,y]),$ (5)
$\displaystyle~{}\Big{(}\delta_{\rm II}(f)\Big{)}(x,y,z)$ $\displaystyle=$
$\displaystyle
D_{\rho,\mu}(x,y)f(z)+\mu(y,z)f(x)-\mu(x,z)f(y)-f(\left\llbracket
x,y,z\right\rrbracket),\quad\forall x,y,z\in\mathfrak{g}.$

######  Proposition 2.8.

([32]) With the notations above, for any $f\in C^{1}_{\rm
LieY}(\mathfrak{g},V)$, we have

$\displaystyle\delta_{\rm I}\Big{(}\delta_{\rm I}(f)),\delta_{\rm
II}(f)\Big{)}=0\quad{\rm and}\quad\delta_{\rm II}\Big{(}\delta_{\rm
I}(f)),\delta_{\rm II}(f)\Big{)}=0.$

Moreover, for all $(f,g)\in C^{p}_{\rm LieY}(\mathfrak{g},V)~{}(p\geqslant
2)$, we have

$\displaystyle\delta_{\rm I}\Big{(}\delta_{\rm I}(f,g)),\delta_{\rm
II}(f,g)\Big{)}=0\quad{\rm and}\quad\delta_{\rm II}\Big{(}\delta_{\rm
I}(f,g)),\delta_{\rm II}(f,g)\Big{)}=0.$

In some literature, this conclusion is denote by $\delta\circ\delta=0.$

##  3\. Cohomology and Linear deformations of $\mathsf{LieYRep}$ pairs

In this section, we establish the cohomology theory of $\mathsf{LieYRep}$
pairs, and use this type of cohomology to classify linear deformations of
$\mathsf{LieYRep}$ pairs.

###  3.1. Cohomology of $\mathsf{LieYRep}$ pairs

Recall that in this paper, we use a terminology $\mathsf{LieYRep}$ pair to
denote a pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$,
where
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
is a Lie-Yamaguti algebra, and $(V;\rho,\mu)$ is a representation of
$\mathfrak{g}$. We also denote a $\mathsf{LieYRep}$ pair by
$(\mathfrak{g},\rho,\mu)$ without ambiguity.

For the moment, let $\mathfrak{g}$ and $V$ be vector spaces, where elements in
$\mathfrak{g}$ are denoted by $x,y,z,x_{i},y_{i}$ and those in $V$ by
$u,v,w,u_{i},v_{i}$. Denote by
${\mathcal{C}}^{\bullet}(\mathfrak{g},\rho,\mu)=\oplus_{n=0}^{\infty}{\mathcal{C}}^{n}(\mathfrak{g},\rho,\mu)$,
where ${\mathcal{C}}^{0}(\mathfrak{g},\rho,\mu)$ is defined to be $0$,
${\mathcal{C}}^{1}(\mathfrak{g},\rho,\mu)$ is defined to be
$\mathfrak{gl}(\mathfrak{g})\times\mathfrak{gl}(V)$, and for $n\geqslant 1$,

$\displaystyle{\mathcal{C}}^{n+1}(\mathfrak{g},\rho,\mu)$ $\displaystyle:=$
$\displaystyle\Big{(}\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n},\mathfrak{g})\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n}\otimes\mathfrak{g},\mathfrak{g})\Big{)}$
$\displaystyle\oplus\Big{(}\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)}_{n},V)\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n}\otimes
V,V)\Big{)}$
$\displaystyle\oplus\Big{(}\oplus_{i=1}^{n-1}\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)\otimes\cdots\wedge^{2}\mathfrak{g}}_{n},V)\times\Big{(}\oplus_{j=1}^{n}\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)\otimes\cdots\wedge^{2}\mathfrak{g}}_{n}\otimes\mathfrak{g},V)\Big{)}\Big{)},$

where $\mathfrak{g}\wedge V$ occurring in the third term is at the $i$-th
position.

Assume that $n\geqslant 1$ now, and
$(f,g)\in{\mathcal{C}}^{n+1}(\mathfrak{g},\rho,\mu)$.

  * (a)

For a pair of given maps

$(f,g)\in\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n},\mathfrak{g})\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n}\otimes\mathfrak{g},\mathfrak{g}),$

we define a pair of linear maps $(\hat{f},\hat{g})\in C_{\rm
LieY}^{n+1}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus V)$ to be

$\displaystyle\hat{f}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n})\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}f(x_{1}\wedge y_{1},\cdots,x_{n}\wedge
y_{n}),0\Big{)},$
$\displaystyle\hat{g}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n}),(x_{n+1},u_{n+1})\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}g(x_{1}\wedge y_{1},\cdots,x_{n}\wedge
y_{n},x_{n+1}),0\Big{)}.$

  * (b)

For a pair of given maps

$(f,g)\in\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)}_{n},V)\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n}\otimes
V,V),$

we define a pair of linear maps $(\hat{f},\hat{g})\in C_{\rm
LieY}^{n+1}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus V)$ to be

$\displaystyle\hat{f}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n})\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}0,f(x_{1}\wedge
y_{1},\cdots,x_{n-1}\wedge y_{n-1},(x_{n},v_{n}))-f(x_{1}\wedge
y_{1},\cdots,x_{n-1}\wedge y_{n-1},(y_{n},u_{n}))\Big{)},$
$\displaystyle\hat{g}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n}),(z,w)\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}0,g(x_{1}\wedge y_{1},\cdots,x_{n}\wedge
y_{n},w)-g(x_{1}\wedge y_{1},\cdots,z\wedge y_{n},u_{n})+g(x_{1}\wedge
y_{1},\cdots,z\wedge x_{n},v_{n})\Big{)}.$

  * (c)

For a pair of given maps

$(f,g)\in\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)\otimes\cdots\otimes\wedge^{2}\mathfrak{g}}_{n},V)\times\mathrm{Hom}(\underbrace{\wedge^{2}\mathfrak{g}\otimes\cdots\otimes(\mathfrak{g}\wedge
V)\otimes\cdots\wedge^{2}\mathfrak{g}}_{n}\otimes\mathfrak{g},V),$

we define a pair of linear maps $(\hat{f},\hat{g})\in C_{\rm
LieY}^{n+1}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus V)$ to be

$\displaystyle\hat{f}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n})\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}0,f(x_{1}\wedge
y_{1},\cdots,(x_{i},v_{i})\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n}))-f(x_{1}\wedge
y_{1},\cdots,(y_{i},u_{i}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n}))\Big{)},$
$\displaystyle\hat{g}\Big{(}(x_{1},u_{1})\wedge(y_{1},v_{1}),\cdots,(x_{n},u_{n})\wedge(y_{n},v_{n}),(z,w)\Big{)}$
$\displaystyle=$ $\displaystyle\Big{(}0,g(x_{1}\wedge
y_{1},\cdots,(x_{i},v_{i}),\cdots,x_{n}\wedge y_{n},z)-g(x_{1}\wedge
y_{1},\cdots,(y_{i},u_{i}),\cdots,x_{n}\wedge y_{n},z)\Big{)}.$

Moreover, for a given linear map
$f=(f_{1},f_{2})\in{\mathcal{C}}^{1}(\mathfrak{g},\rho,\mu)=\mathrm{Hom}(\mathfrak{g},\mathfrak{g})\times\mathrm{Hom}(V,V)$,
we define a linear map $\hat{f}\in C_{\rm LieY}^{1}(\mathfrak{g}\oplus
V,\mathfrak{g}\oplus V)$ to be

$\displaystyle\hat{f}(x,u)=(f_{1}(x),f_{2}(u)),\quad\forall(x,u)\in\mathfrak{g}\oplus
V.$

A pair of linear maps $(\hat{f},\hat{g})$ (resp. $\hat{f}$) defined above is
called the lift of $(f,g)$ (resp. $f$). It is obvious that
${\mathcal{C}}^{n}(\mathfrak{g},\rho,\mu)$ is a subspace of $C^{n}_{\rm
LieY}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus V)$ via the lift map.

Let $(\mathfrak{g},\rho,\mu)$ be a $\mathsf{LieYRep}$ pair. For
$(f,g)=\Big{(}(f_{1},g_{1}),(f_{2},g_{2}),(f_{3},g_{3})\Big{)}\in{\mathcal{C}}^{n+1}(\mathfrak{g},\rho,\mu)~{}(n\geqslant
1)$, write the coboundary map

$\Delta:{\mathcal{C}}^{n+1}(\mathfrak{g},\rho,\mu)\to{\mathcal{C}}^{n+2}(\mathfrak{g},\rho,\mu)$

as

$\Delta(f,g)=\Big{(}\big{(}\Delta(f,g)\big{)}_{1},\big{(}\Delta(f,g)\big{)}_{2},\big{(}\Delta(f,g)\big{)}_{3}\Big{)},$

where

  * •

$\Big{(}\Delta(f,g)\Big{)}_{1}=\delta_{\mathsf{LieY}}(f_{1},g_{1})$ is the
coboundary map given in (2) and (3) with coefficients in the adjoint
representation.

  * •

$\Big{(}\Delta(f,g)\Big{)}_{2}$ is defined to be

$\displaystyle\Big{(}\Delta_{\rm
I}(f,g)\Big{)}_{2}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},(x,v))$
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}\rho(x)g_{2}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},v)+\rho(g_{1}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},x))v-g_{2}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},\rho(x)v)\Big{)}$
$\displaystyle+\sum_{k=1}^{n}(-1)^{k+1}D(x_{k},y_{k})f_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n},(x,v))$
$\displaystyle+\sum_{1\leqslant k<l\leqslant
n}(-1)^{k}f_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n},(x,v))$
$\displaystyle+\sum_{1\leqslant
k<n+1}(-1)^{k}f_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n},(\left\llbracket
x_{k},y_{k},x\right\rrbracket,v)+(x,D(x_{k},y_{k})v)),$
$\displaystyle\Big{(}\Delta_{\rm
II}(f,g)\Big{)}_{2}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n+1},v)$
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}D(g_{1}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},x_{n+1}),y_{n+1})v-D(g_{1}(\mathfrak{X}_{1},\cdots,\mathfrak{X}_{n},y_{n+1}),x_{n+1})v\Big{)}$
$\displaystyle+\sum_{k=1}^{n+1}(-1)^{k+1}D(x_{k},y_{k})g_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n+1},v)$
$\displaystyle+\sum_{1\leqslant k<l\leqslant
n+1}(-1)^{k}g_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n+1},v)$
$\displaystyle+\sum_{k=1}^{n+1}(-1)^{k}g_{2}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n+1},D(x_{k},y_{k})v).$

  * •

$\Big{(}\Delta(f,g)\Big{)}_{3}$ is defined to be

$\displaystyle\Big{(}\Delta_{\rm
I}(f,g)\Big{)}_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n+1})$
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}\rho(x_{n+1})g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n},y_{n+1})-\rho(y_{n+1})g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n},x_{n+1})$
$\displaystyle-
g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,[x_{n+1},y_{n+1}]\Big{)}$
$\displaystyle+\sum_{k=1,\atop k\neq
i}^{n}(-1)^{k+1}D(x_{k},y_{k})f_{3}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n+1})$
$\displaystyle+\sum_{k<l\leqslant n+1,\atop k,l\neq
i}(-1)^{k}f_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n+1})$
$\displaystyle+(-1)^{i}\mu(x_{i},f_{1}(\mathfrak{X}_{1},\cdots,\hat{(x_{i},v)},\cdots,\mathfrak{X}_{n+1}))v$
$\displaystyle+\sum_{k>i}(-1)^{i}f_{3}(\mathfrak{X}_{1},\cdots,\hat{(x_{i},v)},\cdots,(x_{k},\mu(x_{i},y_{k})v)-(y_{k},\mu(x_{i},x_{k})v),\cdots,\mathfrak{X}_{n+1})$
$\displaystyle+\sum_{k<i}(-1)^{k}f_{3}(\mathfrak{X}_{1},\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{i-1},(\left\llbracket
x_{k},y_{k},x_{i}\right\rrbracket,v)+(x_{i},D(x_{k},y_{k})v),\mathfrak{X}_{i+1},\cdots,\mathfrak{X}_{n+1}),$
$\displaystyle\Big{(}\Delta_{\rm
II}(f,g)\Big{)}_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n+1},x)$
$\displaystyle=$
$\displaystyle(-1)^{n}\Big{(}\mu(y_{n+1},x)g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n+1},x_{n+1})-\mu(x_{n+1},x)g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\mathfrak{X}_{n+1},y_{n+1})\Big{)}$
$\displaystyle+\sum_{k=1,\atop k\neq
i}^{n+1}(-1)^{k+1}D(x_{k},y_{k})g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n+1},x)$
$\displaystyle+\sum_{k<l\leqslant n+1,\atop k,l\neq
i}(-1)^{k}g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{k}\circ\mathfrak{X}_{l},\cdots,\mathfrak{X}_{n+1},x)$
$\displaystyle+\sum_{k=1}^{n+1}g_{3}(\mathfrak{X}_{1},\cdots,(x_{i},v),\cdots,\hat{\mathfrak{X}}_{k},\cdots,\mathfrak{X}_{n+1},\left\llbracket
x_{k},y_{k},x\right\rrbracket)$
$\displaystyle+(-1)^{i}\Big{(}\mu(x_{i},g_{1}(\mathfrak{X}_{1},\cdots,\hat{(x_{i},v)},\cdots,\mathfrak{X}_{n+1},x))v-g_{2}(\mathfrak{X}_{1},\cdots,\hat{(x_{i},v)},\cdots,\mathfrak{X}_{n+1},\mu(x_{i},x)v)$
$\displaystyle\quad+g_{3}(\mathfrak{X}_{1},\cdots,\hat{(x_{i},v)},\cdots,(y_{l},\mu(x_{i},x_{l})v)-(x_{l},\mu(x_{i},y_{l})v),\cdots,\mathfrak{X}_{n+1},x)\Big{)}.$

For a pair of given maps
$f=\Big{(}f_{1},f_{2}\Big{)}\in{\mathcal{C}}^{1}(\mathfrak{g},\rho,\mu)=\mathrm{Hom}(\mathfrak{g},\mathfrak{g})\times\mathrm{Hom}(V,V)$,
the coboundary map

$\Delta:{\mathcal{C}}^{1}(\mathfrak{g},\rho,\mu)\to{\mathcal{C}}^{2}(\mathfrak{g},\rho,\mu)$

is written as $\Big{(}\Delta(f)_{1},\Delta(f)_{2},\Delta(f)_{3}\Big{)}$, where

  * •

$\Big{(}\Delta(f)\Big{)}_{1}=\delta_{\mathsf{LieY}}f_{1}$ is the coboundary
map given in (4) and (5) with coefficients in the adjoint representation.

  * •

$\Big{(}\Delta(f)\Big{)}_{2}$ is given as follows

$\displaystyle\Big{(}\Delta_{\rm I}(f)\Big{)}_{2}(x,v)$ $\displaystyle=$
$\displaystyle\rho(x)f_{2}(v)+\rho(f_{1}(x))v-f_{2}(\rho(x)v),$
$\displaystyle\Big{(}\Delta_{\rm II}(f)\Big{)}_{2}(x,y,v)$ $\displaystyle=$
$\displaystyle D(x,y)f_{2}(v)+D(f_{1}(x),y)v+D(x,f_{1}(y))v-f_{2}(D(x,y)v).$

  * •

$\Big{(}\Delta(f)\Big{)}_{3}$ is given as follows

$\displaystyle\Big{(}\Delta_{\rm I}(f)\Big{)}_{3}(x,v)$ $\displaystyle=$
$\displaystyle\rho(x)f_{2}(v)+\rho(f_{1}(x))v-f_{2}(\rho(x)v),$
$\displaystyle\Big{(}\Delta_{\rm II}(f)\Big{)}_{3}(v,x,y)$ $\displaystyle=$
$\displaystyle\mu(x,y)f_{2}(v)+\mu(f_{1}(x),y)v+\mu(x,f_{1}(y))v-f_{2}(\mu(x,y)v).$

######  Theorem 3.1.

With the above notations, we have $\Delta\circ\Delta=0$. Thus we obtain a
well-defined complex
$({\mathcal{C}}^{\bullet}(\mathfrak{g},\rho,\mu),\Delta)$.

_Sketch of Proof._ Consider the semidirect product Lie-Yamaguti algebra
$(\mathfrak{g}\ltimes
V,[\cdot,\cdot]_{\ltimes},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\ltimes})$,
and take its corresponding cohomology
$(C^{\bullet}_{\mathsf{LieY}}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus
V),\delta)$ with coefficients in the adjoint representation. It is easy to see
that $({\mathcal{C}}^{\bullet}(\mathfrak{g},\rho,\mu),\Delta)$ is a subcomplex
of $(C^{\bullet}_{\mathsf{LieY}}(\mathfrak{g}\oplus V,\mathfrak{g}\oplus
V),\delta)$. Thus for any $(f,g)\in{\mathcal{C}}^{n}(\mathfrak{g},\rho,\mu)$,
we have that
$(\Delta\circ\Delta)(f,g)_{1}=(\Delta\circ\Delta)(f,g)_{2}=(\Delta\circ\Delta)(f,g)_{3}=0$.
The conclusion thus follows. ∎

######  Definition 3.2.

Let $(\mathfrak{g},\rho,\mu)$ be a $\mathsf{LieYRep}$ pair. We call the
cochain complex $({\mathcal{C}}^{\bullet}(\mathfrak{g},\rho,\mu),\Delta)$ the
cohomology of the $\mathsf{LieYRep}$ pair $(\mathfrak{g},\rho,\mu)$. The
corresponding $n$-th cohomology group is denoted by

$\mathcal{H}^{n}(\mathfrak{g},\rho,\mu)=\mathcal{Z}^{n}(\mathfrak{g},\rho,\mu)/\mathcal{B}^{n}(\mathfrak{g},\rho,\mu),$

where $\mathcal{Z}^{n}(\mathfrak{g},\rho,\mu)$ and
$\mathcal{B}^{n}(\mathfrak{g},\rho,\mu)$ are sets of $n$-cocycles and
$n$-coboundaries respectively.

###  3.2. Linear deformations of $\mathsf{LieYRep}$ pairs

In this subsection, we characterize linear deformations of $\mathsf{LieYRep}$
pairs by the cohomology theory established in the previous subsection. The
following definition is standard.

######  Definition 3.3.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot]{{}_{\mathfrak{g}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{g}}),(V;\rho_{V},\mu_{V})\Big{)}$
and
$\Big{(}(\mathfrak{h},[\cdot,\cdot]{{}_{\mathfrak{h}}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\mathfrak{h}}),(W;\rho_{W},\mu_{W})\Big{)}$
be two $\mathsf{LieYRep}$ pairs. A homomorphism from
$(\mathfrak{g},\rho_{V},\mu_{V})$ to $(\mathfrak{h},\rho_{W},\mu_{W})$ is a
pair $(\phi,\varphi)$, where $\phi:\mathfrak{g}\to\mathfrak{h}$ is a Lie-
Yamaguti algebra homomorphism and $\varphi:V\to W$ is a homomorphism of
representations from $(V;\rho,\mu)$ to $(W;\varrho,\varpi)$, i.e.,

(6) $\displaystyle\varphi\Big{(}\rho_{V}(x)v\Big{)}$ $\displaystyle=$
$\displaystyle\rho_{W}\Big{(}\phi(x)\Big{)}\big{(}\varphi(v)\big{)},$ (7)
$\displaystyle\varphi\Big{(}\mu_{V}(x,y)v\Big{)}$ $\displaystyle=$
$\displaystyle\mu_{W}\Big{(}\phi(x),\phi(y)\Big{)}\big{(}\varphi(v)\big{)},\quad\forall
x,y\in\mathfrak{g},~{}v\in V.$

By Eqs. (1), (6), and (7), we deduce that

(8) $\displaystyle\varphi\Big{(}D_{V}(x,y)v\Big{)}$ $\displaystyle=$
$\displaystyle
D_{W}\Big{(}\phi(x),\phi(y)\Big{)}\big{(}\varphi(v)\big{)},\quad\forall
x,y\in\mathfrak{g},~{}v\in V,$

where $D_{V}=D_{\rho_{V},\mu_{V}}$ and $D_{W}=D_{\rho_{W},\mu_{W}}$.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair. Let
$\phi:\wedge^{2}\mathfrak{g}\to\mathfrak{g},~{}\varphi_{i}:\wedge^{2}\mathfrak{g}\otimes\mathfrak{g}\to\mathfrak{g}$
and
$\varrho:\mathfrak{g}\to\mathfrak{gl}(V),~{}\varpi_{i}:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(V)~{}(i=1,2)$
be linear maps. Consider a $t$-parameterized family of bracket operations and
linear maps:

$\displaystyle[x,y]_{t}$ $\displaystyle=$ $\displaystyle[x,y]+t\phi(x,y),$
$\displaystyle~{}\left\llbracket x,y,z\right\rrbracket_{t}$ $\displaystyle=$
$\displaystyle\left\llbracket
x,y,z\right\rrbracket+t\varphi_{1}(x,y,z)+t^{2}\varphi_{2}(x,y,z),$
$\displaystyle\rho_{t}(x)$ $\displaystyle=$
$\displaystyle\rho(x)+t\varrho(x),$ $\displaystyle\mu_{t}(x,y)$
$\displaystyle=$ $\displaystyle\mu(x,y)+t\varpi_{1}(x,y)+t^{2}\varpi_{2}(x,y)$

By (1), we have

$D_{t}:=D_{\rho_{t},\mu_{t}}=D+t\mathcal{D}_{1}+t^{2}\mathcal{D}_{2}$

where

$\displaystyle\mathcal{D}_{1}(x,y)$ $\displaystyle=$
$\displaystyle\varpi_{1}(y,x)-\varpi_{1}(x,y)+[\rho(x),\varrho(y)]+[\varrho(x),\rho(y)]-\varrho([x,y]),$
$\displaystyle~{}\mathcal{D}_{2}(x,y)$ $\displaystyle=$
$\displaystyle\varpi_{2}(y,x)-\varpi_{2}(x,y)+[\varrho(x),\varrho(y)].$

It is easy to see that both $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ are skew-
symmetric, and so is $D_{t}$.

######  Definition 3.4.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair. With the above notations, if
$\Big{(}(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}),(V,\rho_{t},\mu_{t})\Big{)}$
are a family of $\mathsf{LieYRep}$ pairs for all $t$, we say that
$(\phi,\varphi_{i};\varrho,\varpi_{i})~{}(i=1,2)$ generates a linear
deformation of the $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.

######  Proposition 3.5.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair. If a collection of linear maps
$(\phi,\varphi_{i};\varrho,\varpi_{i})~{}(i=1,2)$ generates a linear
deformation of the $\mathsf{LieYRep}$ pair $(\mathfrak{g},\rho,\mu)$, then we
have

  * (i)

$\Big{(}(\phi,\varphi_{1}),(\varrho,\mathcal{D}_{1}),(0,0)\Big{)}\in{\mathcal{C}}^{2}(\mathfrak{g},\rho,\mu)$
is a $2$-cocycle of $\Delta$, i.e.,

$\Delta\Big{(}(\phi,\varphi_{1}),(\varrho,\mathcal{D}_{1}),(0,0)\Big{)}=0;$

  * (ii)

$\Big{(}(\mathfrak{g},\phi,\varphi_{2}),(V;\varrho,\varpi_{2})\Big{)}$ is a
$\mathsf{LieYRep}$ pair.

###### Proof.

The computation is straightforward, so we omit the details. ∎

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair, and suppose that
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
and
$(\mathfrak{g},[\cdot,\cdot]_{t}^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}^{\prime};\rho_{t}^{\prime},\mu_{t}^{\prime})$
are two linear deformations of
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
In the following, we denote $D_{\rho_{t},\mu_{t}}$ and
$D_{\rho_{t}^{\prime},\mu_{t}^{\prime}}$ by $D_{t}$ and $D_{t}^{\prime}$
respectively.

######  Definition 3.6.

  * (i)

Two linear deformations
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
and
$(\mathfrak{g},[\cdot,\cdot]_{t}^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}^{\prime};\rho_{t}^{\prime},\mu_{t}^{\prime})$
of $(\mathfrak{g},\rho,\mu)$ are called equivalent if there exists an
isomorphism $({{\rm{Id}}}_{\mathfrak{g}}+tN,{{\rm{Id}}}_{V}+tS)$ from
$(\mathfrak{g},[\cdot,\cdot]_{t}^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}^{\prime};\rho_{t}^{\prime},\mu_{t}^{\prime})$
to
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$.

  * (ii)

A linear deformation
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
is said to be trivial if it is equivalent to
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket;\rho,\mu)$
itself.

Suppose that two deformations
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
and
$(\mathfrak{g},[\cdot,\cdot]_{t}^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}^{\prime};\rho_{t}^{\prime},\mu_{t}^{\prime})$
are equivalent, for
$(N,S)\in{\mathcal{C}}^{1}(\mathfrak{g},\rho,\mu)=\mathfrak{gl}(\mathfrak{g})\times\mathfrak{gl}(V)$,
by Definition 3.3 and Definition 3.6, we obtain that

(9) $\displaystyle({{\rm{Id}}}_{\mathfrak{g}}+tN)[x,y]_{t}^{\prime}$
$\displaystyle=$
$\displaystyle[({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y]_{t},$
(10) $\displaystyle({{\rm{Id}}}_{\mathfrak{g}}+tN)\left\llbracket
x,y,z\right\rrbracket_{t}^{\prime}$ $\displaystyle=$
$\displaystyle\left\llbracket({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y,({{\rm{Id}}}_{\mathfrak{g}}+tN)z\right\rrbracket_{t},$
(11) $\displaystyle({{\rm{Id}}}_{V}+tS)\circ\rho_{t}^{\prime}(x)$
$\displaystyle=$
$\displaystyle\rho_{t}\Big{(}({{\rm{Id}}}_{\mathfrak{g}}+tN)x\Big{)}\circ({{\rm{Id}}}_{V}+tS),$
(12) $\displaystyle({{\rm{Id}}}_{V}+tS)\circ\mu_{t}^{\prime}(x,y)$
$\displaystyle=$
$\displaystyle\mu_{t}\Big{(}({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y\Big{)}\circ({{\rm{Id}}}_{V}+tS),$
(13) $\displaystyle({{\rm{Id}}}_{V}+tS)\circ D_{t}^{\prime}(x,y)$
$\displaystyle=$ $\displaystyle
D_{t}\Big{(}({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y\Big{)}\circ({{\rm{Id}}}_{V}+tS),$

which implies that

$\displaystyle\phi^{\prime}(x,y)-\phi(x,y)$ $\displaystyle=$
$\displaystyle[Nx,y]+[x,Ny]-N[x,y]=\Big{(}\Delta_{\rm
I}(N,S)\Big{)}_{1}(x,y),$
$\displaystyle\varphi_{1}^{\prime}(x,y,z)-\varphi_{1}(x,y,z)$ $\displaystyle=$
$\displaystyle\left\llbracket Nx,y,z\right\rrbracket+\left\llbracket
x,Ny,z\right\rrbracket+\left\llbracket x,y,Nz\right\rrbracket-N\left\llbracket
x,y,z\right\rrbracket=\Big{(}\Delta_{\rm II}(N,S)\Big{)}_{1}(x,y,z),$
$\displaystyle\varrho^{\prime}(x)v-\varrho(x)v$ $\displaystyle=$
$\displaystyle\rho(x)(Sv)+\rho(Nx)v-S(\rho(x)v)=\Big{(}\Delta_{\rm
I}(N,S)\Big{)}_{2}(x,v),$
$\displaystyle\varpi_{1}^{\prime}(x,y)v-\varpi_{1}(x,y)v$ $\displaystyle=$
$\displaystyle\mu(x,Ny)v+\mu(Nx,y)v+\mu(x,y)(Sv)-S(\mu(x,y)v)=\Big{(}\Delta_{\rm
II}(N,S)\Big{)}_{2}(x,y,v),$
$\displaystyle\mathcal{D}_{1}^{\prime}(x,y)v-\mathcal{D}_{1}(x,y)v$
$\displaystyle=$ $\displaystyle
D(x,Ny)v+D(Nx,y)v+D(x,y)(Sv)-S(D(x,y)v)=\Big{(}\Delta_{\rm
II}(N,S)\Big{)}_{3}(v,x,y).$

Thus, we have the following theorem.

######  Theorem 3.7.

If two linear deformations
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
and
$(\mathfrak{g},[\cdot,\cdot]_{t}^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}^{\prime};\rho_{t}^{\prime},\mu_{t}^{\prime})$
of a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
are equivalent, which are generated by $(\phi,\varphi_{i};\varrho,\varpi_{i})$
and
$(\phi^{\prime},\varphi_{i}^{\prime};\varrho^{\prime},\varpi_{i}^{\prime})$
respectively. Then $(\phi+\varrho,\varphi_{1}+\varpi_{1})$ and
$(\phi^{\prime}+\varrho^{\prime},\varphi_{1}^{\prime}+\varpi_{1}^{\prime})$
are in the same cohomology class of $\mathcal{H}^{2}(\mathfrak{g},\rho,\mu)$.

By a straightforward computation, a linear deformation
$(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t};\rho_{t},\mu_{t})$
of a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
is trivial if and only if the following equalities hold:

(14) $\displaystyle\begin{cases}\phi(x,y)&=[Nx,y]+[x,Ny]-N[x,y],\\\
~{}N\phi(x,y)&=[Nx,Ny],\\\ \varphi_{1}(x,y,z)&=\left\llbracket
Nx,y,z\right\rrbracket+\left\llbracket x,Ny,z\right\rrbracket+\left\llbracket
x,y,Nz\right\rrbracket-N\left\llbracket x,y,z\right\rrbracket,\\\
\varphi_{2}(x,y,z)&=\left\llbracket Nx,Ny,z\right\rrbracket+\left\llbracket
x,Ny,Nz\right\rrbracket+\left\llbracket
Nx,y,Nz\right\rrbracket-N\varphi_{1}(x,y,z),\\\
N\varphi_{2}(x,y,z)&=\left\llbracket Nx,Ny,Nz\right\rrbracket,\\\ \end{cases}$
(15) $\displaystyle\begin{cases}\varrho(x)&=\rho(Nx)+\rho(x)\circ
S-S\circ\rho(x),\\\ \rho(Nx)\circ S&=S\circ\varrho(x),\\\
\varpi_{1}(x,y)&=\mu(Nx,y)+\mu(x,Ny)+\mu(x,y)\circ S-S\circ\mu(x,y),\\\
\varpi_{2}(x,y)&=\mu(Nx,y)\circ S+\mu(x,Ny)\circ
S+\mu(Nx,Ny)-S\circ\varpi_{1}(x,y),\\\ ~{}\mu(Nx,Ny)\circ
S&=S\circ\varpi_{2}(x,y).\end{cases}$

######  Definition 3.8.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair, $N\in\mathfrak{gl}(\mathfrak{g})$ and
$S\in\mathfrak{gl}(V)$ linear maps on $\mathfrak{g}$ and $V$ respectively. If
Eqs. (14)-(15) hold, then $(N,S)$ is called a Nijenhuis structure on
$(\mathfrak{g},\rho,\mu)$.

######  Remark 3.9.

Note that (14) means that $N$ is a Nijenhuis operator on the Lie-Yamaguti
algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$,
where $\phi:=[\cdot,\cdot]_{N}$ and
$\varphi_{2}:=\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N}$ are
deformed brackets. Besides,
$([\cdot,\cdot]{{}_{N}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$
is a Lie-Yamaguti algebra structure on $\mathfrak{g}$ and thus $N$ is a
homomorphism from Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot]{{}_{N}},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$
to
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
See [26] for more details of Nijenhuis operators on Lie-Yamaguti algebras.

Note that Eqs. (15) is equivalent to the following equalities:

(16) $\displaystyle\rho(Nx)(Sv)$ $\displaystyle=$ $\displaystyle
S\Big{(}\rho(x)(Sv)-\rho(Nx)v\Big{)}-S^{2}(\rho(x)v),$ (17)
$\displaystyle\mu(Nx,Ny)(Sv)$ $\displaystyle=$ $\displaystyle
S\Big{(}\mu(Nx,y)(Sv)+\mu(x,Ny)(Sv)+\mu(Nx,Ny)v\Big{)}$
$\displaystyle-S^{2}\Big{(}\mu(Nx,y)v+\mu(x,Ny)v+\mu(x,y)(Sv)\Big{)}+S^{3}\Big{(}\mu(x,y)v\Big{)}.$

######  Lemma 3.10.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then we have

(18) $\displaystyle D(Nx,Ny)(Sv)$ $\displaystyle=$ $\displaystyle
S\Big{(}D(Nx,y)(Sv)+D(x,Ny)(Sv)+D(Nx,Ny)v\Big{)}$
$\displaystyle-S^{2}\Big{(}D(Nx,y)v+D(x,Ny)v+D(x,y)(Sv)\Big{)}+S^{3}\Big{(}D(x,y)v\Big{)}.$

###### Proof.

The proof only involves computation, so we omit the details. ∎

######  Example 3.11.

Let $N$ be a Nijenhuis operator on a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
Then the pair $(N,N)$ is a Nijenhuis structure on the $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(\mathfrak{g};\mathrm{ad},\mathfrak{R})\Big{)}$.

Obviously, a trivial deformation of a $\mathsf{LieYRep}$ pair gives rise to a
Nijenhuis structure. Furthermore, the converse is also valid. First, we need a
lemma.

######  Lemma 3.12.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair, and $\mathfrak{h}$ and $W$ vector spaces, where
$\mathfrak{h}$ is endowed with a pair of two operations
$([\cdot,\cdot]^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{\prime})$
and $W$ is endowed with two linear maps
$\varrho:W\to\mathfrak{gl}(\mathfrak{h}),~{}\varpi:\otimes^{2}W\to\mathfrak{gl}(\mathfrak{h})$.
If there exist two isomorphisms between vector spaces
$f:\mathfrak{h}\to\mathfrak{g}$ and $g:W\to V$, such that

$\displaystyle f([x,y]^{\prime})$ $\displaystyle=$ $\displaystyle[f(x),f(y)],$
$\displaystyle~{}f(\left\llbracket x,y,z\right\rrbracket^{\prime})$
$\displaystyle=$ $\displaystyle\left\llbracket
f(x),f(y),f(z)\right\rrbracket,$ $\displaystyle~{}g(\varrho(x)w)$
$\displaystyle=$ $\displaystyle\rho(f(x))(g(w)),$
$\displaystyle~{}g(\varpi(x,y)w)$ $\displaystyle=$
$\displaystyle\mu(f(x),f(y))(g(w)),\quad\forall x,y,z\in\mathfrak{h},w\in W.$

then
$\Big{(}(\mathfrak{h},[\cdot,\cdot]^{\prime},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{\prime}),(W;\varrho,\varpi)\Big{)}$
is a $\mathsf{LieYRep}$ pair

###### Proof.

It is a direct computation, so we omit the details. ∎

######  Theorem 3.13.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then a linear deformation can be obtained by setting for any
$x,y,z\in\mathfrak{g}$

(19) $\displaystyle\begin{cases}\phi(x,y)&=[Nx,y]+[x,Ny]-N[x,y],\\\
\varphi_{1}(x,y,z)&=\left\llbracket Nx,y,z\right\rrbracket+\left\llbracket
x,Ny,z\right\rrbracket+\left\llbracket x,y,Nz\right\rrbracket-N\left\llbracket
x,y,z\right\rrbracket,\\\ \varphi_{2}(x,y,z)&=\left\llbracket
Nx,Ny,z\right\rrbracket+\left\llbracket
x,Ny,Nz\right\rrbracket+\left\llbracket
Nx,y,Nz\right\rrbracket-N\varphi_{1}(x,y,z),\\\
\varrho(x)&=\rho(Nx)+\rho(x)\circ S-S\circ\rho(x),\\\
\varpi_{1}(x,y)&=\mu(Nx,y)+\mu(x,Ny)+\mu(x,y)\circ S-S\circ\mu(x,y),\\\
\varpi_{2}(x,y)&=\mu(Nx,y)\circ S+\mu(x,Ny)\circ
S+\mu(Nx,Ny)-S\circ\varpi_{1}(x,y).\end{cases}$

Moreover, this deformation is trivial.

###### Proof.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$(\mathfrak{g},\rho,\mu)$. Therefore, for any $t$, there holds that

$\displaystyle({{\rm{Id}}}_{\mathfrak{g}}+tN)[x,y]_{t}$ $\displaystyle=$
$\displaystyle[({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y],$
$\displaystyle({{\rm{Id}}}_{\mathfrak{g}}+tN)\left\llbracket
x,y,z\right\rrbracket_{t}$ $\displaystyle=$
$\displaystyle\left\llbracket({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y,({{\rm{Id}}}_{\mathfrak{g}}+tN)z\right\rrbracket,$
$\displaystyle({{\rm{Id}}}_{V}+tS)\circ\rho_{t}(x)$ $\displaystyle=$
$\displaystyle\rho\Big{(}({{\rm{Id}}}_{\mathfrak{g}}+tN)x\Big{)}\circ({{\rm{Id}}}_{V}+tS),$
$\displaystyle({{\rm{Id}}}_{V}+tS)\circ\mu_{t}(x,y)$ $\displaystyle=$
$\displaystyle\mu\Big{(}({{\rm{Id}}}_{\mathfrak{g}}+tN)x,({{\rm{Id}}}_{\mathfrak{g}}+tN)y\Big{)}\circ({{\rm{Id}}}_{V}+tS),\quad\forall
x,y,z\in\mathfrak{g}.$

Then for $t$ sufficiently small, ${{\rm{Id}}}_{\mathfrak{g}}+tN$ and
${{\rm{Id}}}_{V}+tS$ are isomorphisms between vector spaces. By Lemma 3.12, we
have that
$\Big{(}(\mathfrak{g},[\cdot,\cdot]_{t},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{t}),(V,\rho_{t},\mu_{t})\Big{)}$
is a $\mathsf{LieYRep}$ pair, which means that the linear maps
$(\phi,\varphi_{i};\varrho,\varpi_{i})~{}(i=1,2)$ given by Eqs. (19) generate
a linear deformation. It is obvious that this deformation is trivial. This
completes the proof. ∎

The following proposition demonstrates that a Nijenhuis structure gives rise
to a Nijenhuis operator on the semidirect product Lie-Yamaguti algebra.

######  Proposition 3.14.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then $N+S$ is a Nijenhuis operator on the semidirect product Lie-Yamaguti
algebra $\mathfrak{g}\ltimes V$.

###### Proof.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then we have

$\displaystyle[(N+S)(x+u),(N+S)(y+v)]_{\ltimes}-(N+S)\Big{(}[(N+S)(x+u),y+v]_{\ltimes}+[x+u,(N+S)(y+v)]_{\ltimes}\Big{)}$
$\displaystyle~{}~{}+(N+S)^{2}[x+u,y+v]_{\ltimes}$ $\displaystyle=$
$\displaystyle\Big{(}[Nx,Ny]+\rho(Nx)(Sv)-\rho(Ny)(Su)\Big{)}-(N+S)\Big{(}[Nx,y]+\rho(Nx)v-\rho(y)(Su)$
$\displaystyle~{}~{}+[x,Ny]+\rho(x)(Sv)-\rho(Ny)u\Big{)}+(N^{2}+S^{2})\Big{(}[x,y]+\rho(x)v-\rho(y)u\Big{)}$
$\displaystyle=$
$\displaystyle\Big{(}[Nx,Ny]-N\big{(}[Nx,y]+[x,Ny]-N[x,y]\big{)}\Big{)}+\Big{(}\rho(Nx)(Sv)-S\big{(}\rho(Nx)v-\rho(x)(Sv)+S(\rho(x)v)\big{)}\Big{)}$
$\displaystyle-\Big{(}\rho(Ny)(Su)-S\big{(}\rho(Ny)u-\rho(y)(Su)+S(\rho(y)u)\big{)}\Big{)}$
$\displaystyle=$ $\displaystyle 0.$

Moreover, we also have

$\displaystyle\left\llbracket(N+S)(x+u),(N+S)(y+v),(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle=$ $\displaystyle\left\llbracket
Nx,Ny,Nz\right\rrbracket+D(Nx,Ny)(Sw)+\mu(Ny,Nz)(Su)-\mu(Nx,Nz)(Sv),$

and

$\displaystyle(N+S)\Big{(}\left\llbracket(N+S)(x+u),(N+S)(y+v),z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket(N+S)(x+u),y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}\Big{)}$
$\displaystyle~{}~{}-(N+S)^{2}\Big{(}\left\llbracket(N+S)(x+u),y+v,z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),z+w\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket
x+u,y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}\Big{)}+(N+S)^{3}\left\llbracket
x+u,y+v,z+w\right\rrbracket_{\ltimes}$ $\displaystyle=$
$\displaystyle(N+S)\Big{(}\left\llbracket
Nx,Ny,z\right\rrbracket+D(Nx,Ny)w+\mu(Ny,z)(Su)-\mu(Nx,z)(Sv)$
$\displaystyle~{}~{}+\left\llbracket
Nx,y,Nz\right\rrbracket+D(Nx,y)(Sw)+\mu(y,Nz)(Su)-\mu(Nx,Nz)v$
$\displaystyle~{}~{}+\left\llbracket
x,Ny,Nz\right\rrbracket+D(x,Ny)(Sw)+\mu(Ny,Nz)u-\mu(x,Nz)(Sv)\Big{)}$
$\displaystyle~{}~{}-(N^{2}+S^{2})\Big{(}\left\llbracket
Nx,y,z\right\rrbracket+D(Nx,y)w+\mu(y,z)(Su)-\mu(Nx,z)v$
$\displaystyle~{}~{}+\left\llbracket
x,Ny,z\right\rrbracket+D(x,Ny)w+\mu(Ny,z)u-\mu(x,z)(Sv)$
$\displaystyle~{}~{}+\left\llbracket
x,y,Nz\right\rrbracket+D(x,y)(Sw)+\mu(y,Nz)u-\mu(x,Nz)v\Big{)}$
$\displaystyle~{}~{}+(N^{3}+S^{3})\Big{(}\left\llbracket
x,y,z\right\rrbracket+D(x,y)w+\mu(y,z)u-\mu(x,z)v\Big{)}$ $\displaystyle=$
$\displaystyle N\Big{(}\left\llbracket Nx,Ny,z\right\rrbracket+\left\llbracket
Nx,y,Nz\right\rrbracket+\left\llbracket x,Ny,Nz\right\rrbracket\Big{)}$
$\displaystyle~{}~{}-N^{2}\Big{(}\left\llbracket
Nx,y,z\right\rrbracket+\left\llbracket x,Ny,z\right\rrbracket+\left\llbracket
x,y,Nz\right\rrbracket\Big{)}+N^{3}\left\llbracket x,y,z\right\rrbracket$
$\displaystyle~{}~{}+S\Big{(}D(Nx,Ny)w+D(Nx,y)(Sw)+D(x,Ny)(Sw)\Big{)}$
$\displaystyle~{}~{}-S^{2}\Big{(}D(Nx,y)w+D(x,Ny)w+D(x,y)(Sw)\Big{)}+S^{3}(D(x,y)w)$
$\displaystyle~{}~{}+S\Big{(}\mu(Ny,z)(Su)+\mu(y,Nz)(Su)+\mu(Ny,Nz)u\Big{)}$
$\displaystyle~{}~{}-S^{2}\Big{(}\mu(y,z)(Su)+\mu(Ny,z)u+\mu(y,Nz)u\Big{)}+S^{3}(\mu(y,z)u)$
$\displaystyle~{}~{}-S\Big{(}\mu(Nx,z)(Sv)+\mu(Nx,Nz)v+\mu(x,Nz)(Sv)\Big{)}$
$\displaystyle~{}~{}+S^{2}\Big{(}\mu(Nx,z)v+\mu(x,z)(Sv)+\mu(x,Nz)v\Big{)}-S^{3}(\mu(x,z)v),$

which implies that

$\displaystyle\left\llbracket(N+S)(x+u),(N+S)(y+v),(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle=$
$\displaystyle(N+S)\Big{(}\left\llbracket(N+S)(x+u),(N+S)(y+v),z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket(N+S)(x+u),y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}\Big{)}$
$\displaystyle~{}~{}-(N+S)^{2}\Big{(}\left\llbracket(N+S)(x+u),y+v,z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),z+w\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket
x+u,y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}\Big{)}+(N+S)^{3}\left\llbracket
x+u,y+v,z+w\right\rrbracket_{\ltimes}.$

This completes the proof. ∎

We introduce two linear maps $\hat{\rho}:\mathfrak{g}\to\mathfrak{gl}(V)$ and
$\hat{\mu}:\otimes^{2}\mathfrak{g}\to\mathfrak{gl}(V)$ by

(20) $\displaystyle\hat{\rho}(x)$ $\displaystyle=$
$\displaystyle\rho(Nx)+\rho(x)\circ S-S\circ\rho(x),$ (21)
$\displaystyle\hat{\mu}(x,y)$ $\displaystyle=$ $\displaystyle\mu(Nx,y)\circ
S+\mu(x,Ny)\circ S+\mu(Nx,Ny)$
$\displaystyle-S\Big{(}\mu(Nx,y)+\mu(x,Ny)+\mu(x,y)\circ
S\Big{)}+S^{2}\circ\mu(x,y),$

######  Lemma 3.15.

With the above notations, we have

(22) $\displaystyle\hat{D}(x,y):=D_{\hat{\rho},\hat{\mu}}$ $\displaystyle=$
$\displaystyle D(Nx,y)\circ S+D(x,Ny)\circ S+D(Nx,Ny)$
$\displaystyle-S\Big{(}D(Nx,y)+D(x,Ny)+D(x,y)\circ S\Big{)}+S^{2}\circ
D(x,y).$

###### Proof.

The proof also involves computation. ∎

######  Proposition 3.16.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then
$\Big{(}(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N}),(V;\hat{\rho},\hat{\mu})\Big{)}$
is a $\mathsf{LieYRep}$ pair, where
$(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$
is the deformed Lie-Yamaguti algebra. Thus $(N,S)$ is a homomorphism from
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N}),\\\
(V;\hat{\rho},\hat{\mu})\Big{)}$ to
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.

###### Proof.

It was proved in [26] that
$(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$
is a Lie-Yamaguti algebra. What is left is to prove that
$(V;\hat{\rho},\hat{\mu})$ is a representation on
$(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$.
Indeed, by Proposition 4.10, we have

$\displaystyle~{}[x+u,y+v]_{N+S}$ $\displaystyle=$
$\displaystyle[(N+S)(x+u),y+v]_{\ltimes}+[x+u,(N+S)(y+v)]_{\ltimes}-(N+S)[x+u,y+v]_{\ltimes}$
$\displaystyle=$ $\displaystyle[Nx,y]+[x,Ny]-N[x,y]$
$\displaystyle+\Big{(}\rho(Nx)v+\rho(x)(Sv)-S(\rho(x)v)\Big{)}-\Big{(}\rho(Ny)u+\rho(y)(Su)-S(\rho(y)u)\Big{)}$
$\displaystyle=$ $\displaystyle[x,y]_{N}+\hat{\rho}(x)v-\hat{\rho}(y)u,$

and by Lemma 3.15, we also have that

$\displaystyle\left\llbracket x+u,y+v,z+w\right\rrbracket_{N+S}$
$\displaystyle=$
$\displaystyle\left\llbracket(N+S)(x+u),(N+S)(y+v),z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket(N+S)(x+u),y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}-(N+S)\Big{(}\left\llbracket(N+S)(x+u),y+v,z+w\right\rrbracket_{\ltimes}+\left\llbracket
x+u,(N+S)(y+v),z+w\right\rrbracket_{\ltimes}$
$\displaystyle~{}~{}+\left\llbracket
x+u,y+v,(N+S)(z+w)\right\rrbracket_{\ltimes}\Big{)}+(N+S)^{2}\left\llbracket
x+u,y+v,z+w\right\rrbracket_{\ltimes}$ $\displaystyle=$
$\displaystyle\Big{(}\left\llbracket Nx,Ny,z\right\rrbracket+\left\llbracket
Nx,y,Nz\right\rrbracket+\left\llbracket x,Ny,Nz\right\rrbracket$
$\displaystyle~{}~{}-N\big{(}\left\llbracket
Nx,y,z\right\rrbracket+\left\llbracket x,Ny,z\right\rrbracket+\left\llbracket
x,y,Nz\right\rrbracket\big{)}+N^{2}\left\llbracket
x,y,z\right\rrbracket\Big{)}$
$\displaystyle+\Big{(}D(Nx,Ny)w+D(Nx,y)(Sw)+D(x,Ny)(Sw)$
$\displaystyle~{}~{}-S\big{(}D(Nx,y)w+D(x,Ny)w+D(x,y)(Sw)\big{)}+S^{2}(D(x,y)w)\Big{)}$
$\displaystyle~{}~{}+\Big{(}\mu(Ny,z)(Su)+\mu(y,Nz)(Su)+\mu(Ny,Nz)u$
$\displaystyle~{}~{}-S\big{(}\mu(y,z)(Su)+\mu(Ny,z)u+\mu(y,Nz)u\big{)}+S^{2}(\mu(y,z)u)\Big{)}$
$\displaystyle~{}~{}-\Big{(}\mu(Nx,z)(Sv)+\mu(Nx,Nz)v+\mu(x,Nz)(Sv)$
$\displaystyle~{}~{}+S\big{(}\mu(Nx,z)v+\mu(x,z)(Sv)+\mu(x,Nz)v\big{)}-S^{3}(\mu(x,z)v)\Big{)}$
$\displaystyle=$ $\displaystyle\left\llbracket
x,y,z\right\rrbracket_{N}+\hat{D}(x,y)w+\hat{\mu}(y,z)u-\hat{\mu}(x,z)v,$

which implies that $(V;\hat{\rho},\hat{\mu})$ is a representation on
$(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$.
∎

##  4\. Relative Rota-Baxter-Nijenhuis structures and compatible relative
Rota-Baxter operators

###  4.1. Relative Rota-Baxter-Nijenhuis structures on $\mathsf{LieYRep}$
pairs

In this subsection, we introduce the notion of relative Rota-Baxter-Nijenhuis
structures and discuss several properties of relative Rota-Baxter-Nijenhuis
structures. First, let us recall some notions in [27] about relative Rota-
Baxter operators on Lie-Yamaguti algebras.

######  Definition 4.1.

([27]) Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra with a representation $(V;\rho,\mu)$ and
$T:V\to\mathfrak{g}$ a linear map. If $T$ satisfies

$\displaystyle[Tu,Tv]$ $\displaystyle=$ $\displaystyle
T\Big{(}\rho(Tu)v-\rho(Tv)u\Big{)},$ $\displaystyle\left\llbracket
Tu,Tv,Tw\right\rrbracket$ $\displaystyle=$ $\displaystyle
T\Big{(}D(Tu,Tv)w+\mu(Tv,Tw)u-\mu(Tu,Tw)v\Big{)},\quad\forall u,v,w\in V,$

then we call $T$ a relative Rota-Baxter operator on $\mathfrak{g}$ with
respect to the representation $(V;\rho,\mu)$.

######  Lemma 4.2.

([27]) Let $T$ be a relative Rota-Baxter operator on a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to $(V;\rho,\mu)$. Then
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$
is a Lie-Yamaguti algebra, where

(23) $\displaystyle[u,v]^{T}$ $\displaystyle=$
$\displaystyle\rho(Tu)v-\rho(Tv)u,$ (24) $\displaystyle\left\llbracket
u,v,w\right\rrbracket^{T}$ $\displaystyle=$ $\displaystyle
D_{\rho,\mu}(Tu,Tv)w+\mu(Tv,Tw)u-\mu(Tu,Tw)v,\quad\forall u,v,w\in V.$

Thus $T$ is a Lie-Yamaguti algebra homomorphism from
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$
to
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
Furthermore,
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$
is the sub-adjacent Lie-Yamaguti algebra of the pre-Lie-Yamaguti algebra
$(V,*^{T},\\{\cdot,\cdot,\cdot\\}^{T})$ defined by

(25) $\displaystyle
u*^{T}v=\rho(Tu)v,\quad\\{u,v,w\\}^{T}=\mu(Tv,Tw)u,\quad\forall u,v,w\in V.$

With the above notations, let $S\in\mathfrak{gl}(V)$ be a linear map on $V$,
then the deformed bracket
$([\cdot,\cdot]_{S}^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{S}^{T})$
of the sub-adjacent Lie-Yamaguti algebra
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$
is given as follows:

$\displaystyle[u,v]_{S}^{T}$ $\displaystyle=$
$\displaystyle[Su,v]^{T}+[u,Sv]^{T}-S[u,v]^{T},$ $\displaystyle\left\llbracket
u,v,w\right\rrbracket_{S}^{T}$ $\displaystyle=$ $\displaystyle\left\llbracket
Su,Sv,w\right\rrbracket^{T}+\left\llbracket
Su,v,Sw\right\rrbracket^{T}+\left\llbracket
u,Sv,Sw\right\rrbracket^{T}\Big{)}$ $\displaystyle-S\Big{(}\left\llbracket
Su,v,w\right\rrbracket^{T}+\left\llbracket
u,Sv,w\right\rrbracket^{T}+\left\llbracket
u,v,Sw\right\rrbracket^{T}\Big{)}+S^{2}\left\llbracket
u,v,w\right\rrbracket^{T},\quad\forall u,v,w\in V.$

Moreover, define a pair of brackets
$([\cdot,\cdot]_{\hat{\rho}}^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\hat{\mu}}^{T})$
to be

$\displaystyle[u,v]_{\hat{\rho}}^{T}$ $\displaystyle=$
$\displaystyle\hat{\rho}(Tu)v-\hat{\rho}(Tv)u,$ $\displaystyle\left\llbracket
u,v,w\right\rrbracket_{\hat{\mu}}^{T}$ $\displaystyle=$
$\displaystyle\hat{D}(Tu,Tv)w+\hat{\mu}(Tv,Tw)u-\hat{\mu}(Tu,Tw)v,$

respectively, where $\hat{\rho},~{}\hat{\mu}$ and $\hat{D}$ are given by
(20)-(22). Note that in general
$([\cdot,\cdot]_{S}^{T},~{}\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{S}^{T})$,
and
$([\cdot,\cdot]_{\hat{\rho}}^{T},~{}\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\hat{\mu}}^{T})$
are not necessarily Lie-Yamaguti algebra structures.

######  Definition 4.3.

Let $(N,S)$ be a Nijenhuis structure on a $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
and let $T:V\longrightarrow\mathfrak{g}$ be a relative Rota-Baxter operator on
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to $(V;\rho,\mu)$. If for all $u,v,w\in V$, the following
conditions are satisfied:

(26) $\displaystyle N\circ T$ $\displaystyle=$ $\displaystyle T\circ S,$ (27)
$\displaystyle~{}[u,v]_{S}^{T}$ $\displaystyle=$ $\displaystyle[u,v]^{N\circ
T},$ (28) $\displaystyle\left\llbracket u,v,w\right\rrbracket_{S}^{T}$
$\displaystyle=$ $\displaystyle\left\llbracket u,v,w\right\rrbracket^{N\circ
T}.$

Then we call the triple $(T,S,N)$ a relative Rota-Baxter-Nijenhuis structure
on the $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.

######  Lemma 4.4.

Let $(T,S,N)$ be a relative Rota-Baxter-Nijenhuis structure on a
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then we have

$\displaystyle~{}[u,v]_{S}^{T}$ $\displaystyle=$
$\displaystyle[u,v]_{\hat{\rho}}^{T},$ $\displaystyle\left\llbracket
u,v,w\right\rrbracket_{S}^{T}$ $\displaystyle=$ $\displaystyle\left\llbracket
u,v,w\right\rrbracket_{\hat{\mu}}^{T}.$

###### Proof.

It follows from (26) and a direct computation. ∎

Thus if $(T,S,N)$ is a relative Rota-Baxter-Nijenhuis structure on a
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),\\\
(V;\rho,\mu)\Big{)}$, then the brackets
$([\cdot,\cdot]_{S}^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{S}^{T})$
and
$([\cdot,\cdot]_{\hat{\rho}}^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\hat{\mu}}^{T})$
are equal. Moreover, we have the following conclusion.

######  Theorem 4.5.

Let $(T,S,N)$ be a relative Rota-Baxter-Nijenhuis structure on a
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then $S$ is a Nijenhuis operator on the sub-adjacent Lie-Yamaguti algebra
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$.
Thus the brackets
$([\cdot,\cdot]_{S}^{T},~{}\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{S}^{T})$
and
$([\cdot,\cdot]_{\hat{\rho}}^{T},~{}\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{\hat{\mu}}^{T})$
are the Lie-Yamaguti algebra structures on $V$.

###### Proof.

Since $(T,S,N)$ is a relative Rota-Baxter-Nijenhuis structure, then we have

$\displaystyle[Su,Sv]^{T}\stackrel{{\scriptstyle\eqref{induce1}}}{{=}}\rho(TSu)(Sv)-\rho(TSv)(Su)$
$\displaystyle\stackrel{{\scriptstyle\eqref{ON1},\eqref{Nijstru1}}}{{=}}$
$\displaystyle
S\Big{(}\rho(TSu)v+\rho(Tu)(Sv)\Big{)}-S^{2}\Big{(}\rho(Tu)v\Big{)}$
$\displaystyle-S\Big{(}\rho(TSv)u+\rho(Tv)(Su)\Big{)}+S^{2}\Big{(}\rho(Tv)u\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{induce1}}}{{=}}$ $\displaystyle
S\Big{(}[Su,v]^{T}+[u,Sv]^{T}\Big{)}-S^{2}[u,v]^{T},$

and

$\displaystyle\left\llbracket Su,Sv,Sw\right\rrbracket^{T}$
$\displaystyle\stackrel{{\scriptstyle\eqref{induce2}}}{{=}}$ $\displaystyle
D(TSu,TSv)(Sw)+\mu(TSv,TSw)(Su)-\mu(TSu,TSw)(Sv)$
$\displaystyle\stackrel{{\scriptstyle\eqref{ON1},\eqref{Nijstru2},\eqref{Nijstru3}}}{{=}}$
$\displaystyle S\Big{(}D(TSu,Tv)(Sw)+D(Tu,TSv)(Sw)+D(TSu,TSv)w\Big{)}$
$\displaystyle-S^{2}\Big{(}D(TSu,Tv)w+D(Tu,TSv)w+D(Tu,Tv)(Sw)\Big{)}+S^{3}(D(Tu,Tv)w)$
$\displaystyle+S\Big{(}\mu(TSv,Tw)(Su)+\mu(Tv,TSw)(Su)+\mu(TSv,TSw)u\Big{)}$
$\displaystyle-S^{2}\Big{(}\mu(TSv,Tw)u+\mu(Tv,TSw)u+\mu(Tv,Tw)(Su)\Big{)}+S^{3}(\mu(Tv,Tw)u)$
$\displaystyle-S\Big{(}\mu(TSu,Tw)(Sv)+\mu(Tu,TSw)(Sv)+\mu(TSu,TSw)v\Big{)}$
$\displaystyle+S^{2}\Big{(}\mu(TSu,Tw)v+\mu(Tu,TSw)v+\mu(Tu,Tw)(Sv)\Big{)}-S^{3}(\mu(Tu,Tw)v)$
$\displaystyle\stackrel{{\scriptstyle\eqref{induce2}}}{{=}}$ $\displaystyle
S\Big{(}\left\llbracket Su,Sv,w\right\rrbracket^{T}+\left\llbracket
Su,v,Sw\right\rrbracket^{T}+\left\llbracket
u,Sv,Sw\right\rrbracket^{T}\Big{)}$ $\displaystyle-S^{2}\Big{(}\left\llbracket
Su,v,w\right\rrbracket^{T}+\left\llbracket
u,Sv,w\right\rrbracket^{T}+\left\llbracket
Su,v,Sw\right\rrbracket^{T}\Big{)}+S^{3}\left\llbracket
u,v,w\right\rrbracket^{T}.$

Thus, $S$ is a Nijenhuis operator on the Lie-Yamaguti algebra
$(V,[\cdot,\cdot]^{T},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket^{T})$.
∎

######  Theorem 4.6.

Let $(T,S,N)$ be a relative Rota-Baxter-Nijenhuis structure on a
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.
Then we have

  * (i)

$T$ is a relative Rota-Baxter operator on the deformed Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot]_{N},\left\llbracket\cdot,\cdot,\cdot\right\rrbracket_{N})$
with respect to a representation $(V;\hat{\rho},\hat{\mu})$;

  * (ii)

$N\circ T$ is a relative Rota-Baxter operator on the Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to a representation $(V;\rho,\mu)$.

###### Proof.

Since $T$ is a relative Rota-Baxter operator on the Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to a representation $(V;\rho,\mu)$ and $N\circ T=T\circ S$, we
have

$\displaystyle~{}T[u,v]_{\hat{\rho}}^{T}$ $\displaystyle=$ $\displaystyle
T[u,v]_{S}^{T}=T\Big{(}[Su,v]^{T}+[u,Sv]^{T}-S[u,v]^{T}\Big{)}$
$\displaystyle=$ $\displaystyle[TSu,Tv]+[Tu,TSv]-T\circ S[u,v]^{T}$
$\displaystyle=$ $\displaystyle[NTu,Tv]+[Tu,NTv]-N\circ T[u,v]^{T}$
$\displaystyle=$ $\displaystyle[NTu,Tv]+[Tu,NTv]-N[Tu,Tv]$ $\displaystyle=$
$\displaystyle[Tu,Tv]_{N},$

and

$\displaystyle T\left\llbracket
u,v,w\right\rrbracket_{\hat{\mu}}^{T}=T\left\llbracket
u,v,w\right\rrbracket_{S}^{T}$ $\displaystyle=$ $\displaystyle
T\Big{(}\left\llbracket Su,Sv,w\right\rrbracket^{T}+\left\llbracket
Su,v,Sw\right\rrbracket^{T}+\left\llbracket u,Sv,Sw\right\rrbracket^{T}$
$\displaystyle-S\left\llbracket Su,v,w\right\rrbracket^{T}-S\left\llbracket
u,Sv,w\right\rrbracket^{T}-S\left\llbracket
u,v,Sw\right\rrbracket^{T}+S^{2}\left\llbracket
u,v,w\right\rrbracket^{T}\Big{)}$ $\displaystyle=$
$\displaystyle\left\llbracket TSu,TSv,w\right\rrbracket+\left\llbracket
TSu,Tv,TSw\right\rrbracket+\left\llbracket Tu,TSv,TSw\right\rrbracket$
$\displaystyle-T\circ S\left\llbracket Su,TSv,w\right\rrbracket^{T}-T\circ
S\left\llbracket Su,Sv,w\right\rrbracket-T\circ S\left\llbracket
u,v,Sw\right\rrbracket^{T}+T\circ S^{2}\left\llbracket
u,v,w\right\rrbracket^{T}$ $\displaystyle=$ $\displaystyle\left\llbracket
NTu,NTv,w\right\rrbracket+\left\llbracket
NTu,Tv,NTw\right\rrbracket+\left\llbracket Tu,NTv,NTw\right\rrbracket$
$\displaystyle-N\Big{(}\left\llbracket
NTu,Tv,Tw\right\rrbracket^{T}+\left\llbracket
NTu,Tv,NTw\right\rrbracket+\left\llbracket
Tu,Tv,NTw\right\rrbracket^{T}\Big{)}+N^{2}\left\llbracket
Tu,Tv,Tw\right\rrbracket$ $\displaystyle=$ $\displaystyle\left\llbracket
Tu,Tv,Tw\right\rrbracket_{N}.$

This proves (i). By (27) and (28), we have that

$N\circ T[u,v]^{N\circ T}=N\circ T[u,v]_{S}^{T}=N[Tu,Tv]_{N}=[NTu,NTv],$

and

$N\circ T\left\llbracket u,v,w\right\rrbracket^{N\circ T}=N\circ
T\left\llbracket u,v,w\right\rrbracket_{S}^{T}=N\left\llbracket
Tu,Tv,Tw\right\rrbracket_{N}=\left\llbracket NTu,NTv,NT.w\right\rrbracket$

This proves (ii). ∎

###  4.2. Compatible relative Rota-Baxter operators on Lie-Yamaguti algebras

In this subsection, we introduce the notion of compatible relative Rota-Baxter
operators on Lie-Yamaguti algebra, and we show that a relative Rota-Baxter-
Nijenhuis structure gives rise to a pair of relative Rota-Baxter operators on
Lie-Yamaguti algebras under a certain condition.

######  Definition 4.7.

Let $T_{1},~{}T_{2}:V\to\mathfrak{g}$ be two relative Rota-Baxter operators on
a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot]\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to a representation $(V;\rho,\mu)$. If for all $k_{1}$ and
$k_{2}$, $k_{1}T_{1}+k_{2}T_{2}$ is still a relative Rota-Baxter operator on
$\mathfrak{g}$, then we say that $T_{1}$ and $T_{2}$ are compatible.

By a direct computation, we have the following proposition.

######  Proposition 4.8.

Let $T_{1},~{}T_{2}:V\to\mathfrak{g}$ be two relative Rota-Baxter operators on
a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot]\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
Then $T_{1}$ and $T_{2}$ are compatible if and only if the following equations
are satisfied:

$\displaystyle[T_{i}u,T_{j}v]+[T_{j}u,T_{i}v]$ $\displaystyle=$ $\displaystyle
T_{i}\Big{(}\rho(T_{j}u)v-\rho(T_{j}v)u\Big{)}+T_{j}\Big{(}\rho(T_{i}u)v-\rho(T_{i}v)u\Big{)},$
$\displaystyle\left\llbracket
T_{i}u,T_{i}v,T_{j}w\right\rrbracket+\left\llbracket
T_{i}u,T_{j}v,T_{i}w\right\rrbracket+\left\llbracket
T_{j}u,T_{i}v,T_{i}w\right\rrbracket$ $\displaystyle=$ $\displaystyle
T_{i}\Big{(}D(T_{i}u,T_{j}v)w+\mu(T_{i}v,T_{j}w)u-\mu(T_{i}u,T_{j}w)v$
$\displaystyle+D(T_{j}u,T_{i}v)w+\mu(T_{j}v,T_{i}w)u-\mu(T_{j}u,T_{i}w)v\Big{)}$
$\displaystyle+T_{j}\Big{(}D(T_{i}u,T_{i}v)w+\mu(T_{i}v,T_{i}w)u-\mu(T_{i}u,T_{i}w)v\Big{)},\quad(i,j=1,2).$

The following proposition shows that a pair of compatible relative Rota-Baxter
operators with an invertible condition yields a Nijenhuis operator.

######  Proposition 4.9.

Let $T_{1},~{}T_{2}:V\to\mathfrak{g}$ be two compatible relative Rota-Baxter
operators on a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
with respect to a representation $(V;\rho,\mu)$. If $T_{2}$ is invertible,
then $N=T_{1}\circ T_{2}^{-1}$ is a Nijenhuis operator on the Lie-Yamaguti
algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.

###### Proof.

Since $T_{2}$ is invertible, for all $x,y,z\in\mathfrak{g}$, there exist
$u,v,w\in V$, such that $x=T_{2}u,~{}y=T_{2}v,~{}z=T_{2}w$. Thus it is
sufficient to show that

(29) $\displaystyle~{}[NT_{2}u,NT_{2}v]$ $\displaystyle=$ $\displaystyle
N\Big{(}[NT_{2}u,T_{2}v]+[T_{2}u,NT_{2}v]\Big{)}-N^{2}[T_{2}u,T_{2}v],$ (30)
$\displaystyle~{}\left\llbracket NT_{2}u,NT_{2}v,NT_{2}w\right\rrbracket$
$\displaystyle=$ $\displaystyle N\Big{(}\left\llbracket
NT_{2}u,NT_{2}v,T_{2}w\right\rrbracket+\left\llbracket
NT_{2}u,T_{2}v,NT_{2}w\right\rrbracket+\left\llbracket
T_{2}u,NT_{2}v,NT_{2}w\right\rrbracket\Big{)}$
$\displaystyle-N^{2}\Big{(}\left\llbracket
NT_{2}u,T_{2}v,T_{2}w\right\rrbracket+\left\llbracket
T_{2}u,NT_{2}v,T_{2}w\right\rrbracket+\left\llbracket
T_{2}u,T_{2}v,NT_{2}w\right\rrbracket\Big{)}$
$\displaystyle+N^{3}\left\llbracket T_{2}u,T_{2}v,T_{2}w\right\rrbracket.$

Since $T_{1}=N\circ T_{2}$ is a relative Rota-Baxter operator, the left hand
side of Eqs. (29) and (30) are

$NT_{2}\Big{(}\rho(NT_{2}u)v-\rho(NT_{2}v)u\Big{)}$

and

$NT_{2}\Big{(}D(NT_{2}u,NT_{2}v)w+\mu(NT_{2}v,NT_{2}w)u-\mu(NT_{2}u,NT_{2}w)v\Big{)}$

respectively. Since $T_{1}=N\circ T_{2}$ and $T_{2}$ are compatible, by
Proposition 4.8, we have

$\displaystyle[NT_{2}u,T_{2}v]+[T_{2}u,NT_{2}v]$ (31) $\displaystyle=$
$\displaystyle
T_{2}\Big{(}\rho(NT_{2}u)v-\rho(NT_{2}v)u\Big{)}+NT_{2}\Big{(}\rho(T_{2}u)v-\rho(T_{2}v)u\Big{)}$
$\displaystyle=$ $\displaystyle
T_{2}\Big{(}\rho(NT_{2}u)v-\rho(NT_{2}v)u\Big{)}+N[T_{2}u,T_{2}v].$

The action of $N$ on both sides of (31) yields Eq (29). On the other hand, as
for (30) we have

$\displaystyle\left\llbracket
NT_{2}u,NT_{2}v,T_{2}w\right\rrbracket+\left\llbracket
T_{2}u,NT_{2}v,NT_{2}w\right\rrbracket+\left\llbracket
NT_{2}u,T_{2}v,NT_{2}w\right\rrbracket$ $\displaystyle=$ $\displaystyle
T_{2}\Big{(}D(NT_{2}u,NT_{2}v)w+\mu(NT_{2}v,NT_{2}w)u-\mu(NT_{2}u,NT_{2}w)v\Big{)}$
$\displaystyle+NT_{2}\Big{(}D(NT_{2}u,T_{2}v)w+\mu(NT_{2}v,T_{2}w)u-\mu(NT_{2}u,T_{2}w)v$
$\displaystyle+D(T_{2}u,NT_{2}v)w+\mu(T_{2}v,NT_{2}w)u-\mu(T_{2}u,NT_{2}w)v\Big{)},$

and

$\displaystyle\left\llbracket
NT_{2}u,T_{2}v,T_{2}w\right\rrbracket+\left\llbracket
T_{2}u,NT_{2}v,T_{2}w\right\rrbracket+\left\llbracket
T_{2}u,T_{2}v,NT_{2}w\right\rrbracket$ $\displaystyle=$ $\displaystyle
NT_{2}\Big{(}D(T_{2}u,T_{2}v)w+\mu(T_{2}v,T_{2}w)u-\mu(T_{2}u,T_{2}w)v\Big{)}$
$\displaystyle+T_{2}\Big{(}D(NT_{2}u,T_{2}v)w+\mu(NT_{2}v,T_{2}w)u-\mu(NT_{2}u,T_{2}w)v$
$\displaystyle+D(T_{2}u,NT_{2}v)w+\mu(T_{2}v,NT_{2}w)u-\mu(T_{2}u,NT_{2}w)v\Big{)}.$

Thus the right hand side of Eq (30) is

$NT_{2}\Big{(}D(NT_{2}u,NT_{2}v)w+\mu(NT_{2}v,NT_{2}w)u-\mu(NT_{2}u,NT_{2}w)v\Big{)},$

which is exactly the left hand side of (30). This finishes the proof. ∎

Given a relative Rota-Baxter-Nijenhuis structure $(T,S,N)$, by Theorem 4.6,
$T\circ S=N\circ T$ is a relative Rota-Baxter operator. However, in general,
$T$ and $T\circ S$ are not compatible spontaneously. Indeed, we can show that
they are compatible under a certain condition.

######  Proposition 4.10.

Let $(T,S,N)$ be a relative Rota-Baxter-Nijenhuis structure on a
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
satisfying

(32) $D(Tu,Tv)(Sw)+\mu(Tv,Tw)(Su)-\mu(Tu,Tw)(Sv)=S\big{(}\left\llbracket
u,v,w\right\rrbracket^{T}\big{)},\quad\forall u,v,w\in\mathfrak{g}.$

Then $T$ and $T\circ S$ are compatible relative Rota-Baxter operators.

###### Proof.

Since $T$ is a relative Rota-Baxter operator, we have

$\displaystyle[TSu,Tv]+[Tu,TSv]$ $\displaystyle=$ $\displaystyle
T\Big{(}\rho(TSu)v-\rho(Tv)(Su)+\rho(Tu)(Sv)-\rho(TSv)u\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{ON2}}}{{=}}$ $\displaystyle
T\Big{(}\rho(TSu)v-\rho(TSv)u\Big{)}+TS\Big{(}\rho(Tu)v-\rho(Tv)u\Big{)}.$

Moreover, Eqs. (28) and (32) imply that

$\displaystyle D(TSu,Tv)(Sw)+\mu(TSv,Tw)(Su)-\mu(TSu,Tw)(Sv)$
$\displaystyle+D(Tu,TSv)(Sw)+\mu(Tv,TSw)(Su)-\mu(Tu,TSw)(Sv)$ $\displaystyle=$
$\displaystyle S\Big{(}D(TSu,Tv)w+\mu(TSv,Tw)u-\mu(TSu,Tw)v$
$\displaystyle+D(Tu,TSv)w+\mu(Tv,TSw)u-\mu(Tu,TSw)v\Big{)}.$

By the fact that $T$ is a relative Rota-Baxter operator again, we have

$\displaystyle\left\llbracket TSu,TSv,Tw\right\rrbracket+\left\llbracket
TSu,Tv,TSw\right\rrbracket+\left\llbracket Tu,TSv,TSw\right\rrbracket$
$\displaystyle=$ $\displaystyle
T\Big{(}D(TSu,TSv)w+\mu(TSv,Tw)(Su)-\mu(TSu,Tw)(Sv)$
$\displaystyle+D(TSu,Tv)(Sw)+\mu(Tv,TSw)(Su)-\mu(TSu,TSw)v$
$\displaystyle+D(Tu,TSv)(Sw)+\mu(TSv,TSw)u-\mu(Tu,TSw)(Sv)\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{strong1}}}{{=}}$ $\displaystyle
T\Big{(}D(TSu,TSv)w+\mu(TSv,TSw)u-\mu(TSu,TSw)v\Big{)}$ $\displaystyle+T\circ
S\Big{(}D(TSu,Tv)w+\mu(TSv,Tw)u-\mu(TSu,Tw)v$
$\displaystyle+D(Tu,TSv)w+\mu(Tv,TSw)u-\mu(Tu,TSw)v\Big{)}.$

Similarly, we deduce from (32) that

$\displaystyle\left\llbracket Tu,TSv,Tw\right\rrbracket+\left\llbracket
Tu,Tv,TSw\right\rrbracket+\left\llbracket TSu,Tv,Tw\right\rrbracket$
$\displaystyle=$ $\displaystyle T\circ
S\Big{(}D(Tu,Tv)w+\mu(Tv,Tw)u-\mu(Tu,Tw)v\Big{)}$
$\displaystyle+T\Big{(}\rho(TSu,Tv)w+\mu(TSv,Tw)u-\mu(TSu,Tw)v+D(Tu,TSv)w$
$\displaystyle+\mu(Tv,TSw)u-\mu(Tu,TSw)v\Big{)}.$

By Proposition 4.8, the conclusion thus follows. ∎

######  Remark 4.11.

In the context of Lie algebras or associative algebras, for any
$k\in\mathbb{N}$, $T_{k}=N^{k}\circ T$ are relative Rota-Baxter operators and
what is more, $T_{k}$ and $T_{l}$ are compatible. However, this property fails
in the context of $3$-Lie algebras ([35]) and Lie-Yamaguti algebras, which
leads to a different content of the relative Rota-Baxter-Nijenhuis structures
on ternary case.

##  5\. $r$-matrix-Nijenhuis structures and Rota-Baxter-Nijenhuis structures

In this section, we introduce the notions of $r$-matrix-Nijenhuis structures
and Rota-Baxter-Nijenhuis structures on Lie-Yamaguti algebras and investigate
the relationship between them. If $(V;\rho,\mu)$ is a representation of a Lie-
Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$,
then the dual representation of $(V;\rho,\mu)$ is given by

$\Big{(}V^{*};\rho^{*},-\mu^{*}\tau\Big{)},$

where $\rho^{*},~{}\mu^{*}$ are dual to $-\rho,~{}-\mu$ respectively, and
$\tau:\otimes^{2}\mathfrak{g}\to\otimes^{2}\mathfrak{g}$ is the switching map,
i.e.,

$\tau(x\otimes y)=y\otimes x,\quad\forall x\otimes
y\in\otimes^{2}\mathfrak{g}.$

See [27] for ore details about dual representation of Lie-Yamaguti algebras.

######  Example 5.1.

([27]) Let $(\mathfrak{g};\mathrm{ad},\mathfrak{R})$ be the adjoint
representation of a Lie-Yamaguti algebra $\mathfrak{g}$ given in Example 2.6,
then $(\mathfrak{g}^{*};\mathrm{ad}^{*},-\mathfrak{R}^{*}\tau)$ is the dual
representation of $(\mathfrak{g};\mathrm{ad},\mathfrak{R})$, which is called
the coadjoint representation.

First, we introduce the notions of dual Nijenhuis structures and relative
Rota-Baxter-dual-Nijenhuis structures on Lie-Yamaguti algebras.

######  Definition 5.2.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$
be a $\mathsf{LieYRep}$ pair, $N\in\mathfrak{gl}(\mathfrak{g})$,
$S\in\mathfrak{gl}(V)$, and $T:V\longrightarrow\mathfrak{g}$ a relative Rota-
Baxter operator on $\mathfrak{g}$ with respect to $(V;\rho,\mu)$.

  * (i)

A pair $(N,S)$ is called a dual Nijenhuis structure if $(N,S^{*})$ is a
Nijenhuis structure on the $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V^{*};\rho^{*},-\mu^{*}\tau)\Big{)}$.

  * (ii)

If moreover, $T$ and dual Nijenhuis structure $(N,S)$ satifies Eqs. (26)-(28),
the triple $(T,S,N)$ is called a relative Rota-Baxter-dual-Nijenhuis structure
on $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(V;\rho,\mu)\Big{)}$.

Note that a pair $(N,S)$ is a dual Nijenhuis structure is equivalent to that
$N$ is a Nijenhuis operator on
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
and for all $x,y\in\mathfrak{g},v\in V$, the following conditions are
satisfied:

$\displaystyle\rho(Nx)(Sv)$ $\displaystyle=$ $\displaystyle
S\big{(}\rho(Nx)v-\rho(x)(Sv)\Big{)}+\rho(x)S^{2}v,$
$\displaystyle\mu(Nx,Ny)(Sv)$ $\displaystyle=$ $\displaystyle
S\Big{(}\mu(Nx,Ny)v-\mu(Nx,y)(Sv)-\mu(x,Ny)(Sv)+\mu(x,y)(S^{2}v)\Big{)}$
$\displaystyle+\mu(Nx,y)(S^{2}v)+\mu(x,Ny)(S^{2}v)-\mu(x,y)(S^{3}v).$

######  Example 5.3.

Let $N$ be a Nijenhuis operator on a Lie-Yamaguti algebra
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$.
Then the pair $(N,N^{*})$ is a dual Nijenhuis structure on the
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(\mathfrak{g}^{*};\mathrm{ad}^{*},-\mathfrak{R}^{*}\tau)\Big{)}$.

######  Remark 5.4.

Note that in the context of Lie algebras or associative algebras, a relative
Rota-Baxter-Nijenhuis structure $(T,N,S)$ is also a relative Rota-Baxter-dual-
Nijenhuis structure under the condition that $T$ is invertible. However, this
property fails in the the context of $3$-Lie algebras or Lie-Yamaguti algebras
even though $T$ is invertible. This is another different contents of ternary
operations.

The notion of the classical Lie-Yamaguti $r$-matrices was introduced in [37],
and we call it an $r$-matrix in the sequel. Moreover, we proved that a skew-
symmetric $2$-tensor $\pi\in\wedge^{2}\mathfrak{g}$ is an $r$-matrix if and
only if the induced map
$\pi^{\sharp}:\mathfrak{g}^{*}\longrightarrow\mathfrak{g}$ is a relative Rota-
Baxter operator with respect to the coadjoint representation, where
$\pi^{\sharp}$ is defined to be

(34)
$\displaystyle\left\langle\pi^{\sharp}(\alpha),\beta\right\rangle=\pi(\alpha,\beta),\quad\forall\alpha,\beta\in\mathfrak{g}^{*}.$

In the following, we introduce the notion of $r$-matrix-Nijenhuis structures
on Lie-Yamaguti algebras.

######  Definition 5.5.

Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra, $\pi\in\wedge^{2}\mathfrak{g}$ an $r$-matrix, and
$N:\mathfrak{g}\longrightarrow\mathfrak{g}$ a Nijenhuis operator on
$\mathfrak{g}$. A pair $(\pi,N)$ is called an $r$-matrix-Nijenhuis structure
on $\mathfrak{g}$ if for all $\alpha,\beta,\gamma\in\mathfrak{g}^{*}$, they
satisfy

(35) $\displaystyle N\circ\pi^{\sharp}$ $\displaystyle=$
$\displaystyle\pi^{\sharp}\circ N^{*},$ (36)
$\displaystyle~{}[\alpha,\beta]^{N\circ\pi^{\sharp}}$ $\displaystyle=$
$\displaystyle[\alpha,\beta]^{\pi^{\sharp}}_{N^{*}},$ (37)
$\displaystyle~{}\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{N\circ\pi^{\sharp}}$
$\displaystyle=$
$\displaystyle\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}_{N^{*}},$

where $\pi^{\sharp}$ is given by (34).

######  Remark 5.6.

It is easy to see that an $r$-matrix-Nijenhuis structure $(\pi,N)$ on
$\mathfrak{g}$ is a special relative Rota-Baxter-dual-Nijenhuis structure on
the $\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(\mathfrak{g}^{*},\mathrm{ad}^{*},-\mathfrak{R}^{*}\tau)\Big{)}$:

$\Big{(}T=\pi^{\sharp},S=N^{*},N\Big{)}.$

Similar to the case of $r$-matrix-Nijenhuis structures, we give the notion of
Rota-Baxter-Nijenhuis structures as follows.

######  Definition 5.7.

Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a Lie-Yamaguti algebra. Suppose that $R:\mathfrak{g}\to\mathfrak{g}$ is a
Rota-Baxter operator and $N:\mathfrak{g}\to\mathfrak{g}$ is a Nijenhuis
operator on $\mathfrak{g}$. A pair $(R,N)$ is a Rota-Baxter-Nijenhuis
structure on $\mathfrak{g}$ if for all $x,y,z\in\mathfrak{g}$, they satisfy

(38) $\displaystyle N\circ R$ $\displaystyle=$ $\displaystyle R\circ N,$ (39)
$\displaystyle~{}[x,y]^{N\circ R}$ $\displaystyle=$
$\displaystyle[x,y]^{R}_{N},$ (40) $\displaystyle~{}\left\llbracket
x,y,z\right\rrbracket^{N\circ R}$ $\displaystyle=$
$\displaystyle\left\llbracket x,y,z\right\rrbracket^{R}_{N},$

######  Remark 5.8.

It is not hard to see that a Rota-Baxter-Nijenhuis structure $(R,N)$ on
$\mathfrak{g}$ is a special relative Rota-Baxter-Nijenhuis structure on the
$\mathsf{LieYRep}$ pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),(\mathfrak{g};\mathrm{ad},\mathfrak{R})\Big{)}$:

$\Big{(}T=R,S=N,N\Big{)}.$

######  Definition 5.9.

([16]) A quadratic Lie-Yamaguti algebra is a pair
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),B\Big{)}$,
where
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
is a Lie-Yamaguti algebra, and $B\in\otimes^{2}\mathfrak{g}^{*}$ is a
nondegenerate symmetric bilinear form satisfying the following invariant
conditions

$\displaystyle B([x,y],z)$ $\displaystyle=$ $\displaystyle-B(y,[x,z]),$
$\displaystyle B(\left\llbracket x,y,z\right\rrbracket,w)$ $\displaystyle=$
$\displaystyle B(x,\left\llbracket w,z,y\right\rrbracket),\quad\forall
x,y,z\in\mathfrak{g}.$

Then $B$ induces a bijection $B^{\sharp}:\mathfrak{g}^{*}\to\mathfrak{g}$
defined by

$B(x,y)=\langle(B^{\sharp})^{-1}(x),y\rangle,\quad\forall x,y\in\mathfrak{g}.$

In [27], by the invariance of $B$, it is proved that

(41) $\displaystyle B^{\sharp}\Big{(}\mathrm{ad}^{*}_{x}\alpha\Big{)}$
$\displaystyle=$
$\displaystyle\mathrm{ad}_{x}\Big{(}B^{\sharp}(\alpha)\Big{)},$ (42)
$\displaystyle~{}-B^{\sharp}\Big{(}\mathfrak{R}^{*}(y,x)\alpha\Big{)}$
$\displaystyle=$
$\displaystyle\mathfrak{R}(x,y)\Big{(}B^{\sharp}(\alpha)\Big{)},\quad\forall\alpha\in\mathfrak{g}^{*}.$

Moreover, it is easy to see that

(43) $\displaystyle B^{\sharp}\Big{(}\mathfrak{L}^{*}(x,y)\alpha\Big{)}$
$\displaystyle=$
$\displaystyle\mathfrak{L}(x,y)\Big{(}B^{\sharp}(\alpha)\Big{)},\quad\forall\alpha\in\mathfrak{g}^{*}.$

A skew-symmetric endomorphism of $(\mathfrak{g},B)$ is a linear map
$R:\mathfrak{g}\to\mathfrak{g}$ such that $R\circ
B^{\sharp}:\mathfrak{g}^{*}\to\mathfrak{g}$ is skew-symmetric in the sense
that

$\displaystyle\langle\alpha,R\circ
B^{\sharp}(\beta)\rangle+\langle\beta,R\circ
B^{\sharp}(\alpha)\rangle=0,\quad\forall\alpha,\beta\in\mathfrak{g}^{*},$

The following theorem illustrates the relationship between $r$-matrix-
Nijenhuis structures and Rota-Baxter-Nijenhuis structures.

######  Theorem 5.10.

Let
$\Big{(}(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket),B\Big{)}$
be a quadratic Lie-Yamaguti algebra,
$N:\mathfrak{g}\longrightarrow\mathfrak{g}$ a Nijenhuis operator on
$\mathfrak{g}$ and $R:\mathfrak{g}\longrightarrow\mathfrak{g}$ a skew-
symmetric endomorphism of $(\mathfrak{g},B)$. Set $\pi^{\sharp}=R\circ
B^{\sharp}$ and assume that $B$ and $N$ are compatible, i.e.,

(44) $\displaystyle B^{\sharp}\circ N^{*}=N\circ B^{\sharp}.$

Then we have

  * (i)

If $(R,N)$ is a Rota-Baxter-Nijenhuis structure on $\mathfrak{g}$, then
$(\pi,N)$ is an $r$-matrix-Nijenhuis structure on $\mathfrak{g}$.

  * (ii)

If $(\pi,N)$ is an $r$-matrix-Nijenhuis structure on the Lie-Yamaguti algebra
$\mathfrak{g}$, then the pair $(R=\pi^{\sharp}\circ(B^{\sharp})^{-1},N)$ is a
Rota-Baxter-Nijenhuis structure on $\mathfrak{g}$.

###### Proof.

First, by (38) and (44), it is sasy to see that
$N\circ\pi^{\sharp}=\pi^{\sharp}\circ N^{*}$. Next, we show that
$\pi^{\sharp}:=R\circ B^{\sharp}$ is a relative Rota-Baxter operator with
respect to the coadjoint representation, which is equivalent to that
$\pi\in\wedge^{2}\mathfrak{g}$ is an $r$-matrix. Indeed, by the invariance of
$B$, we have

$\displaystyle\pi^{\sharp}\Big{(}\mathrm{ad}^{*}_{\pi^{\sharp}(\alpha)}\beta-\mathrm{ad}^{*}_{\pi^{\sharp}(\beta)}\alpha\Big{)}$
$\displaystyle=$ $\displaystyle R\circ
B^{\sharp}\Big{(}\mathrm{ad}^{*}_{\pi^{\sharp}(\alpha)}\beta-\mathrm{ad}^{*}_{\pi^{\sharp}(\beta)}\alpha\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{inva1}}}{{=}}$ $\displaystyle
R\Big{(}\mathrm{ad}_{\pi^{\sharp}(\alpha)}B^{\sharp}(\beta)-\mathrm{ad}_{\pi^{\sharp}(\beta)}B^{\sharp}(\alpha)\Big{)}$
$\displaystyle=$ $\displaystyle R\Big{(}[R\circ
B^{\sharp}(\alpha),B^{\sharp}(\beta)]+[B^{\sharp}(\alpha),R\circ
B^{\sharp}(\beta)]\Big{)}$ $\displaystyle=$ $\displaystyle[R\circ
B^{\sharp}(\alpha),R\circ
B^{\sharp}(\beta)]=[\pi^{\sharp}(\alpha),\pi^{\sharp}(\beta)],$

and similarly

$\displaystyle\pi^{\sharp}\Big{(}\mathfrak{L}^{*}(\pi^{\sharp}(\alpha),\pi^{\sharp}(\beta))\gamma-\mathfrak{R}^{*}(\pi^{\sharp}(\gamma),\pi^{\sharp}(\beta))\alpha+\mathfrak{R}^{*}(\pi^{\sharp}(\gamma),\pi^{\sharp}(\alpha))\beta\Big{)}$
$\displaystyle=$ $\displaystyle
R\Big{(}\left\llbracket\pi^{\sharp}(\alpha),\pi^{\sharp}(\beta),B^{\sharp}(\gamma)\right\rrbracket+\left\llbracket
B^{\sharp}(\alpha),\pi^{\sharp}(\beta),\pi^{\sharp}(\gamma)\right\rrbracket-\left\llbracket
B^{\sharp}(\beta),\pi^{\sharp}(\alpha),\pi^{\sharp}(\gamma)\right\rrbracket\Big{)}$
$\displaystyle=$ $\displaystyle\left\llbracket R\circ
B^{\sharp}(\alpha),R\circ B^{\sharp}(\beta),R\circ
B^{\sharp}(\gamma)\right\rrbracket=\left\llbracket\pi^{\sharp}(\alpha),\pi^{\sharp}(\beta),\pi^{\sharp}(\gamma)\right\rrbracket.$

What is left is to show
$[\alpha,\beta]^{N\circ\pi^{\sharp}}=[\alpha,\beta]^{\pi^{\sharp}}_{N^{*}}$
and
$\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{N\circ\pi^{\sharp}}=\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}_{N^{*}}$.

Let
$\alpha=(B^{\sharp})^{-1}(x),~{}\beta=(B^{\sharp})^{-1}(y),~{}\gamma=(B^{\sharp})^{-1}(z)$,
by (41)-(43), we have

$\displaystyle~{}[\alpha,\beta]^{\pi^{\sharp}}$ $\displaystyle=$
$\displaystyle\mathrm{ad}^{*}_{\pi^{\sharp}((B^{\sharp})^{-1}(x))}(B^{\sharp})^{-1}(y)-\mathrm{ad}^{*}_{\pi^{\sharp}((B^{\sharp})^{-1}(y))}(B^{\sharp})^{-1}(x)$
$\displaystyle\stackrel{{\scriptstyle\eqref{inva1}}}{{=}}$
$\displaystyle\mathrm{ad}^{*}_{R(x)}(B^{\sharp})^{-1}(y)-\mathrm{ad}^{*}_{R(y)}(B^{\sharp})^{-1}(x)$
$\displaystyle=$
$\displaystyle(B^{\sharp})^{-1}\Big{(}[R(x),y]-[R(y),x]\Big{)}=(B^{\sharp})^{-1}[x,y]^{R},$

and

$\displaystyle\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle=$
$\displaystyle\mathfrak{L}^{*}\Big{(}\pi^{\sharp}(B^{\sharp})^{-1}(x),\pi^{\sharp}(B^{\sharp})^{-1}(y)\Big{)}\Big{(}(B^{\sharp})^{-1}(z)\Big{)}-\mathfrak{R}^{*}\Big{(}\pi^{\sharp}(B^{\sharp})^{-1}(z),\pi^{\sharp}(B^{\sharp})^{-1}(y)\Big{)}\Big{(}(B^{\sharp})^{-1}(x)\Big{)}$
$\displaystyle+\mathfrak{R}^{*}\Big{(}(\pi^{\sharp}(B^{\sharp})^{-1}(z),\pi^{\sharp}(B^{\sharp})^{-1}(x)\Big{)}\Big{(}(B^{\sharp})^{-1}(y)\Big{)}$
$\displaystyle=$
$\displaystyle\mathfrak{L}^{*}\Big{(}R(x),R(y)\Big{)}\Big{(}(B^{\sharp})^{-1}(z)\Big{)}-\mathfrak{R}^{*}\Big{(}R(z),R(y)\Big{)}\Big{(}(B^{\sharp})^{-1}(x)\Big{)}+\mathfrak{R}^{*}\Big{(}R(z),R(x)\Big{)}\Big{(}(B^{\sharp})^{-1}(y)\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{inva2},\eqref{inva3}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\Big{(}\left\llbracket
R(x),R(y),z\right\rrbracket+\left\llbracket
x,R(y),R(z)\right\rrbracket-\left\llbracket
y,R(x),R(z)\right\rrbracket\Big{)}=(B^{\sharp})^{-1}\left\llbracket
x,y,z\right\rrbracket^{R}.$

Thus we have

(45)
$\displaystyle~{}[(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y)]^{\pi^{\sharp}}$
$\displaystyle=$ $\displaystyle(B^{\sharp})^{-1}[x,y]^{R},$ (46)
$\displaystyle\left\llbracket(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y),(B^{\sharp})^{-1}(z)\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle=$ $\displaystyle(B^{\sharp})^{-1}\left\llbracket
x,y,z\right\rrbracket^{R}.$

Hence, by (44), (45), and (46), we have

$\displaystyle[\alpha,\beta]^{N\circ\pi^{\sharp}}-[\alpha,\beta]^{\pi^{\sharp}}_{N^{*}}=[\alpha,\beta]^{N\circ\pi^{\sharp}}-\Big{(}[N^{*}(\alpha),\beta]^{\pi^{\sharp}}+[\alpha,N^{*}(\beta)]^{\pi^{\sharp}}-N^{*}[\alpha,\beta]^{\pi^{\sharp}}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{con1}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\Big{(}[x,y]^{N\circ
R}\Big{)}-\Big{(}[N^{*}((B^{\sharp})^{-1}(x)),(B^{\sharp})^{-1}(y)]^{\pi^{\sharp}}+[(B^{\sharp})^{-1}(x),N^{*}((B^{\sharp})^{-1}(y))]^{\pi^{\sharp}}$
$\displaystyle-N^{*}[(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y)]^{\pi^{\sharp}}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{app}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\Big{(}[x,y]^{N\circ
R}\Big{)}-\Big{(}[(B^{\sharp})^{-1}N(x),(B^{\sharp})^{-1}(y)]^{\pi^{\sharp}}+[(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}N(y)]^{\pi^{\sharp}}$
$\displaystyle-N^{*}[(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y)]^{\pi^{\sharp}}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{con1}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\Big{(}[x,y]^{N\circ
R}\Big{)}-\Big{(}(B^{\sharp})^{-1}[Nx,y]^{R}+(B^{\sharp})^{-1}[x,Ny]^{R}-N^{*}(B^{\sharp})^{-1}[x,y]^{R}\Big{)}$
$\displaystyle=$ $\displaystyle(B^{\sharp})^{-1}\Big{(}[x,y]^{N\circ
R}\Big{)}-(B^{\sharp})^{-1}\Big{(}[Nx,y]^{R}+[x,Ny]^{R}-N[x,y]^{R}\Big{)}$
$\displaystyle=$ $\displaystyle(B^{\sharp})^{-1}\Big{(}[x,y]^{N\circ
R}-[x,y]^{R}_{N}\Big{)}=0,$

and

$\displaystyle\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{N\circ\pi^{\sharp}}-\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}_{N^{*}}$
$\displaystyle=$
$\displaystyle\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{N\circ\pi^{\sharp}}-\Big{(}\left\llbracket
N^{*}(\alpha),N^{*}(\beta),\gamma\right\rrbracket^{\pi^{\sharp}}+\left\llbracket
N^{*}(\alpha),\beta,N^{*}(\gamma)\right\rrbracket^{\pi^{\sharp}}+\left\llbracket\alpha,N^{*}(\beta),N^{*}(\gamma)\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle-N^{*}\big{(}\left\llbracket
N^{*}(\alpha),\beta,\gamma\right\rrbracket^{\pi^{\sharp}}+\left\llbracket\alpha,N^{*}(\beta),\gamma\right\rrbracket^{\pi^{\sharp}}+\left\llbracket\alpha,\beta,N^{*}(\gamma)\right\rrbracket^{\pi^{\sharp}}\big{)}+(N^{*})^{2}\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{con2}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\left\llbracket x,y,z\right\rrbracket^{N\circ
R}-\Big{(}\left\llbracket(B^{\sharp})^{-1}N(x),(B^{\sharp})^{-1}N(y),(B^{\sharp})^{-1}(z)\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle+\left\llbracket(B^{\sharp})^{-1}N(x),(B^{\sharp})^{-1}(y),(B^{\sharp})^{-1}N(z)\right\rrbracket^{\pi^{\sharp}}+\left\llbracket(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}N(y),(B^{\sharp})^{-1}N(z)\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle-N^{*}\big{(}\left\llbracket(B^{\sharp})^{-1}N(x),(B^{\sharp})^{-1}(y),(B^{\sharp})^{-1}(z)\right\rrbracket^{\pi^{\sharp}}+\left\llbracket(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}N(y),(B^{\sharp})^{-1}(z)\right\rrbracket^{\pi^{\sharp}}$
$\displaystyle+\left\llbracket(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y),(B^{\sharp})^{-1}N(z)\right\rrbracket^{\pi^{\sharp}}\big{)}+(N^{*})^{2}\left\llbracket(B^{\sharp})^{-1}(x),(B^{\sharp})^{-1}(y),(B^{\sharp})^{-1}(z)\right\rrbracket^{\pi^{\sharp}}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle\eqref{con2}}}{{=}}$
$\displaystyle(B^{\sharp})^{-1}\left\llbracket x,y,z\right\rrbracket^{N\circ
R}-\Big{(}(B^{\sharp})^{-1}\left\llbracket
N(x),N(y),z\right\rrbracket^{R}+(B^{\sharp})^{-1}\left\llbracket
N(x),y,N(z)\right\rrbracket^{R}+(B^{\sharp})^{-1}\left\llbracket
x,N(y),N(z)\right\rrbracket^{R}$
$\displaystyle-N^{*}\circ(B^{\sharp})^{-1}\big{(}\left\llbracket
N(x),y,z\right\rrbracket^{R}+\left\llbracket
x,N(y),z\right\rrbracket^{R}+\left\llbracket
x,y,N(z)\right\rrbracket^{R}\big{)}+(N^{*})^{2}(B^{\sharp})^{-1}\left\llbracket
x,y,z\right\rrbracket^{R}\Big{)}$ $\displaystyle=$
$\displaystyle(B^{\sharp})^{-1}\left\llbracket x,y,z\right\rrbracket^{N\circ
R}-(B^{\sharp})^{-1}\Big{(}\left\llbracket
N(x),N(y),z\right\rrbracket^{R}+\left\llbracket
N(x),y,N(z)\right\rrbracket^{R}+\left\llbracket
x,N(y),N(z)\right\rrbracket^{R}$ $\displaystyle-N\big{(}\left\llbracket
N(x),y,z\right\rrbracket^{R}+\left\llbracket
x,N(y),z\right\rrbracket^{R}+\left\llbracket
x,y,N(z)\right\rrbracket^{R}\big{)}+N^{2}\left\llbracket
x,y,z\right\rrbracket^{R}\Big{)}$ $\displaystyle=$
$\displaystyle(B^{\sharp})^{-1}\left\llbracket x,y,z\right\rrbracket^{N\circ
R}-(B^{\sharp})^{-1}\left\llbracket
x,y,z\right\rrbracket^{R}_{N}=(B^{\sharp})^{-1}\Big{(}\left\llbracket
x,y,z\right\rrbracket^{N\circ R}-\left\llbracket
x,y,z\right\rrbracket^{R}_{N}\Big{)}=0,$

which implies that

$[\alpha,\beta]^{N\circ\pi^{\sharp}}=[\alpha,\beta]^{\pi^{\sharp}}_{N^{*}}\quad{\rm
and}\quad\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{N\circ\pi^{\sharp}}=\left\llbracket\alpha,\beta,\gamma\right\rrbracket^{\pi^{\sharp}}_{N^{*}}.$

Thus $(\pi,N)$ is an $r$-matrix-Nijenhuis structure. This proves (i).

(ii) can be proved similarly. ∎

We give some examples of Rota-Baxter-Nijenhuis structures on Lie-Yamaguti
algebras to end up this section.

######  Example 5.11.

Let $\mathfrak{g}=C^{\infty}([0,1])$ endowed with the following operations

$\displaystyle[f,g](x)$ $\displaystyle=$ $\displaystyle
f(x)g^{\prime}(x)-g(x)f^{\prime}(x),$ $\displaystyle\left\llbracket
f,g,h\right\rrbracket(x)$ $\displaystyle=$ $\displaystyle
f(x)g^{\prime}(x)h^{\prime}(x)-g(x)f^{\prime}(x)h^{\prime}(x)-h(x)\Big{(}f(x)g^{\prime\prime}(x)-g(x)f^{\prime\prime}(x)\Big{)},\quad\forall
x\in[0,1],$

for all $f,g,h\in\mathfrak{g}$. Then
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
forms a Lie-Yamaguti algebra. For all $f\in C^{\infty}([0,1])$, the integral
operator $R:\mathfrak{g}\longrightarrow\mathfrak{g}$ defined to be

$R(f)(x):=\int_{0}^{x}f(t){\mathrm{d}}t,\quad\forall x\in[0,1]$

is a Rota-Baxter operator on $\mathfrak{g}$, and
$N:\mathfrak{g}\longrightarrow\mathfrak{g}$ given by

$N(f)(x):=\lambda f(x),\quad\forall x\in[0,1]$

is a Nijenhuis operator for $\lambda\in\mathbb{K}$. Then $(R,N)$ is a Rota-
Baxter-Nijenhuis structure on $\mathfrak{g}$.

######  Example 5.12.

Let
$(\mathfrak{g},[\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$
be a $2$-dimensional Lie-Yamaguti algebra with a basis $\\{e_{1},e_{2}\\}$,
where $([\cdot,\cdot],\left\llbracket\cdot,\cdot,\cdot\right\rrbracket)$ is
defined to be

$[e_{1},e_{2}]=e_{1},\quad\left\llbracket
e_{1},e_{2},e_{2}\right\rrbracket=e_{1}.$

Then $(R,N)$ is a Rota-Baxter-Nijenhuis structure on $\mathfrak{g}$, where

$\displaystyle R=\begin{pmatrix}0&a\\\ 0&0\end{pmatrix},\quad
N=\begin{pmatrix}0&\lambda\\\ 0&0\end{pmatrix}.$

## References

  * [1] E. Abdaoui, S. Mabrouk, and A. Makhlouf, Rota-Baxter operators on pre-Lie superalgebras, _Bull. Malays. Math. Sci. Soc._ 42 (2019), no.4, 1567-1606.
  * [2] C. Bai, O. Bellier, L. Guo, and X. Ni, Splitting of operations, Manin products, and Rota-Baxter operators, _Int. Math. Res. Not._ (2013), no. 3, 485-524.
  * [3] C. Bai, L. Guo, and Y. Sheng, Bialgebras, the classical Yang-Baxter equations and Manin triples for 3-Lie algebras, _Adv. Theor. Math. Phys._ 23 (2019) 27-74.
  * [4] R. Bai, L. Guo, J. Li, and Y. Wu, Rota-Baxter 3-Lie algebras, _J. Math. Phys._ 54 (2013) 064504, 14 pp.
  * [5] G. Baxter, An analytic problem whose solution follows from a simple algebraic identity, _Pacific J. Math._ 10 (1960), 731-742.
  * [6] P. Benito, M. Bremner, and S. Madariaga, Symmetric matrices, orthogonal Lie algebras and Lie-Yamaguti algebras, _Linear Multilinear Algebra_ 63 (2015) 1257-1281.
  * [7] P. Benito, C. Draper, and A. Elduque, Lie-Yamaguti algebras related to $\mathcal{G}_{2}$, _J. Pure Appl. Algebra_ 202 (2005) 22-54.
  * [8] P. Benito, A. Elduque, and F. Mart$\rm\acute{i}$n-Herce, Irreducible Lie-Yamaguti algebras, _J. Pure Appl. Algebra_ 213 (2009) 795-808.
  * [9] P. Benito, A. Elduque, and F. Mart$\rm\acute{i}$n-Herce, Irreducible Lie-Yamaguti algebras of generic type, _J. Pure Appl. Algebra_ 215 (2011) 108-130.
  * [10] I. Dorfman, Dirac structures and integrability of nonlinear evolution equations, Wiley, Chichester (1993).
  * [11] B. Fuchssteiner and A. S. Fokas, Symplectic structures, their B$\rm\ddot{a}$cklund transformations and hereditary symmetries, _Physica D_ 4 (1981/82) 47-66.
  * [12] L. Guo, An introduction to Rota-Baxter algebra. Surveys of Modern Mathematics, 4. International Press, Somerville, MA; Higher Education Press, Beijing, 2012. xii+226 pp.
  * [13] Y. Hu, J. Liu, and Y. Sheng, Kupershmidt-(dual-)Nijenhuis structures on a Lie algebra with a representation. _J. Math. Phys._ 59 (2018) 081702.
  * [14] J. Liu, Y. Sheng, Y. Zhou, and C. Bai, Nijenhuis operators on $n$-Lie algebras. _Commun. Theor. Phys._ 65 (2016), 659-670.
  * [15] J. Liu and Q. Wang, Pre-Lie analogues of Poisson-Nijenhuis structures and Maurer-Cartan equations, arXiv:2004.02098.
  * [16] M. Kikkawa, On Killing-Ricci forms of Lie triple algebras, _Pacific J. Math._ 96 (1981), no. 1, 153-161.
  * [17] M. K. Kinyon and A. Weinstein, Leibniz algebras, Courant algebroids and multiplications on reductive homogeneous spaces, _Amer. J. Math._ 123 (2001) 525-550.
  * [18] Y. Kosmann-Schwarzbach and F. Magri, Poisson-Nijenhuis structures, _Ann. Inst. Henri Poincare_ , Sect. A 53 (1990) 35-81.
  * [19] Y. Kosmann-Schwarzbach and V. Rubtsov, Compatible structures on Lie algebroids and Monge-Amp$\rm\grave{e}$re operators, _Acta. Appl. Math._ 109 (2010) 101-135.
  * [20] B. A. Kupershmidt, What a classical $r$-matrix really is, _J. Nonlinear Math. Phys._ 6 (1999) no.4, 448-488.
  * [21] J. Liu, C. Bai, and Y. Sheng, Compatible ${\mathcal{O}}$-operators on bimodules over associative algebras. _J. Algebra_ 532 (2019) 80-118.
  * [22] F. Magri and C. Morosi, A geometrical characterization of integrable hamiltonian systems through the theory of Poisson-Nijenhuis manifolds, Quaderno S/19, Milan, 1984.
  * [23] A. Nijenhuis and R. W. Richardson, Deformations of Lie algebra structures, _J. Math. Mech._ 17 (1967) 89-105.
  * [24] K. Nomizu, Invariant affine connections on homogeneous spaces, _Amer. J. Math._ 76 (1954) 33-65.
  * [25] Z. Ravanpak, A. Rezaei-Aghdam, and G. Haghighatdoost, Invariant Poisson-Nijenhuis structures on Lie groups and classification, _Int. J. Geom. Methods Mod. Phys._ 15(4) (2018) 1850059.
  * [26] Y. Sheng, J. Zhao, and Y. Zhou, Nijenhuis operators, product structures and complex structures on Lie-Yamaguti algebras, _J. Algebra Appl._ Vol. 20, No. 8 (2021) 2150146, 22 pages.
  * [27] Y. Sheng and J. Zhao, Relative Rota-Baxter operators and symplectic structures on Lie-Yamaguti algebras, to appear in _Commun. Algebra_.
  * [28] N. Takahashi, Modules over quadratic spaces and representations of Lie-Yamaguti algebras, arXiv:2010.05564.
  * [29] R. Tang, C. Bai, L. Guo and Y. Sheng, Deformations and their controlling cohomologies of ${\mathcal{O}}$-operators. _Commun. Math. Phys._ 368 (2019) 665-700.
  * [30] K. Uchino, Quantum analogy of Poisson geometry, related dendriform algebras and Rota-Baxter operators. _Lett. Math. Phys._ 85 (2008) 91-109.
  * [31] K. Yamaguti, On the Lie triple system and its generalization, _J. Sci. Hiroshima Univ. Ser. A_ 21 (1957/1958) 155-160.
  * [32] K. Yamaguti, On cohmology groups of general Lie triple systems, _Kumamoto J. Sci. A_ 8 (1967/1969) 135-146.
  * [33] K. Yamaguti, On the cohomology space of Lie triple systems, _Kumamoto J. Sci. A_ 5(1) (1960) 44-52.
  * [34] T. Zhang and J. Li, Deformations and extension of Lie-Yamaguti algebras, _Linear Multilinear Algebra._ 63 (2015) 2212-2231.
  * [35] J. Zhao, J. Liu, and Y. Sheng, Cohomologies and relative Rota-Baxter-Nijenhuis structures on $3$-$\mathsf{LieRep}$ pairs, to appear in _Linear Multilinear Algebra_.
  * [36] J. Zhao and Y. Qiao, Cohomology and deformations of relative Rota-Baxter operators on Lie-Yamaguti algebras, arXiv: 2204.04872.
  * [37] J. Zhao and Y. Qiao, The classical Lie-Yamaguti Yang-Baxter equation and Lie-Yamaguti bialgebras, Preprint.