# Powers and roots of partial isometric covariant representations

Dimple Saini* Centre for mathematical and Financial Computing, Department of
Mathematics, The LNM Institute of Information Technology, Rupa ki Nangal,
Post-Sumel, Via-Jamdoli Jaipur-302031, (Rajasthan) India
<EMAIL_ADDRESS><EMAIL_ADDRESS>,  Harsh Trivedi  Centre for
mathematical and Financial Computing, Department of Mathematics, The LNM
Institute of Information Technology, Rupa ki Nangal, Post-Sumel, Via-Jamdoli
Jaipur-302031, (Rajasthan) India<EMAIL_ADDRESS><EMAIL_ADDRESS>and  Shankar Veerabathiran  Institute of Mathematical
Sciences, A CI of Homi Bhabha National Institute, CIT Campus, Taramani,
Chennai, 600113, Tamilnadu, INDIA<EMAIL_ADDRESS>
###### Abstract.

Isometric covariant representations play an important role in the study of
Cuntz-Pimsner algebras. In this article, we study partial isometric covariant
representations and explore under what conditions powers and roots of partial
isometric covariant representations are also partial isometric covariant
representations.

###### Key words and phrases:

Hilbert $C^{*}$-modules, covariant representations, Structure theorems of
partial isometries, tuples of operators

###### 2010 Mathematics Subject Classification:

46L08, 47A65, 47L55, 47L80.

##  1\. Introduction

The classical result of Wold [24] asserts that: A given isometry on a Hilbert
space is either a shift, or a unitary, or breaks uniquely as a direct summand
of both. Let us denote by $B(\mathcal{H})$ the $C^{*}$-algebra of all bounded
linear operators on a Hilbert space $\mathcal{H},$ and for $V\in
B(\mathcal{H})$ we denote the kernel of $V$ and the range of $V$ by $N(V)$ and
$R(V)$, respectively. A bounded linear operator $V$ on a Hilbert space is said
to be partial isometry if it is isometry on the orthogonal complement of
$N({V}).$ It is clear that $V$ is a partial isometry if and only if $VV^{*}$
is a (final) projection onto $R(V)$ if and only if $V^{*}V$ is a (initial)
projection onto $R(V^{*}).$ It is well known that the sum and multiplication
of partial isometries need not be partial isometries. Murray and von Neumann
[18] discussed the classification of factors using the sum of partial
isometries. In the following result Erdelyi [4] studied when the product of
two partial isometries is also a partial isometry:

######  Theorem 1.1.

Suppose that $V_{1}$ and $V_{2}$ are two partial isometries acting on a
Hilbert space $\mathcal{H}.$ Let $T=V_{1}V_{2}.$ Then the following conditions
are equivalent:

  * (1)

$T$ is a partial isometry.

  * (2)

Initial space of $V_{1},$ $R(V_{1}^{*}),$ is invariant under $P_{R(V_{2})},$
that is,

$V_{2}V_{2}^{*}R(V_{1}^{*})\subseteq R(V_{1}^{*}).$

  * (3)

$R(V_{2})$ is invariant under $P_{R(V_{1}^{*})},$ that is,

$V_{1}^{*}V_{1}R(V_{2})\subseteq R(V_{2}).$

  * (4)

The product $P_{R(V_{1}^{*})}P_{R(V_{2})}$ is idempotent operator.

A bounded linear operator $V$ on a Hilbert space is called power partial
isometry if $V^{n}$ is a partial isometry for all $n\geq 1.$ Halmos and Wallen
[13] explored power partial isometries and proved that every power partial
isometry uniquely decomposes into a direct sum of a pure isometry, a unitary
operator, a pure co-isometry and a truncated shift. Ezzahraoui, Mbekhta, Salhi
and Zerouali [5] characterized the powers of a partial isometry in terms of
orthogonal complement of kernels. For $n\in\mathbb{N},$ we denote
$I_{n}:=\\{1,2,\dots,n\\}.$

######  Theorem 1.2.

Suppose that $V$ is a partial isometry acting on a Hilbert space
$\mathcal{H}.$ Then the following conditions are equivalent:

  * (1)

$V^{k}$ is a partial isometry for all $k\in I_{n}.$

  * (2)

$V(N(V^{k})^{\perp})\subseteq N(V^{k-1})^{\perp}$ for all $k\in I_{n}.$

Moreover, if $V$ is a power partial isometry if and only if
$V(N(V^{k+1})^{\perp})\subseteq N(V^{k})^{\perp}$ for all $k\geq 1.$

Gupta [12] analyzed the converse question and proved that if $V\in
B(\mathcal{H})$ is a contraction with $N(V^{2})=N(V),$ and if $V^{n}$ is a
partial isometry for some $n\geq 2,$ then $V$ is a partial isometry.
Ezzahraoui, Mbekhta, Salhi and Zerouali [5] generalized the Gupta’s result in
the following way:

######  Theorem 1.3.

Suppose that $V$ is a contraction on a Hilbert space $\mathcal{H}.$ Let
$V^{n}$ is a partial isometry for some $n\geq 2.$ Then the following
conditions are equivalent:

  * (1)

$V$ is a partial isometry.

  * (2)

    * (a)

For every $h\in N(V^{n})\ominus N(V),$ we have $\|Vh\|=\|h\|.$

    * (b)

$V(N(V^{n})^{\perp})$ is orthogonal to $V(N(V^{n})\ominus N(V)).$

In a noncommutative context, Cuntz [3] investigated a $C^{*}$-algebra,
famously known as the Cuntz algebra, which is generated by row isometry. The
tuple $(V_{1},\dots,V_{n})$ of isometries on a Hilbert space $\mathcal{H}$ is
said to be row isometry if $V_{i}$’s have orthogonal range. Frazho [10]
introduced the Wold decomposition for two isometries with orthogonal range,
and Popescu [20] extended this decomposition to encompass the scenario of an
infinite sequence of isometries with orthogonal final spaces. Using isometric
covariant representations of $C^{*}$-correspondences Pimsner in [19] extended
the construction of Cuntz-Kreiger algebras.

Muhly and Solel derived the Wold decomposition for isometric covariant
representations in [17], which is an analogue of Popescu’s Wold decomposition
[20] for row isometry. In [23], Viselter studied powers of partial isometric
covariant representations and introduced relatively isometric covariant
representations to explore conditions for a covariant representation of a
subproduct system to extend to a $C^{*}$-representation of the Toeplitz
algebra. Our aim in this article is to study conditions so that powers and
roots of partial isometric covariant representations are also partial
isometric covariant representations.

The plan of the article is as follows: In Section 1, we recall the basic setup
and definitions. In Section 2, we study hypothesis under which the
multiplication of partial isometric representations of $C^{*}$-correspondences
becomes partial isometric representations. In Section 3, we analyze under
which assumptions the powers and roots of partial isometric representations
become partial isometric representations.

###  1.1. Preliminaries and Notations

Let $E$ be a Hilbert $C^{*}$-module over a $C^{*}$-algebra $\mathcal{A}.$ We
say that $E$ is a $C^{*}$-correspondence over $\mathcal{A}$ if it is equipped
with a left $\mathcal{A}$-module structure induced by a non zero
$*$-homomorphism $\phi:\mathcal{A}\to B^{a}(E),$ that is,

$a\xi:=\phi(a)\xi\quad\quad\xi\in E,a\in\mathcal{A},$

where $B^{a}(E)$ denotes the algebra of all adjointable operators on $E.$ If
the closed linear span of $\\{\langle u,v\rangle:u,v\in E\\}$ equals
$\mathcal{A},$ then we say that $E$ is full. In this article, every
$*$-homomorphism considered is nondegenerate.

######  Definition 1.4.

Assume $\sigma:\mathcal{A}\to B(\mathcal{H})$ to be a representation and
$V:E\to B(\mathcal{H})$ to be a linear map. The pair $(\sigma,V)$ is called a
covariant representation of $E$ on $\mathcal{H}$ (cf. [16]) if

$V(a\xi c)=\sigma(a)V(\xi)\sigma(c)\quad\quad(\xi\in E,a,c\in\mathcal{A}).$

A covariant representation $(\sigma,V)$ is called completely bounded covariant
(or simply say, c.b.c.) representation if $V$ is completely bounded. Moreover,
$(\sigma,V)$ is called isometric if
$V(\zeta)^{*}V(\xi)=\sigma(\langle\zeta,\xi\rangle)$ for all $\zeta,\xi\in E.$

The next result from Muhly and Solel [16] which is important to classify the
c.b.c. representations of $C^{*}$-correspondences.

######  Lemma 1.5.

The map $(\sigma,V)\mapsto\widetilde{V}$ provides a one-to-one correspondence
between the set of all completely contractive covariant (respectively, c.b.c.)
representations $(\sigma,V)$ of $E$ on $\mathcal{H}$ and the set of all
contractive (respectively, bounded) linear operators
$\widetilde{V}:E\otimes_{\sigma}\mathcal{H}\to\mathcal{H}$ defined by

$\widetilde{V}(\xi\otimes h)=V(\xi)h\quad\quad(h\in\mathcal{H},\xi\in E),$

with $\widetilde{V}(\phi(a)\otimes I_{\mathcal{H}})=\sigma(a)\widetilde{V}$
for every $a\in\mathcal{A}$. Further, $(\sigma,V)$ is isometric if and only if
$\widetilde{V}$ is isometry.

For all $m\in\mathbb{N}_{0}$, $E^{\otimes
m}=E\otimes_{\phi}\dots\otimes_{\phi}E$ ($m$-times) (with $E^{\otimes
0}=\mathcal{A}$) is a $C^{*}$-correspondence over $\mathcal{A}$, where the
left action of $\mathcal{A}$ on $E^{\otimes m}$ given by

$\phi_{m}(a)(\xi_{1}\otimes\dots\otimes\xi_{m})=a\xi_{1}\otimes\dots\otimes\xi_{m},\>\>\>\>\xi_{i}\in
E,a\in\mathcal{A}.$

For $m\in\mathbb{N},$ define $\widetilde{V}_{m}:E^{\otimes
m}\otimes\mathcal{H}\to\mathcal{H}$ by

$\widetilde{V}_{m}(\xi_{1}\otimes\dots\otimes\xi_{m}\otimes h)=V(\xi_{1})\dots
V(\xi_{m})h,\quad h\in\mathcal{H},\xi_{i}\in E.$

The left action of $\mathcal{A}$ on $\mathcal{F}(E)$ is defined by

$\phi_{\infty}(a)\left(\oplus_{m\geq 0}\xi_{m}\right)=\oplus_{m\geq
0}\phi_{m}(a)\xi_{m},\>\>\xi_{m}\in E^{\otimes m},$

where $\mathcal{F}(E)=\bigoplus_{m\geq 0}E^{\otimes m}$ is a _Fock module_
[9]. Now, we defined the _creation operator_ $V_{\xi},\xi\in E$ on
$\mathcal{F}(E)$ by

$V_{\xi}(\eta)=\xi\otimes\eta,\>\>\eta\in E^{\otimes m},m\geq 0.$

Let $\mathcal{T}{(E)}$ be a $C^{*}$-algebra in $B^{a}(\mathcal{F}(E)),$ known
as the Toeplitz algebra, which is generated by
$\\{\phi_{\infty}(a)\\}_{a\in\mathcal{A}}$ and $\\{V_{\xi}\\}_{\xi\in E}.$ The
following result is due to Pimsner [19] which is useful to classify the
isometric covariant representations:

######  Theorem 1.6.

Suppose that $(\sigma,V)$ is an isometric covariant representation of $E$ on
$\mathcal{H}.$ Then the map

$\begin{cases}\phi_{\infty}(a)\mapsto\sigma(a),&a\in\mathcal{A};\\\
V_{\xi}\mapsto V(\xi),&\xi\in E\end{cases}$

uniquely extends to a $C^{*}$-representation of $\mathcal{T}(E)$ on
$\mathcal{H}.$ Conversely, let $\pi:\mathcal{T}(E)\mapsto B(\mathcal{H})$ be a
$C^{*}$-representation. If $\sigma(a)$ is defined to be
$\pi(\phi_{\infty}(a))$ for $a\in\mathcal{A},$ and $V(\xi)$ is defined to be
$\pi(V_{\xi})$ for $\xi\in E,$ then $(\sigma,V)$ is the isometric covariant
representation of $E$ on $\mathcal{H}.$

The following notion of partial isometric covariant representation plays an
important role in this paper:

######  Definition 1.7.

A completely bounded covariant representation $(\sigma,V)$ of ${E}$ on
$\mathcal{H}$ is called partial isometric if $\|\widetilde{V}x\|=\|x\|$ for
all $x\in N(\widetilde{V})^{\perp}$.

Suppose that $(\sigma,V)$ is a completely bounded covariant representation of
$E$ on $\mathcal{H}$ such that $R(\widetilde{V})$ is closed range. If

$\widetilde{V}^{\dagger}\widetilde{V}\widetilde{V}^{\dagger}=\widetilde{V}^{\dagger},\quad\widetilde{V}\widetilde{V}^{\dagger}\widetilde{V}=\widetilde{V},\quad(\widetilde{V}\widetilde{V}^{\dagger})^{*}=\widetilde{V}\widetilde{V}^{\dagger},\quad(\widetilde{V}^{\dagger}\widetilde{V})^{*}=\widetilde{V}^{\dagger}\widetilde{V},$

then we say that $\widetilde{V}^{\dagger}$ is the Moore-Penrose inverse [21,
2, 11] of $(\sigma,V).$ For $m\in\mathbb{N},$ define
$\widetilde{V}^{\dagger(m)}:\mathcal{H}\to E^{\otimes m}\otimes\mathcal{H}$ by

$\widetilde{V}^{\dagger(m)}:=(I_{E^{\otimes
m-1}}\otimes\widetilde{V}^{\dagger})(I_{E^{\otimes
m-2}}\otimes\widetilde{V}^{\dagger})\dots(I_{E}\otimes\widetilde{V}^{\dagger})\widetilde{V}^{\dagger}.$

######  Remark 1.8.

[14] Suppose that $(\sigma,V)$ is a c.b.c. representation of $E$ on
$\mathcal{H}$. Then the following conditions are equivalent:

  1. (1)

$\widetilde{V}$ is a partial isometry.

  2. (2)

$\widetilde{V}^{*}$ is a partial isometry.

  3. (3)

$\widetilde{V}\widetilde{V}^{*}\widetilde{V}=\widetilde{V}.$

  4. (4)

$\widetilde{V}^{*}\widetilde{V}=P_{{R}(\widetilde{V}^{*})}$ (initial
projection of $\widetilde{V}$).

  5. (5)

$\widetilde{V}\widetilde{V}^{*}=P_{{R}(\widetilde{V})}$ (final projection of
$\widetilde{V}$).

  6. (6)

$\widetilde{V}^{\dagger}=\widetilde{V}^{*}.$

######  Remark 1.9.

Let $(\sigma,V)$ be a partial isometric representation of $E$ on
$\mathcal{H}.$ If $\mathcal{K}$ is a $(\sigma,V)$-reducing subspace of
$\mathcal{H},$ then $(\sigma,V)|_{\mathcal{K}}$ is also a partial isometric
representation (see [13, Lemma 1]) of $E$ on $\mathcal{K}.$

######  Definition 1.10.

A completely bounded covariant representation $(\sigma,V)$ of $E$ on
$\mathcal{H}$ is called power partial isometric representation if each
$\widetilde{V}_{n}$ is a partial isometry for all $n\in\mathbb{N}.$

##  2\. Product of Partial isometric covariant representations

The aim of this section is to study the product of partial isometric
representations of $C^{*}$-correspondences. Let $(\sigma,V^{(1)})$ and
$(\sigma,V^{(2)})$ be two partial isometric representations of
$C^{*}$-correspondences $E_{1}$ and $E_{2}$ on $\mathcal{H},$ respectively.
Define a c.b.c. representation $(\sigma,V)$ of $E_{1}\otimes E_{2}$ on
$\mathcal{H}$ by
$\widetilde{V}:=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}).$ If
$\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})=\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(1)},$
then $(\sigma,V)$ is a partial isometric representation. Indeed,

$\displaystyle\widetilde{V}\widetilde{V}^{*}\widetilde{V}$
$\displaystyle=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})=\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})$
$\displaystyle=\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(2)})=\widetilde{V}.$

The following result is a generalization of [13, Lemma 2].

######  Theorem 2.1.

Let $(\sigma,V^{(1)})$ and $(\sigma,V^{(2)})$ be two partial isometric
representations of $C^{*}$-correspondences $E_{1}$ and $E_{2}$ on
$\mathcal{H},$ respectively. Then the following conditions are equivalent:

  1. (1)

$\widetilde{V}$ = $\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})$
is a partial isometry.

  2. (2)

The projections $\mathfrak{E}:=\widetilde{V}^{{(1)}^{*}}\widetilde{V}^{(1)}$
and
$\mathfrak{F}:=(I_{E_{1}}\otimes{\widetilde{V}^{(2)}}\widetilde{V}^{{(2)}^{*}})$
commute.

###### Proof.

Since $\mathfrak{E}$ and $\mathfrak{F}$ commute, we get

$\displaystyle\widetilde{V}\widetilde{V}^{*}\widetilde{V}$
$\displaystyle=\widetilde{V}^{{(1)}}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}}\widetilde{V}^{{(2)}^{*}})\widetilde{V}^{{(1)}^{*}}\widetilde{V}^{{(1)}}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}})=\widetilde{V}^{{(1)}}\mathfrak{F}\mathfrak{E}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}})$
$\displaystyle=\widetilde{V}^{{(1)}}\mathfrak{E}\mathfrak{F}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}})=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})=\widetilde{V}.$

Conversely, suppose that $\widetilde{V}$ =
$\widetilde{V}\widetilde{V}^{*}\widetilde{V},$ then
$\widetilde{V}^{{(1)}}\mathfrak{F}\mathfrak{E}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}})=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}),$
and hence

$\displaystyle\mathfrak{E}\mathfrak{F}\mathfrak{E}\mathfrak{F}=\widetilde{V}^{{(1)}^{*}}\widetilde{V}^{{(1)}}\mathfrak{F}\mathfrak{E}(I_{E_{1}}\otimes\widetilde{V}^{{(2)}}\widetilde{V}^{{(2)}^{*}})=\widetilde{V}^{{(1)}^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{{(2)}^{*}})=\mathfrak{E}\mathfrak{F}.$

Therefore $\mathfrak{E}\mathfrak{F}$ is idempotent. Since
$\|\mathfrak{E}\mathfrak{F}\|$ $\leq 1,$ we have $\mathfrak{E}\mathfrak{F}$ is
self-adjoint, and thus $\mathfrak{E}$ and $\mathfrak{F}$ commute. ∎

The following result is regarding the product of a finite number of partial
isometric representations which is a generalization of [4, Theorem 2]:

######  Theorem 2.2.

Let $E_{1},\dots,E_{n}$ be $C^{*}$-correspondences. For each $i\in I_{n},$ let
$(\sigma,V^{(i)})$ be a partial isometric representation of $E_{i}$ on
$\mathcal{H}.$ For $i\in I_{n},$ we define a c.b.c. representation
$(\sigma,T^{(i)})$ of a $C^{*}$-correspondence $E_{1}\otimes\dots\otimes
E_{i}$ on $\mathcal{H}$ by

$T^{(i)}(\xi_{1}\otimes\xi_{2}\otimes\dots\otimes\xi_{i})h=V^{(1)}(\xi_{1})V^{(2)}(\xi_{2})\dots
V^{(i)}(\xi_{i})h\quad\quad(\xi_{1}\in E_{1},\dots,\xi_{i}\in
E_{i},h\in\mathcal{H}).$

Then the following conditions are equivalent:

  1. (1)

$(\sigma,T^{(2)}),\dots,(\sigma,T^{(n)})$ are partial isometric
representations.

  2. (2)

Range of $\widetilde{T}^{(i-1)^{*}}$ is invariant under
$P_{{R}(I_{E_{1}\otimes\dots\otimes E_{i-1}}\otimes\widetilde{V}^{(i)})}$,
that is,

$(I_{E_{1}\otimes\dots\otimes
E_{i-1}}\otimes\widetilde{V}^{(i)}\widetilde{V}^{(i)^{*}}){R}(\widetilde{T}^{(i-1)^{*}})\subseteq{R}(\widetilde{T}^{(i-1)^{*}})\quad
for\quad all\quad i\in I_{n}\setminus\\{1\\}.$

  3. (3)

Range of $(I_{E_{1}\otimes\dots\otimes E_{i}}\otimes\widetilde{V}^{(i+1)})$ is
invariant under $P_{{R}(\widetilde{T}^{(i)^{*}})}$, that is,

$\widetilde{T}^{(i)^{*}}\widetilde{T}^{(i)}{R}(I_{E_{1}\otimes\dots\otimes
E_{i}}\otimes\widetilde{V}^{(i+1)})\subseteq{R}(I_{E_{1}\otimes\dots\otimes
E_{i}}\otimes\widetilde{V}^{(i+1)})\quad for\quad all\quad i\in I_{n-1}.$

  4. (4)

The operator products
$P_{{R}(\widetilde{T}^{(i)^{*}})}P_{{R}(I_{E_{1}\otimes\dots\otimes
E_{i}}\otimes\widetilde{V}^{(i+1)})},i\in I_{n-1}$ are idempotent.

###### Proof.

We will prove it using the mathematical induction. For $n=2$ we have

$(1)\Rightarrow(2)$ We will prove it using contradiction. Suppose $(2)$ does
not hold, then there exists a nonzero element
$\eta\in{R}(\widetilde{T}^{(2)^{*}})=(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})R(\widetilde{V}^{(1)^{*}})$
such that
$\zeta=(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta\notin{R}(\widetilde{V}^{(1)^{*}}).$
Note that $\langle\zeta,\zeta\rangle$=
$\langle(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta,(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta\rangle$=$\langle\eta,\eta\rangle.$
Since $\widetilde{V}_{2}$ is a partial isometry we have

$\langle\eta,\eta\rangle=\langle\widetilde{T}^{(2)}\eta,\widetilde{T}^{(2)}\eta\rangle=\langle\widetilde{V}^{(1)}\zeta,\widetilde{V}^{(1)}\zeta\rangle<\langle\zeta,\zeta\rangle=\langle\eta,\eta\rangle.$

This is a contradiction, so
$(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}){R}(\widetilde{V}^{(1)^{*}})\subseteq{R}(\widetilde{V}^{(1)^{*}}).$

$(2)\Rightarrow(1)$ Suppose $(2)$ holds, then
$\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}$ is the orthogonal projection onto
$(I_{E_{1}}\otimes\widetilde{V}^{(2)}){R}(\widetilde{T}^{(2)^{*}}).$ Let
$\eta\in{R}(\widetilde{T}^{(2)^{*}})$ $\subseteq$
${R}(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})$ we have

$\langle\widetilde{T}^{(2)}\eta,\widetilde{T}^{(2)}\eta\rangle=\langle\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta,\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta\rangle=\langle(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta,(I_{E_{1}}\otimes\widetilde{V}^{(2)})\eta\rangle=\langle\eta,\eta\rangle.$

$(1)\Rightarrow(3)$ Suppose that $\widetilde{T}^{(2)}$ is a partial isometry,
then $\widetilde{T}^{(2)^{*}}$ is also a partial isometry. Apply condition
$(2)$ to the adjoint
$\widetilde{T}^{(2)^{*}}=(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}},$
we get $(3).$

$(3)\Rightarrow(1)$ Suppose $(3)$ holds, then
$(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})$ is the
orthogonal projection onto $\widetilde{V}^{(1)^{*}}{R}(\widetilde{T}^{(2)})$.
Let $h\in{R}(\widetilde{T}^{(2)})\subseteq{R}(\widetilde{V}^{(1)})$ we get

$\langle\widetilde{T}^{(2)^{*}}h,\widetilde{T}^{(2)^{*}}h\rangle=\langle(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}h,(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}h\rangle=\langle\widetilde{V}^{(1)^{*}}h,\widetilde{V}^{(1)^{*}}h\rangle=\langle
h,h\rangle.$

Therefore $\widetilde{T}^{(2)^{*}}$ is a partial isometry, and hence
$\widetilde{T}^{(2)}$ is a partial isometry.

$(1)\Rightarrow(4)$ Suppose that $\widetilde{T}^{(2)}$ is a partial isometry,
then
$\widetilde{T}^{(2)}=\widetilde{T}^{(2)}\widetilde{T}^{(2)^{*}}\widetilde{T}^{(2)},$
that is,

$\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}).$

Pre-multiply by $\widetilde{V}^{(1)^{*}}$ and post-multiply by
$(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})$ in the last inequality, we get

$\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})=\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}).$

This implies that
$P_{{R}(\widetilde{V}^{(1)^{*}})}P_{{R}(I_{E_{1}}\otimes\widetilde{V}^{(2)})}$
is idempotent.

$(4)\Rightarrow(1)$ Since
$P_{R(\widetilde{V}_{1}^{*})}P_{R(I_{E_{1}}\otimes\widetilde{V}_{2})}$ is
idempotent, we have

$\displaystyle\widetilde{T}^{(2)}\widetilde{T}^{(2)^{*}}\widetilde{T}^{(2)}$
$\displaystyle=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})$
$\displaystyle=\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(2)})$
$\displaystyle=\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}\widetilde{V}^{(2)})=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})=\widetilde{T}^{(2)}.$

Therefore $\widetilde{T}^{(2)}$ is a partial isometry. Suppose that
$(1),(2),(3)$ and $(4)$ are equivalent for $n=m.$ Consider

$\widetilde{T}^{(m+1)}=\widetilde{T}^{(m)}(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)}),$

where $\widetilde{V}^{(m+1)}$ is a partial isometry. Then
$\widetilde{T}^{(m+1)}$ is a partial isometry if and only if any of the
following conditions hold:

  * (1)

$(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)}\widetilde{V}^{(m+1)^{*}}){R}(\widetilde{T}^{(m)^{*}})\subseteq{R}(\widetilde{T}^{(m)^{*}}).$

  * (2)

$\widetilde{T}^{(m)^{*}}\widetilde{T}^{(m)}{R}(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)})\subseteq{R}(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)}).$

  * (3)

$(P_{{R}(\widetilde{T}^{(m)^{*}})}P_{{R}(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)})})^{2}=P_{{R}(\widetilde{T}^{(m)^{*}})}P_{{R}(I_{E_{1}\otimes\dots\otimes
E_{m}}\otimes\widetilde{V}^{(m+1)})}.$

This completes the proof. ∎

######  Remark 2.3.

Let $E_{1},\dots,E_{n}$ be $C^{*}$-correspondences. For each $i\in I_{n},$ let
$(\sigma,V^{(i)})$ be a partial isometric representation of $E_{i}$ on
$\mathcal{H}.$ Define a c.b.c. representation $(\sigma,T^{(n)})$ of a
$C^{*}$-correspondence $E_{1}\otimes\dots\otimes E_{n}$ on $\mathcal{H}$ by

$T^{(n)}(\xi_{1}\otimes\xi_{2}\otimes\dots\otimes\xi_{n})h:=V^{(1)}(\xi_{1})V^{(2)}(\xi_{2})\dots
V^{(n)}(\xi_{n})h\quad\quad(\xi_{1}\in E_{1},\dots,\xi_{n}\in
E_{n},h\in\mathcal{H}).$

Then
${\widetilde{T}^{(n)}}=\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})\dots(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)})$ is a partial isometry if and only if

${\widetilde{T}^{(n)^{\dagger}}}=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{\dagger}})\dots(I_{E_{1}}\otimes\widetilde{V}^{(2)^{\dagger}})\widetilde{V}^{(1)^{\dagger}}.$

###### Proof.

Let ${\widetilde{T}^{(n)}}$ be a partial isometry, then
${\widetilde{T}^{(n)^{\dagger}}}={\widetilde{T}^{(n)^{*}}}.$ It follows that

$\displaystyle{\widetilde{T}^{(n)^{\dagger}}}$
$\displaystyle={\widetilde{T}^{(n)^{*}}}=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{*}})\dots\widetilde{V}^{(1)^{*}}=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{\dagger}})\dots\widetilde{V}^{(1)^{\dagger}}.$

Conversely, suppose that
${\widetilde{T}^{(n)^{\dagger}}}=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{\dagger}})\dots(I_{E_{1}}\otimes\widetilde{V}^{(2)^{\dagger}})\widetilde{V}^{(1)^{\dagger}},$
then

$\displaystyle{\widetilde{T}^{(n)^{*}}}$
$\displaystyle=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{*}})\dots\widetilde{V}^{(1)^{*}}=(I_{E_{1}\otimes\dots\otimes
E_{n-1}}\otimes\widetilde{V}^{(n)^{\dagger}})\dots\widetilde{V}^{(1)^{\dagger}}={\widetilde{T}^{(n)^{\dagger}}}.$

Thus ${\widetilde{T}^{(n)}}$ is a partial isometry. ∎

Suppose that $(\sigma,V^{(1)})$ is a completely contractive covariant
representation of $E_{1}$ on $\mathcal{H}.$ Then the operator matrix
$\begin{bmatrix}\widetilde{V}^{(1)}&(I_{H}-\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}})^{1/2}\\\
0&0\\\
\end{bmatrix}:(E_{1}\otimes\mathcal{H})\oplus\mathcal{H}\to\mathcal{H}\oplus\mathcal{H}$
is a partial isometry.

######  Theorem 2.4.

Let $(\sigma,V^{(1)})$ and $(\sigma,V^{(2)})$ be two completely contractive
covariant representations of $E_{1}$ and $E_{2}$ on $\mathcal{H},$
respectively. Then the operator matrix

$M=\begin{bmatrix}\widetilde{V}^{(1)}(I_{E_{1}}\otimes\widetilde{V}^{(2)})&\widetilde{V}^{(1)}(I_{E_{1}}\otimes(I_{H}-\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}\\\
0&0\\\ \end{bmatrix}$

is a partial isometry if and only if $(\sigma,V^{(1)})$ is a partial isometric
representation.

###### Proof.

Let $(\sigma,V^{(1)})$ be a partial isometric representation. Note that

$\displaystyle
MM^{*}=\begin{bmatrix}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}&0\\\
0&0\end{bmatrix}.$

Since $\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}$ is a projection, $MM^{*}$
is also a projection. Thus $M$ is a partial isometry.

Conversely, suppose that $M$ is a partial isometry, then $M^{*}MM^{*}=M^{*}.$
Since

$\displaystyle
M^{*}MM^{*}=\begin{bmatrix}(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}&0\\\
(I_{E_{1}}\otimes(I_{H}-\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}&0\end{bmatrix}$

and

$\displaystyle
M^{*}=\begin{bmatrix}(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}&0\\\
(I_{E_{1}}\otimes(I_{H}-\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}\widetilde{V}^{(1)^{*}}&0\end{bmatrix},$

we get

(2.1)
$\displaystyle(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}=(I_{E_{1}}\otimes\widetilde{V}^{(2)^{*}})\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}$

and

(2.2)
$\displaystyle(I_{E_{1}}\otimes(I_{H}-\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}\widetilde{V}^{(1)^{*}}=(I_{E_{1}}\otimes(I_{H}-\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}.$

Now pre-multiply by $(I_{E_{1}}\otimes\widetilde{V}^{(2)})$ in Equation (2.1)
and
$(I_{{E_{1}}\otimes\mathcal{H}}-(I_{E_{1}}\otimes\widetilde{V}^{(2)}\widetilde{V}^{(2)^{*}}))^{1/2}$
in Equation (2.2), we get
$\widetilde{V}^{(1)^{*}}=\widetilde{V}^{(1)^{*}}\widetilde{V}^{(1)}\widetilde{V}^{(1)^{*}}.$
Therefore $(\sigma,V^{(1)})$ is a partial isometric representation. ∎

##  3\. Powers and roots of partial isometric representations

In this section we will discuss the powers and roots of partial isometric
representations. From Theorem 2.2, we proved that if $(\sigma,V)$ is a partial
isometric representation of $E$ on $\mathcal{H}$ such that
$(I_{E^{\otimes(i-1)}}\otimes\widetilde{V}\widetilde{V}^{*}){N}(\widetilde{V}_{i-1})\subseteq{N}(\widetilde{V}_{i-1})$
for every $i\leq n$ , then $\widetilde{V}_{i}$ is a partial isometry for every
$i\leq n.$

The following result gives an equivalent condition of Theorem 2.2, which is an
analogue of [5, Proposition 1].

######  Proposition 3.1.

Suppose $(\sigma,V)$ is a partial isometric representation of a
$C^{*}$-correspondence $E$ on $\mathcal{H}.$ Then the following conditions are
equivalent:

  1. (1)

$(I_{E^{\otimes(n-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{n})^{\perp}\subseteq{N}(\widetilde{V}_{n-1})^{\perp}$
for all $n\in\mathbb{N}.$

  2. (2)

$(I_{E^{\otimes(n-1)}}\otimes\widetilde{V}\widetilde{V}^{*}){N}(\widetilde{V}_{n-1})\subseteq{N}(\widetilde{V}_{n-1})$
for all $n\in\mathbb{N}.$

###### Proof.

For $n\in\mathbb{N},$ we have

$\displaystyle(I_{E^{\otimes(n-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{n})^{\perp}$
$\displaystyle=(I_{E^{\otimes(n-1)}}\otimes\widetilde{V})\widetilde{V}_{n}^{*}\mathcal{H}=(I_{E^{\otimes(n-1)}}\otimes\widetilde{V}\widetilde{V}^{*})\widetilde{V}_{n-1}^{*}\mathcal{H}$
$\displaystyle=(I_{E^{\otimes(n-1)}}\otimes\widetilde{V}\widetilde{V}^{*}){N}(\widetilde{V}_{n-1})^{\perp}.$

It gives $(1)$ is equivalent to $(2).$ ∎

The next result is a combination of Theorem 2.2 and Proposition 3.1.

######  Theorem 3.2.

Let $(\sigma,V)$ be a partial isometric representation of $E$ on
$\mathcal{H}$. Then the following conditions are equivalent:

  1. (1)

$\widetilde{V}_{i}$ is a partial isometry for every $i\in I_{n}.$

  2. (2)

$(I_{E^{\otimes(i-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{i})^{\perp}\subseteq{N}(\widetilde{V}_{i-1})^{\perp}$
for every $i\in I_{n}.$

Moreover, $(\sigma,V)$ is a power partial isometric representation of $E$ on
$\mathcal{H}$ if and only if $(I_{E^{\otimes
i}}\otimes\widetilde{V}){N}(\widetilde{V}_{i+1})^{\perp}\subseteq{N}(\widetilde{V}_{i})^{\perp}$
for every $i\in\mathbb{N}.$

###### Proof.

We will prove it using the mathematical induction. For $n=1,$ it is obvious.
Suppose that our assumption holds for $n=m.$ Now, we want to prove it is true
for $n=m+1.$ Let $\eta\in$ ${N}(\widetilde{V}_{m+1})^{\perp}$ $\subseteq$
${N}(I_{E^{\otimes m}}\otimes\widetilde{V})^{\perp},$ since $(I_{E^{\otimes
m}}\otimes\widetilde{V}){N}(\widetilde{V}_{m+1})^{\perp}\subseteq{N}(\widetilde{V}_{m})^{\perp},$
we have $(I_{E^{\otimes
m}}\otimes\widetilde{V})\eta\in{N}(\widetilde{V}_{m})^{\perp}$. Since
$(I_{E^{\otimes m}}\otimes\widetilde{V})$ and $\widetilde{V}_{m}$ are partial
isometries, we get

$\|\widetilde{V}_{m+1}\eta\|=\|\widetilde{V}_{m}(I_{E^{\otimes
m}}\otimes\widetilde{V})\eta\|=\|(I_{E^{\otimes
m}}\otimes\widetilde{V})\eta\|=\|\eta\|.$

Therefore $\widetilde{V}_{m+1}$ is a partial isometry.

Conversely, suppose that $\widetilde{V}_{i}$ is a partial isometry for $i\in
I_{m+1}.$ From Theorem 2.2, we obtain

$(I_{E^{\otimes(i-1)}}\otimes\widetilde{V}\widetilde{V}^{*}){N}(\widetilde{V}_{i-1})\subseteq{N}(\widetilde{V}_{i-1})\quad\quad
for\quad all\quad i\in I_{m+1}.$

From Proposition 3.1,
$(I_{E^{\otimes(i-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{i})^{\perp}\subseteq{N}(\widetilde{V}_{i-1})^{\perp}$
for every $i\in I_{m+1}.$ ∎

######  Corollary 3.3.

Let $(\sigma,V)$ be a partial isometric representation of $E$ on $\mathcal{H}$
such that $\widetilde{V}_{n}$ is a partial isometry for some $n\geq 2.$ If
$(I_{E^{\otimes
n}}\otimes\widetilde{V}){N}(\widetilde{V}_{n+1})^{\perp}\subseteq{N}(\widetilde{V}_{n})^{\perp}$,
then $\widetilde{V}_{n+1}$ is a partial isometry.

######  Definition 3.4.

A completely bounded covariant representation $(\sigma,V)$ of $E$ on
$\mathcal{H}$ is said to be regular if its range, $R(\widetilde{V}),$ is
closed and
$N(\widetilde{V})\subseteq{E}\otimes{{{R}^{\infty}{({\widetilde{V}})}}}.$ We
denote ${{{R}^{\infty}{({\widetilde{V}})}}}:=\bigcap_{n\geq
0}{{{R}{(\widetilde{V}_{n})}}}$ is the generalized range of $(\sigma,V).$

Suppose that $(\sigma,V)$ is a c.b.c. representation of ${E}$ on
$\mathcal{H}.$ Let ${S}:\mathcal{H}\to E\otimes\mathcal{H}$ be a bounded
linear operator. If ${S}\widetilde{V}{S}={S}$ and
$\widetilde{V}{S}\widetilde{V}=\widetilde{V},$ then we say that $S$ is
generalized inverse of $\widetilde{V}.$

######  Lemma 3.5.

Suppose that $(\sigma,V)$ is a regular representation of $E$ on $\mathcal{H},$
and let $S$ be a generalized inverse of $\widetilde{V}$. Then $(I_{E^{\otimes
m}}\otimes S){N}(\widetilde{V}_{m})\subseteq{N}(\widetilde{V}_{m+1})$ for all
$m\in\mathbb{N}.$

###### Proof.

Since $(\sigma,V)$ is a regular, let $x\in$
${N}(\widetilde{V}_{m})\subseteq{E^{\otimes m}}\otimes{R}(\widetilde{V}),$
there exists $\eta\in E^{\otimes m+1}\otimes\mathcal{H}$ such that
$x=(I_{E^{\otimes m}}\otimes\widetilde{V})\eta.$ It gives

$\displaystyle\widetilde{V}_{m+1}(I_{E^{\otimes m}}\otimes S)x$
$\displaystyle=\widetilde{V}_{m}(I_{E^{\otimes
m}}\otimes\widetilde{V}S\widetilde{V})\eta=\widetilde{V}_{m}(I_{E^{\otimes
m}}\otimes\widetilde{V})\eta=\widetilde{V}_{m}(x)=0.$

Therefore $(I_{E^{\otimes m}}\otimes
S){N}(\widetilde{V}_{m})\subseteq{N}(\widetilde{V}_{m+1}).$ ∎

Now we present the connection between partial isometric and power partial
isometric representations.

######  Corollary 3.6.

Suppose that $(\sigma,V)$ is a regular representation of $E$ on $\mathcal{H}$.
Then $(\sigma,V)$ is a partial isometric representation if and only if
$(\sigma,V)$ is a power partial isometric representation.

###### Proof.

Suppose that $(\sigma,V)$ is a partial isometric representation, then
$\widetilde{V}^{*}$ is the generalized inverse of $\widetilde{V}$. From Lemma
3.5, $(I_{E^{\otimes
m}}\otimes\widetilde{V}^{*}){N}(\widetilde{V}_{m})\subseteq{N}(\widetilde{V}_{m+1})$
for all $m\in\mathbb{N},$ or equivalently, $(I_{E^{\otimes
m}}\otimes\widetilde{V}){N}(\widetilde{V}_{m+1})^{\perp}\subseteq{N}(\widetilde{V}_{m})^{\perp}$
for all $m\in\mathbb{N}.$ Then from Theorem 3.2, $(\sigma,V)$ is a power
partial isometric representation. ∎

######  Example 3.7.

Consider $\mathcal{H}$ is a Hilbert space with orthonormal basis
$\\{e_{m}:\>m\geq 0\\}$ and $E$ is an $n$-dimensional Hilbert space with
orthonormal basis $\\{\delta_{i}\\}_{i\in I_{n}}.$ Suppose that $(\rho,S^{w})$
is the unilateral weighted shift c.b.c. representation (see [21, 22]) of $E$
on $\mathcal{H}$ given by

$\rho(b)=bI_{\mathcal{H}}\>\>\mbox{and}\>\>\>S^{w}(\delta_{i})=V_{i},\>\>b\in\mathbb{C},$

where the operator $V_{i}(e_{m})=w_{i,m}e_{nm+i}$ for all $i\in I_{n},m\geq 0$
and $\\{w_{i,m}:i\in I_{n},\>\>m\geq 0\\}$ is the bounded set of nonnegative
real numbers. Let $B\subseteq\mathbb{N}$ and $(\alpha_{m})_{m\in\mathbb{N}}$
be the sequence such that

$\begin{cases}\alpha_{m}=0&\text{if }{m\in B};\\\ \alpha_{m}=1&\text{if
}{m\notin B}.\end{cases}$

The unilateral weighted shift is defined by

$S^{w,{B}}(\delta_{i})(e_{m})=V_{i,B}(e_{m})=w_{i,m}\alpha_{m}e_{nm+i}\quad\mbox{for
all}\quad m\geq 0.$

It is easy to check that $N(V_{i,B})=\overline{span}\\{e_{m}:\>m\in B\\}$ and
$N(V_{i,B}^{k})=\bigcup_{p=1}^{k}\overline{span}\\{e_{m}:\>n^{p-1}m+\sum_{l=2}^{p}n^{p-l}i\in
B\\}$ for every $i\in I_{n}$ and $k\in\mathbb{N}.$ This follows that

$V_{i,B}(N(V_{i,B}^{k+1})^{\perp})\subseteq
N(V_{i,B}^{k})^{\perp}\quad\mbox{for all}\quad i\in I_{n}.$

The following result is a combination of Example 3.7 and Theorem 3.2:

######  Proposition 3.8.

  1. (1)

$(\rho,S^{w,{B}})$ is a partial isometric representation of $E$ on
$\mathcal{H}$ if and only if $w_{i,m}=1$ for every $m\notin B$ and $i\in
I_{n}.$

  2. (2)

Every partial isometric unilateral weighted shift c.b.c. representation
$(\rho,S^{w,{B}})$ of $E$ on $\mathcal{H}$ is a power partial isometric
representation.

The next result is a generalization of [5, Theorem 5].

######  Theorem 3.9.

Let $(\sigma,V)$ be a completely contractive covariant representation of a
full $C^{*}$-correspondence $E$ on $\mathcal{H}.$ Suppose that
$\widetilde{V}_{k}$ is a partial isometry for some $k\geq 2.$ Then the
following conditions are equivalent:

  1. (1)

$(\sigma,V)$ is a partial isometric representation.

  2. (2)

$(a)$ For all
$\eta\in{N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})$
we have

$\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|=\|\eta\|.$

$(b)$
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})^{\perp})\perp(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$

###### Proof.

$(1)\Rightarrow(2).$ Suppose that $\widetilde{V}$ is a partial isometry, then
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})$ is also a partial isometry. Let
$\eta\in{N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\subseteq{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})^{\perp},$
then $\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|=\|\eta\|.$ Next we
want to prove that
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})^{\perp})\perp(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$
Let $\eta\in$
${N}(\widetilde{V}_{k})^{\perp}\subseteq{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})^{\perp}$
and
$\zeta\in{N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\subseteq{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})^{\perp},$
then $\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|=\|\eta\|$ and
$\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\zeta\|=\|\zeta\|.$ Since
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})$ is a partial isometry, we have

$\displaystyle\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta+(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\zeta\|^{2}$
$\displaystyle=\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})(\eta+\zeta)\|^{2}=\|\eta+\zeta\|^{2}=\|\eta\|^{2}+\|\zeta\|^{2}$
$\displaystyle=\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|^{2}+\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\zeta\|^{2}.$

It follows that
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})^{\perp})\perp(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$

$(2)\Rightarrow(1).$ Suppose $(2)$ holds. Since $\widetilde{V}_{k}$ is a
partial isometry and $\widetilde{V}$ is a contraction. For
$\eta\in{N}(\widetilde{V}_{k})^{\perp}$, we have

$\|\eta\|=\|\widetilde{V}_{k}\eta\|=\|\widetilde{V}_{k-1}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|\leq\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|\leq\|\eta\|.$

This implies that
$\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|=\|\eta\|$ for all
$\eta\in{N}(\widetilde{V}_{k})^{\perp}.$ Since
${N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})^{\perp}={N}(\widetilde{V}_{k})^{\perp}\oplus({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$
For each $\eta\in{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})^{\perp},$ we
have
$\eta=\eta_{1}+\eta_{2}\in{N}(\widetilde{V}_{k})^{\perp}\oplus({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$
From $(2)$ we get

$\displaystyle\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|^{2}=\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta_{1}+(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta_{2}\|^{2}$
$\displaystyle=\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta_{1}\|^{2}+\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta_{2}\|^{2}=\|\eta_{1}\|^{2}+\|\eta_{2}\|^{2}=\|\eta\|^{2}.$

Therefore $(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})$ is a partial isometry.
Since $E$ is full and $\sigma$ is nondegenerate, $(\sigma,V)$ is a partial
isometric representation. ∎

The following result is a generalization of [12, Theorem A].

######  Remark 3.10.

Let $(\sigma,V)$ be a completely contractive covariant representation of a
full $C^{*}$-correspondence $E$ on $\mathcal{H}.$ Suppose that
$\widetilde{V}_{k}$ is a partial isometry for some $k\geq 2.$ If
${N}(I_{E}\otimes\widetilde{V})={N}(\widetilde{V}_{2}),$ then $(\sigma,V)$ is
a partial isometric representation.

######  Remark 3.11.

Let $(\sigma,V)$ be a completely contractive covariant representation of a
full $C^{*}$-correspondence $E$ on $\mathcal{H}.$ Suppose that
$\widetilde{V}_{k}$ is a partial isometry for some $k\geq 2$ such that

  * (1)

$\|(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})\eta\|=\|\eta\|$ for all
$\eta\in{N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V}).$

  * (2)

$(I_{E^{\otimes(k-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{k})^{\perp}\subseteq{N}(\widetilde{V}_{k-1})^{\perp}.$

Then $(\sigma,V)$ is a partial isometric representation. Indeed, let
$\eta\in{N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V}),$
then
$(I_{E^{\otimes(k-1)}}\otimes\widetilde{V})\eta\in{N}(\widetilde{V}_{k-1}).$
Since
$(I_{E^{\otimes(k-1)}}\otimes\widetilde{V}){N}(\widetilde{V}_{k})^{\perp}\subseteq{N}(\widetilde{V}_{k-1})^{\perp},$
we obtain
$(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})^{\perp})\perp(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})({N}(\widetilde{V}_{k})\ominus{N}(I_{E^{\otimes{k-1}}}\otimes\widetilde{V})).$

###  3.1. Wold-type decomposition for regular partial isometric
representations

Ezzahraoui, Mbekhta and Zerouali [6] gave a Wold-type decomposition for
regular operators using growth condition. Rohilla, Veerabathiran, and Trivedi
[21] generalized this decomposition for regular covariant representations with
growth condition. In [8], Ezzahraoui, Mbekhta, and Zerouali proved a Wold-type
decomposition for bi-regular operators. Saini [22] generalized this
decomposition for bi-regular covariant representations.

Now we recall the following notion of bi-regular covariant representation and
it’s Wold-type decomposition from [22].

######  Definition 3.12.

A regular representation $(\sigma,V)$ of ${E}$ on $\mathcal{H}$ is said to be
bi-regular if $N(I_{E^{\otimes n}}\otimes\widetilde{V}^{\dagger})$ is
contained in $R(\widetilde{V}^{\dagger{(n)}})$ for every $n\in\mathbb{N}.$

Note that if $(\sigma,V)$ is a regular representation of ${E}$ on
$\mathcal{H},$ then its adjoint $\widetilde{V}^{*}$ is also regular. That is,
$N(I_{E^{\otimes n}}\otimes\widetilde{V}^{*})\subseteq
R(\widetilde{V}_{n}^{*})$ for all $n\in\mathbb{N}.$ Suppose that $(\sigma,V)$
is a regular partial isometric representation of ${E}$ on $\mathcal{H},$ then
$\widetilde{V}^{\dagger}=\widetilde{V}^{*},$ and hence $(\sigma,V)$ is bi-
regular.

######  Definition 3.13.

Suppose that $(\sigma,V)$ is a completely bounded covariant representation of
$E$ on $\mathcal{H}$ such that $R(\widetilde{V})$ is closed range. The Cauchy
dual $\widetilde{V}^{\prime}:E\otimes\mathcal{H}\to\mathcal{H}$ of
$(\sigma,V)$ [22] is given by

$\widetilde{V}^{\prime}:=\widetilde{V}(\widetilde{V}^{*}\widetilde{V})^{\dagger}.$

Suppose that $(\sigma,V)$ is a partial isometric representation of $E$ on
$\mathcal{H}.$ Then by [22, Proposition 3.3],
$\widetilde{V}^{\prime}=\widetilde{V}.$

######  Proposition 3.14.

Suppose that $(\sigma,V)$ is a bi-regular representation of $E$ on
$\mathcal{H}.$ Then

$\mathcal{H}=[\mathcal{H}\ominus\widetilde{V}(E\otimes\mathcal{H})]_{\widetilde{V}}\oplus
R^{\infty}({\widetilde{V}^{\prime}})=[\mathcal{H}\ominus\widetilde{V}(E\otimes\mathcal{H})]_{\widetilde{V}^{\prime}}\oplus
R^{\infty}({\widetilde{V}}).$

######  Corollary 3.15.

Suppose that $(\sigma,V)$ is a regular partial isometric representation of
${E}$ on $\mathcal{H}.$ Then

$\mathcal{H}=[\mathcal{H}\ominus\widetilde{V}(E\otimes\mathcal{H})]_{\widetilde{V}}\oplus
R^{\infty}({\widetilde{V}}).$

### Acknowledgement

Dimple Saini is supported by UGC fellowship (File No:16-6(DEC.
2018)/2019(NET/CSIR)). Harsh Trivedi is supported by MATRICS-SERB Research
Grant, File No: MTR/2021/000286, by SERB, Department of Science & Technology
(DST), Government of India. Shankar Veerabathiran thanks IMSc Chennai for
postdoc fellowship. Saini and Trivedi acknowledge the DST-FIST program (Govt.
of India) FIST - No. SR/FST/MS-I/2018/24.

## References

  * [1] C. Badea and M. Mbekhta, _Operators similar to partial isometries_ , Acta Sci. Math. (Szeged) 71 (2005), 663-680.
  * [2] A. Ben-Israel and T. N. E. Greville, _Generalized Inverses Theory and applications, Second edition_ , Canadian Mathematic Society, Springer-Verlag, New York, Inc, 2003.
  * [3] J. Cuntz, _Simple $C^{*}$-algebras generated by isometries_, Comm. Math. Phys. 57(2) (1977), 173-185.
  * [4] I. Erdelyi, _Partial isometries closed under multiplication on Hilbert spaces_ , J. Math. Anal. Appl. 22 (1968), 546-551.
  * [5] H. Ezzahraoui, M. Mbekhta, A. Salhi and E. H. Zerouali, _A note on roots and powers of partial isometries_ , Archiv der Mathematik 110(3) (2018), 251-259.
  * [6] H. Ezzahraoui, M. Mbekhta and E. H. Zerouali, _Wold-type decomposition for some regular operators_ , Journal of Mathematical Analysis and Applications 430.1 (2015), 483-499.
  * [7] H. Ezzahraoui, M. Mbekhta and E. H. Zerouali, _On the Cauchy dual of closed range operators_ , Acta Sci. Math. (Szeged), 85 (2019), 231-248.
  * [8] H. Ezzahraoui, M. Mbekhta and E. H. Zerouali, _Wold-type decomposition for bi-regular operators_ , Acta Sci. Math. (Szeged) 87 (2021), 463-483.
  * [9] N. J. Fowler, _Discrete product systems of Hilbert bimodules_ , Pacific J. Math. 204 (2002), no. 2, 335-375.
  * [10] A. E. Frazho, _Complements to models for noncommuting operators_ , J. Funct. Anal. 59 (1984), 445-461.
  * [11] C. W. Groetsch, _Generalized inverses of linear operators: representation and approximation_ , Monographs and Textbooks in Pure and Applied Mathematics, No. 37. Marcel Dekker, Inc. New York-Basel, 1977.
  * [12] B. C. Gupta, _A note on partial sometries. II_ , Indian J. Pure Appl. Math. 11 (1980), 208-211.
  * [13] P. R. Halmos and L. J. Wallen, _Powers of partial isometries_ , J. Math. Mech. 19 (1969/1970), 657-663.
  * [14] C. S. Kubrusly, _The elements of operator theory_ , Birkhauser-Springer, New York, (2011).
  * [15] E. C. Lance, _Hilbert $C^{*}$-modules_, London Mathematical Society Lecture Note Series, vol. 210, Cambridge University Press, Cambridge, (1995).
  * [16] P. S. Muhly and B. Solel, _Tensor algebras over $C^{*}$-correspondences: representations, dilations, and $C^{*}$-envelopes_, J. Funct. Anal. 158(2) (1998), 389-457.
  * [17] P. S. Muhly and B. Solel, _Tensor Algebras, Induced Representations, and the Wold Decomposition_ , Canad. J. Math 51(4) (1999), 850-880.
  * [18] F. J. Murray and J. v. Neumann. _On rings of operators_ , Annals of Mathematics (1936): 116-229.
  * [19] M. V. Pimsner, _A class of $C^{*}$-algebras generalizing both Cuntz-Krieger algebras and crossed products by ${\bf Z}$_, Free probability theory (Waterloo, ON, 1995), 189-212, Fields Inst. Commun., 12, Amer. Math. Soc., Providence, RI, (1997).
  * [20] G. Popescu, _Isometric dilations for infinite sequences of noncommuting operators_ , Transactions of the American Mathematical Society 316 (1989), no. 2, 523–536.
  * [21] A. Rohilla, S. Veerabathiran and H. Trivedi, _Regular covariant representations and their Wold-type decomposition_ , J. Math. Anal. Appl. 522(2) (2023), p.126997, 33.
  * [22] D. Saini, _Cauchy dual and Wold-type decomposition for bi-regular covariant representations_ , Acta Scientiarum Mathematicarum (2024): 1-22.
  * [23] A. Viselter, _Covariant representations of subproduct systems_ , Proc. Lond. Math. Soc. (3) 102 (2011), no. 4, 767-800.
  * [24] H. Wold, _A study in the analysis of stationary time series_ , Almquist and Wiksell, Uppsala, 1938.


# Multi-vehicle Platoon Overtaking Using NoisyNet Multi-Agent Deep Q-Learning
Network

Lv He<EMAIL_ADDRESS>School of Automation and Electronic Information,
Xiangtan University, Xiangtan 411105, China

###### Abstract

With the recent advancements in Vehicle-to-Vehicle communication technology,
autonomous vehicles are able to connect and collaborate in platoon, minimizing
accident risks, costs, and energy consumption. The significant benefits of
vehicle platooning have gained increasing attention from the automation and
artificial intelligence areas. However, few studies have focused on platoon
with overtaking. To address this problem, the NoisyNet multi-agent deep
Q-learning algorithm is developed in this paper, which the NoisyNet is
employed to improve the exploration of the environment. By considering the
factors of overtake, speed, collision, time headway and following vehicles, a
domain-tailored reward function is proposed to accomplish safe platoon
overtaking with high speed. Finally, simulation results show that the proposed
method achieves successfully overtake in various traffic density situations.

###### keywords:

Multi-vehicle platoon; overtake; multi-agent reinforcement learning; mixed
traffic.

††journal: Neurocomputing

##  1 Introduction

In recent years, autonomous vehicles (AVs) and their technologies have
received extensive attention worldwide. Autonomous driving has stronger
perception and shorter reaction time compared to human driving. There is no
human driver behavior such as fatigue driving, and it is safer for long-
distance driving. By using advanced Vehicle-to-Vehicle communication
technologies, AVs are able to share information with each other and cooperate
in dynamic driving tasks. Through sharing information about the environment
[1, 2], locations, and actions, it will improve the driving safety [3], reduce
the traffic congestion [4], and decrease the energy consumption [5]. Multi-
vehicle collaboration and overtaking are two important topics for AVs.

Figure 1: Illustration of the considered platoon overtaking traffic scenario.
AVs (blue) and HDVs (green) coexist in the straight lane.

###  1.1 Multi-vehicle Collaboration

The cooperation of multiple vehicles is a promising way to improve traffic
efficiency and reduce congestion, and many research works have been reported.
By integrating CARLA [6] and SUMO [7], OpenCDA [8, 9] was proposed, which
supports both cooperative driving automation prototyping and regular
autonomous driving components. Based on OpenCDA, a series of multi-vehicle
collaboration works focused on various fields were studied, e.g.,
collaborative perception [10, 11, 1, 2, 12, 13, 14], planning [15],
localization, and safety system [16]. Coordinated strategies between
autonomous vehicles could improve transportation efficiency and reduce
unnecessary waiting time for passengers [17]. So that people have more time to
do more meaningful things. Reasonable coordination of multiple vehicles in
different road environments can improve traffic safety [18], improve traffic
efficiency [19], and reduce fuel consumption [20]. These optimized indexes are
of great significance to social traffic operations [21]. Based on the
development of autonomous multi-vehicle coordination, in order to improve the
transportation capacity of multiple vehicles, and reduce fuel consumption. A
lot of useful techniques have been developed, and one of the very useful
techniques is platoon [22], which has been studied in great detail. Platoon
driving refers to the situation where multiple vehicles coordinate and the
rear vehicles follow the front vehicles at a short distance.

Reinforcement learning is a powerful method for decision-making, which has
been applied to address autonomous driving problems in recent years. Multi-
agent reinforcement learning algorithms can coordinate agents effectively and
explore a large number of potential different environments quickly. It not
only enables multi-agent to adapt to the dense and complex dynamic driving
environment but also enables multi-agents to make effective collaborative
decisions [23].

###  1.2 Reinforcement Learning for Overtaking

Overtaking is an important way for AVs to improve driving efficiency,
especially in mixed-traffic environments that contain AVs and human-driven
vehicles (HDVs). There are some works [24, 25, 26] using reinforcement
learning to handle single-vehicle overtaking problems. When making overtaking
decisions, the agent needs to consider that other vehicles are in the vicinity
of the agent and that different vehicles among them are traveling at different
speeds. This requires agents to have multiple abilities to deal with
overtaking problems [26]. Experienced human drivers can handle overtaking
problems better, [27] used curriculum reinforcement learning to make the agent
perform overtaking operations comparable to experienced human drivers. When
the platoon cannot travel at a relatively high or expected speed in the
traffic flow, it will lead to a reduction in the efficiency of vehicle
transportation. The platoon needs to speed up to overtake the slow vehicles in
front and reach the destination faster. As shown in Fig.1, platoon overtaking
requires not only close coordination among members of the platoon but also the
prevention of collision with other human vehicles around the platoon during
the process of overtaking. The reinforcement learning algorithm with the more
effective exploration of the environment can realize and improve the platoon’s
performance during overtaking.

However, to the best of our knowledge, using RL, especially multi-agent RL,
for the AV platoon overtaking problem has rarely been studied. It is still an
open and challenging problem, which motivates our studies in this paper.
Inspired by NoisyNet [28], we added the parameterized factorised Gaussian
noise to the linear layer networks weights of the multi-agent deep Q-learning
network, which induced stochasticity in the agent’s policy that can be used to
aid efficient exploration. The factorised Gaussian noise parameters are
learned by gradient descent along with the weights of the remaining networks.
The longitudinal vehicle following distance in a platoon cannot be designed to
be a fixed value. AVs at the end of a fixed-distance platoon may collide with
nearby HDVs in overtaking passes, which will make the platoon much less safe
when overtaking. Self-driving platoon faces the possibility of encountering
other HDVs in the process of driving and overtaking, and need a suitable
safety distance to adjust the self-driving vehicles policy to reduce the risk
of collision. With the consideration of the above factors, a domain-tailored
customized reward function is designed to achieve high-speed safe platoon
overtaking. In order to reduce fluctuations arosen from the vehicle following
reward, coefficients are added to the same lane following reward and the
following distance interval reward, respectively. The total reward curve is
more likely to converge after adding the coefficients, which is extremely
helpful for multi-agents to learn stable policy. The contributions of the
paper are summarized as follows:

  * 1.

The NoisyNet based multi-agent deep Q network (NoisyNet-MADQN) algorithm is
developed for multi-vehicle platoon overtaking. By adding the parameterized
factorised Gaussian noise to the linear layer networks weights of the multi-
agent deep Q-learning network, which induced stochasticity in the agent’s
policy that improves the exploration efficiency. The parameters of the
factorised Gaussian noise are learned with gradient descent along with the
remaining network weights. To reduce this computational overload, we select
factorised Gaussian noise, which reduces the computational time for generating
random numbers in the NoisyNet-MADQN algorithm.

  * 2.

By considering the factors of overtaking, speed, collision, time headway and
vehicle following, the domain-tailored customized platoon overtaking is
designed. The safety distance is designed in the vehicle-following reward to
reduce the risk of collisions between the platoon and nearby HDVs while
straight driving and overtaking. It is able to reduce collision rate of the
AVs in the safety distance by the reward adaptively adjusting the distance
they maintain from the AV in front.

The rest of the paper is organized as follows. In Section II, the preliminary
works of RL and the NoisyNet are presented. Section III presents the reward
function’s design and the platoon overtaking algorithm. Section IV presents
the experiments and results. Finally, Section V concludes our work.

##  2 Problem Formulation

###  2.1 Preliminary of Reinforcement Learning

In reinforcement learning, the agent’s goal is to learn the optimal policy
$\pi^{*}$ that maximizes the cumulative future rewards
$R_{t}=\begin{matrix}\sum_{k=0}^{T}\gamma^{k}r_{t+k}\end{matrix}$, where $t$
is the time step, $r_{t+k}$ is the reward at time step $t+k$, and
$\gamma\in(0,1]$ is the discount factor that quantifies the relative
importance of future rewards. At time step $t$, the agent observes the state
$s_{t}\in\boldsymbol{S}\subseteq\mathbb{R}^{n}$, selects an action
$a_{t}\in\boldsymbol{A}\subseteq\mathbb{R}^{m}$, and receives a reward signal
$r_{t}\subseteq\mathbb{R}$. $n$ represents the total number of agents, $i\in
n$.

####  2.1.1 Action Space

An agent’s action space $\boldsymbol{A_{i}}$ is defined as a set of high-level
control decisions. Decision-making behaviors include turning left, turning
right, idling, speeding up, and slowing down.

####  2.1.2 State Space

The state of agent ${i}$, $\boldsymbol{S}_{i}$, is defined as a matrix of
dimension $\boldsymbol{N_{\mathcal{N}}}_{i}\times\boldsymbol{m}$, where
$\boldsymbol{N_{\mathcal{N}}}_{i}$ is the number of observed vehicles and
$\boldsymbol{m}$ is the number of features. $\boldsymbol{Ispresent}$ is a
binary variable that indicates whether there are other observable vehicles in
the vicinity of 150 meters from the ego vehicle. $\boldsymbol{x}$ is the
observed longitudinal position of the vehicle relative to the ego vehicle.
$\boldsymbol{y}$ represents the lateral position of the observed vehicle
relative to the ego vehicle. $\boldsymbol{v_{x}}$ and $\boldsymbol{v_{y}}$
represent the longitudinal and lateral speeds of the observed vehicle relative
to the ego vehicle, respectively.

In the highway simulator, we assume that the ego vehicle can only obtain
information about neighboring vehicles within 150 meters of the longitudinal
distance of the ego vehicle. In the considered two-lane scenario (see Fig .1),
the neighboring vehicle is located in the lane and its neighboring lanes
closest to the ego vehicle, with the $2^{th}$ AV as the ego vehicle and its
neighboring vehicles as the $1^{th}$ AV, $3^{th}$ AV, $6^{th}$ HDV, and
$7^{th}$ HDV.

####  2.1.3 Reward Distribution

In this paper, the NoisyNet multi-agent deep Q-learning is developed. As a
multi-agent algorithm, since the vehicles in the platoon are the same type of
vehicles, we assume that all the agents share the same network structure and
parameters. Our algorithm aims to maximize the overall reward. To solve the
communication overhead and credit assignment problems [29], we use the
following local reward design [30]. So, the reward for the $i^{th}$ agent at
time $t$ is defined as:

$r_{i,t}=\frac{1}{\left|{V_{i}}\right|}\sum_{j\in V_{i}}r_{j,t},$ (1)

where $\left|V_{i}\right|$ denotes the cardinality of a set containing the ego
vehicle and its close neighbors. This reward design includes only the rewards
of the agents most relevant to the success or failure of the task [31].

###  2.2 Multi-agent Reinforcement Learning (MARL) and NoisyNets

This subsection will focus on the NoisyNets and MARL. In the NoisyNet, its
neural network weights and biases are perturbed by a function of noise
parameters. These parameters are adjusted according to gradient descent [28].
They assume that $y=f_{\theta}(x)$ is a neural network parameterized by a
vector of noise parameters $\theta$ that accepts input $x$ and output $y$. In
our experiments, we assume that there are $n$ AVs in the experimental
environment. Each AV represents an agent, the $i^{th}$ AV represents the
$i^{th}$ agent, $i\in n$. $x_{i}$ represents the observed state of the
$i^{th}$ agent, $y_{i}$ represents the action of the $i^{th}$ agent. The noise
parameter $\theta_{i}$ is denoted as
$\theta_{i}\overset{def}{=}\mu_{i}+\Sigma_{i}\odot\varepsilon_{i}$,
$\zeta_{i}\overset{def}{=}(\mu_{i},\Sigma_{i})$ is a set of learnable
parameter vectors, $\varepsilon_{i}$ is a zero-mean noise vector with fixed
statistics, and $\odot$ denotes element multiplication. The loss of the neural
network is wrapped by the expectation of the noise
$\varepsilon_{i}:\bar{L}(\zeta_{i})\overset{def}{=}\mathbb{E}[L(\theta_{i})]$.
Then, the set of parameters $\zeta_{i}$ is optimized. Consider the linear
layers of the neural networks with $p$ inputs and $q$ outputs in these
experiments, represented by

$y_{i}=w_{i}x_{i}+b_{i},$ (2)

where $x_{i}\in\mathbb{R}^{p}$ are the layers inputs,
$w_{i}\in\mathbb{R}^{q\times p}$ the weight matrix, and
$b_{i}\in\mathbb{R}^{q}$ the bias. The corresponding noisy linear layers are
defined as:

$y_{i}\overset{def}{=}(\mu_{i}^{w_{i}}+\sigma_{i}^{w_{i}}\odot\varepsilon_{i}^{w_{i}})x_{i}+\mu_{i}^{b_{i}}+\sigma_{i}^{b_{i}}\odot\varepsilon_{i}^{b_{i}},$
(3)

where $\mu_{i}^{w_{i}}+\sigma_{i}^{w_{i}}\odot\varepsilon_{i}^{w_{i}}$ and
$\mu_{i}^{b_{i}}+\sigma_{i}^{b_{i}}\odot\varepsilon_{i}^{b_{i}}$ replace
correspondingly $w_{i}$ and $b_{i}$ in Eq.(2). The parameters
$\mu_{i}^{w_{i}}\in\mathbb{R}^{q\times p}$,
$\mu_{i}^{b_{i}}\in\mathbb{R}^{q}$, $\sigma_{i}^{w_{i}}\in\mathbb{R}^{q\times
p}$, $\sigma_{i}^{b_{i}}\in\mathbb{R}^{q}$, are learnable whereas
$\varepsilon_{i}^{w_{i}}\in\mathbb{R}^{q\times p}$ and
$\varepsilon_{i}^{b_{i}}\in\mathbb{R}^{q}$ are noise random variables.
DeepMind introduced two types of Gaussian noise: independent Gaussian noise
and factorised Gaussian noise. The computation overhead for generating random
numbers in the algorithm is particularly prohibitive in the case of single-
thread agents. To reduce the computation overhead for generating random
numbers in the multi-agent deep Q-learning network, we selected factorised
Gaussian noise.

We factorize $\varepsilon_{j,k}^{w}$, use $p$ unit Gaussian variables
$\varepsilon_{j}$ for the noise of the inputs and $q$ unit Gaussian variables
$\varepsilon_{k}$ for the noise of the outputs. Each $\varepsilon_{j,k}^{w}$
and $\varepsilon_{k}^{b}$ can then be written as:

$\varepsilon_{j,k}^{w}=f(\varepsilon_{j})f(\varepsilon_{k}),$ (4)
$\varepsilon_{k}^{b}=f(\varepsilon_{k}),$ (5)

where $f$ is a real-valued function. In this experiment, we used
$f(x)=\text{sgn}(x)\sqrt{\left|x\right|}$. We can obtain the loss of multiple
noise networks. $\bar{L_{i}}(\zeta_{i})=\mathbb{E}[L_{i}(\theta_{i})]$,
present the expectation of multiple gradients can be obtained directly from:

$\nabla\bar{L_{i}}(\zeta_{i})=\nabla\mathbb{E}[L_{i}(\theta_{i})]=\mathbb{E}[\nabla_{\mu_{i},\Sigma_{i}}L(\mu_{i}+\Sigma_{i}\odot\varepsilon_{i})].$
(6)

Using a Monte Carlo approximation to the above gradients, taking samples
$\xi_{i}$ at each step of optimization:

$\nabla\bar{L_{i}}(\zeta_{i})\approx\nabla_{\mu_{i},\Sigma_{i}}L(\mu_{i}+\Sigma_{i}\odot\xi_{i}).$
(7)

In this work, we will no longer use $\epsilon$-greed, The policy greedily
optimizes the (randomised) action-value function. Then the fully connected
layers of the value network are parameterized to the noisy network, where the
parameters are extracted from the noisy network parameter distribution after
each replay step. Before each action, the noisy network parameters will be
resampled, so that each action step of the algorithm can be optimized. In the
target networks, the parameterized action-value function
$Q(s_{i},a_{i},\varepsilon_{i};\zeta_{i})$ and
$Q(s_{i},a_{i},\varepsilon^{{}^{\prime}}_{i};\zeta^{-}_{i})$ can be regarded
as a random variable when the linear layers in the network are replaced by the
noisy layers. The outer expectation is with respect to the distribution of the
noise variables $\varepsilon$ for the noisy value function
$Q(s_{i},a_{i},\varepsilon_{i};\zeta_{i})$ and the noise variable
$\varepsilon^{{}^{\prime}}$ for the noisy target value function
$Q(s_{i},a_{i},\varepsilon^{{}^{\prime}}_{i};\zeta^{-}_{i})$. So the NoisyNet-
MADQN loss:

$\bar{L_{i}}(\zeta_{i})=\mathbb{E}\left[\mathbb{E}_{(s,a,r,s_{t+1})\sim
D}[r+\gamma\mathbf{max}Q(s_{i},a_{i},\varepsilon^{{}^{\prime}}_{i};\zeta^{-}_{i})-Q(s_{i},a_{i},\varepsilon_{i};\zeta_{i})]^{2}\right].$
(8)

##  3 NoisyNet-MADQN for platoon overtaking

###  3.1 Reward function design

This subsection proposes a novel reward function for reinforcement learning
algorithms to implement platoon overtaking. Reward functions are crucial for
reinforcement learning models. By designing the reward function, we can guide
the learning of RL agents to achieve our purpose.

####  3.1.1 The overtake and speed evaluation

The vehicles will choose to drive at high speed driven by the reward, which
will improve efficiency and allow more vehicles to reach their destination
faster. When the speed of the front HDVs is less than the AVs, the platoon
leader will increase the speed to overtake the front slower vehicles to get
more rewards for completing the overtaking behavior. The other AVs in the
platoon will also overtake the low-speed HDVs in front of them because of the
following reward and speed reward. They follow the leader closely to form a
platoon overtake. So the speed reward can also be seen as an overtaking
reward. Therefore we still define the overtaking and speed reward for this
vehicle as follows:

$r_{os}=\frac{v_{t}-v_{min}}{v_{max}-v_{min}},$ (9)

where ${v_{t}}$, ${v_{min}}$ = 20 m/s, and ${v_{max}}$ = 30 m/s are the
current, minimum, and maximum speeds of the ego vehicle, respectively.

####  3.1.2  The collision penalty design

Safety is the most critical factor in autonomous driving: if a collision
occurs, the collision evaluation $\boldsymbol{r_{c}}$ is set to -1. If there
is no collision, $\boldsymbol{r_{c}}$ is set to 0. The collision evaluation is
defined as

$r_{c}=\begin{cases}-1,&\text{ collision;}\\\ 0,&\text{ safety}.\end{cases}$
(10)

####  3.1.3 The time headway evaluation

The time headway evaluation is defined as

$r_{h}=log\frac{d_{headway}}{t_{h}v_{t}},$ (11)

where $d_{headway}$ is the distance headway and $t_{h}$ is a predefined time
headway threshold. As such, the ego vehicle will get penalized when the time
headway is less than $t_{h}$ and rewarded only when the time headway is
greater than $t_{h}$. In this paper, we choose $t_{h}$ as 1.2 s as suggested
in [32].

####  3.1.4 The vehicles following evaluation

To keep the AVs in the platoon. So, the vehicles following evaluation is
defined as

$r_{f}=\begin{cases}0.3k_{1}{\left|C[i+1].p[0]-C[i].p[0]\right|}/({t_{h}v_{max})},&\text{
$\left|C[i+1].p[0]-C[i].p[0]\right|$ $\leq$ 2$v_{max}$; }\\\ 0.7k_{2},&\text{
$C[i].p[1]=C[i+1].p[1]$},\end{cases}$ (12)

where $c[i]$ represents the $i^{th}$ AV, and $p[0]$,$p[1]$ represents the
longitudinal and lateral coordinates of the AV, respectively.

Figure 2: Illustration of the considered platoon overtaking traffic scenario.
AVs (blue) and HDVs (green) coexist in the straight lane. The long green bar
and the long red bar represent the safety distance and the danger distance
respectively

When the longitudinal distance between the AVs and the vehicle ahead of it in
the platoon at time $t$ is kept within 60 $m$, the reward obtained at this
time is $0.3k_{1}{\left|C[i+1].p[0]-C[i].p[0]\right|}/({t_{h}v_{max}})$.
According to the time headway, we defined the danger distance as the distance
range when the vehicle travels in a straight line at maximum speed for 1.2
$s$. The safety distance is defined as the distance range of 1.2$\sim$2 $s$
when the vehicle travels in a straight line at the maximum speed. As shown in
Fig .2. When the AVs are at the safety distance, the AVs get more rewards.
When the distance between AVs is at the danger distance, the closer the two
AVs are to each other, the less reward they will receive. They also get the
penalty because of the time headway evaluation. The situations encountered by
the self-driving platoon are classified into two types: platoon straight
driving and platoon overtaking. HDVs can have an impact on the safety of the
platoon while straight driving and overtaking. In Fig .2, when the $7^{th}$
HDV enters between the $2^{th}$ AV and the $3^{th}$ AV, the fixed following
distance of the rear AVs in order to follow the front AVs when the self-
driving platoon leader accelerates will increase the risk of collision between
the $3^{th}$ AV and the $7^{th}$ HDV. When the platoon leader in the self-
driving platoon overtakes the $6^{th}$ HDV to obtain a greater overtake and
speed reward. The AVs behind it follow closely. The $2^{th}$ AV and the
$6^{th}$ HDV are very close to each other, and if the following distance is
designed as a constant value, then they have a very high chance of collision.
Based on the above, so we designed the safety distance to reduce the risk of
following when overtaking. AVs can automatically adjust their following
distance through training. After the training, the self-driving platoon
vehicles will choose the safer and more effective action at time $t$ when
faced with such situations.

When the rear AVs in the platoon follow the front AVs in the same lane, at
time $t$ the reward obtained is 0.7. Set the reward for keeping the AVs in the
same lane to be larger than the reward for keeping the safety distance between
the platoon. This is to ensure that when the platoon leader changes lanes to
overtake, the following AVs can also change lanes in time to overtake.

In our training, an episode has 100 steps, and the total reward accumulated by
the acquisition fluctuates too much, which is not conducive to the agent
obtaining a stable policy. So we add the weight $k_{1}$ to the reward obtained
by keeping the distance with the vehicle ahead and the weight $k_{2}$ to the
reward function of keeping in the same lane with the vehicle ahead in the
platoon, respectively. $k_{1}$ and $k_{2}$ have values of 0.25 and 0.3,
respectively. The fluctuation of the total reward becomes smaller after we add
the weights, and it is easier for the agent to obtain a stable policy.
Designing the weight $k_{2}$ to be slightly larger than $k_{1}$ can guide the
vehicles behind the platoon leader to keep up with the platoon leader when the
platoon leader overtakes. The smaller $k_{1}$ enables AVs to take advantage of
the safety distance to avoid the risk of collision when AVs in a platoon
encounter nearby HDVs while overtaking.

####  3.1.5  Total reward

The reward function $\boldsymbol{r_{i}}$ is necessary for training multiple
agents to behave as we desire. Since our goal is to keep the platoon while
overtaking other vehicles. Therefore, the reward for the $\boldsymbol{i}^{th}$
agent at time step $\boldsymbol{t}$ is defined as follows:

$r_{i,t}=w_{c}r_{c}+w_{os}r_{os}+w_{h}r_{h}+w_{f}r_{f}.$ (13)

Among them, $\boldsymbol{w_{c}}$, $\boldsymbol{w_{os}}$, $\boldsymbol{w_{h}}$,
and $\boldsymbol{w_{f}}$ are the positive weight scalars corresponding to the
collision assessment $\boldsymbol{w_{c}}$, the overtake and speed evaluation
$\boldsymbol{w_{os}}$, the time headway evaluation $\boldsymbol{w_{h}}$, and
AVs following evaluation $\boldsymbol{w_{f}}$, respectively. Since safety is
the most important criterion, we made the $\boldsymbol{w_{c}}$ heavier than
others. $\boldsymbol{w_{f}}$ is second only to $\boldsymbol{w_{c}}$ and higher
than the other two weights. The coefficients $\boldsymbol{w_{c}}$,
$\boldsymbol{w_{os}}$, $\boldsymbol{w_{h}}$, and $\boldsymbol{w_{f}}$ for the
reward function are set as 200, 1, 4, and 5.

###  3.2  Noisy network Multi-Agent Deep Q-learning Network

Figure 3: The structure of the NoisyNet-MADQN network design is proposed,
$i=1,2,...,N$. And the numbers in parentheses indicate the size of the layers.

The leader of the platoon needs to choose the optimal policy $\pi^{*}$ when
maintaining the platoon with the followers during overtaking, which improves
the safety and efficiency of driving during overtaking. In the platoon
overtaking scenario, we added the parameterized factorised Gaussian noise to
linear layer network weights of the multi-agent deep Q-learning network.
Factorised Gaussian noise parameters are learned by gradient descent together
with the remaining network weights. The computational overhead associated with
single-thread agents is especially prohibitive. To overcome this computational
overhead we select factorised Gaussian noise, which reduces the computational
time for generating random numbers in the multi-agent deep Q-learning network.
Fig.3 shows the network structure of our algorithm, in which states separated
by physical units are first processed by separate 64-neuron fully connected
(FC) layers. Then all hidden units are combined and fed into the 128-neuron FC
layer. Based on this method, each agent in the multi-agent improves the
exploration of the environment. Our algorithm can obtain a larger optimal
Q-value function for the same level of traffic compared to the original
algorithm to achieve a better policy. More rewards for the platoon also show
that our algorithm can do better than the baseline in platoon overtaking.
Algorithm 3.2 is the detailed procedure of NoisyNet-MADQN.

Algorithm 1 NoisyNet-MADQN

1: Initialize replay buffers $D_{1},...,D_{n}$ to the capacities
$N_{1},...N_{n}$, action-value functions $Q_{1},...Q_{n}$ and target action-
value functions $\hat{Q}^{(1)},...,\hat{Q}^{(n)}$ , $\varepsilon$ set of
random variables of the network, $\zeta$ initial network parameters,
$\zeta^{-}$ initial target network parameters, $N_{T}$ training batch size,
$N^{-}$ target network replacement frequency

2: for $episode$ = {1, …$M$} do

3: Initialize state $s_{0}^{(1),..,(n)}$ $\sim$ $Env$

4: for $t$ = {1, …$T$} do

5: Set $s^{(1),..,(n)}$ $\leftarrow$ $s_{0}^{(1),..,(n)}$

6: Sample a noisy network $\xi_{1}$ $\sim$ $\varepsilon_{1}$, …,$\xi_{n}$
$\sim$ $\varepsilon_{n}$

7: Select
$a_{t}^{1}=\mathbf{max}_{a}Q_{1}(s,a,\xi_{1};\zeta_{1}),...,a_{t}^{(n)}=\mathbf{max}_{a}Q_{n}(s,a,\xi^{(n)};\zeta^{(n)})$

8: Execute joint action ($a_{t}^{(1)}$, …,$a_{t}^{(n)}$ ) and observe reward
$r^{(1)}_{t},...,r^{(n)}_{t}$ , and new state
($s^{(1)}_{t+1},...,s^{(n)}_{t+1}$)

9: Store transition($s_{t}^{(1)}$, $a_{t}^{(1)}$,
$r_{t}^{(1)}$,$s_{t+1}^{(1)}$, …, $s_{t}^{(n)}$, $a_{t}^{(n)}$,
$r_{t}^{(n)}$,$s_{t+1}^{(n)}$) in $D_{1},...,D_{n}$

10: Sample a minibatch of $N_{T}$ transitions
(($s_{j}^{(1)}$,$a_{j}^{(1)}$,$r_{j}^{(1)}$,$s_{j+1}^{(1)}$) $\sim$
$D_{1})_{j=1}^{N_{T}}$,
…,(($s_{j}^{(n)}$,$a_{j}^{(n)}$,$r_{j}^{(n)}$,$s_{j+1}^{(n)}$) $\sim$
$D_{n})_{j=1}^{N_{T}}$

11: Sample the noisy variable for the online network $\xi_{1}$ $\sim$
$\varepsilon_{1}$, …,$\xi_{n}$ $\sim$ $\varepsilon_{n}$

12: Sample the noisy variable for the target network $\xi^{{}^{\prime}}_{1}$
$\sim$ $\varepsilon_{1}$, …,$\xi^{{}^{\prime}}_{n}$ $\sim$ $\varepsilon_{n}$

13: for $j$ $\in$ {1, …,$N_{T}$ } do

14: if $s_{j}$ is a terminal state then

15: $\hat{Q}$ $\leftarrow$ $r_{j}$

16: else

17: $\hat{Q}$ $\leftarrow$ $r_{j}$ \+ $\gamma$ $\mathbf{max}$
$Q$($s_{j}$,$a_{j}$,$\xi^{{}^{\prime}}$ ; $\zeta^{-}$)

18: end if

19: Perform a gradient descent step on ($\hat{Q}_{1}$ \-
$Q_{1}(s_{j}^{(1)},a_{j}^{(1)}$,$\xi_{1}$;$\zeta_{1}))^{2}$, …, ($\hat{Q}_{n}$
\- $Q_{n}(s_{j}^{(n)},a_{j}^{(n)}$,$\xi_{n}$;$\zeta_{n}))^{2}$

20: end for

21: if $t$ $\equiv$ $0$ $(modeN^{-})$ then

22: Update the target network $\zeta_{1}^{-}=\zeta_{1}$ , …,
$\zeta_{n}^{-}=\zeta_{n}$

23: end if

24: end for

25: end for

##  4 Experiments and Discussion

In this section, the effectiveness of the proposed method is verified by
simulation. Some implementation details of the experiments are given, and we
evaluate the performance of the proposed MARL algorithm in terms of training
effectiveness for overtaking in the considered road scenario shown in Fig .1.
The experimental results are also discussed.

For cost and feasibility considerations, we conducted experiments in the
simulator. We use an open-source simulator developed on highway-env [33] and
modify it as needed. The simulator is capable of simulating the driving
environment and vehicle sensors. These vehicles randomly appear on the highway
with different initial speeds of 20-30 m/s. And take random actions.

###  4.1 Experimental Settings

In order to fully demonstrate the effectiveness of our proposed method. Three
traffic density levels were used to evaluate the effectiveness of the proposed
method, corresponding to low, middle, and high levels of traffic congestion.
We train the POMARL algorithm for 200 episodes by applying two different
random seeds. The same random seed is shared among agents. These experiments
were performed on an ubuntu server with a 2.7GHz Intel Core i5 processor and
16GB of RAM. The number of vehicles in different traffic modes is shown in
Table 1.

Table 1:  Traffic density modes.

Density | AVs | HDVs | Explanation
---|---|---|---
1 | 4 | 1-2 | low level
2 | 4 | 2-3 | middle level
3 | 4 | 3-4 | high level

Fig .4 shows the comparison between our algorithm and the baseline algorithm.
It shows that NoisyNet-MADQN performs better than MADQN. We put the rewards
obtained by the two algorithms respectively under low-level, mid-level, and
high-level to make a comprehensive curve comparison. We tabulate these reward
values and divide the rewards within 200 episodes into five intervals. The
value of each interval is their average. In Table 2, the NoisyNet-MADQN
obtains higher rewards than MADQN most of the time. Fig .5 shows snapshots of
the platoon overtaking at low, middle, and high levels, respectively. It shows
that our method can be applied to solve platoon overtaking. From the
comparison of NoisyNet-MADQN and baseline algorithm rewards, it can be found
that the proposed new algorithm obtains better results in platoon overtaking.
It shows that the platoon has achieved better results in both driving
efficiency, safety, and convoy coordination.

(a) Low level

(b) Middle level

(c) High level

Figure 4: Evaluation curves during training with different algorithms for different traffic levels. Table 2: Reward compare Density | Method | 1 $\sim$ 40 | 41 $\sim$ 80 | 81 $\sim$ 120 | 121 $\sim$ 160 | 161 $\sim$ 200
---|---|---|---|---|---|---
low level | NoisyNet-MADQN | -14.31 | 28.11 | 36.03 | 44.32 | 52.51
| MADQN | -5.02 | 23.47 | 28.02 | 27.95 | 33.48
middle level | NoisyNet-MADQN | -6.13 | 12.33 | 34.38 | 45.61 | 50.53
| MADQN | -15.75 | 28.51 | 19.77 | 17.48 | 29.01
high level | NoisyNet-MADQN | -4.48 | 19.28 | 33.52 | 51.58 | 52.34
| MADQN | -19.13 | 28.62 | 26.60 | 31.72 | 31.29

(a) Platoon overtaking at low level

(b) Platoon overtaking at middle level

(c) Platoon overtaking at high level

Figure 5: Platoon overtaking in the simulation environment

##  5 Conclusions

This paper proposes the NoisyNet multi-agent deep Q-learning network. The
computation overhead for generating random numbers in the algorithm is
particularly prohibitive in the case of single-thread agents. To reduce the
computational overhead for generating random numbers in the multi-agent deep
Q-learning network we selected factorised Gaussian noise. By adding the
parameterized factorised Gaussian noise to the linear layer networks weights
of the multi-agent deep Q-learning network, the induced randomness of the
agent’s policy can be used to help effective exploration. The parameters of
the factorised Gaussian noise are learned with gradient descent along with the
remaining network weights. To prove the effectiveness of our proposed
algorithm, the proposed algorithm is compared with the baseline in the platoon
overtaking tasks. By considering overtake, speed, collision, time headway and
following vehicles factors, a domain-tailored reward function is proposed to
accomplish safe platoon overtaking with high speed. The safety distance in the
vehicle following evaluation allows the vehicles in the platoon to adjust the
following distance to avoid collision when the platoon faces the nearby HDVs
inserting into the platoon. It also allows the rear vehicle in the platoon to
avoid collision with the nearby HDVs when following the overtaking vehicle. We
compared it with the existing baseline algorithm at three different traffic
densities and showed that it performs better than the baseline, achieving
reasonable results.

## References

  * [1] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, J. Ma, Cobevt: Cooperative bird’s eye view semantic segmentation with sparse transformers, arXiv preprint arXiv:2207.02202, 2022.
  * [2] W. Chen, R. Xu, H. Xiang, L. Liu, J. Ma, Model-agnostic multi-agent perception framework, arXiv preprint arXiv:2203.13168, 2022.
  * [3] M. A. Javed, E. B. Hamida, On the interrelation of security, qos, and safety in cooperative its, IEEE Transactions on Intelligent Transportation Systems 18 (7) (2017) 1943–1957.
  * [4] Y. Wang, E. Sarkar, W. Li, M. Maniatakos, S. E. Jabari, Stop-and-go: Exploring backdoor attacks on deep reinforcement learning-based traffic congestion control systems, IEEE Transactions on Information Forensics and Security 16 (2021) 4772–4787.
  * [5] Y. Zhang, Z. Ai, J. Chen, T. You, C. Du, L. Deng, Energy-saving optimization and control of autonomous electric vehicles with considering multiconstraints, IEEE Transactions on Cybernetics 52 (10) (2022) 10869–10881.
  * [6] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, V. Koltun, Carla: An open urban driving simulator, in: Conference on robot learning, PMLR, 2017, pp. 1–16.
  * [7] M. Behrisch, L. Bieker, J. Erdmann, D. Krajzewicz, Sumo–simulation of urban mobility: an overview, in: Proceedings of SIMUL 2011, The Third International Conference on Advances in System Simulation, ThinkMind, 2011.
  * [8] R. Xu, Y. Guo, X. Han, X. Xia, H. Xiang, J. Ma, OpenCDA: an open cooperative driving automation framework integrated with co-simulation, in: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE, 2021, pp. 1155–1162.
  * [9] R. Xu, H. Xiang, X. Han, X. Xia, Z. Meng, C.-J. Chen, C. Correa-Jullian, J. Ma, The opencda open-source ecosystem for cooperative driving automation research, IEEE Transactions on Intelligent Vehicles.
  * [10] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, J. Ma, Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 2583–2589.
  * [11] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, J. Ma, V2x-vit: Vehicle-to-everything cooperative perception with vision transformer, in: S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, T. Hassner (Eds.), Computer Vision – ECCV 2022, Springer Nature Switzerland, Cham, 2022, pp. 107–124.
  * [12] R. Xu, J. Li, X. Dong, H. Yu, J. Ma, Bridging the domain gap for multi-agent perception, arXiv preprint arXiv:2210.08451, 2022.
  * [13] X. Cai, W. Jiang, R. Xu, W. Zhao, J. Ma, S. Liu, Y. Li, Analyzing infrastructure lidar placement with realistic lidar, arXiv preprint arXiv:2211.15975.
  * [14] J. Li, R. Xu, X. Liu, J. Ma, Z. Chi, J. Ma, H. Yu, Learning for vehicle-to-vehicle cooperative perception under lossy communication, arXiv preprint arXiv:2212.08273.
  * [15] X. Han, R. Xu, X. Xia, A. Sathyan, Y. Guo, P. Bujanović, E. Leslie, M. Goli, J. Ma, Strategic and tactical decision-making for cooperative vehicle platooning with organized behavior on multi-lane highways, Transportation Research Part C: Emerging Technologies 145 (2022) 103952.
  * [16] H. Xiang, R. Xu, X. Xia, Z. Zheng, B. Zhou, J. Ma, V2xp-asg: Generating adversarial scenes for vehicle-to-everything perception, arXiv preprint arXiv:2209.13679, 2022.
  * [17] K.-F. Chu, A. Y. S. Lam, V. O. K. Li, Joint rebalancing and vehicle-to-grid coordination for autonomous vehicle public transportation system, IEEE Transactions on Intelligent Transportation Systems 23 (7) (2022) 7156–7169.
  * [18] F. Wang, Y. Chen, A novel hierarchical flocking control framework for connected and automated vehicles, IEEE Transactions on Intelligent Transportation Systems 22 (8) (2021) 4801–4812.
  * [19] N. Goulet, B. Ayalew, Distributed maneuver planning with connected and automated vehicles for boosting traffic efficiency, IEEE Transactions on Intelligent Transportation Systems 23 (8) (2022) 10887–10901.
  * [20] A. Validi, C. Olaverri-Monreal, Simulation-based impact of connected vehicles in platooning mode on travel time, emissions and fuel consumption, in: 2021 IEEE Intelligent Vehicles Symposium (IV), 2021, pp. 1150–1155.
  * [21] Y. Zhang, R. Hao, T. Zhang, X. Chang, Z. Xie, Q. Zhang, A trajectory optimization-based intersection coordination framework for cooperative autonomous vehicles, IEEE Transactions on Intelligent Transportation Systems 23 (9) (2022) 14674–14688.
  * [22] T. Sturm, C. Krupitzer, M. Segata, C. Becker, A taxonomy of optimization factors for platooning, IEEE Transactions on Intelligent Transportation Systems 22 (10) (2021) 6097–6114.
  * [23] Z. Bai, P. Hao, W. ShangGuan, B. Cai, M. J. Barth, Hybrid reinforcement learning-based eco-driving strategy for connected and automated vehicles at signalized intersections, IEEE Transactions on Intelligent Transportation Systems 23 (9) (2022) 15850–15863.
  * [24] Y. Yu, C. Lu, L. Yang, Z. Li, F. Hu, J. Gong, Hierarchical reinforcement learning combined with motion primitives for automated overtaking, in: 2020 IEEE Intelligent Vehicles Symposium (IV), 2020, pp. 1–6.
  * [25] M. Kaushik, V. Prasad, K. M. Krishna, B. Ravindran, Overtaking maneuvers in simulated highway driving using deep reinforcement learning, in: 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 1885–1890.
  * [26] D. C. K. Ngai, N. H. C. Yung, A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers, IEEE Transactions on Intelligent Transportation Systems 12 (2) (2011) 509–522.
  * [27] Y. Song, H. Lin, E. Kaufmann, P. Dürr, D. Scaramuzza, Autonomous overtaking in gran turismo sport using curriculum reinforcement learning, in: 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 9403–9409.
  * [28] M. Fortunato, M. G. Azar, B. Piot, J. Menick, M. Hessel, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, S. Legg, Noisy networks for exploration, in: International Conference on Learning Representations, 2018.
  * [29] R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018\.
  * [30] D. Chen, Z. Li, Y. Wang, L. Jiang, Y. Wang, Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic, arXiv preprint arXiv:2105.05701, 2021.
  * [31] I. ElSayed-Aly, S. Bharadwaj, C. Amato, R. Ehlers, U. Topcu, L. Feng, Safe multi-agent reinforcement learning via shielding, arXiv preprint arXiv:2101.11196, 2021.
  * [32] T. Ayres, L. Li, D. Schleuning, D. Young, Preferred time-headway of highway drivers, in: ITSC 2001. 2001 IEEE Intelligent Transportation Systems. Proceedings (Cat. No. 01TH8585), IEEE, 2001, pp. 826–829.
  * [33] E. Leurent, An environment for autonomous driving decision-making, https://github.com/eleurent/highway-env (2018).