#  Tracking Consensus of Networked Random Nonlinear Multi-agent Systems with
Intermittent Communications

Ali Azarbahram Ali Azarbahram is with the Department of Mechanical and Process
Engineering, Rheinland-Pfälzische Technische Universität Kaiserslautern-
Landau, 67663 Kaiserslautern, Germany. Current contact via email address:
<EMAIL_ADDRESS>
###### Abstract

The paper proposes an intermittent communication mechanism for the tracking
consensus of high-order nonlinear multi-agent systems (MASs) surrounded by
random disturbances. Each collaborating agent is described by a class of high-
order nonlinear uncertain strict-feedback dynamics which is disturbed by a
wide stationary process representing the external noise. The resiliency level
of this networked control system (NCS) to the failures of physical devices or
unreliability of communication channels is analyzed by introducing a linear
auxiliary trajectory of the system. More precisely, the unreliability of
communication channels sometimes makes an agent incapable of sensing the local
information or receiving it from neighboring nodes. Therefore, an intermittent
communication scheme is proposed among the follower agents as a consequence of
employing the linear auxiliary dynamics. The closed-loop networked system
signals are proved to be noise-to-state practically stable in probability
(NSpS-P). It has been justified that each agent follows the trajectory of the
corresponding local auxiliary virtual system practically in probability. The
simulation experiments finally quantify the effectiveness of our proposed
approach in terms of providing a resilient performance against unreliability
of communication channels and reaching the tracking consensus.

###### Index Terms:

networked control systems (NCSs), tracking consensus, intermittent
communications over network, wide stationary process, multi-agent systems.

##  I Introduction

The cooperative control of multi-agent systems (MASs) has certainly been among
the most referred topics in the field of systems engineering in the past
decade [1, 2, 3, 4, 5]. The control of collaborative MASs is inevitably linked
with the concept of networked control systems (NCSs) [6, 7, 8, 9]. For
instance, when a networked group of interconnected agents performs a specific
mission, the resiliency of network to measurement failures in transmission
channels is considered among the essential concerns that should be taken into
account. Moreover, the real-world MASs are usually surrounded by environmental
disturbances and the nature of these external forces is in fact random. The
irrefutable presence of environmental disturbances has highlighted the
significance of investigating and analyzing the stability of nonlinear MASs
modeled by stochastic [10, 11, 12, 13] or random [14, 2] differential
equations. With this general overview of the multi-agent NCSs challenges, now
we review some of the most important technical considerations regarding the
implementation of random networked MASs in more details.
In real MAS applications, the induced disturbances by different factors
certainly degrade the overall performance of system and in some cases cause
instability and even system damage. In prior methods for controlling nonlinear
MASs, time varying deterministic external disturbances are considered and
compensated (see for instance, [15]). However, environmental disturbances
contain stochastic components. The stability analysis and control of nonlinear
MASs modeled by stochastic differential equations including standard Wiener
process is widely studied in recent years [10, 13]. In theoretical approaches,
white noise originated possess are favorable due to their convenience in
analysis. However, such a process fails to fully model many realistic
situations. Besides, a standard Wiener process is not strictly differentiable.
Accordingly, colored noise processes have attracted much attention in recent
years [14, 16, 17, 18, 19]. Although the cooperative control of nonlinear MASs
subject to colored noise disturbances is recently addressed in [18] and [19],
agents sometimes cannot well sense the local information or receive it from
neighboring nodes due to the failure of physical devices or unreliability of
communication channels in real NCS implementations. Therefore, it is
reasonable to assume that each agent can make the measurements only
intermittently and the control system is supposed to compensate this effect
[20]. More precisely, investigating the cooperative performance of MASs
subject to the wide stationary processes (which stands for the colored noise)
is more favorable to be addressed while other NCSs limitations have also been
taken into account.
The intermittent control approach is known as a discontinuous mechanism that
is applied periodically on the system [21, 22]. This control method has been
widely studied in recent years and showed significant superiority in terms of
increasing the flexibility and being cost-efficient compared to traditional
continuous control schemes [20]. The intermittent control has different
practical applications in various real-world scenarios including a single
system [23, 24, 25, 26] and collaborative MASs [21, 22]. The state-dependent
intermittent control approaches are proposed in [23, 24] for general classes
of linear networked system. Howbeit, the nature of almost all real systems are
intrinsically nonlinear. The intermittent control schemes are designed for
single nonlinear networked systems in [25, 26]. Subsequently, the
collaborative performance of nonlinear MASs is recently studied in [21, 22] by
introducing the intermittent mechanisms. However, the proposed approaches in
[21, 22] are just valid for the networks of general second-order and first-
order nonlinear MASs, respectively. Additionally, the environmental
disturbances are not taken into account in the nonlinear dynamics of studied
MASs in [21, 22].
By the above discussion, this paper proposes an intermittent communication
approach for the tracking consensus of uncertain high-order nonlinear MASs
subject to environmental loads described by wide stationary process. According
to the authors’ best knowledge, this is the first instance that an
intermittent data transmission mechanism is introduced for a class of strict-
feedback MASs surrounded by random disturbance. The main contributions of this
work are clarified in what follows. (i) Compared to the cooperative control
designs in [18] and [19] for nonlinear MASs subject to environmental loads
that is described wide stationary process, the tracking consensus method in
this paper is resilient to the failure of physical devices or unreliability of
communication channels to better address the concerns of real NCS
implementations. (ii) In comparison to the studied intermittent control
approaches in [21] and [22], the agents dynamics are described in this paper
by uncertain high-order strict-feedback nonlinear random differential
equations including wide stationary process to cover more real scenarios.
The remaining paper is organized as follows. The preliminaries and problem
formulation is first given in section II. By modeling the intermittent
communications over the network and stating that the objectives should be
realized in two steps, the control architecture is constructed to meet the
desired performance and then the closed-loop system stability is analyzed. In
section IV, the effectiveness of the proposed method using a numerical example
is investigated. The last section finally concludes the paper.

##  II Preliminaries and Problem Formulation

Throughout this paper, vectors and matrices are denoted in the bold font and
scalars by normal font. For an undirected graph of $N$ followers,
${\bm{\mathcal{A}}}:=[a_{i,j}]\in\mathbb{R}^{N\times N}$ is the follower
adjacency (or connectivity) matrix with element $a_{i,j}$. ${N}_{i}$ is the
neighbouring set for node $i$. If follower $i$ can directly obtain the
information from the leader, then $b_{i}>0$; otherwise $b_{i}=0$, and
${\bm{\mathcal{B}}}:=diag[b_{1},...,b_{N}]$ $\in\mathbb{R}^{N\times N}$.
Furthermore, the Laplacian matrix is $\bm{\mathcal{L}}$.
The following notations are also used. $\mathbb{R}$: the set of real scalars;
$\mathbb{N}$: the set of natural number; $\mathbb{R}^{n}$: the set of real
column vectors with dimension $n$; $|x|$: absolute value of $x\in\mathbb{R}$;
$||\bm{x}||$: Euclidean norm of the column vector $\bm{x}\in\mathbb{R}^{n}$;
${\rm T}$: the transpose of vector or square matrix; $\otimes$: the Kronecker
product; $\bm{1}_{N}$: column vector of order $N$ with one entries.
Random Dynamics: A random differential equation which typically describes a
system in the presence of colored noise is described as

$\displaystyle\dot{x}={h}({x},t)+g({x},t)\xi(t),$ (1)

in which scalar $x$ is the state and $\xi(t)\in\mathbb{R}$ is a wide
stationary process that represents the colored noise and stands for a noise
with uneven power spectral density function. The nonlinear terms are defined
by an unknown function ${h}({x},t)$ and, a known function $g({x},t)$ that
represents the coefficient of the colored noise.
Definition 1[14]: Let the continuous zero-mean process $\xi(t)$ is
$\mathcal{F}_{t}$-adapted and satisfies $\sup_{t\geq
t_{0}}E_{p}(|\xi(t)|^{2})\leq\xi^{\ast}$ for a positive constant $\xi^{\ast}$.
The described random system (1) is called noise-to-state practically stable in
probability (NSpS-P), if for any positive constant $l_{0}$ there exists a
class-$\scriptstyle\mathcal{K}\mathcal{L}$ functions $r(\bullet,\bullet)$ and
class-$\scriptstyle\mathcal{K}$ function $d(\bullet)$ such that for any
initial condition $x_{0}$ and for all $t\in[t_{0}\leavevmode\nobreak\
\leavevmode\nobreak\ \infty)$, one has

$\displaystyle P(|x(t)|\leq r(|x_{0}|,t-t_{0})+d(\varpi))\geq 1-l_{0}$ (2)

where $\varpi=l^{[a]}\xi^{\ast}+l^{[b]}$ with $l^{[a]}>0$ and $l^{[b]}\geq 0$.
Lemma 1[14]: Suppose that there exists a function $\mathcal{V}\in C^{1}$ and
positive constants $c_{\gamma},a_{1},a_{2},l^{[a]},l^{[b]}$. The random system
(1) is said to have an NSpS-P unique global solution if

$\displaystyle\begin{cases}a_{1}|x|^{2}\leq\mathcal{V}\leq a_{2}|x|^{2},\\\
\dot{\mathcal{V}}\leq-
c_{\gamma}\mathcal{V}+l^{[a]}|\xi|^{2}+l^{[b]}.\end{cases}$ (3)

Furthermore if (3) holds, then
$\lim_{t\rightarrow\infty}E_{p}(\mathcal{V})\leq\frac{\varpi}{c_{\gamma}}$.

###  II-1 Problem formulation

A network of $N$ uncertain nonlinear follower agents labeled with
$i,i=1,\ldots,N$ are supposed to reach the racking consensus with respect to
the virtual leader signal $z_{r}\in\mathbb{R}$. Let
$\bm{x}_{i}\in\mathbb{R}^{n}$ with $\bm{x}_{i}=[x_{i,1},\thinspace
x_{i,2},\ldots,\thinspace x_{i,n}]^{\rm{T}}$ denote the state vector for the
$i$-th agent. The dynamics of each uncertain follower agent subject to random
disturbance is described by the following high-order nonlinear differential
equations

$\displaystyle\begin{cases}\dot{x}_{i,q}=x_{i,q+1}+f_{i,q}(\bar{\bm{x}}_{i,q})+\bm{g}_{i,q}^{\rm{T}}(\bar{\bm{x}}_{i,q})\bm{\xi}_{i},\\\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ q=1,\ldots,n-1,\\\
\dot{x}_{i,n}=u_{i}+f_{i,n}(\bar{\bm{x}}_{i,n})+\bm{g}_{i,n}^{\rm{T}}(\bar{\bm{x}}_{i,n})\bm{\xi}_{i},\\\
z_{i}=x_{i,1},\end{cases}$ (4)

where $u_{i}$ and $z_{i}$ are respectively the control input and the output of
the $i$-th agent.
$f_{i,q}(\bar{\bm{x}}_{i,q}):\mathbb{R}^{q}\rightarrow\mathbb{R}$ is an
unknown smooth function with $\bar{\bm{x}}_{i,q}=[x_{i,1},\thinspace
x_{i,2},\ldots,\thinspace x_{i,q}]^{\rm{T}}$. $\bm{\xi}_{i}\in\mathbb{R}^{m}$
is a continuous zero-mean and $\mathcal{F}_{t}$-adapted process representing
the colored noise that satisfies $\sup_{t\geq
t_{0}}E_{p}(\|\bm{\xi}_{i}\|^{2})\leq\xi_{i}^{\ast}$ for a positive and
bounded constant $\xi_{i}^{\ast}$. Furthermore,
$\bm{g}_{i,q}(\bar{\bm{x}}_{i,q}):\mathbb{R}^{q}\rightarrow\mathbb{R}^{m}$ is
a known and differentiable function.

######  Assumption 1.

The leader trajectory $z_{r}$ is $n$-times continuously differentiable and
there exists a positive constants $c_{z}$ such that $|z_{r}^{(l)}|\leq c_{z},$
for all $l=1,\ldots,n$ where $z_{r}^{(l)}$ stands for the $l$-th derivative of
$z_{r}$.

###  II-2 Distributed Virtual Systems

Suppose that a virtual system is defined locally for each agent $i$, which
evolves according to the following ordinary differential equation

$\displaystyle\dot{\bm{\eta}}_{i}=\bm{A}\bm{\eta}_{i}+\bm{B}\bm{K}\bm{\zeta}_{i},$
(5)

where $\bm{\eta}_{i}\in\mathbb{R}^{n}$ with
$\bm{\eta}_{i}=[\eta_{i,1},\thinspace\eta_{i,2},\ldots,\thinspace\eta_{i,n}]^{\rm{T}}$
is the objective trajectory of the virtual system.
$\bm{\zeta}_{i}\in\mathbb{R}^{n}$ is an intermittent (with respect to the
communication modes) distributed virtual signal which aims at forcing the
system (5) to achieve the tracking consensus with respect to the leader’s
information. $\bm{K}\in\mathbb{R}^{1\times n}$ is a control gain vector to be
chosen according to the design requirements. The pair $(\bm{A},\bm{B})$ is
stabilizable. The matrix $\bm{A}\in\mathbb{R}^{n\times n}$ and the vector
$\bm{B}\in\mathbb{R}^{n}$ are defined as

$\displaystyle\bm{A}=\kern-3.00003pt\left[\begin{matrix}\bm{0}_{(n-1)\times
1}&\bm{I}_{(n-1)\times(n-1)}\\\
0&\bm{0}_{1\times(n-1)}\end{matrix}\right],\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \bm{B}=[\bm{0}_{1\times(n-1)}\leavevmode\nobreak\
\leavevmode\nobreak\ \thinspace 1]^{\rm{T}}.$

Figure 1: Schematic time diagram for the intermittent communication operation
modes.

##  III Design and Analysis

Let $\tau^{[0]},\tau^{[1]},\ldots,\tau^{[\kappa]},\tau^{[\kappa+1]},\ldots$
denote the time sequence at which the communication network switches to the ON
mode. In our problem formulation, the first communicating ON mode instant is
set to $\tau^{[0]}=0$. Subsequent to each ON mode interval, the networked
system experiences an OFF mode of a particular duration. Let
$\tau^{[0,e]},\tau^{[1,e]},\ldots,\tau^{[\kappa,e]},\tau^{[\kappa+1,e]},\ldots$
denote the time sequence at which the communication network switches to the
OFF mode. The overall time sequence of switches in order of occurrence is as
follows:
$\tau^{[0]},\tau^{[0,e]},\tau^{[1]},\tau^{[1,e]},\ldots,\tau^{[\kappa]},\tau^{[\kappa,e]},\ldots$.
To recapitulate, the following holds true regarding the communication
intermittent modes

$\displaystyle\mathscr{M}:\begin{cases}\mathcal{O}^{[\kappa]}:\leavevmode\nobreak\
t\in[\tau^{[\kappa]},\tau^{[\kappa,e]})\leavevmode\nobreak\
\leavevmode\nobreak\ &\text{ON Mode},\\\
\mathcal{F}^{[\kappa]}:\leavevmode\nobreak\
t\in[\tau^{[\kappa,e]},\tau^{[\kappa+1]})\leavevmode\nobreak\
\leavevmode\nobreak\ &\text{OFF Mode},\end{cases}$ (6)

for $\kappa\in\mathbb{N}$. The total duration of two consecutive On and OFF
mode intervals is denoted by
${T}^{[\kappa]}=\tau^{[\kappa+1]}-\tau^{[\kappa]}$ which satisfies
$0<{T}^{[\kappa]}\leq{T}$ for a positive constant ${T}$. The duration of a
particular ON mode interval for communication network is also defined as
$\mathscr{G}^{[\kappa]}=\tau^{[\kappa,e]}-\tau^{[\kappa]}$. Easily, the
duration of a particular OFF mode interval for communication network is
derived as $\mathscr{C}^{[\kappa]}={T}^{[\kappa]}-\mathscr{G}^{[\kappa]}$. To
better illustrate the time notation regarding the intermittent operation
modes, a schematic diagram is provided in Fig. 1.

Design the intermittent distributed virtual control signal $\bm{\zeta}_{i}(t)$
as

$\displaystyle\bm{\zeta}_{i}=\begin{cases}\sum_{j\in{N}_{i}}a_{i,j}(\bm{\eta}_{j}-\bm{\eta}_{i})+b_{i}(\bar{\bm{z}}_{r}-\bm{\eta}_{i}),&t\in\mathcal{O}^{[\kappa]},\\\
\bm{0}_{n\times 1},&t\in\mathcal{F}^{[\kappa]},\end{cases}$ (7)

where $\bar{\bm{z}}_{r}=[z_{r},\thinspace
z_{r}^{\prime},\thinspace\ldots,z_{r}^{(n-1)}]^{\rm{T}}$.

Let
$\bm{\varsigma}=[\bm{\varsigma}_{1}^{\rm{T}},\thinspace\ldots,\thinspace\bm{\varsigma}_{N}^{\rm{T}}]^{\rm{T}}$
and
$\bm{\eta}=\big{[}\bm{\eta}_{1}^{\rm{T}},\thinspace\ldots,\thinspace\bm{\eta}_{N}^{\rm{T}}\big{]}^{\rm{T}}$,
where $\bm{\varsigma}_{i}=\bm{\eta}_{i}-\bar{\bm{z}}_{r}$. A Lyapunov function
candidate is then chosen as

$\mathcal{V}_{e}=\bm{\varsigma}^{\rm{T}}(\mathbf{I}_{N}\otimes\bm{P})\bm{\varsigma}$

in which $\bm{P}$ is a positive definite matrix that satisfies the following
inequalities

$\displaystyle\small\begin{cases}\bm{A}^{\rm{T}}\bm{P}+\bm{P}\bm{A}-2\bm{P}\bm{B}\bm{B}^{\rm{T}}\bm{P}+c_{1}\bm{I}<0,\\\
\bm{A}^{\rm{T}}\bm{P}+\bm{P}\bm{A}-c_{3}\bm{P}<0,\end{cases}$ (8)

where $c_{1}$ and $c_{3}$ are some positive design constants. According to
(7), for all $t\in\mathcal{O}^{[\kappa]}$, the time derivative of the Lyapunov
function candidate $\mathcal{V}_{e}$ gives

$\displaystyle\dot{\mathcal{V}}_{e}$
$\displaystyle=\bm{\varsigma}^{\rm{T}}(\mathbf{I}_{N}\otimes(\bm{A}^{\rm{T}}\bm{P}+\bm{P}\bm{A})-(\bm{\mathcal{L}}+\bm{\mathcal{B}})\otimes
2\bm{P}\bm{B}\bm{K})\bm{\varsigma}$
$\displaystyle-2\bm{\varsigma}^{\rm{T}}(\mathbf{1}_{N}\otimes
z_{r}^{(n)}\bm{P}\bm{B}).$ (9)

The control gain vector is designed as $\bm{K}=c_{0}\bm{B}^{\rm{T}}\bm{P}$
where the positive integer $c_{0}$ is chosen such that
$c_{0}\lambda_{\min}(\bm{\mathcal{L}}+\bm{\mathcal{B}})\geq 1$. The time
derivative of the Lyapunov function candidate $\dot{\mathcal{V}}_{e}$
satisfies

$\dot{\mathcal{V}}_{e}\leq-
c_{\alpha,1}\bm{\varsigma}^{\rm{T}}\bm{\varsigma}+c_{\beta}$

with the constants

$c_{\alpha,1}=c_{1}-\frac{c_{z}N\|\bm{P}\|}{c_{2}}$

and

$c_{\beta}=c_{2}c_{z}\|\bm{P}\|$

and positive constant $c_{2}$. Hence,

$\dot{\mathcal{V}}_{e}(t)\leq-\delta_{\alpha}\mathcal{V}_{e}(t)+c_{\beta}$

with

$\delta_{\alpha}=\frac{c_{\alpha,1}}{\|\bm{P}\|}.$

For $t\in\mathcal{O}^{[\kappa]}$, it holds that

$\displaystyle\mathcal{V}_{e}(t)\leq\mathcal{V}_{e}(\tau^{[\kappa]})e^{-\delta_{\alpha}(t-\tau^{[\kappa]})}+\frac{c_{\beta}}{\delta_{\alpha}}(1-e^{-\delta_{\alpha}(t-\tau^{[\kappa]})}).$
(10)

For all $t\in\mathcal{F}^{[\kappa]}$, let

$\delta_{\beta}=c_{3}+\frac{c_{z}N}{c_{2}}.$

The time derivative of the Lyapunov function candidate gives

$\dot{\mathcal{V}}_{e}(t)\leq\delta_{\beta}\mathcal{V}_{e}(t)+c_{\beta}.$

Consequently, it holds that

$\displaystyle\mathcal{V}_{e}(t)\leq\mathcal{V}_{e}(\tau^{[\kappa,e]})e^{\delta_{\beta}(t-\tau^{[\kappa,e]})}+\frac{c_{\beta}}{\delta_{\beta}}(e^{\delta_{\beta}(t-\tau^{[\kappa,e]})}-1).$
(11)

Let

$\Lambda^{[\kappa]}=\delta_{\alpha}(\tau^{[\kappa,e]}-\tau^{[\kappa]})-\delta_{\beta}(\tau^{[\kappa+1]}-\tau^{[\kappa,e]}),$

and

$\varkappa=(\frac{c_{\beta}}{\delta_{\beta}}+\frac{c_{\beta}}{\delta_{\alpha}})e^{\delta_{\beta}{T}}-\frac{c_{\beta}}{\delta_{\beta}}.$

Therefore, there exists a ${T}$ for which $\varkappa>0$.

######  Theorem 1.

Under Assumption 1, consider the auxiliary trajectory of $i$-th agent in (5).
Suppose that the data is intermittently transmitted among the agents according
to the intermittent rule in (7). Then, design the control gain
$\bm{K}=c_{0}\bm{B}^{\rm{T}}\bm{P}$ for the objective trajectory. If for all
$\kappa\in\mathbb{N}$

$\displaystyle(\delta_{\alpha}+\delta_{\beta})\mathscr{G}^{[\kappa]}-\delta_{\beta}{T}^{[\kappa]}>0,$
(12)

holds true, then the communication network is resilient to be switched to the
OFF mode with a maximum duration of

$\mathscr{C}^{[\kappa]}\approx(\mathscr{G}^{[\kappa]})(\frac{\delta_{\alpha}+\delta_{\beta}}{\delta_{\beta}}-1)\times
100\%.$

Also, it holds that
$\lim\nolimits_{t\rightarrow\infty}|\eta_{i}-z_{r}|\leq\varepsilon^{[e]}$, for
a positive constant $\varepsilon^{[e]}$.

###### Proof.

By recursion, the following is derived

$\displaystyle{\mathcal{V}}_{e}(\tau^{[\kappa+1]})$
$\displaystyle\leq[{\mathcal{V}}_{e}(\tau^{[\kappa]})-\frac{c_{\beta}}{\delta_{\alpha}}]e^{-\Lambda^{[\kappa]}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
+(\frac{c_{\beta}}{\delta_{\beta}}+\frac{c_{\beta}}{\delta_{\alpha}})e^{\delta_{\beta}(\tau^{[\kappa+1]}-\tau^{[\kappa,e]})}-\frac{c_{\beta}}{\delta_{\beta}}$
$\displaystyle\leq[{\mathcal{V}}_{e}(\tau^{[\kappa]})-\frac{c_{\beta}}{\delta_{\alpha}}]e^{-\Lambda^{[\kappa]}}+\varkappa$
$\displaystyle\leq{\mathcal{V}}_{e}(t_{0})e^{-\sum_{l=0}^{\kappa}\Lambda^{[l]}}-\frac{c_{\beta}}{\delta_{\alpha}}[e^{-\sum_{l=0}^{\kappa}\Lambda^{[l]}}$
$\displaystyle+e^{-\sum_{l=1}^{\kappa}\Lambda^{[l]}}+\ldots+e^{-\sum_{l=\kappa}^{\kappa}\Lambda^{[l]}}]$
$\displaystyle+\varkappa[e^{-\sum_{l=1}^{\kappa}\Lambda^{[l]}}+e^{-\sum_{l=2}^{\kappa}\Lambda^{[l]}}+\ldots$
$\displaystyle+e^{-\sum_{l=\kappa}^{\kappa}\Lambda^{[l]}}]+\varkappa$
$\displaystyle\leq{\mathcal{V}}_{e}(t_{0})e^{-\sum_{l=0}^{\kappa}\Lambda^{[l]}}+\varkappa[e^{-\sum_{l=1}^{\kappa}\Lambda^{[l]}}$
$\displaystyle+e^{-\sum_{l=2}^{\kappa}\Lambda^{[l]}}+\ldots+e^{-\sum_{l=\kappa}^{\kappa}\Lambda^{[l]}}]+\varkappa.$
(13)

Let $\Lambda=\min\limits_{0\leq l\leq\kappa}\Lambda^{[l]}$. Then,

$\displaystyle{\mathcal{V}}_{e}(\tau^{[\kappa+1]})$
$\displaystyle\leq{\mathcal{V}}_{e}(t_{0})e^{-(\kappa+1)\Lambda}+\varkappa[e^{-\kappa\Lambda}+\ldots+e^{-\Lambda}]+\varkappa$
$\displaystyle\leq[{\mathcal{V}}_{e}(t_{0})+\frac{\varkappa
e^{\Lambda}}{1-e^{\Lambda}}]e^{-(\kappa+1)\Lambda}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
-\frac{\varkappa}{1-e^{\Lambda}}+\varkappa.$ (14)

For all $t>0$, there exists $\iota\in\mathbb{N}$ such that
$t\in[\tau^{[\iota+1]},\tau^{[\iota+2]})$. Since, $t<(\iota+2){T}$, it also
holds that

$-(\iota+1)\Lambda\leq-\frac{\Lambda}{{T}}t-\Lambda.$

For $t\in[\tau^{[\iota+1]},\tau^{[\iota+1,e]})$, it holds that

$\displaystyle{\mathcal{V}}_{e}(t)$
$\displaystyle\leq\mathcal{V}_{e}(\tau^{[\iota+1]})e^{-\delta_{\alpha}(t-\tau^{[\iota+1]})}+\frac{c_{\beta}}{\delta_{\alpha}}(1-e^{-\delta_{\alpha}(t-\tau^{[\iota+1]})})$
$\displaystyle\leq\mathcal{V}_{e}(\tau^{[\iota+1]})+\frac{c_{\beta}}{\delta_{\alpha}}$
$\displaystyle\leq[{\mathcal{V}}_{e}(t_{0})+\frac{\varkappa
e^{\Lambda}}{1-e^{\Lambda}}]e^{-(\iota+1)\Lambda}-\frac{\varkappa}{1-e^{\Lambda}}+\varkappa+\frac{c_{\beta}}{\delta_{\alpha}}$
$\displaystyle\leq\pi_{1}e^{-\frac{\Lambda}{{T}}t}+\varepsilon_{1},$ (15)

where

$\pi_{1}={\mathcal{V}}_{e}(t_{0})e^{-\Lambda}+\frac{\varkappa}{1-e^{\Lambda}}$

and

$\varepsilon_{1}=-\frac{\varkappa}{1-e^{\Lambda}}+\varkappa+\frac{c_{\beta}}{\delta_{\alpha}}.$

Similarly for $t\in[\tau^{[\iota+1,e]},\tau^{[\iota+2]})$, it holds that

$\displaystyle{\mathcal{V}}_{e}(t)\leq\pi_{2}e^{-\frac{\Lambda}{{T}}t}+\varepsilon_{2},$
(16)

where

$\pi_{2}=e^{\delta_{\beta}{T}}[{\mathcal{V}}_{e}(t_{0})e^{-\Lambda}+\frac{\varkappa}{1-e^{\Lambda}}]$

and

$\varepsilon_{2}=e^{\delta_{\beta}{T}}[-\frac{\varkappa}{1-e^{\Lambda}}+\varkappa+\frac{c_{\beta}}{\delta_{\alpha}}]+\varkappa.$

Let $\pi^{\star}=\max\\{\pi_{1},\pi_{2}\\}$ and
$\varepsilon^{\star}=\max\\{\varepsilon_{1},\varepsilon_{2}\\}$. Then,

$\displaystyle{\mathcal{V}}_{e}(t)\leq\pi^{\star}e^{-\frac{\Lambda}{{T}}t}+\varepsilon^{\star},$
(17)

for all $t>0$. If (12) holds true, then $\Lambda>0$ and

$\max\\{\mathscr{C}^{[\kappa]}\\}\approx(\mathscr{G}^{[\kappa]})(\frac{\delta_{\alpha}+\delta_{\beta}}{\delta_{\beta}}-1).$

Hence, the closed-loop systems error $\bm{\varsigma}_{i}$ exponentially
converge to $\varepsilon^{\star}$. Accordingly, the state vector of each agent
$i$ exponentially converges to a bounded region around $\bar{\bm{z}}_{r}$ as
time tends to infinity, i.e.,

$\lim\nolimits_{t\rightarrow\infty}|\eta_{i}-z_{r}|\leq\varepsilon^{[e]}.$

This completes the proof. ∎

To continue, the following closed-loop system errors are defined

$\displaystyle e_{i,1}=z_{i}-\eta_{i,1},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
e_{i,q}=x_{i,q}-\alpha_{i,q-1}$ (18)

where $q=2,\ldots,n$ and $\alpha_{i,q-1}$ is the virtual input to be designed
later. We will derive the control laws and adaptive rules for the follower
agents in what follows.
Let $\bar{f}_{i,1}=f_{i,1}(\bar{\bm{x}}_{i,1})-\eta_{i,2}$ and
$\bar{f}_{i,q}=f_{i,q}(\bar{\bm{x}}_{i,q})-\dot{\alpha}_{i,q-1}$ for
$q=2,\ldots,n$. The neural networks (NNs)[27] approach, is in most cases
utilized locally to approximate unknown functions of the form $\bar{f}_{i,q}$
for $q=1,\ldots,n$. Hence, for positive constants $\epsilon_{i,q}$, it follows
that $\bar{f}_{i,q}=\bm{\theta}_{i,q}^{\rm
T}\bm{\varphi}_{i,q}+\epsilon_{i,q}$. $\bm{\theta}_{i,q}$ is an unknown
vector. The activation vectors of Gaussian basis functions are denoted by
$\bm{\varphi}_{i,q}$. Additionally, one has
$|\epsilon_{i,q}|^{2}\leq\epsilon_{i,q}^{\ast}$, for some unknown positive
bounded parameters $\epsilon_{i,q}^{\ast}$. Since $\bm{\theta}_{i,q}$ is an
unknown constant vector, the approximation errors can be defined as

$\tilde{\bm{\theta}}_{i,q}=\bm{\theta}_{i,q}-\hat{\bm{\theta}}_{i,q}$

where $\hat{\bm{\theta}}_{i,q}$ stands for the approximated value of
$\bm{\theta}_{i,q}$. By applying Young’s inequality one gets

$e_{i,q}e_{i,q+1}\leq\frac{\rho_{i,q}^{2}}{2}e_{i,q}^{2}+\frac{1}{2\rho_{i,q}^{2}}e_{i,q+1}^{2}$

for $q=1,\ldots,n-1$, and the followings hold true for $q=1,\ldots,n$

$\displaystyle\begin{cases}e_{i,q}\epsilon_{i,q}\leq\frac{\rho_{i,q}^{2}}{2}e_{i,q}^{2}+\frac{1}{2\rho_{i,q}^{2}}\epsilon_{i,q}^{\ast},\\\
e_{i,q}\bm{g}_{i,q}^{\rm{T}}(\bar{\bm{x}}_{i,q})\bm{\xi}_{i}\leq\frac{\rho_{i,q}^{2}}{2}e_{i,q}^{2}\|\bm{g}_{i,q}\|^{2}+\frac{1}{2\rho_{i,q}^{2}}\|\bm{\xi}_{i}\|^{2},\end{cases}$

with positive design constants $\rho_{i,q},\leavevmode\nobreak\ q=1,\ldots,n$.
The actual control input $u_{i}$ is designed as follows

$\displaystyle u_{i}$
$\displaystyle=-(\mathcal{K}_{i,n}+\frac{\rho_{i,n}^{2}}{2}\|\bm{g}_{i,n}\|^{2})e_{i,n}-\hat{\bm{\theta}}_{i,n}^{\rm
T}\bm{\varphi}_{i,n},$ (19)

and the $q$-th virtual input $\alpha_{i,q}$ for $q=1,\ldots,n-1$, and the
$q$-th adaptive rule for $q=1,\ldots,n$ are

$\displaystyle\alpha_{i,q}$
$\displaystyle=-(\mathcal{K}_{i,q}+\frac{\rho_{i,q}^{2}}{2}\|\bm{g}_{i,q}\|^{2})e_{i,q}-\hat{\bm{\theta}}_{i,q}^{\rm
T}\bm{\varphi}_{i,q},$ (20) $\displaystyle\dot{\hat{\bm{\theta}}}_{i,q}$
$\displaystyle=\gamma_{i,q}({e}_{i,q}\bm{\varphi}_{i,q}-\sigma_{i,q}\hat{\bm{\theta}}_{i,q}),$
(21)

where $\sigma_{i,q}$ is a $\sigma$-modification parameter and
$\mathcal{K}_{i,q}$ is a positive gain to be chosen according to the control
objectives. The design procedure is now complete and the following Theorem is
provided to summarize the results.

######  Theorem 2.

Consider a group of uncertain nonlinear MASs defined in (4). Design the
virtual signals and adaptive rules in (20), (21). Furthermore, consider that
the actuator updates are done according to (19). Then, the tracking errors are
NSpS-P with a unique global solution. Furthermore,

$\lim\nolimits_{t\rightarrow\infty}E_{p}(|z_{i}-z_{r}|)<\varepsilon_{i,v}$

for all $i\in V$ and a positive constant $\varepsilon_{i,v}$ i.e., the
tracking consensus performance can be achieved.

###### Proof.

Let $\Delta_{i,q}=\mathcal{K}_{i,q}-\rho_{i,q}^{2}$ for $q=1,\ldots,n-1$ and
$\Delta_{i,n}=\mathcal{K}_{i,n}-0.5\rho_{i,n}^{2}$. By applying Young’s
inequality, it follows that

$\sigma_{i,q}\tilde{\bm{\theta}}_{i,q}^{\rm
T}\bm{\theta}_{i,q}\leq\frac{1}{2}\sigma_{i,q}\tilde{\bm{\theta}}_{i,q}^{\rm
T}\tilde{\bm{\theta}}_{i,q}+\frac{1}{2}\sigma_{i,q}\|\bm{\theta}_{i,q}\|^{2}.$

A Lyapunov function candidate is chosen as

$\displaystyle\mathcal{V}_{i}=\sum_{q=1}^{n}\frac{1}{2}e_{i,q}^{2}+\sum_{q=1}^{n}\frac{1}{2\gamma_{i,q}}\tilde{\bm{\theta}}_{i,q}^{\rm
T}\tilde{\bm{\theta}}_{i,q},$ (22)

where $\gamma_{i,q}$ is a positive design parameter. Then, the time derivative
of $\mathcal{V}_{i}$ gives

$\displaystyle\dot{\mathcal{V}}_{i}\leq$
$\displaystyle-\Delta_{i,1}e_{i,1}^{2}-\sum_{q=2}^{n}(\Delta_{i,q}-\frac{1}{2\rho_{i,q-1}^{2}})e_{i,q}^{2}$
$\displaystyle-\frac{1}{2}\sum_{q=1}^{n}\sigma_{i,q}\tilde{\bm{\theta}}_{i,q}^{\rm
T}\tilde{\bm{\theta}}_{i,q}+l_{i}^{[a]}\|\bm{\xi}_{i}\|^{2}+l_{i}^{[b]},$ (23)

where $l_{i}^{[a]}=\sum_{q=1}^{n}\frac{1}{2\rho_{i,q}^{2}}$ and

$l_{i}^{[b]}=\sum_{q=1}^{n}\frac{1}{2\rho_{i,q}^{2}}\epsilon_{i,q}^{\ast}+\sum_{q=1}^{n}\frac{1}{2}\sigma_{i,q}\|\bm{\theta}_{i,q}\|^{2}.$

Let

$\displaystyle\small c_{i,\gamma}=\min\bigg{\\{}$ $\displaystyle
2\Delta_{i,1},\big{[}2(\Delta_{i,q}-\frac{1}{2\rho_{i,q-1}^{2}})\big{]}_{q=2,\ldots,n,}$
$\displaystyle\big{[}\frac{\gamma_{i,q}}{\sigma_{i,q}}\big{]}_{q=1,\ldots,n.}\bigg{\\}}.$

Hence,

$\displaystyle\dot{\mathcal{V}}_{i,o}\leq-
c_{i,\gamma}\mathcal{V}_{i,o}+l_{i}^{[a]}\|\bm{\xi}_{i}\|^{2}+l_{i}^{[b]}.$
(24)

Therefore, by defining $\varpi_{i}=l_{i}^{[a]}\xi_{i}^{\ast}+l_{i}^{[b]}$ and
applying Lemma 1, it holds that

$\lim\nolimits_{t\rightarrow\infty}E_{p}(\mathcal{V}_{i,o})\leq\varpi_{i}/c_{i,\gamma}.$

Hence, the error signals $e_{i,q}$, and $\tilde{\bm{\theta}}_{i,q}$ for
$q=1,\ldots,n$ are all NSpS-P with a unique global solution according to Lemma
1. Accordingly,
$\lim\nolimits_{t\rightarrow\infty}E_{p}(|z_{i}-\eta_{i,1}|)<\varepsilon_{i,x}$
for a positive constant $\varepsilon_{i,x}$. From (17) it follows that
$\lim\nolimits_{t\rightarrow\infty}|\eta_{i}-z_{r}|\leq\varepsilon^{[e]}$.
Therefore,

$\lim\nolimits_{t\rightarrow\infty}E_{p}(|z_{i}-z_{r}|)<\varepsilon_{i,v}$

and the tracking consensus performance can ultimately be achieved. This
completes the proof. ∎

Figure 2: The communication graph.  Figure 3: Consensus trajectory tracking:
$\eta_{i,q}$ with respect to $z_{r}$ and $z_{r}^{\prime}$.  Figure 4:
Consensus trajectory tracking: $x_{i,1}$ with respect to $z_{r}$.

##  IV Simulation results

In this section, the simulation results are reported subsequent to applying
the proposed method in this paper to a network of $N=4$ uncertain nonlinear
second-order agents (i.e., $n=2$) defined in (4). The agents are initially
stationary positioned randomly in $[-2,2]$.
The unknown smooth heterogeneous nonlinear functions are defined as
$f_{i,1}(\bar{\bm{x}}_{i,1})=0.5ix_{i,1}\sin(x_{i,1})\cos(x_{i,1})$ and
$f_{i,2}(\bar{\bm{x}}_{i,2})=0.9ix_{i,1}\sin(x_{i,2})\cos(0.3x_{i,1})$. The
virtual leader signal is $z_{r}=\sin(0.5t)$ and therefore $c_{z}=1$. The
random disturbances coefficient functions with $m=1$ are
${g}_{i,1}(\bar{\bm{x}}_{i,1})=0.5x_{i,1}\sin(x_{i,1})$ and
${g}_{i,2}(\bar{\bm{x}}_{i,2})=0.5x_{i,1}\sin(x_{i,1})\cos(x_{i,2})$. In the
simulation results, the random disturbances $\xi_{i}$ are produced by

$\displaystyle\dot{\xi}_{i}(t)=\frac{1}{\mathcal{X}}[-\xi_{i}(t)+w_{i}(t)],$
(25)

in which $w_{i}(t)$ is generated by band-limited white noise block with noise
power $A=1$ and correlation time $t_{c}=0.1$. The number of seed for
$w_{i}(t)$ is $[23341]$, $[34243]$, $[23343]$, and $[34241]$, respectively for
$i=1,\ldots,4$.
The communication graph for the considered system is depicted in Fig. 2. From
the graph topology,
$\lambda_{\min}(\bm{\mathcal{L}}+\bm{\mathcal{B}})=0.1981$. With $c_{0}=6$, it
holds that $c_{0}\lambda_{\min}(\bm{\mathcal{L}}+\bm{\mathcal{B}})\geq 1$. A
feasible solution to (8) gives

$\displaystyle\bm{P}=\kern-3.00003pt\left[\begin{matrix}22.9454&3.1623\\\
3.1623&3.6280\end{matrix}\right],$

with $c_{1}=20$, $c_{2}=10$ and $c_{3}=3$. The control gain vector is chosen
as $\bm{K}=[18.9737\leavevmode\nobreak\ \leavevmode\nobreak\ \thinspace
21.7679]$. Furthermore, $\delta_{\alpha}=0.4529$ and $\delta_{\beta}=3.4$.
Additionally, from (12),

$\max\\{\mathscr{C}^{[\kappa]}\\}\approx(\mathscr{G}^{[\kappa]})\times
13.2\%.$

In the simulation results, we choose
$\mathscr{C}^{[\kappa]}=0.9\max\\{\mathscr{C}^{[\kappa]}\\}$ and the duration
of a particular ON mode interval for communication network is randomly
selected from $[0.5,2]$. The other design parameters are also selected as
$\mathcal{K}_{i,q}=15$, $\sigma_{i,q}=0.5$, and $\gamma_{i,q}=10$ for
$q=1,\ldots,n$.
The simulation is performed for a total time period of $20$ seconds with the
sampling time-step one milliseconds. The tracking consensus of virtual and
real systems are respectively depicted in Fig. 3 and Fig. 4, in which the OFF
mode intervals are highlighted in gray color. Otherwise, the system is
performing the tasks in ON mode intervals. From the simulation experiments, it
can be seen that the tracking performance is achieved while the communications
among neighboring agents are intermittent. The effects of random disturbance
are also compensated with the proposed approach in this paper.

##  V Conclusion

An intermittent communication mechanism for the tracking consensus of high-
order nonlinear MASs surrounded by random disturbances was addressed in this
paper. The resiliency level to the failure of physical devices or
unreliability of communication channels was analyzed by introducing a linear
auxiliary trajectory of the system. The closed-loop networked system signals
were proved to be NSpS-P. It has been justified that each agent follows the
trajectory of the corresponding local auxiliary virtual system practically in
probability.

## References

  * [1] R. Olfati-Saber, J. A. Fax, and R. M. Murray, “Consensus and cooperation in networked multi-agent systems,” Proceedings of the IEEE, vol. 95, no. 1, pp. 215–233, 2007.
  * [2] A. Azarbahram, A. Amini, and N. Pariz, “Event-triggered tracking formation of networked nonlinear intelligent transportation systems surrounded by random disturbances,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 21959–21970, 2022.
  * [3] M. Shahvali, A. Azarbahram, M.-B. Naghibi-Sistani, and J. Askari, “Bipartite consensus control for fractional-order nonlinear multi-agent systems: An output constraint approach,” Neurocomputing, vol. 397, pp. 212–223, 2020.
  * [4] A. Amini, A. Azarbahram, and M. Sojoodi, “$h_{\infty}$ consensus of nonlinear multi-agent systems using dynamic output feedback controller: an lmi approach,” Nonlinear Dynamics, vol. 85, pp. 1865–1886, 2016.
  * [5] M. Shahvali, A. Azarbahram, and N. Pariz, “Adaptive output consensus of nonlinear fractional-order multi-agent systems: a fractional-order backstepping approach,” International Journal of General Systems, vol. 52, no. 2, pp. 147–168, 2023.
  * [6] B. Wu, M. D. Lemmon, and H. Lin, “Formal methods for stability analysis of networked control systems with ieee 802.15.4 protocol,” IEEE Transactions on Control Systems Technology, vol. 26, no. 5, pp. 1635–1645, 2018.
  * [7] A. Azarbahram and A. Amini, “Secure event-triggered tracking consensus under denial of service: A resilient periodically intermittent approach,” IEEE Control Systems Letters, vol. 6, pp. 2786–2791, 2022.
  * [8] A. Amini, A. Asif, A. Mohammadi, and A. Azarbahram, “Sampled-data dynamic event-triggering control for networked systems subject to dos attacks,” IEEE Transactions on Network Science and Engineering, 2021.
  * [9] A. Amini, A. Azarbahram, A. Mohammadi, and A. Asif, “Resilient event-triggered average consensus under denial of service attack and uncertain network,” in 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT), pp. 291–296, IEEE, 2019.
  * [10] W. Zou, C. K. Ahn, and Z. Xiang, “Event-triggered consensus tracking control of stochastic nonlinear multiagent systems,” IEEE Systems Journal, vol. 13, no. 4, pp. 4051–4059, 2019.
  * [11] A. Azarbahram, N. Pariz, M.-B. Naghibi-Sistani, and R. Kardehi Moghaddam, “Event-triggered formation control of n-link networked stochastic robotic manipulators,” Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, vol. 236, no. 5, pp. 927–943, 2022.
  * [12] A. Azarbahram, N. Pariz, M.-B. Naghibi-Sistani, and R. Kardehi Moghaddam, “Platoon of uncertain unmanned surface vehicle teams subject to stochastic environmental loads,” International Journal of Adaptive Control and Signal Processing, vol. 36, no. 3, pp. 729–750, 2022.
  * [13] W. Zou, P. Shi, Z. Xiang, and Y. Shi, “Consensus tracking control of switched stochastic nonlinear multiagent systems via event-triggered strategy,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 3, pp. 1036–1045, 2020.
  * [14] Z. Wu, H. R. Karimi, and P. Shi, “Practical trajectory tracking of random lagrange systems,” Automatica, vol. 105, pp. 314–322, 2019.
  * [15] Y. Li, Y. Wu, and S. He, “Distributed consensus control for a group of autonomous marine vehicles with nonlinearity and external disturbances,” Neurocomputing, vol. 443, pp. 380–387, 2021.
  * [16] L. Yao, W. Zhang, and X.-J. Xie, “Stability analysis of random nonlinear systems with time-varying delay and its application,” Automatica, vol. 117, p. 108994, 2020.
  * [17] H. Zhang, R. Xi, Y. Wang, S. Sun, and J. Sun, “Event-triggered adaptive tracking control for random systems with coexisting parametric uncertainties and severe nonlinearities,” IEEE Transactions on Automatic Control, pp. 1–1, 2021.
  * [18] H. Wang, W. Li, and M. Tang, “Distributed output tracking of nonlinear multi-agent systems perturbed by second-order moment processes,” Neurocomputing, vol. 452, pp. 789–795, 2021.
  * [19] X. Bao, H. Wang, and W. Li, “Containment control for upper-triangular nonlinear multi-agent systems perturbed by second-order moment processes,” IEEE Access, vol. 9, pp. 21102–21111, 2021.
  * [20] C. Li, G. Feng, and X. Liao, “Stabilization of nonlinear systems via periodically intermittent control,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 54, no. 11, pp. 1019–1023, 2007.
  * [21] W. Guo and H. Xiao, “Distributed consensus of the nonlinear second-order multi-agent systems via mixed intermittent protocol,” Nonlinear Analysis: Hybrid Systems, vol. 30, pp. 189–198, 2018.
  * [22] H. Li, J. Zhang, L. Jing, and W. Ying, “Neural-network-based adaptive quasi-consensus of nonlinear multi-agent systems with communication constrains and switching topologies,” Nonlinear Analysis: Hybrid Systems, vol. 35, p. 100833, 2020.
  * [23] N. Xavier and B. Bandyopadhyay, “Practical sliding mode using state depended intermittent control,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 68, no. 1, pp. 341–345, 2020.
  * [24] Q. Wang and Y. He, “Time-triggered intermittent control of continuous systems,” International Journal of Robust and Nonlinear Control, vol. 31, no. 14, pp. 6867–6879, 2021.
  * [25] Y. Yang and Y. He, “Non-fragile observer-based robust control for uncertain systems via aperiodically intermittent control,” Information Sciences, vol. 573, pp. 239–261, 2021.
  * [26] B. Liu, M. Yang, T. Liu, and D. J. Hill, “Stabilization to exponential input-to-state stability via aperiodic intermittent control,” IEEE Transactions on Automatic Control, vol. 66, no. 6, pp. 2913–2919, 2020.
  * [27] F. L. Lewis, K. Liu, and A. Yesildirek, “Neural net robot controller with guaranteed tracking performance,” IEEE Trans. Neural Netw., vol. 6, no. 3, pp. 703–715, 1995.


# NICE: CVPR 2023 Challenge on Zero-shot Image Captioning

Taehoon Kim
Pyunghwan Ahn
Sangyun Kim
Sihaeng Lee
Mark Marsden
Alessandra Sala
Seung Hwan Kim
Bohyung Han
Kyoung Mu Lee
Honglak Lee
Kyounghoon Bae
Xiangyu Wu
Yi Gao
Hailiang Zhang
Yang Yang
Weili Guo
Jianfeng Lu
Youngtaek Oh
Jae Won Cho
Dong-Jin Kim
In So Kweon
Junmo Kim
Wooyoung Kang
Won Young Jhoo
Byungseok Roh
Jonghwan Mun
Solgil Oh
Kenan Emir Ak
Gwang-Gook Lee
Yan Xu
Mingwei Shen
Kyomin Hwang
Wonsik Shin
Kamin Lee
Wonhark Park
Dongkwan Lee
Nojun Kwak
Yujin Wang
Yimu Wang
Tiancheng Gu
Xingchang Lv
Mingmao Sun

###### Abstract

In this report, we introduce NICE (New frontiers for zero-shot Image
Captioning Evaluation) project111https://nice.lgresearch.ai/ and share the
results and outcomes of 2023 challenge. This project is designed to challenge
the computer vision community to develop robust image captioning models that
advance the state-of-the-art both in terms of accuracy and fairness. Through
the challenge, the image captioning models were tested using a new evaluation
dataset that includes a large variety of visual concepts from many domains.
There was no specific training data provided for the challenge, and therefore
the challenge entries were required to adapt to new types of image
descriptions that had not been seen during training. This report includes
information on the newly proposed NICE dataset, evaluation methods, challenge
results, and technical details of top-ranking entries. We expect that the
outcomes of the challenge will contribute to the improvement of AI models on
various vision-language tasks.

††Affiliation and role of each author are listed in the appendix.

(a) View from sandy beach of picturesque sunset horizon over sea surf lapping
on shore

(b) 3D concept of online banking on mobile phone showing services currencies
and graphs

(c) St. Sebastian’s Church, Ramsau, near Berchtesgaden, Bavaria, Germany

(d) A group of young friends having a barbecue outdoors next to a tent

Figure 1: Example images and corresponding captions in NICE dataset.

##  1 Introduction

Zero-shot image captioning is a task that general purpose vision-language
models must perform well, as it requires both the visual understanding of the
scene and the ability to describe it in natural language. Automatic generation
of image descriptions makes a variety of applications possible, such as
improved image search with the help of natural language, better detection of
inappropriate contents on the web, and explanation of visual contents to
visually impaired people. In most real-world scenarios, images from unseen
environments are frequently given to perform these actions, which makes zero-
shot image captioning essential.

In the earlier days, image captioning models were trained on curated datasets
which consist of training and testing data from the same domain or categories
[8, 30]. These models inevitably have limited capability to recognize a whole
new concept and describe it. For better utility of image captioning models,
several datasets [1] were suggested in order to test the model with images
from unseen categories. This evaluation process has put the existing models on
harder testing conditions, leading to development of image captioning models
that can understand general scenes.

Although there are a few benchmarks which target for zero-shot image
captioning evaluation, each of them lacks in one or more requirements, which
includes large size of the dataset, variety of categories, and quality of
language descriptions. The size of the dataset is essential to test the models
with adequate number of images to guarantee reliability, and variety of
categories is required so that the tested models are not fitted to perform
well on a few certain concepts. Also, as the prediction shall be usually
compared with the ground truth captions using text comparison metrics, the
quality of the ground truth captions must be guaranteed to be accurate and of
high quality.

Through NICE 2023 challenge, we made a newly curated NICE dataset, which
consists of 26k images and corresponding high-quality captions, publicly
available. Furthermore, we did not provide a specific training data for the
challenge, which forces the models to be trained to generalize well, thus
achieving zero-shot capability. Even though this challenge was a newly
organized one, 51 and 31 teams competed in the validation and test phase,
respectively, and the top-scoring entries showed very small differences in the
final score. The following sections of the report includes the information on
the challenge, evaluation methods, results, and proposed approaches by top-
scoring entries.

##  2 Challenge

In this challenge, a newly curated dataset was made publicly available, named
NICE dataset, and then the challenge was organized to evaluate image
captioning capability of AI models. This section introduces the dataset,
evaluation method, challenge phases, and results.

Rank Username Bleu_1 Bleu_2 Bleu_3 Bleu_4 ROUGE_L CIDEr METEOR SPICE 1 jinx
56.0839 (3) 46.5881 (2) 40.0586 (2) 35.1730 (2) 55.5685 (2) 325.7216 (1)
29.1455 (2) 44.4351 (2) 2 stack-top 58.0129 (1) 47.8769 (1) 40.9018 (1)
35.7796 (1) 56.3780 (1) 324.9277 (2) 30.0329 (1) 45.5456 (1) 3 PEJI 56.4908
(2) 46.5371 (3) 39.6859 (3) 34.5996 (3) 54.9832 (3) 316.2290 (3) 28.9407 (3)
43.8281 (3) 4 calisolo 55.7849 (4) 45.4753 (4) 38.4468 (4) 33.1970 (4) 52.9229
(4) 287.6926 (4) 27.9402 (4) 41.3584 (4) 5 img_capt 53.6418 (7) 42.8248 (7)
35.6963 (6) 30.5766 (5) 51.6134 (7) 278.1607 (5) 27.0087 (7) 40.9747 (6) 6
kyominhwang 54.1975 (6) 43.0211 (6) 35.4753 (7) 30.0185 (7) 52.3251 (5)
274.6941 (6) 27.3464 (5) 41.0430 (5) 7 Mtop 54.4547 (5) 43.3951 (5) 35.8492
(5) 30.2877 (6) 52.1572 (6) 270.5980 (7) 27.2156 (6) 40.7732 (7) 8 Yongsik
51.3670 (8) 40.2552 (8) 32.7835 (8) 27.4779 (8) 50.5170 (8) 255.9013 (8)
26.1637 (8) 39.4600 (8) 9 TXHmercury 50.7199 (10) 39.0901 (11) 31.1348 (12)
25.3470 (13) 50.0612 (9) 239.0106 (9) 25.3394 (9) 38.5370 (9) 10 zero_score
50.1327 (12) 39.4831 (9) 32.4496 (9) 27.4089 (9) 48.0671 (12) 238.3091 (10)
24.8445 (12) 37.0757 (10) 11 sungbin.son 48.2634 (14) 37.8367 (13) 31.0821
(13) 26.3040 (10) 46.5434 (14) 229.5613 (11) 23.8719 (15) 35.3885 (14) 12
ss501 50.4477 (11) 38.8727 (12) 31.1372 (11) 25.6216 (12) 48.1135 (11)
228.8920 (12) 24.7030 (13) 36.5725 (12) 13 danielchoi 50.7946 (9) 39.1847 (10)
31.4011 (10) 25.8194 (11) 48.1828 (10) 226.1749 (13) 24.8638 (11) 36.8950 (11)
14 Hi1988 48.9767 (13) 37.1830 (14) 29.2179 (14) 23.4479 (14) 47.8742 (13)
217.7578 (14) 24.0982 (14) 36.4749 (13) 15 BraveGirls 46.4913 (15) 34.2065
(15) 25.9219 (15) 19.9941 (15) 46.1757 (15) 180.6635 (15) 24.9094 (10) 33.2203
(15) 16 mobled37 39.8284 (16) 27.7706 (16) 20.0732 (16) 14.7538 (16) 39.5366
(16) 134.2035 (16) 18.9348 (16) 27.7624 (16) 17 rjsdn 35.2991 (17) 23.4908
(17) 15.9875 (17) 10.9201 (17) 37.0862 (17) 113.8025 (17) 16.9700 (17) 25.3106
(17) 18 snow0 31.7156 (20) 19.2009 (19) 12.2571 (18) 7.9731 (18) 31.8328 (18)
92.8288 (18) 14.5823 (19) 22.6518 (18) 19 soeun 27.3832 (25) 16.1707 (23)
10.1247 (22) 6.4948 (21) 29.3049 (21) 78.9487 (19) 12.9487 (22) 20.3599 (21)
20 do0ori 32.1687 (19) 18.3737 (20) 11.0992 (20) 6.8483 (20) 30.1717 (20)
77.8300 (20) 13.9382 (20) 20.6390 (20) 21 zhengmaozong 33.7145 (18) 19.5054
(18) 11.7051 (19) 7.0696 (19) 30.5224 (19) 75.3119 (21) 15.0924 (18) 20.8646
(19) 22 yongsu23 26.9424 (27) 15.3106 (25) 9.3395 (24) 5.8594 (23) 28.2218
(22) 73.1277 (22) 12.2745 (24) 19.4324 (22) 23 challang 30.8059 (21) 17.0075
(22) 9.8781 (23) 5.8459 (24) 27.7463 (23) 65.9372 (23) 13.0716 (21) 18.9231
(23) 24 k1101jh 24.3603 (29) 13.4100 (28) 8.0403 (28) 5.0202 (27) 26.3008 (27)
65.7418 (24) 11.2266 (28) 17.6973 (24) 25 gamy 28.6025 (23) 15.3838 (24)
9.0163 (25) 5.4843 (25) 26.9439 (25) 64.0570 (25) 12.0974 (25) 17.6462 (25) 26
alsaco 27.0615 (26) 14.4850 (27) 8.4428 (26) 5.1177 (26) 26.6241 (26) 64.0352
(26) 11.2880 (27) 17.0905 (26) 27 Yebin46 30.3189 (22) 17.4146 (21) 10.5292
(21) 6.3576 (22) 27.6034 (24) 59.7099 (27) 12.5039 (23) 16.5475 (27) 28
Rainy21 27.4143 (24) 14.6171 (26) 8.3537 (27) 4.9207 (28) 23.6640 (28) 49.8930
(28) 11.7443 (26) 15.9822 (28) 29 Runbor 25.0107 (28) 11.8485 (29) 6.0223 (29)
3.1465 (29) 21.7858 (29) 32.5173 (29) 9.5540 (29) 11.5799 (29) 30 Merander
22.4721 (30) 9.8276 (30) 4.7389 (30) 2.3512 (30) 20.1784 (30) 26.2615 (30)
8.3881 (30) 10.4668 (30) 31 kathmanducow 15.9741 (31) 6.4685 (31) 2.6371 (31)
1.1054 (31) 14.4777 (31) 10.2777 (31) 5.7737 (31) 4.1072 (31)

Table 1: NICE 2023 Challenge results are presented. The numbers in the
parentheses are individual ranking based on the metric of each column. The
final rankings are determined by CIDEr score.

###  2.1 Dataset

The images and corresponding captions used in this challenge were provided by
Shutterstock. This new large-scale evaluation dataset consists of
approximately 26k high quality images, along with associated metadata. It
includes a wide breadth of concepts from various categories. With this
dataset, the participants of the challenge were expected to take a
longitudinal evaluation across a variety of metrics to comparatively assess
the performances of different zero-shot image captioning models. Some example
images in NICE dataset are shown in Figure 1. We did not provide specific set
of training data for the challenge, as we aim for zero-shot image captioning,
in which AI models can perform image captioning on the new data that was never
seen in the training stage.

###  2.2 Evaluation metrics

There were several evaluation metrics used for this challenge. The evaluation
metric with the first priority was CIDEr [33] score, which is short for
Consensus-based Image Description Evaluation. CIDEr score calculates the
similarity score between two sentences by weighting each n-gram with the TF-
IDF (Term Frequency Inverse Document Frequency) values. This scheme sets the
weight of an n-gram higher if it appears in the sentence but not frequently in
the whole set of captions, thus estimating the importance of that n-gram
higher. CIDEr score is currently one of the most popular metrics for text
comparison, and we chose it as the top priority evaluation metric for the
challenge. In case of a tie, we used SPICE [2], METEOR [4], ROUGE [18], Bleu
[24] in the order of priority.

###  2.3 Challenge phases

Validation phase: From February to April 2023, validation server was provided
so that the participants could upload their prediction results and the server
would calculate the scores by comparing them to the ground truth captions. In
this phase, participants were able to access the validation data, which
includes the image and the ground truth captions. As this is the first
challenge on the task, we provided ground truth captions in order to let
participants get acquainted with the data format and build strategies for the
preparation of the final challenge entry.

Figure 2: Overall Architecture of team ”no”. Their solution consists of four
main stages, which includes Pre-training, Coarse-tuning, Fine-tuning and
Model-ensemble. The training data for the first three stages are all collected
from the large-scale Laion-5B dataset.

Test phase: In April 2023, test server was open and allowed participants to
upload prediction results up to 5 times during the phase. The score was
calculated and the entry with the best CIDEr score was uploaded on the open
leaderboard222https://codalab.lisn.upsaclay.fr/competitions/10248#results. In
the test phase, the ground truth captions were not accessible and only the
test scores were shown to the participants.

###  2.4 Challenge results

The results of the challenge is presented in Table 1. There were 31 teams that
participated in the challenge, and the final decision of the ranking was based
on CIDEr score. The top-ranking entry scored 325.72, and the following entries
scored 324.93, 316.23, and so on. Furthermore, the first ranking entry did not
score the best in the other metrics, which shows that each entry had strong
and weak points.

##  3 Proposed Approaches

###  3.1 1st rank : no

For the model level, they used OFA [34] as their base model. As shown in
Figure 2, the overall architecture consists of three parts, namely Pre-
training, Coarse-tuning and Fine-tuning. Pre-training stage aims to align a
wide range of extensively visual concepts and store sufficient vision-language
knowledge through contrastive learning, image captioning pre-train objectives.
Coarse-tuning stage utilizes a small-scale external dataset similar to the
competition domain, which can learn a large variety of novel concepts. Fine-
tuning stage further compresses the dataset in the last stage and adds it to
the competition validation dataset.

For the data level, they collected external training data from LAION-5B [28],
a large-scale CLIP-filtered image-text dataset. In the Pre-training stage, 1M
image-text pairs are collected from the specific url(thumbx.shutterstock.com,
editorial01.shutterstock.com, etc.). In the Coarse-tuning and Fine-tuning
stage, they used all the competition images to retrieve external data from
LAION-5B through Clip-retrieval333https://github.com/rom1504/clip-retrieval
library based on similarity. For each image query, they retrieved top-30 and
top-10 image-text pairs, retain 120k samples for Coarse-tuning and 12k for
Fine-tuning with the 5k validation dataset.

In addition, they introduced contrastive learning [17], similarity-bucket,
retrieval-augmented [37] strategies to the model. Contrastive learning aims to
learn better single-modal representations before fusion and align visual and
text concepts in Pre-training stage. Similarity-bucket strategy provides
different similarity-prompts to the vision-language model in the all stages,
which can control the model to generate the most matched and high-quality
caption for a given image. Retrieval-augmented strategy provides a mini
knowledge-base for each image-text pair during the training part. The model
can not only extract visual features such as objects, attributes, and
relationships of the image, but also explicitly align the information of the
image with the knowledge in the knowledge-base.

###  3.2 2nd rank : Retriever

Figure 3: Overview of Retriever framework. It improves image captioning under
zero-shot settings in two stages: (a) retrieval-based dataset discovery for
training and (b) retrieval-based fusion conditioned on examples similar to
input queries.

Captions in NICE dataset often contain new concepts like camera angle
descriptions and proper nouns like place names, which are difficult to predict
under zero-shot settings. Motivated from retrieval-augmented models [20, 5],
the Retriever framework aims to complement such limited data condition by
efficiently utilizing external knowledge in model training and inference. As
shown in 3, it enhances typical captioning model in two stages. Firstly, to
construct a dataset for training, an explicit retrieval module samples a set
of image-text pairs from the memory, which can closely mimic the target
distribution of captions. Secondly, the knowledge associated with the input
query is explicitly combined into the model via retrieval-based fusion. These
two processes are introduced as below, while the technical report [23]
includes more details.

Dataset Discovery The base captioning model BLIP-2 [16] exhibits poor
performance when evaluated on NICE dataset. This necessitates to discover data
samples for training that are relevant to the target distribution. To that
end, given a query image, a retrieval module is employed to retrieve a set of
related examples from a dataset saved in the external memory via a $k$-nearest
neighbor (kNN) search. This module is applied to the NICE dataset of size $N$,
yielding total $kN$ images with corresponding captions from the external
memory. After a deduplication step, the unique image-text pairs are utilized
for further finetuning the BLIP-2.

Retrieval-based fusion Given a query image either during training or
inference, a set of value embeddings is produced by encoding the caption of
the $k$ retrieved image-text pairs. These embeddings can provide rich
contextual information complementary to the knowledge in the original model.
They are aggregated and then concatenated with the query feature. After the
fusion, the new feature is further passed to the remaining captioning
pipeline; Q-Former followed by the LLM decoder in BLIP-2 [16]. This results in
more accurate captions, and the entire Retriever framework achieves CIDEr
324.9 on NICE test split.

###  3.3 3rd rank : Kakaobrain-MMU

Figure 4:  The architecture of captioning model. First, the control signal is
computed by bucketing CLIP similarity during the training phase, and set to a
constant one during the test phase. Then, it is concatenated to the image
embeddings and is fed into a cross-attention-based caption decoder model as
key-value features. This figure is inspired and slightly modified from [14].

The main components of their approach are three-fold: 1) NoC framework [14],
2) three-stage training pipeline, and 3) consensus-based model ensemble.
Detailed explanations for each method will be described in the following
sections.

Training Algorithm Since they trained their models using large-scale web-
crawled image-text paired datasets that contain inherent noises, _i.e_.,
misaligned pairs, they utilized their prior work, Noise-aware Captioning (NoC)
framework [14], as a primary training algorithm, illustrated in Figure 4. In a
nutshell, NoC incorporates conditional modeling into a captioning model to
model alignment levels of the input image-text pairs. During the training
phase, NoC utilizes the CLIP similarity between an image-text pair as an
additional input of a control signal indicating the alignment level. Then, at
the inference phase, they feed a control signal of a desired alignment level
into the captioning model to produce semantically accurate captions that are
closely aligned with input images.

Figure 5: The overview of the method proposed by team Otsuka AI.

Three-stage Training Given web-crawled image-text data and NICE validation
set, they trained their model with a three-stage pipeline: 1) pre-training on
large-scale data, 2) fine-tuning on retrieved data, and 3) fine-tuning on NICE
validation set. The model trained in each stage is used to initialize the
model in the next stage. At the first stage, they pre-trained captioning
models with the following datasets444They trained a captioning model for each
dataset.: CC15M [29, 7], COYO-700M [6], COYO-100M (a subset of COYO-700M
having CLIP similarity higher than 0.3), LAION-45M [28] (filtered from LAION-
Aesthetics-en V1 52M), and 5) LAION-120M [28] (filtered from LAION-Aesthetics
Predictor V2 600M). The filtering strategy for LAION-45M and -120M was based
on aspect ratio, image size, CLIP similarity, text length, etc. Then they
fine-tuned the pre-trained model using retrieved data. Inspired by [19], they
retrieved the most NICE-relevant image-text pairs from the pre-training data.
For each query sample (captions in NICE validation split), they retrieved
1,000 T2T and 1,000 T2I samples from the union of COYO and LAION datasets
using CLIP ViT-B/32 [26] and FAISS [13]. This makes the captioning models
likely to generate captions more similar to captions in NICE validation set.
Finally, they further fine-tuned their captioning models from stage 2 with the
NICE validation set to more tightly align the caption style with the ones in
the NICE challenge.

Consensus-based Ranking for Ensemble Given $N$ captioning models, they
generated a set of captions $\mathcal{C}_{N}=\\{c_{1},c_{2},...,c_{N}\\}$ for
an input image where individual models generate a caption with a beam search
strategy. Then, for a caption $c_{n}$ from $n^{\text{th}}$ captioning model,
they computed a consensus score [21, 9] $s_{n}$ that is defined as an averaged
similarity to all the other captions
($c_{m}\in{\mathcal{C}_{N}}\backslash\\{c_{n}\\}$) as follows:

$s_{n}=\frac{1}{|\mathcal{C}_{N}|-1}\sum_{c_{m}\in{\mathcal{C}_{N}}\backslash\\{c_{n}\\}}\text{sim}(c_{n},c_{m}),$
(1)

where $\text{sim}(c_{n},c_{m})$ is the CIDEr score between two captions
$c_{n}$ and $c_{m}$. Finally, the caption of the highest consensus score is
chosen as their final output for the input image.

###  3.4 4th rank : Otsuka AI

In the NICE dataset, named entities such as geographical locations (e.g.,
Germany, Italy) pose a challenging task for inferring through images. For
instance, deducing that the habitat of highland cattle is Germany based on
pictures requires substantial foundational knowledge. In this work, problem is
approached from the perspective of generating dialogue specialized for the
NICE dataset, resembling persona conversations. The distinguished image
captioning model OFA [34] is fine-tuned to be specialized for NICE.

The term ”Levels” represents the entirety of the methodology, where soft
prompts [25] are effectively utilized to categorize the strength of hints
present in example captions accessible by the model. Based on the output layer
values of the OFA encoder, the captions of the four most similar images to the
input image are retrieved from the pool of 5,000 validation images. Each
caption serves as a powerful hint for generating predictions. To replicate the
linguistic style of the NICE dataset, the four captions are employed akin to
few-shot prompts.

Figure 6: The proposed architecture from CLAS team.

As a method to specify the level of hint in the prompt and facilitate the
model’s judgment, soft prompts are introduced. The values of cosine similarity
between the input image and the four example images were normalized to provide
their corresponding levels. These soft prompts are structured into four levels
based on cosine similarity and three levels based on public id difference.
Soft prompts form part of the broader utilization of various soft prompt
techniques, as seen in visual prompting [12], symbol tuning [36], and others.

###  3.5 5th rank : CLAS

Their approach revolves around the BLIP-2 architecture [16], that combines
several strategies to improve performance. The BLIP-2 architecture effectively
utilizes image features by incorporating the Querying Transformer (Q-Former)
along with state-of-the-art language models, such as OPT [39]. Q-Former is
pretrained in two stages: the representation and generative learning stages,
allowing the extraction of a fixed number of output features from the image
encoder, regardless of the input image resolution. Their image encoder is
based on ViT-G/14 [10], while the language model is based on OPT-2.7b.

For fine-tuning, they used the validation set and the ”a picture of” prompt,
which helps the model converge faster and generate better captions. The
network is fine-tuned on the validation set for 200 epochs. They employed FP16
mixed precision training and the Low-Rank Adaptation (LoRA) technique [11] to
allow for adaptation without updating all model parameters. After fine-tuning
the network with cross-entropy loss, they utilized CIDEr optimization based on
self-critical sequence training [27]. This approach leverages the output of
its own test-time inference algorithm to normalize the rewards it experiences.

Considering the high computational requirements of their model, conventional
ensemble methods that average or rank sequences were not practical. Instead,
they adopted an ensemble of 10 models, each trained with varying learning
rates and epochs, using both cross-entropy and CIDEr optimization methods. To
rank the models, they employed a model ranking system based on the confidence
score of each generated caption. This confidence score is calculated by
considering the probability of each word in the caption obtained through
greedy sampling. The caption with the highest probability is selected for the
output.

###  3.6 6th rank : MKC

Figure 7: Proposed Adapter structure

They finetuned a large Vision-Language Pre-trained model, BLIP-2 ViT-g OPT
6.7B to cope with various concepts in the NICE dataset. To avoid catastrophic
forgetting and reduce the number of parameters to train, they added adapters
to BLIP-2 image encoder and only update the attached adapter, layernorm and
Q-former using EMA(Exponential Moving Average) method. For finetuning, they
divided the training process into two stages: 1) finetuning with the CC3M
dataset and 2) finetuning with the NICE validation and the CC3M dataset mixed
together.

Figure 8: Instructed image caption generation.

They plugged an adapter [15] with zero convolution [38] at each layer in ViT
[10]. Figure 7 shows the structure of the adapter. The weights in zero
convolutions would progressively grow from zero to optimal parameters, which
helped the model slowly adapt to the CC3M dataset. They also optimized the
layernorm [3] inspired by the context of domain adaptation [31, 22]. Due to
the various domains in NICE dataset, the method in domain adaptation could
positively affect the performance. Lastly, Q-former in the original BLIP-2 was
also optimized. The main role of the Q-former is to reduce the modality gap
between image and natural language. Since the image feature extractor was
decided to be trained, it was reasonable to optimize the Q-former as well.
They also applied the EMA method to prevent catastrophic forgetting. They
built a teacher model with parameter $\theta_{T}$, and regularized a student
model, $\theta_{S}$, with consistency loss [32]. Consistency
loss($Loss_{con}(\theta)$) between a teacher model($f(x;\theta_{T})$) and a
student model($f(x;\theta_{S})$) was defined as follows:

$\displaystyle\mathcal{L}_{con}(\theta)=\beta
KL(f(x;\theta_{T}),f(x;\theta_{S}))$ (2)

where $\beta$ is a hyperparameter for controlling the strength of consistency
between a teacher and a student model, $x$ denotes an input image, and $f$
denotes the model architecture. The overall loss function for the whole
finetuning process was defined as follows:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{ce}+\mathcal{L}_{con}$ (3)

where $\mathcal{L}_{ce}$ is a standard cross-entropy loss. At the first
finetuning stage, only the CC3M dataset was used. Then in the second stage,
the mixture of NICE validation dataset and the CC3M dataset was used.

In this work, they presented the method of light-weighted finetuning in the
field of Vision-Language, especially aimed at the image captioning task.
Choosing BLIP-2 as the baseline model, adapters with zero convolution and
optimization of layer normalization were applied to perform efficient domain-
robust finetuning. Their model achieved up to 274.69 CIDEr score on the test
case, demonstrating the effectiveness of the proposed finetuning methodology.

###  3.7 7th rank : Mtop

Team Mtop’s solution for the NICE 2023 challenge revolves around data
augmentation and finetuning of the BEiT3 [35] model with the augmented data.
Due to the limitations in computational resources, they employed a minimal-
effort approach. Firstly, they generated additional caption data by learning
different styles. Next, they performed finetuning of the BEiT3 model using
this augmented data without resorting to external datasets. To prevent
catastrophic forgetting, a novel caption correction method was adopted. Their
submission secured the 7th position in the challenge, achieving a notable
score of 270 points.

Details of Approach BEiT3, pretrained on image-text pairs from datasets like
CC12M, CC3M, SBU, COCO, and VG, served as the foundation for their approach.
However, they recognized the significant differences in caption styles between
these datasets and NICE, resulting in distributional discrepancies. To address
this, they utilized BLIP2 to measure the similarity with NICE and selected
image-text pairs that closely resembled the NICE dataset. These pairs enabled
zero-shot learning during finetuning. One of the key challenges encountered
during finetuning was catastrophic forgetting and being misled by the training
data. To mitigate this issue, they employed instructed captions generated by
large language models to guide the caption generation process. This approach
led to improved performance and better results.

Key Contributions Their submission not only advances Image Captioning and
Zero-shot Learning but also introduces a novel caption correction method to
alleviate the impact of catastrophic forgetting. By finetuning the BEiT3 model
with carefully selected augmented image-text pairs, they demonstrated the
importance of considering target styles of captions for this task.
Experimental results highlight the significance of caption correction and
finetuning with relevant styles in achieving superior performance.

##  4 Conclusion

Through NICE challenge 2023, a new dataset for zero-shot image captioning
evaluation was proposed, and various approaches were attempted to
appropriately adjust the AI models that had been trained on other datasets to
the new evaluation dataset. We hope to continue this line of research and
contribute to more challenging tasks to be performed by vision-language
models. The proposed methods presented a wide range of insight on adapting the
pretrained model to a specific domain of data, without sufficient training
data from the target domain. This research field is expected to dive deeper
into the real-world vision-language problems where the input visual data must
be described in language of various styles.

## References

  * [1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.  Nocaps: Novel object captioning at scale.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948–8957, 2019.
  * [2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer, 2016.
  * [3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [4] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
  * [5] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.  Improving language models by retrieving from trillions of tokens.  In International conference on machine learning, pages 2206–2240. PMLR, 2022.
  * [6] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  COYO-700M: Image-Text Pair Dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * [7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts.  In CVPR, 2021.
  * [8] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [9] Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick.  Exploring nearest neighbor approaches for image captioning.  arXiv preprint arXiv:1505.04467, 2015.
  * [10] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023.
  * [11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [12] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In European Conference on Computer Vision, pages 709–727. Springer, 2022.
  * [13] Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with gpus.  IEEE Transactions on Big Data, 7(3):535–547, 2019.
  * [14] Wooyoung Kang, Jonghwan Mun, Sungjun Lee, and Byungseok Roh.  Noise-aware learning from web-crawled image-text data for image captioning.  arXiv preprint arXiv:2212.13563, 2022.
  * [15] Taehoon Kim, Mark Marsden, Pyunghwan Ahn, Sangyun Kim, Sihaeng Lee, Alessandra Sala, and Seung Hwan Kim.  Large-scale bidirectional training for zero-shot image captioning.  arXiv preprint arXiv:2211.06774, 2022.
  * [16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [17] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [18] Chin-Yew Lin.  ROUGE: A package for automatic evaluation of summaries.  In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  * [19] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li.  Learning customized visual models with retrieval-augmented knowledge.  arXiv preprint arXiv:2301.07094, 2023.
  * [20] Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel.  Retrieval augmented classification for long-tail visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6959–6969, 2022.
  * [21] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille.  Deep captioning with multimodal recurrent neural networks (m-rnn).  ICLR, 2015.
  * [22] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan.  Towards stable test-time adaptation in dynamic wild world.  arXiv preprint arXiv:2302.12400, 2023.
  * [23] Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, and Junmo Kim.  Technical report of nice challenge at cvpr 2023: Retrieval-based data discovery and fusion for zero-shot image captioning.  https://ytaek-oh.github.io/assets/pdf/nice_report.pdf.
  * [24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
  * [25] Guanghui Qin and Jason Eisner.  Learning how to ask: Querying lms with mixtures of soft prompts.  arXiv preprint arXiv:2104.06599, 2021.
  * [26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning Transferable Visual Models From Natural Language Supervision.  In ICML, 2021.
  * [27] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel.  Self-critical sequence training for image captioning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024, 2017.
  * [28] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [29] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In ACL, 2018.
  * [30] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
  * [31] Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi.  Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization.  arXiv preprint arXiv:2303.01904, 2023.
  * [32] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  Advances in neural information processing systems, 30, 2017.
  * [33] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
  * [34] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  In International Conference on Machine Learning, pages 23318–23340. PMLR, 2022.
  * [35] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  arXiv preprint arXiv:2208.10442, 2022.
  * [36] Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al.  Symbol tuning improves in-context learning in language models.  arXiv preprint arXiv:2305.08298, 2023.
  * [37] Zhuolin Yang, Wei Ping, Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, et al.  Re-vilm: Retrieval-augmented visual language model for zero and few-shot image captioning.  arXiv preprint arXiv:2302.04858, 2023.
  * [38] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [39] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.

##  Appendix A Organizers

Organizing committee
Seung Hwan Kim,
LG AI Research,
<EMAIL_ADDRESS>
Kyoung Mu Lee,
Seoul National University,
<EMAIL_ADDRESS>
Bohyung Han,
Seoul National University,
<EMAIL_ADDRESS>
Alessandra Sala,
Shutterstock,
<EMAIL_ADDRESS>

Technical committee
Sihaeng Lee,
LG AI Research,
<EMAIL_ADDRESS>
Taehoon Kim,
LG AI Research,
<EMAIL_ADDRESS>
Pyunghwan Ahn,
LG AI Research,
<EMAIL_ADDRESS>
Sangyun Kim,
LG AI Research,
<EMAIL_ADDRESS>
Mark Marsden,
Shutterstock,
<EMAIL_ADDRESS>

##  Appendix B Challenge participants

Team no
Xiangyu Wu,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>
Yi Gao,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>
Hailiang Zhang,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>
YangYang,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>
Weili Guo,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>
Jianfeng Lu,
Nanjing University of Science and Technology,
<EMAIL_ADDRESS>

Team Retriever
Youngtaek Oh,
Korea Advanced Institute of Science and Technology,
<EMAIL_ADDRESS>
Jae Won Cho,
Korea Advanced Institute of Science and Technology,
<EMAIL_ADDRESS>
Dong-Jin Kim,
Hanyang University,
<EMAIL_ADDRESS>
In So Kweon,
Korea Advanced Institute of Science and Technology,
<EMAIL_ADDRESS>
Junmo Kim,
Korea Advanced Institute of Science and Technology,
<EMAIL_ADDRESS>

Team Kakaobrain-MMU
Wooyoung Kang,
Kakao Brain,
<EMAIL_ADDRESS>
Won Young Jhoo,
Kakao Brain,
<EMAIL_ADDRESS>
Byungseok Roh,
Kakao Brain,
<EMAIL_ADDRESS>
Jonghwan Mun,
Kakao Brain,
<EMAIL_ADDRESS>

Team Otsuka AI
Solgil Oh,
Wooribank,
<EMAIL_ADDRESS>

Team CLAS
Kenan Emir Ak,
Amazon Inc.,
<EMAIL_ADDRESS>
Gwang-Gook Lee,
Amazon Inc.,
<EMAIL_ADDRESS>
Yan Xu,
Amazon Inc.,
<EMAIL_ADDRESS>
Mingwei Shen,
Amazon Inc.,
<EMAIL_ADDRESS>

Team MKC
Kyomin Hwang,
Seoul National University,
<EMAIL_ADDRESS>
Wonsik Shin,
Seoul National University,
<EMAIL_ADDRESS>
Kamin Lee,
Seoul National University,
<EMAIL_ADDRESS>
Wonhark Park,
Seoul National University,
<EMAIL_ADDRESS>
Dongkwan Lee,
Seoul National University,
<EMAIL_ADDRESS>
Nojun Kwak,
Seoul National University,
<EMAIL_ADDRESS>

Team Mtop
Yujin Wang,
Tsinghua University,
<EMAIL_ADDRESS>
Yimu Wang,
University of Waterloo,
<EMAIL_ADDRESS>
Tiancheng Gu,
University of Sydney,
<EMAIL_ADDRESS>
Xingchang Lv,
Illinois Institute of Technology,
<EMAIL_ADDRESS>
Mingmao Sun,
University of California, Berkeley,
<EMAIL_ADDRESS>