solutionSolutionsolutionfile solutionfile[CapacityIRS]

# Ergodic Capacity of IRS-Assisted MIMO Systems with Correlation and Practical
Phase-Shift Modeling

Anastasios Papazafeiropoulos A. Papazafeiropoulos is with the Communications
and Intelligent Systems Research Group, University of Hertfordshire, Hatfield
AL10 9AB, U. K., and with the SnT at the University of Luxembourg, Luxembourg.
This work was supported by the University of Hertfordshire’s 5-year Vice
Chancellor’s Research Fellowship and by the National Research Fund,
Luxembourg, under the project RISOTTI. E-mail<EMAIL_ADDRESS>
###### Abstract

We focus on the maximization of the exact ergodic capacity (EC) of a point-to-
point multiple-input multiple-output (MIMO) system assisted by an intelligent
reflecting surface (IRS). In addition, we account for the effects of
correlated Rayleigh fading and the intertwinement between the amplitude and
the phase shift of the reflecting coefficient of each IRS element, which are
usually both neglected despite their presence in practice. Random matrix
theory tools allow to derive the probability density function (PDF) of the
cascaded channel in closed form, and subsequently, the EC, which depend only
on the large-scale statistics and the phase shifts. Notably, we optimize the
EC with respect to the phase shifts with low overhead, i.e., once per several
coherence intervals instead of the burden of frequent necessary optimization
required by expressions being dependent on instantaneous channel information.
Monte-Carlo (MC) simulations verify the analytical results and demonstrate the
insightful interplay among the key parameters and their impact on the EC.

###### Index Terms:

Intelligent reflecting surface (IRS), ergodic capacity, MIMO communication,
correlated Rayleigh fading, beyond 5G networks.

##  I Introduction

Intelligent reflecting surface (IRS) has emerged as a promising green and
cost-effective solution towards the sustainable growth of next-generation
wireless networks [1]. An IRS is a planar meta-surface, which consists of a
large number of nearly passive elements that are managed by a smart
controller. The role of each element is the induction of an independent phase
shift and/or amplitude attenuation to the impinging electromagnetic waves. In
[2], it was shown that the amplitude and phase response are intertwined, while
this was neglected by most works. The proper design of the reflecting
coefficients enhances the performance by constructive and destructive addition
of the signals reflected by IRS to increase the desired signal power and
mitigate the co-channel interference, respectively.

The study of IRS performance gains has attracted significant attention under
various application scenarios and objectives [3, 4, 5, 6, 7]. For instance, in
[3], a minimization of the total transmit power took place by optimizing the
transmit and reflecting beamforming (RB). Also, in [5], we suggested the use
of multiple distributed IRSs and compared two practical scenarios, namely, a
large number of finite size IRSs and a finite number of large IRSs to show
which implementation scenario is more advantageous. Moreover, in [6], we
studied the impact of hardware impairments at both the transceiver and IRS
sides in a general IRS-assisted multi-user multiple-input single-output (MISO)
system with correlated Rayleigh fading, which is usually neglected in relevant
works (e.g., see [1, 3]), although it should be taken into account [8].
Furthermore, in [7], the assumptions required to integrate massive multiple-
input multiple-output (MIMO) systems with IRS were studied and the achievable
rate was obtained.

In this direction, most existing works on IRS-assisted systems have focused on
single-input single-output (SISO) or MISO systems, while limited research has
been devoted to MIMO communication, e.g., see [9, 10, 11]. In particular,
regarding the study of capacity, the capacity limit of point-to-point MIMO
IRS-assisted systems was studied in [9] and the capacity region with non-
orthogonal multiple access (NOMA) was characterized in [10]. In [11], the
ergodic capacity (EC) was studied for SISO channels. Moreover, in[12], the EC
for millimeter wave (mmWave) MIMO systems was investigated but not obtained in
closed-form. Similarly, in [13, 14], only upper bounds for the EC were
derived.

The previous observations motivate the topic of this paper, which is the
derivation of the exact EC of MIMO IRS-assisted systems for Rayleigh channels
under practical considerations while performing robust optimization. Contrary
to [9, 10, 11], we have accounted for both correlated fading and the
intertwinement between the amplitude and phase-shift of each element, which
have been disregarded in most previous works. Also, [11] considered only SISO
channels, while we focus on MIMO channels. Furthermore, [12] focused on mmWave
systems and [13, 14] derived just upper bounds, while these works did not
focus on the exact analysis of the EC. To this end, we have achieved to obtain
the probability density function (PDF) of an IRS-assisted MIMO channel with
correlation in a simple closed-form expression, and we have optimized the EC
by considering a practical phase-shift model. Moreover, the optimization in
[9, 10] relied on instantaneous channel state information (CSI), while our
analytical expression depends only on large-scale statistics as in [5, 6, 7],
and thus, it can be optimized once per several coherence intervals. Hence, our
approach is of great value since it achieves to reduce considerably the signal
overhead, which becomes prohibitive as the number of IRS elements increases.
Finally, we shed light on the impact of the fundamental system parameters on
the EC. For example, we show its degradation due to correlated Rayleigh
fading.

##  II System Model

We consider a MIMO communication system assisted by a two-dimensional
rectangular IRS dynamically adjusted by the IRS controller, where the
transmitter and the receiver have $M$ and $K$, antennas, respectively, as
shown in Fig. 1. The IRS, equipped with $N=N_{\mathrm{H}}N_{\mathrm{V}}$
nearly passive reflecting elements, enables the communication between the
transmitter and the receiver, where $N_{\mathrm{H}}$ and $N_{\mathrm{V}}$
denote the elements per row and per column of the IRS. Each element has size
$d_{\mathrm{H}}d_{\mathrm{V}}$, where $d_{\mathrm{H}}$ is its horizontal width
and $d_{\mathrm{V}}$ is the vertical height.

Figure 1: An IRS-assisted MIMO communication system with $M$ antennas at the
transmitter, $N$ IRS elements, and $K$ antennas at the receiver.

We assume narrowband transmission on quasi-static block-fading channels, where
in each block all the channels remain constant. In particular, let
${\mathbf{H}}_{1}=[{\mathbf{h}}_{1,1}\ldots,{\mathbf{h}}_{1,M}]\in\mathbb{C}^{N\times
M}$ be the channel matrix between the transmitter and the IRS with
${\mathbf{h}}_{1,i}\in\mathbb{C}^{N\times 1}$ for $i=1,\ldots,M$ being its
column vectors. Similarly, ${\mathbf{H}}_{2}\in\mathbb{C}^{K\times N}$
expresses the channel matrix between the IRS and the receiver. The subscripts
$1$ and $2$ correspond to the transmitter-IRS and IRS-receiver links,
respectively. Despite that many works, e.g., [3], assumed independent Rayleigh
fading model, we take into account the spatial correlations of all nodes,
which are unavoidable in practice and affect the performance [15, 8].
Especially, in [8], a correlation model, being suitable for IRS, was
presented.

By relying on the Kronecker model to account for the spatial correlation of
the MIMO channel for each link, the correlated Rayleigh fading distribution
for the $i$th link can be written as

$\displaystyle{\mathbf{H}}_{i}$
$\displaystyle={\mathbf{R}}_{i}^{\frac{1}{2}}{\mathbf{X}}_{i}{\mathbf{T}}_{i}^{\frac{1}{2}},~{}\mathrm{for}~{}i=1,2$
(1)

where ${\mathbf{R}}_{1}\in\mathbb{C}^{N\times N}$,
${\mathbf{T}}_{1}\in\mathbb{C}^{M\times M}$ and
${\mathbf{R}}_{2}\in\mathbb{C}^{K\times K}$,
${\mathbf{T}}_{2}\in\mathbb{C}^{N\times N}$ describe the deterministic
nonnegative semi-definite correlation matrices at the receive, transmit side
of link $1$ and $2$, respectively. Basically, ${\mathbf{R}}_{1}$ and
${\mathbf{T}}_{2}$, corresponding to the IRS, are identical. Notably, all the
correlation matrices, assumed to be known by the network, can be obtained
through existing estimation methods (see e.g., [16]). For the IRS, we adopt
the suitable correlation model presented in [8], while the correlations at the
transmitter and the receiver correspond to conventional linear antenna array
modeling (see e.g., [6]). Moreover, ${\mathbf{X}}_{1}\in\mathbb{C}^{N\times
M}$ and ${\mathbf{X}}_{2}\in\mathbb{C}^{K\times N}$ express the corresponding
fast-fading matrices with elements distributed as
$\mathcal{CN}\left(0,1\right)$. Also, $\beta_{1}$ and $\beta_{2}$ denote the
path-losses of the transmitter-IRS and IRS-receiver links, respectively.

Let
${\bm{\Theta}}=\mathrm{diag}\left(\alpha_{1}e^{j\phi_{1}},\ldots,\alpha_{N}e^{j\phi_{N}}\right)\in\mathbb{C}^{N\times
N}$ denote the reflecting beamforming (RBM) being diagonal and representing
the response of the $N$ IRS elements, where $\phi_{n}\in[-\pi,\pi)$ and
$\alpha_{n}(\phi_{n})\in[0,1]$ express the phase-shift and amplitude
coefficient of element $n$, respectively. Contrary to many previous works that
assumed lossless metasurfaces ($\alpha_{n}(\phi_{n})=1$), we consider that the
amplitude coefficient is phase-dependent based on the load or surface
impedances as mentioned in [2]. Especially, we consider that the amplitude
response is explicitly expressed as a function of the phase shift as

$\displaystyle\alpha_{n}(\phi_{n})=(1-\kappa_{\min})\left(\frac{\sin(\phi_{n}-\vartheta)+1}{2}\right)^{\\!\\!\xi}+\kappa_{\min},$
(2)

where $\kappa_{\min}\geq 0$ is the minimum amplitude, $\vartheta\geq 0$ is the
horizontal distance between $-\pi/2$ and $\kappa_{\min}$, which expresses the
distance between $-\pi/2$ and the phase shift $\phi_{n}$ that minimizes
$\sin(\phi_{n}-\vartheta)$ (i.e., $\phi_{n}=-\pi/2+\vartheta$), and $\xi$ is
the steepness of the function curve. If $\kappa_{\min}=1$ or $\xi=0$, (2)
results in the ideal phase-shift model, i.e., $\alpha_{n}(\phi_{n})=1$.

We assume that signals reflected by the IRS two or more times are ignored and
that no direct signal exists due to blockage from obstacles such as buildings.
Thus, the effective channel matrix is given by
${\mathbf{H}}_{2}{\bm{\Theta}}{\mathbf{H}}_{1}$. The received signal vector
${\mathbf{y}}\in\mathbb{C}^{K\times 1}$ by the receiver is given by

$\displaystyle{\mathbf{y}}={\mathbf{H}}_{2}{\bm{\Theta}}{\mathbf{H}}_{1}{\mathbf{x}}+{\mathbf{z}},$
(3)

where ${\mathbf{x}}$ is the transmit signal vector by the transmitter and
${\mathbf{z}}\sim{\cal C}{\cal N}({\mathbf{0}},\sigma^{2}{\mathbf{I}}_{M})$ is
the additive white Gaussian noise (AWGN) vector at the receiver with
$\sigma^{2}$ expressing the average noise power.

##  III Ergodic Capacity Analysis

By assuming that the transmitter has no channel knowledge while the receiver
has perfect CSI of both ${\mathbf{H}}_{1}$ and ${\mathbf{H}}_{2}$111It is
worthwhile to mention that by assuming receive RF chains with sensing
abilities integrated into the IRS, the acquisition of the CSI of individual
channels is feasible as suggested recently in [17]. On this ground, the
consideration of the imperfect CSI scenario, which is more practical, is the
topic of future work., the EC in (b/s/Hz) of the MIMO-assisted system in (3)
is expressed as

$\displaystyle
C=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{K}+\frac{\mathrm{SNR}}{M}{\mathbf{H}}_{2}{\bm{\Theta}}{\mathbf{H}}_{1}{\mathbf{Q}}{\mathbf{H}}_{1}^{{\scriptscriptstyle\mathsf{H}}}{\bm{\Theta}}^{{\scriptscriptstyle\mathsf{H}}}{\mathbf{H}}_{2}^{{\scriptscriptstyle\mathsf{H}}}\big{missing})\big{]},$
(4)

where the expectation in (4) is over ${\mathbf{H}}_{1}$, ${\mathbf{H}}_{2}$
while ${\mathbf{Q}}$ denotes the capacity-achieving input covariance, being
normalized by its energy per dimension and given by

$\displaystyle{\mathbf{Q}}=\frac{\mathbb{E}[{\mathbf{x}}{\mathbf{x}}^{{\scriptscriptstyle\mathsf{H}}}]}{\frac{1}{M}\mathbb{E}[\|{\mathbf{x}}\|^{2}]}.$
(5)

Note that the normalization guarantees that $\mathbb{E}[\tr({\mathbf{Q}})]=M$.
Also, we have

$\displaystyle\mathrm{SNR}=\frac{\mathbb{E}[\|{\mathbf{x}}\|^{2}]}{\frac{1}{K}\sigma^{2}}.$
(6)

Substitution of the channel expressions described by (1) into (4) gives (7).
Next, by denoting ${\bm{\Lambda}}_{{\mathbf{R}}_{i}}$,
${\bm{\Lambda}}_{{\mathbf{T}}_{i}}$, and ${\mathbf{P}}$ the diagonal
eigenvalue matrices corresponding to the matrices ${\mathbf{R}}_{i}$,
${\mathbf{T}}_{i}$, and ${\mathbf{Q}}$, the EC can be written as in (8). Note
that ${\mathbf{P}}$ describes the capacity-achieving power allocation.

$\displaystyle C$
$\displaystyle=\mathbb{E}\big{[}\\!\log_{2}\det\big({\bm{\mathrm{I}}}_{K}+\frac{\mathrm{SNR}}{M}{\mathbf{R}}_{2}{\mathbf{X}}_{2}{\mathbf{T}}_{2}^{\frac{1}{2}}{\bm{\Theta}}{\mathbf{R}}_{1}{\mathbf{X}}_{1}{\mathbf{T}}_{1}^{\frac{1}{2}}{\mathbf{Q}}{\mathbf{T}}_{1}^{\frac{1}{2}}{\mathbf{X}}_{1}^{{\scriptscriptstyle\mathsf{H}}}{\mathbf{R}}_{1}^{\frac{1}{2}}{\bm{\Theta}}^{{\scriptscriptstyle\mathsf{H}}}{\mathbf{T}}_{2}^{\frac{1}{2}}{\mathbf{X}}_{2}^{{\scriptscriptstyle\mathsf{H}}}\big{missing})\big{]}$
(7)
$\displaystyle=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{K}+\frac{\mathrm{SNR}}{M}{\bm{\Lambda}}_{{\mathbf{R}}_{2}}{\mathbf{X}}_{2}{\bm{\Lambda}}_{{\mathbf{T}}_{2}}^{\frac{1}{2}}{\bm{\Theta}}{\bm{\Lambda}}_{{\mathbf{R}}_{1}}^{\frac{1}{2}}{\mathbf{X}}_{1}{\bm{\Lambda}}_{{\mathbf{T}}_{1}}{\mathbf{P}}{\mathbf{X}}_{1}^{{\scriptscriptstyle\mathsf{H}}}{\bm{\Lambda}}_{{\mathbf{R}}_{1}}^{\frac{1}{2}}{\bm{\Theta}}^{{\scriptscriptstyle\mathsf{H}}}{\bm{\Lambda}}_{{\mathbf{T}}_{2}}^{\frac{1}{2}}{\mathbf{X}}_{2}^{{\scriptscriptstyle\mathsf{H}}}\big{missing})\big{]}.$
(8)

It is generally known that even the derivation of the PDF of the unordered
eigenvalue of
${\mathbf{H}}_{i}{\mathbf{H}}_{i}^{{\scriptscriptstyle\mathsf{H}}}$, where
${\mathbf{H}}_{i}$ is given as in (1), is a challenging problem that remains
unsolved. In the case of point-to-point MIMO channel, special cases of
correlated fading are considered, where correlation is assumed only at either
side of the transmitter or the receiver [15]. Also, it is known that doubly
correlated Rayleigh fading is not amenable to tractable manipulations [18].

Taking these facts into account, we notice that the capacity expression in (9)
includes two random matrices ${\mathbf{X}}_{1}$, ${\mathbf{X}}_{2}$ while
double correlated Rayleigh channels are assumed. To tackle this difficulty, we
assume correlation only at the transmitter or the receiver side while we
perform singular value decomposition (SVD) to the channel of the other link.
Next, we obtain the conditional PDF of the unordered eigenvalue of
${\mathbf{G}}{\mathbf{G}}^{{\scriptscriptstyle\mathsf{H}}}$, denoted by
$f_{\lambda|{\mathbf{L}}}$, and finally, we derive the marginal PDF of the
unordered eigenvalue. The derivation requires to consider cases, where either
${\bm{\Lambda}}_{{\mathbf{R}}_{2}}={\bm{\mathrm{I}}}_{N}$ or
${\bm{\Lambda}}_{{\mathbf{T}}_{1}}{\mathbf{P}}={\bm{\mathrm{I}}}_{M}$ as in
[15]. Note that, due to the reciprocity of MIMO channel, correlation at either
the receiver or transmitter is equivalent. The only difference is that the
former case including ${\bm{\Lambda}}_{{\mathbf{R}}_{2}}$ is determined
completely by the receive correlation while the latter case depends on both
the transmit correlation and the power allocation.

###  III-A Main Results

Let us start with ${\bm{\Lambda}}_{{\mathbf{R}}_{2}}={\bm{\mathrm{I}}}_{N}$,
then after applying the SVD to ${\mathbf{X}}_{1}$, i.e.,
${\mathbf{X}}_{1}={\mathbf{U}}_{1}{\mathbf{D}}_{1}{\mathbf{V}}_{1}^{{\scriptscriptstyle\mathsf{H}}}$,
where
${\mathbf{D}}_{1}=\text{diag}\left(\right)\left(\lambda_{1},\ldots,\lambda_{\min(M,N)}\right)$
is a diagonal matrix with diagonal elements being the singular values in
increasing order while the matrices ${\mathbf{U}}_{1}\in\mathbb{C}^{N\times
N}$ and ${\mathbf{V}}_{1}\in\mathbb{C}^{M\times M}$ are unitary and contain
the corresponding eigenvectors, we obtain

$\displaystyle C$
$\displaystyle=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{K}+\frac{\mathrm{SNR}}{M}{\mathbf{X}}_{2}{\bm{\Psi}}{\mathbf{X}}_{2}^{{\scriptscriptstyle\mathsf{H}}}\big{missing})\big{]},$
(9)

where

$\displaystyle\\!\\!\\!\\!\\!{\bm{\Psi}}\\!=\\!\\!\begin{cases}\text{diag}\left(\right)\\!\left(\gamma_{1}\\!(\phi_{1})\lambda_{1}^{2},\ldots,\gamma_{N}(\phi_{N})\lambda_{N}^{2}\right)&\\!\\!\\!\\!\\!N\leq
M\\\
\text{diag}\left(\right)\big{(}\\!\gamma_{1}\\!(\phi_{1})\lambda_{1}^{2},\ldots,\gamma_{M}(\phi_{M})\lambda_{M}^{2},\underbrace{\\!0,\ldots,0}_{N-M}\\!\big{)}&\\!\\!\\!\\!\\!N>M\end{cases}\\!\\!\\!\\!$
(10)

with
$\gamma_{i}(\phi_{i})=p_{i}\alpha_{i}^{2}(\phi_{i})\lambda_{{\mathbf{T}}_{1},i}\lambda_{{\mathbf{R}}_{1},i}\lambda_{{\mathbf{T}}_{2},i}$.
Note that we have taken into account the invariance of ${\mathbf{X}}_{2}$
under left and right unitary transformation and that to achieve capacity
(optimal transmit strategy), the eigenvectors of ${\mathbf{Q}}$ must coincide
with those of the transmit correlation ${\mathbf{T}}_{1}$ [15]. Since the
transmitter has no channel knowledge, equal power allocation to each transmit
antenna is the most reasonable strategy, i.e., choosing
${\mathbf{Q}}=\frac{P}{M}{\bm{\mathrm{I}}}_{M}$ (see [18] and references
therein). Hence, the required EC optimization below concerns only the phase
shifts. More concretely, we can write

$\displaystyle C$
$\displaystyle=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{K}+\frac{\mathrm{SNR}}{M}\hat{{\mathbf{X}}}_{2}\hat{{\mathbf{L}}}\hat{{\mathbf{X}}}_{2}^{{\scriptscriptstyle\mathsf{H}}}\big{missing})\big{]},$
(11)

where
$\hat{{\mathbf{X}}}_{2}\sim\mathcal{CN}\left({\mathbf{0}},{\bm{\mathrm{I}}}_{K}\otimes{\bm{\mathrm{I}}}_{q}\right)$
and
$\hat{{\mathbf{L}}}=\text{diag}\left(\right)\left(\gamma_{i}(\phi_{i})\lambda_{i}^{2}\right)_{i=1}^{q}$
with ${q}=\min\left(M,N\right)$.

Similarly, if we set
${\bm{\Lambda}}_{{\mathbf{T}}_{1}}{\mathbf{P}}={\bm{\mathrm{I}}}_{M}$ and
apply the SVD to ${\mathbf{X}}_{2}$, which means
${\mathbf{X}}_{2}={\mathbf{U}}_{2}{\mathbf{D}}_{2}{\mathbf{V}}_{2}^{{\scriptscriptstyle\mathsf{H}}}$,
where
${\mathbf{D}}_{2}=\text{diag}\left(\right)\left(\tilde{\lambda}_{1},\ldots,\tilde{\lambda}_{\min(K,N)}\right)$
is a diagonal matrix with diagonal elements being the singular values in
increasing order while ${\mathbf{U}}_{2}\in\mathbb{C}^{K\times K}$ and
${\mathbf{V}}_{2}\in\mathbb{C}^{N\times N}$ are unitary matrices, we obtain

$\displaystyle C$
$\displaystyle=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{M}+\frac{\mathrm{SNR}}{M}{\mathbf{X}}_{1}^{{\scriptscriptstyle\mathsf{H}}}\tilde{{\bm{\Psi}}}{\mathbf{X}}_{1}\big{missing})\big{]},$
(12)

where

$\displaystyle\\!\\!\\!\\!\\!\tilde{{\bm{\Psi}}}\\!=\\!\\!\begin{cases}\text{diag}\left(\right)\\!\left(\tilde{\gamma}_{1}\\!(\phi_{1})\tilde{\lambda}_{1}^{2},\ldots,\tilde{\gamma}_{N}(\phi_{N})\tilde{\lambda}_{N}^{2}\right)&\\!\\!\\!\\!\\!\\!N\leq
K\\\
\text{diag}\left(\right)\big{(}\\!\tilde{\gamma}_{1}\\!(\phi_{1})\tilde{\lambda}_{1}^{2},\ldots,\tilde{\gamma}_{K}(\phi_{K})\tilde{\lambda}_{K}^{2},\underbrace{\\!0,\ldots,0}_{N-K}\\!\big{)}&\\!\\!\\!\\!\\!N>K\end{cases}\\!\\!\\!\\!$
(13)

with
$\tilde{\gamma}_{i}(\phi_{i})=p_{i}\alpha_{i}^{2}(\phi_{i})\lambda_{{\mathbf{R}}_{1},i}\lambda_{{\mathbf{R}}_{2},i}\lambda_{{\mathbf{T}}_{2},i}$
and the invariance of ${\mathbf{X}}_{1}$ considered under left and right
unitary transformation. In a similar way to (11), the capacity can be
described by

$\displaystyle C$
$\displaystyle=\mathbb{E}\big{[}\log_{2}\det\big({\bm{\mathrm{I}}}_{M}+\frac{\mathrm{SNR}}{M}\tilde{{\mathbf{X}}}_{1}^{{\scriptscriptstyle\mathsf{H}}}\tilde{{\mathbf{L}}}\tilde{{\mathbf{X}}}_{1}\big{missing})\big{]},$
(14)

where
$\tilde{{\mathbf{X}}}_{1}\sim\mathcal{CN}\left({\mathbf{0}},{\bm{\mathrm{I}}}_{M}\otimes{\bm{\mathrm{I}}}_{\tilde{q}}\right)$
and
$\tilde{{\mathbf{L}}}=\text{diag}\left(\right)\big{(}\tilde{\gamma}_{i}(\phi_{i})\tilde{\lambda}_{i}^{2}\big{)}_{i=1}^{\tilde{q}}$
with $\tilde{q}=\min\left(K,N\right)$.

We observe that (11) and (14) have a similar expression. Hence, it is
sufficient to compute one of them, e.g., (11). We can rewrite it in an
equivalent way as

$\displaystyle\hat{C}$
$\displaystyle={s}\\!\int_{0}^{\infty}\\!\log_{2}\big{(}1+\frac{\mathrm{SNR}}{M}\hat{\lambda}\big{)}f_{\hat{\lambda}}(\hat{\lambda})\mathrm{d}\hat{\lambda},$
(15)

where ${s}=\min(K,q)$, $\hat{\lambda}$ is an unordered eigenvalue of the
random matrix
$\hat{{\mathbf{X}}}_{2}\hat{{\mathbf{L}}}\hat{{\mathbf{X}}}_{2}^{{\scriptscriptstyle\mathsf{H}}}$,
and $f_{\hat{\lambda}}(\hat{\lambda})$ is its PDF. Notably, there is no
expression for the distribution of $\hat{\lambda}$. We notice that (15)
resembles [18, Eq. 11] but includes a different form of the diagonal matrix,
which can not result in our expressions as a special case. Remarkably, (15)
provides the EC an IRS-assisted MIMO channel with correlation in terms of
large-scale statistics.

For this reason, it is required first to obtain a new exact closed-form
expression for the joint PDF of $0\leq c_{1}<\cdots<c_{q}<\infty$, where
$c_{i}=\gamma_{i}(\phi_{i})\hat{\lambda}_{i}^{2}$ are the elements of
$\hat{{\mathbf{L}}}$.

###### Lemma 1

The joint pdf of
$\hat{{\mathbf{L}}}=\text{diag}\left(\right)\left(c_{1},\ldots,c_{q}\right)$,
where $0\leq c_{1}<\cdots<c_{q}<\infty$, is given by

$\displaystyle
f_{\hat{{\mathbf{L}}}}\left(c_{1},\ldots,c_{q}\right)=\mathcal{K}\prod_{i=1}^{q}\frac{c_{i}^{p-q}e^{-\frac{c_{i}}{\gamma_{i}}}}{\gamma_{i}^{p-q+1}}\prod_{i<j}^{q}\left(\frac{c_{j}}{\gamma_{j}}-\frac{c_{i}}{\gamma_{i}}\right)^{2},$
(16)

where
$\mathcal{K}=\left(\prod_{i=1}^{q}\Gamma\left(q-i+1\right)\Gamma\left(p-i+1\right)\right)^{-1}$
with $p=\max(K,N)$.

###### Proof:

See Appendix A. ∎

Taking into account Lemma 1, we obtain $f_{\hat{\lambda}}(\hat{\lambda})$
according to the following theorem.

###### Theorem 1

The marginal PDF of an unordered eigenvalue $\hat{\lambda}$ of
${\mathbf{X}}_{1}^{{\scriptscriptstyle\mathsf{H}}}\tilde{{\mathbf{L}}}{\mathbf{X}}_{1}$
is given by

$\displaystyle\\!f_{\hat{\lambda}}(\hat{\lambda})\\!=\\!\frac{\mathcal{K}}{s\\!\prod_{i<j}^{q}\\!\left(\gamma_{j}\\!-\\!\gamma_{i}\right)}$
$\displaystyle\sum_{l=1}^{q}\sum_{k=q-s+1}^{q}\frac{\hat{\lambda}^{(M+p-2q+2k+l-3)/2}}{\Gamma\\!\left(M\\!-\\!q\\!+\\!k\right)\gamma_{l}^{\frac{p-M+l-1}{2}}}$
$\displaystyle\times\mathrm{K}_{p-M+l-1}(2\sqrt{\hat{\lambda}/\gamma_{l}})G_{l,k},$
(17)

where $G_{l,k}$ is the $(l,k)th$ cofactor of a $q\times q$ matrix whose
$(m,n)th$ entry is $\\{{\mathbf{G}}\\}_{m,n}=\Gamma(p-q+m+n-1)$.

###### Proof:

See Appendix B. ∎

###### Remark 1

Theorem 1 presents the PDF of an IRS-assisted MIMO channel with correlation in
a simple closed-form expression. Based on this theorem, the EC in (15) depends
on the number of antennas at the transmitter, the receiver, and the number or
IRS elements together with the eigenvalues of the correlation matrices. A
notable dependence in (17) concerns the phase-shift and amplitude in terms of
$\alpha_{i}^{2}(\phi_{i})$.

###  III-B RBM optimization

The EC can be maximized in terms of the phase shifts. Relying on the common
assumption of infinite resolution phase shifters, we formulate the
optimization problem as

$\displaystyle\begin{split}&\\!\\!\\!(\mathcal{P}1)~{}\max_{{\bm{\Phi}}}~{}~{}\hat{C}\\\
&~{}~{}~{}~{}~{}~{}\mathrm{s.t}~{}-\pi\leq\phi_{n}\leq\pi,~{}~{}n\\!=\\!1,\dots,q,\end{split}$
(18)

where $\hat{C}$ is given by (15) based on (11) or (14).

The problem $(\mathcal{P}1)$ concerns a non-convex maximization with respect
to $\phi_{n}$. Taking the expression of $f_{\hat{\lambda}}(\hat{\lambda})$
into account, the optimization of (15) is basically a constrained maximization
problem whose solution could be given by means of the projected gradient
ascent until convergence to a stationary point [6]. The convergence is
guaranteed due to the power constraint. Since the dependence on phase shifts
is hidden only in $f_{\hat{\lambda}}(\hat{\lambda})$, below, we focus on their
optimization. According to the algorithm, let
${\mathbf{s}}^{i}=[\phi_{1}^{i},\ldots,\phi_{q}^{i}]^{{\scriptscriptstyle\mathsf{T}}}$
denote the vector including the phases at step $i$. The next iteration point
results in the increase of $f_{\hat{\lambda}}(\hat{\lambda})$ towards to its
convergence. Hence, we have

$\displaystyle\tilde{{\mathbf{s}}}^{i+1}$
$\displaystyle={\mathbf{s}}^{i}+\mu{\mathbf{q}}^{i},$ (19)
$\displaystyle{\mathbf{s}}^{i+1}$
$\displaystyle=\exp\left(j\arg\left(\tilde{{\mathbf{s}}}^{i+1}\right)\right),$
(20)

where $\mu$ is the step size and ${\mathbf{q}}^{i}$ is the adopted ascent
direction at step $i$ with
$[{\mathbf{q}}^{i}]_{n}=\partialderivative{f_{\hat{\lambda}}(\hat{\lambda})}{\phi_{n}}$.
This derivative is provided by Lemma 2 below. The solution is found by
formulating the projection problem
$\min_{|\phi_{n}|=1,n=1,\ldots,N}\|{\mathbf{s}}-\tilde{{\mathbf{s}}}\|^{2}$
based on (19) and (20) under the constraint in (18). Note that the suitable
step size requires computation at each iteration, which is achieved by means
of the backtracking line search [19].

###### Lemma 2

The derivative of $f_{\hat{\lambda}}(\hat{\lambda})$ with respect to
$\phi_{n}$ is provided by

$\displaystyle\\!\partialderivative{f_{\hat{\lambda}}(\hat{\lambda})}{\phi_{n}}\\!=\\!-\frac{p_{n}\alpha_{n}(\phi_{n})\lambda_{{\mathbf{T}}_{1},n}\lambda_{{\mathbf{R}}_{1},n}\lambda_{{\mathbf{T}}_{2},n}\mathcal{K}}{s\\!\prod_{i<j}^{q}\\!\left(\gamma_{j}\\!-\\!\gamma_{i}\right)}(1-\kappa_{\min})\xi$
$\displaystyle\times\cos(\phi_{n}-\vartheta)\left(\frac{\sin(\phi_{n}-\vartheta)+1}{2}\right)^{\\!\\!\xi-1}\big{(}\tr\big({\mathbf{V}}^{-1}\partialderivative{{\mathbf{V}}}{\gamma_{n}}\big{missing})$
$\displaystyle\times\\!\\!\sum_{l=1}^{q}\sum_{k=q-s+1}^{q}\\!\\!\frac{\hat{\lambda}^{(M+p-2q+2k+l-3)/2}}{\Gamma\\!\left(M\\!-\\!q\\!+\\!k\right)\\!\gamma_{l}^{\frac{p\\!-\\!M\\!+\\!l\\!-\\!1}{2}}}\mathrm{K}_{p-M+l-1}(2\sqrt{\\!\hat{\lambda}/\gamma_{l}})G_{l,k}$
$\displaystyle+\sum_{k=q-s+1}^{q}\\!\\!\frac{\hat{\lambda}^{(2M+\bar{p}-2q+2k-2)/2}}{\Gamma\\!\left(M\\!-\\!q\\!+\\!k\right)\\!\gamma_{n}^{\frac{\bar{p}}{2}}}G_{n,k}\big{(}\frac{\bar{p}}{2\gamma_{n}}\mathrm{K}_{\bar{p}}(2\sqrt{\\!\hat{\lambda}/\gamma_{n}})$
$\displaystyle-\frac{1}{\gamma_{n}}\sqrt{\frac{\hat{\lambda}}{\gamma_{n}}}\big{(}\mathrm{K}_{\bar{p}-1}(2\sqrt{\\!\hat{\lambda}/\gamma_{n}})+\mathrm{K}_{\bar{p}+1}(2\sqrt{\\!\hat{\lambda}/\gamma_{n}})\\!\big{)}\\!\big{)}\\!\big{)},$
(21)

where $\bar{p}=p-M+n-1$, ${\mathbf{V}}$ is the Vandermonde matrix satisfying
$\det({\mathbf{V}})=\prod_{i<j}^{q}\left(\gamma_{j}-\gamma_{i}\right)$, and
$\partialderivative{{\mathbf{V}}}{\gamma_{n}}$ is a matrix whose elements are
the derivatives of ${\mathbf{V}}$ with respect to $\gamma_{n}$.

###### Proof:

The computation of the derivative is straightforward since
$f_{\hat{\lambda}}(\hat{\lambda})$ is in closed-form. We notice that the
dependence on the phase shifts is inside $\gamma_{l}$ in terms of
$\alpha^{2}_{l}(\phi_{l})$. Hence, the derivative is obtained as in (21) by
using the derivative of (2), [20, Eq. 46], which gives the derivative of a
determinant, and [21, Eq. 03.04.20.0014.02], which gives the derivative of
$K_{v}(\cdot)$ after some simple algebraic manipulations. ∎

##  IV Numerical Results

We consider uniform linear arrays (ULAs) for the configuration of both the
transmitter and the receiver, while the IRS consists of a uniform planar array
(UPA). Based on the 3GPP Urban Micro (UMi) scenario from TR36.814 for a
carrier frequency of $2.5$ GHz and noise level $-80$ dBm, the path losses for
${\mathbf{H}}_{1}$ and ${\mathbf{H}}_{2}$ are generated based on the NLOS
version [22]. Specifically, therein, the overall path loss for the IRS-
assisted link is $\beta=\beta_{1}\beta_{2}$, where
$\beta_{i}=C_{i}d_{i}^{-\nu_{i}},~{}i=1,2$ with $C_{1}=26$ dB, $C_{2}=28$ dB,
$\nu_{1}=2.2$, $\nu_{2}=3.67$. The variables $d_{1}=8~{}\mathrm{m}$ and
$d_{2}=60~{}\mathrm{m}$ express the distances between the transmitter and the
IRS, and the IRS and the receiver, respectively. As can be seen, since the IRS
is closer to the transmitter, the path-loss exponent of this link is lower
because fewer obstacles are expected. We use $5$ dBi antennas at the
transmitter ($M=4$) and the receiver ($K=4$), and their correlation matrices
are given by [6]. Unless otherwise stated, we consider the following values.
The correlation matrix for the IRS is given by [8], where
$d_{\mathrm{H}}\\!=\\!d_{\mathrm{V}}\\!=\\!\lambda/4$. The phase shift model
parameters are $\kappa_{\min}=0.8$, $\xi=1.6$, and $\vartheta=0.43\pi$ [2].
The figures correspond to
${\bm{\Lambda}}_{{\mathbf{R}}_{2}}={\bm{\mathrm{I}}}_{N}$, while similar
observations can be extracted in the case
${\bm{\Lambda}}_{{\mathbf{T}}_{1}}{\mathbf{P}}={\bm{\mathrm{I}}}_{M}$ since we
do not focus on the transceiver design but on the impact of the IRS.
Furthermore, Monte-Carlo (MC) simulations coincide with the analytical results
in all cases, which corroborates our analysis.

Figure 2: EC of an IRS-assisted MIMO system versus the number of IRS elements
$N$ for varying IRS correlation ( $\kappa_{\min}=0.8$, $\xi=1.6$).  Figure 3:
EC of an IRS-assisted MIMO system versus the number of IRS elements $N$ for
varying phase-shift model parameters (
$\mathrm{d_{\mathrm{H}}=d_{\mathrm{V}}=\lambda/8}$).

Fig. 2 illustrates the EC versus the number of IRS elements. In general, we
observe that $\hat{C}$ increases with the number of IRS elements. Moreover,
given specific values for the phase-shift model, we study the impact of the
IRS correlation. We observe that the unrealistic independent Rayleigh fading
gives the highest $\hat{C}$, while the decrease of the distance between the
IRS elements (higher correlation) results in a lower capacity. Also, we depict
the scenarios with no IRS and with random phase shifts, i.e., not optimized.
The former corresponds to a horizontal line as $N$ increases since the
capacity is independent of $N$, while the latter shows that the phase-shift
optimization is clearly very beneficial during the IRS implementation.

In Fig. 3, we depict the EC for different phase-shift settings. The “dashed”
and “solid” lines correspond to variation of $\kappa_{\min}$ and $\xi$,
respectively. For the sake of comparison, we have also shown the case of ideal
phase-shift model ( $\kappa_{\min}=1$, $\xi=0$). We observe that when
$\kappa_{\min}$ increases, EC increases. On the contrary, when $\xi$
increases, $\hat{C}$ decreases. Notable, these observations coincide with the
results in [11] for SISO-IRS assisted channels.

##  V Conclusion

In this paper, we derived the PDF and the EC of a point-to-point MIMO system
assisted by an IRS under the unavoidable realistic conditions of correlated
Rayleigh fading and correlation between the amplitude and the phase-shift of
each IRS element. Also, we provided the maximization of the EC with respect to
the phase shifts of the IRS elements. This maximization is advantageous since
it is accompanied by reduced overhead and computational complexity by
exploiting the dependence of the EC by just large-scale statistics. The
analytical results are verified by MC simulations and show how the various
system parameters affect the EC. For example, correlated fading results in
performance degradation. Future works on EC should account for Rician fading
conditions, and possibly, the existence of a direct path between the
transmitter and the receiver.

##  Appendix A Proof of Lemma 1

The joint PDF of
${\mathbf{W}}=\text{diag}\left(\right)\left(w_{1},\ldots,w_{q}\right)$, where
$w_{i}=\hat{\lambda}_{i}^{2}$ and $p=\max(K,N)$, is given by [23] as

$\displaystyle\\!\\!\\!\\!f_{{\mathbf{W}}}\\!\left(w_{1},\ldots,w_{q}\right)\\!=\\!\mathcal{K}e^{-\sum_{i=1}^{q}\\!w_{i}}\\!\prod_{i=1}^{q}\\!w_{i}^{p-q}\prod_{i<j}^{q}\\!\\!\left(w_{j}-w_{i}\right)^{2}\\!.$
(22)

Taking into account that $\hat{\lambda}_{i}^{2}=c_{i}/\gamma_{i}(\phi_{i})$,
we obtain the joint PDF of
$\hat{{\mathbf{L}}}=\text{diag}\left(\right)\left(c_{1},\ldots,c_{q}\right)$
after applying a vector transformation to (22) as

$\displaystyle f_{\hat{{\mathbf{L}}}}\\!\left(c_{1},\ldots,c_{q}\right)$
$\displaystyle=f_{{\mathbf{W}}}\\!\left(\frac{c_{1}}{\gamma_{1}},\ldots,\frac{c_{q}}{\gamma_{q}}\right)$
$\displaystyle\times|{\mathbf{J}}\left(\left(w_{1},\ldots,w_{q}\right)\to\left(c_{1},\ldots,c_{q}\right)\right)|,$
(23)

where the Jacobian transformation is evaluated as

$\displaystyle|{\mathbf{J}}\left(\left(w_{1},\ldots,w_{q}\right)\to\left(c_{1},\ldots,c_{q}\right)\right)|=\prod_{i=1}^{q}\frac{1}{\gamma_{i}}.$
(24)

Substitution of (22) and (24) into (23) gives the joint PDF of
$\hat{{\mathbf{L}}}$ in (16).

##  Appendix B Proof of Theorem 1

The unconditional PDF of an unordered eigenvalue of
${\mathbf{X}}_{1}^{{\scriptscriptstyle\mathsf{H}}}\tilde{{\mathbf{L}}}{\mathbf{X}}_{1}$
is obtained by taking the expectation over $\tilde{{\mathbf{L}}}$, i.e.,
$\mathbb{E}_{\tilde{{\mathbf{L}}}}[f_{\hat{\lambda}|\tilde{{\mathbf{L}}}}(\hat{\lambda})]$.
For this reason, we employ the conditional unordered eigenvalue PDF from [18,
Lemma 1] after expressing it as

$\displaystyle\\!\\!\\!\\!f_{\hat{\lambda}|\tilde{{\mathbf{L}}}}(\hat{\lambda})\\!=\\!\frac{1}{s\prod_{i<j}^{q}\\!\left(c_{j}\\!-\\!c_{i}\right)}\\!\\!\sum_{k=q-s+1}^{q}\\!\\!\frac{\hat{\lambda}^{M-q+k-1}}{\Gamma\\!\left(M\\!-\\!q\\!+\\!k\right)}\det\\!\left({\mathbf{D}}_{k}\right)\\!,$
(25)

where ${\mathbf{D}}_{k}$ is a $q\times q$ matrix with entries

$\displaystyle\\{{\mathbf{D}}_{k}\\}_{m,n}=\begin{cases}c_{m}^{n-1},&~{}~{}~{}n\neq
k,\\\ e^{-\hat{\lambda}/c_{m}}c_{m}^{q-M-1},&~{}~{}~{}n=k.\end{cases}$ (26)

The unconditional PDF is derived as

$\displaystyle f_{\hat{\lambda}}(\hat{\lambda})$
$\displaystyle=\frac{\mathcal{K}}{s}\sum_{k=q-s+1}^{q}\frac{\hat{\lambda}^{M-q+k-1}}{\Gamma\left(M-q+k\right)}\mathcal{I}_{k},$
(27)

where

$\displaystyle\mathcal{I}_{k}$
$\displaystyle\\!=\\!\\!\\!\displaystyle\int_{0\leq
c_{1}<\cdots<c_{q}<\infty}\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\\!\det\\!\left({\mathbf{D}}_{k}\right)\\!\prod_{l=1}^{q}\\!\frac{c_{l}^{p-q}e^{-\frac{c_{l}}{\gamma_{l}}}}{\gamma_{l}^{p-q+1}}\\!\prod_{i<j}^{q}\frac{\\!\left(\frac{c_{j}}{\gamma_{j}}\\!-\\!\frac{c_{i}}{\gamma_{i}}\\!\right)^{2}}{\left(c_{j}-c_{i}\right)}\mathrm{d}c_{1}\cdots\mathrm{d}c_{q}.$

Meanwhile, according to the matrix theory, it holds that
$\prod_{i<j}^{q}\left(\theta_{j}-\theta_{i}\right)=\det({\mathbf{X}})$, where
${\mathbf{X}}$ is a Vandermonde matrix with entry given by $\theta_{i}^{j-1}$.
Based on this property, we obtain

$\displaystyle\mathcal{I}_{k}=\frac{1}{\prod_{i<j}^{q}\left(\gamma_{j}-\gamma_{i}\right)}\det\left({\mathbf{Y}}_{k}\right),$
(28)

where ${\mathbf{Y}}_{k}$ is a $q\times q$ matrix with entries

$\displaystyle\\!\\!\\!\\!\\{{\mathbf{Y}}_{k}\\}_{m,n}=\\!\begin{cases}\int_{0}^{\infty}x^{p-q+m+n-2}e^{-x}\mathrm{d}x,&\\!\\!\\!\\!\\!~{}~{}~{}n\neq
k,\\\
\int_{0}^{\infty}e^{-\hat{\lambda}/\gamma_{l}x}x^{p-M+m-2}e^{-x}\mathrm{d}x,&\\!\\!\\!\\!\\!~{}~{}~{}n=k.\end{cases}$
(29)

Note that in (29), we have made a change of variables, i.e., we have set
$x=c_{l}/\gamma_{l}$. The integrals are evaluated based on [24, Eq.3.381.4]
and [24, Eq.3.471.9] as

$\displaystyle\int_{0}^{\infty}x^{p-q+m+n-2}e^{-x}\mathrm{d}x$
$\displaystyle=\Gamma(p-q+m+n-1),$ (30)
$\displaystyle\int_{0}^{\infty}e^{-\hat{\lambda}/\gamma_{l}x}x^{\tilde{p}-1}e^{-x}\mathrm{d}x$
$\displaystyle=2\left(\frac{\hat{\lambda}}{\gamma_{l}}\right)^{\\!\\!\frac{\tilde{p}}{2}}\mathrm{K}_{\tilde{p}}(2\sqrt{\hat{\lambda}/\gamma_{l}}),$
(31)

where $\tilde{p}=p-M+m-1$, $K_{v}(\cdot)$ is the modified Bessel function of
the second kind [24, Eq. 8.432.6]. Substitution of (30) and (31) into (29)
together with the application of Laplace’s expansion provides the desired
result.

## References

  * [1] E. Basar _et al._ , “Wireless communications through reconfigurable intelligent surfaces,” _IEEE Access_ , vol. 7, pp. 116 753–116 773, 2019\.
  * [2] S. Abeywickrama _et al._ , “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” _IEEE Trans. Commun._ , vol. 68, no. 9, pp. 5849–5863, 2020.
  * [3] Q. Wu and R. Zhang, “Intelligent reflecting surface enhanced wireless network via joint active and passive beamforming,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 11, pp. 5394–5409, 2019.
  * [4] A. M. Elbir _et al._ , “Deep channel learning for large intelligent surfaces aided mm-Wave massive MIMO systems,” _IEEE Wireless Commun. Lett._ , vol. 9, no. 9, pp. 1447–1451, 2020.
  * [5] A. Papazafeiropoulos _et al._ , “Coverage probability of distributed IRS systems under spatially correlated channels,” vol. 10, no. 8, pp. 1722–1726, 2021.
  * [6] A. Papazafeiropoulos _et al._ , “Intelligent reflecting surface-assisted MU-MISO systems with imperfect hardware: Channel estimation, beamforming design,” _IEEE Trans. Wireless Commun._ , 2021.
  * [7] Z. Wang _et al._ , “Massive MIMO communication with intelligent reflecting surface,” _arXiv preprint arXiv:2107.04255_ , 2021.
  * [8] E. Björnson and L. Sanguinetti, “Rayleigh fading modeling and channel hardening for reconfigurable intelligent surfaces,” _IEEE Wireless Commun. Lett._ , vol. 10, no. 4, pp. 830–834, 2021.
  * [9] S. Zhang and R. Zhang, “Capacity characterization for intelligent reflecting surface aided MIMO communication,” _IEEE J. Sel. Areas Commun._ , vol. 38, no. 8, pp. 1823–1838.
  * [10] X. Mu _et al._ , “Capacity and optimal resource allocation for IRS-assisted multi-user communication systems,” _IEEE Trans. Commun._ , vol. 69, no. 6, pp. 3771–3786, 2021.
  * [11] Y. Zhang _et al._ , “Performance analysis of RIS-aided systems with practical phase shift and amplitude response,” _IEEE Trans. Veh. Tech._ , vol. 70, no. 5, pp. 4501–4511, 2021.
  * [12] C. Guo _et al._ , “Maximum ergodic capacity of intelligent reflecting surface assisted MIMO wireless communication system,” in _International Conference on Communications and Networking in China_. Springer, 2019, pp. 331–343.
  * [13] F. Yang _et al._ , “Intelligent reflecting surface-assisted mmWave communication exploiting statistical CSI,” in _IEEE International Conference on Communications (ICC)_ , 2020, pp. 1–6.
  * [14] J. Wang _et al._ , “Joint transmit beamforming and phase shift design for reconfigurable intelligent surface assisted MIMO systems,” _IEEE Trans. Cogn. Commun. Net._ , vol. 7, no. 2, pp. 354–368, 2021.
  * [15] G. Alfano _et al._ , “Capacity of MIMO channels with one-sided correlation,” in _Eighth IEEE International Symposium on Spread Spectrum Techniques and Applications-Programme and Book of Abstracts_ , pp. 515–519.
  * [16] D. Neumann, M. Joham, and W. Utschick, “Covariance matrix estimation in massive MIMO,” _IEEE Signal Process. Lett._ , vol. 25, no. 6, pp. 863–867, 2018.
  * [17] B. Zheng, C. You, and R. Zhang, “Efficient channel estimation for double-IRS aided multi-user MIMO system,” vol. 69, no. 6, pp. 3818–3832.
  * [18] S. Jin _et al._ , “Ergodic capacity analysis of amplify-and-forward MIMO dual-hop systems,” _IEEE Trans. on Inform. Theory_ , vol. 56, no. 5, pp. 2204–2224, 2010.
  * [19] S. Boyd, S. P. Boyd, and L. Vandenberghe, _Convex optimization_. Cambridge university press, 2004.
  * [20] K. B. Petersen and M. S. Pedersen, “The matrix cookbook, nov 2012,” _URL http://www2. imm. dtu. dk/pubdb/p. php_ , vol. 3274, p. 14, 2012.
  * [21] T. W. f. s. Wolfram. [Online]. Available: http://functions.wolfram.com
  * [22] 3GPP, “Further advancements for E-UTRA physical layer aspects (Release 9),” 3GPP TS 36.814, Tech. Rep., 2017.
  * [23] A. T. James, “Distributions of matrix variates and latent roots derived from normal samples,” _The Ann. of Math. Statist._ , pp. 475–501, 1964.
  * [24] I. S. Gradshteyn and I. M. Ryzhik, “Table of integrals, series, and products,” _Alan Jeffrey and Daniel Zwillinger (eds.), Seventh edition (Feb 2007)_ , vol. 885, 2007.

solutionfile


# Hierarchical Semantic Tree Concept Whitening for Interpretable Image
Classification

Haixing Dai*, Lu Zhang*, Lin Zhao, Zihao Wu, Zhengliang Liu, David Liu,
Xiaowei Yu, Yanjun Lyu, Changying Li, Ninghao Liu, Tianming Liu, Dajiang Zhu *
Co-first authors.
Haixing Dai, Lin Zhao, Zihao Wu, Zhengliang Liu, Ninghao Liu, Tianming Liu and
Changying Li are with the Department of Computer Science, University of
Georgia, Athens, GA, USA. (e-mail: hd54134, lin.zhao, zw63397,zl18864,
ninghao.liu<EMAIL_ADDRESS>changyingl@gmail.com).
Lu Zhang, Xiaowei Yu, Yanjun Lyu and Dajiang Zhu are with the Department of
Computer Science and Engineering, The University of Texas at Arlington,
Arlington, TX, USA. (e-mail: lu.zhang2, xxy1302<EMAIL_ADDRESS><EMAIL_ADDRESS>
David Weizhong Liu is with Athens Academy, Athens, GA, USA.(e-mail:
<EMAIL_ADDRESS>
###### Abstract

With the popularity of deep neural networks (DNNs), model interpretability is
becoming a critical concern. Many approaches have been developed to tackle the
problem through post-hoc analysis, such as explaining how predictions are made
or understanding the meaning of neurons in middle layers. Nevertheless, these
methods can only discover the patterns or rules that naturally exist in
models. In this work, rather than relying on post-hoc schemes, we proactively
instill knowledge to alter the representation of human-understandable concepts
in hidden layers. Specifically, we use a hierarchical tree of semantic
concepts to store the knowledge, which is leveraged to regularize the
representations of image data instances while training deep models. The axes
of the latent space are aligned with the semantic concepts, where the
hierarchical relations between concepts are also preserved. Experiments on
real-world image datasets show that our method improves model
interpretability, showing better disentanglement of semantic concepts, without
negatively affecting model classification performance.

###### Index Terms:

Explainable AI (XAI), hierarchical tree of semantic concepts, image embedding,
image interpretation.

##  I Introduction

Machine learning interpretability has recently received considerable attention
in various domains [1, 2, 3, 4]. An important challenge that arises with deep
neural networks (DNNs) is the opacity of semantic meanings of data
representations in hidden layers. Several types of methods have been proposed
to tackle the problem. First, recent works have shown that some neurons could
be aligned with certain high-level semantic patterns in data [5, 6]. Second,
it is possible to extract concept vectors [7] or clusters [8] to identify
semantic meanings from latent representations. However, these methods are
built upon the assumption that semantic patterns are already learned by DNNs,
and the models would admit the post-hoc method of a specific form. There is no
guarantee that the assumption holds true for any model, especially when
meaningful patterns or rules may not be manifested in the model, thus leading
to over-interpretation [9, 3]. Meanwhile, although many post-hoc explanation
methods are proposed with the expectation of improving or debugging models, it
is challenging to achieve this goal in practice. Although we could collect
human annotations to guide prediction explanations and improve model
credibility [10, 11], manually labeling or checking semantic concepts is
rather difficult. Unlike explaining individual predictions, which is a local
and instance-level task, extracting concepts provides a global understanding
of models, where manual inspection of such interpretation is time-consuming
and much harder, if not impossible.

Instead of relying on post-hoc approaches, we aim to instill interpretability
as a constraint into model establishment. For example, explanation
regularization is proposed in [12], but it constrains gradient magnitude
instead of focusing on semantic concepts. Meanwhile, $\beta$-VAE and its
variants [13, 14] add independence constraints to learn disentangled factors
in latent representations, but it is difficult to explicitly specify and align
latent dimensions with semantic meanings. Ideally, we want to construct DNNs
whose latent space could tell us how it is encoding concepts. The recent
decorrelated batch normalization (DBN) method [15] normalizes representations,
providing an end-to-end technique for manipulating representations, but it is
not directly related to interpretability.

In this work, we propose a novel Hierarchical Semantic Tree Concept Whitening
(HaST-CW) model to decorrelate the latent representations in image
classification for disentangling concepts with hierarchical relations. The
idea of our work is illustrated in Fig. 1. Specifically, we define each
concept as one class of objects, where the concepts are of different
granularities and form a hierarchical tree structure. We decorrelate the
activations of neural network layers, so that each concept is aligned with one
or several latent dimensions. Unlike the traditional DBN method (Fig. 1a),
which treats different concepts as independent, our method is able to leverage
the hierarchically related organization of label concepts inherent in domain
knowledge (Fig. 1b). The consideration of relations between different concepts
is crucial in many real-world applications [16, 17]. For example, in the
healthcare domain, the relationship of different disease stages (concepts) may
reflect the progression of the disease, which is significant for reversing
pathology [18, 19, 20, 21]. Also, in the precision agriculture domain [2, 22],
real-time monitoring of interactions of multiple agricultural objects
(concepts) with each other and with the environment are crucial in maintaining
agro-ecological balance [2]. In our model, a novel semantic constraint (SC)
loss function is designed to regularize representations. As a result, the data
representations of two concepts with higher semantic similarity will be closer
with each other in the latent space. Moreover, a new hierarchical concept
whitening (HCW) method is proposed to decorrelate different label concepts
hierarchically. We evaluated the proposed HaST-CW model using a novel
agriculture image dataset called Agri-ImageNet. The results suggest that our
model could preserve the semantic relationship between the label concepts, and
provide a clear understanding of how the network gradually learns the concept
in different layers, without hurting classification performance.

Figure 1: The intuition behind HaST-CW. (a) Distribution of discrete concepts
in the latent space after applying concept whitening. (b) Distribution of
hierarchical concepts after applying HaST-CW.

##  II Related Work

Post-Hoc Interpretation. Post-Hoc interpretation can be divided into
approaches that explain predictions or models [1, 3]. Prediction-oriented
interpretation aims to develop faithful and robust measures to quantify
feature importance towards individual predictions for identifying those
features (e.g., pixels, super-pixels, words) that made most contributions [23,
24, 25, 26, 27, 28]. Model-oriented interpretation analyzes behaviors of
neural networks either by characterizing the function of model components [29,
5, 30] or analyzing semantic concepts from latent representations [7, 8, 31,
32]. The proposed method also targets concept-level interpretation in deep
neural networks. Different from post-hoc techniques that focus on discovering
existing patterns in models, the newly proposed HaST-CW proactively injects
concept-related knowledge into training and disentangles different concepts to
promote model interpretability.

Inherently Interpretable Models. Another school of thought favors building
inherently explainable machine learning models [33, 34]. Some approaches
design models that highlight prototypical features of samples as
interpretation. For example, Chen et al. [35] classifies images by dissecting
images into parts and comparing these components to similar prototypes towards
prediction. Li et al. [36] designs an encoder-decoder framework to allow
comparisons between inputs and the learned prototypes in latent space. Some
other works such as $\beta$-VAE and its variants [13, 14] regularize
representation learning for autoencoders to produce disentangled factors in
representation dimensions, but the semantic meaning of each dimension remains
unknown without further manual inspection. In contrast, our method attempts to
explicitly align latent dimensions with specific semantic concepts contained
in external knowledge. A recent technique called Concept Whitening (CW) [34]
constrains the latent space, after revising Batch Whitening [37, 38], such
that it aligns with predefined classes. Our method attempts to infuse more
complex knowledge of concept relations into representation learning.

Applying Whitening to Computer Vision. Whitening is a standard image
preprocessing technique, which refers to transforming the covariance matrix of
input vectors into the identity matrix. In fact, the well-known Batch
Normalization [39] can be regarded as a variant of whitening where only the
normalization process is retained. There are many works in deep learning that
describe the effectiveness of whitening [40, 41, 42] and the process of
finding the whitening matrix [43]. Our work further takes semantics into
consideration during the whitening process towards more interpretable
representation learning.

##  III Methodology

###  III-A Overview

The proposed HaST-CW model aims to preserve the underlying hierarchical
relationship of label concepts, as well as to disentangle these concepts by
decorrelating their latent representations. To achieve this goal, we leverage
the hierarchical tree structure of the label concepts extracted from specific
domain knowledge (Sec. III-B). Then, the obtained structure of label concepts
is used as prior knowledge to be instilled into the model for guiding the
representation learning process. There are two key components in the knowledge
instillation process – the hierarchical concept whitening (HCW) module and the
semantic constraint (SC) loss, which will be elaborated in Sec. III-C and Sec.
III-D, respectively.

Figure 2: Hierarchical Tree Structure of Concepts.

###  III-B The Hierarchical Semantic Tree of Concepts

In this work, we used a newly collected and curated Agri-ImageNet dataset to
develop and evaluate the HaST-CW model. There are 9173 high quality images in
Agri-ImageNet, covering 21 different types of agricultural objects. Taking
each type of agricultural object as one class, we have 21 label concepts in
total. Some pairs of agriculture objects have the supertype-subtype
relationship between them, so we obtain the parent-child relationship between
the corresponding labels. As a result, a tree structure is built to represent
the underlying hierarchically related organization of label concepts, which is
shown in Fig. 2. Two concepts connected in the tree structure means they have
parent-child relationship, where the parent is located at the lower hierarchy
level. Besides the parent-child relation, we further introduce two notions –
brother and cousin. If two concepts have the same parent, then they are
brothers. If the parents of two concepts are brothers, then the two concepts
are cousins. According to the laws of inheritance: (1) objects with the
parent-child relation should be more similar than those with the uncle-child
relation (vertical parent-child relationship); and (2) the traits of brothers
should be more similar than cousins (horizontal brother-cousin relationship).
An effective model should be able to capture both of the vertical relationship
and horizontal relationship, so that the representation of any concept in the
latent space should be closer to its parent than uncles, and closer to
brothers than cousins. For our HaST-CW model shown in Fig. 3, a new HCW module
(Sec. III-C) is proposed to preserve the vertical relationship, and a novel SC
loss (Sec. III-D) is proposed to preserve the horizontal relationship.

Figure 3: The architecture of HaST-CW model.

###  III-C Hierarchical Concept Whitening

The hierarchical concept whitening (HCW) module is one of the key components
in the HaST-CW model, which aims to disentangle different label concepts while
preserving their underlying hierarchical relationship. Specifically, in this
work, the set of label concepts were denoted by $C=\\{C_{i}\\}_{i=1}^{N_{c}}$,
where $C_{i}$ represents the $i^{th}$ concept and $N_{c}=21$ is the number of
concepts. For $C_{i}$, its parent, children, brothers and cousins were denoted
as $C_{i.\mathcal{P}}$, $\\{C_{i.children}\\}$, $\\{C_{i.\mathcal{B}}\\}$ and
$\\{C_{i.\mathcal{C}}\\}$, respectively. A dataset is denoted as
$\mathcal{D}\\{\textbf{x}_{i},y_{i}\\}^{n}_{i=1}$. We use
$\textbf{X}^{C_{i}}=\\{\textbf{x}_{j}^{C_{i}}\\}_{j=1}^{n_{i}}$ to denote the
set of $i^{th}$-class samples labeled by $C_{i}$.

In traditional whitening transformation [34], during the training process,
data samples are first fed into the model in mini-batches to obtain the latent
representation matrix $\textbf{Z}_{d\times n}$, where $n$ is the mini-batch
size and $d$ is the dimension of latent representation. We use ResNet as the
model backbone in this work. Then a transformation $\psi$ is applied to
decorrelate and standardize $\textbf{Z}_{d\times n}$:

$\psi(\textbf{Z})=\textbf{W}(\textbf{Z}-\mu{\textbf{1}_{n\times 1}}^{T}),$ (1)

where $\textbf{W}_{d\times d}$ is the orthogonal whitening matrix, and
$\mu=\frac{1}{n}\sum^{n}_{i=1}\textbf{z}_{i}$ is the sample mean. A property
of representation whitening is that $\textbf{Q}^{\textbf{T}}\textbf{W}$ is
still a valid whitening matrix if Q is an orthogonal matrix. We leverage this
property for interpretable representation learning. In our model, besides
decorrelation and standardization, we expect that the transformed
representation of samples from concept $C_{i}$, namely
$\textbf{Q}^{\textbf{T}}\psi(\textbf{Z}^{C_{i}})$, can align well with the
$i^{th}$ axis of latent space. Meanwhile, the underlying hierarchical
relationship of concepts should also be preserved in their latent
representations. That is, we need to find an orthogonal matrix
$\textbf{Q}=[\textbf{q}_{1},\textbf{q}_{2},\dots,\textbf{q}_{N_{c}}]$ with two
requirements: (1) $\textbf{Z}^{C_{i}}$ should be most activated by
$\textbf{q}_{i}$, i.e., the $i^{th}$ column of Q; (2) $\textbf{Z}^{C_{i}}$
should also be activated by $\\{\textbf{q}_{c}\\}$, where
$c\in\\{C_{i.children}\\}$ is the child of concept $C_{i}$. The first
constraint makes the representation align together with the corresponding
concept dimension, and the second one maintains the vertical parent-child
relationship between concepts. To this end, the optimization problem can be
formulated as:

$\displaystyle\max_{\textbf{q}_{1},\dots\textbf{q}_{N_{c}}}$
$\displaystyle\sum^{N_{c}}_{i=1}[{\frac{1}{n_{i}}\textbf{q}^{T}_{i}\psi(\textbf{Z}^{C_{i}})\textbf{1}_{n_{i}\times
1}+}$ $\displaystyle\sum_{c\in\\{C_{i.children}\\}}{\frac{1}{n_{i}\times
N_{cd}}({\textbf{q}_{c}})^{T}\psi(\textbf{Z}^{C_{i}})\textbf{1}_{n_{i}\times
1}}],$ (2) $\displaystyle s.t.\quad\textbf{Q}^{T}\textbf{Q}=\textbf{I}_{d},$

where $N_{cd}=|\\{C_{i.children}\\}|$ is the number of child concepts of
$C_{i}$. To solve this optimization problem with the orthogonality constraint,
a gradient descent method with the curvilinear search algorithm [44] is
adopted. With the whitening matrix W and rotation orthogonality matrix Q,
HaST-CW can replace any batch normalization layer in deep neural networks. The
details of representation whitening for HaST-CW is summarized in Algorithm 1.

The overall training pipeline of our HaST-CW model is shown in Algorithm 2. We
adopt an alternative training scheme. In the first stage, the deep neural
network is trained with the traditional classification loss. In the second
stage, we solve for Q to align representation dimension with semantic
concepts. The two stages work alternatively during the training process. The
classification loss of the first stage is defined as:

$\min_{\theta,\omega,\textbf{W},\mu,}\frac{1}{m}\sum^{m}_{i=1}\ell(g(\textbf{Q}^{T}\psi(\Phi(\textbf{x}_{i};\theta);\textbf{W},\mu);\omega);y_{i}),$
(3)

where $\Phi(\cdot)$ and $g(\cdot)$ are layers before and after the HaST-CW
module parameterized by $\theta$ and $\omega$, respectively. $\psi(\cdot)$ is
the whitening transformation parameterized by the sample mean $\mu$ and
whitening matrix W. The rotation orthogonal matrix Q will be updated according
to Sec. III-C in the second stage. The operation of
$\textbf{Q}^{T}\psi(\cdot)$ forms the HCW module. During the first training
stage, Q will be fixed and other parameters ($\theta,\omega,\textbf{W},\mu$)
will be optimized according to Eq. 3 to minimize the classification error. The
first stage will take $T_{thre}$ mini batches (we set $T_{thre}=30$ in
experiments). After that, Q will be updated by the Cayley transform [44]:

$\textbf{Q}^{\prime}=(\textbf{I}+\frac{\eta}{2}\textbf{A})^{-1}(\textbf{I}-\frac{\eta}{2}\textbf{A})\textbf{Q},$
(4) $\textbf{A}=\textbf{G}\textbf{Q}^{T}-\textbf{Q}{\textbf{G}}^{T},$ (5)

where A is a skew-symmetric matrix. G is the gradient of the concept alignment
loss, which is defined in Algorithm 2. $\eta$ is the learning rate. At the end
of the second stage, an updated $\textbf{Q}^{\prime}$ will participate in the
first training stage of the next iteration.

Algorithm 1 Forward Pass of HCW Module

1: Input: mini-batch input $\textbf{Z}\in\mathbb{R}^{d\times n}$

2: Optimization Variables: orthogonal matrix $\textbf{Q}\in\mathbb{R}^{d\times
d}$

3: Output: whitened representations $\hat{\textbf{Z}}\in\mathbb{R}^{d\times
n}$

4: The batch mean: $\mu=\frac{1}{n}\textbf{Z}\cdot\textbf{1}$

5: The centered representations:
$\textbf{Z}_{\textbf{C}}=\textbf{Z}-\mu\cdot\textbf{1}^{T}$

6: Calculate ZCA-whitening matrix W

7: Calculate the whitened representation:
$\hat{\textbf{Z}}=\textbf{Q}^{T}\textbf{W}\textbf{Z}_{\textbf{C}}$

Algorithm 2 The Overall Framework of HaST-CW

1: Input: Training dataset
$\mathcal{D}_{T}=\\{\textbf{x}_{i},y_{i}\\}^{n}_{i=1}$, Concept datasets
$\mathcal{D}_{C}=\;\\{\textbf{X}^{C_{1}}$, $\textbf{X}^{C_{2}}$, …,
$\textbf{X}^{C_{N_{c}}}\\}$

2: Optimization Variables:
$\textbf{W},\textbf{Q},\theta,\mu,\omega,\textbf{G}=[\textbf{g}_{1},\textbf{g}_{2},\dots,\textbf{g}_{N_{c}}]$

3: Hyperparameters: $\beta,\eta$

4: for $t=1\;to\;T$ do

5: Randomly sample a mini-batch$\\{\textbf{x}_{i},y_{i}\\}^{n}_{i=1}$from
$\mathcal{D}_{T}$

6: Do one step of SGD w.r.t $\theta$ and $\omega$ on the loss
$\frac{1}{n}\sum^{n}_{i=1}\ell(g(\textbf{Q}^{T}\psi(\Phi(\textbf{x}_{i};\theta);\textbf{W},\mu);\omega);y_{i})$

7: Update W and $\mu$ by exponential moving average

8: if $t$ mod $T_{thre}$ = 0 then

9: for $i\in\\{1,2,\dots,N_{c}\\}$ do

10: Sample mini-batches $\\{\textbf{x}_{j}^{C_{i}},y_{j}\\}^{n_{i}}_{j=1}$
from $\mathcal{D}_{C}$

11:
$\textbf{g}_{i}=-\frac{1}{n_{i}}\sum^{n_{i}}_{j=1}\psi(\Phi(\textbf{x}^{C_{i}}_{j};\theta);\textbf{W},\mu)$

12: $C_{child}\in\\{C_{i.children}\\}$,and $child\in\\{1,2,\dots,N_{c}\\}$

13: $N_{child}=|\\{C_{i.children}\\}|$

14: for $child$ do

15: Sample mini-batches
$\\{\textbf{x}_{j}^{C_{child}},y_{j}\\}^{n_{child}}_{j=1}$

16: $\textbf{g}_{child}=-\frac{1}{n_{child}\times
N_{child}}\sum^{n_{child}}_{j=1}\psi(\Phi(\textbf{x}^{C_{child}}_{j};\theta);W,\mu)$

17: end for

18: end for

19: end if

20: end for

###  III-D Semantic Constraint Loss

Besides preserving the vertical parent-child relationship of concepts, we
further model the horizontal relation between concepts that are at the same
hierarchy level (i.e., brothers or cousins). Different from the HCW in Sec.
III-C that focuses on concept alignment, here we directly control the distance
between representations of different concepts with the horizontal relation
[45, 46]. To this end, we propose a Semantic Constraint (SC) loss to model the
horizontal brother-cousin relationship as below:

$\displaystyle\quad\quad\quad{\mathcal{L}}_{SC}={\alpha\mathcal{L}}_{\mathcal{B}}+{\beta\mathcal{L}}_{\mathcal{C}},$
(6) $\displaystyle{\mathcal{L}}_{\mathcal{B}}=\sum_{j}$
$\displaystyle\sum_{{\mathcal{B}}_{i}\in{\\{C_{i.\mathcal{B}}\\}}}\sum_{k}max\\{0,m_{\mathcal{B}}-d({\textbf{z}}^{C_{i}}_{j},{\textbf{z}}^{\mathcal{B}_{i}}_{k})\\},$
$\displaystyle{\mathcal{L}}_{\mathcal{C}}=\sum_{j}$
$\displaystyle\sum_{{\mathcal{B}}_{i}\in{\\{C_{i.\mathcal{B}}\\}}}\sum_{{\mathcal{C}}_{i}\in{\\{C_{i.\mathcal{C}}\\}}}\sum_{k}\sum_{l}max\\{0,d({\textbf{z}}^{C_{i}}_{j},{\textbf{z}}^{\mathcal{B}_{i}}_{k})$
$\displaystyle-d$
$\displaystyle({\textbf{z}}^{C_{i}}_{j},{\textbf{z}}^{\mathcal{C}_{i}}_{l})+m_{\mathcal{C}}\\}.$

There are two components in the SC loss and their contributions are controlled
by two hyperparameters – $\alpha$ and $\beta$. The first term
${\mathcal{L}}_{\mathcal{B}}$ is a contrastive loss, which takes a pair of
image representations labeled by two brother concepts as input and enlarges
the distance between them. It uses a hyperparameter $m_{\mathcal{B}}$ to
control the distance. The distance between two concepts increases when
$m_{\mathcal{B}}$ is set larger.
${\mathcal{B}}_{i}\in{\\{C_{i.\mathcal{B}}\\}}$ denotes one of the brothers of
concept $C_{i}$. The second term ${\mathcal{L}}_{\mathcal{C}}$ is a triplet
loss. It takes three inputs: the anchor image representation
${\textbf{z}}^{C_{i}}_{j}$, the image representation
${\textbf{z}}^{\mathcal{B}_{i}}_{k}$ labeled by brother concept of the anchor,
and the image representation ${\textbf{z}}^{\mathcal{C}_{i}}_{l}$ labeled by
cousin concept of the anchor. ${\mathcal{C}}_{i}\in{\\{C_{i.\mathcal{C}}\\}}$
denotes the cousins of concept $C_{i}$. The triplet loss encourages the
anchor-brother distance to be smaller compared with the anchor-cousin distance
in representation space. In this way, the distance of image representations
from brother classes tends to be smaller than the distance of image
representations from cousin classes. The gap between the two types of distance
is controlled by the margin value $m_{\mathcal{C}}$. Consequently, the
hierarchical concept whitening module, together with the SC loss, enables the
latent representations of concepts with similar semantics to be close with
each other in the latent space.

###  III-E Latent Feature Maps Activation

The proposed HaST-CW model can generate latent representations
($\hat{\textbf{z}}_{i}$) for input images ($\textbf{x}_{i}$) at each neural
network layer by
$\hat{\textbf{z}}_{i}=\textbf{Q}^{T}\psi(\Phi(\textbf{x}_{i};\theta);\textbf{W},\mu)$.
The latent representation can be used to assess the interpretability of the
learning process by measuring the degree of activation of
$\hat{\textbf{z}}_{i}$ at different concept dimensions (i.e.
$\\{\textbf{q}_{i}\\}$). In the implementation, $\Phi(\cdot)$ is a CNN based
deep network, whose convolution output
$\textbf{z}_{i}=\Phi(\textbf{x}_{i};\theta)$ is a tensor with the dimension
$\textbf{z}_{i}\in R^{d\times h\times w}$. Since $\hat{\textbf{z}}_{i}$ is
calculated by $\hat{\textbf{z}}_{i}=\textbf{Q}^{T}\psi(\textbf{z}_{i})$ where
$\textbf{Q}^{T}\in R^{d\times d}$, we obtain $\hat{\textbf{z}}_{i}\in
R^{d\times h\times w}$, where $d$ is the channel dimension and $h\times w$ is
the feature map dimension. The hierarchical concept whitening operation
$\textbf{Q}^{T}\psi(\cdot)$ is conducted upon the $d$ feature maps. Therefore,
different feature maps contain the information of whether and where the
concept patterns exist in the image. However, as a tensor the feature map
cannot directly measure the degree of concept activation. To solve this
problem and at the same time to reserve both of the high-level and low-level
information, we first apply the max pooling on the feature map and then use
the mean value of the downsteam feature map to represent the original one. By
this way, we reshape the original feature map $\textbf{z}_{i}\in R^{d\times
h\times w}$ to $\textbf{z}_{i}^{\prime}\in R^{d\times 1}$. Finally,
$\textbf{z}_{i}^{\prime}$ is used to measure the activation of image
$\textbf{x}_{i}$ at each concept dimension.

##  IV Experiments

In the experiments section, we first visually demonstrate how our method can
effectively learn and hierarchically organize concepts in the latent space
(Sec. IV-B). We also show that (Sec. IV-C), compared to existing concept
whitening methods, HaST-CW not only separates concepts, but also can separate
groups of semantically related concepts in the latent space. After that, we
discuss the advantages offered by our method with quantitative results and
intuitive examples (Sec. IV-D) compared with baselines, including the CW
module and ablated versions of our method.

###  IV-A Experimental Setting

####  IV-A1 Data Preparation

In this work, we use a newly collected and curated Agri-ImageNet dataset to
evaluate the proposed HaST-CW model. In total, 9173 images from 21 classes are
used in our experiments. Each image is labeled with the class at the highest
possible hierarchy level. For example, an image of Melrose apple will be
labeled as ”Melrose” rather than the superclass ”Apple”. Then we divide images
per class into three parts by 60%/20%/20% for a standardized
training/validation/test splitting. Because the resolution of the original
images can range from 300 to 5000, we adopt the following steps to normalize
the image data: 1) we first lock aspect ratio and resize the images to make
the short edge to be 256; 2) During each training epoch, the images in the
training and validation datasets are randomly cropped into 224×224; 3) During
testing process, images in the test dataset are center cropped to be of size
224×224; 4) After cropping, the pixel values of images are normalized to
[0,1]. Then, the whole training dataset is divided into two parts
($\mathcal{D}_{T}$ and $\mathcal{D}_{C}$ in Algorithm 2). $\mathcal{D}_{C}$ is
the concept dataset used to update the matrix Q in the second stage (Eq. 4).
It is created by randomly selecting 64 images from each class in the training
dataset. The remaining images in the training dataset $\mathcal{D}_{T}$ are
used in the first stage to train the model parameters (Eq. 3).

####  IV-A2 Model Setting

In this work, we use several ResNet structures [47] to extract features from
images, including ResNet18 and ResNet50. During the training process, the two-
stage training scheme adopts a 30-to-1 ratio to alternatively train the whole
framework. In this case, after 30 mini batches of continuous training, the
model will pause and the rotation orthogonal matrix Q will be optimized at the
next mini batch. Two hyper-parameters $\alpha$ and $\beta$ in the SC loss are
set to be 1.0. Adam optimizer is used to train the whole model with a learning
rate of 0.1, a batch size of 64, a weight decay of 0.01, and a momentum rate
of 0.9.

###  IV-B Visualization of Semantic Map

To illustrate the learned semantic hierarchical structure, we show the
representations extracted from the latent hidden layer of all the samples in
Figure 4. For better visualization, we use Uniform Manifold Approximation and
Projection (UMAP) [48] to project the representations to a two-dimensional
space. All the images are color coded using the 17 sub-concepts which are
defined on the left of Figure 4. The top panel shows the result using CW
method. In general, all the concepts are assembled as small groups, but
neither semantic relations nor hierarchical structures have been learned. We
highlight the super-concept of “Weed” (black) and three sub-concepts ( “Apple
Golden” - green, “Apple Fuji” - red and “Apple Melrose” - blue) in the right
column. We can see that the three types of apple (sub-concepts) are evenly
distributed along with other fruits samples. The bottom panel shows our HaST-
CW results. All the different concepts successfully keep their distinct
cluster patterns as CW result. After our two-stage training process to instill
the semantic and hierarchical knowledge, the three types of apple images have
been pulled together and form a new concept (“Apple” with orange circle) at a
higher level. Moreover, the newly learned concept of “Apple” simultaneously
possesses sufficient distance to “Weed” (different super-concept) and
maintains relatively close relations to “Strawberry”, “Orange”, “Mango” as
well as other types of “Fruit”. This result demonstrates the effectiveness of
our hierarchical semantic concept learning framework, without negatively
affecting the overall classification performance.

Figure 4: UMAP visualization of the latent space embedding with Agri-ImageNet
images, colored according to the legend of image labels on the left. The top
panel shows the results of the CW method and we highlight the super-concept
“Weed” (black) and three sub-concepts. As shown in the bottom panel, we apply
the same rules to the output of HaST-CW and visualize the results. In
addition, we draw an orange circle that encapsulates three types of apples to
represent the super-concept “Apple”.

###  IV-C Efficiency and Accuracy of Concept Alignment

In this section, we compare the learning efficiency and accuracy of the
proposed HaST-CW with that of the conventional CW method. We track the
alignment between image representations and their corresponding concepts at
each layer. Specifically, we randomly select six concepts, and for each
concept we sort and select the top five images whose representations show the
strongest activation at the corresponding concept axis. We show the results at
both shallow and deep layers (layer 4 vs. layer 8) in Figure 5. From the
results of layer 4 (the left column) we can see that most of the top five
images obtained by conventional CW (the rows marked by green box) are
mismatched with the corresponding concepts. For example, the five images under
the concept of “Apple-Melrose” obtained by CW are from the “Weed” class. The
five images under the concept of “Snake Weed” are actually from other subclass
of “Weed”. Moreover, this situation continues in the following layers and has
not been changed until layer 8. On the contrary, with the help of our designed
semantic constraint loss, our HaST-CW (the rows marked by orange boxes) can
learn the intrinsic concept faster and achieves the best performance at an
earlier training stage (e.g., at a shallow layer). This result demonstrates
that by paralleling multiple HCW layers the proposed HaST-CW model can capture
the high-level features more efficiently.

Figure 5: Top 5 activation images of each concept. The image panel is divided
into two sets of columns: the left set of columns contains the results of
layer 4 (a shallow layer), whereas the right set of columns holds the results
of layer 8 (a deeper layer). Each concept covers two rows that correspond to
the results of the conventional CW (marked by green boxes) and the proposed
HaST-CW (marked by orange boxes), respectively.

To further demonstrate the alignment between images and the corresponding
concepts, we project each image in the test dataset into a latent space where
each concept can be represented by an axis. To visualize the alignments at
different concept hierarchies (Figure 2), we show three pairs of concepts
which belong to different hierarchical levels as examples: “Apple-
Melrose”-“Apple-Fuji” is from hierarchy 3 (H-3), “Snake Weed”-“Parkinsonia” is
from hierarchy 2 (H-2), and “Weed”-“Apple” crosses hierarchies 1 and 2
(“Weed”: H-1, “Apple”: H-2). Within each concept pair, a two-dimensional space
has been built by taking the two concepts as axes. Thus, each image can be
mapped into the space by calculating the similarity between image
representation and the two concept representations. The results are shown in
Figure 6. Different rows correspond to different methods and the concept axes
(space) are defined at the bottom.

Figure 6: Data distribution in the concept latent space. Three pairs of
concepts corresponding to different semantic hierarchy levels are selected.
For each concept pair, a two-dimensional space is built by taking the concepts
as axes. To visualize the alignments between images and the concepts, the
images are projected into the two-dimensional space by similarity values
between image representations and the two concept representations. Different
rows in the figure panel correspond to different methods and the concept axes
(space) are defined at the bottom.

The first column of Figure 6 shows the data distribution in the two-
dimensional space of “Apple-Melrose”-“Apple-Fuji” concept pair. The images
belonging to Apple-Melrose class should have the highest similarity with the
concept of “Apple-Melrose”, and thereby they should be located at the right-
bottom corner. Similarly, the images of Apple-Fuji class should be located at
the left-top corner. The other images should distribute in the space according
to the similarity with the two concepts. For example, compared to images of
fruit-related classes, images of weed-related classes will have lower semantic
similarity with the two concepts, so they should locate near the origin point
(left-bottom corner). As shown in the first column, the two models which adopt
the HaST-CW method (the second and third rows) can better follow the above-
mentioned patterns. While in the CW model (the first row), nearly all the
images are gathered at the right-bottom corner. This may be due to the high
similarity between the two concepts considered, since they share the same
super-class of “Apple”. As a result, CW model may be limited in distinguishing
different classes with high semantic similarity. A similar situation happens
in the second column with the concept pair of “Snake Weed”-“Parkinsonia”.
These results suggest that compared to CW method, HaST-CW can better capture
the subtle differences of semantic-related classes.

The third column shows the results of the concept pair of two super-classes:
“Weed” and “Apple”. As each of the super-class concept contains multiple sub-
classes, the intra-class variability is greater. Our proposed HaST-CW,
together with the SC loss (the third row), can effectively capture the common
visual features and project the “Weed” and “Apple” images to the left-top and
right-bottom, respectively. At the same time, the images belonging to
different sub-classes under “Weed” and “Apple” are assembled as blocks instead
of scattered along the diagonal line. In the other two methods, especially in
the CW method (the first row), the images of “Weed” class spread out over a
wide range along the vertical axis. This result suggests that the proposed
HaST-CW with SC loss can effectively model both the inter- and intra- class
similarity.

###  IV-D Interpretable Image Classification

In this section, we compare the classification performance of the proposed
HaST-CW method and the SC loss function with the conventional CW method using
different backbones: ResNet18 and ResNet50. The results are summarized in
Table I. Different rows correspond to different model settings. Within each
model setting, we repeat the experiments for five times to reduce the effect
of random noise. The mean and variance of accuracy (ACC.) are reported in the
fourth column. From the results, we can see that the classification
performance is slightly better than the other three model settings. This
result indicates that the proposed HaST-CW model can improve the
interpretability without hurting predictive performance.

TABLE I:  Comparison of Classification Performance.
Module | Backbone | Loss | Acc.
---|---|---|---
CW | ResNet18 | ${\mathcal{L}}_{CE}$ |  63.48 $\pm$ 0.68
CW | ResNet50 | ${\mathcal{L}}_{CE}$ |  69.25 $\pm$ 3.93
HaST-CW | ResNet50 | ${\mathcal{L}}_{CE}$ |  69.30 $\pm$ 3.75
HaST-CW | ResNet50 | ${\mathcal{L}}_{CE}+{\mathcal{L}}_{SC}$ |  69.49 $\pm$ 3.20

To track and visualize the classification process, we randomly select two
images from Apple-Melrose class and Snake Weed class. The activation values
between each image with the six relevant concepts are calculated and
normalized to [0, 1]. The images, concepts and activation values are organized
into a hierarchical activation tree. The results are shown in Figure 7. We
could observe that the activation values of each image correctly represent the
semantic relationship between the images and the concepts. For example, in
Figure 7 (a), the image located at the root is from Snake Weed class which is
a subclass of Weed. The activation values of the image are consistent with
this relationship and possess the highest activation values on the two
concepts – “Weed” and “Snake Weed”.

Figure 7: Hierarchical activation tree. We randomly select two images from the
Apple-Melrose class and the Snake Weed class. For each image, activation
values corresponding to the 6 concepts are calculated and normalized to [0,
1]. The highest activation values are highlighted with red along the
hierarchical path.

##  V Conclusion and Future Work

In this study, we propose a new HaST-CW and demonstrate its superiority over
Concept Whitening [34]. HaST-CW decorrelates representations in the latent
space and aligns concepts with corresponding dimensions. In addition, it
correctly groups concepts at different granularity levels in the latent space
and preserves hierarchical structures of concepts of interest. By doing so, we
can interpret concepts better and observe the semantic relationships among
concepts.

We believe there are many possibilities for future work. One promising
direction is automatically learning concepts from data. In this scenario, we
can jointly learn possible concepts from common abstract features among images
and how to represent these learned concepts in the latent space. For example,
it might be possible to develop unsupervised or weakly-supervised methods to
automatically learn the concept tree from data. By jointly learning concepts,
their representations, and relations, the model may discover more data-driven
semantic structures.

HaST-CW can also be extended with post-hoc interpretability strategies (such
as saliency-based methods that highlight focused areas used for
classification). Such explanations at the concept level can provide a more
global view of model behaviors.

In addition, while this work focuses on the the natural image domain, the idea
of leveraging hierarchical knowledge to guide representation learning is
generalizable to other domains such as natural language processing [22, 49,
50] and medical image analysis [51, 52, 53, 54, 55, 56, 57, 58, 59]. Exploring
knowledge-infused learning in different domains [60, 61, 62, 63, 64, 65, 66]
and tasks [67, 68, 69, 70, 71], including innovative applications [72, 73,
74], is an interesting future direction.

In conclusion, as deep learning models become increasingly complex, model
interpretability is crucial for understanding behaviors, gaining trust, and
enabling human-AI collaboration. Our work complements previous work and lays a
solid foundation for further exploration.

## References

  * [1] M. Du, N. Liu, and X. Hu, “Techniques for interpretable machine learning,” _Communications of the ACM_ , 2019.
  * [2] M. De Clercq, A. Vats, and A. Biel, “Agriculture 4.0: The future of farming technology,” _Proceedings of the World Government Summit, Dubai, UAE_ , pp. 11–13, 2018.
  * [3] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu, “Interpretable machine learning: definitions, methods, and applications,” _arXiv preprint arXiv:1901.04592_ , 2019.
  * [4] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang, “Concept bottleneck models,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5338–5348.
  * [5] C. Olah, A. Mordvintsev, and L. Schubert, “Feature visualization,” _Distill_ , vol. 2, no. 11, p. e7, 2017.
  * [6] B. Zhou, D. Bau, A. Oliva, and A. Torralba, “Interpreting deep visual representations via network dissection,” _IEEE transactions on pattern analysis and machine intelligence_ , 2018.
  * [7] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas _et al._ , “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),” in _ICML_ , 2018.
  * [8] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim, “Towards automatic concept-based explanations,” in _NeurIPS_ , 2019.
  * [9] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” _Nature Machine Intelligence_ , vol. 1, no. 5, pp. 206–215, 2019.
  * [10] J. Wang, J. Oh, H. Wang, and J. Wiens, “Learning credible models,” in _KDD_ , 2018.
  * [11] C.-H. Chang, G. A. Adam, and A. Goldenberg, “Towards robust classification model by counterfactual and invariant data generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [12] A. S. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients,” in _Thirty-second AAAI conference on artificial intelligence_ , 2018.
  * [13] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework.” _ICLR_ , 2017.
  * [14] R. T. Chen, X. Li, R. Grosse, and D. Duvenaud, “Isolating sources of disentanglement in vaes,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2019, pp. 2615–2625.
  * [15] L. Huang, D. Yang, B. Lang, and J. Deng, “Decorrelated batch normalization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 791–800.
  * [16] S. Rezayi, H. Dai, Z. Liu, Z. Wu, A. Hebbar, A. H. Burns, L. Zhao, D. Zhu, Q. Li, W. Liu _et al._ , “Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2022, pp. 269–278.
  * [17] S. Rezayi, Z. Liu, Z. Wu, C. Dhakal, B. Ge, C. Zhen, T. Liu, and S. Li, “Agribert: knowledge-infused agricultural language models for matching food and nutrition,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence_ , vol. 7, 2022, pp. 5150–5156.
  * [18] L. Zhang, L. Wang, and D. Zhu, “Representing alzheimer’s disease progression via deep prototype tree,” _arXiv preprint arXiv:2102.06847_ , 2021.
  * [19] ——, “Jointly analyzing alzheimer’s disease related structure-function using deep cross-model attention network,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 563–567.
  * [20] L. Wang, L. Zhang, and D. Zhu, “Learning latent structure over deep fusion model of mild cognitive impairment,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 1039–1043.
  * [21] H. Dai, Y. Li, Z. Liu, L. Zhao, Z. Wu, S. Song, Y. Shen, D. Zhu, X. Li, S. Li _et al._ , “Ad-autogpt: An autonomous gpt for alzheimer’s disease infodemiology,” _arXiv preprint arXiv:2306.10095_ , 2023.
  * [22] S. Rezayi, Z. Liu, Z. Wu, C. Dhakal, B. Ge, H. Dai, G. Mai, N. Liu, C. Zhen, T. Liu _et al._ , “Exploring new frontiers in agricultural nlp: Investigating the potential of large language models for food applications,” _arXiv preprint arXiv:2306.11892_ , 2023.
  * [23] M. T. Ribeiro, S. Singh, and C. Guestrin, “Why should i trust you?: Explaining the predictions of any classifier,” in _KDD_ , 2016.
  * [24] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” _arXiv preprint arXiv:1706.03825_ , 2017\.
  * [25] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017.
  * [26] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” _PloS one_ , 2015.
  * [27] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in _NIPS_ , 2017.
  * [28] A. Ghorbani, A. Abid, and J. Zou, “Interpretation of neural networks is fragile,” in _AAAI_ , 2019.
  * [29] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” _arXiv preprint arXiv:1312.6034_ , 2013.
  * [30] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _European conference on computer vision_. Springer, 2014, pp. 818–833.
  * [31] A. Bau, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass, “Identifying and controlling important neurons in neural machine translation,” in _International Conference on Learning Representations_ , 2018.
  * [32] J. Mu and J. Andreas, “Compositional explanations of neurons,” _arXiv preprint arXiv:2006.14032_ , 2020.
  * [33] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” _Nature Machine Intelligence_ , vol. 1, no. 5, pp. 206–215, 2019.
  * [34] Z. Chen, Y. Bei, and C. Rudin, “Concept whitening for interpretable image recognition,” _Nature Machine Intelligence_ , vol. 2, no. 12, pp. 772–782, 2020.
  * [35] C. Chen, O. Li, C. Tao, A. J. Barnett, J. Su, and C. Rudin, “This looks like that: deep learning for interpretable image recognition,” _arXiv preprint arXiv:1806.10574_ , 2018.
  * [36] O. Li, H. Liu, C. Chen, and C. Rudin, “Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [37] L. Huang, D. Yang, B. Lang, and J. Deng, “Decorrelated batch normalization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 791–800.
  * [38] A. Siarohin, E. Sangineto, and N. Sebe, “Whitening and coloring batch transform for gans,” _arXiv preprint arXiv:1806.00420_ , 2018.
  * [39] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [40] M. Cogswell, F. Ahmed, R. Girshick, L. Zitnick, and D. Batra, “Reducing overfitting in deep networks by decorrelating representations,” _arXiv preprint arXiv:1511.06068_ , 2015.
  * [41] K. K. Pal and K. Sudeep, “Preprocessing for image classification by convolutional neural networks,” in _2016 IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT)_. IEEE, 2016, pp. 1778–1781.
  * [42] P. Luo, “Learning deep architectures via generalized whitened neural networks,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2238–2246.
  * [43] G. Desjardins, K. Simonyan, R. Pascanu, and K. Kavukcuoglu, “Natural neural networks,” _arXiv preprint arXiv:1507.00210_ , 2015.
  * [44] Z. Wen and W. Yin, “A feasible method for optimization with orthogonality constraints,” _Mathematical Programming_ , vol. 142, no. 1, pp. 397–434, 2013.
  * [45] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 539–546.
  * [46] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 815–823.
  * [47] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [48] L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” _arXiv preprint arXiv:1802.03426_ , 2018.
  * [49] Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu _et al._ , “Summary of chatgpt/gpt-4 research and perspective towards the future of large language models,” _arXiv preprint arXiv:2304.01852_ , 2023.
  * [50] H. Cai, W. Liao, Z. Liu, X. Huang, Y. Zhang, S. Ding, S. Li, Q. Li, T. Liu, and X. Li, “Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training,” _arXiv preprint arXiv:2211.02849_ , 2022.
  * [51] X. Li, L. Zhang, Z. Wu, Z. Liu, L. Zhao, Y. Yuan, J. Liu, G. Li, D. Zhu, P. Yan _et al._ , “Artificial general intelligence for medical imaging,” _arXiv preprint arXiv:2306.05480_ , 2023.
  * [52] X.-A. Bi, K. Chen, S. Jiang, S. Luo, W. Zhou, Z. Xing, L. Xu, Z. Liu, and T. Liu, “Community graph convolution neural network for alzheimer’s disease classification and pathogenetic factors identification,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [53] Z. Liu, M. He, Z. Jiang, Z. Wu, H. Dai, L. Zhang, S. Luo, T. Han, X. Li, X. Jiang _et al._ , “Survey on natural language processing in medical image analysis.” _Zhong nan da xue xue bao. Yi xue ban= Journal of Central South University. Medical Sciences_ , vol. 47, no. 8, pp. 981–993, 2022\.
  * [54] J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y. Liu, S. Zhang _et al._ , “Review of large vision models and visual prompt engineering,” _arXiv preprint arXiv:2307.00855_ , 2023.
  * [55] L. Zhang, Z. Liu, L. Zhang, Z. Wu, X. Yu, J. Holmes, H. Feng, H. Dai, X. Li, Q. Li _et al._ , “Segment anything model (sam) for radiation oncology,” _arXiv preprint arXiv:2306.11730_ , 2023.
  * [56] S. Zhang, E. Shi, L. Wu, R. Wang, S. Yu, Z. Liu, S. Xu, T. Liu, and S. Zhao, “Differentiating brain states via multi-clip random fragment strategy-based interactive bidirectional recurrent neural network,” _Neural Networks_ , 2023\.
  * [57] N. Qiang, Q. Dong, J. Gao, J. Li, S. Zhang, H. Liang, Y. Sun, B. Ge, Z. Liu, Z. Wu _et al._ , “A deep learning method for autism spectrum disorder identification based on interactions of hierarchical brain networks,” _Available at SSRN 4309357_.
  * [58] H. Dai, Q. Li, L. Zhao, L. Pan, C. Shi, Z. Liu, Z. Wu, L. Zhang, S. Zhao, X. Wu _et al._ , “Graph representation neural architecture search for optimal spatial/temporal functional brain network decomposition,” in _International Workshop on Machine Learning in Medical Imaging_. Springer, 2022, pp. 279–287.
  * [59] Y. Liu, E. Ge, M. He, Z. Liu, S. Zhao, X. Hu, D. Zhu, T. Liu, and B. Ge, “Discovering dynamic functional brain networks via spatial and channel-wise attention,” _arXiv preprint arXiv:2205.09576_ , 2022.
  * [60] L. Zhao, L. Zhang, Z. Wu, Y. Chen, H. Dai, X. Yu, Z. Liu, T. Zhang, X. Hu, X. Jiang _et al._ , “When brain-inspired ai meets agi,” _arXiv preprint arXiv:2303.15935_ , 2023.
  * [61] J. Holmes, Z. Liu, L. Zhang, Y. Ding, T. T. Sio, L. A. McGee, J. B. Ashman, X. Li, T. Liu, J. Shen _et al._ , “Evaluating large language models on a highly-specialized topic, radiation oncology physics,” _arXiv preprint arXiv:2304.01938_ , 2023.
  * [62] Z. Liu, X. He, L. Liu, T. Liu, and X. Zhai, “Context matters: A strategy to pre-train language model for science education,” _arXiv preprint arXiv:2301.12031_ , 2023.
  * [63] C. Ma, Z. Wu, J. Wang, S. Xu, Y. Wei, Z. Liu, L. Guo, X. Cai, S. Zhang, T. Zhang _et al._ , “Impressiongpt: an iterative optimizing framework for radiology report summarization with chatgpt,” _arXiv preprint arXiv:2304.08448_ , 2023.
  * [64] Z. Liu, A. Zhong, Y. Li, L. Yang, C. Ju, Z. Wu, C. Ma, P. Shu, C. Chen, S. Kim _et al._ , “Radiology-gpt: A large language model for radiology,” _arXiv preprint arXiv:2306.08666_ , 2023.
  * [65] Y. Ding, H. Feng, Y. Yang, J. Holmes, Z. Liu, D. Liu, W. W. Wong, N. Y. Yu, T. T. Sio, S. E. Schild _et al._ , “Deep-learning based fast and accurate 3d ct deformable image registration in lung cancer,” _Medical Physics_ , 2023.
  * [66] Y. Ding, Z. Liu, H. Feng, J. Holmes, Y. Yang, N. Yu, T. Sio, S. Schild, B. Li, and W. Liu, “Accurate and efficient deep neural network based deformable image registration method in lung cancer,” in _MEDICAL PHYSICS_ , vol. 49, no. 6. WILEY 111 RIVER ST, HOBOKEN 07030-5774, NJ USA, 2022, pp. E148–E148.
  * [67] W. Liao, Z. Liu, H. Dai, S. Xu, Z. Wu, Y. Zhang, X. Huang, D. Zhu, H. Cai, T. Liu _et al._ , “Differentiate chatgpt-generated and human-written medical texts,” _arXiv preprint arXiv:2304.11567_ , 2023.
  * [68] Z. Liu, X. Yu, L. Zhang, Z. Wu, C. Cao, H. Dai, L. Zhao, W. Liu, D. Shen, Q. Li _et al._ , “Deid-gpt: Zero-shot medical text de-identification by gpt-4,” _arXiv preprint arXiv:2303.11032_ , 2023.
  * [69] W. Liao, Z. Liu, H. Dai, Z. Wu, Y. Zhang, X. Huang, Y. Chen, X. Jiang, D. Zhu, T. Liu _et al._ , “Mask-guided bert for few shot text classification,” _arXiv preprint arXiv:2302.10447_ , 2023.
  * [70] Z. Wu, L. Zhang, C. Cao, X. Yu, H. Dai, C. Ma, Z. Liu, L. Zhao, G. Li, W. Liu _et al._ , “Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task,” _arXiv preprint arXiv:2304.09138_ , 2023.
  * [71] T. Zhong, Y. Wei, L. Yang, Z. Wu, Z. Liu, X. Wei, W. Li, J. Yao, C. Ma, X. Li _et al._ , “Chatabl: Abductive learning via natural language interaction with chatgpt,” _arXiv preprint arXiv:2304.11107_ , 2023.
  * [72] H. Dai, C. Ma, Z. Liu, Y. Li, P. Shu, X. Wei, L. Zhao, Z. Wu, D. Zhu, W. Liu _et al._ , “Samaug: Point prompt augmentation for segment anything model,” _arXiv preprint arXiv:2307.01187_ , 2023.
  * [73] L. Zhang, J. M. Holmes, Z. Liu, S. A. Vora, T. T. Sio, C. E. Vargas, N. Y. Yu, S. R. Keole, S. E. Schild, M. Bues _et al._ , “Beam mask and sliding window-facilitated deep learning-based accurate and efficient dose prediction for pencil beam scanning proton therapy,” _arXiv preprint arXiv:2305.18572_ , 2023.
  * [74] H. Dai, Z. Liu, W. Liao, X. Huang, Z. Wu, L. Zhao, W. Liu, N. Liu, S. Li, D. Zhu _et al._ , “Chataug: Leveraging chatgpt for text data augmentation,” _arXiv preprint arXiv:2302.13007_ , 2023.