# Asymptotic Behavior of Multiplicative Spherical Integrals and S-transform

Jonathan Husson  Université de Lyon, ENSL, CNRS, France  Jonathan.Husson@ens-
lyon.fr

###### Abstract.

In this note, we study the asymptotics of a spherical integral that is a
multiplicative counterpart to the well-known Harish-Chandra Itzykson Zuber
integral. This counterpart can also be expressed in terms the Heckman-Opdam
hypergeometric function. When the argument of this spherical integral is of
finite support and of order $N$, these asymptotics involve a modified version
of the $S$-transform of the limit measure of the matrix argument and its
largest eigenvalue. To prove our main result, we are leveraging a technique of
successive conditionning used in [8]. In particular we prove in a
"mathematically rigorous" manner a result from [14] in the case $\beta=1,2$
and we generalize it for multiple arguments.

This work was supported in part by ERC Project LDRAM : ERC-2019-ADG Project
884584

##  1\. Introduction

If $A$ and $B$ are two real or complex self-adjoint $N\times N$ matrices, the
Harish-Chandra-Itzykson-Zuber integral is defined as:

$HCIZ(A,B):=\int_{\mathcal{U}_{N}^{\beta}}\exp(NTr(AUBU^{*}))dU$

where $dU$ is the Haar measure on either the orthogonal or the unitary group.
Introduced by Harish-Chandra in [10], and studied then by Itzykson and Zuber
in [13] who gave an explicit formula in the complex case, this integral has
proven to be an interesting object in a wide variety of fields, from algebraic
geometry [6, 15] to physics [4]. In random matrix theory, it is a powerful
tool when one looks at the large deviations of the spectrum. In [9] Guionnet
an Maida proved that when $A_{N}$ and $B_{N}$ are $N\times N$ matrices and
$B_{N}$ is a matrix of rank one, the asymptotical behavior of
$HCIZ(A_{N},B_{N})$ for large $N$ depends on the largest eigenvalue of $A_{N}$
and on the limit of its empirical measure through its $R$-transform. This
result has since been extended by Guionnet and the author in [8] to matrices
$B_{N}$ with finite rank. It has been instrumental in the proofs of large
deviations principles for the largest eigenvalue in various random matrix
models [7, 12, 1]. The HCIZ integrals are also related to Schur polynomials
and the theory of special functions. In this paper, we study a multiplicative
counterpart of the HCIZ integral that was comparatively less studied. This
integral is defined for a self-adjoint definite $N\times N$ positive matrix,
$X$ and $\bar{\theta}=(\theta_{1},...,\theta_{N})\in\mathbb{R}^{N}$ by:

$I_{N}(X,\bar{\theta})=\int_{\mathcal{U}_{N}^{\beta}}\prod_{i=1}^{N}\det([U^{*}XU]_{i}^{\theta_{i}-\theta_{i+1}})^{\frac{\beta
N}{2}}dU$

where $[A]_{i}$ is the top left $i\times i$ block of $A$. The "multiplicative"
denomination comes from the fact that if one takes two matrices $X$ and $Y$
and $U$ a Haar-distributed unitary matrix, one has:

$\int_{\mathcal{U}_{N}^{\beta}}I_{N}(Z(U),\bar{\theta})dU=I_{N}(X,\bar{\theta})I_{N}(Y,\bar{\theta}).$

where $Z(U)$ is a diagonal matrix with the same spectrum as $U^{*}XUY$. When
$\theta_{i}=\lambda_{i}$ with $\lambda$ a signature (that is a $N$-tuple of
integers such that $\lambda_{i}\geq...\geq\lambda_{N}$), this spherical
integral can be expressed in function of the Jack polynomials
$j_{\lambda}^{(\beta/2)}$. If we set $\textbf{x}=(x_{1},...,x_{N})$ the
eigenvalues of $X$ then we have:

$I_{N}(X,\lambda)=\frac{j_{\lambda}^{(\frac{\beta}{2})}(\textbf{x})}{j^{(\frac{\beta}{2})}_{\lambda}(\textbf{1}_{N})}$

where $\textbf{1}_{N}=(1,...,1)$. In particular, when $\beta=2$, we get back
the Schur polynomials since $j^{(1)}_{\lambda}=s_{\lambda}$. For arguments
that are not signatures, this spherical integral can also be expressed as a
shifted version of the Heckman-Opdam hypergeometric function
$\mathcal{F}^{(\beta)}_{\textbf{a}}(z)$ [11, 16]. Namely, we have that:

$I_{N}(X,\bar{\theta})=\mathcal{F}^{(\beta)}_{\ln\textbf{x}}(\bar{\theta}-\rho)$

where $\ln\textbf{x}=(\ln x_{1},...,\ln x_{N})$ and $\rho_{i}=\beta/2(i-1)$.
The asymptotics of Schur and Jack polynomials have been investigated in the
literature, (see for instance [5] for Schur polynomials and [2, 3] for Jack
polynomials and McDonald polynomials). However, our result deal with the
logarithm of this integral divided by $N$ when the argument $\bar{\theta}$ has
finite support whereas the preceding results of [5] deal with the case of
$\ln\textbf{x}$ having finite support. Another way to see this is to notice
that we are interested in limits of the form:

$\lim_{N\to\infty}\frac{1}{N}\ln\mathcal{F}^{(\beta)}_{\textbf{a}}\Big{(}\frac{\beta}{2}(\theta_{1}N,\theta_{2}N-1,...,\theta_{k}N-k+1,-k,...,-N+1)\Big{)}.$

Whereas previous results from [5, 8] deal with limits of a different
renormalization of the Heckman-Opdam function, that is:

$\lim_{N\to\infty}\frac{1}{N}\ln\frac{\mathcal{F}^{(\beta)}_{\textbf{a}}(\frac{\beta}{2}(\theta_{1}N,\theta_{2}N,..\theta_{k}N,0,0,...,0))}{\mathcal{F}^{(\beta)}_{\textbf{a}}(0,0,0,...,0)}.$

We direct the reader to [14] and the reference and the references therein for
more information on the links between the quantity $I_{N}$ and special
functions. In [14], Mergny and Potters study the asymptotics of these
integrals for a one-dimensional parameter $\bar{\theta}$ (that is a parameter
whose support is at most on the first coordinate). This study reveals a
modified version of the $S$-transform of the eigenvalue distribution and the
largest eigenvalue. They also conjecture that for finite-dimensional
parameters, the behavior of the log of this integral should be the sum of the
one-dimensional behaviors. In this note we prove again the result of Potters
and Mergny, (whose original proof make use of non-convergent contour integral)
with mathematically rigorous tools and we also address their conjecture. It
turns out that in this finite rank case, we have the same phenomenon as the
one described for the HCIZ integral in [8]. Asymptotically, the behavior of
the log is indeed the sum of the one-dimensional behaviors but where we pair
the $j$-th largest of the positive $\theta_{i}$ with the $j$-th largest
eigenvalue of $X$ and the $j$-th smallest of the negative $\theta_{i}$ with
the $j$-th smallest eigenvalue of $X$ (which correspond the the conjecture
formulated in [14] if the extremal eigenvalues stick to the support of the
limiting measure).

This proof follows the general philosophy of [8] and we will do a recursion on
the dimension of the argument $\bar{\theta}$. After the definition of the
multiplicative spherical integral and the statement of the result in section
2, we will initialize our recursion and prove the result in the one-
dimensional case in section 3. For this we will proceed then in two steps:
first, in subsection 3.1 we will discretize the spectrum of the matrix $X_{N}$
in order to treat a simpler, finite-dimensional large deviation problem
involving Dirichlet laws, and then in subsection 3.2, we will treat the
generic case using discrete approximations and continuity arguments. Then, in
section 4, we will assume that our theorem stands for the $k-1$-dimensional
case and we will prove it for the $k$-dimensional one. There again we proceed
first by discretization in subsection 4.1 and then approximation and
continuity in subsection 4.2. In subsection 4.1, we will use our recursion
hypothesis by conditioning the first column of $U$ and reducing the problem to
the study of another $k-1$-dimensional integral
$I_{N-1}(\tilde{\theta},Y_{N})$, where the spectrum of $Y_{N}$ is interlaced
with the spectrum of $X_{N}$. Here, the main difference with [8] is that we
need to take into account the fact that contrary to the HCIZ integral, this
integral is _not_ symmetrical in its argument $\theta$ which adds a bit of
technicality in subsection 4.1. Furthermore the interlacing relationship
between the spectra of $X_{N}$ and $Y_{N}$ will also be different.

###  1.1. Notations

We will denote $\mathcal{H}_{N}^{\beta}$ the set of symmetric matrices of
$\mathcal{M}_{N}(\mathbb{R})$ if $\beta=1$ and the set of Hermitian matrices
of $\mathcal{M}_{N}(\mathbb{C})$ if $\beta=2$. We also denote
$\mathcal{H}_{N}^{\beta,+}$ and $\mathcal{H}_{N}^{\beta,++}$ respectively the
positive and definite positive matrices of $\mathcal{H}_{N}^{\beta}$. We will
also denote $\mathcal{U}_{N}^{\beta}$ the orthogonal group if $\beta=1$ and
the unitary group when $\beta=2$. For a matrix $X$ in
$\mathcal{H}_{N}^{\beta}$ and $i\in\llbracket 1,N\rrbracket$, we will denote
$\lambda_{i}(X)$ its $i$-th eigenvalue, $\chi_{X}(t)=\det(t-X)$ its
characteristic polynomial and
$\mu_{X}:=N^{-1}\sum_{i=1}^{N}\delta_{\lambda_{i}(X)}$ its eigenvalue
distribution. For $n\in\mathbb{N}^{*}$, we denote $\mathbb{S}^{\beta n-1}$
either the real sphere of dimension $n$ or the complex sphere of dimension
$n$. We will also denote $\mathbb{R}^{+}$ the set of non-negative real numbers
and $\mathbb{R}^{+,*}$ the set $\mathbb{R}^{+}\setminus\\{0\\}$.

If $\mu$ is a probability measure on $\mathbb{R}$, we will denote $r(\mu)$ the
rightmost point of its support and $l(\mu)$ the leftmost point of its support.
We denote $G_{\mu}$ its Stieljes transform, that is, for $z\in\mathbb{C}$
outside of the support of $\mu$:

$G_{\mu}(z)=\int_{t\in\mathbb{R}}\frac{1}{z-t}d\mu(t).$

Furthermore if $z=r(\mu)$ or $z=l(\mu)$, we define $G_{\mu}(z)$ as the
following limits on the real line:

$G_{\mu}(r(\mu))=\lim_{z\to r(\mu)^{+}}G_{\mu}(z)\text{ and
}G_{\mu}(l(\mu))=\lim_{z\to l(\mu)^{-}}G_{\mu}(z).$

In particular, we allow $G_{\mu}(r(\mu))=+\infty$ and
$G_{\mu}(l(\mu))=-\infty$.

If, $\mu\in\mathcal{P}(\mathbb{R}^{+})$, we also denote
$T_{\mu}(z)=zG_{\mu}(z)-1$. We define $T_{\mu}(r(\mu))$ and $T_{\mu}(l(\mu))$
as we did for $G_{\mu}$. We also denote, for $\tilde{S}_{\mu}$ its modified
$S$-transform defined on
$[T_{\mu}(l(\mu)),T_{\mu}(r(\mu))]\setminus\\{0,-1\\}$ by:

$\tilde{S}_{\mu}(z)=\frac{z}{z+1}T_{\mu}^{-1}(z)$

where $T_{\mu}^{-1}$ is the inverse function of $T_{\mu}$ defined on
$[T_{\mu}(l(\mu)),T_{\mu}(r(\mu))]\setminus\\{0\\}$. If $l(\mu)>0$, doing a
Taylor expansion of $T_{\mu}^{-1}$ in $-1$ show that we can extend
$\tilde{S}_{\mu}$ analytically in $-1$ by $\tilde{S}_{\mu}(-1)=(\int
d\mu(t)/t)^{-1}$ . We can also prove that $\tilde{S}$ can be analytically
extended on $0$ by $\tilde{S}_{\mu}(0)=\int td\mu(t)$. Furthermore, derivating
$\tilde{S}$ gives that it is an increasing function on
$]T_{\mu}(l(\mu)),T_{\mu}(r(\mu))[$.

##  2\. Definitions and statement of the result

Given the preceding notations, we can define the multiplicative spherical
integral on the group $\mathcal{U}^{\beta}_{N}$.

######  Definition 2.1.

For $\beta=1\text{ or }2$, $X_{N}\in\mathcal{H}_{N}^{\beta,++}$, $k\leq N$.
$\overline{\theta}=(\theta_{1},\theta_{2},...,\theta_{k})\in\mathbb{R}^{k}$ :

$I_{N}(\overline{\theta},X_{N})=\int_{\mathcal{U}^{\beta}_{N}}\Delta_{\overline{\theta}}(U^{*}X_{N}U)^{\beta
N/2}dU$

where $dU$ is the Haar measure on $\mathcal{U}_{N}^{\beta}$ and
$\Delta_{a}(M)$ is defined for $M\in\mathcal{H}_{N}^{\beta,++}$ and
$a\in\mathbb{R}^{k}$ by

$\Delta_{a}(M)=\prod_{i=1}^{k}\det([M]_{i})^{a_{i}-a_{i+1}}$

where $[M]_{i}$ is the $i\times i$ matrix obtained by extracting the first $i$
rows and the first $i$ columns of $M$ and with the convention $a_{k+1}=0$.

Here is our main result regarding the behavior of the multiplicative spherical
integral.

######  Theorem 2.1.

Let $\beta=1$ for the real case and $\beta=2$ for the complex case. Let
$\overline{\theta}=(\theta_{1},...,\theta_{k})\in\mathbb{R}^{k}$, $\sigma$ a
permutation of $\llbracket 1,k\rrbracket$ and $m\in\llbracket 1,k\rrbracket$
such that:

$\theta_{\sigma(1)}\leq\theta_{\sigma(2)}\leq...\leq\theta_{\sigma(m)}\leq
0\leq\theta_{\sigma(m+1)}\leq...\leq\theta_{\sigma(k)}.$

Let $(X_{N})_{N\in\mathbb{N}}$ be a sequence of matrices in
$\mathcal{H}^{\beta,++}_{N}$ such that there exists $M\in\mathbb{R}^{+}$ so
that $||X_{N}||,||X_{N}^{-1}||\leq M$ for all $N$ and such that there exists
$\lambda_{1},...\lambda_{k}\in\mathbb{R}^{+,*}$ such that:

$\forall i\in[1,m],\lim_{N\to\infty}\lambda_{i}(X_{N})=\lambda_{i}$

and

$\forall i\in[m+1,k],\lim_{N\to\infty}\lambda_{N+i-k}(X_{N})=\lambda_{i}$

then:

$\lim_{N\to\infty}\frac{1}{N}\log
I_{N}(\bar{\theta},X_{N})=\frac{\beta}{2}\sum_{i=1}^{k}J(\theta_{\sigma(i)},\lambda_{i},\mu)$

where $J$ is defined for $(\theta,\lambda,\mu)$ in the following set :

$\\{(\theta,\lambda,\mu)\in\mathbb{R}^{+}\times\mathbb{R}^{+,*}\times\mathcal{P}(\mathbb{R}^{+}):r(\mu)\leq\lambda\\}\cup\\{(\theta,\lambda,\mu)\in\mathbb{R}^{-}\times\mathbb{R}^{+,*}\times\mathcal{P}(\mathbb{R}^{+}):l(\mu)\geq\lambda,r(\mu)<+\infty\\}$

by

$J(\theta,\lambda,\mu):=(\theta+1)\log
c-\log|\theta|-\int\log\Big{|}\frac{\theta+1}{\theta}c-x\Big{|}d\mu(x)$

for $\theta\neq 0,-1$ with:

$c:=\begin{cases}\frac{\theta}{\theta+1}\lambda\text{ when }\theta\leq
T_{\mu}(\lambda)\leq 0\text{ and }0\leq T_{\mu}(\lambda)\leq\theta\\\
\tilde{S}_{\mu}(\theta)\text{ otherwise}.\end{cases}$

For $\theta=-1$, we can extend $J$ by continuity in $\theta$ by letting:

$J(-1,\lambda,\mu)=-\int\log|x|d\mu(x).$

For $\theta=0$, we can extend $J$ by continuity by letting :

$J(0,\lambda,\mu)=0.$

One can notice in particular that for values of $\theta$ between $0$ and
$T_{\mu}(\lambda)$, $J(\theta,\lambda,\mu)$ does not depend on $\lambda$. In
particular, if we differentiate in $\theta$, we can see that

$\frac{\partial
J}{\partial\theta}(\theta,\lambda,\mu)=\log\tilde{S}_{\mu}(\theta).$

Therefore:

######  Remark 2.2.

If $\theta\in[0,T_{\mu}(\lambda)]$ for $\theta>0$ or if
$\theta\in[T_{\mu}(\lambda),0]$ for $\theta<0$, we have:

$J(\theta,\lambda,\mu)=\int_{0}^{\theta}\log\tilde{S}_{\mu}(t)dt.$

This expression is similar to the expression found in [9, Theorem 6] which
expresses the limit of classical spherical integrals using the integral of the
$R$-transform. Before further studying this integral, one can make a first
useful remark about its regularity with respect to the parameters
$\overline{\theta}$ and the eigenvalues of $X_{N}$.

######  Remark 2.3.

  * Let $\overline{\theta},\overline{\phi}\in\mathbb{R}^{k}$ and $X_{N},Y_{N}\in\mathcal{H}_{N}^{\beta,++}$ two definite positive matrices.

  * •

If there is $\epsilon>0$ such that for all $i\in[1,N]$,

$e^{-\epsilon}\leq\frac{\lambda_{i}(X_{N})}{\lambda_{i}(Y_{N})}\leq
e^{\epsilon}$

then

$\left|I_{N}(X_{N},\overline{\theta})-I_{N}(Y_{N},\overline{\theta})\right|\leq
M\epsilon$

where $M=k\max_{i=1}^{k-1}|\theta_{i+1}-\theta_{i}|.$

  * •

if for all $i\in[1,k]$

$|\theta_{i}-\phi_{i}|\leq\epsilon$

then

$\left|I_{N}(X_{N},\overline{\phi})-I_{N}(X_{N},\overline{\theta})\right|\leq
M^{\prime}\epsilon$

where $M^{\prime}=2k\max_{i=1}^{N}|\log\lambda_{i}(X_{N})|.$

###### Proof.

For the first point, let us compare $\det([X_{N}]_{i})$ and
$\det([Y_{N}]_{i})$ for $i\leq k$. There exists an orthonormal basis
$e_{1},...,e_{i}\in\mathbb{C}^{i}$ such that:

$\det([X_{N}]_{i})=\prod_{j=1}^{i}\langle e_{j},[X_{N}]_{i}e_{j}\rangle.$

But the hypothesis implies that for every $u\in\mathbb{C}^{N}$, $\langle
u,X_{N}u\rangle\geq e^{-\epsilon}\langle u,Y_{N}u\rangle$ and this imply that
for every $u\in\mathbb{C}^{i}$, $\langle u,[X_{N}]_{i}u\rangle\geq
e^{-\epsilon}\langle u,[Y_{N}]_{i}u\rangle$. Therefore, we can write:

$\det([X_{N}]_{i})\geq e^{-i\epsilon}\prod_{j=1}^{i}\langle
e_{j},[Y_{N}]_{i}e_{j}\rangle\geq e^{-i\epsilon}\det([Y_{N}]_{i}).$

Where we used Hadamard’s inequality for the last inequality. Switching the
roles of $X_{N}$ and $Y_{N}$, we also have that
$\det([X_{N}]_{i})\leq\exp(i\epsilon)\det([Y_{N}]_{i})$. It follows then
easily that
$\exp(M\epsilon)\Delta_{\overline{\theta}}(Y_{N})\geq\Delta_{\overline{\theta}}(X_{N})\geq\exp(-M\epsilon)\Delta_{\overline{\theta}}(Y_{N})$
which implies our result.

For the second point, it suffices to remark that:

$\exp(-M^{\prime}/2)\leq\det[X_{N}]_{i}\leq\exp(M^{\prime}/2).$

∎

##  3\. Proof in the case $k=1$

We will begin by proving the Theorem 2.1 in the case $k=1$ assuming that the
spectra of the $X_{N}$ are all included in some finite set
$\\{\mu_{1},...,\mu_{l}\\}$ in the subsection 3.1. Then, in the subsection
3.2, we will generalize this to general matrices using approximations and
continuity arguments for the integral $I_{N}$ and its limit $J$. This case
will be refered as the one-dimensional case.

###  3.1. The case of spectra included in a finite set

Using the unitary invariance of the Haar measure, without loss of generality,
we can assume that $X_{N}$ is diagonal. In this subsection, we assume that
there exists $0<\mu_{1}...<\mu_{l}$ and integers $\alpha_{i}(N)$ such that
$\alpha_{1}(N)+...+\alpha_{l}(N)=N$ and
$\lim_{N\to\infty}\alpha_{i}(N)/N=\alpha_{i}\geq 0$ so that $X_{N}$ is of the
form

$X_{N}=diag(\underbrace{\mu_{1},...,\mu_{1}}_{\alpha_{1}(N)\text{ times
}},...,\underbrace{\mu_{l},...,\mu_{l}}_{\alpha_{l}(N)\text{ times }}).$

There, we have the convergence of the empirical $\mu_{X_{N}}$ toward
$\mu=\sum_{i=1}^{l}\alpha_{i}\delta_{\mu_{i}}$. Furthermore, let
$\theta\in\mathbb{R}$. If $\theta\geq 0$ we assume that $\alpha_{l}(N)>0$ for
$N$ large enough and if $\theta<0$, we assume that $\alpha_{1}(N)>0$. Then we
want to prove the following lemma which corresponds to Theorem 2.1 for $k=1$

######  Lemma 3.1.

If $\theta\geq 0$, then:

$\lim_{N\to\infty}\frac{1}{N}\log
I_{N}(\theta,X_{N})=\frac{\beta}{2}J(\theta,\mu_{l},\mu).$

If $\theta\leq 0$, then:

$\lim_{N\to\infty}\frac{1}{N}\log
I_{N}(\theta,X_{N})=\frac{\beta}{2}J(\theta,\mu_{1},\mu).$

###### Proof.

We will denote $I_{i}^{N}$ the following intervals:

$I_{i}^{N}=\rrbracket\sum_{j=1}^{i-1}\alpha_{j}(N),\sum_{j=1}^{i}\alpha_{j}(N)\rrbracket.$

If we denote $e$ the first column of $U$, we have that

$\det([U^{*}X_{N}U]_{1})^{\frac{\beta\theta N}{2}}=\langle
e,X_{N}e\rangle^{\frac{\beta\theta
N}{2}}=\Big{(}\sum_{j=1}^{l}\mu_{i}\gamma_{i}\Big{)}^{\frac{\beta
N\theta}{2}},$

where the $\gamma^{N}_{i}$ are defined as follows:

$\gamma^{N}_{i}=\sum_{j\in I^{N}_{i}}|e_{j}|^{2}.$

$(\gamma^{N}_{1},...,\gamma^{N}_{l})$ is a random variable whose distribution
is a Dirichlet law of parameters
$(\beta\alpha_{1}(N)/2,...,\beta\alpha_{l}(N)/2)$. Therefore, following for
instance [8, Theorem 11] it satisfies a large deviations principle on the
$l-1$-dimensional simplex
$K:=\\{\gamma\in(\mathbb{R}^{+})^{l}:\gamma_{1}+...+\gamma_{l}=1\\}$ with good
rate function:

$\frac{\beta}{2}H(\gamma)=-\frac{\beta}{2}\sum_{i=1}^{l}\alpha_{i}\log\frac{\gamma_{i}}{\alpha_{i}}.$

Therefore, using Varadhan’s lemma we have that:

$\lim_{N\to\infty}\frac{1}{N}\log
I_{N}(\theta,X_{N})=\frac{\beta}{2}\sup_{\gamma\in K}f_{\theta}(\gamma)\text{
where}f_{\theta}(\gamma)=\Big{[}\theta\log\Big{(}\sum_{i=1}^{l}\gamma_{i}\mu_{i}\Big{)}-H(\gamma)\Big{]}.$

Since the function $f_{\theta}$ is proper, and continuous from $K$ to
$\mathbb{R}\cup\\{-\infty\\}$ and since it is upper bounded by
$|\theta\max\\{\log\mu_{1},\log\mu_{l}\\}|$, its supremum is necessarily
achieved at some $\overline{\gamma}$. Since a priori, there may be several
maximum arguments, we choose one $\overline{\gamma}$ arbitrarily. However we
will end up proving that there is indeed only one maximal argument possible.

Let us first deal with the case of a positive $\theta$. Let $A$ be the set of
indices $i$ such that $\alpha_{i}=0$. Then we can assume $\bar{\gamma}_{i}=0$
for all $i\in A\setminus\\{l\\}$. Indeed for any $\gamma\in K$, if we
construct $\psi\in K$ by letting $\psi_{l}=\sum_{i\in
A\setminus\\{l\\}}\gamma_{i}+\gamma_{l}$, $\psi_{i}=0$ for $i\in
A\setminus\\{l\\}$ and $\psi_{i}=\gamma_{i}$ for all other indices, we have
easily that $f_{\theta}(\psi)\geq f_{\theta}(\gamma)$. Therefore, up to
reasoning on $K\cap\mathbb{R}^{\llbracket
1,l\rrbracket\setminus(A\setminus\\{l\\})}$ rather that on $K$, we can assume
that $\alpha_{i}>0$ for all $i$ except maybe $l$. Then, there are two
possibilities: either $\bar{\gamma}_{l}=0$ or $\bar{\gamma}_{l}>0$. If $\nabla
f_{\theta}$ is the gradient of $f_{\theta}$ in the ambient space
$\mathbb{R}^{l}$, we have that:

$\forall i=1,...,l,\,\nabla
f_{\theta}(\gamma)_{i}=\frac{\theta\mu_{i}}{\sum_{j=1}^{l}\gamma_{j}\mu_{j}}+\frac{\alpha_{i}}{\gamma_{i}}.$

Then, writing the critical point equation for each cases, we have that there
exists $a\in\mathbb{R}$ such that respectively:

$\displaystyle\text{ if }\bar{\gamma}_{k}=0,$ $\displaystyle\,\forall
i=1,...,l-1,\,\nabla f_{\theta}(\bar{\gamma})_{i}=a,$ $\displaystyle\text{ if
}\bar{\gamma}_{k}>0,$ $\displaystyle\,\forall i=1,...,l,\,\nabla
f_{\theta}(\bar{\gamma})_{i}=a.$

In both cases, it implies $\sum_{i=1}^{l}\bar{\gamma}_{i}\nabla
f_{\theta}(\bar{\gamma})_{i}=a$ and we get that $a=\theta+1$. If we let
$c=\sum_{j=1}^{l}\mu_{j}\bar{\gamma}_{j}$ we have that for every
$i=1,...,l-1$:

$\frac{\theta\mu_{i}}{c}+\frac{\alpha_{i}}{\bar{\gamma}_{i}}=\theta+1.$

We remind that if $\alpha_{i}>0$ then $\bar{\gamma}_{i}>0$. Indeed, if not we
would have $f_{\theta}(\bar{\gamma})=-\infty$. Let us assume that
$\bar{\gamma}_{l}>0$ and $\alpha_{l}=0$ then:

$c=\frac{\theta\mu_{l}}{\theta+1}$

and by definition of $c$, we must have:

$\displaystyle c$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{l}\mu_{i}\bar{\gamma}_{i}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{l-1}\frac{\mu_{i}\alpha_{i}}{\theta+1-\theta\frac{\mu_{i}}{c}}+\mu_{l}\bar{\gamma}_{l}$
$\displaystyle>$
$\displaystyle\frac{1}{\theta+1}\sum_{i=1}^{l-1}\frac{\mu_{i}\alpha_{i}}{1-\frac{\mu_{i}}{\mu_{l}}}$
$\displaystyle>$
$\displaystyle\frac{c}{\theta}\sum_{i=1}^{l-1}\frac{\mu_{i}\alpha_{i}}{\mu_{l}-\mu_{i}}$
$\displaystyle>$ $\displaystyle\frac{c}{\theta}T_{\mu}(\mu_{l}).$

Therefore, we necessarily have $\theta>T_{\mu}(\mu_{l})$. If now we assume
that either $\bar{\gamma}_{l}=0$ (and therefore $\alpha_{l}=0$) or
$\alpha_{l}>0$ (and therefore $\bar{\gamma}_{l}>0$), then we have

$\displaystyle c$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{l}\mu_{i}\bar{\gamma}_{i}$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{l}\frac{\mu_{i}\alpha_{i}}{\theta+1-\theta\frac{\mu_{i}}{c}}$

so that

$\theta=T_{\mu}(\frac{\theta+1}{\theta}c).$

Furthermore, since $\alpha_{i}>0$, for $i=1,...,l-1$, we necessarily have
$\overline{\gamma}_{i}>0$ for $i=1,...,l-1$. Therefore, if
$u=(-(l-1)^{-1},...,-(l-1)^{-1},1))$, $\overline{\gamma}+\epsilon u\in K$ for
$\epsilon>0$ small enough and so, differentiating in $\epsilon\to 0^{+}$, we
must have that:

$\langle\nabla f_{\theta}(\bar{\gamma}),u\rangle\leq 0$

which implies that

$\theta\frac{\mu_{l}}{c}-(\theta+1)\leq 0$

that is $\mu_{l}\leq\frac{\theta+1}{\theta}c$ so that, applying $T_{\mu}$ we
have $\theta\leq T_{\mu}(\mu_{l})$. Therefore, since at least one maximal
argument $\bar{\gamma}$ always exists, we have the following dichotomy for
every possible $\bar{\gamma}$:

  * •

If $\theta\leq T_{\mu}(\mu_{l})$, then either $\bar{\gamma}_{l}=0$ or
$\alpha_{l}>0$ and in both cases
$c=\frac{\theta}{\theta+1}T_{\mu}^{(-1)}(\theta)=\tilde{S}_{\mu}(\theta)$.

  * •

If $\theta>T_{\mu}(\mu_{l})$, then $\bar{\gamma}_{l}>0$ and
$c=\frac{\theta}{\theta+1}\mu_{l}$.

In each case, we remind that we have for $i=1,...,l-1$:

$\bar{\gamma}_{i}=\frac{\mu_{i}\alpha_{i}}{\theta+1-\theta\frac{\mu_{i}}{c}}=\frac{c}{\theta}\frac{\alpha_{i}\mu_{i}}{\frac{\theta+1}{\theta}c-\mu_{i}}.$

and

$\bar{\gamma}_{l}=1-\sum_{i=1}^{l-1}\bar{\gamma}_{i}.$

Therefore, there can be only one possible $\bar{\gamma}$ and then substituting
$\bar{\gamma}$ and $c$ in $f_{\theta}(\gamma)$ we have:

$\displaystyle\lim_{N\to\infty}\frac{1}{N}\log I_{N}(\theta,X_{N})$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}\Big{[}(\theta+1)\log
c-\log|\theta|-\sum_{i=1}^{l-1}\alpha_{i}\log\Big{|}\frac{\theta+1}{\theta}c-\mu_{i}\Big{|}\Big{]}$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}J(\theta,\lambda,\mu)$

where $J$ is indeed the function defined in Theorem 2.1.

For $\theta\leq 0$, we have the same analysis by replacing $\mu_{l}$ by
$\mu_{1}$ (and thus assuming that $\alpha_{i}>0$ for all $i>1$). We end up
then with the following values for $c$:

  * •

If $\theta\geq T_{\mu}(\mu_{1})$, then
$c=\frac{\theta}{\theta+1}T_{\mu}^{(-1)}(\theta)=\tilde{S}_{\mu}(\theta)$
(with the continuous extension in $\theta=-1$,
$\tilde{S}_{\mu}(-1)=\Big{(}\int\frac{d\mu(x)}{x}\Big{)}^{-1}$).

  * •

If $\theta<T_{\mu}(\mu_{1})$, then $c=\frac{\theta}{\theta+1}\mu_{1}$. Then
again, substituting $\hat{\gamma}$ and $c$, we obtain again the function $J$.

∎

###  3.2. Generalization to a generic limit $\mu$

Now we want to generalize our result to matrices with a general limit measure
$\mu$. First we will need that the function $J$ is continuous in the argument
$(\theta,\lambda_{,}\mu)$.

######  Lemma 3.2.

Let $\kappa>0$. The function $J$ is continuous on the sets:

$\mathcal{D}^{+}:=\\{(\theta,\lambda,\mu)\in\mathbb{R}^{+}\times\mathbb{R}^{+,*}\times\mathcal{P}(\mathbb{R}^{+}),\lambda\geq
r(\mu),l(\mu)>0\\}$

and on the set

$\mathcal{D}^{-}:=\\{(\theta,\lambda,\mu)\in\mathbb{R}^{-}\times\mathbb{R}^{+,*}\times\mathcal{P}(\mathbb{R}^{+}),\lambda\leq
l(\mu),r(\mu)<\kappa\\}$

where $\mathcal{D}$ is endowed with the topology induced by the product
topology where we consider the weak topology on $\mathcal{P}(\mathbb{R}^{+})$.

###### Proof.

Let $d=\frac{\theta+1}{\theta}c$ so that

$J(\theta,\lambda,\mu)=(\theta+1)\ln
d+\theta\ln|\theta|-(\theta+1)\ln(|\theta+1|)-\int\ln|d-x|d\mu(x).$

Let us first prove the continuity on $\mathcal{D}^{+}$. Differentiating in
$\lambda$, we get that

$\frac{\partial
J}{\partial\lambda}=\begin{cases}\frac{\theta+1}{\lambda}-G_{\mu}(\lambda)\text{
if }\lambda\geq T_{\mu}^{-1}(\theta)\\\ 0\text{ if not }\end{cases}.$

This implies that since $G_{\mu}(\lambda)\geq\frac{1}{\lambda}$.

$0\leq\frac{\partial
J}{\partial\lambda}(\theta,\lambda,\mu)\leq\frac{\theta}{\lambda}.$

so that on $\mathcal{D}^{+}$:

$|J(\theta,\lambda,\mu)-J(\theta,\lambda^{\prime},\mu)|\leq\theta|\ln\lambda-\ln\lambda^{\prime}|.$

The same way, differentiating in $\theta$ we have:

$\frac{\partial J}{\partial\theta}(\theta,\lambda,\mu)=\ln c.$

For $\theta\leq T_{\mu}(\lambda)$, since $\tilde{S}$ is increasing, we have
that $c=\tilde{S}(\theta)\leq\lambda$. If not, since $\theta>0$,
$c=\frac{\theta}{\theta+1}\lambda\leq\lambda$. Therefore:

$0\leq\frac{\partial J}{\partial\theta}(\theta,\lambda,\mu)\leq\ln\lambda.$

And then we use that $c\leq\lambda$ to have that

$|J(\theta,\lambda,\mu)-J(\theta^{\prime},\lambda,\mu)|\leq\ln\lambda|\theta-\theta^{\prime}|.$

Therefore, we have a continuity modulus uniform on every compact on
$\mathcal{D}^{+}$ for the variables $(\theta,\lambda)$ in the sense that for
each such compact $K\subset\mathcal{D}^{+}$, the exists some increasing
function $g_{K}:\mathbb{R}^{+}\to\mathbb{R}^{+}$ that tends to $0$ in $0$ such
that if $(\theta,\lambda,\mu),(\theta^{\prime},\lambda^{\prime},\mu)\in K$
such that $|\lambda-\lambda^{\prime}|+|\theta-\theta^{\prime}|\leq\epsilon$
then

$|J((\theta,\lambda,\mu))-J((\theta^{\prime},\lambda^{\prime},\mu))|\leq
g_{K}(\epsilon).$

‘Therefore it suffices to prove continuity with respect to $\mu$. Using the
continuity modulus with respect to $\lambda$, it suffices to prove the
continuity on the domains:

$\mathcal{D}_{M,A,\epsilon}^{+}:=\\{(\theta,\lambda,\mu)\in[0,A]\times[M^{-1},M]\times\mathcal{P}(\mathbb{R}^{+}),\lambda\geq
e^{\epsilon}r(\mu),0\leq l(\mu)\\}$

for $M>0,\epsilon>0,A>0$. If
$((\theta_{n},\lambda_{n},\mu_{n}))_{n\in\mathbb{N}}$ is a sequence of
elements of $\mathcal{D}_{M,A,\epsilon}^{+}$ converging toward converging
toward some $(\theta,\lambda,\mu)\in\mathcal{D}_{M,A,\epsilon}^{+}$, we see
that for $n$ large enough
$(\theta,\lambda,\mu_{n})\in\mathcal{D}_{M+1,A+1,\epsilon/2}^{+}$ and using
the uniform continuity modulus on $\mathcal{D}_{M+1,A+1,\epsilon/2}^{+}$, we
see that it is sufficient to show that $J(\theta,\lambda,\mu_{n})$ converges
toward $J(\theta,\lambda,\mu)$.

If $\theta=0$, the result is trivial, so we can assume that $\theta>0$. Then
let $d_{n}=\frac{\theta+1}{\theta}c_{n}$, where $c_{n}$ is defined the same as
the quantity $c$ for $\lambda_{,}\theta,\mu_{n}$ instead of
$\lambda,\theta,\mu$. First we have $\lim_{n\to\infty}d_{n}=d$. Indeed, if
$\theta>T_{\mu}(\lambda)$, using the fact that $\lambda\geq
e^{\epsilon/2}r(\mu_{n})$ for $n$ large enough and the weak convergence of
$\mu_{n}$, we have that
$\lim_{n\to\infty}T_{\mu_{n}}(\lambda)=T_{\mu}(\lambda)$, and therefore
$\theta>T_{\mu_{n}}(\lambda)$ and $d_{n}=\lambda$ for $n$ large enough. If
$\theta<T_{\mu}(\lambda)$, the same argument leads to
$\theta<T_{\mu_{n}}(\lambda)$ for $n$ large enough. In that case, for every
$\eta>0$, we can find $\theta^{+}$ and $\theta^{-}$ such that
$\theta^{-}<\theta<\theta^{+}$ and $T_{\mu}^{-1}(\theta^{-})\geq
T_{\mu}^{-1}(\theta)-\eta>e^{\epsilon/2}r(\mu)$ and
$T_{\mu}^{-1}(\theta^{+})\leq T_{\mu}^{-1}(\theta)+\eta$. If we denote
$d^{\pm}=T_{\mu}^{-1}(\theta^{\pm})$, using the weak convergence of $\mu_{n}$
and the fact that $d^{-}\geq e^{\epsilon/2}r(\mu_{n})$ for $n$ large enough,
we have that $T_{\mu_{n}}(d^{-})\geq\theta\geq T_{\mu_{n}}(d^{+})$ and
therefore $d^{-}\leq d_{n}\leq d^{+}$ for $n$ large enough which proves
$\lim_{n\to\infty}d_{n}=d$. The case $T_{\mu}(\lambda)=\theta$ can be easily
deduced by uniform continuity in $\theta$. To conclude then, we only need to
prove the convergence of the term $\int\log|d_{n}-x|d\mu_{n}$. One can see
that $d_{n}\geq\lambda>e^{\epsilon/2}r(\mu_{n})$. Therefore we have $d_{n}\geq
e^{\epsilon/2}r(\mu)$. The sequence of function $x\mapsto\log|d_{n}-x|$
converges uniformly on $]0,e^{\epsilon/3}r(\mu)]$ and one concludes with the
weak convergence of $\mu_{n}$. The proof is the same for $\mathcal{D}^{-}$
except one needs the assumption $r(\mu_{n})\leq\kappa$ to prove the uniform
convergence of the $x\mapsto\log|d_{n}-x|$ on some
$[e^{-\epsilon/3}l(\mu),\kappa]$ and that one needs also to consider the case
$\theta=-1$ apart from the generic case.

∎

To proceed from the case with $\mu$ with a finite support to the case of a
diffuse support, we first consider the case with $\mu$ with a continuous
repartition function. We will assume that $\theta\geq 0$ (the case $\theta\leq
0$ is similar). Let us take $(X_{N})_{N\in\mathbb{N}}$ a sequence of matrices
such that their eigenvalue distributions converge toward $\mu$, that
$\lambda_{N}(X_{N})$ converges toward some $\lambda_{1}$ and that
$||X_{N}||,||X_{N}^{-1}||\leq M$ for some $M$. For $\epsilon>0$ and let
$X_{N}^{(\epsilon)}$ be the modification of $X_{N}$ where we replace its
$i$-th eigenvalue $\lambda_{i}(X_{N})$ by
$\lambda_{i}^{(\epsilon)}(X_{N})=(1+\epsilon)^{n_{0}}$ where $n_{0}$ is the
smallest integer such that $\lambda_{i}(X_{N})\leq(1+\epsilon)^{n_{0}}$. Then
the eigenvalue distribution of $X_{N}^{\epsilon}$ converges toward
$\mu^{(\epsilon)}=\sum_{k\in\mathbb{N}^{*}}\alpha_{k}\delta_{(1+\epsilon)^{k}}$
where $\alpha_{k}=\mu([(1+\epsilon)^{k-1},(1+\epsilon)^{k}])$ thanks to the
continuity hypothesis on the repartition function of $\mu$. Since
$\lambda_{N}(X_{N})$ converges toward $\lambda_{1}$,
$\lambda_{N}(X_{N}^{(\epsilon)})$ is constant for $N$ large enough and equal
to $\lambda_{1}^{(\epsilon)}$ (defined the same way as
$\lambda_{i}(X_{N}^{(\epsilon)})$). Then $X_{N}^{(\epsilon)}$ satisfies the
conditions of section 3.1 and Lemma 3.1 applies. Therefore:

$\lim_{N\to\infty}\frac{1}{N}\log
I_{N}(X^{(\epsilon)}_{N},\theta)=\frac{\beta}{2}J(\theta,\lambda_{1}^{(\epsilon)},\mu^{(\epsilon)}).$

With our hypothesis on $\mu$, $\lim_{\epsilon\to 0}\mu^{(\epsilon)}=\mu$ and
$\lim_{\epsilon\to 0}\lambda_{1}^{(\epsilon)}=\lambda_{1}$. Furthermore, using
Remark 2.3, we can see that:

$\frac{1}{N}|\log I_{N}(\theta,X_{N})-\log
I_{N}(\theta,X^{(\epsilon)}_{N})|\leq\frac{\beta}{2}\theta\log(1+\epsilon).$

Using the continuity of the limit $J$, one concludes Theorem 2.1 stands in
this case.

To relax the condition on the repartition function of $\mu$ we can do the same
thing by replacing $\lambda_{i}(X_{N})$ by
$\lambda_{i}^{(\epsilon)}(X_{N})=\lambda_{i}(X_{N})(1+\epsilon\delta_{i})$
where $\delta_{i}\in[0,1]$ and are sampled so that the eigenvalue distribution
of $X_{N}^{(\epsilon)}$ converges towards $\mu^{(\epsilon)}$ which is the
independent product of $\mu$ with a uniform law on the segment
$[1,1+\epsilon]$. This law has a continuous repartition function and we can
again apply the approximation arguments above.

##  4\. The case of $k\geq 2$

To prove the result in the case $k\geq 2$, we will proceed recursively on $k$
by conditioning on the first column of $U$. This will bring us back the case
$k-1$. As in the one-dimensional case, we will first proceed by dealing with
the case where the eigenvalues of $X_{N}$ are in a finite set in subsection
4.1 and then we will extend the result to general $\mu$ in the same fashion as
in the one-dimensional case in subsection 4.2.

Therefore, for the rest of this section, we let $k>1$ be an integer and we
assume that the Theorem 2.1 is satisfied for $k-1$.

###  4.1. Spectra included in a finite set

Let $\bar{\theta}\in\mathbb{R}^{l}$, $\sigma$ a permutation of $[2,k-1]$ and
$m\in[1,k-1]$ such that
$\theta_{\sigma(1)}\leq\theta_{\sigma(2)}\leq...\theta_{\sigma(m)}\leq
0\leq\theta_{\sigma(m+1)}\leq...\leq\theta_{\sigma(k-1)}$. Let
$p\in\mathbb{N}$ such that
$\theta_{\sigma(p)}\leq\theta_{1}\leq\theta_{\sigma(p+1)}$. We will first
assume that $\theta_{1}\geq 0$ which implies that we can choose $p\geq m$. At
the end of the proof, we will discuss the differences that arise when
$\theta_{1}\leq 0$

We will assume that there is a finite number of reals
$\mu_{-m}<\mu_{-m+1}<...<\mu_{l+k-m}$ such that:

$X_{N}=diag(\mu_{-m},...,\mu_{-1},\underbrace{\mu_{0},...,\mu_{0}}_{\alpha_{0}(N)\text{
times }},...,\underbrace{\mu_{l},...,\mu_{l}}_{\alpha_{l}(N)\text{ times
}},\mu_{l+1},...,\mu_{l+k-m})$

and where $\lim\alpha_{i}(N)/N=\alpha_{i}\geq 0$.

Therefore, given these notation we are trying to establish is the following
lemma:

######  Lemma 4.1.

$\displaystyle\lim_{N\to\infty}\frac{1}{N}\log I_{N}(\overline{\theta},X_{N})$
$\displaystyle=\frac{\beta}{2}J(\theta_{1},\mu_{l+p-m+1},\mu)+\frac{\beta}{2}\sum_{i=1}^{m}J(\theta_{\sigma(i)},\mu_{i-m-1},\mu)$
$\displaystyle+\frac{\beta}{2}\sum_{i=m+1}^{p}J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)+\frac{\beta}{2}\sum_{i=p+1}^{k-1}J(\theta_{\sigma(i)},\mu_{i-m+l+1},\mu)$

###### Proof.

First, we can write the matrix $U^{*}X_{N}U$ by blocks as follows

$U^{*}X_{N}U=\begin{pmatrix}a&c^{*}\\\ c&\tilde{X}_{N-1}\end{pmatrix}$

where $a\in\mathbb{R}^{+,*}$, $c\in\mathbb{C}^{N-1}$ and
$\tilde{X}_{N-1}\in\mathcal{H}^{\beta,++}_{N-1}$. Then one can see that:

$T^{*}_{N}U^{*}X_{N}UT_{N}=\begin{pmatrix}a&0\\\
0&\tilde{X}_{N-1}-\frac{cc^{*}}{a}\end{pmatrix}$

where

$T_{N}=\begin{pmatrix}1&-\frac{c^{*}}{a}\\\ 0&I_{N-1}\end{pmatrix}.$

Due to the the fact that $T_{N}$ is upper triangular, this implies that for
$i\in[1,N]$:

$[T^{*}_{N}]_{i}[U^{*}X_{N}U]_{i}[T_{N}]_{i}=\begin{pmatrix}a&0\\\
0&[\tilde{X}_{N-1}-\frac{cc^{*}}{a}]_{i-1}\end{pmatrix}$

If we denote $Y_{N}=\tilde{X}_{N-1}-\frac{cc^{*}}{a}$, using that
$\det[T_{N}]_{i}=1$, we have that $\det([U^{*}X_{N}U]_{i})=a\det(Y_{N})$ and
we can express $\Delta_{\bar{\theta}}(X_{N})$ as a function of $a$ and
$Y_{N}$.

$\Delta_{\bar{\theta}}(X_{N})=a^{\theta_{1}-\theta_{2}}\prod_{i=2}^{k}\det[U^{*}X_{N}U]_{i}^{\theta_{i}-\theta_{i-1}}=a^{\theta_{1}-\theta_{2}}\prod_{i=2}^{k}a^{\theta_{i}-\theta_{i+1}}\det[Y_{N}]_{i}^{\theta_{i}-\theta_{i+1}}=a^{\theta_{1}}\Delta_{\tilde{\theta}}(Y_{N})$

where
$\tilde{\theta}=(\theta_{2},\theta_{3},....,\theta_{k})\in\mathbb{R}^{k-1}$.
Furthermore, one can easily see that the spectrum of $Y_{N}$ only depends on
the first column of $U$ and that once conditioned on this that column that we
denote $e$, the law of $Y_{N}$ is invariant by conjugation by any element of
$\mathcal{U}^{\beta}_{N-1}$. Therefore, since $a=\langle e,X_{N}e\rangle$ we
deduce that:

(1) $\displaystyle I_{N}(\bar{\theta},X_{N})$ $\displaystyle=$
$\displaystyle\int_{U\in\mathcal{U}_{N}^{(\beta)}}\langle
e,X_{N}e\rangle^{\frac{\beta
N}{2}\theta_{1}}\Delta_{\tilde{\theta}}(Y_{N})^{\frac{\beta N}{2}}dU$
$\displaystyle=$ $\displaystyle\int_{e\in\mathbb{S}^{\beta N-1}}\langle
e,X_{N}e\rangle^{\frac{\beta
N}{2}\theta_{1}}I_{N-1}\Big{(}\frac{N}{N-1}\tilde{\theta},Y_{N}\Big{)}de.$

Let us first determine the spectrum of $Y_{N}$ in function of $e$ and its
components. Let us denote $P$ the characteristic polynomial of $X_{N}$. Using
the linearity in the first column, we deduce:

$P(z)=z\chi_{\tilde{X}_{N}}+\det\begin{pmatrix}-a&-c^{*}\\\
-c&z-\tilde{X}_{N-1}\end{pmatrix}=z\chi_{\tilde{X}_{N}}-a\chi_{Y_{N}}(z)$

where we used the Schur complement formula. In the same way we defined the
$I^{N}_{i}$ and the $\gamma_{i}^{N}$ in subsection 3.1, we can also do it
here:

$\forall
i=-m,...,k+l-m\,,I_{i}^{N}:=\\{j\in[1,N],\lambda_{j}(X_{N})=\mu_{i}\\}\text{
and }\gamma_{i}^{N}=\sum_{j\in I_{i}}|e_{j}|^{2}.$

Then using Weyl interlacing relationships, we know that:

$\frac{\chi_{\tilde{X}_{N}}(z)}{P(z)}=\sum_{i=-m}^{l+k-m}\frac{\gamma^{N}_{i}}{z-\mu_{i}}$

therefore we have that:

$\frac{\chi_{Y_{N}}(z)}{P(z)}=\sum_{i=-m}^{l+k-m}\frac{\gamma^{N}_{i}\mu_{i}}{z-\mu_{i}}.$

Therefore, if we consider $Q$ the polynomial:

(2) $Q(z)=\sum_{i=-m}^{l+k-m}\gamma^{N}_{i}\mu_{i}\prod_{j\neq i}(z-\mu_{i})$

and if we call
$\chi_{-m}\leq...\leq\chi_{-1}\leq\mu^{\prime}_{1}\leq...\mu^{\prime}_{l}\leq\chi_{1}\leq...\leq\chi_{k-m-1}$.
its roots, then the spectrum of $Y_{N}$ has the following form:

(3)
$Spec(Y_{N})=\\{\chi_{-m},...,\chi_{-1},\underbrace{\mu_{0},...,\mu_{0}}_{\alpha_{0}(N)-1},\mu^{\prime}_{0},\underbrace{\mu_{1},...,\mu_{1}}_{\alpha_{1}(N)-1},....,\mu^{\prime}_{l-1},\underbrace{\mu_{l},...,\mu_{l}}_{\alpha_{l}(N)-1},\mu^{\prime}_{l},\chi_{1},...,\chi_{k-m-1}\\}$

and the roots of $Q$ satisfy:

(4)
$\mu_{-m}\leq\chi_{-m}\leq\mu_{-m+1}\leq...\leq\mu_{1}\leq\mu^{\prime}_{1}\leq...\leq\mu_{l-1}\leq\mu^{\prime}_{l-1}\\\
\leq\mu_{l}\leq\mu^{\prime}_{l}\leq\mu_{l+1}\leq\chi_{1}\leq\mu_{l+2}\leq...\leq\chi_{k-m-1}\leq\mu_{k-m}.$

In particular the spectrum of $Y_{N}$ is also interlaced with the spectrum of
$X_{N}$. Therefore the empirical eigenvalue distribution of $Y_{N}$ also
converges towards $\mu$. Furthermore, the $\chi_{i}$ and $\mu_{i}$ are
continuous functions of the $\gamma^{N}_{i}$ (since they are roots of a
polynomial whose coefficients are continuous in the $\gamma_{i}^{N}$) and if
the $\gamma^{N}_{i}$ are all positive, they are respectively the $m$ smallest
and $k-m-1$ largest solutions of the following equation in $z$.

(5) $\sum_{i=-m}^{k+l-m}\frac{\gamma^{N}_{i}\mu_{i}}{z-\mu_{i}}=0$

Given the form of the spectrum of $Y_{N}$ one can see
$I_{N}(\tilde{\theta},Y_{N})$ as a function of the $\mu^{\prime}_{i}$ and
$\chi_{i}$ and therefore as a function of of the $\gamma_{i}^{N}$.

Then, using the recursion hypothesis, that is that Theorem 2.1 is satisfied
for $k-1$, Remark 2.3 that provides the equicontinuity of
$I(\tilde{\theta},Y_{N})$ in the $\mu^{\prime}_{i}$ and $\chi_{i}$, and the
continuity of the function $J$ in its first two arguments, one can prove the
following lemma:

######  Lemma 4.2.

Let $\mathcal{E}_{N}$ be the subset of matrices in
$\mathcal{H}_{N-1}^{\beta,++}$ whose spectrum is of the form described by the
inequalities (3) with the $\mu^{\prime}_{i}$ and $\chi_{i}$ satisfying the
equation (4). Then we have:

$\lim_{N\to\infty}\sup_{Y\in\mathcal{E}_{N}}\Big{|}\frac{1}{N}\log
I_{N-1}(\frac{N}{N-1}\tilde{\theta},Y)-\sum_{i=1}^{m}J(\theta_{\sigma(i)},\chi_{i-m+1},\mu)-\sum_{i=m+1}^{k-1}J(\theta_{\sigma(i)},\chi_{i-m},\mu)\Big{|}=0.$

If we look at the law of the $\gamma_{i}^{N}$, we have again that they follow
a Dirichlet law with parameters:

$(\underbrace{\frac{\beta}{2},...,\frac{\beta}{2}}_{m\text{ times
}},\frac{\beta\alpha_{0}(N)}{2},...,\frac{\beta\alpha_{l}(N)}{2},\underbrace{\frac{\beta}{2},...,\frac{\beta}{2}}_{k-m\text{
times }})$

and therefore it follows a large deviation principle on the simplex $K$ with
the rate function:

$\gamma\mapsto\frac{\beta}{2}\sum_{i=0}^{l}\alpha_{i}\log\frac{\gamma_{i}}{\alpha_{i}}.$

(where the indexation for the coordinates of $\gamma$ begins at $-m$ and ends
at $k-m+l$ as for the $\gamma_{i}^{N}$). Therefore, using this large deviation
principle in conjunction with the preceding lemma and with the equation (1),
we can use Varadhan’s lemma on $I_{N}(\bar{\theta},X_{N})$ so the limit of
$N^{-1}\log I_{N}(\bar{\theta},X_{N})$ exists and is equal to the following
supremum that we will denote $J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$:

$J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})=\frac{\beta}{2}\sup_{\gamma_{i}\geq
0,\atop\sum\gamma_{i}=1}\Big{[}\theta_{1}\log\Big{(}\sum_{i=-m}^{l+k-m}\mu_{i}\gamma_{i}\Big{)}+\sum_{i=1}^{m}J(\theta_{\sigma(i)},\chi_{i-m+1},\mu)+\sum_{i=m+1}^{k-1}J(\theta_{\sigma(i)},\chi_{i-m},\mu)+\sum_{i=0}^{l}\alpha_{i}\ln\frac{\gamma_{i}}{\alpha_{i}}\Big{]}$

where the $\chi_{i}$ are viewed as continuous functions of the $\gamma_{i}$
(that is as the $m$ smallest and $k-m-1$ largest roots of the polynomial $Q$
defined in equation (2) where one replace $\gamma_{i}^{N}$ by $\gamma_{i}$).
By continuity, we can take the supremum on the subset of $\gamma$ such that
$\gamma_{i}>0$ for all $i$. Then since in this case, the $\chi_{i}$ are each
the unique solution to equation (5) on their corresponding intervals, we can
instead take the preceding supremum over the couples $(\gamma,\chi)$ over the
following domain:

$\mathcal{D}=\Big{\\{}(\gamma,\chi)\in(\mathbb{R}^{+,*})^{l+k+1}\times\prod_{i=-m}^{-1}]\mu_{i},\mu_{i+1}[\times\prod_{i=1}^{k-m-1}]\mu_{l+i},\mu_{l+i+1}[:\\\
\sum_{i=-m}^{k+l-m}\gamma_{i}=1,\forall
j\in\llbracket-m,-1\rrbracket\cup\llbracket
1,k-m-1\rrbracket,\sum_{i=-m}^{k+l-m}\frac{\gamma_{i}\mu_{i}}{\chi_{j}-\mu_{i}}=0\Big{\\}}$

And so we have :

$J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})=\frac{\beta}{2}\sup_{(\gamma,\chi)\in\mathcal{D}}\Big{[}\theta_{1}\log\Big{(}\sum_{i=-m}^{l+k-m}\mu_{i}\gamma_{i}\Big{)}+\sum_{i=1}^{m}J(\theta_{\sigma(i)},\chi_{i-m+1},\mu)+\sum_{i=m+1}^{k-1}J(\theta_{\sigma(i)},\chi_{i-m},\mu)+\sum_{i=0}^{l}\alpha_{i}\ln\frac{\gamma_{i}}{\alpha_{i}}\Big{]}$

We are now going to make a change of variables and introduce the new
$\overline{\gamma}_{i}$ in order to decouple the terms depending on the
$\chi_{i}$ from the terms depending on the $\overline{\gamma}_{i}$. Namely we
will let for $i\in\llbracket 0,l+1\rrbracket$:

$\overline{\gamma}_{i}=\prod_{j=-m}^{-1}\frac{\chi_{j}(\mu_{j}-\mu_{i})}{\mu_{j}(\chi_{j}-\mu_{i})}\prod_{j=1}^{k-m-1}\frac{\chi_{j}(\mu_{l+j+1}-\mu_{i})}{\mu_{l+j+1}(\chi_{j}-\mu_{i})}\gamma_{i}.$

Let us prove that the fonction
$\Phi:(\gamma,\chi)\mapsto(\overline{\gamma},\chi)$ is a bijection of
$\mathcal{D}$ onto the domain $\mathcal{D}^{\prime}$ defined as:

$\mathcal{D}^{\prime}=\Big{\\{}(\overline{\gamma},\chi)\in(\mathbb{R}^{+,*})^{l+2}\times\prod_{i=-m}^{-1}]\mu_{i},\mu_{i+1}[\times\prod_{i=1}^{k-m-1}]\mu_{l+i},\mu_{l+i+1}[:\sum_{i=0}^{l+1}\overline{\gamma}_{i}=1\Big{\\}}.$

First let us check that $\Phi(\mathcal{D})\subset\mathcal{D}^{\prime}$. It is
easy to check that each term of the product that defines
$\overline{\gamma}_{i}$ is positive so that $\overline{\gamma}_{i}>0$. Then
let us define $F$ the following rational function:

$F(Y)=\prod_{j=-m}^{-1}\frac{\chi_{j}(Y\mu_{j}-1)}{\mu_{j}(Y\chi_{j}-1)}\prod_{j=1}^{k-m-1}\frac{\chi_{j}(\mu_{l+j+1}Y-1)}{\mu_{l+j+1}(Y\chi_{j}-1)}$

so that $\overline{\gamma}_{i}=F(\mu_{i}^{-1})\gamma_{i}$. If we write down
the partial fraction decomposition of $F$, we have that there exists a family
of reals $a_{j}$ such that:

$F(Y)=1+\sum_{j=-m}^{-1}\frac{a_{j}}{Y\chi_{j}-1}+\sum_{j=1}^{k-m-1}\frac{a_{j}}{Y\chi_{j}-1}.$

Furthermore, since $F(\mu_{i}^{-1})=0$ for $i<0$ and $i>l+1$, we have that

$\displaystyle\sum_{i=0}^{l+1}\overline{\gamma}_{i}$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{l=1}F(\mu_{i}^{-1})\gamma_{i}$ $\displaystyle=$
$\displaystyle\sum_{i=-m}^{l+k-m}F(\mu_{i}^{-1})\gamma_{i}$ $\displaystyle=$
$\displaystyle\sum_{i=-m}^{l+k-m}\gamma_{i}+\sum_{j=-m}^{-1}\sum_{i=-m}^{l+k-m}\frac{a_{j}\mu_{i}\gamma_{i}}{\chi_{j}-\mu_{i}}+\sum_{j=1}^{k-m-1}\sum_{i=-m}^{l+k-m}\frac{a_{j}\mu_{i}\gamma_{i}}{\chi_{j}-\mu_{i}}$
$\displaystyle=$ $\displaystyle 1$

Where we used the interlacing equation satisfied by $\gamma$ and $\chi$ in the
last line. Furthermore, $F(\mu_{i}^{-1})>0$ so $\overline{\gamma}_{i}>0$. To
prove the bijection, one can refer to [8] where a similar change of variable
is made in the proof of proposition 14. In particular we have that for
$i\in\llbracket-m,l+k-m\rrbracket\setminus\llbracket 0,l+1\rrbracket$,
$\gamma_{i}=-(\sum_{j=0}^{l+2}G_{i}(\mu_{j}^{-1})\overline{\gamma_{j}})/G_{i}(\mu_{j}^{-1})$
where $G_{i}(X)=F(X)(X\mu_{i}-1)^{-1}$.

Let us now compare $\sum\mu_{i}\gamma_{i}$ with
$\sum\mu_{i}\overline{\gamma}_{i}$, for this, let us write down the partial
fraction decomposition for $F(Y)/Y$:

$\frac{F(Y)}{Y}=\frac{b}{Y}+\sum_{j=-m}^{-1}\frac{b_{j}}{Y\chi_{j}-1}+\sum_{j=1}^{k-m-1}\frac{b_{j}}{Y\chi_{j}-1}$

where

$b=\prod_{j=-m}^{-1}\frac{\chi_{j}}{\mu_{j}}\prod_{j=1}^{k-m-1}\frac{\chi_{j}}{\mu_{l+j+1}}$

so that, substituting $Y$ for $\mu_{i}^{-1}$, multiplying by $\gamma_{i}$ and
summing, one obtains:

$\sum_{i=0}^{l+1}\mu_{i}\overline{\gamma_{i}}=b\sum_{i=-m}^{l+k-m}\mu_{i}\gamma_{i}.$

Therefore, we can write that

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$
$\displaystyle\frac{\beta}{2}\sup_{(\overline{\gamma},\chi)\in\mathcal{D}^{\prime}}\Big{[}\theta_{1}\ln\Big{(}\sum_{i=0}^{l+1}\mu_{i}\overline{\gamma}_{i}\Big{)}+\sum_{i=0}^{l}\alpha_{i}\ln\frac{\overline{\gamma}_{i}}{\alpha_{i}}$
$\displaystyle+$
$\displaystyle\sum_{i=1}^{m}\Big{(}J(\theta_{\sigma(i)},\chi_{i-m+1},\mu)+(\theta_{1}+1)\ln\mu_{i-m+1}-(\theta_{1}+1)\ln\chi_{i-m+1}$
$\displaystyle+$
$\displaystyle\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{j}-\chi_{i-m+1})-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{j}-\mu_{i-m+1})\Big{)}$
$\displaystyle+$
$\displaystyle\sum_{i=m+1}^{k-1}\Big{(}J(\theta_{\sigma(i)},\chi_{i-m},\mu)+(\theta_{1}+1)\ln\mu_{l+i-m+1}-(\theta_{1}+1)\ln\chi_{i-m}$
$\displaystyle+$
$\displaystyle\sum_{j=0}^{l}\alpha_{j}\ln(\chi_{i-m}-\mu_{j})-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m+1}-\mu_{j})\Big{)}\Big{]}.$

We define the quantities $H_{i}$ for $i=1,...,k-1$ as follows :

$H_{i}(\chi)=J(\theta_{\sigma(i)},\chi,\mu)-(\theta_{1}+1)\ln\chi+\sum_{j=0}^{l}\alpha_{j}\ln(|\mu_{j}-\chi|)$

so that:

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$
$\displaystyle\frac{\beta}{2}\sup_{(\overline{\gamma},\chi)\in\mathcal{D}^{\prime}}\Big{[}\theta_{1}\ln\Big{(}\sum_{i=0}^{l+1}\mu_{i}\overline{\gamma}_{i}\Big{)}+\sum_{i=0}^{l}\alpha_{i}\ln\frac{\overline{\gamma}_{i}}{\alpha_{i}}$
$\displaystyle+$
$\displaystyle\sum_{i=1}^{m}\Big{(}H_{i}(\chi_{i-m+1})+(\theta_{1}+1)\ln\mu_{i-m+1}-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{j}-\mu_{i-m+1})\Big{)}$
$\displaystyle+$
$\displaystyle\sum_{i=m+1}^{k-1}\Big{(}H_{i}(\chi_{i-m})+(\theta_{1}+1)\ln\mu_{l+i-m}-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m}-\mu_{j})\Big{)}\Big{]}.$

Using the fact that the domain $\mathcal{D}^{\prime}$ is a product, this can
be further simplified in:

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}\sup_{\overline{\gamma}_{i}\geq
0\atop\sum_{i}\overline{\gamma}_{i}=1}\Big{[}\theta_{1}\ln\Big{(}\sum_{i=0}^{l+1}\mu_{i}\overline{\gamma}_{i}\Big{)}+\sum_{i=0}^{l}\alpha_{i}\ln\frac{\overline{\gamma}_{i}}{\alpha_{i}}\Big{]}$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=1}^{m}\Big{(}\sup_{\chi\in]\mu_{i-m+1},\mu_{i-m+2}[}H_{i}(\chi)+(\theta_{1}+1)\ln\mu_{i-m+1}-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{j}-\mu_{i-m+1})\Big{)}$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=m+1}^{k-1}\Big{(}\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)+(\theta_{1}+1)\ln\mu_{l+i-m+1}-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m+1}-\mu_{j})\Big{)}.$

Here, following the analysis of the same optimization problem performed in
subsection 3.1, the first line is going to be equal to
$J(\theta_{1},\mu_{l+1},\mu)$. Let us determine the variations of $H_{i}$ on
the different intervals considered. For $i=1,...,m$ we have that $H_{i}$ is
decreasing and therefore its supremum on $]\mu_{i-m-1},\mu_{i-m}[$ is
$H_{i}(\theta_{1},\mu_{i-m-1})$ and the term of index $i$ simplifies in
$J(\theta_{\sigma(i)},\mu_{i-m-1},\mu)$.

For $i=m+1,...,k-1$, if $\chi\leq T_{\mu}^{-1}(\theta_{\sigma(i)})$,
$\chi\mapsto J(\theta_{\sigma(i)},\chi,\mu)$ is constant and therefore:

$H_{i}^{\prime}(\chi)=-\frac{\theta_{1}+1}{\chi}+\sum_{j=0}^{l}\alpha_{j}\frac{1}{\chi-\mu_{j}}=G_{\mu}(\chi)-\frac{\theta_{1}+1}{\chi}.$

This derivative is positive or null if $\chi\leq T_{\mu}^{-1}(\theta_{1})$ and
negative or null if $\chi\geq T_{\mu}^{-1}(\theta_{1})$. Now, if $\chi\geq
T_{\mu}^{-1}(\theta_{\sigma(i)})$, we have

$H_{i}^{\prime}(\chi)=\frac{\theta_{\sigma(i)}-\theta_{1}}{\chi}.$

Let us look first at the terms of indices $i\geq p+1$, that is such that
$\theta_{\sigma(i)}\geq\theta_{1}$. Then, since
$T_{\mu}^{-1}(\theta_{\sigma(i)})\leq T_{\mu}^{-1}(\theta_{1})$, $H_{i}$ is
increasing and therefore, on any bounded intervals attains its supremum for
values that tends to its rightmost point, and we have:

$\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)=H(\mu_{l+i-m+1}).$

Using the expression of $H_{i}$, we have that the term of index $i$ in the
preceding sum simplifies in $J(\theta_{\sigma(i)},\mu_{l+i-m+1})$. We end up
with the simplified expression:

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}J(\theta_{1},\mu_{l+1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=1}^{m}J(\theta_{\sigma(i)},\mu_{i-m-1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=m+1}^{p}\Big{(}\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)+(\theta_{1}+1)\ln\mu_{l+i-m+1}-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m+1}-\mu_{j})\Big{)}$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=p+1}^{k-1}J(\theta_{\sigma(i)},\mu_{i-m+l+1},\mu).$

Now let us look at the variation of $H_{i}$ for $m+1\leq i\leq p$. Since
$0\leq\theta_{\sigma(i)}\leq\theta_{1}$, $T_{\mu}^{-1}(\theta_{1})\leq
T_{\mu}^{-1}(\theta_{\sigma(i)})$ and $H_{i}$ is increasing till
$T_{\mu}^{-1}(\theta_{1})$ and decreasing after. We let

$q=\sup\\{i\in\llbracket
m+1,p+1\rrbracket,T_{\mu}(\mu_{l+i-m})\geq\theta_{1}\\}$

We will assume here that $q\neq-\infty,p+1$, we leave those two edge cases to
the reader since they are in fact easier to deal with.

For every $i<q$ we have that:

$\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)=H_{i}(\mu_{l+i-m+1})=J(\theta_{\sigma(i)},\mu_{l+i-m+1},\mu)-(\theta_{1}+1)\ln\mu_{l+i-m+1}+\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m+1}-\mu_{j})$

Furthermore since $\theta_{\sigma(i)}\leq\theta_{1}\leq
T_{\mu}^{-1}(\mu_{l+i-m+1})$, we have
$J(\theta_{\sigma(i)},\mu_{l+i-m+1},\mu)=J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)$
and therefore the term of index $i$ in the sum above is
$J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)$.

For $i=q$, we have that

$\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)=H_{i}(T_{\mu}^{-1}(\theta_{1}))$

and therefore the term of index $i=q$ is:

$J(\theta_{\sigma(q)},T_{\mu}^{-1}(\theta_{1}),\mu)+(\theta_{1}+1)(\ln\mu_{l+q-m+1}-\ln
T_{\mu}^{-1}(\theta_{1}))-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+q-m+1}-\mu_{j})+\sum_{j=0}^{l}\alpha_{j}\ln(T_{\mu}^{-1}(\theta_{1})-\mu_{j})$

Furthermore, looking back at the expression of $J$, we have that the
expression above is:

$J(\theta_{\sigma(q)},\mu_{l+q-m},\mu)+J(\theta_{1},\mu_{l+q-m+1},\mu)-J(\theta_{1},T^{-1}_{\mu}(\theta_{1}),\mu).$

For every $i>q$, we have that

$\sup_{\chi\in]\mu_{l+i-m},\mu_{l+i-m+1}[}H_{i}(\chi)=H_{i}(\mu_{l+i-m}).$

And therefore, the term of index $i$ is:

$J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)+(\theta_{1}+1)(\ln\mu_{l+i-m+1}-\ln\mu_{l+i-m})-\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m+1}-\mu_{j})+\sum_{j=0}^{l}\alpha_{j}\ln(\mu_{l+i-m}-\mu_{j}).$

There again, this term is equal to:

$J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)+J(\theta_{1},\mu_{l+i-m+1},\mu)-J(\theta_{1},\mu_{l+i-m},\mu).$

Rearranging the terms in $J$, we have

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}J(\theta_{1},\mu_{l+1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=1}^{m}J(\theta_{\sigma(i)},\mu_{i-m-1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=m+1}^{q-1}J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)$
$\displaystyle+$ $\displaystyle
J(\theta_{\sigma(q)},T_{\mu}^{-1}(\theta_{1}),\mu)+J(\theta_{1},\mu_{l+q-m+1},\mu)-J(\theta_{1},T_{\mu}^{-1}(\theta_{1}),\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=q+1}^{p}J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)+J(\theta_{1},\mu_{l+i-m+1},\mu)-J(\theta_{1},\mu_{l+i-m},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=p+1}^{k-1}J(\theta_{\sigma(i)},\mu_{i-m+l+1},\mu)$

Since $\mu_{l+1}\leq T_{\mu}^{-1}(\theta_{1})$ (we assumed that $q>m+1$),
$J(\theta_{1},\mu_{l+1},\mu)=J(\theta_{1},T_{\mu}^{-1}(\theta_{1}),\mu)$ and
the sum above simplifies in

$\displaystyle J(\bar{\theta},\\{\mu_{i}\\},\\{\alpha_{i}\\})$
$\displaystyle=$ $\displaystyle\frac{\beta}{2}J(\theta_{1},\mu_{l+p-m+1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=1}^{m}J(\theta_{\sigma(i)},\mu_{i-m-1},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=m+1}^{q-1}J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)$
$\displaystyle+$ $\displaystyle J(\theta_{\sigma(q)},\mu_{l+q-m},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=q+1}^{p}J(\theta_{\sigma(i)},\mu_{l+i-m},\mu)$
$\displaystyle+$
$\displaystyle\frac{\beta}{2}\sum_{i=p+1}^{k-1}J(\theta_{\sigma(i)},\mu_{i-m+l+1},\mu)$

which is the expected limit and therefore the lemma is proved.

∎

For $\theta_{1}\leq 0$, the computation remain the same except that the role
of the indices $i$ between $1$ and $m$ and the role of the indices $i$ between
$m+1$ and $k$ are switched.

###  4.2. Case of a general $\mu$

To generalize our result, one first need a multi-dimensional version of the
continuity lemma 3.2:

######  Lemma 4.3.

Let $\kappa>0$. The function

$(\theta,\lambda,\mu)\mapsto\sum_{i=1}^{k}J(\theta_{i},\lambda_{i},\mu)$

defined on the following domain:

$\mathcal{D}^{m,k-m}:=\\{(\theta,\lambda,\mu)\in(\mathbb{R}^{-})^{m}\times(\mathbb{R}^{+})^{k-m}\times(\mathbb{R}^{+,*})^{k}\times\mathcal{P}(\mathbb{R}^{+}):\lambda_{1}\leq...\leq\lambda_{m}\leq
l(\mu)\leq r(\mu)\leq\lambda_{m+1}\leq...\leq\lambda_{k}\leq\kappa\\}$

endowed with the topology induced by the product topology, is continuous.

This lemma can be easily be proved from Lemma 3.2. Then, using again
continuity arguments, we can see that the result of the previous subsection
still stand for sequences of matrices $(X_{N})_{N\in\mathbb{N}}$ of the same
form where we relaxed the conditions on the $\mu_{i}$ to only
$\mu_{-m}\leq\mu_{-m+1}\leq...\leq\mu_{l+k-m}$. Then for general eigenvalue
distribution $\mu$ we proceed the same way as in subsection 3.2, by
approximating uniformly the eigenvalues.

##  5\. Conclusion and open questions

In this note, we proved one of the main results of [14] in the case
$\beta=1,2$ as well as the conjecture given as the last Remark of section 2 of
[14]. An interesting thing that this result teaches us is that, although the
lack of symmetry of the integral in the $\theta_{i}$ disappears asymptotically
for parameters $\bar{\theta}$ whose support lie on the $k$ first indices, it
is no more the case if the support is not restricted to $\llbracket
1,k\rrbracket$, even if it remains finite. To see this, one can take some
sequence $X_{N}$ satisfying the hypothesis of Theorem 2.1 whose largest
eigenvalue converges toward $\lambda$ and whose eigenvalue distribution
converges toward $\mu$, and let $\theta=(1,0,...,0)$ and
$\theta^{\prime}=(0,....,0,1)$. then, one can write that

$\Delta_{\frac{\beta
N}{2}\theta^{\prime}}(U^{*}X_{N}U)=\Big{(}\frac{\det(U^{*}X_{N}U)}{\det([U^{*}X_{N}U]_{N-1})}\Big{)}^{\frac{\beta
N}{2}}=(U^{*}X^{-1}_{N}U)_{1,1}^{-\frac{\beta N}{2}}$

and therefore:

$I_{N}(\theta^{\prime},X_{N})=I_{N}(-\theta,X_{N}^{-1})$

Thus

$\lim_{N\to\infty}N^{-1}\log
I_{N}(\theta^{\prime},X_{N})=\frac{\beta}{2}J\Big{(}-1,\lambda^{-1},\frac{1}{\mu}\Big{)}$

where $1/\mu$ is the push-forward of $\mu$ by $x\mapsto 1/x$. But, since
$-1\geq T_{1/\mu}(1/\lambda)$, one has:

$J\Big{(}-1,\lambda^{-1},\frac{1}{\mu}\Big{)}=-\int\log|x|d\frac{1}{\mu}(x)=\int\log
xd\mu(x)$

This quantity is in general different from $J(1,\lambda,\mu)$ (since in
particular, it never depends on $\lambda$ whatever its value or the value of
$\mu$). An interesting result would be to study given $\theta\in\mathbb{R}$
and $\theta^{(M)}=(0,...,0,\underbrace{\theta}_{\text{position }M},....0)$,
the behavior of $N^{-1}\log I_{N}(\theta^{(M(N))},X_{N})$ depending of the
behavior of the sequence $M(N)$.

Another natural question already asked by Mergny and Potters is the question
of the behavior of the integral for arguments $\bar{\theta}$ whose support
tends to $\infty$. For instance, if
$\bar{\theta}^{N}=(\theta_{1}^{N},...,\theta_{k(N)}^{N})$ with $k(N)$ some
sequence whose limit is $+\infty$, can we prove the convergence of
$(k(N)N)^{-1}\log I_{N}(\bar{\theta}^{N},X_{N})$ toward some limit and then
does the lack of symmetry in $\theta$ disappears at $\infty$?

## References

  * [1] Giulio Biroli and Alice Guionnet.  Large deviations of spiked random matrices.  To appear in ECP, 2020.
  * [2] Cesar Cuenca.  Pieri integral formula and asymptotics of jack unitary characters, 2017\.
  * [3] Cesar Cuenca.  Asymptotic formulas for macdonald polynomials and the boundary of the (q,t)-gelfand-tsetlin graph.  Symmetry, Integrability and Geometry: Methods and Applications, Jan 2018.
  * [4] Bertrand Eynard.  Eigenvalue distribution of large random matrices, from one matrix to several coupled matrices.  Nuclear Physics B, 506(3):633–664, Dec 1997.
  * [5] Vadim Gorin and Greta Panova.  Asymptotics of symmetric polynomials with applications to statistical mechanics and representation theory.  The Annals of Probability, 43(6), Nov 2015.
  * [6] Ian P. Goulden, Mathieu Guay-Paquet, and Jonathan Novak.  Monotone hurwitz numbers and the hciz integral.  Annales Mathématiques Blaise Pascal, 21(1):71–89, 2014.
  * [7] Alice Guionnet and Jonathan Husson.  Large deviations for the largest eigenvalue of Rademacher matrices.  Ann. Probab., 48(3):1436–1465, 2020.
  * [8] Alice Guionnet and Jonathan Husson.  Asymptotics of k dimensional spherical integrals and applications.  arXiv:2101.01983, 2021.
  * [9] Alice Guionnet and Mylène Maıda.  A fourier view on the r-transform and related asymptotics of spherical integrals.  Journal of functional analysis, 222(2):435–490, 2005.
  * [10] Harish-Chandra.  Invariant differential operators on a semisimple Lie algebra.  Proc. Nat. Acad. Sci. U.S.A., 42:252–253, 1956.
  * [11] G. J. Heckman and E. M. Opdam.  Root systems and hypergeometric functions. I.  Compositio Mathematica, 64(3):329–352, 1987.
  * [12] Jonathan Husson.  Large deviations for the largest eigenvalue of matrices with variance profiles.  arXiv:2002.01010, 2020.
  * [13] Claude Itzykson and Jean-Bernard Zuber.  The planar approximation. II.  J. Math. Phys., 21:411–421, 1980.
  * [14] Pierre Mergny and Marc Potters.  Asymptotic behavior of the multiplicative counterpart of the harish-chandra integral and the $s$-transform.  arXiv:2007.09421, 2021.
  * [15] Jonathan Novak.  On the complex asymptotics of the hciz and bgw integrals.  arXiv:2006.04304, 2020.
  * [16] Eric M. Opdam.  Lecture notes on Dunkl operators for real and complex reflection groups, volume 8.  Tokyo: Mathematical Society of Japan, 2000.


# A Feature Set of Small Size for the PDF Malware Detection

Ran Liu  Univ. of Maryland, Baltimore County<EMAIL_ADDRESS>and  Charles
Nicholas  Univ. of Maryland, Baltimore County<EMAIL_ADDRESS>
###### Abstract.

Machine learning (ML)-based malware detection systems are becoming
increasingly important as malware threats increase and get more sophisticated.
PDF files are often used as vectors for phishing attacks because they are
widely regarded as trustworthy data resources, and are accessible across
different platforms. Therefore, researchers have developed many different PDF
malware detection methods. Performance in detecting PDF malware is greatly
influenced by feature selection. In this research, we propose a small features
set that don’t require too much domain knowledge of the PDF file. We evaluate
proposed features with six different machine learning models. We report the
best accuracy of 99.75% when using Random Forest model. Our proposed feature
set, which consists of just 12 features, is one of the most conciseness in the
field of PDF malware detection. Despite its modest size, we obtain comparable
results to state-of-the-art that employ a much larger set of features.

††copyright: none

##  1\. Introduction

The flexibility and portability of PDF files makes them a popular target for
malware attacks. Over time, different approaches have been proposed to detect
PDF malware. Machine learning and neural network based models have
particularly shown promise in these detection tasks. However, the performance
of the model relies on the quality of the feature set chosen(Liu et al.,
2023). Features used in malware detection are grouped into two categories:
dynamic and static. Dynamic features are obtained from monitoring program
execution, such as APIs called, instructions executed, or IP addresses
accessed. Conversely, static features are obtained through static analysis.
Both categories have some limitations. Dynamic features need to be executed in
a sandbox environment, where some sophisticated malware can detect the sandbox
environment and consequently alter their behaviors. Static features, on the
other hand, can be obfuscated by attackers using evasion techniques, making
the detection challenging. This raises concerns that some commonly used
features have been thoroughly investigated by attackers. If attackers have
exploited these features to perform a successful evasive attack, PDF malware
detection systems built on the same or similar features set might become
vulnerable. This highlights the importance of the usage of PDF-specific
features, which may reduce the attack surface. Earlier research, including
PDFRate, have employed some PDF-specific features such as the number and the
occurrence of a specific PDF objects for model training, which obtained
promising accuracy in PDF malware detection(Parkour, [n. d.])(Maiorca et al.,
2015). Nevertheless, most of these features requires a large amount of domain
knowledge to extract. Moreover, their feature sets are large and complex,
which may potentially lead to over-fitting. Consequently, it’s desirable to
have a simple and small PDF-specific features set that may achieve detection
accuracy comparable to more complex features.

In this paper, we limit the scope of PDF-specific features to those that are
unique to PDF files, hence excluding most dynamic features such as system call
sequences, API call sequences, and some static features such as binary code.
Furthermore, we exclude features that need extensive domain knowledge for PDF
files, which means that most keyword-based features, including JavaScript code
in PDF files, are not used in our research.

PDF files can be viewed as a set of interconnected objects. Some work, such as
Hidost’s, extracts the tree structure of the PDF and uses the binary counts
for these paths as features(Šrndić and Laskov, 2016). Our previous work showed
that such tree structures contain sequential relationships and can be used to
train the Time Series Model for PDF malware detection. In this paper, we
propose a novel set of graph features to accurately detect PDF malware. We
investigated multiple types of graph tree features by parsing a PDF file into
tree representative features. For specific feature types, our research
demonstrated statistical differences between benign and malicious PDFs. Using
the proposed feature set to train a machine learning model, we show
empirically that the model can successfully detect 99.75% of PDF malware
samples with only 12 features. The primary contributions of our study are the
introduction of one of the smallest PDF specific feature sets. We have
conducted a thorough investigation and performance analysis of the ML - models
based on these proposed features. Furthermore, we benchmarked our results
against state-of-the-arts, indicating that our feature set is promising.

###  1.1. PDF file Structure

A PDF file is structured using interconnected modules known as objects, which
are made up of four parts: a header, a body, a cross-reference table, and a
trailer, as shown in Figure 1.

  * •

Header: The header contains information about the PDF version and is marked
with the ’%’ symbol.

  * •

Body: The body, as the primary section of the PDF file, consists of objects
that define all the operations performed by the file. These objects, which
include both indirect and direct objects, characterize the functionality of
each object using keywords marked with ’/’. For example, ’/Length’ and
’/Filter’ are such keywords. Indirect objects, which start with a numeric
identifier like ’4 0 R’, contain information in a directory and can be
referenced by other objects. For example, an object starting with ’1 0 R’ can
be referred to by other objects using ’1’, the sequence number. This structure
allows for the interconnection of objects. The generation number, usually set
to 0, is represented by the second digit, although it can be other number in
some cases. Objects are usually end with the ’endobj’ marker. A specific type
of object, known as a stream, starts with the keyword ’stream’ and ends with
’endstream’ and ’endobj’. The content of the stream object, which includes
elements like images and texts, is encoded using filters.

  * •

Cross-reference table: This table contains the location references for each
object. A PDF parser uses this table to find the object reference in the
memory for parsing. The cross-reference table, marked by ’Xref’ followed by
numbers, indicates the total number of objects in the references with its last
number. For instance, ’0 16’ indicates a total of 16 objects in the cross-
reference table.

  * •

Trailer: trailer contains information about the file such as the number of
objects using keyword ’/Size’. It also contains a reference to the root object
using keyword /Root and metadata using keyword /Info. The file structure
organizes the logical access order for the PDF file. When a PDF reader
application accesses a PDF file, it first locates the trailer to find the root
object. Then the parser uses a cross-reference table to parse each indirect
object to decompress all data. In this way, the content of the PDF is made
visible to the user. When a modification happens in the PDF file, like
inserting a page into the PDF, a new body, trailer, and cross-reference table
will be appended to the original file accordingly, and a new version number
will be generated. However, because the cross-reference table sets a strict
boundary for each object, removing objects from the previous version can cause
errors. Thus, an attacker is more likely to add features instead of removing
features.

Each object is labeled with a number, allowing it to be referenced by other
objects. Object information is stored in the cross-reference table, while the
trailer indicates the root object’s number and the cross-reference table’s
location. By querying the cross-reference table, catalog objects can be found.
The catalog object serves as the entire document’s root object, containing the
PDF document’s outline and the page group object’s reference.

Figure 1. PDF Structure

###  1.2. Related Work

In the field of machine learning-based PDF malware detection, two primary
types of features are commonly used - static and dynamic features. The dynamic
features are obtained by running the PDF in a controlled environment, which
allows for the collection of PDF running behaviors such as sequences of system
calls and API calls(Rieck et al., [n. d.])(Liu and Nicholas, 2023). However,
most dynamic features are typically not distinctive to PDF files, and the
building of a robust sandbox environment increases the complexity of the
detection process. Despite the promising results in malware detection tasks
using dynamic features, our work primarily focuses on the use of static
features. Previously used dynamic and static features require significant
amounts of domain knowledge for feature extraction. In contrast, our goal is
to employ features that demand minimal domain knowledge and are unique to PDF
files.

There are three categories of static features. The first type of static
features are obtained from a keyword-based analysis, which involves searching
for predefined keywords such as ’/Javascript’, ’/OpenAction’, ’/GoTo’, ’/URI’,
and ’/RichMedia’. These keywords are often associated with malicious code
injection. Features can include the number of keywords or simply their
presence. The malicious payloads are usually inserted into objects associated
with such keywords, making them useful features for PDF malware detection. The
second type of static features are obtained through a tree structure-based
analysis. This method constructs the object tree representation to capture the
connections between items. The tree structure can provide insights into the
hierarchical connections between the objects of the PDF file, which may reveal
malware-related behaviors. Lastly, static features can be obtained using code-
based analysis, which focuses on malicious strings and functions in Javascript
code. As PDF malware often manipulates Javascript code to execute malicious
activities, the presence of specific code strings or functions may indicate
the presence of malware. PDF malware detectors may employ one or more of the
features described above. One such example is Hidost(Šrndić and Laskov, 2016),
a system that uses the Poppler PDF parser(FreeDesktop.org., 2018) to extract
tree structural paths of objects in a PDF file, which are then used as
features in the classification process. Hidost is implemented with two
different models: Support Vector Machine (SVM) and Random Forest (RF)(Šrndic
and Laskov, 2013)(Šrndić and Laskov, 2016). SVM is a supervised learning model
that creates an optimal hyperplane to separate different labels. RF is a meta-
estimator that integrates different decision trees to improve classification
accuracy. The researchers trained their model using a dataset of 10,000
randomly selected files, maintaining a malicious-to-benign ratio of 1:1. The
complete PDF dataset comprised 407,037 benign and 32,567 malicious files. The
Hidost system has 99.8% accuracy and less than 0.06% false positive rate for
both models.

The PDFrate classifier is implemented using an RF algorithm with 99% accuracy
and 0.2% false positive rate over the Contagio malware dataset(Parkour, [n.
d.]). PDFrate uses the metadata, which includes the names of the files’
authors, the size of the file, its location, and the amount of certain
keywords, and the content of the PDF files as features. The authors manually
define the feature set, which has 202 features in all, including counts for
different keywords and specific fields in the PDF. Examples include the number
of characters in the author field, the quantity of “endobj” keywords, the
total number of pixels in all the photos, the quantity of JavaScript markers,
etc. The Mimicus implementation of PDFrate, claiming to get a close
approximation, only makes use of 135 of these features. The two versions of
PDFrate, PDFrate-v1 and PDFrate-v2, each use a different machine learning
model(Yerima et al., 2022)(Smutz and Stavrou, 2016). The classifiers in
PDFrate-v2 use mutual agreement to implement an ensemble technique. The term
“uncertain” is introduced into the classifier voting, where rates of 25–50%
are regarded as benign uncertainty and rates of 50–75% as malicious
uncertainty.

PjScan is a tool that concentrates on examining JavaScript code(Laskov and
Šrndić, 2011). It uses Poppler(FreeDesktop.org., 2018) as a parser to extract
tokens from JavaScript code, and a one-class SVM as a classifier. PjScan
achieved 85% detection accuracy. Malware Slayer is available in two
variants(Maiorca et al., 2012)(Maiorca et al., 2015). The original Slayer
extracts keyword features from PDF files using a pattern recognition method
and labels samples using a random forest algorithm. Slayer NEO uses the
PeePDF(Tonn and Bandla, 2013) and Origami(Sogeti ESEC Lab, 2015) parsers to
extract structural data as features and the AdaBoost algorithm for
classification.

Maryam et al. proposed a PDF malware detection system based on stacking
learning(Issakhani et al., 2022). Their approach is based on the idea that
combining different classifiers could produce improved accuracy as each
classifier operates based on unique data assumptions. Their feature set
included ten general features, such as PDF size and title character count, as
well as structural features such as the amount of keywords and objects. These
extracted features were initially fed into a base layer consisting of SVM,
Random Forest, MLP, and AdaBoost. The prediction outputs from this layer were
subsequently fed into a meta-layer featuring Logistic Regression, K-Nearest
Neighbors, and Decision Trees. Their reported metrics on the hybird dataset
including Contagio dataset were impressive: an accuracy of 99.98%, precision
of 99.84%, recall of 99.89%, and an F1 score of 99.86%.

##  2\. Statistical Analysis of a Feature Set

We now introduce our proposed feature set. We use the Contagio dataset, which
includes 9,000 benign and 10,982 malicious PDF samples, to extract
features(Parkour, [n. d.]). This dataset was chosen due to its accessibility
and its large number of labeled samples. We used the pdfrw library to extract
tree structure paths for each file, and while some samples were corrupted, we
were able to successfully extract path objects from 7,396 benign PDF files and
10,814 malicious PDF files(Xu et al., 2016). Our feature selection strategy is
to minimize required domain knowledge during feature extraction. Consequently,
despite the promising results achieved by keyword-based features in other
research, we chose not to use them. We have already noted that a PDF can be
represented as a tree structure of objects, prompting our investigation into
graph features. Our selection of features was facilitated by comparing mean
values, standard deviations with 95% CI and Quantiles. This led us to select
the following features:

  * •

Distribution of children per node: the average (avg children), median (median
children) and variance of children per node (var children).

  * •

Number of leaves in the tree (num leaves).

  * •

Number of nodes (num nodes).

  * •

The depth of the tree (depth).

  * •

Average degree (avg degree).

  * •

Degree assortativity coefficient (degree assortativity).111Degree
assortativity refers to the tendency for nodes of high or low degree to be
connected to other nodes of high or low degree, respectively.

  * •

The average shortest path length (avg shortest path).

  * •

How nodes in a graph tend to cluster together (avg clustering coefficient).

  * •

Graph density (density).

We applied statistical analysis to investigate the proposed features of benign
and malicious PDF files. The key statistical metrics in our investigation
were: the 75% quartile, median 50%, 25% quartile, 95% CI mean and 95% CI
standard deviation.

Table 1. Quantiles for the Proposed Feature Set for Benign PDFs Benign PDF | Quantiles |
---|---|---
| 75% | 50% | 25%
avg children | 1.9416 | 1.5218 | 1.44
avg clustering coefficient | 0.0640 | 0.0185 | 0.097
avg degree | 3 | 3 | 2
avg shortest path | 1.0667 | 0.7118 | 0.5421
degree assortativity | -0.3781 | -0.4298 | -0.5569
density | 0.0215 | 0.0083 | 0.0068
depth | 4 | 4 | 4
median children | 0 | 0 | 0
num edges | 459 | 309 | 189
num leaves | 188 | 158 | 71
num nodes | 260 | 199 | 135.25
var children | 158.2680 | 94.4440 | 51.1347
Table 2. Quantiles for the Proposed Feature Set for Malicious PDFs Malicious PDF | Quantiles |
---|---|---
| 75% | 50% | 25%
avg children | 1.7619 | 1.5758 | 1.5714
avg clustering coefficient | 0.0698 | 0.0413 | 0.0374
avg degree | 3 | 3 | 3
avg shortest path | 0.6 | 0.5595 | 0.4542
degree assortativity | -0.2821 | -0.3126 | -0.3405
density | 0.0881 | 0.0786 | 0.0686
depth | 4 | 4 | 4
median children | 1 | 1 | 1
num edges | 40 | 37 | 33
num leaves | 10 | 10 | 10
num nodes | 27 | 21 | 21
var children | 4.1814 | 4.0272 | 3.6734
Table 3. 95% CI Mean Comparison for the Proposed Feature Set | Benign | Malicious
---|---|---
| 95% CI Mean | 95% CI Mean
avg children | 1.8601 - 1.9044 | 1.6359 - 1.6469
avg clustering coefficient | 0.0372 - 0.0392 | 0.0501 - 0.0512
avg degree | 3.1385 - 3.2299 | 2.8545 - 2.8762
avg shortest path | 0.8079 - 0.8279 | 0.5430 - 0.5498
degree assortativity | -0.4589 - -0.4541 | -0.3071 - -0.3033
density | 0.0205 - 0.0213 | 0.0764 - 0.0772
depth | 4.2561 - 4.2831 | 3.9594 - 3.9692
median children | 0.1506 - 0.1942 | 0.9405 - 0.9612
num edges | 441.5762 - 471.6670 | 43.8435 - 45.6364
num leaves | 152.1234 - 157.9772 | 12.2298 - 12.7902
num nodes | 201.8999 - 209.8423 | 25.4634 - 26.2143
var children | 114.7908 - 122.0534 | 4.8081 - 5.34606
Table 4. 95% CI Standard Deviation Comparison for the Proposed Feature Set | Benign | Malicious
---|---|---
| 95% CI SD | 95% CI SD
avg children | 0.9561 - 0.9874 | 0.2881 - 0.2959
avg clustering coefficient | 0.0427 - 0.0441 | 0.0274 - 0.0282
avg degree | 1.9738 - 2.0385 | 0.5662 - 0.5815
avg shortest path | 0.4358 - 0.4500 | 0.1759 - 0.1806
degree assortativity | 0.1097 - 0.1133 | 0.1005 - 0.1032
density | 0.0230 - 0.0238 | 0.02047 - 0.0210
depth | 0.5867 - 0.6059 | 0.2579 - 0.2648
median children | 0.9462 - 0.9772 | 0.5416 - 0.5563
num edges | 649.6877 - 670.9714 | 46.9322 - 48.2001
num leaves | 126.3856 - 130.5260 | 14.6701 - 15.0664
num nodes | 171.4772 - 177.0948 | 19.6561 - 20.1871
var children | 156.8088 - 161.9459 | 14.0821 - 14.4626

Note that Table 1, Table 3, Table 2 and Table 4 show a significant difference
in the proposed feature set between benign and malicious PDFs. The statistical
difference between benign and malicious PDFs is further visualized in the box
plots ( Figure 2 and Figure 3), providing clear graphical representations of
these differences. We excluded the median degree, depth, and median children
from the box plots since they are discrete values.

Figure 2. Box plot of the benign PDFs. Features are on the y-axis.  Figure 3.
Box plot of the malicious PDFs. Features are on the y-axis.

##  3\. Experimental Results

###  3.1. PDF Malware Detection Results

Having shown a significant difference in the proposed feature set between
benign and malicious PDFs, we move on to demonstrate the feature set’s utility
in experimental evaluation. we use the Contagio dataset and the pdfrw library
to extract tree structure paths for each file, and (as mentioned above) we
were able to successfully extract path objects from the 7,396 benign PDF files
and 10,814 malicious PDF files(Xu et al., 2016). We use scikit-learn to
evaluate the model’s performance(Pedregosa et al., 2012). The confusion matrix
we used is shown as Table 5

Table 5. Confusion Matrix. | Detected as Benign | Detected as Malicious
---|---|---
Benign | True Positives (TP) | False Negatives (FN)
Malicious | False Positives (FP) | True Negatives (TN)

We apply 5-fold cross validation to 6 machine learning classifiers trained
with the proposed 12 graph features. To show that our results are
statistically significant, we report 95% Confidence Interval (95% CI) of the
Precision, Recall in Table 6 and 95% CI F1 score in Table 7. The recall value
indicates the ratio of samples have been correctly classified. We report the
best recall value for the malicious class with Random Forest, indicating that
99.73% to 99.77% of the malicious samples are correctly detected, while the
benign class’s recall value is 0.9987 - 0.9991, which means that 99.87% to
99.91% of benign samples are correctly classified.

Table 6. 95% CI Precision and Recall for each label. Classifier | Label | Precision | Recall
---|---|---|---
XGBoost | Malicious | 0.9939 - 0.9977 | 0.9958 - 0.9966
Benign | 0.9982 - 0.9987 | 0.9973 - 0.9991
Naive Bayes | Malicious | 0.8769 - 0.9564 | 0.8000 - 0.9750
Benign | 0.8698 - 0.9683 | 0.7850 - 0.9824
Multi-layer Perceptron | Malicious | 0.9871 - 0.9911 | 0.9811 - 0.9940
Benign | 0.9946 - 0.9963 | 0.9946 - 0.9963
Decision Tree (J48) | Malicious | 0.9963 - 0.9973 | 0.9946 - 0.9982
Benign | 0.9972 - 0.9980 | 0.9959 - 0.9986
Random Forest | Malicious | 0.9966 - 0.9982 | 0.9973 - 0.9977
Benign | 0.9987 - 0.9991 | 0.9987 - 0.9991
Simple Logistic | Malicious | 0.9739 - 0.9879 | 0.9820 - 0.9824
Benign | 0.9878 - 0.9885 | 0.9831 - 0.9917
Table 7. 95% CI F1 Score for each label. Classifier | Label | F1 Score
---|---|---
XGBoost | Malicious | 0.9953 - 0.9968
Benign | 0.9980 - 0.9986
Naive Bayes | Malicious | 0.8712 - 0.9234
Benign | 0.8671 - 0.9227
Multi-layer Perceptron | Malicious | 0.9861 - 0.9906
Benign | 0.9946 - 0.9963
Decision Tree (J48) | Malicious | 0.9959 - 0.9972
Benign | 0.9970 - 0.9979
Random Forest | Malicious | 0.9970 - 0.9979
Benign | 0.9987 - 0.9991
Simple Logistic | Malicious | 0.9781 - 0.9849
Benign | 0.9854 - 0.9901

We report the Accuracy, True Positive Rate (TPR), False Positive Rate (FPR),
False Negative Rate (FNR) and True Negative Rate (TNR) in Table 8. The result
indicates our proposed features have good overall performance. We report the
best accuracy of 0.9975 when using Random Forest model.

Table 8. Accuracy, True Positive Rate (TPR), False Positive Rate (FPR), False Negative Rate (FNR) and True Negative Rate (TNR) with the Proposed Features Set. Classifier | Accuracy | TPR | FPR | FNR | TNR
---|---|---|---|---|---
XGBoost | 0.9964 | 0.9980 | 0.0042 | 0.0020 | 0.9958
Naive Bayes | 0.9006 | 0.7850 | 0.0268 | 0.2150 | 0.9732
Multi-layer Perceptron | 0.9926 | 0.9919 | 0.0088 | 0.0081 | 0.9912
Decision Tree | 0.9967 | 0.9959 | 0.0019 | 0.0041 | 0.9981
Random Forest | 0.9975 | 0.9980 | 0.0023 | 0.0020 | 0.9977
Simple Logistic | 0.9822 | 0.9817 | 0.0180 | 0.0183 | 0.9820

###  3.2. Comparison with Other Works

For model performance comparison, we select models with reported results in
the literature for direct comparison.

Table 9. Comparison with other models. Model | Precision | Recall | F1-Score | Accuracy
---|---|---|---|---
Model(Yerima et al., 2022) | 0.9970 | 0.9970 | 0.9970 | 0.9965
Model(Abu Al-Haija et al., 2022) | 0.9880 | 0.9890 | 0.9885 | 0.9884
Model(Issakhani et al., 2022) | 0.9888 | 0.9887 | 0.9877 | 0.9869
Our work | 0.9973 | 0.9974 | 0.9974 | 0.9975

We report the best result obtained by applying the Random Forest model. Table
9 presents the comparison of the evaluation metrics to other work. The results
show that our work beats other work while being significantly smaller than the
feature set they use.

The main weaknesses of the feature set we propose is that it is vulnerable to
some evasive attacks. Detectors that employ this feature set can be
compromised through the insertion, deletion, or alteration of a subtree. In
order to enhance the robustness of the detector, one potential strategy is to
enrich the feature set and diversify the feature types employed. We observe
that the parser did not successfully parse all objects for some malware
specimens. The effectiveness of the our approach is influenced by the quality
of parsed PDF objects.

##  4\. Conclusion

In this work, we introduced a new features set for PDF malware detection that
based on PDF tree structure. Our work aimed to address the need of finding a
small features set without needing too much domain knowledge of the PDF file.
Our work might serve as a baseline for the future investigation. We do not
expect our work can replace the current used static and dynamic features now
or in the future, but our work might inspire researchers with an alternative
way to build a malware detection model. In the future task, we plan to explore
the other features to improve the overall performance and enhance the
robustness.

## References

  * (1)
  * Abu Al-Haija et al. (2022) Qasem Abu Al-Haija, Ammar Odeh, and Hazem Qattous. 2022\.  PDF Malware Detection Based on Optimizable Decision Trees.  _Electronics_ 11, 19 (9 2022), 3142\.  https://doi.org/10.3390/electronics11193142
  * FreeDesktop.org. (2018) FreeDesktop.org. 2018\.  Poppler.
  * Issakhani et al. (2022) Maryam Issakhani, Princy Victor, Ali Tekeoglu, and Arash Lashkari. 2022.  PDF Malware Detection based on Stacking Learning. In _Proceedings of the 8th International Conference on Information Systems Security and Privacy_. SCITEPRESS - Science and Technology Publications, 562–570.  https://doi.org/10.5220/0010908400003120
  * Laskov and Šrndić (2011) Pavel Laskov and Nedim Šrndić. 2011.  Static detection of malicious JavaScript-bearing PDF documents. In _Proceedings of the 27th Annual Computer Security Applications Conference_. ACM, New York, NY, USA, 373–382.  https://doi.org/10.1145/2076732.2076785
  * Liu et al. (2023) Ran Liu, Maksim Eren, and Charles Nicholas. 2023.  Can Feature Engineering Help Quantum Machine Learning for Malware Detection?  (5 2023).
  * Liu and Nicholas (2023) Ran Liu and Charles Nicholas. 2023.  _IMCDCF: An Incremental Malware Detection Approach Using Hidden Markov Models_.  Technical Report. MALWARE TECHNICAL EXCHANGE MEETING (MTEM) 2021.
  * Maiorca et al. (2015) Davide Maiorca, Davide Ariu, Igino Corona, and Giorgio Giacinto. 2015.  _A structural and content-based approach for a precise and robust detection of malicious PDF files; A structural and content-based approach for a precise and robust detection of malicious PDF files_.  Technical Report.  http://www.mozilla.org/rhino
  * Maiorca et al. (2012) Davide Maiorca, Giorgio Giacinto, and Igino Corona. 2012\.  A Pattern Recognition System for Malicious PDF Files Detection.  510–524.  https://doi.org/10.1007/978-3-642-31537-4{_}40
  * Parkour ([n. d.]) Mila Parkour. [n. d.].  Malicious Documents Archive for Signature Testing and Research - Contagio Malware Dump.
  * Pedregosa et al. (2012) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Andreas Müller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2012.  Scikit-learn: Machine Learning in Python.  (1 2012).
  * Rieck et al. ([n. d.]) Konrad Rieck, Philipp Trinius, Carsten Willems, and Thorsten Holz. [n. d.].  _Automatic Analysis of Malware Behavior using Machine Learning_.  Technical Report.  http://www.iospress.nl,
  * Smutz and Stavrou (2016) Charles Smutz and Angelos Stavrou. 2016.  When a Tree Falls: Using Diversity in Ensemble Classifiers to Identify Evasion in Malware Detectors. In _Proceedings 2016 Network and Distributed System Security Symposium_. Internet Society, Reston, VA.  https://doi.org/10.14722/ndss.2016.23078
  * Sogeti ESEC Lab (2015) Sogeti ESEC Lab. 2015\.  Origami.
  * Šrndic and Laskov (2013) Nedim Šrndic and Pavel Laskov. 2013.  Detection of Malicious PDF Files Based on Hierarchical Document Structure. In _Proceedings of the 20th Annual Network & Distributed System Security Symposium_. Citeseer, 1–16.
  * Šrndić and Laskov (2016) Nedim Šrndić and Pavel Laskov. 2016.  Hidost: a static machine-learning-based detector of malicious files.  _EURASIP Journal on Information Security_ 2016, 1 (12 2016), 22.  https://doi.org/10.1186/s13635-016-0045-0
  * Tonn and Bandla (2013) Trevor Tonn and Kiran Bandla. 2013.  PhoneyPDF.
  * Xu et al. (2016) Weilin Xu, Yanjun Qi, and David Evans. 2016.  Automatically Evading Classifiers: A Case Study on PDF Malware Classifiers. In _Proceedings 2016 Network and Distributed System Security Symposium_. Internet Society, Reston, VA.  https://doi.org/10.14722/ndss.2016.23115
  * Yerima et al. (2022) Suleiman Y. Yerima, Abul Bashar, and Ghazanfar Latif. 2022\.  Malicious PDF detection Based on Machine Learning with Enhanced Feature Set. In _2022 14th International Conference on Computational Intelligence and Communication Networks (CICN)_. IEEE, 486–491.  https://doi.org/10.1109/CICN56167.2022.10008374