# A modification of the Prudnikov and Laguerre polynomials

S. Yakubovich
Department of Mathematics, Fac. Sciences of University of Porto,
Rua do Campo Alegre, 687; 4169-007 Porto (Portugal)

###### Abstract.

A two-parameter sequence of orthogonal polynomials
$\\{P_{n}(x;\lambda,t)\\}_{n\geq 0}$ with respect to the weight function
$x^{\alpha}e^{-\lambda x}\rho_{\nu}(xt),\ \alpha>-1,\ \lambda,t\geq 0,\
\rho_{\nu}(x)=2x^{\nu/2}K_{\nu}(2\sqrt{x}),\ x>0,\nu\geq 0$, where
$K_{\nu}(z)$ is the modified Bessel function, is investigated. The case
$\lambda=0$ corresponds to the Prudnikov polynomials and $t=0$ is related to
the Laguerre polynomials. A special one-parameter case
$\\{P_{n}(x;1-t,t)\\}_{n\geq 0},\ t\in[0,1]$ is analyzed as well.

###### Key words and phrases:

Orthogonal polynomials, modified Bessel function, Mellin transform, associated
Laguerre polynomials

###### 2000 Mathematics Subject Classification:

33C47, 33C10, 44A15, 42C05

E-mail<EMAIL_ADDRESS>
The work was partially supported by CMUP, which is financed by national funds
through FCT(Portugal), under the project with reference UIDB/00144/2020.

##  1\. Introduction and preliminary results

Recently [7] the author gave an interpretation of the Prudnikov sequence of
orthogonal polynomials with the weight function $x^{\alpha}\rho_{\nu}(x)$,
where $\rho_{\nu}(x)=2x^{\nu/2}K_{\nu}(2\sqrt{x}),\ x>0,\nu\geq 0,\ \alpha>-1$
and $K_{\nu}(z)$ is the modified Bessel function [1], Vol. II, in terms of the
so-called composition orthogonality with respect to the measure
$x^{\nu}e^{-x}dx$ related to the Laguerre polynomials [2]. Our main goal here
is to characterize a two-parameter sequence of orthogonal polynomials
$\\{P_{n}(x;\lambda,t)\\}_{n\geq 0}$ with respect to the weight function
$x^{\alpha}e^{-\lambda x}\rho_{\nu}(xt),\ \alpha>-1,\ \lambda,t\geq 0$,
satisfying the following orthogonality conditions

$None$ $\int_{0}^{\infty}P_{m}(x;\lambda,t)P_{n}(x;\lambda,t)e^{-\lambda
x}\rho_{\nu}(xt)x^{\alpha}dx=\delta_{n,m},$

where $\delta_{n,m},\ n,m\in\mathbb{N}_{0}$ is the Kronecker symbol. Up to a
normalization factor conditions (1.1) are equivalent to the following
equalities, respectively,

$None$ $\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1,\ \quad n\in\mathbb{N}.$

The function $\rho_{\nu}$ has the Mellin-Barnes integral representation

$None$ $\rho_{\nu}(x)=\frac{1}{2\pi
i}\int_{\gamma-i\infty}^{\gamma+i\infty}\Gamma(\nu+s)\Gamma(s)x^{-s}ds\ ,\quad
x,\gamma\in\mathbb{R}_{+},\ \nu\in\mathbb{R},$

where $\Gamma(z)$ is Euler’s gamma-function [1], Vol. I. Moreover, the
Parseval equality for the Mellin transform [5] permits to write (1.5) as the
Laplace integral representation for this function. In fact, we obtain

$None$ $\rho_{\nu}(x)=\int_{0}^{\infty}y^{\nu-1}e^{-y-x/y}dy,\quad x>0,\
\nu\in\mathbb{R}.$

Since $\rho_{\nu}(0)=\Gamma(\nu),\ \nu>0$, the polynomials
$P_{n}(x;\lambda,0),\ \lambda>0$ are associated with Laguerre polynomials,
and, correspondingly, $P_{n}(x;0,t),\ t>0,\ n\in\mathbb{N}_{0}$ are related to
the Prudnikov polynomials [7]. The asymptotic behavior of the modified Bessel
function at infinity and near the origin [1], Vol. II gives the corresponding
values for the function $\rho_{\nu},\ \nu\in\mathbb{R}$. Precisely, we have

$None$ $\rho_{\nu}(x)=O\left(x^{(\nu-|\nu|)/2}\right),\ x\to 0,\ \nu\neq
0,\quad\rho_{0}(x)=O(\log x),\ x\to 0,$

$None$ $\rho_{\nu}(x)=O\left(x^{\nu/2-1/4}e^{-2\sqrt{x}}\right),\
x\to+\infty.$

The moments for the weight $e^{-\lambda x}\rho_{\nu}(xt),\
\lambda^{2}+t^{2}\neq 0$ can be calculated via integral (1.4). Indeed, we have

$I_{n}=\int_{0}^{\infty}e^{-\lambda
x}\rho_{\nu}(xt)x^{n+\alpha}dx=\lambda^{-n-\alpha-1}\int_{0}^{\infty}e^{-x}x^{n+\alpha}\int_{0}^{\infty}u^{\nu-1}e^{-u-xt/(\lambda
u)}dudx$

$None$
$=\Gamma(1+n+\alpha)\int_{0}^{\infty}{u^{\nu+n+\alpha}e^{-u}\over(\lambda
u+t)^{n+\alpha+1}}du,\quad n\in\mathbb{N}_{0}.$

where the interchange of the order of integration is justified by Fubini’s
theorem. The latter integral is expressed in terms of the Tricomi function
$\Psi(a,b;z)$ (cf. [1], Vol. I and [4], Vol. I, Entry 2.3.6.9). Hence we get
finally the values from (1.7)

$None$
$I_{n}=\lambda^{-\nu-\alpha-n-1}t^{\nu}\Gamma(n+\nu+\alpha+1)\Gamma(n+\alpha+1)\Psi\left(1+n+\alpha+\nu,\
1+\nu;{t\over\lambda}\right).$

The quotient of the scaled Macdonald functions $\rho_{\nu},\rho_{\nu+1}$ is
given by the Ismail integral [3]

$None$
${\rho_{\nu}(x)\over\rho_{\nu+1}(x)}={1\over\pi^{2}}\int_{0}^{\infty}{y^{-1}dy\over(x+y)\left[J_{\nu+1}^{2}(2\sqrt{y})+Y^{2}_{\nu+1}(2\sqrt{y})\right]},$

where $J_{\nu}(z),Y_{\nu}(z)$ are Bessel functions of the first and second
kind, respectively [1]. Moreover, Entry 2.19.4.13 in [4], Vol. II represents
the product $x^{n}\rho_{\nu}(x)$ in terms of the associated Laguerre
polynomials [2]. Precisely, it has

$None$ ${(-1)^{n}x^{n}\over n!}\
\rho_{\nu}(x)=\int_{0}^{\infty}y^{\nu+n-1}e^{-y-x/y}L_{n}^{\nu}(y)dy,\quad
n\in\mathbb{N}_{0}.$

Appealing to the Riemann-Liouville fractional integral [6]

$None$
$\left(I_{-}^{\alpha}f\right)(x)={1\over\Gamma(\alpha)}\int_{x}^{\infty}(t-x)^{\alpha-1}f(t)dt,\quad{\rm
Re}\alpha>0$

and Entry 2.16.3.8 in [1], Vol. II, we get the formula

$None$ $\rho_{\nu}(x)=\left(I_{-}^{\nu}\rho_{0}\right)(x).$

Besides, the index law for fractional integrals immediately implies

$None$
$\rho_{\nu+\mu}(x)=\left(I_{-}^{\nu}\rho_{\mu}\right)(x)=\left(I_{-}^{\mu}\rho_{\nu}\right)(x).$

The corresponding definition of the fractional derivative presumes the
relation $D^{\mu}_{-}=-DI_{-}^{1-\mu}$. Hence for the ordinary $n$-th
derivative of $\rho_{\nu}$ we find

$None$ $D^{n}\rho_{\nu}(x)=(-1)^{n}\rho_{\nu-n}(x),\quad n\in\mathbb{N}_{0}.$

Recalling the Mellin-Barnes integral (1.3) and reduction formula for the
gamma-function it is not difficult to derive the recurrence relation

$None$
$\rho_{\nu+1}(x)=\nu\rho_{\nu}(x)+x\rho_{\nu-1}(x),\quad\nu\in\mathbb{R}.$

As it follows from the theory of orthogonal polynomials [2], a sequence
$\\{p_{n}\\}_{n\geq 0}$ satisfies the three term recurrence relation

$None$ $xp_{n}(x)=A_{n+1}p_{n+1}(x)+B_{n}p_{n}(x)+A_{n}p_{n-1}(x),$

where $p_{-1}(x)\equiv 0,\ p_{n}(x)=a_{n}x^{n}+b_{n}x^{n-1}+\dots,$ and

$None$ $A_{n}={a_{n-1}\over a_{n}},\quad B_{n}={b_{n}\over
a_{n}}-{b_{n+1}\over a_{n+1}}.$

As a consequence of (1.16) the Christoffel-Darboux formula takes place

$None$
$\sum_{k=0}^{n}p_{k}(x)p_{k}(y)=A_{n+1}\frac{p_{n+1}(x)p_{n}(y)-p_{n}(x)p_{n+1}(y)}{x-y}.$

Denoting the weight function as $\omega(x)=e^{-\lambda
x}\rho_{\nu}(xt)x^{\alpha}$ we establish a differential equation whose
solution is $\omega$.

Lemma 1. The weight function $\omega$ satisfies the following second order
differential equation

$None$ $x^{2}\omega^{\prime\prime}-(2(\alpha-\lambda
x)+\nu-1)x\omega^{\prime}+\left(\left(\alpha-\lambda
x\right)^{2}+x(\lambda(1-\nu)-1)+\alpha\nu\right)\omega=0.$

###### Proof.

In fact, recalling (1.14), (1.15), we have

$[xt\omega]^{\prime}={d\over dx}\left[x^{\alpha}e^{-\lambda
x}(\rho_{\nu+2}(xt)-(\nu+1)\rho_{\nu+1}(xt))\right]$

$None$ $=t\left(\alpha-\lambda x+\nu+1\right)\omega-e^{-\lambda
x}\rho_{\nu+1}(xt)x^{\alpha}.$

Hence with one more differentiation it gives

$None$ $[xt\omega]^{\prime\prime}=t(1-\lambda)\omega+t\left(\alpha-\lambda
x+\nu+1\right)\omega^{\prime}-\left({\alpha\over x}-\lambda\right)e^{-\lambda
x}\rho_{\nu+1}(xt)x^{\alpha}.$

Combining (1.20), (1.21), we find after simplification

$[x\omega]^{\prime\prime}=\left(2(\alpha-\lambda
x)+\nu+1\right)\omega^{\prime}-\left(\left({\alpha\over
x}-\lambda\right)\left(\alpha-\lambda x+\nu\right)+\lambda-1\right)\omega.$

Therefore fulfilling the differentiation and multiplying by $x$, we arrive at
(1.19).

∎

##  2\. Differential-difference properties

We begin this section with the following result.

Theorem 1. Let $\alpha>0,\ \lambda,t,\nu\geq 0,\ \lambda^{2}+t^{2}\neq
0,n\in\mathbb{N}$. Orthogonal polynomials $P_{n}(x;\lambda,t)$ satisfy the
integro-differential-difference equation

$x{\partial\over\partial
x}[P_{n}(x;\lambda,t)]=={xtA_{n}\over\pi^{2}}\int_{0}^{\infty}\int_{0}^{\infty}\bigg{[}P_{n}(x;\lambda,t)P_{n-1}(y;\lambda,t)-P_{n}(y;\lambda,t)P_{n-1}(x;\lambda,t)\bigg{]}$

$\times{P_{n}(y;\lambda,t)\ e^{-\lambda y}\rho_{\nu}(yt)y^{\alpha}\over
u(y+u)(x+u)\left[J_{\nu}^{2}(2\sqrt{u})+Y^{2}_{\nu}(2\sqrt{u})\right]}dudy$

$None$ $-\alpha A_{n}\bigg{[}a_{n-1,0}\ P_{n}(x;\lambda,t)-a_{n,0}\
P_{n-1}(x;\lambda,t)\bigg{]}\int_{0}^{\infty}P_{n}(y;\lambda,t)e^{-\lambda
y}\rho_{\nu}(yt)y^{\alpha-1}dy,$

where $A_{n}$ is defined in
$(1.17),P_{n}(x;\lambda,t)=a_{n}x^{n}+b_{n}x^{n-1}+\dots+a_{n,0}$ and,
clearly, polynomial coefficients are functions of $\lambda,t$.

###### Proof.

Since ${\partial\over\partial x}[P_{n}(x;\lambda,t)]$ is a polynomial of
degree $n-1$, we write it in the form

$None$ ${\partial\over\partial
x}[P_{n}(x;\lambda,t)]=\sum_{k=0}^{n-1}c_{n,k}P_{k}(x;\lambda,t),$

where, owing to the orthogonality,

$None$ $c_{n,k}=\int_{0}^{\infty}{\partial\over\partial
x}[P_{n}(x;\lambda,t)]P_{k}(x;\lambda,t)\ e^{-\lambda
x}\rho_{\nu}(xt)x^{\alpha}dx.$

Then, integrating by parts with the use of the orthogonality (1.1) and
relation (1.14), we eliminate integrated terms by virtue of the asymptotic
formulas (1.5), (1.6) to obtain

$c_{n,k}=t\int_{0}^{\infty}P_{n}(x;\lambda,t)P_{k}(x;\lambda,t)\ e^{-\lambda
x}\rho_{\nu-1}(xt)x^{\alpha}dx$

$None$ $-\alpha\int_{0}^{\infty}P_{n}(x;\lambda,t)P_{k}(x;\lambda,t)\
e^{-\lambda x}\rho_{\nu}(xt)x^{\alpha-1}dx.$

In the meantime, we easily observe from the orthogonality that

$\int_{0}^{\infty}P_{n}(y;\lambda,t)\sum_{k=0}^{n-1}P_{k}(y;\lambda,t)P_{k}(x;\lambda,t)e^{-\lambda
y}\rho_{\nu}(yt)y^{\alpha}\ {\rho_{\nu-1}(xt)\over\rho_{\nu}(xt)}dy=0,$

$\int_{0}^{\infty}P_{n}(y;\lambda,t)\sum_{k=0}^{n-1}P_{k}(y;\lambda,t)P_{k}(x;\lambda,t)e^{-\lambda
y}\rho_{\nu}(yt)y^{\alpha}x^{-1}dy=0.$

Therefore from (1.2), (1.9), (2.2), (2.4) and Christoffel-Darboux formula
(1.18) we derive

${\partial\over\partial
x}[P_{n}(x;\lambda,t)]=-\alpha\sum_{k=0}^{n-1}P_{k}(x;\lambda,t)\int_{0}^{\infty}P_{n}(y;\lambda,t)P_{k}(y;\lambda,t)\
e^{-\lambda y}\rho_{\nu}(yt)y^{\alpha}\left[{1\over y}-{1\over x}\right]dy$

$+t\sum_{k=0}^{n-1}P_{k}^{\nu}(x,t)\int_{0}^{\infty}P_{n}(y;\lambda,t)P_{k}(y;\lambda,t)\
e^{-\lambda
y}\rho_{\nu}(yt)y^{\alpha}\bigg{[}{\rho_{\nu-1}(yt)\over\rho_{\nu}(yt)}-{\rho_{\nu-1}(xt)\over\rho_{\nu}(xt)}\bigg{]}dy$

$=-{\alpha A_{n}\over x}\
\int_{0}^{\infty}P_{n}(y;\lambda,t)\bigg{[}P_{n}(x;\lambda,t)P_{n-1}(y;\lambda,t)-P_{n}(y;\lambda,t)P_{n-1}(x;\lambda,t)\bigg{]}\
e^{-\lambda y}\rho_{\nu}(yt)y^{\alpha-1}dy$

$+tA_{n}\int_{0}^{\infty}P_{n}(y;\lambda,t)\bigg{[}P_{n}(x;\lambda,t)P_{n-1}(y;\lambda,t)-P_{n}(y;\lambda,t)P_{n-1}(x;\lambda,t)\bigg{]}$

$\times{e^{-\lambda y}\rho_{\nu}(yt)y^{\alpha}\over
x-y}\bigg{[}{\rho_{\nu-1}(yt)\over\rho_{\nu}(yt)}-{\rho_{\nu-1}(xt)\over\rho_{\nu}(xt)}\bigg{]}dy$

$={tA_{n}\over\pi^{2}}\int_{0}^{\infty}\int_{0}^{\infty}\bigg{[}P_{n}(x;\lambda,t)P_{n-1}(y;\lambda,t)-P_{n}(y;\lambda,t)P_{n-1}(x;\lambda,t)\bigg{]}$

$\times{P_{n}(y;\lambda,t)\ e^{-\lambda y}\rho_{\nu}(yt)y^{\alpha}\over
u(y+u)(x+u)\left[J_{\nu}^{2}(2\sqrt{u})+Y^{2}_{\nu}(2\sqrt{u})\right]}dudy$

$-{\alpha A_{n}\over x}\ \bigg{[}a_{n-1,0}\ P_{n}(x;\lambda,t)-a_{n,0}\
P_{n-1}(x;\lambda,t)\bigg{]}\int_{0}^{\infty}P_{n}(y;\lambda,t)e^{-\lambda
y}\rho_{\nu}(yt)y^{\alpha-1}dy.$

This completes the proof of Theorem 1.

∎

Now, assuming that polynomial coefficients are continuously differentiable
functions of $\lambda,t$, we differentiate through equalities (1.2) with
respect to $\lambda$ and $t$ under the integral sign and integrate by parts
with the use of the orthogonality and equalities (1.14), (1.15). We note that
the differentiation under the integral sign in (1.2) can be motivated by the
absolute and uniform convergence in the subset of $\mathbb{R}^{2}\
M=\left\\{(\lambda,t)\in\mathbb{R}^{2}_{+}\left|\right.\lambda\geq\lambda_{0}>0,\
t\geq t_{0}>0\right\\}.$ Hence we obtain from (1.2) three types of
orthogonality conditions

$\int_{0}^{\infty}{\partial\over\partial\lambda}[P_{n}(x;\lambda,t)]e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+m}dx$

$None$ $-\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx=0,\quad m=0,1,\dots,n-1,$

$\int_{0}^{\infty}t{\partial\over\partial t}[P_{n}(x;\lambda,t)]e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+m}dx$

$None$ $-\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu+1}(xt)\
x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1,$

$\lambda\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx+\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda
x}\rho_{\nu+1}(xt)\ x^{\alpha+m}dx$

$None$ $-\int_{0}^{\infty}{\partial\over\partial
x}[P_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\ x^{\alpha+m+1}dx=0,\quad
m=0,1,\dots,n-1.$

Then as a direct consequence of (2.6), (2.7) we find the equalities

$\int_{0}^{\infty}\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\ x^{\alpha+m}dx$

$None$ $+\lambda\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+m+1}dx=0,\quad m=0,1,\dots,n-1.$

Furthermore, (1.16), (2.5), (2.8) and the differentiation through (1.1) for
$n=m$ by $\lambda$ and by $t$ yield

$None$
$\int_{0}^{\infty}\bigg{[}{\partial\over\partial\lambda}[P_{n}(x;\lambda,t)]-A_{n}P_{n-1}(x;\lambda,t)\bigg{]}e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1,$

$None$ $\int_{0}^{\infty}\bigg{[}\left(t{\partial\over\partial
t}-x{\partial\over\partial x}\right)[P_{n}(x;\lambda,t)]+\lambda
A_{n}P_{n-1}(x;\lambda,t)\bigg{]}e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1,$

$None$
$2\int_{0}^{\infty}P_{n}(x;\lambda,t){\partial\over\partial\lambda}[P_{n}(x;\lambda,t)]e^{-\lambda
x}\rho_{\nu}(xt)\
x^{\alpha}dx-\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+1}dx=0,$

$None$ $2\int_{0}^{\infty}P_{n}(x;\lambda,t){\partial\over\partial
t}[P_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha}dx-\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu-1}(xt)\ x^{\alpha+1}dx=0.$

As a result we are ready to prove the following theorems.

Theorem 2. Let $\alpha>-1,\ \lambda,t,\nu\geq 0,\ \lambda^{2}+t^{2}\neq
0,n\in\mathbb{N}$. Orthogonal polynomials $P_{n}(x;\lambda,t)$ satisfy the
first order ordinary and partial differential-difference equations

$None$ ${\partial\over\partial\lambda}[P_{n}(x;\lambda,t)]-{1\over
a_{n}}{\partial
a_{n}\over\partial\lambda}P_{n}(x;\lambda,t)-A_{n}P_{n-1}(x;\lambda,t)=0,$

$None$ $\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P_{n}(x;\lambda,t)]-\bigg{[}{t\over a_{n}}\ {\partial
a_{n}\over\partial t}-n\bigg{]}P_{n}(x;\lambda,t)+\lambda
A_{n}P_{n-1}(x;\lambda,t)=0.$

###### Proof.

Indeed, equations (2.13), (2.14) are direct consequences of the uniqueness and
orthogonality conditions (2.9), (2.10), respectively. In fact, polynomials
within the brackets under the integral sign are of degree $n$ and up to
constants are equal to $P_{n}$. These constants, in turn, can be found,
equating coefficients in front of $x^{n}$.

∎

Equating coefficients in front of $x^{n-1}$ in (2.13), (2.14) to zero and
dividing the obtained equalities by $a_{n}$, we find after simplification

Corollary 1. The following differential recurrence relations for coefficients
$a_{n},b_{n},A_{n},B_{n}$ hold

$None$ ${\partial\over\partial\lambda}\left({b_{n}\over
a_{n}}\right)=A_{n}^{2},$

$None$ ${\partial\over\partial t}\left({tb_{n}\over a_{n}}\right)=-\lambda
A_{n}^{2},$

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}\right)B_{n}+B_{n}=0,$ $None$
$\left(\lambda{\partial\over\partial\lambda}-t{\partial\over\partial
t}\right)\left({b_{n}\over a_{n}}\right)-{b_{n}\over a_{n}}=2\lambda
A_{n}^{2},$

$None$ $\left(\lambda{\partial\over\partial\lambda}-t{\partial\over\partial
t}\right)B_{n}-B_{n}=2\lambda\left[A_{n}^{2}-A_{n+1}^{2}\right].$

We note that relations (2.17)-(2.19) follow immediately from (2.15), (2.16)
via (1.17). Moreover, as we will see, identities (2.11), (2.12) generate

Theorem 3. Let $\alpha>-1,\ t>0,\ \lambda,\nu\geq 0,\ n\in\mathbb{N}_{0}$.
Then it has the equalities

$None$ $B_{n}={2\over a_{n}}{\partial a_{n}\over\partial\lambda},$

$None$ ${2t\over a_{n}}{\partial a_{n}\over\partial t}=2n+\alpha+1-\lambda
B_{n},$

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}\right)a_{n}=\left(n+{\alpha+1\over 2}\right)a_{n},$

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}\right)A_{n}+A_{n}=0.$

###### Proof.

Identity (2.20) follows from (2.11), orthogonality (1.1), the three term
recurrence relation (1.16) and, as a consequence, from the integral

$None$ $\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+n}dx={1\over a_{n}}.$

In order to prove (2.21), we have from (2.12) via (1.14), (1.15), (2.24) and
integration by parts

$None$ ${2\over a_{n}}{\partial a_{n}\over\partial t}-{\alpha+1\over
t}+{\lambda\over t}\ B_{n}-{2\over
t}\int_{0}^{\infty}P_{n}(x;\lambda,t){\partial\over\partial
x}[P_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\ x^{\alpha+1}dx=0.$

Using again (2.24), it is easily seen that the latter integral is equal to
$n$. Hence we end up with (2.21), and equalities (2.22) , (2.23) are immediate
consequences of (2.20), (2.21).

∎

Further, differentiating through by $\lambda$ and by $t$ the three term
recurrence relation (1.16) and employing (2.17), (2.23), we arrive at the
following result.

Corollary 2. Let $\alpha>-1,\ \lambda,t,\nu\geq 0,\ \lambda^{2}+t^{2}\neq
0,n\in\mathbb{N}_{0},m\in\mathbb{Z}$. Orthogonal polynomials
$P_{n}(x;\lambda,t)$ obey the partial differential-difference recurrence
relation of the form

$\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+m\mathbb{E}\right)xP_{n}(x;\lambda,t)=A_{n+1}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)P_{n+1}(x;\lambda,t)$

$None$
$+B_{n}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)P_{n}(x;\lambda,t)+A_{n}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)P_{n-1}(x;\lambda,t),$

where $\mathbb{E}$ is the identity operator.

The differential-difference recurrence relation (2.26) can be extended for the
integer power of the operator
$\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+m\mathbb{E},\ m\in\mathbb{Z}$. Namely, employing (2.17), (2.23) and the
method of mathematical induction, it is not difficult to establish the
following differential-difference recurrence relation

$\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+m\mathbb{E}\right)^{r}xP_{n}(x;\lambda,t)=A_{n+1}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)^{r}P_{n+1}(x;\lambda,t)$

$None$
$+B_{n}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)^{r}P_{n}(x;\lambda,t)+A_{n}\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}+(m-1)\mathbb{E}\right)^{r}P_{n-1}(x;\lambda,t),$

where $r\in\mathbb{N}_{0}$.

Now, recalling Theorem 2, we multiply (2.13) by $\lambda$ and sum with (2.14)
to get

Corollary 3. Let $\alpha>-1,\ \lambda,t,\nu>0,\ n\in\mathbb{N}_{0}$.
Orthogonal polynomials $P_{n}(x;\lambda,t)$ satisfy the partial differential
equation of the first order

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}\right)P_{n}(x;\lambda,t)-x{\partial\over\partial
x}[P_{n}(x;\lambda,t)]-{\alpha+1\over 2}P_{n}(x;\lambda,t)=0.$

Moreover, analogously to $(2.27)$ for the operator powers it has the relation

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}-{\alpha+1\over
2}\right)^{m}P_{n}(x;\lambda,t)=\left(x{\partial\over\partial
x}\right)^{m}P_{n}(x;\lambda,t),\quad m\in\mathbb{N}_{0}.$

Equating coefficient in front of $x^{n-1}$ to zero, we find a companion of the
equality (2.22)

$None$ $\left(\lambda{\partial\over\partial\lambda}+t{\partial\over\partial
t}\right)b_{n}=\left(n+{\alpha-1\over 2}\right)b_{n}.$

Further use of the three term recurrence relation (1.16) and the orthogonality
(1.1) generates more values of the integrals similar to (2.24) and identities
between polynomial coefficients. In fact, we have

Lemma 2. Writing the polynomial $P_{n}$ in the form

$P_{n}(x;\lambda,t)=a_{n}x^{n}+b_{n}x^{n-1}+d_{n}x^{n-2}+\hbox{lower
degrees},$

the following values of integrals take place

$None$ $\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+2}dx=A_{n+1}^{2}+B_{n}^{2}+A_{n}^{2},$

$None$ $\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+n+1}dx=-{b_{n+1}\over a_{n+1}a_{n}},$

$None$ $\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+n+2}dx={b_{n+2}b_{n+1}\over a_{n+2}a_{n+1}a_{n}}-{d_{n+2}\over
a_{n+2}a_{n}},$

$None$ $\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+3}dx=A_{n+1}^{2}\left[B_{n+1}+2B_{n}\right]+A_{n}^{2}B_{n-1}+\left[2A_{n}^{2}+B^{2}_{n}\right]B_{n}$

and the identity

$None$ ${d_{n}\over a_{n}}-{d_{n+2}\over a_{n+2}}-{b_{n+1}\over
a_{n+1}}\bigg{[}B_{n}+B_{n+1}\bigg{]}=A_{n+1}^{2}+B_{n}^{2}+A_{n}^{2}.$

###### Proof.

Identity (2.31) is a direct consequence of (1.16), (1.1). In the same manner
one gets (2.32), (2.33), involving the polynomial $P_{n+1},\ P_{n+2}$,
respectively. Then a combination with (2.31) and (1.17) yields (2.35).
Finally, the value of the integral (2.34) can be obtained, writing it via
(1.16), namely,

$\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+3}dx$

$=\int_{0}^{\infty}[A_{n+1}P_{n+1}(x;\lambda,t)+B_{n}P_{n}(x;\lambda,t)+A_{n}P_{n-1}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+1}dx,$

and appealing to (1.1), (1.16).

∎

On the other hand, we observe from (2.12), (2.21)

$None$ $\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu+1}(xt)\ x^{\alpha}dx=2n+\alpha+\nu+1-\lambda B_{n}.$

Then, we employ (1.14), (1.15), (2.31), (2.32) and integration by parts to get

$\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda x}\rho_{\nu+1}(xt)\
x^{\alpha+1}dx=\nu B_{n}+t\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu-1}(xt)\ x^{\alpha+2}dx$

$=\left(\alpha+2+\nu\right)B_{n}-\lambda\left[A_{n+1}^{2}+B_{n}^{2}+A_{n}^{2}\right]+2\int_{0}^{\infty}P_{n}(x;\lambda,t){\partial\over\partial
x}[P_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\ x^{\alpha+2}dx$

$=\left(\alpha+2+\nu\right)B_{n}-\lambda\left[A_{n+1}^{2}+B_{n}^{2}+A_{n}^{2}\right]+2(n-1){b_{n}\over
a_{n}}-2n{b_{n+1}\over a_{n+1}},$

i.e. we derive the identity

$None$ $\int_{0}^{\infty}[P_{n}(x;\lambda,t)]^{2}e^{-\lambda
x}\rho_{\nu+1}(xt)\
x^{\alpha+1}dx=\left(\alpha+2(n+1)+\nu\right)B_{n}-\lambda\left[A_{n+1}^{2}+B_{n}^{2}+A_{n}^{2}\right]-2{b_{n}\over
a_{n}}.$

Now, returning to equalities (2.8), we observe that for $\lambda\neq 0$ the
second integral is zero when $m=0,1,\dots,n-2$. Consequently, if $t=0$, we end
up with a modification of Laguerre polynomials $\hat{L}^{\alpha}_{n}$, getting
the known property

$None$ ${d\over dx}[P_{n}(x;\lambda,0)]=\hat{L}^{\alpha+1}_{n-1}(x;\lambda),$

and can be calculated explicitly, invoking properties of Laguerre polynomials.
Otherwise, in the case $t\neq 0$, we write (2.8) in the form

$None$ $\int_{0}^{\infty}\left(t{\partial\over\partial
t}-x{\partial\over\partial x}\right)[P^{\alpha}_{n}(x;\lambda,t)]e^{-\lambda
x}\rho_{\nu}(xt)\ x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-2,$ $None$
$\int_{0}^{\infty}\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P^{\alpha}_{n}(x;\lambda,t)]e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+n-1}dx\neq 0.$

This means that the sequence $\left\\{\left(t{\partial\over\partial
t}-x{\partial\over\partial x}\right)[P_{n}(x;\lambda,t)]\right\\}_{n\geq 0}$
is quasi-orthogonal with respect to the weight $e^{-\lambda
x}\rho_{\nu}(xt)x^{\alpha}$. The latter property is confirmed, in particular,
by virtue of the recurrence relation (2.14) as a combination of the above
quasi-orthogonal polynomials of degree $n$ in terms of their orthogonal
counterparts $P_{n},P_{n-1}$.

##  3\. Integral-diference equations. Composition orthogonality

In this section we will establish the integral-difference equations for the
sequence $\\{P_{n}\\}_{n\geq 0}$ and characterize it in terms of the
composition orthogonality for the operator polynomials $P_{n}(\theta)$, where
$\theta=xDx,\ D={d\over dx}$ and the usual product of polynomials
$P_{n}(x)P_{m}(x)$ is substituted by the composition of operators
$P_{n}(\theta)P_{m}(\theta)$. This notion was introduced in [7] and as we will
see below, the orthogonality relations (1.2) can be rewritten in the
composition sense by virtue of the key property for the operator $\theta$,
namely, its nonnegative integer power can be represented in the form

$None$ $\theta^{m}=x^{m}D^{m}x^{m},\quad m\in\mathbb{N}_{0}.$

Indeed, basing on differential-difference equations (2.13), (2.14), we have

Theorem 4. Let $\alpha>-1,\ \nu>0,\ \lambda,t,\geq 0,\ \lambda^{2}+t^{2}\neq
0,n\in\mathbb{N}$. Orthogonal polynomials $P_{n}(x;\lambda,t)$ satisfy the
following integral-difference equations

$None$ $P_{n}(x;\lambda,t)=\int_{0}^{\lambda}\exp\left({1\over
2}\int_{\xi}^{\lambda}B_{n}(y,t)dy\right)A_{n}(\xi,t)P_{n-1}(x;\xi,t)d\xi+\exp\left({1\over
2}\int_{0}^{\lambda}B_{n}(y,t)dy\right)P_{n}(x;0,t),$

$P_{n}(x;\lambda,t)=\int_{0}^{t}\left[x{\partial\over\partial
x}[P_{n}(x;\lambda,y)]-nP_{n}(x;\lambda,y)-\lambda
A_{n}(\lambda,y)P_{n-1}(x;\lambda,y)\right]{dy\over ya_{n}(\lambda,y)}$

$None$
$+(-1)^{n}\lambda^{-n-(1+\alpha)/2}\bigg{[}n!\Gamma(n+\alpha+1)\Gamma(\nu)\bigg{]}^{1/2}a_{n}(\lambda,t)\hat{L}^{\alpha}_{n}(x;\lambda).$

###### Proof.

The first equation (3.2) is obtained, solving the non-homogeneous ordinary
differential-difference equation (2.13) in terms of $\lambda$ with respect to
$P_{n}$, and using identity (2.20). Concerning the second equation, we will
resolve (2.14) under the initial condition (cf. (2.38))
$P_{n}(x;\lambda,0)=\hat{L}^{\alpha}_{n}(x;\lambda)$. Therefore it is
straightforward to write from (2.14) via integration the equality

$P_{n}(x;\lambda,t)=\int_{0}^{t}\left[x{\partial\over\partial
x}[P_{n}(x;\lambda,y)]-nP_{n}(x;\lambda,y)-\lambda
A_{n}(\lambda,y)P_{n-1}(x;\lambda,y)\right]{dy\over ya_{n}(\lambda,y)}$

$None$ $+{a_{n}(\lambda,t)\over
a_{n}(\lambda,0)}\hat{L}^{\alpha}_{n}(x;\lambda),$

where the integral converges via (2.14) and since $a_{n}\neq 0.$ Our goal now
is to find the value $a_{n}(\lambda,0)$. At the same time the three term
recurrence relation (1.16) for the modified Laguerre polynomials can be
written explicitly, because due to (2.21), (2.23) and the orthogonality for
the normalized Laguerre polynomials we find

$None$ $B_{n}(\lambda,0)={1\over\lambda}[2n+\alpha+1],\quad\lambda>0,$

$None$
$A_{n}(\lambda,0)={A_{n}(1,0)\over\lambda}=-{(n(n+\alpha))^{1/2}\over\lambda},\quad\lambda>0.$

Hence, taking into account (1.1) with $t=0$, (1.17) and (3.6), we obtain

$None$ $\prod_{k=1}^{n}A_{k}(\lambda,0)={a_{0}(\lambda,0)\over
a_{n}(\lambda,0)}={(-1)^{n}\over\lambda^{n}}(n!(1+\alpha)_{n})^{1/2},\
\lambda>0,$

$None$
$a_{0}(\lambda,0)={\lambda^{(1+\alpha)/2}\over(\Gamma(\nu)\Gamma(1+\alpha))^{1/2}},\quad\nu,\lambda>0,$

where $(z)_{n}$ is the Pochhammer symbol. Thus it yields the formula

$None$
$a_{n}(\lambda,0)={(-1)^{n}\lambda^{n+(1+\alpha)/2}\over\left(n!\Gamma(n+\alpha+1)\Gamma(\nu)\right)^{1/2}}.$

Substituting this value in (3.4), we end up with (3.3), completing the proof
of Theorem 4.

∎

Concerning the composition orthogonality of the sequence $\\{P_{n}\\}_{n\geq
0}$, we establish the following result.

Theorem 5. Let $\alpha>-1,\ \lambda,t,\nu\geq 0,\ \lambda^{2}+t^{2}\neq 0.$
The sequence $\\{P_{n}\\}_{n\geq 0}$ is compositionally orthogonal in the
sense of Laguerre, i.e. the corresponding orthogonality conditions have the
form

$None$ $\int_{0}^{\infty}y^{\nu}e^{-y}P_{n}\left({\theta\over
t};\lambda,t\right)\theta^{m}\left\\{{\Gamma(1+\alpha)\
y^{\alpha}\over(\lambda y+t)^{\alpha+1}}\right\\}dy=0,\quad m=0,1,\dots,n-1,\
\quad n\in\mathbb{N},$

where $\theta=yDy$.

###### Proof.

Recalling integral representation (1.10) of the function $\rho_{\nu}$ in terms
of Laguerre polynomials, we substitute its right-hand side in (1.2) and
interchange the order of integration by Fubini’s theorem via the absolute
convergence of the iterated integral. Hence we write

$None$ $\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m}dx=(-1)^{m}m!\int_{0}^{\infty}y^{\nu+m-1}e^{-y}L_{m}^{\nu}(y)\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-x(\lambda+t/y)}x^{\alpha}dxdy.$

Then, invoking the Rodrigues formula form Laguerre polynomials, we observe
(see (3.1))

$None$
$m!y^{\nu+m}e^{-y}L_{m}^{\nu}(y)=\theta^{m}\left\\{y^{\nu}e^{-y}\right\\}.$

Moreover, the differentiation by $y$ under the integral sign via the absolute
and uniform convergence allows to deduce the following operator formula

${1\over
y}\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-x(\lambda+t/y)}x^{\alpha}dx=P^{\nu}_{n}\left({\theta\over
t};\lambda,t\right)\left\\{{1\over
y}\int_{0}^{\infty}e^{-x(\lambda+t/y)}x^{\alpha}dx\right\\}$

$None$ $=P^{\nu}_{n}\left({\theta\over
t};\lambda,t\right)\left\\{{\Gamma(1+\alpha)\ y^{\alpha}\over(\lambda
y+t)^{\alpha+1}}\right\\}.$

Plugging (3.12), (3.13) in the right-hand side of (3.11), we integrate by
parts, eliminating the integrated terms. Thus we derive

$\int_{0}^{\infty}P_{n}(x;\lambda,t)e^{-\lambda x}\rho_{\nu}(xt)\
x^{\alpha+m}dx=(-1)^{m}\int_{0}^{\infty}\theta^{m}\left\\{y^{\nu}e^{-y}\right\\}P^{\nu}_{n}\left({\theta\over
t};\lambda,t\right)\left\\{{\Gamma(1+\alpha)\ y^{\alpha}\over(\lambda
y+t)^{\alpha+1}}\right\\}dy$

$=\int_{0}^{\infty}y^{\nu}e^{-y}P^{\nu}_{n}\left({\theta\over
t};\lambda,t\right)\theta^{m}\left\\{{\Gamma(1+\alpha)\
y^{\alpha}\over(\lambda y+t)^{\alpha+1}}\right\\}dy.$

This implies (3.10) and completes the proof.

∎

##  4\. One-parameter case $\lambda=1-t$

Let us consider the sequence of orthogonal polynomials
$\\{P_{n}(x;1-t,t)\\}_{n\geq 0}$ with respect to the weight
$e^{-(1-t)x}\rho_{\nu}(xt),\ 0\leq t\leq 1$. The respective orthogonality
conditions (1.2) will take the form

$None$ $\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1,\ \quad n\in\mathbb{N}.$

Then, making a differentiation with respect to parameter, we deduce from (4.1)
similar to (2.5), (2.6)

$\int_{0}^{\infty}{\partial\over\partial
t}[P_{n}(x;1-t,t)]e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m}dx+\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx$

$None$ $-\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu-1}(xt)\
x^{\alpha+m+1}dx=0,\quad m=0,1,\dots,n-1.$

Now, employing again (1.15) and orthogonality (4.1), we find from the latter
equalities (4.2)

$\int_{0}^{\infty}t{\partial\over\partial
t}[P_{n}(x;1-t,t)]e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m}dx+t\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx$

$None$ $-\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu+1}(xt)\
x^{\alpha+m}dx=0,\quad m=0,1,\dots,n-1.$

On the other hand, integrating by parts in (4.1) and using (1.14), (1.15) and
the orthogonality, it gives (cf. (2.7))

$(1-t)\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx+\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu+1}(xt)\
x^{\alpha+m}dx$

$None$ $-\int_{0}^{\infty}{\partial\over\partial
x}[P_{n}(x;1-t,t)]e^{-(1-t)x}\rho_{\nu}(xt)\ x^{\alpha+m+1}dx=0,\quad
m=0,1,\dots,n-1.$

Eliminating the integral with $\rho_{\nu+1}$ via (4.3), we get

$\int_{0}^{\infty}\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P_{n}(x;1-t,t)]e^{-(1-t)x}\rho_{\nu}(xt)\ x^{\alpha+m}dx$

$None$ $+\int_{0}^{\infty}P_{n}(x;1-t,t)e^{-(1-t)x}\rho_{\nu}(xt)\
x^{\alpha+m+1}dx=0,\quad m=0,1,\dots,n-1.$

This yields immediately an analog of the differential-difference equation
(2.14)

$None$ $\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P_{n}(x;1-t,t)]-\bigg{[}{t\over a_{n}}\ {\partial a_{n}\over\partial
t}-n\bigg{]}P_{n}(x;1-t,t)+A_{n}P_{n-1}(x;1-t,t)=0.$

Equating coefficients in front of $x^{n-1}$ in (4.6), we derive the Toda-type
equations (cf. (2.18), (2.19))

$None$ ${\partial\over\partial t}\left({tb_{n}\over a_{n}}\right)=-A_{n}^{2},$

$None$ ${\partial\over\partial
t}\left(tB_{n}\right)=\left[A_{n+1}^{2}-A_{n}^{2}\right].$

Finally, as in (2.39), (2.40) we conclude from (4.5) that the sequence
$\left\\{\left(t{\partial\over\partial t}-x{\partial\over\partial
x}\right)[P_{n}(x;1-t,t)]\right\\}_{n\geq 0}$ is quasi-orthogonal with respect
to the weight $e^{-(1-t)x}\rho_{\nu}(xt)x^{\alpha}$.

## References

  * [1] A. Erdélyi,W. Magnus, F. Oberhettinger, and F.G. Tricomi, Higher Transcendental Functions,Vols. I and II, McGraw-Hill, NewYork, London, Toronto, 1953.
  * [2] T.S. Chihara, An Introduction to Orthogonal Polynomials, Gordon and Breach, New York, London, 1978.
  * [3] M.E.H. Ismail, Bessel functions and the infinite divisibility of the student $t$-distribution, Ann. Prob. 5 (1977), 582-585.
  * [4] A. P. Prudnikov, Yu. A. Brychkov and O.I. Marichev, Vol. I: Elementary Functions, Gordon and Breach, New York, London, 1986; Vol. II: Special Functions, Gordon and Breach, New York, London, 1986; Vol. III: More Special Functions, Gordon and Breach, New York, London, 1990.
  * [5] E.C. Titchmarsh, An Introduction to the Theory of Fourier Integrals, Clarendon Press, Oxford, 1937.
  * [6] S. Yakubovich and Yu. Luchko, The Hypergeometric Approach to Integral Transforms and Convolutions, Kluwer Academic Publishers, Mathematics and Applications. Vol.287, 1994.
  * [7] S. Yakubovich, Orthogonal polynomials with ultra-exponential weight functions: an explicit solution to the Ditkin-Prudnikov problem, Constr. Approx. 53 (2021), no. 1, 1-38.


# Size Matters

Mats L. Richter  Wolf Byttner  Ulf Krumnack  Ludwig Schallner  Justin Shenk

###### Abstract

Fully convolutional neural networks can process input of arbitrary size by
applying a combination of downsampling and pooling. However, we find that
fully convolutional image classifiers are not agnostic to the input size but
rather show significant differences in performance: presenting the same image
at different scales can result in different outcomes. A closer look reveals
that there is no simple relationship between input size and model performance
(no ‘bigger is better’), but that each each network has a preferred input
size, for which it shows best results. We investigate this phenomenon by
applying different methods, including spectral analysis of layer activations
and probe classifiers, showing that there are characteristic features
depending on the network architecture. From this we find that the size of
discriminatory features is critically influencing how the inference process is
distributed among the layers.

Machine Learning, ICML

##  1 Introduction

The relation between the neural architecture and the resolution of the input
image is an often overlooked property of convolutional neural network (CNN)
classifier. Early designs like Krizhevsky et al. (2012); Simonyan & Zisserman
(2014) and Szegedy et al. (2015) chose square resolutions between $224\times
224$ and $299\times 299$ pixels mainly as a compromise between image size and
computational efficiency. In recent publications like Huang et al. (2018) the
resolution is increased up to $600\times 600$ pixels.

Furthermore Tan & Le (2019) show experimentally that the input resolution
increases the performance reliably on various architectures. The intuitive
explanation offered by authors like Szegedy et al. (2016) for this phenomenon
is the addition of greater detail gained by increasing the size of the image.
These details potentially yield additional features that can be detected by
the classifier, resulting in higher performance.

Tan & Le (2019) introduces also the idea that the size of the input image
needs to match the neural architecture for optimal efficiency and performance;
they use the size of the input image as one of 3 linearly related properties
for scaling up neural network architectures.

###  1.1 Contribution

In this work we investigate the following questions addressing the
relationship between input size and model architecture:

  * •

Has the size of the input image an effect on the predictive performance of CNN
classifiers? Answer: yes, it is shown that not just increasing the amount of
details but just altering the image resolution can improve classification
accuracy (Section 3.1).

  * •

Does the size of the input image influence how the information is processed in
the network? Answer: yes, we can show that processing significantly differs
depending on input size (Section 3.2) and the size of the depicted objects
(Section 3.3).

  * •

Can we know in advance which layers will contribute to the quality of the
prediction based on the input size? Answer: For strictly sequential
architectures the receptive field size allows to identify unproductive layers
(Section 3.4).

  * •

Do residual connections influence the observed behavior? Yes, residual
connections can help to involve more layers in the inference process (Section
3.5).

##  2 Background

###  2.1 Fully Convolutional Networks

This work focuses on fully convolutional neural network classifiers; the
current de facto standard for CNN classifier architectures (He et al., 2015;
Szegedy et al., 2016; Tan & Le, 2019; Iandola et al., 2014). While convolution
is a local operation that can in principle deal with input of arbitrary height
and width, the fully connected dense layers used for the output operate
globally and hence enforce specific input and output shapes. Therefore
architectures such as AlexNet have a fixed input size and require changes to
the architecture in order to change the input size (Krizhevsky et al., 2012).
In contrast, fully convolutional networks only apply a stack of convolutional
filters to the input, resulting in feature maps whose size depends on the size
of the input data. These feature maps can be summarized, usually by global
average pooling, to obtain a feature vector of fixed dimensionality that can
be used for further processing, e.g. classification (Szegedy et al., 2016).
This makes fully convolutional neural networks effectively agnostic to the
size of the input.

Being able to change the size of the input image without altering the
architecture of the CNN is important for avoiding potential artifacts induced
by architectural changes. For this reason we also modify non-fully
convolutional architectures such as the VGG-family of networks by Simonyan &
Zisserman (2014) to be fully convolutional. This is done by simply replacing
the readout, which reshapes the tensor produced by a convolutional layer into
a vector by a global average pooling layer (Lin et al., 2013).

###  2.2 Probe Classifiers

Probe classifiers were initially proposed by Alain & Bengio (2016) as a tool
for analyzing how the solution quality progresses while the data is propagated
through neural networks. In order to do this, logistic regression ”probes” are
trained on the same task as the model, using the output of individual layers
from the trained model as input. Since the softmax layer and a cross-entropy
minimizing logistic regression effectively solve the same task, we can use
these probes to see how the quality of the intermediate solution evolves
during the forward pass. Usually, probe accuracy will be low for the first
layers and increase throughout the network to approximate the model accuracy
in the final layers, as the example shown in figure 1 demonstrates.

###  2.3 Saturation and Tail Patterns

Saturation is another technique for expressing the activity of a neural
network layer in a single number, introduced by Richter et al. (2020). The
saturation metric of a layer is obtained by computing the number of
eigendirections in the output required to explain 99% of the total variance
and dividing this number by the dimensionality of the layer’s output.111In
case of convolutional layers, each position of the convolutional filter is
considered a single sample and the number of output channels is considered the
dimensionality of the data. Similar to the accuracy of a probe classifier,
this results in a value bound between 0 and 1. Intuitively saturation measures
a percentage of how much the output space of a layer is “filled” or
“saturated” with the data. Richter et al. (2020) show that interesting
patterns can be observed when the saturation of every layer is plotted in
sequence of the forward pass. Important for our work is the “tail pattern”,
which can be seen in figure 1, as it indicates an inefficiently distributed
forward pass.

(a)  Figure 1: VGG16 trained on Cifar10 with $32\times 32$ pixel input size.
The model exhibits a tail pattern starting from Conv8. Note that the probe
performance in the tail is roughly equal to the performance of the model
itself.

A tail pattern is a sub-sequence of layers, generally found close to the input
or the output of the network, with significantly less saturation than all
other layers. By observing probe performances we find that these layers
generally do not contribute significantly to the inference process by
devolving to quasi-pass-through layers over the course of training. We will
refer to these layers as unproductive layers or unproductive sequences.

###  2.4 The receptive field size

The receptive field size is a property of a convolutional layer inside a
neural network architecture. The receptive field size describes the maximum
height and width of a square area222Technically a rectangle, however since
square kernels are the norm we can make this simplification. on an arbitrarily
large input image. Only pixels contained in this area may influence the output
of a single convolution operation. In other words, the receptive field size
describes the spatial upper bound of visual patterns detectable by the
respective layer.

For convolutional neural networks with a sequential structure (no multiple
pathways during the forward pass) the receptive field size can be computed
analytically. We refer to the receptive field $r$ of the $l$th layer of
sequential network structure as $r_{l}$ (with $r_{0}=1$, which is the
”receptive field” of the input). For all layers $l>0$ in the convolutional
part of a sequential network the receptive field can be computed with the
following formula:

$r_{l}=r_{l-1}+((k_{l}-2)\prod^{l-1}_{i=0}s_{i})$ (1)

where $r_{l-1}$ is the receptive field of the previous layer, $k_{l}$ refers
to the kernel size of layer $l$ (with potential dilation already accounted
for) and $s_{i}$ the stride size of layer $i$. In the simplest case, which is
the first convolutional layer of the neural architecture, $r_{1}$ is exactly
the size of the kernel. When convolutional layers are stacked, the receptive
field expands for every layer $l$ with a kernel size $k_{l}\neq 1$. The stride
size $s_{l}$ of a layer $l$ furthermore has a multiplicative effect on the
growth rate of the receptive field, since a stride effectively downsamples the
feature map.

While the ResNet-family of networks by He et al. (2015) is not strictly
sequential due to their residual connections, we compute the receptive field
size by simply ignoring the residual connections. We can do this in the
context of this work because residual connections do not change the receptive
field size and we are only interested in the receptive field size as a spatial
upper bound for the integration of information into single output positions.

###  2.5 Choice of Models

The experiments are primarily conducted on VGG and ResNet-style architectures
originally developed by Simonyan & Zisserman (2014) and He et al. (2015).

The purpose of the VGG-style architectures in this work is to serve as a
baseline of simple sequential CNN architectures. The VGG architectures are
modified by using a global average pooling layer in the readout in order to
make them fully convolutional. Furthermore we use variants that utilize batch-
normalization after each layer, which has become a widespread pattern in many
more recent architectures (He et al., 2015; Tan & Le, 2019; Huang et al.,
2018).

For more complex architectures we choose the ResNet family and will mainly
focus on ResNet18 for analysis in the main part of this work. The reason for
this decision is two-fold. First, ResNet networks utilize many ideas of modern
neural architectures, like the building block design and residual connections,
while being rather simple in their structure (Tan & Le, 2019; Iandola et al.,
2014; Szegedy et al., 2016). Second, the ResNet family and ResNet18333For the
sake of consistency and comparability, when we refer to ResNet models like
ResNet18, we specifically refer to the ImageNet versions of these
architectures. In the initial publication, also a Cifar10-Version of ResNet
models is proposed, which deviates drastically in the receptive field size and
in performance on small image datasets. in particular are easy to visualize,
while many more complex derivative architectures feature a more non-sequential
structure and a much larger amount of consecutive layers, making the
experimental results harder to read and interpret.

To demonstrate that the claims made in this work generalize we reproduce some
results on additional models such as EfficentNet-B0, ResNet34 and ResNet50 and
provide them either in the main part of this work or in the supplementary
material.

###  2.6 Training Setup

Training on any dataset is conducted using a batch size of 64 (see
supplementary material for more details). If no information on the input size
is given it can be assumed that the native resolution of the dataset is fed
into the network. The training data is augmented by random cropping,
horizontal flipping and channel wise normalization of the data points using
ImageNet computed means and standard deviations of every color channel. When
images are resized linear interpolation is used and the resize is applied
after data augmentation and any other preprocessing. We use the stochastic
gradient decent optimizer for training. The learning rate is set initially to
0.1 and reduced after one and two thirds of the trained epochs by a factor 10.
The models are trained for 30 epochs with the exception of ResNet50, which is
trained for 60 epochs due to slow convergence. No hyperparameter optimization
or fine tuning on hyperparameters is conducted, since it is not the goal of
the paper to achieve maximum or state-of-the-art performance. We find that
changing the learning rate, optimizer, batch size, data augmentation and
number of epochs trained does not influence the conclusions of the experiments
in any significant way as long as the model is able to converge to a solution
better than chance-level (see supplementary material).

###  2.7 Datasets

Multiple datasets are used to conduct and verify the experiments presented in
this work. Experiments that require small base images were conducted on
Cifar10 and reproduced on MNIST and TinyImageNet (Le & Yang, 2015; Krizhevsky
et al., 2010; LeCun & Cortes, 2010). Experiments requiring images of higher
resolution were conducted on iNaturalist as well as ImageNet (Horn et al.,
2017; Russakovsky et al., 2015).

##  3 Experiments

A series of experiments has been conducted to investigate the relationship of
input size and model architecture. We first demonstrate that input size
affects the predictive performance of models even though no additional
information is added by scaling the images. We then explore this effect
further by reproducing these results on Cifar10 and analyzing the saturation
and probe performance. Based on these results we investigate the role of
locality of discriminatory features in the image. Finally we discuss the role
of the receptive field in these observations and analyze the influence of
residual connections.

###  3.1 Image Size and additional Details have independent effects on the
predictive performance

Tan & Le (2019) showed that increasing the input resolution of a model leads
to increased performance. The intuitive explanation for this effect is that
adding details (i.e. information) leads to better performance. The working
hypothesis is that more details is the only factor increasing performance when
increasing the image size. We investigate by increasing the image size, with
or without added detail.

We train multiple models on ImageNet and iNaturalist in three different
settings A, B and C. Models of set A, are trained on images of the original
design size of $224\times 224$ pixels, providing the performance baseline. Set
B models are trained on images of size $32\times 32$. We expect the relative
performance of these models to drop significantly in all scenarios, since the
image detail is reduced. The third set of models C is trained on images that
are first resized to $32\times 32$ pixels; then up-sampled to $224\times 224$
pixels. These images have as much detail as the images used for training the
models of set B while keeping the larger size used for training the models of
set A. According to the working hypothesis, performance should not increase
relative to groups two.

Table 1: Relative Top1-Accuracy: The table shows the predictive performance of trained models relative to a baseline model. The baseline uses the same model and architecture and is trained on images resized to $224\times 224$ pixels. Training the models on $32\times 32$ pixel images results in less than half the predictive performance in all tested scenarios relative to the baseline. Resizing the $32\times 32$ images back to $224\times 224$ results in a significant recovery of lost performance in all tested scenarios, despite no information is added by upscaling. Model | Dataset | downscaled | upscaled
---|---|---|---
|  | $32\times 32$ | $224\times 224$
VGG16 | ImageNet | 15.35% | 66.08%
VGG16 | iNaturalist | 36.83% | 58.38%
ResNet18 | ImageNet | 45.74% | 67.39%
ResNet18 | iNaturalist | 30.4% | 52.68%
ResNet50 | ImageNet | 28.21% | 71.8%
ResNet50 | iNaturalist | 33.87% | 54.38%
EfficientNet-B0 | ImageNet | 19.32% | 64.45%
EfficientNet-B0 | iNaturalist | 15.34% | 63.79%

While performance suffers drastically when reducing the input size, we can
also see in table 1 that decreasing, then increasing the image size regains
some of the lost performance with all models. Based on this observation we can
conclude that the size of the input images has an independent effect on the
predictive performance. This also confirms observations made by Richter et al.
(2020), were the same effect was observed on the Food101 dataset.

###  3.2 Input size affects the inference process of the CNN

To further understand what effect the input size has on the trained model and
its performance, we train the same models on Cifar10. Cifar10 has a native
resolution of $32\times 32$, therefore no additional details can be added by
increasing the size of the image. We train ResNet18 on Cifar10s native
resolution as well as $224\times 224$ pixels. We also train the model a third
time using an input size of $1024\times 1024$ pixels, to understand what
happens when the image is drastically oversized.

(a) $32\times 32$ (Cifar10 native resolution)

(b) $224\times 224$ (ResNet standard)

(c) $1024\times 1024$

Figure 2: Changing the input size changes how the inference is distributed
among the layers. The resolution ResNet18 was designed for (b) distributes the
inference most evenly, while too small (a) and too large (c) resolution shift
the bulk of the inference process to early and later layers respectively,
resulting in worse predictive performance.

The results in figure 2 show that the inference process is distributed
differently among the layers depending on the input size. Best predictive
performance is achieved at $224\times 224$, where inference is also
distributed most evenly. At a substantially smaller size of $32\times 32$
pixels, the performance suffers, dropping from 92.79% accuracy to 84.64%. We
also observe in figure 2a that only the first (highly saturated) two thirds of
the network improve the predictive performance of the probes substantially
from layer to layer. The rest of the network is lower saturated and does not
contribute to improving the quality of the solution.444The anomalies of single
layers dropping drastically in probe performance are an artifact of residual
connections. This was first observed by Alain & Bengio (2016) and also
described by Richter et al. (2020) Thus shrinking the input size has shifted
the bulk of the qualitative inference process to the front of the network.
Similarly, when drastically increasing the resolution to $1024\times 1024$
pixels (see figure 2c), the opposite effect can be observed. The qualitative
inference process is shifted closer to the output, while the first half of the
network does not substantially improve the predictive performance. The impact
on the predictive performance is also negative, reducing the accuracy from
92.79% to 86.77% compared to the model trained on $224\times 224$ pixel input
images.

###  3.3 The role of the object size in the relation of model and input
resolution

Processing similar features with different sizes is a common challenge in the
related field of object detection, where objects may strongly vary in size and
position on the image (Redmon et al., 2015; Redmon & Farhadi, 2016, 2018;
Bochkovskiy et al., 2020). One possible solution is the pyramid network
concept introduced in (Han et al., 2016), which is also employed in popular
models like YoloV3 by Redmon & Farhadi (2018). By utilizing feature maps from
different parts of the network the model becomes more stable when the object
size varies significantly. This indicates that processing objects of different
size is handled by different parts of the network. Based on our previous
observations, we hypothesize that the uneven distribution of the inference is
caused by the size of features on the images used for classification.

(a) $32\times 32$ without canvas

(b) $160\times 160$ with canvas

(c) $160\times 160$ upsampled

Figure 3: A tail of unproductive layers can be produced by placing the Cifar10
images on a $160\times 160$ pixel canvas. This indicates that locality of
discriminatory features (essentially the size of the object) is responsible
for the observed effect on the inference process.

To verify this experimentally we train models on a modified version of the
Cifar10 dataset. In this version the Cifar10 images are randomly placed on a
black canvas of $160\times 160$ pixels. This restricts the size of the object
of interest and therefore all potentially discriminatory features to a square
region of 32 pixels per side. If the absolute size of the object is
influencing which parts of the network process the information, the resulting
tail pattern and evolution of the probe performances should be similar to the
same model trained on Cifar10 on its native resolution. The results in figure
3 show that this is the case. Saturation and probe performances behave very
similar to training the model on $32\times 32$ pixels, while scaling the
entire image to $160\times 160$ pixels distributes the inference process very
differently (see figure 3c). The same effect can be reproduced with other
models and datasets (see supplementary material).

We can conclude from this that the relevance of the input size when training
classifiers comes from the size of the discriminatory features detected by the
model. In other words, the size of the image is a heuristic for the size of
the object of interest. This also means that the results depicted in this work
are unlikely to generalize to tasks where objects of interest are more
localized and varied like in most object detection datasets.

###  3.4 The role of the receptive field in relation to the object size

From the previous section we can derive that features of different sizes are
recognized by different layers in the model. In this section we will
investigate the causes of this phenomenon from an architectural point of view.

From the perspective of the architecture, the receptive field of a layer can
be considered an upper bound for the size of recognizable features. Since the
receptive field expands with every layer that has stride and/or kernel size
greater than 1, increasingly larger features can be recognized. We hypothesize
that for simple, sequential architectures like the VGG-family of networks
555We define simple architectures as networks that are strictly sequential
without multiple pathways, featuring only convolutional layers (with ReLU and
BatchNorm) and downsampling (max-pooling or otherwise) layers in the feature
extractor. The feature extractor is separated by a global average pooling
layer and followed only by fully connected layers., the receptive field is the
dominating factor influencing presence and position of unproductive
subsequences of layers.

(a) VGG11

(b) VGG13

(c) VGG16

(d) VGG19

Figure 4: The first convolutional layer processing input from a layer with a
receptive field size greater than the input size can be seen as a border
(marked with a black vertical line) between the ”solving” and the
”compressing” stage of the network. Only the highly saturated solving stage
improves on the intermediate solutions as shown by the probes. The depicted
models are from the VGG-family of networks by Simonyan & Zisserman (2014) and
trained on Cifar10 with an input size of $32\times 32$ pixels.

We study this property of the receptive field by training the VGG-family of
networks on Cifar10 and add the receptive field as an additional information
to our analysis.

(a) ResNet18 (no res. connections)

(b) VGG19 (dilation=3)

Figure 5: The behavior described in figure 4 translates to different
sequential architectures as well. For instance ResNet18 with disabled residual
connections uses mainly strided convolutional layers and has a significantly
faster receptive field expansion due to the first two layers being
downsampling layers. The VGG19 variant displayed here has the border layer
shifted closer to the input compared to the original version displayed in
figure 4, since the dilated convolutions increase the kernel size and
therefore the expansion of the receptive field. All models are trained on
Cifar10 with $32\times 32$ pixel input size.

In figure 4 we mark the first layer that processes input from a layer with a
receptive field size greater than the input size with a black border. We will
refer to this layer as the border layer. For all architectures, this border
separates layers contributing to the inference process from layers that do not
contribute to the quality of the inference significantly. This suggests that a
layer in a simple, sequential architecture can only substantially improve the
performance when novel information is integrated into positions on its feature
map.

We investigate this observation further by testing architectures, which alter
the receptive field size in different ways than the VGG-family of networks. We
test the effect of dilated convolutions by training a modified VGG19 with a
dilation rate of 2 in all convolutional layers. This modification effectively
increases the kernel sizes of all layers, which in turn increases the
receptive field size without changing the number of parameters. We also turn
ResNet18 into a sequential model by removing the residual connections. The
ResNet-family mainly uses strided convolutions for downsampling and also
features more downsampling layers that are spaced differently compared to VGG-
style models. This results in a much faster growth of the receptive field as
well as a higher overall receptive field size.

From the results in figure 5 we can see that the behavior of both models is
consistent with previous observations.

(a) Pre Border

(b) Border

(c) Post Border

(d) Last Layer

(e)

Figure 6: The heatmaps display the relative performance of probes trained on
individual positions of feature map to the performance of the model. The
depicted model is VGG16 trained on Cifar10 and displayed also in figure 4. The
depicted layers are the 5th, 8th, 9th and the last convolutional layer. We
observe that the performance of probes on early layers is best in the center,
where the receptive field can integrate the most information since the
receptive field will not exceed the bounds of the image. This is still true
for the border layer. However, all four inner positions of this feature map
integrate information from the entire image and have reached the performance
of the softmax output of the model as a result. The last two depicted layers
visualize the activity in the low saturated tail. As the receptive field
expands more positions on the feature map encode information from the entire
image and achieve similar probe performances. This homogenisation of partial
solutions results in a decreasing number of high variance eigendirections
which in turn results in low saturation. This explains the low saturated tail
of layers seen on networks trained on too low resolution.

Finally, we investigate how the solution develops inside the feature maps of
different parts of the network. We modify our method of logistic regression
probes by applying a single probe to every position of every layer’s feature
map. We then compute the relative performance of the probes by dividing the
probe accuracy with the network‘s accuracy (both evaluated on the test set).
By doing so we can visualize the quality of the partial solutions contained in
every position of the feature map based on their position. We observe in
figure 6 that the central positions on the feature map generally perform best
in early layers, while outer positions perform increasingly worse, with the
corner positions generally being the worst. We suspect that the receptive
field is at least partially responsible for this, since outer positions on the
feature map will receive more black padding and thus less information with the
receptive field expansion as a center pixel.

Another interesting observation in figure 6 is that the center most positions
on the feature map contain partial solutions roughly equivalent to the
performance of the entire model. As the saturation drops and layers become
part of the low saturated tail, the partial solution quality becomes
increasingly homogeneous across feature map positions. In the last layer each
vector on the feature map is as linearly separable with regards to the
classification task as the average pooled solution. We conclude based on these
measurements that this homogenisation of partial solution quality is also
responsible for the drop in saturation.

Based on these observations we can conclude that simple, sequential neural
networks may develop two stages of inference when the image is smaller than
the receptive field size of the model: The first being the solving stafe,
where the data is processed incrementally to achieve loss minimization. The
second stage, starting from the border layer, is the compressing stage. This
stage compresses the latent space by homogenisation of the partial solutions
for every position in the feature map.

###  3.5 The role of residual connections

Residual connections are a popular component used in many neural architectures
(He et al., 2015; Iandola et al., 2014; Szegedy et al., 2016; Tan & Le, 2019).
According to He et al. (2015) the networks with residual connections are able
to add ”deltas” to the existing representation of the data rather than
transforming it entirely. The residual connection itself does not expand or
change the receptive field. After the residual connection is added to the
output of a convolutional layer, information based on multiple receptive field
sizes is present in the feature map. This allows features based on lower
receptive field sizes to ”skip” layers and to be processed later in the
network, resulting in the network being able to distribute the inference on
more layers.

The aforementioned claims about the effects of residual connections suggest,
that models utilizing residual connections are able to utilize layers for
improving the prediction after the border layer. The border layer is the first
layer to receive an input produced by a layer with a receptive field size
larger than the image. In sequential architectures it is also the layer
separating the solving and compressing stage of the model.

(a) ResNet18

(b) ResNet34

Figure 7: Residual connections allow networks to utilize layers past the
border layer (marked with the horizontal bar). The zick-zack-pattern and drops
of probe performances observed in both networks are artifacts previously
observed by Richter et al. (2020) and Alain & Bengio (2016). These indicate
that the networks attempt to ”skip” the convolutional layers in the low
saturated tail. It is also worth noting that these skips are only present on
later layers which are part of the tail.

We can confirm this by looking at the results in figure 7. Both networks
improve the probe performances and stay highly saturated long after the border
layer has processed the data. We can attribute this behavior to the residual
connections, since we tested ResNet18 without them in the previous section
(see figure 5), resulting in behavior consistent with other sequential
architectures (and lower performance).

While the presence of residual connections has a positive effect on the
predictive performance of ResNet18 (84.61% accuracy with and 79.05% accuracy
without residual connections), the performance still remains worse than VGG-
style models. We attribute this to lower overall receptive field size of
VGG19, which is only 252 pixels. ResNet models downsample more aggressively in
the beginning, resulting in receptive field sizes of 413 (ResNet18) and over
800 pixels (Resnet34) respectively. This suggests that matching the receptive
field size with the input size remains important when using residual
connections in the architecture.

(a) Cifar10 optimized ResNet18

(b) Cifar10 optimized ResNet34

Figure 8: Cifar10 optimized version ResNet18 and 34 have a roughly quartered
receptive field size in every layer. These networks still display similar
behavior as the ImageNet optimized version when it comes to the border layer.
However, the reduction in the receptive field size has further removed the low
saturated tail and lead in both cases to a steady increase in performance over
the entire networks structure.

Therefore it should be possible to improve the performance of the (ImageNet
optimized) ResNet18 and 34 by reducing the size of the receptive field. This
was implicitly done by He et al. (2015), who proposed additionally to the
ImageNet optimized versions of ResNet a Cifar10 optimized variant. The Cifar10
variant of ResNet has no max-pooling layer and the first layer has its stride
size in kernel size halved. By doing so the receptive field of ResNet18 for
example was reduced from 413 (ImageNet optimized) to 109 pixels (Cifar10
optimized). The effect of these reductions can be seen in figure 8: the
proportion of low saturated layers is drastically reduced for both models and
the inference process is now well/more evenly distributed. Thus the
performance increases in both cases compared to the ImageNet optimized models.
The accuracy of ResNet18 improves from 84.61% (ImageNet optimized) to 91.95%
(Cifar10 optimized) ResNet34 improves from 82.76% to 92.21%. This example
illustrates that a mismatch of dataset and architecture can be resolved by
altering the architecture in a way that brings the receptive field size closer
to the input resolution.

##  4 Conclusion

For the longest time in deep learning, developing increasingly bigger and more
complex models to improve performance has been the norm. Our work provides a
counterpoint, showing that the neural architecture and the visual properties
of the dataset must match in order to avoid inefficiencies like layers not
contributing to the quality of the solution and sub-optimal predictive
performance. We show that the receptive field size is a primary architectural
factor in this interaction. We further show that the residual connections can
counteract (but not entirely mitigate) mismatches of input size and receptive
field size by allowing more layers to contribute to the inference
qualitatively. Finally, we demonstrate that probe classifiers and saturation
are useful tools to analyze the relationship between dataset and model
architecture allowing the optimization of CNN design in a principled way. We
conclude that the size of the input image matters for optimal predictive and
computational performance.

## References

  * Alain & Bengio (2016) Alain, G. and Bengio, Y.  Understanding intermediate layers using linear classifier probes.  _ArXiv_ , abs/1610.01644, 2016.
  * Bochkovskiy et al. (2020) Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y. M.  YOLOv4: Optimal speed and accuracy of object detection, 2020.
  * Bossard et al. (2014) Bossard, L., Guillaumin, M., and Van Gool, L.  Food-101 – mining discriminative components with random forests.  In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), _Computer Vision – ECCV 2014_ , pp. 446–461, Cham, 2014. Springer International Publishing.  ISBN 978-3-319-10599-4.
  * Han et al. (2016) Han, D., Kim, J., and Kim, J.  Deep pyramidal residual networks.  _CoRR_ , abs/1610.02915, 2016.  URL http://arxiv.org/abs/1610.02915.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.
  * Horn et al. (2017) Horn, G. V., Aodha, O. M., Song, Y., Shepard, A., Adam, H., Perona, P., and Belongie, S. J.  The iNaturalist challenge 2017 dataset.  _CoRR_ , abs/1707.06642, 2017.  URL http://arxiv.org/abs/1707.06642.
  * Huang et al. (2018) Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, Q. V., and Chen, Z.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  _CoRR_ , abs/1811.06965, 2018.  URL http://arxiv.org/abs/1811.06965.
  * Iandola et al. (2014) Iandola, F. N., Moskewicz, M. W., Karayev, S., Girshick, R. B., Darrell, T., and Keutzer, K.  DenseNet: Implementing efficient convnet descriptor pyramids.  _CoRR_ , abs/1404.1869, 2014.  URL http://arxiv.org/abs/1404.1869.
  * Iandola et al. (2016) Iandola, F. N., Moskewicz, M. W., Ashraf, K., Han, S., Dally, W. J., and Keutzer, K.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1mb model size.  _CoRR_ , abs/1602.07360, 2016.  URL http://arxiv.org/abs/1602.07360.
  * Krizhevsky et al. (2010) Krizhevsky, A., Nair, V., and Hinton, G.  CIFAR-10 (canadian institute for advanced research), 2010.  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E.  ImageNet classification with deep convolutional neural networks.  In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q. (eds.), _Advances in Neural Information Processing Systems 25_ , pp. 1097–1105. Curran Associates, Inc., 2012.
  * Le & Yang (2015) Le, Y. and Yang, X.  Tiny ImageNet visual recognition challenge, 2015.
  * LeCun & Cortes (2010) LeCun, Y. and Cortes, C.  MNIST handwritten digit database.  http://yann.lecun.com/exdb/mnist/, 2010.  URL http://yann.lecun.com/exdb/mnist/.
  * Lin et al. (2013) Lin, M., Chen, Q., and Yan, S.  Network in network, 2013.  URL http://arxiv.org/abs/1312.4400.  cite arxiv:1312.4400Comment: 10 pages, 4 figures, for iclr2014.
  * Redmon & Farhadi (2016) Redmon, J. and Farhadi, A.  YOLO9000: better, faster, stronger.  _CoRR_ , abs/1612.08242, 2016.  URL http://arxiv.org/abs/1612.08242.
  * Redmon & Farhadi (2018) Redmon, J. and Farhadi, A.  YOLOv3: An incremental improvement.  _CoRR_ , abs/1804.02767, 2018.  URL http://arxiv.org/abs/1804.02767.
  * Redmon et al. (2015) Redmon, J., Divvala, S. K., Girshick, R. B., and Farhadi, A.  You only look once: Unified, real-time object detection.  _CoRR_ , abs/1506.02640, 2015.  URL http://arxiv.org/abs/1506.02640.
  * Richter et al. (2020) Richter, M. L., Shenk, J., Byttner, W., Arpteg, A., and Huss, M.  Feature space saturation during training, 2020.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L.  ImageNet large scale visual recognition challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _CoRR_ , abs/1409.1556, 2014.
  * Szegedy et al. (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A.  Going deeper with convolutions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015.
  * Szegedy et al. (2016) Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z.  Rethinking the inception architecture for computer vision.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2818–2826, 2016.  doi: 10.1109/CVPR.2016.308.
  * Tan & Le (2019) Tan, M. and Le, Q.  EfficientNet: Rethinking model scaling for convolutional neural networks.  In Chaudhuri, K. and Salakhutdinov, R. (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 6105–6114. PMLR, 09–15 Jun 2019.
  * Zhang et al. (2017) Zhang, X., Zhou, X., Lin, M., and Sun, J.  ShuffleNet: An extremely efficient convolutional neural network for mobile devices.  _CoRR_ , abs/1707.01083, 2017.  URL http://arxiv.org/abs/1707.01083.

##  Appendix A Code and Experimental Setup

The full code used for conducting all depicted experiments as well as a guide
for reproducing the results can be found here:

https://anonymous.4open.science/r/d04fa611-aa00-4c22-9158-5af363b68c3a/

##  Appendix B Handling convolutional feature maps when computing probe
performance

A practical problem for computing the performance of probes is the size of
convolutional feature maps. In order to make the experiments computationally
tractable, it is necessary to reduce the dimensionality of those feature maps.

When probes were first proposed by Alain & Bengio (2016) two ways of handling
this practical problem are tested, Global Average Pooling (GAP) and randomized
subsampling of the feature maps. However, random sub sampling introduces
artefacts (Richter et al., 2020). Also, Richter et al. (2020) conducted GAP
experiments by looking on the ablative effects of downsampling and adaptive
pooling strategies on VGG and ResNet model trained on input sizes between
$32\times 32$ pixel $224\times 224$ pixel on Food101 and Cifar10 (Bossard et
al., 2014; Krizhevsky et al., 2010). We conclude that the effect on Global
Pooling are diminishing as the feature map increases but the overall structure
does not change. In order to reduce this ablative effect while still making
probe performance computation practical as much as possible we decided on a
$4\times 4$ adaptive average pooling over the feature maps. The experiments
showed diminishingly small changes in the features and the overall performance
levels of probes compared to larger downsampling strategies.

In experiments for showing the evolution of the of the feature map in
conjunction with the receptive field expansion, we also trained probes on each
position of all feature maps, which allows us to generate performance heat
maps, where we are able to show how much information required for solving the
task is contained on a specific position of the feature map.

###  B.1 A more detailed explanation on receptive field size and its
computation

The receptive field size is a property of a layer inside a neural network
architecture. It describes the size (height and width) of an rectangle on an
infinitely large input image. This area contains all pixels that can influence
the output of a single position of the convolutional kernel on the feature
map. Simply speaking, it describes the spatial limits of visual features
detectable by the respective layer.

It is worth noting that the size (in theory) is a rectangle, since the kernels
of CNNs may have an arbitrary height and width. therefore the receptive field
size is technically a 2-tuple consisting of two integers describing height and
width of the rectangle. However, conventionally networks have square kernels
and are fed square images, which are either padded, cropped or scaled to the
correct aspect ratio using data augmentation (Krizhevsky et al., 2012;
Simonyan & Zisserman, 2014; He et al., 2015; Szegedy et al., 2015, 2016; Tan &
Le, 2019; Iandola et al., 2016, 2014). We thus refer to the receptive field
size as an integer for the sake of simplicity.

####  B.1.1 Computing the receptive field

For convolutional neural networks with a sequential structure (no multiple
pathways during the forward pass) the receptive field size can be computed
analytically. We refer to the receptive field $r$ of the $l$th layer of
sequential network structure as $r_{l}$ (with $r_{0}=1$, which is the
”receptive field” of the input). For all layers $l>0$ in the convolutional
part of the network the receptive field can be computed with the following
formula:

$r_{l}=r_{l-1}+((k_{l}-2)\prod^{l-1}_{i=0}s_{i})$ (2)

Where $r_{l-1}$ is the receptive field of the previous layer, $k_{l}$ refers
to the kernel size of layer $l$ (with potential dilation already accounted
for) and $s_{i}$ the stride size of layer $i$. In the simplest case, which is
the first convolutional layer of the neural architecture $r_{1}$ and stride
size of 1, the receptive field is exactly the size of the kernel, therefore
the receptive field size is the kernel size. When convolutional layers are
stacked, the receptive field for every layer $l$ with a kernel size $k_{l}\neq
1$ expands the receptive field for following layers, because the filter
integrates information of an area of the input feature map into a single
output position. Downsampling and pooling layers that reduce the size of the
feature map will have $s_{i}\neq 1$, which has a multiplicative effect on the
receptive field expansion of consecutive layers. In simple cases like VGG16,
where the max-pooling layers are $2\times 2$ filters with $s_{i}=2$, this can
be intuitively imagined as 4 adjacent pixels being summarized into a single
”super pixel”. This effect on the growth of $r$ is also compounding
exponentially when multiple downsampling layers are employed in the neural
architecture, which is generally the case for most popular architectures
(Simonyan & Zisserman, 2014; He et al., 2015; Szegedy et al., 2015, 2016;
Iandola et al., 2014, 2016; Zhang et al., 2017; Tan & Le, 2019; Szegedy et
al., 2016; Iandola et al., 2016).

####  B.1.2 The receptive field in network with multiple pathways

The method described previously allows the computation of the receptive field
for sequential architectures with each layer having at most a single previous
layer. However, since the advent of deep learning multiple pathways have been
common in neural architectures. In this scenario there $s_{i}$ and $r_{l-1}$
may be tuples of values with each element describing the stride size and the
receptive field size of a pathway. However, the common strategies for
funneling multiple paths into a single successive layer are simple element-
wise operation (like additions and multiplications) or concatenations of the
respective stacks of feature maps produced by each path. Both cases have in
common that the size of the feature maps must agree, which is achieved by
either different means influencing the size of the receptive field. However,
we are not interested in knowing exactly which feature map in an output layer
considers which information. We are only interested in the size of the area
that contains all information that can theoretically influence a position in
the feature map. Hence, we ignore all pathways except the one with the largest
receptive field. It is worth noting that we make the assumption that all
pathways downsample the image identically $s_{l,1}=s_{l,2}=...=s_{l,n}$, so
that the growth rate of the receptive field is increased evenly on different
pathways. While this assumption is not true for all possible architectures it
holds true for all architectures considered in this paper and most popular
architectures like VGG, ResNet, InceptionV3 and EfficentNet (Simonyan &
Zisserman, 2014; He et al., 2015; Szegedy et al., 2016; Tan & Le, 2019).

##  Appendix C Resolution Shifts the Distribution of the Inference Process on
different Models

In Section 3.2 we show on a ResNet18 architecture how resolution affects the
distribution of the inference process by training ResNet18 on 3 different
resolutions on Cifar10.

To show that this observation does generalize we repeat the experiment. We use
TinyImageNet as a more complex but still small resolution problem. We further
show that the hypothesis holds also true for VGG16 and ResNet50 models.

###  C.1 VGG16 - Cifar10

(a) $32\times 32$ (Cifar10 native resolution)

(b) $160\times 160$

(c) $1024\times 1024$

Figure 9: Reproduction of the experiment depicted in figure 2 using the VGG16
architecture. Because of the larger memory footprint of the model, the batch
size of this experiment is reduced to 20. The observed pattern stays the same.
Low resolution results in a tail close to the output, while high resolution
exhibits a tail at the input.

The (medium sized) resolution of $160\times 160$ pixels performs best.

###  C.2 ResNet18 - TinyImageNet

TinyImageNet consists of 200 classes of $64\times 64$ images. Therefore the
first experiment is conducted on $64\times 64$ pixel input resolution instead
of $32\times 32$ pixel resolution, since this is the basic resolution of the
dataset.

(a) $64\times 64$ (TinyImageNet native resolution)

(b) $1024\times 1024$

Figure 10: The higher native resolution combined with the residual connection
removes the tail pattern even at $64\times 64$ pixels for ResNet18. We see
that a tail pattern can be produced in the input part of the network when the
resolution is increased drastically to $1024\times 1024$, confirming
observations from Cifar10.

###  C.3 ResNet50 - Cifar10

Due to an error in the setup, the mid-sized image experiment was conducted
with a $160\times 160$ pixel resolution instead of $224\times 224$. However,
even thought the resolution was altered it still shows the exspected behavior
observed with other models.

(a) $32\times 32$ (Cifar10 native resolution)

(b) $160\times 160$ (ResNet standard)

(c) $1024\times 1024$

Figure 11: Producing a tail pattern in the near the input and the output is
also possible for very deep models like ResNet50 with the exspected effects on
the performance. The Bottleneck-Modules with $1\times 1$ convolutions and
varying filter sizes induce additional noise into saturation patterns.

##  Appendix D Object Size Experiments on MNIST and ResNet models

In Section 3.3 we make the claim that the observed relation of input
resolution and the performance of the model is an artifact caused by the
underlying interaction of object size and neural architecture. We repeat the
experiments that demonstrate this claim on ResNet-style models as well as on
the MNIST dataset. As we can see in figure 12 and figure 13 and figure 14, the
results are consistent with the observations shown in the paper in all three
scenarios.

###  D.1 ResNet18 - Cifar10

(a) $32\times 32$ without canvas

(b) $160\times 160$ with canvas

(c) $160\times 160$ upsampled

Figure 12: Random Positioning Experiments conducted with ResNet18 on Cifar10
data.

###  D.2 VGG16 - MNIST

(a) $32\times 32$ without canvas

(b) $160\times 160$ with canvas

(c) $160\times 160$ upsampled

Figure 13: Random Positioning Experiments conducted with VGG16 on MNIST data.

###  D.3 ResNet50 - Cifar10

(a) $32\times 32$ (Cifar10 native resolution)

(b) $160\times 160$ with canvas

(c) $160\times 160$ upsampled

Figure 14: Random Positioning Experiments repeated on ResNet50.

##  Appendix E Additional Experiments Regarding the Relationship of Receptive
Field and Input Resolution

The following experiments show additional observations made regarding the role
of the receptive field size. When we training the models on the (arguably more
difficult) TinyImageNet dataset using the same setup, the behavior stays
consistent with our previous observations (see figure 15). However, if the
problem is easy to solve it is possible for the model to mostly saturate probe
performance before the saturation ”tail” starts. We can see an example of this
in figure 16. When looking closely we can still see very slight improvements
made until the border layers is reached. However, the improvements are hard to
observe, which is also the reason why we only show MNIST results in the
supplementary material.

We also observe that skip-connections have a statistically significant
performance impact and that different combinations of residual connections
distribute the inference process differently. To test this, we disabled
different combinations of residual connections in ResNet18 while training
Cifar10.

###  E.1 TinyImageNet Results

(a) VGG11

(b) VGG13

(c) VGG16

Figure 15: The first convolutional layer processing input from a layer with a
receptive field size greater than the input size marks a border between an
”productive” and an ”unproductive” part of the network. Productive meaning
relatively high saturated and visible improving probe performance. The
depicted models are from the VGG-family of networks by Simonyan & Zisserman
(2014) and trained on Cifar10 with an input size of $32\times 32$ pixels.

###  E.2 MNIST results

(a) VGG11

(b) VGG13

(c) VGG16

(d) VGG19

Figure 16: The first convolutional layer processing input from a layer with a
receptive field size greater than the input size marks a border between an
”productive” and an ”unproductive” part of the network. Productive meaning
relatively high saturated and visible improving probe performance. The
depicted models are from the VGG-family of networks by Simonyan & Zisserman
(2014) and trained on MNIST with an input size of $32\times 32$ pixels.

###  E.3 ResNet18 Cifar10 - Effect of disabling residual connections

Results indicate that removing the first three residual connections harms
performance. Removing the last or second-to-last skip connection is
statistically indistinguishable from not removing any residual connections.
Furthermore, removing the first skip connection harms performance relative to
many other permutations of residual connections. Thus, it seems like early
downsampling decreases network performance and the first skip connection
mitigates that effect. Later residual connections do not have the same effect.

| 2,3,4 | 1,3,4 | 1,2,4 | 1,2,3 | 3,4 | 2,4 | 2,3 | 1,4 | 1,3 | 1,2 | 4 | 1 | ResNet18
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Mean | 82.70 | 83.17 | 83.40 | 83.39 | 82.67 | 82.35 | 82.59 | 83.42 | 81.88 | 82.95 | 81.65 | 82.92 | 83.16
n | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 9 | 9 | 8 | 8
Table 2: ResNet18 with some residual connections removed. ResNet18 has four residual connections, bold numbers indicate which residual connections were enabled. For instance, 2, 3, 4 indicates that residual connections 2, 3 and 4 were present in the network. Mean performance of n Cifar10 trainings. | 2,3,4 | 1,3,4 | 1,2,4 | 1,2,3 | 3,4 | 2,4 | 2,3 | 1,4 | 1,3 | 1,2 | 4 | 1 | ResNet18
---|---|---|---|---|---|---|---|---|---|---|---|---|---
2,3,4 |  | 0.026 | 0.004 | 0.015 | 0.879 | 0.340 | 0.638 | 0.001 | 0.563 | 0.256 | 0.014 | 0.287 | 0.260
1,3,4 | -2.421 |  | 0.254 | 0.380 | 0.018 | 0.028 | 0.013 | 0.167 | 0.363 | 0.273 | 0.001 | 0.177 | 0.983
1,2,4 | -3.297 | -1.178 |  | 0.963 | 0.003 | 0.008 | 0.002 | 0.933 | 0.288 | 0.051 | 0.000 | 0.030 | 0.562
1,2,3 | -2.701 | -0.901 | 0.047 |  | 0.011 | 0.014 | 0.008 | 0.907 | 0.294 | 0.113 | 0.001 | 0.087 | 0.609
3,4 | 0.154 | 2.615 | 3.480 | 2.847 |  | 0.385 | 0.740 | 0.001 | 0.579 | 0.198 | 0.016 | 0.219 | 0.228
2,4 | 0.981 | 2.387 | 2.960 | 2.720 | 0.891 |  | 0.520 | 0.006 | 0.746 | 0.121 | 0.169 | 0.149 | 0.130
2,3 | 0.479 | 2.761 | 3.570 | 2.989 | 0.337 | -0.656 |  | 0.001 | 0.617 | 0.135 | 0.028 | 0.153 | 0.179
1,4 | -3.746 | -1.438 | -0.085 | -0.119 | -3.958 | -3.113 | -3.986 |  | 0.282 | 0.026 | 0.000 | 0.011 | 0.522
1,3 | 0.589 | 0.933 | 1.095 | 1.080 | 0.565 | 0.329 | 0.508 | 1.109 |  | 0.475 | 0.879 | 0.513 | 0.432
1,2 | -1.176 | 1.132 | 2.098 | 1.673 | -1.341 | -1.632 | -1.569 | 2.447 | -0.731 |  | 0.005 | 0.884 | 0.617
4 | 2.750 | 4.126 | 4.613 | 4.264 | 2.671 | 1.437 | 2.402 | 4.819 | 0.154 | 3.291 |  | 0.007 | 0.014
1 | -1.102 | 1.412 | 2.384 | 1.821 | -1.279 | -1.515 | -1.502 | 2.888 | -0.670 | 0.149 | -3.136 |  | 0.579
ResNet18 | -1.167 | 0.021 | 0.592 | 0.522 | -1.253 | -1.595 | -1.407 | 0.655 | -0.805 | -0.510 | -2.779 | -0.568 |
Table 3: Independent two-sample t-test, pairwise mean performance difference
significance test, ResNet18 with some residual connections removed, trained on
Cifar10. ResNet18 has four skip-connections, bold numbers indicate which
residual connections were enabled. For instance, 2, 3, 4 indicates that
residual connections 2, 3 and 4 were present in the network. P-values are in
the upper triangle, t-values in the lower. Statistically significant values
(p¡0.25) are highlighted in green. n=10 or as in the previous table.

##  Appendix F Receptive Field Analysis with Partial Solution Heatmaps

###  F.1 Receptive Field Analysis using Probes on VGG16

(a) VGG16 trained on Cifar10 using $32\times 32$ pixel input size

(b) Conv1

(c) Conv2

(d) Conv3

(e) Conv4

(f) Conv5

(g) Conv6

(h) Conv7

(i) Conv8

(j) Conv9

(k) Conv10

(l) Conv11

(m) Conv12

(n) Conv13 (o)

Figure 17: VGG16 trained on Cifar10 alongside the heatmaps generated from the
relative accuracy of the partial solutions in each layer. Note that the border
layer (Conv8) is the first layer to have partial solutions of equal quality to
the models solution (measured in accuracy).

###  F.2 Receptive Field Analysis using Probes on ResNet18

(a) ResNet18 trained on Cifar10 using $32\times 32$ pixel input size

(b) Conv1

(c) Block1-Conv1

(d) Block1-Conv2

(e) Block2-Conv1

(f) Block2-Conv2

(g) Block3-Conv1

(h) Block3-Conv2

(i) Block4-Conv1

(j) Block4-Conv2

(k) Block5-Conv1

(l) Block5-Conv2

(m) Block6-Conv1

(n) Block6-Conv2

(o) Block7-Conv1

(p) Block7-Conv2

(q) Block8-Conv1

(r) Block8-Conv2

(s)

Figure 18: ResNet18 trained on Cifar10 alongside the heatmaps generated from
the relative accuracy of the partial solutions in each layer. Note that
”skipped” layer (Block6-Conv2 and Block8-Con2) have a worse performance and
worst partial solutions.

###  F.3 Receptive Field Analysis using Probes on ResNet18 optimized for
Cifar10

(a) ResNet18 (Cifar10 Optimized) trained on Cifar10 using $32\times 32$ pixel
input size

(b) Conv1

(c) Block1-Conv1

(d) Block1-Conv2

(e) Block2-Conv1

(f) Block2-Conv2

(g) Block3-Conv1

(h) Block3-Conv2

(i) Block4-Conv1

(j) Block4-Conv2

(k) Block5-Conv1

(l) Block5-Conv2

(m) Block6-Conv1

(n) Block6-Conv2

(o) Block7-Conv1

(p) Block7-Conv2

(q) Block8-Conv1

(r) Block8-Conv2

(s)

Figure 19: Cifar10 optimized ResNet18 trained on Cifar10 alongside the
heatmaps generated from the relative accuracy of the partial solutions in each
layer. Note that no layers are skipped and the solution quality of the partial
solution develops up until the final residual block.