$\displaystyle\frac{J_{m}^{2}\left(\widetilde{\beta}_{n}\right)}{\left\\{1+J_{m}\left(\widetilde{\beta}_{n}\right)\right\\}^{2}}\leq
O_{\mathbb{P}}\left(\max_{1\leq j\leq
p}\operatorname{diam}^{2\kappa}(A_{j})\log(n)\right)\lambda^{-1}.$

By $(ii)$, the RHS of this inequality will be strictly smaller than $1$ for
all large $n$ with high probability, as

$\displaystyle\left(\max_{1\leq j\leq
p}\operatorname{diam}^{2\kappa}(A_{j})\log(n)\right)\lambda^{-1}=\left(\max_{1\leq
j\leq
p}\operatorname{diam}^{2\kappa}(A_{j})\log^{3/2}(n)\right)\log^{-\frac{1}{2}}(n)\lambda^{-1}=O_{\mathbb{P}}\left(\log^{-\frac{1}{2}}(n)\right).$

It follows from part B of Lemma 3 that
$J_{m}(\widetilde{\beta}_{n})=O_{\mathbb{P}}(1)$ and (45) leads to

$\displaystyle\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p}=O_{\mathbb{P}}\left(\log^{2}(n)\max_{1\leq
j\leq p}\operatorname{diam}^{\kappa}(A_{j})\right),$

so that (26) is verified. We have thus completed the proof of (26).

_Step 4._ The last step of our proof involves obtaining the same rate of
convergence for $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ and establishing
that $J_{m}(\widehat{\beta}_{n})=O_{\mathbb{P}}(1)$. For this, we use the
triangle inequality to get

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}\leq\left\|\widehat{\beta}_{n}-\widetilde{\beta}_{n}\right\|_{n,p}+\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p}.$
(46)

Using the definition of $\widetilde{\beta}_{n}$ in (23) we can express

$\left\|\widehat{\beta}_{n}-\widetilde{\beta}_{n}\right\|_{n,p}=\left(1-\gamma_{n}\right)\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}.$

Plugging this into (46) and using (26) we get

$\displaystyle\frac{\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}}{1+\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}}=O_{\mathbb{P}}\left(\log^{2}(n)\left\\{n^{-\frac{m}{2m+d}}+\max_{1\leq
j\leq p}\operatorname{diam}^{\kappa}(A_{j})\right\\}\right).$

Solving for $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ yields

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}(1-o_{\mathbb{P}}(1))=O_{\mathbb{P}}\left(\log^{2}(n)\left\\{n^{-\frac{m}{2m+d}}+\max_{1\leq
j\leq p}\operatorname{diam}^{\kappa}(A_{j})\right\\}\right)$

so that

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}=O_{\mathbb{P}}\left(\log^{2}(n)\left\\{n^{-\frac{m}{2m+d}}+\max_{1\leq
j\leq p}\operatorname{diam}^{\kappa}(A_{j})\right\\}\right),$ (47)

which is the first result of the theorem. To conclude the proof we deduce the
boundedness of $J_{m}(\widehat{\beta}_{n})$ from the boundedness of
$J_{m}(\widetilde{\beta}_{n})$. For this recall that $J_{m}(\cdot)$ is a semi-
norm on $\mathcal{H}^{m}(\mathbbm{R}^{d})$ so that it is homogeneous and
satisfies the triangle inequality. Moreover, by construction, $\gamma_{n}\leq
1$. Therefore, with the help of (23) and (26) we obtain

$\displaystyle\gamma_{n}\,J_{m}(\widehat{\beta}_{n}-\beta_{0})=J_{m}(\gamma_{n}(\widehat{\beta}_{n}-\beta_{0}))\leq
J_{m}(\widetilde{\beta}_{n})+J_{m}(\beta_{0})=O_{\mathbb{P}}(1).$

The boundedness of $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ from (47) now
implies that $\gamma_{n}^{-1}=O_{\mathbb{P}}(1)$ so that
$J_{m}(\widehat{\beta}_{n}-\beta_{0})=O_{\mathbb{P}}(1)$ also. It follows by
another application of the triangle inequality that

$\displaystyle J_{m}\left(\widehat{\beta}_{n}\right)\leq
J_{m}\left(\beta_{0}\right)+J_{m}\left(\widehat{\beta}_{n}-\beta_{0}\right)=O_{\mathbb{P}}(1),$

which is the desired result.

###  A.6 Proof of Corollary 2

Observe that, since the $\\{A_{j}\\}_{j=1}^{p}$ partition $\mathcal{I}$, we
can write

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n}^{2}=\frac{1}{n}\sum_{i=1}^{n}\left|\int_{\mathcal{I}}X_{i}(\mathbf{t})\left(\widehat{\beta}_{n}(\mathbf{t})-\beta_{0}(\mathbf{t})\right)\,\mathrm{d}\,\mathbf{t}\right|^{2}=\frac{1}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}\int_{A_{j}}X_{i}(\mathbf{t})\left(\widehat{\beta}_{n}(\mathbf{t})-\beta_{0}(\mathbf{t})\right)\,\mathrm{d}\,\mathbf{t}\right|^{2},$

so that, using the inequality $|x+y|^{2}\leq 2(|x|^{2}+|y|^{2})$ twice, the
definition of the $d_{i}$ in (4) as well as the definition of the semi-norm
$\|\cdot\|_{n,p}$ in (10), we get

$\displaystyle\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n}^{2}$
$\displaystyle\leq\frac{2}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}\int_{A_{j}}\left\\{X_{i}(\mathbf{t})\left(\widehat{\beta}_{n}(\mathbf{t})-\beta_{0}(\mathbf{t})\right)-X_{i}(\mathbf{t}_{j})\left(\widehat{\beta}_{n}(\mathbf{t}_{j})-\beta_{0}(\mathbf{t}_{j})\right)\right\\}\,\mathrm{d}\,\mathbf{t}\right|^{2}$
$\displaystyle\quad+\frac{2}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\widehat{\beta}_{n}(\mathbf{t}_{j})-\beta_{0}(\mathbf{t}_{j})\right)\mu(A_{j})\right|^{2}$
$\displaystyle\leq\frac{4}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}\int_{A_{j}}\\{X_{i}(\mathbf{t})\widehat{\beta}_{n}(\mathbf{t})-X_{i}(\mathbf{t}_{j})\widehat{\beta}_{n}(\mathbf{t}_{j})\\}\,\mathrm{d}\,\mathbf{t}\right|^{2}$
$\displaystyle\quad+\frac{4}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}\int_{A_{j}}\\{X_{i}(\mathbf{t})\beta_{0}(\mathbf{t})-X_{i}(\mathbf{t}_{j})\beta_{0}(\mathbf{t}_{j})\\}\,\mathrm{d}\,\mathbf{t}\right|^{2}$
$\displaystyle\quad+\frac{2}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\widehat{\beta}_{n}(\mathbf{t}_{j})-\beta_{0}(\mathbf{t}_{j})\right)\mu(A_{j})\right|^{2}$
$\displaystyle=\frac{4}{n}\sum_{i=1}^{n}\left|\sum_{j=1}^{p}\int_{A_{j}}\\{X_{i}(\mathbf{t})\widehat{\beta}_{n}(\mathbf{t})-X_{i}(\mathbf{t}_{j})\widehat{\beta}_{n}(\mathbf{t}_{j})\\}\,\mathrm{d}\,\mathbf{t}\right|^{2}+\frac{4}{n}\sum_{i=1}^{n}d_{i}^{2}+2\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}^{2}.$
(48)

A rate of convergence for $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ was
obtained in Theorem 1. From equation (38) in the proof of that theorem we also
know that $n^{-1}\sum_{i=1}^{n}d_{i}^{2}=O_{\mathbb{P}}(\max_{j\leq
p}\operatorname{diam}^{2\kappa}(A_{j}))$. Hence, to prove the corollary we
only need to determine the order of the first term in the RHS of (A.6).

To bound the first term in (A.6), write

$\displaystyle\left|\sum_{j=1}^{p}\int_{A_{j}}\\{X_{i}(\mathbf{t})\widehat{\beta}_{n}(\mathbf{t})-X_{i}(\mathbf{t}_{j})\widehat{\beta}_{n}(\mathbf{t}_{j})\\}\,\mathrm{d}\,\mathbf{t}\right|$
$\displaystyle\leq\left|\sum_{j=1}^{p}\int_{A_{j}}\left(\widehat{\beta}_{n}(\mathbf{t})-\widehat{\beta}_{n}(\mathbf{t}_{j})\right)X_{i}(\mathbf{t})\,\mathrm{d}\,\mathbf{t}\right|$
$\displaystyle\quad+\left|\sum_{j=1}^{p}\int_{A_{j}}\widehat{\beta}_{n}(\mathbf{t}_{j})(X_{i}(\mathbf{t})-X_{i}(\mathbf{t}_{j}))\,\mathrm{d}\,\mathbf{t}\right|$
$\displaystyle=I+II,$ (49)

say. To bound $I$ in (A.6), let us note that since, by assumption $2m>d+1$,
$\mathcal{H}^{m}(\mathcal{I})$ embeds on the space of Lipschitz continuous
functions on $\mathcal{I}$, which we previously denoted by
$\mathcal{C}^{0,1}(\mathcal{I})$. Since, as shown in Proposition 1,
$\widehat{\beta}_{n}\in\mathcal{H}^{m}(\mathcal{I})$, we therefore have

$\displaystyle\sup_{\mathbf{t},\mathbf{s}\in\mathcal{I}}\frac{\left|\widehat{\beta}_{n}(\mathbf{t})-\widehat{\beta}_{n}(\mathbf{s})\right|}{\left\|\mathbf{t}-\mathbf{s}\right\|_{\mathbbm{R}^{d}}}$
$\displaystyle\leq
C\left\\{\int_{\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|^{2}\,\mathrm{d}\,\mathbf{t}+\sum_{m_{1}+\ldots+m_{d}=m}\binom{m}{m_{1},\ldots,m_{d}}\int_{\mathcal{I}}\left(\frac{\partial^{m}\widehat{\beta}_{n}(\mathbf{t})}{\partial
t_{1}^{m_{1}}\ldots\partial
t_{d}^{m_{d}}}\right)^{2}\,\mathrm{d}\,\mathbf{t}\right\\}^{1/2}$
$\displaystyle\leq
C\left\\{\int_{\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|^{2}\,\mathrm{d}\,\mathbf{t}+J_{m}^{2}(\widehat{\beta}_{n})\right\\}^{1/2},$

for some global $C>0$. By Theorem 1,
$J_{m}(\widehat{\beta}_{n})=O_{\mathbb{P}}(1)$. At the same time, by Theorem
3.4 of Utreras, (1988), there exist $C_{0}$ and $B_{1}$, depending only on
$m,d,\mathcal{I}$ such that

$\displaystyle\int_{\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|^{2}\,\mathrm{d}\,\mathbf{t}\leq
C_{0}\sum_{j=1}^{p}\left|\widehat{\beta}_{n}(\mathbf{t}_{j})\right|^{2}\mu(A_{j})+C_{0}I_{m}^{2}(\widehat{\beta}_{n})\leq
C_{0}J_{m}^{2}\left(\widehat{\beta}_{n}\right).$

By the boundedness of $J_{m}(\widehat{\beta}_{n})$, which also implies the
boundedness of
$\sum_{j=1}^{p}|\widehat{\beta}_{n}(\mathbf{t}_{j})|^{2}\mu(A_{j})$, it now
follows that

$\displaystyle\sup_{\mathbf{t},\mathbf{s}\in\mathcal{I}}\frac{\left|\widehat{\beta}_{n}(\mathbf{t})-\widehat{\beta}_{n}(\mathbf{s})\right|}{\left\|\mathbf{t}-\mathbf{s}\right\|_{\mathbbm{R}^{d}}}=O_{\mathbb{P}}(1),$

from which we obtain

$\displaystyle I\leq
O_{\mathbb{P}}(1)\sup_{\mathbf{t}\in\mathcal{I}}\left|X_{i}(\mathbf{t})\right|\sum_{j=1}^{p}\int_{A_{j}}\left\|\mathbf{t}-\mathbf{t}_{j}\right\|_{\mathbbm{R}^{d}}\,\mathrm{d}\,\mathbf{t}.$

Under (A3),
$\sup_{\mathbf{t}\in\mathcal{I}}|X_{i}(\mathbf{t})|=O_{\mathbb{P}}(1)$. Hence,
we may conclude that

$\displaystyle I=O_{\mathbb{P}}\left(\max_{1\leq j\leq
p}\operatorname{diam}(A_{j})\right).$ (50)

To bound $II$ in (A.6), let us note that for $2m>d$,
$\mathcal{H}^{m}(\mathcal{I})$ also embeds on the space
$\mathcal{C}(\mathcal{I})$ of continuous functions on $\mathcal{I}$, hence

$\displaystyle\sup_{\mathbf{t}\in\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|\leq
C\left\\{\int_{\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|^{2}\,\mathrm{d}\,\mathbf{t}+J_{m}^{2}(\widehat{\beta}_{n})\right\\}^{1/2},$

for some $C>0$. Arguing as with the bound for $I$, we can deduce that
$\sup_{\mathbf{t}\in\mathcal{I}}|\widehat{\beta}_{n}(\mathbf{t})|=O_{\mathbb{P}}(1)$.
Therefore,

$\displaystyle
II\leq\sup_{\mathbf{t}\in\mathcal{I}}\left|\widehat{\beta}_{n}(\mathbf{t})\right|\sum_{j=1}^{p}\int_{A_{j}}\left|X_{i}(\mathbf{t})-X_{i}(\mathbf{t}_{j})\right|\,\mathrm{d}\,\mathbf{t},$

and (A3) yields

$\displaystyle II=O_{\mathbb{P}}\left(\max_{1\leq j\leq
p}\operatorname{diam}^{\kappa}(A_{j})\right).$ (51)

Combining (A.6), (A.6), (50) and (51) with Theorem 1 now leads to the result.

###  A.7 Proof of Theorem 2

The proof of Theorem 2 relies on a suitable decomposition of the empirical
process along with its tightness as a function of $\sigma$ over
$|\sigma-\sigma_{0}|\leq\varepsilon$ for some $\varepsilon>0$. The crucial
elements for our proof are established in Lemma 4 below. For its statement we
adopt our notation from the proof of Theorem 1.

######  Lemma 4.

For some fixed
$(\sigma_{0},\beta_{0})\in\mathbbm{R}\times\mathcal{H}^{m}(\mathbbm{R}^{d})$,
consider the independent processes $V_{i,\beta,\sigma}$ and
$Z_{i,\beta,\sigma}$ given by

$\displaystyle V_{i,\beta,\sigma}$ $\displaystyle=\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\,\mathrm{d}\,t$
$\displaystyle\quad-\mathbb{E}_{\epsilon_{i}}\left\\{\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\,\mathrm{d}\,t\right\\},\quad(i=1\ldots,n),$
$\displaystyle Z_{i,\beta,\sigma}$
$\displaystyle=\left(\psi\left(\frac{\epsilon_{i}}{\sigma}\right)-\mathbb{E}_{\epsilon_{i}}\left\\{\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\right)\langle
X_{i},\beta_{0}-\beta\rangle_{p},\quad(i=1\ldots,n).$

Here, both $V_{i,\beta,\sigma}$ and $Z_{i,\beta,\sigma}$ are indexed by
$\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d})$ with $2m>d$ and
$\sigma\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$ for some small
$\varepsilon>0$ such that $\sigma_{0}-\varepsilon>0$. Then, the following
asymptotic results hold.

  1. A.

$\begin{aligned}
\sup_{\begin{subarray}{c}\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1\\\
|\sigma-\sigma_{0}|\leq\varepsilon\end{subarray}}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}=O_{\mathbb{P}}\left(\sqrt{\log(n)}\right).\end{aligned}$

  2. B.

$\begin{aligned}
\sup_{\begin{subarray}{c}\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1\\\
|\sigma-\sigma_{0}|\leq\varepsilon\end{subarray}}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}Z_{i,\beta,\sigma}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}=O_{\mathbb{P}}\left(\sqrt{\log(n)}\right).\end{aligned}$

###### Proof of Lemma 4.

We begin by proving part A. Consider first the case of a fixed
$\sigma\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$ with $\varepsilon$
small enough so that $\sigma_{0}-\varepsilon>0$. Observe that in this case
Lemma 2 applies, as each $V_{i,\beta,\sigma}$ has mean zero,
$V_{i,\beta_{0},\sigma}=0$ and, for every
$\beta,\beta^{\prime}\in\mathcal{H}^{m}(\mathbbm{R}^{d})$, by (B4) and the
fundamental theorem of calculus, we find that

$\displaystyle\left|\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\,\mathrm{d}\,t-\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta^{\prime}\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\,\mathrm{d}\,t\right|$
$\displaystyle=\left|\int_{d_{i}+\langle
X_{i},\beta_{0}-\beta^{\prime}\rangle_{p}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}\,\mathrm{d}\,t\right|\leq
2\left\|\psi\right\|_{\infty}\left|\langle
X_{i},\beta-\beta^{\prime}\rangle_{p}\right|,$

and constants are trivially uniformly sub-Gaussian. Furthermore, as we
demonstrate in the proof of Theorem 1 (see Step 2 there),

$\displaystyle
H\left(\delta,\\{\beta\in\mathcal{H}^{m}(\mathcal{I}):\|\beta-\beta_{0}\|_{n,p}\leq
1,1+J_{m}(\beta)\leq M\\},\|\cdot\|_{n,p}\right)\lesssim
M^{\frac{d}{2m}}\delta^{1-\frac{d}{2m}},\quad\delta>0,M\geq 1.$

Hence, applying Lemma 2 yields the existence of a $c>0$ such that for all
$T\geq c$

$\displaystyle\mathbb{P}\left(\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}\geq
T\right)\leq c\exp\left[-\frac{T^{2}}{c^{2}}\right].$ (52)

This result is valid for each
$\sigma\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$, but it is not
uniform. To make the result uniform for
$\sigma\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$, notice first that
condition (B4) ensures that for $\varepsilon_{1}=1/(\sigma_{0}+\varepsilon)$
there exists a $D_{\sigma_{0}}$ such that for all
$s\geq\varepsilon_{1},t\geq\varepsilon_{1}$ we have

$\displaystyle\sup_{x\in\mathbbm{R}}\left|\psi(tx)-\psi(sx)\right|\leq
D_{\sigma_{0}}\left|t-s\right|.$ (53)

Split $[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$ into $N$ subintervals
each of them having radius $\leq T/(2\sqrt{n}D_{\sigma_{0}})$ and select
$\\{\sigma_{k}\\}_{k=1}^{N}$, one $\sigma_{k}$ in each one of these $N$
subintervals. By Lemma 2.5 in van de Geer, (2000), we can have

$\displaystyle
N\leq\left(\frac{8\sqrt{n}D_{\sigma_{0}}\varepsilon}{T}+1\right).$ (54)

Observe next that

$\displaystyle\sup_{\begin{subarray}{c}\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1\\\
|\sigma-\sigma_{0}|\leq\varepsilon\end{subarray}}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}$
$\displaystyle\leq\max_{1\leq k\leq
N}\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma_{k}}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}$
$\displaystyle\quad+\sup_{|\sigma-\sigma_{0}|\leq\varepsilon}\min_{1\leq k\leq
N}\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\\{V_{i,\beta,\sigma}-V_{i,\beta,\sigma_{k}}\\}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}},$

where to obtain the second term on the RHS of this inequality we have used the
inequality $|\sup_{x}f(x)-\sup_{x}g(x)|\leq\sup_{x}|f(x)-g(x)|$ for real
valued functions $f$ and $g$. Using (53), for every $\sigma$ satisfying
$|\sigma-\sigma_{k}|\leq T/(2\sqrt{n}D_{\sigma_{0}})$ for some
$k\in\\{1,\ldots,N\\}$, we now see that

$\displaystyle\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\\{V_{i,\beta,\sigma}-V_{i,\beta,\sigma_{k}}\\}\right|$
$\displaystyle\leq\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}+t}{\sigma_{k}}\right)\right\\}\,\mathrm{d}\,t\right|$
$\displaystyle\quad+\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}}{\sigma}\right)-\psi\left(\frac{\epsilon_{i}}{\sigma_{k}}\right)\right\\}\,\mathrm{d}\,t\right|$
$\displaystyle\leq
2D_{\sigma_{0}}\sqrt{n}\left|\sigma-\sigma_{k}\right|\frac{1}{n}\sum_{i=1}^{n}\left|\langle
X_{i},\beta-\beta_{0}\rangle_{p}\right|$ $\displaystyle\leq
T\left\|\beta-\beta_{0}\right\|_{n,p},$

where to obtain the last inequality we have also used the Cauchy-Schwarz
inequality and the definition of $\|\cdot\|_{n,p}$ in (11). Since
$J_{m}(\beta)\geq 0$ for all $\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d})$ and
$d/(2m)<1$, the quotient may be bounded by

$\displaystyle\sup_{|\sigma-\sigma_{0}|\leq\varepsilon}$
$\displaystyle\min_{1\leq k\leq
N}\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\\{V_{i,\beta,\sigma}-V_{i,\beta,\sigma_{k}}\\}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}$
$\displaystyle\quad\leq
T\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\|\beta-\beta_{0}\|_{n,p}}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}$
$\displaystyle\quad\leq
T\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\|\beta-\beta_{0}\|_{n,p}^{\frac{d}{2m}}}{\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}$
$\displaystyle\quad\leq T.$

From this, Boole’s inequality, (52) and (54) we find

$\displaystyle\mathbb{P}\left(\sup_{\begin{subarray}{c}\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1\\\
|\sigma-\sigma_{0}|\leq\varepsilon\end{subarray}}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}\geq
2T\right)$ $\displaystyle\quad\leq\mathbb{P}\left(\max_{1\leq k\leq
N}\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma_{k}}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}\geq
T\right)$
$\displaystyle\quad=\mathbb{P}\left(\bigcup_{k=1}^{N}\left\\{\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma_{k}}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}\geq
T\right\\}\right)$
$\displaystyle\quad\leq\sum_{k=1}^{N}\mathbb{P}\left(\sup_{\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d}):\left\|\beta-\beta_{0}\right\|_{n,p}\leq
1}\frac{\left|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i,\beta,\sigma_{k}}\right|}{\left\|\beta-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\beta)\right\\}^{\frac{d}{2m}}}\geq
T\right)$
$\displaystyle\quad\leq\left(\frac{8\sqrt{n}D_{\sigma_{0}}\varepsilon}{T}+1\right)c\exp\left[-\frac{T^{2}}{c^{2}}\right].$

The result follows by setting $T=\sqrt{C\log(n)}$ for a sufficiently large
$C>0$. To prove part B, use exactly the same arguments given that each
$Z_{i,\beta,\sigma}$ is a centered process with $Z_{i,\beta_{0},\sigma}=0$. ∎

We now move on to the proof of Theorem 2. We only sketch the proof, as most of
the arguments from the proof of Theorem 1 carry over to this case. Let
$L_{n}(\beta,\widehat{\sigma}_{n})$ denote the objective function, that is,

$\displaystyle
L_{n}(\beta,\widehat{\sigma}_{n})=\frac{1}{n}\sum_{i=1}^{n}\rho\left(\frac{Y_{i}-\sum_{j=1}^{p}X_{i}\left(\mathbf{t}_{j}\right)\beta\left(\mathbf{t}_{j}\right)\mu\left(A_{j}\right)}{\widehat{\sigma}_{n}}\right)+\lambda
J_{m}^{2}\left(\beta\right).$

Defining the convex combination
$\widetilde{\beta}_{n}=\gamma_{n}\widehat{\beta}_{n}+(1-\gamma_{n})\beta_{0}$
with $\gamma_{n}=1/(1+\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p})$, we have

$\displaystyle
L_{n}(\widetilde{\beta}_{n},\widehat{\sigma}_{n})\leq\gamma_{n}L_{n}(\widehat{\beta}_{n},\widehat{\sigma}_{n})+(1-\gamma_{n})L_{n}(\beta_{0},\widehat{\sigma}_{n})\leq
L_{n}(\beta_{0},\widehat{\sigma}_{n}).$ (55)

Set

$\displaystyle
M_{n}(\beta,\widehat{\sigma}_{n}):=\frac{1}{n}\sum_{i=1}^{n}\rho\left(\frac{\epsilon_{i}+d_{i}+\sum_{j=1}^{p}X_{i}\left(\mathbf{t}_{j}\right)\left(\beta_{0}\left(\mathbf{t}_{j}\right)-\beta\left(\mathbf{t}_{j}\right)\right)\mu\left(A_{j}\right)}{\widehat{\sigma}_{n}}\right),$

so that
$L_{n}(\beta,\widehat{\sigma}_{n})=M_{n}(\beta,\widehat{\sigma}_{n})+\lambda
J^{2}_{m}(\beta)$. Furthermore, set

$\displaystyle
M(\beta,\widehat{\sigma}_{n}):=\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+d_{i}+\sum_{j=1}^{p}X_{i}\left(\mathbf{t}_{j}\right)\left(\beta_{0}\left(\mathbf{t}_{j}\right)-\beta\left(\mathbf{t}_{j}\right)\right)\mu\left(A_{j}\right)}{\widehat{\sigma}_{n}}\right)\right\\}.$

Rearranging (55) yields

$\displaystyle
M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})+\lambda
J_{m}^{2}(\widetilde{\beta}_{n})$
$\displaystyle\leq\left(M_{n}(\beta_{0},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})\right)$
$\displaystyle-\left(M_{n}(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})\right)+\lambda
J_{m}^{2}(\beta_{0}).$ (56)

As in the proof of Theorem 1, our approach consists of deriving a lower bound
on the LHS of (A.7) in terms of $\|\widetilde{\beta}_{n}-\beta_{0}\|_{n,p}$
and an upper bound on the RHS, also in terms of
$\|\widetilde{\beta}_{n}-\beta_{0}\|_{n,p}$. Combining these two bounds
appropriately will then yield the result:

$\displaystyle\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p}=\log^{2}(n)O_{\mathbb{P}}\left(n^{-\frac{m}{2m+d}}+\max_{1\leq
j\leq p}\operatorname{diam}^{\kappa}(A_{j})\right)\quad\text{and}\quad
J_{m}\left(\widetilde{\beta}_{n}\right)=O_{\mathbb{P}}(1),$ (57)

for $\lambda$ satisfying $(i)$ and $(ii)$. The definition of the convex
combination $\widetilde{\beta}_{n}$ will then allow us to derive the same rate
of convergence for $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ as well as the
boundedness of $J_{m}(\widehat{\beta}_{n})$.

_Step 1_. We begin by deriving a lower bound on the LHS of (A.7). First,
notice that, since $\widehat{\sigma}_{n}\xrightarrow{\mathbb{P}}\sigma_{0}$ by
(B5), we may assume that $|\widehat{\sigma}_{n}-\sigma_{0}|\leq\varepsilon$
for all $\varepsilon>0$ and large $n$. Choose $\varepsilon$ small enough so
that (B6) is satisfied and $\sigma_{0}-\varepsilon>0$. Since
$\widehat{\sigma}_{n}\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$, we
have

$\displaystyle
M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})\geq\inf_{|\sigma-\sigma_{0}|\leq\varepsilon}\left[M(\widetilde{\beta}_{n},\sigma)-M(\beta_{0},\sigma)\right].$

By (B6) and after interchanging expectation and differentiation, which is
permitted by the boundedness of $\psi$, we see that

$\displaystyle\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+d_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)\right\\}$
$\displaystyle=\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)\right\\}+\frac{d_{i}}{\sigma}g_{\sigma}\left(\frac{\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)+O(d_{i}^{2})$
$\displaystyle=\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)\right\\}+O\left(\left|d_{i}\langle
X_{i},\beta_{0}-\beta\rangle_{p}\right|\right)+O(d_{i}^{2}),$

where to obtain the second equality we have used the fact that
$g_{\sigma}(0)=\mathbb{E}\\{\psi(\epsilon_{1}/\sigma)\\}=0$ for every
$\sigma\in[\sigma_{0}-\epsilon,\sigma_{0}+\epsilon]$. Similarly,

$\displaystyle\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+d_{i}}{\sigma}\right)\right\\}=\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}+O(d_{i}^{2}).$

Combining these two expansions and averaging, we find

$\displaystyle M(\beta,\sigma)-M(\beta_{0},\sigma)$
$\displaystyle\geq\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)-\rho\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}-\frac{C}{n}\sum_{i=1}^{n}d_{i}^{2}-$
$\displaystyle\quad-C\left(\frac{1}{n}\sum_{i=1}^{n}d_{i}^{2}\right)^{1/2}\left\|\beta-\beta_{0}\right\|_{n,p},$

by the Cauchy-Schwarz inequality, for some large $C>0$. Reasoning as in (30)
in the proof of Theorem 1, we have

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}_{\epsilon_{i}}\left\\{\rho\left(\frac{\epsilon_{i}+\langle
X_{i},\beta_{0}-\beta\rangle_{p}}{\sigma}\right)-\rho\left(\frac{\epsilon_{i}}{\sigma}\right)\right\\}$
$\displaystyle\geq\varepsilon\frac{\left\|\beta-\beta_{0}\right\|^{2}_{n,p}}{D_{\varepsilon}^{2}\log(n)(1+J_{m}(\beta))^{2}},$
(58)

for every $\beta\in\mathcal{H}^{m}(\mathbbm{R}^{d})$, with high probability.
The RHS of (58) is uniform in
$\sigma\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$. Furthermore, it
follows as in the proof of Theorem 1 that this inequality also holds for
$\widetilde{\beta}_{n}$. Therefore,

$\displaystyle
M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})$
$\displaystyle\geq\inf_{|\sigma-\sigma_{0}|\leq\varepsilon}\left[M(\widetilde{\beta}_{n},\sigma)-M(\beta_{0},\sigma)\right]$
$\displaystyle\geq\frac{\varepsilon\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|^{2}_{n,p}}{D_{\varepsilon}^{2}\log(n)\left(1+J_{m}(\widetilde{\beta}_{n})\right)^{2}}-\frac{C}{n}\sum_{i=1}^{n}d_{i}^{2}$
$\displaystyle\quad-C\left(\frac{1}{n}\sum_{i=1}^{n}d_{i}^{2}\right)^{1/2}\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p},$
(59)

which provides the desired lower bound for (A.7).

_Step 2_. To derive an upper bound for (A.7), use the fundamental theorem of
calculus to write

$\displaystyle\rho\left(\frac{\epsilon_{i}+d_{i}+\langle
X_{i},\beta_{0}-\widetilde{\beta}_{n}\rangle_{p}}{\widehat{\sigma}_{n}}\right)-\rho\left(\frac{\epsilon_{i}+d_{i}}{\widehat{\sigma}_{n}}\right)$
$\displaystyle=\frac{1}{\widehat{\sigma}_{n}}\int_{d_{i}}^{d_{i}+\langle
X_{i},\beta_{0}-\widetilde{\beta}_{n}\rangle_{p}}\left\\{\psi\left(\frac{\epsilon_{i}+t}{\widehat{\sigma}_{n}}\right)-\psi\left(\frac{\epsilon_{i}}{\widehat{\sigma}_{n}}\right)\right\\}\,\mathrm{d}\,t$
$\displaystyle\quad+\frac{1}{\widehat{\sigma}_{n}}\psi\left(\frac{\epsilon_{i}}{\widehat{\sigma}_{n}}\right)\langle
X_{i},\beta_{0}-\widetilde{\beta}_{n}\rangle_{p},$

so that, in the notation of Lemma 4,

$\displaystyle\widehat{\sigma}_{n}\left|\left(M_{n}(\beta_{0},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})\right)-\left(M_{n}(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})\right)\right|\leq\left|\frac{1}{n}\sum_{i=1}^{n}V_{i,\widetilde{\beta}_{n},\widehat{\sigma}_{n}}\right|+\left|\frac{1}{n}\sum_{i=1}^{n}Z_{i,\widetilde{\beta}_{n},\widehat{\sigma}_{n}}\right|.$

An application of Lemma 4 now yields

$\displaystyle\left(M_{n}(\beta_{0},\widehat{\sigma}_{n})-M(\beta_{0},\widehat{\sigma}_{n})\right)$
$\displaystyle-\left(M_{n}(\widetilde{\beta}_{n},\widehat{\sigma}_{n})-M(\widetilde{\beta}_{n},\widehat{\sigma}_{n})\right)$
$\displaystyle=O_{\mathbb{P}}\left(\sqrt{\frac{\log(n)}{n}}\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p}^{1-\frac{d}{2m}}\left\\{1+J_{m}(\widetilde{\beta}_{n})\right\\}^{\frac{d}{2m}}\right),$
(60)

as $\widehat{\sigma}_{n}\in[\sigma_{0}-\varepsilon,\sigma_{0}+\varepsilon]$
for all large $n$. Thus, (A.7) provides the desired upper bound for (A.7).

_Step 3_. Combining (A.7) with (A.7) and (A.7), we find

$\displaystyle\frac{\varepsilon\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|^{2}_{n,p}}{D_{\varepsilon}^{2}\log(n)\left\\{1+J_{m}\left(\widetilde{\beta}_{n}\right)\right\\}^{2}}+\lambda
J_{m}^{2}\left(\widetilde{\beta}_{n}\right)$ $\displaystyle\leq
O_{\mathbb{P}}\left(\sqrt{\frac{\log(n)}{n}}\right)\|\widetilde{\beta}_{n}-\beta_{0}\|_{n,p}^{1-\frac{d}{2m}}\\{1+J_{m}(\widetilde{\beta}_{n})\\}^{\frac{d}{2m}}+\lambda
J_{m}^{2}\left(\beta_{0}\right)$
$\displaystyle\quad+O_{\mathbb{P}}\left(\max_{1\leq j\leq
p}\operatorname{diam}^{2\kappa}(A_{j})\right)$
$\displaystyle\quad+O_{\mathbb{P}}\left(\max_{1\leq j\leq
p}\operatorname{diam}^{\kappa}(A_{j})\right)\left\|\widetilde{\beta}_{n}-\beta_{0}\right\|_{n,p},$

the only difference with the corresponding step in the proof of Theorem 1
being the presence of an additional $\sqrt{\log(n)}$-term on the RHS.
Reasoning along the lines of that proof and making use of $(i)$ and $(ii)$,
(57) can be easily verified.

_Step 4_. The last step in our proof involves establishing the same rate of
convergence for $\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}$ as
$\|\widetilde{\beta}_{n}-\beta_{0}\|_{n,p}$ and using
$J_{m}(\widetilde{\beta}_{n})=O_{\mathbb{P}}(1)$ in order to deduce
$J_{m}(\widehat{\beta}_{n})=O_{\mathbb{P}}(1)$. Both of these facts follow
from identical arguments as in the proof of Theorem 1 and a detailed proof is
thus omitted. This concludes the proof of Theorem 2.

###  A.8 Proof of Theorem 3

To prove that $\widehat{\sigma}_{n}\xrightarrow{\mathbb{P}}\sigma_{0}$ it
suffices to prove that
$\mathbb{P}(\widehat{\sigma}_{n}\geq\sigma_{0}+\varepsilon)\to 0$ and
$\mathbb{P}(\widehat{\sigma}_{n}\leq\sigma_{0}-\varepsilon)\to 0$ for every
$\varepsilon>0$. Consider first the event
$\\{\widehat{\sigma}_{n}\geq\sigma_{0}+\varepsilon\\}$ and recall that
$\widehat{\sigma}_{n}$ is the solution of

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\chi\left(\frac{\epsilon_{i}+d_{i}+(\alpha_{0}-\widehat{\alpha}_{n})+\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\beta_{0}(\mathbf{t}_{j})-\widehat{\beta}_{n}(\mathbf{t}_{j})\right)\mu(A_{j})}{\widehat{\sigma}_{n}}\right)=\frac{1}{2},$
(61)

where $\\{d_{i}\\}_{i=1}^{n}$ are the discretization errors introduced in (4)
and $(\widehat{\alpha}_{n},\widehat{\beta}_{n})$ are the thin-plate estimates.
By the monotonicity of $\chi$, on the set
$\\{\widehat{\sigma}_{n}\geq\sigma_{0}+\varepsilon\\}$ we have

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\chi\left(\frac{\epsilon_{i}+d_{i}+(\alpha_{0}-\widehat{\alpha}_{n})+\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\beta_{0}(\mathbf{t}_{j})-\widehat{\beta}_{n}(\mathbf{t}_{j})\right)\mu(A_{j})}{\sigma_{0}+\varepsilon}\right)>\frac{1}{2}.$
(62)

Now, by a first order Taylor expansion, the LHS of (62) may be rewritten as

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\chi\left(\frac{\epsilon_{i}}{\sigma_{0}+\varepsilon}\right)+\underbrace{\frac{1}{n(\sigma_{0}+\varepsilon)}\sum_{i=1}^{n}\chi^{\prime}\left(\epsilon_{i}^{\star}\right)\left(d_{i}+\left(\alpha_{0}-\widehat{\alpha}_{n}\right)+\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\beta_{0}(\mathbf{t}_{j})-\widehat{\beta}_{n}(\mathbf{t}_{j})\right)\mu(A_{j})\right)}_{S_{n}}$

for some mean values $\left\\{\epsilon_{i}^{\star}\right\\}_{i=1}^{n}$. But,
$\chi^{\prime}$ is bounded and, as we show in the proof of Theorem 1 (see (38)
there), $n^{-1}\sum_{i=1}^{n}|d_{i}|^{2}=o_{\mathbb{P}}(1)$,
$|\widehat{\alpha}_{n}-\alpha_{0}|=o_{\mathbb{P}}(1)$ and
$\|\widehat{\beta}_{n}-\beta_{0}\|_{n,p}=o_{\mathbb{P}}(1)$. Combining these
facts along with the triangle and Cauchy-Schwarz inequalities we now see that

$\displaystyle\left|S_{n}\right|$
$\displaystyle=\left|\frac{1}{n(\sigma_{0}+\varepsilon)}\sum_{i=1}^{n}\chi^{\prime}\left(\epsilon_{i}^{\star}\right)\left(d_{i}+\left(\alpha_{0}-\widehat{\alpha}_{n}\right)+\sum_{j=1}^{p}X_{i}(\mathbf{t}_{j})\left(\beta_{0}(\mathbf{t}_{j})-\widehat{\beta}_{n}(\mathbf{t}_{j})\right)\mu(A_{j})\right)\right|$
$\displaystyle\quad\leq\frac{\left\|\chi^{\prime}\right\|_{\infty}}{\sigma_{0}+\varepsilon}\left(\frac{1}{n}\sum_{i=1}^{n}\left|d_{i}\right|+\left|\widehat{\alpha}_{n}-\alpha_{0}\right|+\frac{1}{n}\sum_{i=1}^{n}\left|\langle
X_{i},\beta_{0}-\widehat{\beta}_{n}\rangle_{p}\right|\right)$
$\displaystyle\quad\leq\frac{\left\|\chi^{\prime}\right\|_{\infty}}{\sigma_{0}+\varepsilon}\left(\left\\{\frac{1}{n}\sum_{i=1}^{n}\left|d_{i}\right|^{2}\right\\}^{1/2}+\left|\widehat{\alpha}_{n}-\alpha_{0}\right|+\left\|\widehat{\beta}_{n}-\beta_{0}\right\|_{n,p}\right)$
$\displaystyle\quad=o_{\mathbb{P}}(1).$

Therefore, on the set $\\{\widehat{\sigma}_{n}\geq\sigma_{0}+\varepsilon\\}$
we have

$\displaystyle\frac{1}{n}\sum_{i=1}^{n}\chi\left(\frac{\epsilon_{i}}{\sigma_{0}+\varepsilon}\right)+S_{n}>\frac{1}{2}.$
(63)

But the summands $\\{\chi(\epsilon_{i}/(\sigma_{0}+\varepsilon))\\}_{i=1}^{n}$
are i.i.d. and bounded so that, by the WLLN,
$n^{-1}\sum_{i=1}^{n}\chi(\epsilon_{i}/(\sigma_{0}+\varepsilon))\xrightarrow{\mathbb{P}}\mathbb{E}\\{\chi(\epsilon_{1}/(\sigma_{0}+\varepsilon))\\}$.
By definition of $\sigma_{0}$ as well as the strict monotonicity of $\chi$,
$\mathbb{E}\\{\chi(\epsilon_{1}/(\sigma_{0}+\epsilon))\\}<1/2$. Since
$S_{n}=o_{\mathbb{P}}(1)$, it now follows that

$\displaystyle\lim_{n\to\infty}\mathbb{P}\left(\widehat{\sigma}_{n}\geq\sigma_{0}+\varepsilon\right)\leq\lim_{n\to\infty}\mathbb{P}\left(\frac{1}{n}\sum_{i=1}^{n}\chi\left(\frac{\epsilon_{i}}{\sigma_{0}+\varepsilon}\right)+S_{n}>\frac{1}{2}\right)=0.$

A similar argument shows that
$\mathbb{P}(\widehat{\sigma}_{n}\leq\sigma_{0}-\varepsilon)\to 0$. Combining
these two limits establishes the result of the theorem.

## References

  * Adams and Fournier, (2003) Adams, R. A. and Fournier, J. J. F. (2003).  Sobolev spaces, volume 140 of Pure and Applied Mathematics (Amsterdam).  Elsevier/Academic Press, Amsterdam, second edition.
  * Adler and Taylor, (2007) Adler, R. J. and Taylor, J. E. (2007).  Random fields and geometry.  Springer Monographs in Mathematics. Springer, New York.
  * Boente et al., (2020) Boente, G., Salibián-Barrera, M., and Vena, P. (2020).  Robust estimation for semi-functional linear regression models.  Comput. Statist. Data Anal., 152:107041, 17.
  * Cardot et al., (2007) Cardot, H., Crambes, C., Kneip, A., and Sarda, P. (2007).  Smoothing splines estimators in functional linear regression with errors-in-variables.  Comput. Statist. Data Anal., 51(10):4832–4848.
  * Cardot et al., (2003) Cardot, H., Ferraty, F., and Sarda, P. (2003).  Spline estimators for the functional linear model.  Statist. Sinica, 13(3):571–591.
  * Conway, (1990) Conway, J. B. (1990).  A course in functional analysis, volume 96 of Graduate Texts in Mathematics.  Springer-Verlag, New York, second edition.
  * Crambes et al., (2009) Crambes, C., Kneip, A., and Sarda, P. (2009).  Smoothing splines estimators for functional linear regression.  Ann. Statist., 37(1):35–72.
  * Cucker and Smale, (2002) Cucker, F. and Smale, S. (2002).  On the mathematical foundations of learning.  Bull. Amer. Math. Soc. (N.S.), 39(1):1–49.
  * Duchon, (1977) Duchon, J. (1977).  Splines minimizing rotation-invariant semi-norms in Sobolev spaces.  In Constructive theory of functions of several variables (Proc. Conf., Math. Res. Inst., Oberwolfach, 1976), volume Vol. 571 of Lecture Notes in Math, pages pp 85–100. Springer, Berlin.
  * Ferraty and Vieu, (2006) Ferraty, F. and Vieu, P. (2006).  Nonparametric functional data analysis: Theory and practice.  Springer Series in Statistics. Springer, New York.
  * Green and Silverman, (1994) Green, P. J. and Silverman, B. W. (1994).  Nonparametric regression and generalized linear models. A roughness penalty approach, volume 58 of Monographs on Statistics and Applied Probability.  Chapman & Hall, London.
  * Gu, (2013) Gu, C. (2013).  Smoothing spline ANOVA models, volume 297 of Springer Series in Statistics.  Springer, New York, second edition.
  * Hall and Horowitz, (2007) Hall, P. and Horowitz, J. L. (2007).  Methodology and convergence rates for functional linear regression.  Ann. Statist., 35(1):70–91.
  * Horváth and Kokoszka, (2012) Horváth, L. and Kokoszka, P. (2012).  Inference for functional data with applications.  Springer Series in Statistics. Springer, New York.
  * Huber and Ronchetti, (2009) Huber, P. J. and Ronchetti, E. M. (2009).  Robust statistics.  Wiley Series in Probability and Statistics. John Wiley & Sons, Inc., Hoboken, NJ, second edition.
  * James et al., (2009) James, G. M., Wang, J., and Zhu, J. (2009).  Functional linear regression that’s interpretable.  Ann. Statist., 37(5A):2083–2108.
  * Kalogridis and Van Aelst, (2019) Kalogridis, I. and Van Aelst, S. (2019).  Robust functional regression based on principal components.  J. Multivariate Anal., 173:393–415.
  * Kalogridis and Van Aelst, (2023) Kalogridis, I. and Van Aelst, S. (2023).  Robust penalized estimators for functional linear regression.  J. Multivariate Anal., 194:Paper No. 105104, 18.
  * Kato, (2012) Kato, K. (2012).  Estimation in functional linear quantile regression.  Ann. Statist., 40(6):3108–3136.
  * Kokoszka and Reimherr, (2017) Kokoszka, P. and Reimherr, M. (2017).  Introduction to functional data analysis.  Texts in Statistical Science Series. CRC Press, Boca Raton, FL.
  * Li and Hsing, (2007) Li, Y. and Hsing, T. (2007).  On rates of convergence in functional linear regression.  J. Multivariate Anal., 98(9):1782–1804.
  * Maronna et al., (2019) Maronna, R. A., Martin, R. D., Yohai, V. J., and Salibián-Barrera, M. (2019).  Robust statistics: Theory and methods (with R).  John Wiley & Sons.
  * Maronna and Yohai, (2013) Maronna, R. A. and Yohai, V. J. (2013).  Robust functional linear regression based on splines.  Comput. Statist. Data Anal., 65:46–55.
  * Nagy, (2023) Nagy, S. (2023).  Package ’RobustSpline’.
  * Nychka et al., (2021) Nychka, D., Furrer, R., Paige, J., and Sain, S. (2021).  fields: Tools for spatial data.  R package version 14.1. https://github.com/dnychka/fieldsRPackage.
  * Qingguo, (2017) Qingguo, T. (2017).  M-estimation for functional linear regression.  Comm. Statist. Theory Methods, 46(8):3782–3800.
  * Ramsay, (1982) Ramsay, J. O. (1982).  When the data are functions.  Psychometrika, 47(4):379–396.
  * Ramsay and Dalzell, (1991) Ramsay, J. O. and Dalzell, C. J. (1991).  Some tools for functional data analysis.  J. Roy. Statist. Soc. Ser. B, 53(3):539–572.
  * Ramsay and Silverman, (2005) Ramsay, J. O. and Silverman, B. W. (2005).  Functional data analysis.  Springer Series in Statistics. Springer, New York, second edition.
  * Reiss and Ogden, (2007) Reiss, P. T. and Ogden, R. T. (2007).  Functional principal component regression and functional partial least squares.  J. Amer. Statist. Assoc., 102(479):984–996.
  * Rousseeuw and Leroy, (1987) Rousseeuw, P. J. and Leroy, A. M. (1987).  Robust regression and outlier detection.  John Wiley & Sons.
  * Shin and Lee, (2016) Shin, H. and Lee, S. (2016).  An RKHS approach to robust functional linear regression.  Statist. Sinica, 26(1):255–272.
  * Stone, (1982) Stone, C. J. (1982).  Optimal global rates of convergence for nonparametric regression.  Ann. Statist., 10(4):1040–1053.
  * Utreras, (1988) Utreras, F. I. (1988).  Convergence rates for multivariate smoothing spline functions.  J. Approx. Theory, 52(1):1–27.
  * van de Geer, (2000) van de Geer, S. (2000).  Empirical Processes in M-Estimation.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, New York, NY.
  * van de Geer, (2002) van de Geer, S. (2002).  $M$-estimation using penalties or sieves.  J. Statist. Plann. Inference, 108(1-2):55–69.
  * Wahba, (1990) Wahba, G. (1990).  Spline models for observational data, volume 59 of CBMS-NSF Regional Conference Series in Applied Mathematics.  Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA.
  * Yohai and Zamar, (1988) Yohai, V. J. and Zamar, R. H. (1988).  High breakdown-point estimates of regression by means of the minimization of an efficient scale.  J. Amer. Statist. Assoc., 83(402):406–413.
  * Yuan and Cai, (2010) Yuan, M. and Cai, T. T. (2010).  A reproducing kernel Hilbert space approach to functional linear regression.  Ann. Statist., 38(6):3412–3444.


# Detecting anxiety from short clips of free-form speech

Prabhat Agarwal
Stanford University
<EMAIL_ADDRESS>
Akshat Jindal
Stanford University
<EMAIL_ADDRESS>
Shreya Singh
Stanford University
<EMAIL_ADDRESS>
(2 June 2020)

# Detecting anxiety from short clips of free-form speech

Prabhat Agarwal
Stanford University
<EMAIL_ADDRESS>
Akshat Jindal
Stanford University
<EMAIL_ADDRESS>
Shreya Singh
Stanford University
<EMAIL_ADDRESS>
(2 June 2020)

###### Abstract

Barriers to accessing mental health assessments including cost and stigma
continues to be an impediment in mental health diagnosis and treatment.
Machine learning approaches based on speech samples could help in this
direction. In this work, we develop machine learning solutions to diagnose
anxiety disorders from audio journals of patients. We work on a novel anxiety
dataset (provided through collaboration with Kintsugi Mindful Wellness Inc.)
and experiment with several models of varying complexity utilizing audio, text
and a combination of multiple modalities. We show that the multi-modal and
audio embeddings based approaches achieve good performance in the task
achieving an AUC ROC score of 0.68-0.69.

##  1 Introduction

Mental and physical health are equally important components of overall health
but there are many barriers to accessing mental health assessments including
cost and stigma. Even when individuals receive professional care, assessments
are intermittent and may be limited partly due to the episodic nature of
psychiatric symptoms. Therefore, machine‐learning technology using speech
samples obtained in the clinic, remotely over telephone or through recording
in personal mobile devices could help improve diagnosis and treatment. In this
work, we study the problem of detecting anxiety disorders through speech.
Since this is a relatively under-studied problem, the dataset is quite small
and hence we plan to explore different approaches to deal with the problem
including using hand-crafted features motivated by psychological studies on
anxiety and pretrained embeddings models. The dataset provided by Kintsugi
provides an interesting opportunity in this area as the recordings are self-
recorded in a comforting environment as opposed to a clinical setting. The
code is publicly available on Github111https://github.com/prabhat1081/Anxiety-
Detection-from-free-form-audio-journals.

The paper is outlined a follows: Section 2 gives details about the existing
literature in this problem domain. In Section 3, we describe the dataset used
for training our models along with the task description. In Section 4, we
provide a detailed explanation of our five models implemented for the task.
Section 5 outline the evaluation routines and presents the study results.
Finally, we conclude our findings in Section 6.

##  2 Related Works

Liu et. al. (Low et al., 2020) provides a systematic review of studies using
speech for automated assessments across a broader range of psychiatric
disorders and summarizes the different datasets, features, and methods that
have been employed in this area. The authors summarize the key acoustic
features used in these models across disorders. Many studies Horwitz et al.
(2013); Quatieri and Malyska (2012); Cummins et al. (2015); Singh et al.
(2019) have observed that depression is linked with a decrease in f0 and f0
range and also leads to jitter and shimmer with an increase in depression
severity. While jitter and shimmer are also correlated with anxiety Özseven et
al. (2018); Silber-Varod et al. (2016), many studies Gilboa-Schechtman et al.
(2014); Özseven et al. (2018); Weeks et al. (2012); Galili et al. (2013) have
found a significant increase in mean f0 in patients with an anxiety disorder.
Most studies surveyed by the authors employed simple machine learning models
like SVM, logistic regression and very few studies used more complex networks
with transfer from speech recognition tasks Singh et al. (2018). We plan to
utilise these features which have been well studied both in the psychology and
machine learning assessment of anxiety and depression in our work and compare
them against deep learning models both in terms of performance and
generalizability.

State-of-the-art works on detecting mental disorders or emotional states
(e.g., anxious vs. calm) from audio data use supervised learning approaches,
which must be “trained” from examples of the sound to be detected Kumari and
Singh (2017). In general, learning such classifiers requires “strongly
labeled” annotated data, where the segments of audio containing the desired
vocal event and the segments not containing that event are clearly indicated.
However getting strongly labelled data is tough an in our project too, we only
have weakly supervised data. Salekin et. al. Salekin et al. (2018); Sebastian
et al. (2019) presents a weakly-supervised framework for detecting symptomatic
individuals from their voice sample. The idea here is to collect long speech
audio samples from individuals already diagnosed with or high in symptoms of
specific mental disorders from situations that may heighten expression of the
symptoms of respective disorders. This type of data is considered “weakly
labeled”, meaning that although they provide information about the presence or
absence of disorder symptoms, they do not provide additional details such as
the precise times in the recording that indicate the disorder etc.

For handling weakly labelled data, they propose a new feature modelling
technique called “NN2Vec” to generate low-dimensional, continuous, and
meaningful representation of speech from long weakly labelled audio data as
the other conventional methods work well only with strongly labelled data. To
complement that, they present a Multiple instance learning paradigm which is a
slight adaptation to classic supervised learning leading to better results.
Their NN2Vec and Bi-LSTM MIL approach achieved an F-1 score of 90.1% and 90%
accuracy in detecting speakers high versus low in social anxiety symptoms.
This F-1 scores is 20.7% higher than of the best baselines they tried.

Recently multi-modal approaches Siriwardhana et al. (2020); Al Hanai et al.
(2018) to the speech analysis have shown promising results. Hanai et. al. Al
Hanai et al. (2018); Mohammed Abdulla et al. (2019) developed an automated
depression-detection algorithm that models interviews between an individual
and agent and learns from sequences of questions and answers without the need
to perform explicit topic modeling of the content Jindal et al. (2023). The
authors used the data of 142 individuals undergoing depression screening
through a human-controlled virtual agent which prompts each individual with a
subset of 170 possible queries that included direct questions (e.g. ‘How are
you?’, ‘Do you consider yourself to be an introvert?’), and dialogic feedback
( e.g. ‘I see’, ‘that sounds great’). The data was from the publicly available
distress analysis and interview corpus (DAIC) and contains audio and text
transcriptions of the spoken interactions Rajan et al. (2023); Rastogi et al.
(2023).

The authors modeled these interactions with audio and text features in Long-
Short Term Memory (LSTM) neural network model to detect depression and through
various experiments and ablation studies showed that not only did a
combination of modalities provide additional discriminative power, but that
they contained complementary information and hence it is quite useful to model
both.

a Anxiety score distribution

b Anxiety level distribution

Figure 1: Distribution of raw anxiety score and inferred anxiety levels

##  3 Dataset and Task Description

For our project, we are working with the Generalized Anxiety Disorder (GAD)
dataset from Kintsugi Mindful Wellness, Inc. The dataset consists of 2263
self-recorded audio journals of users along with corresponding score of the
user on the GAD-7 questionnaire taken by the user after recording the journal.
Each of these files is annotated with an unnormalized anxiety GAD-7 score and
a bucketed score. The unnormalized GAD-7 score ranges from 0 to 21 and Fig. 1a
shows its frequency distribution. These raw anxiety scores are further split
into 4 buckets of anxiety levels, None, mild, moderate and severe,
distribution of which is shown in Fig. 1b. The journals are of varying lengths
with most being around 1-2 minutes long. The distribution of the duration of
the audio journals in the dataset is shown in Fig. 2.

Figure 2: Distribution of the duration of audio files in the dataset

The dataset does not contain human transcripts for the journals and hence we
use the state-of-the-art Wav2Vec2 Baevski et al. (2020) to transcribe the
given audio files and analyze the textual content. In the transcriptions
generated by this model, there are a total of 244K words and 18K unique words.
The average number of words per conversation is 108. Fig.3 shows the frequency
distribution of the transcribed words.

Figure 3: Transcribed words distribution

We further use T5 Raffel et al. (2019) base model fine-tuned on emotion
recognition and sentiment analysis tasks on the transcribed text using
Huggingface Wolf et al. (2019). The emotion recognition model classifies each
transcribed audio file into one of the 5 emotions, namely, anger, fear, joy,
love, sadness while the sentiment analysis model classifies each transcript
into two classes, positive and negative.

Fig. 4 shows visualizes the distribution of different sentiments and emotion
in transcripts of anxiety and non-anxiety classes. We classify an audio
file/datapoint ‘anxiety’ if it belongs to the buckets, mild, moderate or
severe, and consequently, ‘non-anxiety’ are those datapoints which fall in the
None bucket (from Fig.1b).

a Sentiment distribution in anxiety and non–anxiety classes

b Emotion distribution in anxiety and non–anxiety classes

Figure 4: Sentiment and Emotion distribution in anxiety and non–anxiety
classes

###  3.1 Task Description

In this work, we focus on the binary task of detecting whether a user has
anxiety disorder or not given their journal recordings. We group the all
levels of anxiety, namely mild, moderate and severe as anxiety class for this
work. Table 1 shows the train, validation and test split statistics of the
dataset for the binary classification task (Anxiety vs Non-Anxiety).

Class | Train | Valid | Test | Total
---|---|---|---|---
No anxiety | 840 | 149 | 173 | 1162
Anxiety | 790 | 139 | 166 | 1095
Total | 1630 | 288 | 339 | 2257
Table 1: Data statistics for different splits

##  4 Approach

In this study, we sought to model self-recorded hournals of users in order to
detect if the individual has an anxiety disorder based on GAD-7 score Spitzer
et al. (2006). We conducted four sets of experiments with different audio, and
text features extracted from the audio data to predict anxiety.

  1. 1.

Model based hand-crafted audio features

  2. 2.

Model based on transcript embeddings

  3. 3.

Model based on transcript embeddings with sample weights

  4. 4.

Model based on low-dimensional audio embeddings using wav2vec

  5. 5.

Multi-modal model based both both audio and transcript features

Details of each approach is described below and our code for the experiments
can be found online.

###  4.1 Hand-crafted audio features

Work on voice sciences over recent decades has led to a development of
acoustic features for different speech analysis tasks. GeMAPS Eyben et al.
(2015) standard acoustic feature set for various areas of automatic voice
analysis, such as paralinguistic or clinical speech analysis and has proven to
be effective in various standard benchmarks. Hence we use the 88 function
features described in the GeMAPS extracted using the Opensmile tool Eyben et
al. (2010). Based on our analysis of the data, we further extend this feature
set with 2 features (emotion and sentiment) extracted from the generated
transcripts as described in section 3. We train a L1-regularized logistic
regression model on these 90 features extracted from the data to predict if
the individual has anxiety or not.

###  4.2 Transcript embeddings

In this model, we only use the transcripts generated from the audio and use
them for training out anxiety classifier. We don’t use any audio features in
this baseline and solely rely on the textual features, thereby converting this
problem to a natural language domain problem. Specifically, we use
Wave2vec2-Base model Baevski et al. (2020) to transcribe the given audio data.
Upon qualitatively evaluating the generated transcripts, we see that it is a
very close transcription with some grammatical errors. Some examples of the
transcripts are shown in table 2.

I think the best thing that happened for the past week is a being able to get
out a for litbit a besides work i think me magrofere mostly been in our house
and having enjoyed the outdoors …
---
I feel eh but lest less and demotivated there’s lot of work that i need to do
and i folting the ent on my responsibility is at work sometimes i think in my
personal life as well is also just general …
Table 2: Some snippets of generated transcripts from the audio data using
wav2vec2

We embed the generated transcripts in 768 low-dimensional space using the
pretrained sentence embedding model Sentence-BERT Reimers and Gurevych (2019)
which is a RoBERTa model fine–tuned for sentence similarity task. A Gradient
Boosting Classifier (GBC) model is trained on these embeddings to classify
presence of anxiety.

###  4.3 Transcript embeddings with sample weights

In our dataset, in addition to having the binary labels of anxiety we also
have the raw GAD-7 scores for the users which range between $0$ and $21$. We
explore in this model how using these score to assign different weights to
different examples might help the a better anxiety prediction model. We
utilize a simple linear weighting strategy and define the weights as

$w_{i}=\frac{s_{i}+1}{22}$

where $s_{i}$ is the raw GAD-7 score for the $i^{th}$ datapoint in the dataset
and $w_{i}$ is its corresponding weight. Intuitively, by giving a less sample
weight to a ‘non-anxiety’ datapoint and giving a high sample weight to ‘high-
anxiety’ datapoint, we are biasing the model to be reduce the number of false
negatives at the cost of potentially producing more false positives.

###  4.4 Wav2vec audio embeddings

Unsupervised representations have shown to be quite effective in various
machine learning tasks across different domain like image, text and audio.
Hence we explore a model for our task using one of the best unsupervised
representation model for speech data. We use Wav2Vec model Schneider et al.
(2019) to generate 512 dimensional embeddings for the user recordings. The
model takes raw audio at sample rate of 16kHz as input and outputs a sequence
of 512 dimensional embeddings for the input at two levels, $z$ and $c$ as
shown in figure 5. We take the mean of this sequence of embeddings at the $z$
level to construct our final 512 dimensional embedding for the input audio
file. We then train a SVM classifier on these embeddings for our anxiety
prediction task.

Figure 5: Illustration of the model architecture of wav2vec Schneider et al.
(2019). We use the mean of the $z$-layer embeddings for our anxiety model.

###  4.5 Multi-modal model with audio and text

For our final model, we decided to develop a multi-modal classifcation model
combining the information from both the audio and the text transcriptions.
Inspired by the multi-modal model for speech emotion recognition in
Siriwardhana et. al. Siriwardhana et al. (2020) we generate audio features and
text features and use their concatenation to to train a classifier to make
predictions for our task. A overall schematic representation of the model is
shown in figure 6. Details of the different components of the model is
described below.

Figure 6: Schematic for Multi-modal anxiety classification model

  * •

Audio quantizer: VQ-Wav2Vec Baevski et al. (2019): An extension of Wav2Vec
Schneider et al. (2019), VQ-Wav2Vec also learns continuous representations by
optimizing the CPC objective (Contrastive Predictive Coding) Oord et al.
(2018) and then quantizing it either via a Gumbell Softmax or a K-means
approach. For our purposes, we use the pretrained K-means quantizer model.

  * •

Quantized audio sequence embedder: Speech-RoBERTa Liu et al. (2019): Then, the
quantized audio tokens are fed into a pre-trained RoBERTa model, trained via
self-supervision on masked token prediction using quantized vector sequences
from Librispeech-960. This gives us dense representations of the audio signal
(We only retain the $CLS$ (call it $CLS_{speech}$) token embedding which is
the first token embedding) and has been shown to perfrom well for sequence
classification tasks.

  * •

Transcript embedder: RoBERTa Liu et al. (2019): The text features are
generated by tokenizing and feed-forwarding the transcriptions through a
standard RoBERTa model pre-trained on the masked LM task as described in Liu
et al. (2019). This gives us dense representations of the transcriptions (We
only retain the $CLS$ (call it $CLS_{text}$) token embedding which is the
first token embedding).

  * •

SVM Classifier We concatenate $CLS_{text}$ and $CLS_{speech}$ and then train
an SVM classifier on these features to solve the binary classification task.

##  5 Experiments

The results of the experiments, as well as a random baseline are displayed in
Table 3.

Model | Precision | Recall | F1 | AUROC
---|---|---|---|---
Random baseline | 0.50 | 0.48 | 0.49 | 0.50
Audio features | 0.63 | 0.53 | 0.58 | 0.66
Transcipt features | 0.64 | 0.57 | 0.60 | 0.68
Transcript features with sample weights | 0.61 | 0.55 | 0.58 | 0.59
Wav2Vec features | 0.66 | 0.61 | 0.64 | 0.69
Multi-modal model | 0.66 | 0.60 | 0.61 | 0.68
Table 3: Performance of different models

###  5.1 Hand-crafted audio features

The results of the logistic regression model with L1 regularization is shown
in Table 3. The model improves upon the random baseline by around 20%. We
analyse the effect of different features on the model predictions using SHAP
Lundberg and Lee (2017) and a visualization of the same is shown in Fig. 7. As
can be seen from the figure, the top features are loudness, F0, emotion,
sentiment, etc. which have been shown in clinical literature to be signals in
patients suffering from anxiety.

Figure 7: Feature importance of different features for anxiety prediction as
obtained using logistic regression model.

###  5.2 Transcript embeddings

The results of the GBC model trained on transcript embeddings are shown in
Table 3. The model performs similar to the model based on audio features
signifying the importance and effectiveness of text features in mental health
related voice classification systems.

The results of adding sample weights to the model is also shown in Table 3 and
does not improve the performance but rather degrades model performance. Hence
we did not experiment with sample weights for our other models.

###  5.3 Wav2vec audio embeddings

As described in section 4, we embed the audio recording into 512 low
dimensional embedding space and train an SVM classifier to predict anxiety. We
tune the different hyperparameters using the validation set and find $C=10$
for perform the best. The results of the best model on the test set are shown
in Table 3. The ROC curve and the Precision-Recall curve for the model’s
performance on the test set is shown in figure 8. We can see that the model
trained on Wav2vec features improved about 30% over the random baseline. We
also analyse the ROC and precision-recall curve for the model and it is shown
in Fig. 8. As we can see that the precision is almost the same as recall
increases from 0.2 to 0.6 signifying model’s robustness.

a ROC curve

b Precision Recall curve

Figure 8: ROC curve and Precision Recall curve for the Wav2Vec features model

###  5.4 Multi-modal model with audio and text

For this experiment, as described in Section 4, we firstly used a pre-trained
VQ-Wav2Vec model (on Librispeech-960), with a K-means quantizer as provided in
the code associated with Siriwardhana et al. (2020) to quantize our audio
signal. Then we used their pre-trained speech-Roberta model which has the same
architecture as the standard RoBERTa-base model Liu et al. (2019) to obtain
embeddings of size 768 with a max sequence length of 2048. We retain the first
token $CLS_{speech}$ embedding. We then create feature representations for
text by tokenizing and passing it through a roBERTa-large Liu et al. (2019)
model to generate embeddings of size 1024 with a max sequence length of 512.
We combine the $CLS_{text}$ embedding from to $CLS_{speech}$ to create a 1792
dimensional embedding. We then train an SVM classifier on these features to
solve the binary classification task. The results are shown in Table 3. The
model does not improve much over the wav2vec embeddings and understanding the
reason and improving the model is a direction to explore in the future.

##  6 Conclusions

Mental Health is a very important component of overall health which receives
very less attention and importance from the masses. People are more focused
towards physical health and ignore their mental well-being which can be
disastrous if not dealt with in the earlier stages. In this project, we aim to
marry ML-powered speech processing techniques with mental health speech data
to automatically detect anxiety from short clips of free-form speech. Our
learnings are two fold: (i) Firstly, we implemented various ML powered models
of varying complexity on the speech dataset. We start with a random model
which gives us about 50% ROC and treat that as our baseline to improve upon.
Our subsequent models are a mix of only text, only audio and a combination of
both. (ii) We also gained the experience of working with a very small-sized
dataset which can lead to overfitting problems. Hence, our modelling
approaches had to be made powerful enough to learn features from a small
dataset, while also being robust against overfitting.

In the future, we would like to apply techniques like transfer learning for
our task. Our idea is to train our models for depression detection task (which
is a larger dataset) and fine-tune it on the anxiety dataset. Due to
similarities between the nature of anxiety and depression disorders, we
believe this approach will help us build more powerful models for anxiety
detection which are prone to overfitting.

## References

  * Al Hanai et al. (2018) Tuka Al Hanai, Mohammad M Ghassemi, and James R Glass. 2018.  Detecting depression with audio/text sequence modeling of interviews.  In _Interspeech_ , pages 1716–1720.
  * Baevski et al. (2019) Alexei Baevski, Steffen Schneider, and Michael Auli. 2019.  vq-wav2vec: Self-supervised learning of discrete speech representations.  _arXiv preprint arXiv:1910.05453_.
  * Baevski et al. (2020) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _arXiv preprint arXiv:2006.11477_.
  * Cummins et al. (2015) Nicholas Cummins, Stefan Scherer, Jarek Krajewski, Sebastian Schnieder, Julien Epps, and Thomas F Quatieri. 2015.  A review of depression and suicide risk assessment using speech analysis.  _Speech Communication_ , 71:10–49.
  * Eyben et al. (2015) Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. 2015.  The geneva minimalistic acoustic parameter set (gemaps) for voice research and affective computing.  _IEEE transactions on affective computing_ , 7(2):190–202.
  * Eyben et al. (2010) Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010.  Opensmile: the munich versatile and fast open-source audio feature extractor.  In _Proceedings of the 18th ACM international conference on Multimedia_ , pages 1459–1462.
  * Galili et al. (2013) Lior Galili, Ofer Amir, and Eva Gilboa-Schechtman. 2013.  Acoustic properties of dominance and request utterances in social anxiety.  _Journal of social and clinical psychology_ , 32(6):651–673.
  * Gilboa-Schechtman et al. (2014) Eva Gilboa-Schechtman, Lior Galili, Yair Sahar, and Ofer Amir. 2014.  Being “in” or “out” of the game: subjective and acoustic reactions to exclusion and popularity in social anxiety.  _Frontiers in human neuroscience_ , 8:147.
  * Horwitz et al. (2013) Rachelle Horwitz, Thomas F Quatieri, Brian S Helfer, Bea Yu, James R Williamson, and James Mundt. 2013.  On the relative importance of vocal source, system, and prosody in human depression.  In _2013 IEEE International Conference on Body Sensor Networks_ , pages 1–6. IEEE.
  * Jindal et al. (2023) Akshat Jindal, Shreya Singh, and Soham Gadgil. 2023.  Classification for everyone: Building geography agnostic models for fairer recognition.  _arXiv preprint arXiv:2312.02957_.
  * Kumari and Singh (2017) SDGV Akanksha Kumari and Shreya Singh. 2017.  Parallelization of alphabeta pruning algorithm for enhancing the two player games.  _Int. J. Advances Electronics Comput. Sci_ , 4:74–81.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Low et al. (2020) Daniel M Low, Kate H Bentley, and Satrajit S Ghosh. 2020.  Automated assessment of psychiatric disorders using speech: A systematic review.  _Laryngoscope Investigative Otolaryngology_ , 5(1):96–116.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 4765–4774. Curran Associates, Inc.
  * Mohammed Abdulla et al. (2019) G Mohammed Abdulla, Shreya Singh, and Sumit Borar. 2019.  Shop your right size: A system for recommending sizes for fashion products.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , WWW ’19, page 327–334, New York, NY, USA. Association for Computing Machinery.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_.
  * Özseven et al. (2018) Turgut Özseven, Muharrem Düğenci, Ali Doruk, and Hilal İ Kahraman. 2018.  Voice traces of anxiety: acoustic parameters affected by anxiety disorder.  _Archives of Acoustics_ , pages 625–636.
  * Quatieri and Malyska (2012) Thomas F Quatieri and Nicolas Malyska. 2012.  Vocal-source biomarkers for depression: A link to psychomotor activity.  In _Thirteenth Annual Conference of the International Speech Communication Association_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajan et al. (2023) Charles Rajan, Nishit Asnani, and Shreya Singh. 2023.  Shaping political discourse using multi-source news summarization.  _arXiv preprint arXiv:2312.11703_.
  * Rastogi et al. (2023) Chetanya Rastogi, Prabhat Agarwal, and Shreya Singh. 2023.  Exploring graph based approaches for author name disambiguation.  _arXiv preprint arXiv:2312.08388_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Salekin et al. (2018) Asif Salekin, Jeremy W Eberle, Jeffrey J Glenn, Bethany A Teachman, and John A Stankovic. 2018.  A weakly supervised learning framework for detecting social anxiety and depression.  _Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies_ , 2(2):1–26.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _CoRR_ , abs/1904.05862.
  * Sebastian et al. (2019) Abraham Gerard Sebastian, Shreya Singh, P. B. T. Manikanta, T. S. Ashwin, and G. Ram Mohana Reddy. 2019.  Multimodal group activity state detection for classroom response system using convolutional neural networks.  In _Recent Findings in Intelligent Computing Techniques_ , pages 245–251, Singapore. Springer Singapore.
  * Silber-Varod et al. (2016) Vered Silber-Varod, Hamutal Kreiner, Ronen Lovett, Yossi Levi-Belz, and Noam Amir. 2016.  Do social anxiety individuals hesitate more? the prosodic profile of hesitation disfluencies in social anxiety disorder individuals.  _Speech Prosody 2016_ , pages 1211–1215.
  * Singh et al. (2019) Loveperteek Singh, Shreya Singh, Sagar Arora, and Sumit Borar. 2019.  One embedding to do them all.  _arXiv preprint arXiv:1906.12120_.
  * Singh et al. (2018) Shreya Singh, G Mohammed Abdulla, Sumit Borar, and Sagar Arora. 2018.  Footwear size recommendation system.  _arXiv preprint arXiv:1806.11423_.
  * Siriwardhana et al. (2020) Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, and Suranga Nanayakkara. 2020.  Jointly fine-tuning” bert-like” self supervised models to improve multimodal speech emotion recognition.  _arXiv preprint arXiv:2008.06682_.
  * Spitzer et al. (2006) Robert L Spitzer, Kurt Kroenke, Janet BW Williams, and Bernd Löwe. 2006.  A brief measure for assessing generalized anxiety disorder: the gad-7.  _Archives of internal medicine_ , 166(10):1092–1097.
  * Weeks et al. (2012) Justin W Weeks, Chao-Yang Lee, Alison R Reilly, Ashley N Howell, Christopher France, Jennifer M Kowalsky, and Ashley Bush. 2012.  “the sound of fear”: Assessing vocal fundamental frequency as a physiological indicator of social anxiety disorder.  _Journal of anxiety disorders_ , 26(8):811–822.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.