# Optimal Activation Functions for the
Random Features Regression Model

Jianxin Wang
Department of Computer Science
Boston College
<EMAIL_ADDRESS>&José Bento
Department of Computer Science
Boston College
<EMAIL_ADDRESS>
###### Abstract

The asymptotic mean squared test error and sensitivity of the Random Features
Regression model (RFR) have been recently studied. We build on this work and
identify in closed-form the family of Activation Functions (AFs) that minimize
a combination of the test error and sensitivity of the RFR under different
notions of functional parsimony. We find scenarios under which the optimal AFs
are linear, saturated linear functions, or expressible in terms of Hermite
polynomials. Finally, we show how using optimal AFs impacts well established
properties of the RFR model, such as its double descent curve, and the
dependency of its optimal regularization parameter on the observation noise
level.

##  1 Introduction

For many neural network architectures, the test error does not monotonically
increase as a model’s complexity increases but can go down with the training
error both at low and high complexity levels. This phenomenon, the _double
descent curve_ , defies intuition and has motivated new frameworks to explain
it. Explanations have been advanced involving linear regression with random
covariates [1, 2], kernel regression [3, 4], the _neural tangent kernel_ model
[5], and the _Random Features Regression_ (RFR) model [6]. These frameworks
allow queries beyond the generalization power of neural networks. For example,
they have been used to study networks’ robustness properties [7, 8].

One aspect within reach and unstudied to this day is finding optimal
Activation Functions (AFs) for these models. It is known that AFs affect a
network’s approximation accuracy and efforts optimize AFs have been
undertaken. Previous work has justified the choice of AFs empirically, e.g.,
[9], or provided numerical procedures to learn AF parameters, sometimes
jointly with models’ parameters, e.g. [10]. See [11] for commonly used AFs and
Section 2 for how AFs have been previously derived.

We derive for the first time closed-form optimal AFs such that an explicit
objective function involving the asymptotic test error and sensitivity of a
model is minimized. Setting aside empirical and principled but numerical
methods, all past principled and analytical approaches to design AFs focus on
non accuracy related considerations, e.g. [12]. We focus on AFs for the RFR
model and expand its theoretical understanding. We preview a few surprising
conclusions extracted from our main results:

  1. 1.

The optimal AF can be linear, in which case the RFR model is a linear model.
For example, if no regularization is used for training, and for low complexity
models, a linear AF is often preferred if we want to minimize test error. For
high complexity models a non-linear AF is often better;

  2. 2.

A linear optimal AF can destroy the double descent curve behaviour and achieve
small test error with much fewer samples than e.g. a ReLU;

  3. 3.

When, apart from the test error, the sensitivity of a model becomes important,
optimal AFs that without sensitivity considerations were linear can become
non-linear, and vice-versa;

  4. 4.

Using an optimal AF with an arbitrary regularization during training can lead
to the same, or better, test error as using a non-optimal AF, e.g. ReLU, and
optimal regularization.

###  1.1 Problem set up

We consider the effect of AFs on finding an approximation $f$ to a square-
integrable function $f_{d}$ on the $d$-dimensional sphere
$\mathbb{S}^{d-1}(\sqrt{d})$, the function $f_{d}$ having been randomly
generated. The approximation $f$ is to be learnt from training data
$\mathcal{D}=\\{{{\bm{x}}}_{i},y_{i}\\}^{n}_{i=1}$ where
${{\bm{x}}}_{i}\in\mathbb{S}^{d-1}(\sqrt{d})$, the variables
$\\{{\bm{x}}_{i}\\}^{n}_{i=1}$ are i.i.d. uniformly sampled from
$\mathbb{S}^{d-1}(\sqrt{d})$, and $y_{i}=f_{d}({\bm{x}}_{i})+\epsilon_{i}$,
where the noise variables $\\{\epsilon_{i}\\}^{n}_{i=1}$ are i.i.d. with
$\operatorname{\mathbb{E}}(\epsilon_{i})=0,\operatorname{\mathbb{E}}(\epsilon^{2}_{i})=\tau^{2}$,
and $\operatorname{\mathbb{E}}(\epsilon^{4}_{i})<\infty$.

The approximation $f$ is defined according to the RFR model. The RFR model can
be viewed as a two-layer neural network with random first-layer weights
encoded by a matrix ${\bm{\Theta}}\in\mathbb{R}^{N\times d}$ with $i$th row
${\bm{\theta}}_{i}\in\mathbb{R}^{d}$ satisfying
$\|{\bm{\theta}}_{i}\|=\sqrt{d}$, with $\\{{\bm{\theta}}_{i}\\}$ i.i.d.
uniform on $\mathbb{S}^{d-1}(\sqrt{d})$, and with to-be-learnt second-layer
weights encoded by a vector ${\bm{a}}=[a_{i}]^{N}_{i=1}=\mathbb{R}^{N}$.
Unless specified otherwise, the norm $\|\cdot\|$ denotes the Euclidean norm.
The RFR model defines
$f_{{\bm{a}},{\bm{\Theta}}}:\mathbb{S}^{d-1}(\sqrt{d})\mapsto\mathbb{R}$ such
that

$\displaystyle
f_{{\bm{a}},{\bm{\Theta}}}({\bm{x}})=\sum_{i=1}^{N}a_{i}\sigma(\langle{\bm{\theta}}_{i},{\bm{x}}\rangle/\sqrt{d}).$
(1)

where $\sigma(\cdot)$ is the AF that is the target of our study and $\langle
x,y\rangle$ denotes the inner product between vectors $x$ and $y$. When clear
from the context, we write $f_{{\bm{a}},{\bm{\Theta}}}$ as $f$, omitting the
model’s parameters. The optimal weights ${\bm{a}}^{\star}$ are learnt using
ridge regression with regularization parameter $\lambda\geq 0$, namely,

$\displaystyle{\bm{a}}^{\star}={\bm{a}}^{\star}(\lambda,\mathcal{D})=\arg\min_{{\bm{a}}\in{\mathbb{R}}^{N}}\left\\{\frac{1}{n}\sum_{j=1}^{n}\Big{(}y_{j}-\sum_{i=1}^{N}a_{i}\sigma(\langle{\bm{\theta}}_{i},{\bm{x}}_{j}\rangle/\sqrt{d})\Big{)}^{2}+\frac{N\lambda}{d}\,\|{\bm{a}}\|^{2}\right\\}.$
(2)

We will tackle this question: What is the simplest $\sigma$ that leads to the
best approximation of $f_{d}$?

We quantify the simplicity of an AF $\sigma$ with its norm in different
functional spaces. Namely, either
$\|\sigma\|_{1}\triangleq\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z))|),\quad\quad\text{
or }$ (3)
$\|\sigma\|_{2}\triangleq\sqrt{\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})},$
(4)

where $\sigma^{\prime}$ is the derivative of $\sigma$ and the expectations are
with respected to a normal random variable $Z$ with zero mean and unit
variance, i.e. $Z$ $\sim$ $\mathcal{N}(0,1)$. We quantify the quality with
which $f=f_{{\bm{a}}^{\star},\Theta}$ approximates $f_{d}$ via $O$, a linear
combination of the mean squared error and the sensitivity of $f$ to
perturbations in its input (cf. [13]). For $\alpha\in[0,1]$ and ${\bm{x}}$
uniform on $\mathbb{S}^{d-1}(\sqrt{d})$, we define

$O\triangleq(1-\alpha)\mathcal{E}+\alpha\mathcal{S},$ (5)

$\text{ where
}\mathcal{E}\triangleq\operatorname{\mathbb{E}}((f({\bm{x}})-f_{d}({\bm{x}}))^{2}),$
(6)

$\text{ and
}\mathcal{S}\triangleq\operatorname{\mathbb{E}}(\|\nabla_{x}f({\bm{x}})\|^{2}).$
(7)

Like in [6, 14], we operate in an asymptotic setting where $n,d,N\to\infty$,
and have constant ratios between them, namely, $N/d\to\psi_{1}$ and
$n/d\to\psi_{2}$. In this asymptotic setting, it does not matter if in
defining (6) and (7), in addition to taking the expectation with respect to
the test data ${\bm{x}}$, independently of $\mathcal{D}$, we also take
expectations over $\mathcal{D}$ and the random features in RFR. This is
because when $n,d,N\to\infty$ with the ratios defined above, $\mathcal{E}$ and
$\mathcal{S}$ will concentrate around their means [6, 14].

Mathematically, denoting by $\|\sigma\|$ either (4) or (3), our goal is to
study the solutions of the problem

$\min_{\sigma^{\star}}\|\sigma^{\star}\|\text{ subject to
}\sigma^{\star}\in\arg\min_{\sigma}O(\sigma).$ (8)

Notice that the outer optimization only affects the selection of optimal AF in
so far as the inner optimization does not uniquely define $\sigma^{\star}$,
which, as we will later see, it does not.

##  2 Related work

First attempts to optimize AFs include [15], [16], and [17], where genetic and
evolutionary algorithms were used to learn how to numerically combine
different AFs from a library into the same network. More recently, [9] used
reinforcement learning to empirically discover AFs that minimize test
accuracy. Their search was done over AFs that were a combination of basic
units. This work produced the _Swish_ AF. Similarly, [18] defined AFs as the
weighted sum of a pre-defined basis and searched for optimal weights via
training. The authors in [10] provided a theoretical foundation to
simultaneously learn a neural network’s weights and continuous piecewise-
linear AFs. They showed that learning in their framework is compatible with
learning in current existing deep-ReLU, parametric ReLU, APL (adaptive
piecewise-linear) and MaxOut architectures. The work [19] parameterized
continuous piece-wise linear AFs and numerically learnt their parameters to
improve both accuracy and robustness to adversarial perturbations. They
numerically compared the performance of their SPLASH framework with that of
using ReLUs, leaky-ReLUs [20], PReLUs [21], $\tanh$ units, sigmoid units, ELUs
[22], maxout units [23], Swish units, and APL units [24]. Similarly, [25]
parametrized AFs as piece-wise linear units and learnt the AFs parameters to
optimize solving different tasks. [26] proposed an empirical method to learn
variations of ReLUs. [27] studied two-layer neural networks and gave a
condition on the Lipschitz constant of a polynomial AF for the network to
perfectly fit data. They related this condition to the model’s parameter-size
and robustness. They numerically investigated the relationship between the
number of ReLUs in the model and its robustness.

Several papers proposed new AFs and empirically studied their performance
without systematically tuning them. In [12] the authors identified ReLU and
Swish as naturally arising components of a statistical mechanics inspired
model for a neural network. The authors in [28] introduced a “tent”-shaped AF
that improves robustness without adversarial training, while not hurting the
accuracy on non-adversarial examples. [29] proposed a new AF called SRS that
can overcome the non-zero mean, negative missing, and unbounded output in
ReLUs. Their work was purely empirical. [30] developed the SQuared Natural Law
AF. [31] proposed the Piece-wise Linear Unit AF.

The RFR model was introduced by [32] as a way to project input data into a low
dimensional random features space and it has since then been studied
considerably. A great part of the literature has drawn connections between the
expressive power of neural networks and that of the RFR model, often via the
study of Gaussian processes. For example, the work [33] did this in the
context of shallow but infinitely wide neural network and the works [34, 35,
36, 37] did this for deep networks. Other works [38, 39] connected the RFR
model to training a neural network with gradient descent.

In addition to [6, 14], already discussed, other papers studied the
approximation properties of the RFR model. The authors in [40] studied both
the RFR model and the neural tangent kernel model and provided conditions
under which these models can fit polynomials in the raw features up to a
maximum degree. These conditions were provided under two regimes, when
$n\to\infty$ and $N,d$ large but finite, or when $N\to\infty$ and $n,d$ large
but finite. Their results hold under weak assumptions on the AFs. The work [8]
used the RFR model to compute how robust the test error is to distribution
shifts between training and test data. This was done in a high-dimensional
asymptotic limit when random features and training data are normal
distributed. The derivations hold for a generic AF that satisfies some mild
assumptions similar to the assumptions in this paper. The recent work of [7]
characterized the role of overparametrization on the adversarial robustness
for the RFR under an asymptotic regime when learning a linear function using a
RFR model with normal-distributed random weights and normal samples. In their
work, the AF was fixed to a shifted ReLU.

##  3 Background on the asymptotic properties of the RFR model

We review recently derived closed-form expressions for the asymptotic (model’s
dimensions to infinity) mean squared error and sensitivity of the RFR model,
which are the starting point of our work. We first need the following
assumptions, which we assume hold in the theorems in this section.

######  Assumption 1.

We assume that the AF $\sigma$ is weakly differentiable with weak derivative
$\sigma^{\prime}$, it satisfies $|\sigma(u)|,|\sigma^{\prime}(u)|\leq
c_{0}e^{c_{1}|u|}\forall u\in\mathbb{R}$ for some constants
$0<c_{0},c_{1}<\infty$, and that it also satisfies

$\mu_{0}=\operatorname{\mathbb{E}}\\{\sigma(Z)\\},\hskip
11.38092pt\mu_{1}=\operatorname{\mathbb{E}}\\{Z\sigma(Z)\\},\hskip
11.38092pt\mu_{2}=\operatorname{\mathbb{E}}\\{\sigma(Z)^{2}\\},\hskip
11.38092pt\mu_{\star}^{2}=\mu_{2}-\mu_{0}^{2}-\mu_{1}^{2},\hskip
11.38092pt\zeta={\mu_{1}}/{\mu_{\star}},$ (9)

for some $\mu_{0},\mu_{1},\mu_{2}\in\mathbb{R}$, where the expectations are
with respect to $Z$ $\sim$ $\mathcal{N}(0,1)$.

######  Assumption 2.

We assume that $N=N(d)$ and $n=n(d)$ such that the following limits exist in
$(0,\infty)$: $\lim_{d\to\infty}N(d)/d=\psi_{1}$ and
$\lim_{d\to\infty}n(d)/d=\psi_{2}$.

######  Assumption 3.

We assume that $y_{i}=f_{d}({\bm{x}}_{i})+\epsilon_{i}$, where
$\\{\epsilon_{i}\\}_{i\leq n}\sim_{i.i.d.}\mathbb{P}_{\epsilon}$ are
independent of $\\{{\bm{x}}_{i}\\}_{i\leq n}$, with
$\mathbb{E}(\epsilon_{1})=0$, $\mathbb{E}(\epsilon_{1}^{2})=\tau^{2}$,
$\mathbb{E}(\epsilon_{1}^{4})<\infty$, expectations with respect to
$\\{\epsilon_{i}\\}$. Furthermore,

$\displaystyle f_{d}({\bm{x}})=$
$\displaystyle~{}\beta_{d,0}+\langle{\bm{\beta}}_{d,1},{\bm{x}}\rangle+f_{d}^{\mbox{\rm
NL}}({\bm{x}})\,,$ (10)

where $\beta_{d,0}\in{\mathbb{R}}$, ${\bm{\beta}}_{d,1}\in{\mathbb{R}}^{d}$
are deterministic with $\lim_{d\to\infty}\beta_{d,0}^{2}=F_{0}^{2}$,
$\lim_{d\to\infty}\|{\bm{\beta}}_{d,1}\|_{2}^{2}=F_{1}^{2}>0$. The non-linear
$f_{d}^{\mbox{\rm NL}}$ is a centered Gaussian process indexed by
${\bm{x}}\in\mathbb{S}^{d-1}(\sqrt{d})$, with covariance

$\displaystyle\mathbb{E}_{f_{d}^{\mbox{\rm NL}}}\\{f_{d}^{\mbox{\rm
NL}}({\bm{x}}_{1})f_{d}^{\mbox{\rm
NL}}({\bm{x}}_{2})\\}=\Sigma_{d}(\langle{\bm{x}}_{1},{\bm{x}}_{2}\rangle/d),$
(11)

where $\Sigma_{d}(\cdot)$ satisfies $\mathbb{E}_{{\bm{x}}\sim{\rm
Unif}(\mathbb{S}^{d-1}(\sqrt{d}))}\\{\Sigma_{d}(x_{1}/\sqrt{d})\\}=0$,
$\mathbb{E}_{{\bm{x}}\sim{\rm
Unif}(\mathbb{S}^{d-1}(\sqrt{d}))}\\{\Sigma_{d}(x_{1}/\sqrt{d})x_{1}\\}=0$,
where $x_{1}$ is the $1$st component of ${\bm{x}}$. We define the Signal to
Noise Ratio (SNR) $\rho$ by

$\displaystyle\rho=F_{1}^{2}/(F_{\star}^{2}+\tau^{2}),\text{ where
}F_{\star}^{2}\triangleq\lim_{d\to\infty}\Sigma_{d}(1).$ (12)

Informally, $\mu_{\star}$ quantifies how non-linear the AF is (cf. Lemma 4.1),
$\psi_{1}$ quantifies the complexity of the RFR model relative of the
dimension $d$, $\psi_{2}$ quantifies the amount of data used for training
relative to $d$, $\tau^{2}$ is the variance of the observation noise, $F_{1}$
is the magnitude of the linear component of our target function $f_{d}$, which
is controlled by $\beta_{d,1}$, $F_{\star}$ is the magnitude of the non-linear
component $f_{d}^{\mbox{\rm NL}}$ in the target function, and $\rho$ is the
ratio between the magnitude of the linear component and the magnitude of all
of the sources of randomness in the noisy function $f_{d}+\epsilon$. Recall
that all of our results will be derived in the asymptotic regime when
$d\to\infty$.

To obtain closed-form expressions in our main results, we will focus our
analysis on the following three different important regimes: $R_{1}$:
_Ridgeless limit_ regime, when $\lambda\rightarrow 0^{+}$; $R_{2}$: _Highly
overparameterized limit_ , when $\psi_{1}\rightarrow\infty$; $R_{3}$: _Large
sample limit_ , when $\psi_{2}\rightarrow\infty$. In the context of the RFR
model, these regimes were introduced and discussed in [6]. For what follows we
define $\overline{\lambda}\triangleq\lambda/\mu^{2}_{\star}$. Any
“$\lim_{d\to\infty}X=Y$” should be interpreted as $X$ converging to $Y$ in
probability with respect to the training data $\mathcal{D}$, the random
features ${\bm{\Theta}}$, and the random target function $f_{d}$ as
$d\to\infty$.

###  3.1 Asymptotic mean squared test error of the RFR model

The following theorems are a specialization of a more general theorem, Theorem
12 [6], which we include in the Appendix A for completeness.

######  Theorem 1 (Theorem 3 [6]).

The asymptotic test error (6) for regime $R_{1}$ equals

$\mathcal{E}^{\infty}_{R_{1}}\equiv\lim_{\lambda\to
0^{+}}\lim_{d\to\infty}\mathcal{E}=F_{1}^{2}\mathscrsfs{B}_{\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})+(\tau^{2}+F_{\star}^{2})\mathscrsfs{V}_{\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})+F_{\star}^{2},$ (13)

where $\mathscrsfs{B}_{\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})\equiv~{}\mathscrsfs{E}_{1,\mbox{\rm
rless}}/\mathscrsfs{E}_{0,\mbox{\rm rless}}$ ,$\mathscrsfs{V}_{\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})\equiv~{}\mathscrsfs{E}_{2,\mbox{\rm
rless}}/\mathscrsfs{E}_{0,\mbox{\rm rless}}$, and the functions
$\mathscrsfs{E}_{0,\mbox{\rm rless}},\mathscrsfs{E}_{1,\mbox{\rm rless}}$ and
$\mathscrsfs{E}_{2,\mbox{\rm rless}}$ are defined using
$\psi\equiv\min\\{\psi_{1},\psi_{2}\\}$ as follows,

$\displaystyle\mathscrsfs{E}_{0,\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})\equiv~{}-\chi^{5}{\zeta}^{6}+3\chi^{4}{\zeta}^{4}+(\psi_{1}\psi_{2}-\psi_{2}-\psi_{1}+1)\chi^{3}{\zeta}^{6}-2\chi^{3}{\zeta}^{4}-3\chi^{3}{\zeta}^{2}$
$\displaystyle+(\psi_{1}+\psi_{2}-3\psi_{1}\psi_{2}+1)\chi^{2}{\zeta}^{4}+2\chi^{2}{\zeta}^{2}+\chi^{2}+3\psi_{1}\psi_{2}\chi{\zeta}^{2}-\psi_{1}\psi_{2},$
$\displaystyle\mathscrsfs{E}_{1,\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})\equiv~{}\psi_{2}\chi^{3}{\zeta}^{4}-\psi_{2}\chi^{2}{\zeta}^{2}+\psi_{1}\psi_{2}\chi{\zeta}^{2}-\psi_{1}\psi_{2},\text{
and }$ $\displaystyle\mathscrsfs{E}_{2,\mbox{\rm
rless}}({\zeta},\psi_{1},\psi_{2})\equiv~{}\chi^{5}{\zeta}^{6}-3\chi^{4}{\zeta}^{4}+(\psi_{1}-1)\chi^{3}{\zeta}^{6}+2\chi^{3}{\zeta}^{4}+3\chi^{3}{\zeta}^{2}+(-\psi_{1}-1)\chi^{2}{\zeta}^{4}-2\chi^{2}{\zeta}^{2}-\chi^{2},$
$\displaystyle\text{ where
}\chi\equiv-\left(\sqrt{(\psi{\zeta}^{2}-{\zeta}^{2}-1)^{2}+4{\zeta}^{2}\psi}+\psi{\zeta}^{2}-{\zeta}^{2}-1\right)\Big{/}\left(2{\zeta}^{2}\right).$
(14)

######  Remark 1.

As a function of $\psi_{1}$, $\mathcal{E}^{\infty}_{R_{1}}$ has a
discontinuity at $\psi_{1}=\psi_{2}$ called the _interpolation threshold_. For
$\psi_{2}$ high enough, and for $\psi_{1}<\psi_{2}$,
$\mathcal{E}^{\infty}_{R_{1}}$ decreases, reaches a minimum and then explodes
approaching $\psi_{2}$. However, past $\psi_{2}$,
$\mathcal{E}^{\infty}_{R_{1}}$ decreases again with $\psi_{1}$. This _double
descent_ behavior has been observed in many settings and is the target of many
studies, including [6] and references therein.

######  Theorem 2 (Theorem 4 [6]).

The asymptotic test error (6) for regime $R_{2}$ equals

$\mathcal{E}^{\infty}_{R_{2}}\equiv\lim_{\psi_{1}\to\infty}\lim_{d\to\infty}\mathcal{E}=F_{1}^{2}\mathscrsfs{B}_{\mbox{\rm
wide}}({\zeta},\psi_{2},\overline{\lambda})+(\tau^{2}+F_{\star}^{2})\mathscrsfs{V}_{\mbox{\rm
wide}}({\zeta},\psi_{2},\overline{\lambda})+F_{\star}^{2},$ (15)

where $\mathscrsfs{B}_{\mbox{\rm
wide}}({\zeta},\psi_{2},\overline{\lambda})\equiv{(\psi_{2}{\omega_{2}}-\psi_{2})}/{((\psi_{2}-1){\omega_{2}}^{3}+(1-3\psi_{2}){\omega_{2}}^{2}+3\psi_{2}{\omega_{2}}-\psi_{2})}$,
$\mathscrsfs{V}_{\mbox{\rm
wide}}({\zeta},\psi_{2},\overline{\lambda})\equiv{({\omega_{2}}^{3}-{\omega_{2}}^{2})}/{((\psi_{2}-1){\omega_{2}}^{3}+(1-3\psi_{2}){\omega_{2}}^{2}+3\psi_{2}{\omega_{2}}-\psi_{2})}\,$
and

$\displaystyle{\omega_{2}}\equiv-\left(\sqrt{(\psi_{2}{\zeta}^{2}-{\zeta}^{2}{-}\overline{\lambda}\psi_{2}-1)^{2}{+}4\psi_{2}{\zeta}^{2}(\overline{\lambda}\psi_{2}+1)}+\psi_{2}{\zeta}^{2}-{\zeta}^{2}-\overline{\lambda}\psi_{2}-1\right)\Big{/}\left(2(\overline{\lambda}\psi_{2}+1)\right).$
(16)

######  Theorem 3 (Theorem 5 [6]).

The asymptotic test error (6) for regime $R_{3}$ equals

$\displaystyle\mathcal{E}^{\infty}_{R_{3}}\equiv\lim_{\psi_{2}\to\infty}\lim_{d\to\infty}\mathcal{E}=F_{1}^{2}\mathscrsfs{B}_{\mbox{\rm
lsamp}}({\zeta},\psi_{1},\lambda/\mu_{\star}^{2})+F_{\star}^{2}\,,\text{ where
}$ (17)

$\mathscrsfs{B}_{\mbox{\rm
lsamp}}({\zeta},\psi_{1},\overline{\lambda})\equiv{((({\omega_{1}}^{3}-{\omega_{1}}^{2})/{\zeta}^{2})+\psi_{1}{\omega_{1}}-\psi_{1})}/{((\psi_{1}-1){\omega_{1}}^{3}+(1-3\psi_{1}){\omega_{1}}^{2}+3\psi_{1}{\omega_{1}}-\psi_{1})},$
and

$\displaystyle{\omega_{1}}\equiv-\left(\sqrt{(\psi_{1}{\zeta}^{2}-{\zeta}^{2}-\overline{\lambda}\psi_{1}-1)^{2}+4\psi_{1}{\zeta}^{2}(\overline{\lambda}\psi_{1}+1)}+\psi_{1}{\zeta}^{2}-{\zeta}^{2}-\overline{\lambda}\psi_{1}-1\right)\Big{/}\left(2(\overline{\lambda}\psi_{1}+1)\right).$
(18)

###  3.2 Asymptotic sensitivity of the RFR model

We derive a sensitivity formula for regimes $R_{1},R_{2},R_{3}$. Our theorems
are a specialization (proofs in Appendix D) of the more general Theorem 13
that we include in the Appendix A for completeness.

######  Theorem 4.

The sensitivity (7) for regime $R_{1}$ equals

$\mathcal{S}^{\infty}_{R_{1}}\equiv\lim_{\lambda\rightarrow
0^{+}}\lim_{d\to\infty}\mathcal{S}=\frac{\mu^{2}_{1}}{\mu_{\star}^{2}}\left(\frac{F^{2}_{1}\mathscrsfs{D}_{1,{\mbox{\rm
rless}}}({\zeta},\psi_{1},\psi_{2})}{(\chi{\zeta}^{2}-1)\mathscrsfs{D}_{0,{\mbox{\rm
rless}}}({\zeta},\psi_{1},\psi_{2})}+\frac{(F^{2}_{\star}+\tau^{2})\mathscrsfs{D}_{2,{\mbox{\rm
rless}}}({\zeta},\psi_{1},\psi_{2})}{\mathscrsfs{D}_{0,{\mbox{\rm
rless}}}({\zeta},\psi_{1},\psi_{2})}\right),$ (19) $\displaystyle\text{ where
}\mathscrsfs{D}_{0,\mbox{\rm rless}}({\zeta},\psi_{1},\psi_{2})$
$\displaystyle=\chi^{5}{\zeta}^{6}-3\chi^{4}{\zeta}^{4}+(\psi_{1}+\psi_{2}-\psi_{1}\psi_{2}-1)\chi^{3}{\zeta}^{6}+2\chi^{3}{\zeta}^{4}+3\chi^{3}{\zeta}^{2}$
(20)
$\displaystyle+(3\psi_{1}\psi_{2}-\psi_{2}-\psi_{1}-1)\chi^{2}{\zeta}^{4}-2\chi^{2}{\zeta}^{2}-\chi^{2}-3\psi_{1}\psi_{2}\chi{\zeta}^{2}+\psi_{1}\psi_{2}\,,$
$\displaystyle\mathscrsfs{D}_{1,\mbox{\rm rless}}({\zeta},\psi_{1},\psi_{2})$
$\displaystyle=\chi^{6}{\zeta}^{6}-2\chi^{5}{\zeta}^{4}-(\psi_{1}\psi_{2}-\psi_{1}-\psi_{2}+1)\chi^{4}{\zeta}^{6}+\chi^{4}{\zeta}^{4}$
(21)
$\displaystyle+\chi^{4}{\zeta}^{2}-2(1-\psi_{1}\psi_{2})\chi^{3}{\zeta}^{4}-(\psi_{1}+\psi_{2}+\psi_{1}\psi_{2}+1)\chi^{2}{\zeta}^{2}-\chi^{2}\,,$
$\displaystyle\mathscrsfs{D}_{2,\mbox{\rm rless}}({\zeta},\psi_{1},\psi_{2})$
$\displaystyle=-(\psi_{1}-1)\chi^{3}{\zeta}^{4}-\chi^{3}{\zeta}^{2}+(\psi_{1}+1)\chi^{2}{\zeta}^{2}+\chi^{2}\,,$
(22)

$\chi$ is defined as in (14), and $\psi\equiv\min\\{\psi_{1},\psi_{2}\\}$.

######  Theorem 5.

Let ${\omega_{2}}$ equal (16). The sensitivity (7) for regime $R_{2}$ equals

$\displaystyle\mathcal{S}^{\infty}_{R_{2}}\equiv\lim_{\psi_{1}\rightarrow\infty}\lim_{d\to\infty}\mathcal{S}=\frac{{\omega_{2}}^{2}((F^{2}_{\star}+\tau^{2})(-1+{\omega_{2}})+F^{2}_{1}(-1-\psi_{2}+{\omega_{2}}(-1+\psi_{2})))}{(-1+{\omega_{2}})(\psi_{2}-2{\omega_{2}}\psi_{2}+{\omega_{2}}^{2}(-1+\psi_{2}))}.$
(23)

######  Theorem 6.

Let ${\omega_{1}}$ equal (18). The sensitivity (7) for regime $R_{3}$ equals

$\displaystyle\mathcal{S}^{\infty}_{R_{3}}\equiv\lim_{\psi_{2}\rightarrow\infty}\lim_{d\to\infty}\mathcal{S}=F_{1}^{2}(1+({2}/({-1+{\omega_{1}}}))+({\psi_{1}}/({\psi_{1}-2{\omega_{1}}\psi_{1}+{\omega_{1}}^{2}(-1+\psi_{1})}))).$
(24)

##  4 Main results

We will find the simplest AFs that lead to the best trade-off between
approximation accuracy and sensitivity for the RFR model. Mathematically, we
will solve (8). From the theorems in Section 3 we know that $\mathcal{E}$ and
$\mathcal{S}$, and hence $O=(1-\alpha)\mathcal{E}+\alpha\mathcal{S}$, only
depend on the AF via $\mu_{0},\mu_{1},\mu_{2}$. Therefore, we will proceed in
two steps. In Section 4.1, we will fix $\mu_{0},\mu_{1},\mu_{2}$, and find
$\sigma$ with associated values $\mu_{0},\mu_{1},\mu_{2}$ that has minimal
norm, either (4) or (3). In Section 4.2, we will find values of
$\mu_{0},\mu_{1},\mu_{2}$ that minimize
$O=(1-\alpha)\mathcal{E}+\alpha\mathcal{S}$. Together, these specify optimal
AFs for the RFR model.

It is the case that properties of the RFR model other than the test error and
sensitivity also only depend on the AF via $\mu_{0},\mu_{1},\mu_{2}$. One
example is the robustness of the RFR model to disparities between the training
and test data distribution [8]. Although we do not focus on these other
properties, the results in Section (4.1) can be used to generate optimal AFs
for them as well, as long as, similar to in Section 4.2, we can obtain
$\mu_{0},\mu_{1},\mu_{2}$ that optimize these other properties.

We made the decision to, as often as possible, simplify expressions by
manipulating them to expose the signal to noise ratio
$\rho=F^{2}_{1}/(\tau^{2}+F^{2}_{\star})$, $F_{1}>0$, rather than using the
variables $F_{1},\tau$, and $F_{\star}$. The only downside is that conclusions
in the regime $\tau=F_{\star}=0$ require a bit more of effort to be extracted,
often been readable in the limit ${\rho\to\infty}$.

The complete proofs of our main results can be found in Appendix D and their
main ideas below.

###  4.1 Optimal activation functions given fixed $\mu_{0},\mu_{1}$, and
$\mu_{2}$

Since one of our goals is knowing when an optimal AF is linear we start with
the following lemma.

######  Lemma 4.1.

The AF $\sigma$ is linear (almost surely) if and only if
$\mu_{\star}^{2}\triangleq\mu_{2}-\mu^{2}_{1}-\mu^{2}_{0}=0$.

We now state results for the norms (4) and (3). The problem we will solve
under both norms is similar. Let $Z$ $\sim$ $\mathcal{N}(0,1)$. We consider
solving the following functional problem, where $i=1$ or $2$,

$\displaystyle\min_{\sigma}\|\sigma\|_{i}\;\text{ subject
to}\;\operatorname{\mathbb{E}}(\sigma(Z))=\mu_{0},\operatorname{\mathbb{E}}(Z\sigma(Z))=\mu_{1},\operatorname{\mathbb{E}}(\sigma(Z)^{2})=\mu_{2},\text{
with }Z\sim\mathcal{N}(0,1).$ (25)

If $i=2$, we seek solutions over the Guassian-weighted Lebesgue space of twice
weak-differentiable functions that have
$\operatorname{\mathbb{E}}((\sigma(Z))^{2})$ and
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$ defined and finite. If
$i=1$, we seek solutions over the Guassian-weighted Lebesgue space of weak-
differentiable functions that have
$\operatorname{\mathbb{E}}((\sigma(Z))^{2})$ and
$\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z)|)$ defined and finite. The
derivative $\sigma^{\prime}$ is to be understood in a weak sense.

Since $\sigma$ is a one-dimensional function, the requirement of existence of
weak derivative implies that there exists a function $v$ that is absolute
continuous and that agrees with $\sigma$ almost everywhere [41]. Therefore,
any specific solution we propose should be understood as an equivalent class
of functions that agree with $v$ up to a set of measure zero with respect to
the Guassian measure.

######  Theorem 7.

The minimizers of (25) for $i=2$, i.e.
$\|\sigma\|^{2}=\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$, are

$\displaystyle\sigma(x)=ax^{2}+bx+c,\text{ where
}a=\pm{\mu_{\star}}/{\sqrt{2}},b=\mu_{1},\text{ and }c=\mu_{0}-a.$ (26)

In Theorem 7, if $\mu_{\star}=0$ there is only one minimizer, a linear
function. If $\mu_{\star}>0$, there are exactly two minimizers, both quadratic
functions. Note that both minimizers satisfy the growth constraints of
Assumption 1, and hence can be used within the analysis of the RFR model. We
note that quadratic AFs have been empirically studied in the past, e.g. [30].

######  Theorem 8.

One minimizer of (25) for $i=1$, i.e.
$\|\sigma\|=\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z)|)$, is

$\displaystyle\sigma(x)=\mu_{0}+b\max\\{\min\\{x,-s\\},s\\},$ (27)

where $b=\frac{\mu_{1}}{\text{erf}(s/\sqrt{2})}$, erf is the Guass error
function, and $s\in\mathbb{R}$ is the unique solution to the equation

$\zeta^{2}\triangleq{\mu^{2}_{1}}/{\mu^{2}_{\star}}=\left(e^{{s^{2}}/{2}}\text{erf}{\hskip
2.84544pt}^{2}\left({s}/{\sqrt{2}}\right)\right)\Big{/}\left({e^{{s^{2}}/{2}}\left(1-\text{erf}\left({s}/{\sqrt{2}}\right)\right)\left(s^{2}+\text{erf}\left({s}/{\sqrt{2}}\right)\right)-\sqrt{{2}/{\pi}}s}\right).$
(28)

if $\mu_{\star}\neq 0$, and $s=+\infty$ if $\mu_{\star}=0$.

When $\|\sigma\|=\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z)|)$, we can
characterize the complete solution family to (25). These are AFs of the form
$\sigma(x)=a+b\max\\{s,\min\\{t,x\\}\\}$, where $a,b,s$, and $t$ are chosen
such that the constraints in (25) hold. It is possible to explicitly write $a$
and $b$ as a function of $\mu_{0},\mu_{1},s,t$, and express $s,t$ as the
solution of $E(s,t)={\mu^{2}_{1}}/{\mu^{2}_{\star}}$, where $E(\cdot,\cdot)$
has explicit form. In this case, for each $\mu_{0},\mu_{1},\mu_{2}$ there are
an infinite number of optimal AFs since
$E(s,t)={\mu^{2}_{1}}/{\mu^{2}_{\star}}$ has an infinite number of solutions.
ReLU’s are included in this family as $t\to\infty$. The involved lengthy
expressions do not bring any new insights, so we state and prove only Theorem
8, which is a specialization of the general theorem to $s=-t$.

###### Proofs’ main ideas: .

We give the main ideas behind the proof of Theorem 7. The proof of Theorem 8
follows similar techniques. The first-order optimality conditions imply that
$-2x\sigma^{\prime}(x)+2\sigma^{\prime\prime}(x)+\lambda_{1}+\lambda_{2}x+\lambda_{3}\sigma(x)=0$,
where the Lagrange multipliers $\lambda_{1},\lambda_{2}$, and $\lambda_{3}$
must be later chosen such that
$\operatorname{\mathbb{E}}\\{\sigma(Z)\\}=\mu_{0},\operatorname{\mathbb{E}}\\{Z\sigma(Z)\\}=\mu_{1}$,
and $\operatorname{\mathbb{E}}\\{\sigma^{2}(Z)\\}=\mu_{2}$. Using the change
of variable
$\sigma(x)=\tilde{\sigma}(x/\sqrt{2})-\lambda_{1}/\lambda_{3}-x\lambda_{2}/(\lambda_{3}-1)$
we obtain
$-2x\tilde{\sigma}^{\prime}(x)+\tilde{\sigma}^{\prime\prime}(x)+\lambda_{3}\tilde{\sigma}(x)=0$
which is the _Hermite ODE_ , which is well studied in physics, e.g. it appears
in the study of the quantum harmonic oscillator. The cases
$\lambda_{3}\in\\{0,3\\}$ require special treatment. Using a finite
energy/norm condition we can prove that $\lambda_{3}$ is quantized. In
particular $\lambda_{3}=4k,k=1,2,...$, which implies that
$\sigma(x)=-\lambda_{1}/\lambda_{3}-\lambda_{2}x/(\lambda_{3}-2)+cH_{2k}(x/\sqrt{2})$,
where $H_{i}$ is the $i$th _Hermite polynomial_ and $c$ a constant. The
energy/norm is minimal when $k=1$, which implies a quadratic AF. ∎

###  4.2 Activation function parameters for the optimal trade-off between
sensitivity and MSE

We will find AF parameters that minimize a linear combination of sensitivity
and test error. We are interested in an asymptotic analytical treatment in the
three regimes mentioned in Section 3. To be specific, we will compute

$\displaystyle\mathcal{U}_{R_{i}}(\psi_{1},\psi_{2},\tau,\alpha,F_{1},F_{\star},\lambda)\equiv\operatorname*{arg\,min}_{\mu_{0},\mu_{1},\mu_{2}}\;\;\;(1-\alpha)\mathcal{E}^{\infty}_{R_{i}}+\alpha\mathcal{S}^{\infty}_{R_{i}},\text{
where }i=1,2,\text{ or }3.$ (29)

We are not aware of previous work explicitly studying the trade-off between
$\mathcal{E}$ and $\mathcal{S}$ for the RFR model. For the RFR model, the work
of [6] studies only the test error and [14] studies a definition of
sensitivity related but different from ours. Other papers have studied trade-
offs between robustness and error measures related but different than ours and
for other models, e.g. [42, 43].

When $\alpha=1$, problem (29) reduces to minimizing the sensitivity. This has
a trivial solution: the AF, and hence $f$, must be a constant, and
$\mathcal{S}=0$. Therefore, below we focus on the case when $\alpha\in[0,1)$.

Special notation: In Theorem 9 we use the following special notation. Given
two potential choices for AF parameters, say $x$ and $y$, we define $x\sqcup
y$ to mean that $x$ exists and that $y$ might exist or not, and that $x\sqcup
y=y$ if $y$ exists and it leads to a smaller value of
$(1-\alpha)\mathcal{E}+\alpha\mathcal{S}$ than using $x$, and otherwise
$x\sqcup y=x$. Note that $x\sqcup y$ and $y\sqcup x$ make different statements
about the existence of $x$ and $y$. This notation is important to interpret
the results of Table 1 in Theorem 9.

######  Theorem 9.

Let $\alpha\in[0,1)$, $\psi\equiv\min\\{\psi_{1},\psi_{2}\\}$ and
$\overline{\psi}\equiv\max\\{\psi_{1},\psi_{2}\\}$. We have that

$\displaystyle\mathcal{U}_{R_{1}}=\left\\{(\mu_{0},\mu_{1},\mu_{2}):x\mu_{1}^{2}(-1+x+\psi)=\mu^{2}_{\star}(\psi+x)\right\\},\text{
where $x$ is as in Table \ref{theorem_1_table}.}$ (30)

| $\beta_{1}\leq\overline{\psi}$ | $\beta_{2}<\overline{\psi}<\beta_{1}$ | $\beta_{3}<\overline{\psi}\leq\beta_{2}$ | $\overline{\psi}\leq\beta_{3}$
---|---|---|---|---
$(\alpha<\alpha_{L})\land E_{1}$ | $x_{R}$ | $x_{R}\sqcup x_{1}$ | $x_{R}$ | $x_{R}$
$(\alpha<\alpha_{L})\land E_{2}\land(\alpha>\alpha_{C})$ | $x_{1}$ | $x_{1}\sqcup x_{3}$ | $x_{1}$ | $--$
$(\alpha<\alpha_{L})\land E_{2}\land(\alpha<\alpha_{C})$ | $x_{1}$ | $x_{1}\sqcup x_{3}$ | $x_{1}$ | $--$
$(\alpha>\alpha_{L})\land E_{1}\land(\alpha>\alpha_{C})$ | $--$ | $x_{L}$ | $x_{L}$ | $x_{L}$
$(\alpha>\alpha_{L})\land E_{1}\land(\alpha<\alpha_{C})$ | $--$ | $x_{R}$ | $x_{R}$ | $x_{R}$
$(\alpha>\alpha_{L})\land E_{2}$ | $x_{L}$ | $x_{L}\sqcup x_{2}$ | $x_{L}\sqcup x_{2}$ | $x_{L}$
Table 1: The optimal AFs (30) depends on $x$ according to this table. Cells
with “- -” never happen. The values of $x_{1},x_{2},x_{3}$,
$\beta_{1},\beta_{2},\beta_{3},\alpha_{L},\alpha_{C},\alpha_{R}$,
$x_{L},x_{R}$, and the events $E_{1}$ and $E_{2}$ are specified below.

In Table 1, $x_{1},x_{2}$, and $x_{3}$ are the smallest, second smallest and
third smallest roots of a 4th degree polynomial $p(x)$, specified in Appendix
B, in the range $(x_{L},x_{R})\triangleq(-\psi,\min\\{0,1-\psi\\})$, if these
exists. The variables
$\beta_{1},\beta_{2},\beta_{3},\alpha_{L},\alpha_{C},\alpha_{R}$, and the
conditions $E_{1}$ and $E_{2}$ are as follows.

If $\psi_{1}<\psi_{2}$ then $E_{1}=(\alpha<\alpha_{R})$,
$E_{2}=(\alpha>\alpha_{R})$,

$\displaystyle\beta_{1}=\min\\{\psi_{1}-4,-3\psi_{1}\\}-8\sqrt{|1-\psi_{1}|}\max\\{1/\psi_{1},\psi^{3/2}_{1}\\}+8\max\\{1/\psi_{1},\psi^{2}_{1}\\},$
(31)

$\displaystyle\beta_{2}={\psi_{1}(\psi_{1}+2)}/(\psi_{1}+1),$ (32)

$\displaystyle\quad\beta_{3}=\psi_{1}+|1-\psi_{1}|\min\\{\psi_{1},1/\psi_{1}\\},$
(33)

$\displaystyle\alpha_{L}={\psi_{2}}/({\psi_{2}+1+\rho^{-1})},$ (34)

$\displaystyle\quad\alpha_{C}=\psi_{2}/({2\psi_{2}+1-\psi_{1}+\max\\{0,1-\psi_{1}\\}+\rho^{-1}}),$
(35)

$\displaystyle\alpha_{R}={\psi_{2}}/({3\psi_{2}+1-2\min\\{1+\psi_{1},2\psi_{1}\\}+\rho^{-1}}).$
(36)

If $\psi_{1}>\psi_{2}$ and $\psi_{2}\neq 1$ then
$E_{1}=(\psi_{1}{<}B)\lor((\alpha{<}\alpha_{R})\land(B{<}\psi_{1}{<}A))$,
$E_{2}=(\psi_{1}{>}A)\lor((\alpha{>}\alpha_{R})\land(B{<}\psi_{1}{<}A))$,

$\displaystyle\beta_{1}:r(\beta_{1})=0,\text{ where }r\text{ is a 4th degree
polynomial described in Appendix
\ref{app:details_of_theorem_special_case_1}},$ (37)
$\displaystyle\beta_{2}={\psi_{2}}+({\alpha{\psi_{2}}}/({{\psi_{2}}+1+\rho^{-1}})),$
(38)
$\displaystyle\beta_{3}={\psi_{2}}+(\min\\{\psi_{2},1/\psi_{2}\\}{\alpha|{\psi_{2}}-1|^{3}}/{(({\psi_{2}}-1)^{2}+\rho^{-1}({\psi_{2}}+3))}),$
(39)
$\displaystyle\alpha_{L}=({2\psi_{1}-\psi_{2}})/({2\psi_{1}-\psi_{2}+1+\rho^{-1}}),$
(40)
$\displaystyle\alpha_{C}=({\rho\psi_{1}-{\left(\psi_{1}-\psi_{2}\right)}/{\left|1-\psi_{2}\right|}})/({\rho\left(\max\left\\{0,1-\psi_{2}\right\\}+2\psi_{1}-\psi_{2}\right)+1}),$
(41)
$\displaystyle\alpha_{R}=\frac{2\left(\psi_{1}{-}\psi_{2}\right)\max\left\\{1,\psi_{2}\right\\}{-}\rho\left(\psi_{2}{-}1\right)^{2}\psi_{2}}{\left(\psi_{2}{-}1\right)^{2}\left(\rho\left(2\min\left\\{1,\psi_{2}\right\\}{-}2\psi_{1}{+}\psi_{2}\right){-}\rho{-}1\right)},$
(42) $\displaystyle
A=\psi_{2}+(\rho\min\\{1,\psi_{2}\\}(\psi_{2}-1)^{2}/2),\text{ and }$ (43)
$\displaystyle
B=\frac{\rho\left(\psi_{2}{-}1\right){}^{2}\left(2\psi_{2}{+}1{+}2\min\\{\psi_{2}{-}1,0\\}\right){+}2\psi_{2}{-}1{+}2\psi_{2}\max\\{\psi_{2},1\\}{-}\psi^{2}_{2}}{2\rho(\psi_{2}{-}1)^{2}{+}2\max\\{1,\psi_{2}\\}}.$
(44)

If $\psi_{1}>\psi_{2}$ and $\psi_{2}=1$ then $E_{1}=\text{False}$,
$E_{2}=\text{True}$ and eqs. (37)-(44) hold when reading Table 1, except (42),
which is not defined, and (41) which becomes $\alpha_{C}=-\infty$ implying
$(\alpha>\alpha_{C})$ is True.

######  Remark 2.

Excluding trivial scenarios where the optimal AF is a constant, and hence $f$
is also constant, it follows directly from (30) that the optimal AF is linear
if and only if $x=x_{R}$. With this information and Table 1, we have all the
information needed to find exactly when the optimal AF is, or is not, linear.
For regime $R_{1}$, changing $\alpha$ alone can change the optimal AF from
linear to non-linear and vice-versa (see e.g. 3rd column of Table 1), which
justifies the observation 3 in Section 1.

######  Remark 3.

For the cases considered in Table 1, $x$ is unique. When
$\alpha\in\\{\alpha_{R},\alpha_{L},\alpha_{C}\\}$, or when
$(\psi_{1}>\psi_{2})\land(\psi_{1}\in\\{A,B\\})$, we can lose the uniqueness
of $x$. In these circumstances, we can still explicitly characterize the set
of optimal $x$ and the associated set of optimal AFs parameters, but for
simplicity we omit these cases from Theorem 9.

######  Remark 4.

Theorem 9’s proof gives relationships among Table 1’s constants that imply
that (1) no two rows/columns simultaneously hold and (2) in some cases some
cells might not hold. See App. B.

We do not consider $\psi_{1}=\psi_{2}$ in Theorem 9 because it implies
$(1-\alpha)\mathcal{E}^{\infty}_{R_{1}}+\alpha\mathcal{S}^{\infty}_{R_{1}}$ is
not defined. Note that $\psi_{1}=\psi_{2}$ has been called the _interpolation
threshold_ in the context of studying the generalization properties of the RFR
model under vanishing regularization [6]. See Remark 1.

The set (30) of optimal AFs parameters is invariant under scaling and shifting
of $\sigma$. In particular, let $\mu(\sigma)$ be $\mu_{0},\mu_{1},\mu_{2}$
associated with $\sigma$. If $\mu(\sigma)\in\mathcal{U}_{R_{1}}$ then
$\mu(a\sigma+b)\in\mathcal{U}_{R_{1}}$.

Also, when $x\in\\{x_{L},x_{R}\\}$ we can compute the optimal value of the
objective explicitly. For example, if $\psi_{1}<\psi_{2}$ and $x=x_{L}$ the
optimal value of the objective is
$\frac{(1-\alpha)(\psi_{2}(F_{1}^{2}+F_{\star}^{2})+\psi_{1}\tau^{2})}{\psi_{2}-\psi_{1}}$.
If $\psi_{1}>\psi_{2}$ and $x=x_{R}$ the optimal value of the objective is
$\frac{\alpha
F_{1}^{2}(\psi_{1}-\psi_{2})+F_{\star}^{2}((\alpha-1)\psi_{2}-\alpha)+\alpha(\psi_{1}-1)\tau^{2}-\psi_{1}\tau^{2}}{\psi_{1}-\psi_{2}}$
if $\psi_{1}<1$ and
$\frac{F_{1}^{2}(\alpha(2\psi_{1}-1)(\psi_{1}-\psi_{2})+(\psi_{1}-1)\psi_{2})+F_{\star}^{2}((\alpha-1)\psi_{2}-\alpha\psi_{1})-\psi_{1}\tau^{2}}{\psi_{1}-\psi_{2}}$
if $\psi_{1}\geq 1$. This follows from direct substitution.

######  Theorem 10.

Let $\alpha\in[0,1)$. We have that,

$\displaystyle\mathcal{U}_{R_{2}}=\left\\{(\mu_{0},\mu_{1},\mu_{2}):\mu_{1}^{2}(-1+2\psi_{2}-x)(-1+x)=-2(\mu^{2}_{\star}+\lambda\psi_{2})(1+x)\right\\},$
(45)

where $x$ is the unique solution to $p(x)=0$ in the range
$x\in(-1,\min\\{1,-1+2\psi_{2}\\})$, where $p(x)\triangleq
p_{0}+p_{1}x+p_{2}x^{2}+p_{3}x^{3}+p_{4}x^{4}$ with coefficients,

$\displaystyle
p_{0}=8\psi_{2}\rho^{-1}{+}(\alpha{+}4\psi_{2}(-1{+}2\psi_{2})(-1{+}2\alpha)),$
(46)

$\displaystyle\quad
p_{1}=8\psi_{2}\rho^{-1}{+}4((1{-}4\psi_{2})\alpha{+}2\psi^{2}_{2}),$ (47)

$\displaystyle p_{2}=-2(-3\alpha+\psi_{2}(2+4\alpha)),$ (48)

$\displaystyle\quad p_{3}=4\alpha,$ (49)

$\displaystyle\text{ and }\quad p_{4}=\alpha.$ (50)

######  Remark 5.

The only way to get $\mu_{\star}=0$, and hence a linear optimal AF is if $x$
simultaneously satisfies
$\mu_{1}^{2}(-1+2\psi_{2}-x)(-1+x)=-2\lambda\psi_{2}(1+x)$ and $p(x)=0$. Since
the first equation does not depend on $\alpha$, but the zeros of $p(x)=0$
change continuously with $\alpha$, only very special choices of parameters
lead to linear AFs, and in general the overparameterized regime $R_{2}$ does
not have optimal linear AFs.

######  Theorem 11.

Let $\alpha\in[0,1)$. We have that

$\displaystyle\mathcal{U}_{R_{3}}=\left\\{\begin{array}[]{ll}\left\\{(\mu_{0},\mu_{1},\mu_{2}):\mu_{\star}=0\land\mu_{1}=\infty\right\\},\quad\quad\quad\quad\quad\quad\text{
if }\alpha=0\vee(\psi_{1}=1\land 0<\alpha\leq\frac{1}{4})\\\
\left\\{(\mu_{0},\mu_{1},\mu_{2}):\mu_{\star}=0\land\mu^{2}_{1}=\frac{-4\alpha^{2}\lambda+3\alpha\lambda+\sqrt{\alpha}\lambda}{16\alpha^{2}-8\alpha+1}\right\\},\quad\quad\text{
if }\psi_{1}=1\land\alpha>\frac{1}{4}\\\
\left\\{(\mu_{0},\mu_{1},\mu_{2}):\mu_{\star}=0\land\mu_{1}^{2}(-1+2\psi_{1}-x)(-1+x)+2\lambda\psi_{1}(1+x)=0\right\\},\text{
if }\psi_{1}\neq 1\end{array}\right.$ (54)

where $x$ is the unique solution to $p(x)=0$ in the range
$x\in(-1,\min\\{1,-1+2\psi_{2}\\})$, where $p(x)$ is define like in Theorem 10
but with $\rho\to\infty$ and with $\psi_{2}$ replaced by $\psi_{1}$.

######  Remark 6.

The optimal AF is always linear and independent of the noise variables
$F_{\star}$ and $\tau$.

######  Remark 7.

When $\psi_{1}=1,\alpha\leq\frac{1}{4}$ there is no optimal AF inside our AF
search space since no AF can satisfy $\mu_{1}=\infty$. Rather, there exists a
sequence of valid AFs with decreasing $O$ whose $\mu_{1}\to\infty$.

######  Remark 8.

We can compute the optimal objective in closed-form in some scenarios. When
$\alpha=0$ the optimal objective is $F_{\star}^{2}$. When $\psi_{1}=1\land
0<\alpha\leq\frac{1}{4}$, the optimal objective approaches $\alpha
F^{2}_{1}+(1-\alpha)F_{\star}^{2}$ as $\mu_{1}\to\infty$. When
$\psi_{1}=1\land\alpha>\frac{1}{4}$, the optimal objective is
$F^{2}_{1}(4\sqrt{\alpha}-1-3\alpha)+F_{\star}^{2}(1-\alpha)$.

###### Proofs’ main ideas: .

We give the main ideas behind the proof of Theorem 10. The proof of Theorems 9
and 11 follows similar techniques but require more care. The objective $O$
only depends on AF parameters via
$\omega_{2}=\omega_{2}(\psi_{2},\mu^{2}_{1},\mu^{2}_{\star})$. We use the
Möbius transformation $x=(1+\omega_{2})/(\omega_{2}-1)$ such that the infinite
range $\omega_{2}\in[-\infty,0]$ gets mapped to the finite range $x\in[-1,1]$.
We then focus the rest of the proof on minimizing $O=O(x)$ over the range of
$x$. First we show that given that $\mu^{2}_{1},\mu^{2}_{\star}\geq
0,\psi_{1}>0$, the range of $x$ can be reduced to
$x\in[x_{L},x_{R}]\triangleq[-1,\min\\{1,-1+2\psi_{2}\\}]$. Then we compute
${\rm d}O/{\rm d}x$ and ${\rm d}^{2}O/{\rm d}x^{2}$, which turns out to be
rational functions of $x$. We then show that if
$x\in[-1,\min\\{1,-1+2\psi_{2}\\}]$ then ${\rm d}^{2}O/{\rm d}x^{2}>0$, so $O$
is strictly convex. We also show that ${\rm d}O/{\rm d}x<0$ at $x_{L}$ and
${\rm d}O/{\rm d}x>0$ at $x_{R}$, thus $x_{R}$ and $x_{L}$ cannot be
minimizers. Finally, we show that the zeros of the numerator $p(x)$ of the
rational function ${\rm d}O/{\rm d}x$ differ from the denominator’s zeros. So
the optimal $x$ is the unique solution to $p(x)$ in $[x_{L},x_{R}]$. ∎

###  4.3 Important observations

Together, Sections 4.1 and 4.2 explicitly and fully characterize the solutions
of (8) in the ridgeless, overparametrized, and large sample regimes. A few
important observations follow from our theory. In Appendix C we discuss more
on this topic and include details on the observations below.

Observation 1: In regime $R_{1}$, and if $\alpha=0,\psi_{1}<\psi_{2}$, the
optimal AF is linear. This follows from Theorem 9 and Remark 2. Indeed,
expressions simplify and we get that
$p(x)=\rho{\psi_{2}}({\psi_{1}}-(x+{\psi_{1}})^{2})^{2}$ if $\psi_{1}\neq 1$
or $p(x)=-(2+x)^{2}\rho\psi_{2}$ if $\psi_{1}=1$, which implies that $x_{1}$
does not exist (since it would be outside of $(-x_{L},x_{R})$). Hence, the
first row of Table 1 always gives $x=x_{R}$ and the optimal AF is linear.
Also, when $\alpha=0,\psi_{1}<\psi_{2}$, we can explicitly compute the optimal
objective (see paragraph before Theorem 10). If furthermore
$\tau=F_{\star}=0$, we can show that also when $\psi_{1}>\psi_{2}$, $x_{1}$
does not exist and $x=x_{R}$, therefore the optimal objective and optimal AF
when $\psi_{1}>\psi_{2}$ have the same formula as when $\psi_{1}<\psi_{2}$.
Hence, if $\alpha=\tau=F_{\star}=0,\psi_{1}<\psi_{2}$, from the formula one
can conclude that choosing an optimal linear AF destroys the double descent
curve if $\psi_{2}>1$111If $\psi_{2}<1$ the optimal AF is still linear but the
explosion at the interpolation threshold $\psi_{1}=\psi_{2}$ remains., the
test error becoming exactly zero for $\psi_{1}\geq 1$. This contrasts with
choosing a non-linear, sub-optimal, AF which will exhibit a double descent
curve. This justifies observation 1 (low complexity $\psi_{1}<\psi_{2}$) and
observation 2 in Section 1. Figure 1-(A,B) illustrates this and details the
high-complexity ($\psi_{1}>\psi_{2}$) observation.

(A)
$F_{\star}=0$$\tau=0$$\alpha=0$$F_{1}=1$$\psi_{2}=3$$O_{\sigma_{\text{ReLU}}}$$O_{\sigma^{\star}}$$\psi_{1}/\psi_{2}$(B)$F_{\star}=0$$\tau=1$$\alpha=0$$F_{1}=1$$\psi_{2}=3$

$O_{\sigma_{\text{ReLu}}}$

$O_{\sigma_{\text{Linear}}}$$O_{\sigma^{\star}}$$\psi_{1}/\psi_{2}$(C)$F_{1}=1$$O_{\sigma_{\text{ReLU}}}$$\psi_{2}=0$$\tau^{2}=10$$F_{\star}=0$$\alpha=0$$O_{\sigma_{\text{ReLU}}}(\lambda^{\star})=O_{\sigma^{\star}}$$\tau^{2}=5$$O_{\sigma_{\text{ReLU}}}(\lambda^{\star})$>
$O_{\sigma^{\star}}$$\log\lambda$(D)$F_{\star}=0$$O_{\sigma_{\text{ReLU}}}$$\alpha=0$$F_{1}=10$$\psi_{2}=10$$\tau^{2}=5$$O_{\sigma_{\text{ReLU}}}(\lambda^{\star})$$O_{\sigma^{\star}}$$\log\lambda$

Figure 1:  (A) Consider the regime $R_{1}$. In a noiseless setting, if
$\psi_{2}>1$, the evolution of $O$ versus $\psi_{1}$, when an optimal linear
AF $\sigma^{\star}$ is used, can achieve $0$ test error for $\psi_{1}\geq 1$.
However, if a non-linear $\sigma_{\text{ReLU}}$ is used, we observe the
typical double descent curve. (B) Consider the regime $R_{1}$. If there is
observation noise $\tau>0$, the evolution of $O$ versus $\psi_{1}$ with a
linear AF $\sigma_{\text{linear}}$ is only optimal for $\psi_{1}<\psi_{2}$.
For $\psi_{1}>\psi_{2}$, $O$ is optimal for a linear AF until $\psi_{1}<C$
($C=5$ for the parameters here). For $\psi_{1}>C$ a non-linear AF
$\sigma_{\star}$, here close to but different from a ReLU, achieves minimal
$O$. (C) Consider the regime $R_{2}$. When a ReLU is used (green curves), the
evolution of $O$ versus $\lambda$ for both low and high Signal to Noise Ratio
(SNR) $\rho$ is only optimal for a special choice of $\lambda$, achieving the
minimum $O_{\sigma_{\text{ReLU}}}(\lambda^{\star})$. However, also for the
same low and high SNR settings, when an optimal (non-linear) AF is used
(orange curves), we obtain the same, or slightly better, $O_{\sigma^{\star}}$
regardless of any careful choice for $\lambda$. For low SNR ($\tau^{2}$ = 10)
we have $O_{\sigma^{\star}}=O_{\sigma_{\text{ReLU}}}(\lambda^{\star})=0.512$
and for high SNR ($\tau^{2}=5$) we get
$O_{\sigma_{\text{ReLU}}}(\lambda^{\star})=0.0220>O_{\sigma^{\star}}=0.0217$.
(D) In a situation just like in (C) but with even higher SNR, the difference
between the minimum $O$ that can be achieved with a particular choice of
$\lambda$ (blue line ordinate value
$O_{\sigma_{\text{ReLU}}}(\lambda^{\star})$) and the value of $O$ with any
choice of $\lambda$ but with an optimal (non-linear) AF (orange line ordinate
value $O_{\sigma^{\star}}$) becomes clearly visible. We include inside of each
plot the parameters used.

Observation 2: In regime $R_{2}$, looking at Theorem 2 and Theorem 5, one sees
that both $\mathcal{E}$ and $\mathcal{S}$, and hence the objective $O$ (cf.
(5)), only depend on the optimal AF parameters via $\omega_{2}$. In
particular, we can solve (29) by searching for the $\omega_{2}$ that achieves
the smallest objective. Given the definition of
$\omega_{2}=\omega_{2}(\psi_{2},\zeta^{2},\mu_{\star},\lambda)$ in (16),
fixing $\lambda$ and changing $\zeta$ or $\mu_{\star}$ always allows one to
span a larger range of values for $\omega_{2}$ than fixing the AF’s parameters
$\zeta,\mu_{\star}$ and changing $\lambda$. In particular, a tedious
calculation shows that in the first case the achievable range for $\omega_{2}$
is $[\frac{\psi_{2}}{\min\\{0^{-},-1+\psi_{2}\\}},0]$ which contains the range
in the second case which is
$\Big{[}\frac{1}{2}\Big{(}\zeta^{2}\left(-\psi_{2}\right)-\sqrt{\zeta^{2}\left(\psi_{2}\left(\zeta^{2}\left(\psi_{2}-2\right)+2\right)+\zeta^{2}+2\right)+1}+\zeta^{2}+1\Big{)},0\Big{]}$.
This implies that while for a fixed AF one needs to tune $\lambda$ during
learning for best performance, if an optimal AF is used, regardless of
$\lambda$, we always achieve either equal or better performance. This
justifies the observation 4 made in Section 1. This is illustrated in Figure
1-(C,D).

##  5 Conclusion and future work

We solved our task of finding optimal Activation Functions (AFs) for the
Random Features Regression (RFR) model and characterized when these are
linear, or non-linear. Our theory yielded interesting connections between the
best AF to use and the regime in which the RFR model operates: e.g. in some
regimes using a linear AF is optimal; in other regimes we can avoid
hyperparameter tuning by choosing an optimal non-linear AF.

We reduced the gap between the practice and theory of AFs’ design, but parts
remain to be closed. A limitation of our work is that to obtain explicit
formulas we restricted ourselves to three important regimes only. Although a
general and complete explicit treatment is unlikely, we have a plan to extract
in future work a few properties of the optimal AF in a general setting, such
as determining when the optimal AF will, or will not, be linear. Another
limitation is that we could only obtain explicit equations for optimal AFs
under two functional norms in the optimization problem from which we extract
them. We will explore other norms in the future. One research direction for
which we have concrete plans is focusing on the two norms used and adding
higher order moment restrictions to the AF. Some of these higher order
constraints appear in the theoretical analysis of neural models [40].

In future work we plan to extend our theory to extract AFs for the RFR model
that optimize a combination of test error and robustness to test/train
distribution shifts and adversarial attacks. The starting point will be [8]
and [7] discussed in Section 2, the results of [7] needing to be generalized
from a ReLU to general AFs before we can optimize the AFs’ parameters.

## References

  * [1] Mikhail Belkin, Daniel Hsu, and Ji Xu.  Two models of double descent for weak features.  SIAM Journal on Mathematics of Data Science, 2(4):1167–1180, 2020\.
  * [2] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.  Surprises in high-dimensional ridgeless least squares interpolation.  The Annals of Statistics, 50(2):949–986, 2022.
  * [3] Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov.  Does data interpolation contradict statistical optimality?  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1611–1619. PMLR, 2019.
  * [4] Tengyuan Liang and Alexander Rakhlin.  Just interpolate: Kernel “ridgeless” regression can generalize.  The Annals of Statistics, 48(3):1329–1347, 2020.
  * [5] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  Advances in neural information processing systems, 31, 2018.
  * [6] Song Mei and Andrea Montanari.  The generalization error of random features regression: Precise asymptotics and the double descent curve.  Communications on Pure and Applied Mathematics, 75(4):667–766, 2022\.
  * [7] Hamed Hassani and Adel Javanmard.  The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression.  arXiv preprint arXiv:2201.05149, 2022.
  * [8] Nilesh Tripuraneni, Ben Adlam, and Jeffrey Pennington.  Overparameterization improves robustness to covariate shift in high dimensions.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [9] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions, 2017.
  * [10] Michael Unser.  A representer theorem for deep neural networks, 2019.
  * [11] Andrinandrasana David Rasamoelina, Fouzia Adjailia, and Peter Sinčák.  A review of activation function for artificial neural network.  In 2020 IEEE 18th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pages 281–286. IEEE, 2020.
  * [12] Mirco Milletarí, Thiparat Chotibut, and Paolo E. Trevisanutto.  Mean field theory of activation functions in deep neural networks, 2019\.
  * [13] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein.  Sensitivity and generalization in neural networks: an empirical study.  arXiv preprint arXiv:1802.08760, 2018.
  * [14] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al.  Underspecification presents challenges for credibility in modern machine learning.  arXiv preprint arXiv:2011.03395, 2020.
  * [15] Riccardo Poli.  Parallel distributed genetic programming.  University of Birmingham, Cognitive Science Research Centre, 1996.
  * [16] Daniel Weingaertner, Victor K Tatai, Ricardo R Gudwin, and Fernando J Von Zuben.  Hierarchical evolution of heterogeneous neural networks.  In Proceedings of the 2002 Congress on Evolutionary Computation. CEC’02 (Cat. No. 02TH8600), volume 2, pages 1775–1780. IEEE, 2002.
  * [17] Maryam Mahsal Khan, Arbab Masood Ahmad, Gul Muhammad Khan, and Julian F Miller.  Fast learning neural networks using cartesian genetic programming.  Neurocomputing, 121:274–289, 2013.
  * [18] Mohit Goyal, Rajan Goyal, and Brejesh Lall.  Improved polynomial neural networks with normalised activations.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020.
  * [19] Mohammadamin Tavakoli, Forest Agostinelli, and Pierre Baldi.  Splash: Learnable activation functions for improving accuracy and adversarial robustness.  Neural Networks, 140:1–12, 2021.
  * [20] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.  Rectifier nonlinearities improve neural network acoustic models.  In Proc. icml, volume 30, page 3, 2013.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015.
  * [22] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus), 2015.
  * [23] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout networks.  arXiv preprint arXiv:1302.4389, 2013.
  * [24] Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi.  Learning activation functions to improve deep neural networks, 2014.
  * [25] Yucong Zhou, Zezhou Zhu, and Zhao Zhong.  Learning specialized activation functions with the piecewise linear unit, 2021.
  * [26] Chaity Banerjee, Tathagata Mukherjee, and Eduardo Pasiliao Jr.  An empirical study on generalizations of the relu activation function.  In Proceedings of the 2019 ACM Southeast Conference, pages 164–167, 2019.
  * [27] Sébastien Bubeck, Yuanzhi Li, and Dheeraj Nagaraj.  A law of robustness for two-layers neural networks, 2020.
  * [28] Andras Rozsa and Terrance E. Boult.  Improved adversarial robustness by reducing open space risk via tent activations.  2019\.
  * [29] Yuan Zhou, Dandan Li, Shuwei Huo, and Sun-Yuan Kung.  Soft-root-sign activation function.  arXiv preprint arXiv:2003.00547, 2020.
  * [30] Adedamola Wuraola and Nitish Patel.  Sqnl: A new computationally efficient activation function.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2018.
  * [31] Andrei Nicolae.  Plu: The piecewise linear unit activation function.  arXiv preprint arXiv:1809.09534, 2018.
  * [32] Ali Rahimi and Benjamin Recht.  Random features for large-scale kernel machines.  In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
  * [33] Christopher Williams.  Computing with infinite networks.  In M. C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996.
  * [34] Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison.  Deep convolutional networks as shallow gaussian processes.  In International Conference on Learning Representations, 2019.
  * [35] Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein.  Bayesian deep convolutional networks with many channels are gaussian processes.  In International Conference on Learning Representations, 2019.
  * [36] Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani.  Gaussian process behaviour in wide deep neural networks.  In International Conference on Learning Representations, 2018.
  * [37] Tamir Hazan and T. Jaakkola.  Steps toward deep kernel methods from infinite neural networks.  ArXiv, abs/1508.05133, 2015.
  * [38] Amit Daniely, Roy Frostig, and Yoram Singer.  Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [39] Amit Daniely.  Sgd learns the conjugate kernel class of the network.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [40] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari.  Linearized two-layers neural networks in high dimension.  The Annals of Statistics, 49(2):1029–1054, 2021.
  * [41] Walter Rudin et al.  Principles of mathematical analysis, volume 3.  McGraw-hill New York, 1976.
  * [42] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry.  Robustness may be at odds with accuracy.  arXiv preprint arXiv:1805.12152, 2018.
  * [43] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In International conference on machine learning, pages 7472–7482. PMLR, 2019.
  * [44] Izrail Moiseevitch Gelfand, Richard A Silverman, et al.  Calculus of variations.  Courier Corporation, 2000.

##  Appendix A General theorems for the asymptotic mean squared test error and
sensitivity of the RFR model

This appendix is referenced at the start of Section 3.1 and at the start of
Section 3.2.

######  Theorem 12 (Theorem 2 [6]).

If assumptions 1, 2, 3 hold, and for any value of the regularization parameter
$\lambda>0$, the asymptotic test error (6) for the RFR satisfies

$\displaystyle\mathcal{E}\xrightarrow{p}\mathcal{E}^{\infty}\equiv
F_{1}^{2}\mathscrsfs{B}({\zeta},\psi_{1},\psi_{2},\lambda/\mu_{\star}^{2})+(\tau^{2}+F_{\star}^{2})\mathscrsfs{V}({\zeta},\psi_{1},\psi_{2},\lambda/\mu_{\star}^{2})+F_{\star}^{2},$
(55)

where $\xrightarrow{p}$ denotes convergence in probability when $d\to\infty$
with respect to the training data $\mathcal{D}$, the random features
${\bm{\Theta}}$, and the random target function $f_{d}$, and where
$\mathscrsfs{B}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})\equiv~{}{\mathscrsfs{E}_{1}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}/{\mathscrsfs{E}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}$,
$\mathscrsfs{V}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})\equiv~{}{\mathscrsfs{E}_{2}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}/{\mathscrsfs{E}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}$,
and the functions $\mathscrsfs{E}_{0}$, $\mathscrsfs{E}_{1}$,
$\mathscrsfs{E}_{2}$ are defined as follows,

$\displaystyle\mathscrsfs{E}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})\equiv$
$\displaystyle~{}-\chi^{5}{\zeta}^{6}+3\chi^{4}{\zeta}^{4}+(\psi_{1}\psi_{2}-\psi_{2}-\psi_{1}+1)\chi^{3}{\zeta}^{6}-2\chi^{3}{\zeta}^{4}-3\chi^{3}{\zeta}^{2}$
$\displaystyle+(\psi_{1}+\psi_{2}-3\psi_{1}\psi_{2}+1)\chi^{2}{\zeta}^{4}+2\chi^{2}{\zeta}^{2}+\chi^{2}+3\psi_{1}\psi_{2}\chi{\zeta}^{2}-\psi_{1}\psi_{2}\,,$
$\displaystyle\mathscrsfs{E}_{1}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})\equiv$
$\displaystyle~{}\psi_{2}\chi^{3}{\zeta}^{4}-\psi_{2}\chi^{2}{\zeta}^{2}+\psi_{1}\psi_{2}\chi{\zeta}^{2}-\psi_{1}\psi_{2}\,,$
$\displaystyle\mathscrsfs{E}_{2}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})\equiv$
$\displaystyle~{}\chi^{5}{\zeta}^{6}-3\chi^{4}{\zeta}^{4}+(\psi_{1}-1)\chi^{3}{\zeta}^{6}+2\chi^{3}{\zeta}^{4}+3\chi^{3}{\zeta}^{2}+(-\psi_{1}-1)\chi^{2}{\zeta}^{4}-2\chi^{2}{\zeta}^{2}-\chi^{2}\,,$

where
$\chi(\psi_{1},\psi_{2},\overline{\lambda})\equiv\nu_{1}({\mathrm{i}}(\psi_{1}\psi_{2}\overline{\lambda})^{1/2})\cdot\nu_{2}({\mathrm{i}}(\psi_{1}\psi_{2}\overline{\lambda})^{1/2})$,
and $\nu_{1}$ and $\nu_{2}$ are two functions specified in Def. 1 in [6].

######  Theorem 13 (Eq. (19) [14]).

If assumptions 1, 2 and 3 hold, and for any value of the regularization
parameter $\overline{\lambda}\equiv\lambda/\mu_{\star}^{2}>0$, the asymptotic
sensitivity for the RFR, namely equation (7) with
$f=f_{{\bm{a}},{\bm{\Theta}}}$ where $f_{{\bm{a}},{\bm{\Theta}}}$ is defined
as in equations (1) and (2), satisfies

$\mathcal{S}\xrightarrow{p}\mathcal{S}^{\infty}\equiv\mu^{2}_{1}\left(\frac{F_{1}^{2}}{\mu_{\star}^{2}}\cdot\frac{\mathscrsfs{D}_{1}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}{(\chi{\zeta}^{2}-1)\mathscrsfs{D}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}+\frac{\tau^{2}+F_{\star}^{2}}{\mu_{\star}^{2}}\cdot\frac{\mathscrsfs{D}_{2}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}{\mathscrsfs{D}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})}\right),$
(56)

where $\xrightarrow{p}$ denotes convergence in probability when $d\to\infty$
with respect to the training data ${\bm{X}}$, ${\bm{\varepsilon}}{}$, the
random features ${\bm{\Theta}}$, and the random target function $f_{d}$, and
where

$\displaystyle\mathscrsfs{D}_{0}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})$
$\displaystyle=\chi^{5}{\zeta}^{6}-3\chi^{4}{\zeta}^{4}+(\psi_{1}+\psi_{2}-\psi_{1}\psi_{2}-1)\chi^{3}{\zeta}^{6}+2\chi^{3}{\zeta}^{4}+3\chi^{3}{\zeta}^{2}$
(57)
$\displaystyle+(3\psi_{1}\psi_{2}-\psi_{2}-\psi_{1}-1)\chi^{2}{\zeta}^{4}-2\chi^{2}{\zeta}^{2}-\chi^{2}-3\psi_{1}\psi_{2}\chi{\zeta}^{2}+\psi_{1}\psi_{2}\,,$
$\displaystyle\mathscrsfs{D}_{1}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})$
$\displaystyle=\chi^{6}{\zeta}^{6}-2\chi^{5}{\zeta}^{4}-(\psi_{1}\psi_{2}-\psi_{1}-\psi_{2}+1)\chi^{4}{\zeta}^{6}+\chi^{4}{\zeta}^{4}$
(58)
$\displaystyle+\chi^{4}{\zeta}^{2}-2(1-\psi_{1}\psi_{2})\chi^{3}{\zeta}^{4}-(\psi_{1}+\psi_{2}+\psi_{1}\psi_{2}+1)\chi^{2}{\zeta}^{2}-\chi^{2}\,,$
$\displaystyle\mathscrsfs{D}_{2}({\zeta},\psi_{1},\psi_{2},\overline{\lambda})$
$\displaystyle=-(\psi_{1}-1)\chi^{3}{\zeta}^{4}-\chi^{3}{\zeta}^{2}+(\psi_{1}+1)\chi^{2}{\zeta}^{2}+\chi^{2}\,,$
(59)

and where $\chi$ is defined in (14).

##  Appendix B Details regarding Theorem 9

###  B.1 Relationship among the constants in Table 1

This appendix is referenced in Remark 4.

The following relationships can be derived either from the proof of Theorem 9,
or from a direct computation based on the formulas given in Theorem 9. They do
not add critical information to what is proved in Theorem 9 but they give
general rules to exclude some cells in Table 1.

######  Remark 9.

Let $\psi_{1}<\psi_{2}$. If $\psi_{2}<\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{L}<\alpha_{C}<\alpha_{R}$. In this case, the 4th and 5th rows of
Table 1 never happen. If $\psi_{2}>\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{R}<\alpha_{C}<\alpha_{L}$. If
$\psi_{2}=\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{L}=\alpha_{C}=\alpha_{R}$. We always have that
$\alpha_{L},\alpha_{C},\alpha_{R}\in[0,1]$, and
$0\leq\beta_{3}<\beta_{2}<\beta_{1}$. Since $E_{1}$ and $E_{2}$ cannot
simultaneously hold, no two rows can simultaneously hold. Since
$\beta_{3}<\beta_{2}<\beta_{1}$, no two columns can simultaneously hold.
Remark 9 follows from Lemma D.13 used in the proof of Theorem 9.

######  Remark 10.

Let $\psi_{1}>\psi_{2}$. We always have that $\alpha_{L}\in[0,1]$. It also
holds that $\alpha_{C}\in[0,1]$ if and only if
$\psi_{1}\leq\psi_{2}/(1-\rho|1-\psi_{2}|)$. We have that $\alpha_{R}\in[0,1]$
if an only if
$\psi_{1}\geq\psi_{2}+\frac{1}{2}\rho(\psi_{2}-1)^{2}\min\\{\psi_{2},1\\}$ and

$\psi_{1}\leq\frac{-1+\rho(\psi_{2}-1)^{2}(2\psi_{2}+\min\\{2\psi_{2}-1,1\\})+\psi_{2}(2+\max\\{\psi_{2},2-\psi_{2}\\})}{2(\max\\{\psi_{2},1\\}+\rho(\psi_{2}-1)^{2})}.$

Note that if $\psi_{1}>\psi_{2}=1$ then $\alpha_{R}$ is not defined, see
statement of Theorem 9. If we had attempted to use the formulas above we would
have gotten that $\alpha\in[0,1]$ if and only if $1\leq\psi_{1}\leq 1$, this
last condition never being met when $\psi_{1}>\psi_{2}=1$. Also, $A\leq B$
always, where $A$ and $B$ are given in (43) and (44). Furthermore,
$B<\psi_{2}$ if and only if
$1/\rho>\gamma\triangleq\min\\{1,\max\\{0,2\psi_{2}-1\\}\\}$. Since $A\leq B$,
it follows that $E_{1}$ and $E_{2}$ cannot simultaneously hold, and hence no
two rows can simultaneously hold. We always have that
$\beta_{3}<\beta_{2}<\beta_{1}$, therefore no two columns can simultaneously
hold. Remark 10 follows from Lemma D.14 used in the proof of Theorem 9.

###  B.2 Definition of the polynomial $p(x)$ when $\psi_{1}<\psi_{2}$

This appendix is referenced in the statement of Theorem 9 right after Table 1
is introduced.

If $\psi_{1}\neq 1$ then $p(x)\triangleq
p_{0}+p_{1}x+p_{2}x^{2}+p_{3}x^{3}+p_{4}x^{4}+p_{5}x^{5}$ where

$\displaystyle
p_{0}=-(\psi_{1}-1)^{2}\psi_{1}^{2}\left(\rho(\alpha-4\alpha\psi_{1}+(3\alpha-1)\psi_{2})+\alpha\right),$
(60) $\displaystyle
p_{1}=2(\psi_{1}-1)\psi_{1}\left(\rho(\alpha(\psi_{1}(9\psi_{1}-6\psi_{2}-4)+\psi_{2})+2\psi_{1}\psi_{2})-2\alpha\psi_{1}\right),$
(61) $\displaystyle
p_{2}=2\psi_{1}\left(\rho(\alpha+4\alpha\psi_{1}(4\psi_{1}-3)+\psi_{2}(4\alpha-9\alpha\psi_{1}+3\psi_{1}-1))-\alpha(3\psi_{1}-1)\right),$
(62) $\displaystyle
p_{3}=4\psi_{1}\left(\rho(\alpha(7\psi_{1}-2)-3\alpha\psi_{2}+\psi_{2})-\alpha\right),$
(63) $\displaystyle
p_{4}=\rho(\alpha(12\psi_{1}-1)-3\alpha\psi_{2}+\psi_{2})-\alpha,$ (64)
$\displaystyle p_{5}=2\alpha\rho,$ (65)

If $\psi_{1}=1$ then $p(x)\triangleq q_{0}+q_{1}x+q_{2}x^{2}+q_{3}x^{3}$ where

$\displaystyle q_{0}=\rho(-10\alpha+10\alpha\psi_{2}-4\psi_{2})+4\alpha,$ (66)
$\displaystyle q_{2}=\rho(-11\alpha+3\alpha\psi_{2}-\psi_{2})+\alpha,$ (67)

$\displaystyle\quad q_{1}=\rho(-20\alpha+12\alpha\psi_{2}-4\psi_{2})+4\alpha,$
(68) $\displaystyle\quad\text{ and }q_{3}=-2\alpha\rho.$ (69)

###  B.3 Definition of the polynomial $p(x)$ when $\psi_{1}>\psi_{2}$

This appendix is referenced in the statement of Theorem 9 right after Table 1
is introduced.

$\displaystyle p(x)\triangleq
p_{0}+p_{1}x+p_{2}x^{2}+p_{3}x^{3}+p_{4}x^{4}+p_{5}x^{5},\text{ where }$ (70)
$\displaystyle
p_{0}=\psi_{2}^{2}\left(\rho\left(\psi_{2}-1\right){}^{2}\left(2\alpha\psi_{1}-(3\alpha+1)\psi_{2}+\alpha\right)+\left(\alpha\psi_{2}^{2}-2(\alpha+1)\psi_{2}+\alpha+2\psi_{1}\right)\right),$
(71) $\displaystyle
p_{1}=2\psi_{2}\left(\left(\psi_{2}\left(2\alpha\left(\psi_{2}{-}1\right){-}1\right)+\psi_{1}\right){-}\rho\left(\psi_{2}{-}1\right)\left((7\alpha{+}2)\psi_{2}^{2}{-}\psi_{2}\left(4\alpha\psi_{1}{+}3\alpha+1\right){+}\psi_{1}\right)\right),$
(72) $\displaystyle
p_{2}=2\psi_{2}\left(\alpha\left(3\psi_{2}-1\right)-\rho\left((13\alpha+3)\psi_{2}^{2}-2\psi_{2}\left(3\alpha\psi_{1}+5\alpha+1\right)+2\alpha\psi_{1}+\alpha+\psi_{1}\right)\right),$
(73) $\displaystyle
p_{3}=4\psi_{2}\left(\rho\left(2\alpha\left(\psi_{1}+1\right)-(6\alpha+1)\psi_{2}\right)+\alpha\right),$
(74) $\displaystyle
p_{4}=\rho\left(2\alpha\psi_{1}-(11\alpha+1)\psi_{2}+\alpha\right)+\alpha,$
(75) $\displaystyle p_{5}=-2\alpha\rho.$ (76)

###  B.4 Definition of the polynomial $r(x)$ when $\psi_{1}>\psi_{2}$

This appendix is referenced in the statement of Theorem 9 in equation (37).

The coefficient $\beta_{1}$ is such that $r(\beta_{1})=0$, where
$r(x)=r_{0}+r_{1}x+r_{2}x^{2}+r_{3}x^{3}+r_{4}x^{4}$ with

$r_{0}=\rho\psi_{2}^{3}(\rho\psi_{2}(\rho\psi_{2}(16\alpha\rho\psi_{2}-8\alpha(14\alpha\rho+\rho-6)+\rho)+8\alpha(\rho((8\alpha(2\alpha+5)-1)\rho-28\alpha+5)+6))+16\alpha(\alpha\rho+\rho+1)(-4\alpha\rho+\rho+1)^{2}),$
$r_{1}=4\rho\psi_{2}^{2}(\rho\psi_{2}(2\alpha(8\alpha\rho(7-2(\alpha+5)\rho)+3(\rho-5)\rho-18)-\rho\psi_{2}(-56\alpha^{2}\rho+12\alpha\rho\psi_{2}-6\alpha(\rho-6)+\rho))-4\alpha(\rho+1)((2\alpha-3)\rho-3)((4\alpha-1)\rho-1)),$
$r_{2}=2\rho\psi_{2}(-8\alpha^{2}\rho(\rho\psi_{2}(7\rho\psi_{2}-20\rho+14)+7(\rho+1)^{2})+12\alpha(\rho\psi_{2}+\rho+1)(\rho\psi_{2})+2(\rho+1)^{2})+3\rho^{3}\psi_{2}^{2}),$
$r_{3}=-8\alpha\rho(\rho\psi_{2}+\rho+1)(\rho\psi_{2}(2\rho\psi_{2}-3\rho+4)+2(\rho+1)^{2})-4\rho^{4}\psi_{2}^{2},\text{
and}$ $r_{4}=\rho^{4}\psi_{2}.$

##  Appendix C Important observations

This appendix is referenced in Section 4.3.

###  C.1 Reproducing Figure 1

The plots in Figure 1 come directly from our theory. They involve no
experiments and can be obtained using many standard mathematical computing
software tools. Nonetheless, since it does require some effort to code our
equations, we include code to generate Figure 1 in the following anonymous
Google Drive folder:
https://drive.google.com/drive/folders/1JtV6vo_N2jHg9IF1ITzqqP81Y7PKkcvy?usp=sharing.
To generate the plots in Figure 1, run the file named
RunMeToGenerateFigure_1.nb. It runs using Wolfram Mathematica V12. We ran it
in abou using a MacBook Pro with 2.6 GHz 6-Core Intel Core i7 and 32 GB 2667
MHz DDR4. In this machine it takes about 1 second to run.

###  C.2 Some details about observation 1 in Section 4.3

When $\alpha=0$ and $\psi_{1}<\psi_{2}$, and using Theorem 9, we can
explicitly compute the objective
$O=\mathcal{E}_{R_{1}}=(F_{\star}^{2}\psi_{2}+F_{1}^{2}\max\\{1-\psi_{1},0\\}\psi_{2}+\psi_{1}\tau^{2})/(\psi_{2}-\psi_{1})\text{
if }\psi_{1}<\psi_{2}.$ This follows from making $\alpha=0$ in the expressions
in the paragraph before Theorem 10. Observe that, as function of $\psi_{1}$,
the test error $\mathcal{E}_{R_{1}}$ achieves a minimum of
$(F_{\star}^{2}\psi_{2}+\tau^{2})/(-1+\psi_{2})$ at $\psi_{1}=1$ if
$\psi_{2}>1+1/\rho$ and a minimum of
$(F_{1}^{2}\psi_{2}+F_{\star}^{2}\psi_{2})/\psi_{2}$ at $\psi_{1}=0$
otherwise. In particular, if $\tau=F_{\star}=0^{+}$, then the condition
$\psi_{2}>1+1/\rho$ becomes $\psi_{2}>1$.

When $\alpha=0$ and $\psi_{1}>\psi_{2}$ the polynomial that determines $x_{1}$
becomes

$\displaystyle
p(x)=\psi_{2}\left({-}4\psi_{2}x^{3}{-}2\left(\psi_{1}{+}\psi_{2}\left(3\psi_{2}{-}2\right)\right)x^{2}{-}x^{4}{-}2\left(\psi_{2}{-}1\right)\left(\psi_{1}{+}\psi_{2}\left(2\psi_{2}{-}1\right)\right)x{-}\left(\psi_{2}{-}1\right){}^{2}\psi_{2}^{2}\right).$
(77)

One can show that in the range $\psi_{1}>\psi_{2}$ and
$x\in(x_{L},x_{R})=(-\psi_{2},\min\\{0,1-\psi_{2}\\})$, we have $p(x)<0$ and
hence it has no roots. Therefore, $x_{1}$ does not exist. Therefore, if
furthermore we have $\tau=F_{\star}=0$, then condition $E_{2}$ is never true,
so $x$ must be read from the first row in Table 12, which implies that
$x=x_{R}$, and the optimal AF is linear also when $\psi_{1}>\psi_{2}$.

##  Appendix D Proofs

This appendix is referenced in Section 4.

###  D.1 Proof of sensitivity properties of Section 3.2

###### Proof of Theorem 4.

The proof follows directly from Theorem 13 by taking the limit when
$\lambda\to+0$. ∎

###### Proof of Theorem 5.

The proof follows directly from Theorem 13 by taking the limit when
$\psi_{1}\to+\infty$. ∎

###### Proof of Theorem 6.

The proof follows directly from Theorem 13 by taking the limit when
$\psi_{2}\to+\infty$. ∎

###  D.2 Proof of necessary and sufficient condition for linearity of Section
4.1

###### Proof of Lemma 4.1 .

If $\sigma(x)$ is linear function, a direct calculation shows that
$\mu_{\star}=0$. On the other hand, since
$0\leq\operatorname{\mathbb{E}}(\sigma(Z)-\mu_{0}-\mu_{1}Z)^{2}=\mu^{2}_{\star}$,
we have that $\mu_{\star}=0$ implies that $\sigma(Z)=\mu_{0}+\mu_{1}Z$ almost
surely. Since the support of the probability density function of $Z$ is
$\mathbb{R}$, it follows that $\sigma(x)=\mu_{0}+\mu_{1}x$ except on a set of
measure zero in $\mathbb{R}$. ∎

###  D.3 Proof of Theorem 7

To prove Theorem 7, we first need to state and prove a series of intermediary
results.

######  Lemma D.1.

A necessary condition for optimally of $\sigma(x)$ is that

$-2x\sigma^{\prime}(x)+2\sigma^{\prime\prime}(x)+\lambda_{1}+\lambda_{2}x+\lambda_{3}\sigma(x)=0,$
(78)

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ satisfy the following
constraints,

$\displaystyle\lambda_{1}+\lambda_{3}\mu_{0}=0,$ (79)
$\displaystyle-2\mu_{1}+\lambda_{2}+\lambda_{3}\mu_{1}=0,$ (80)
$\displaystyle-2\mathbb{E}((\sigma^{\prime}(Z))^{2})+\lambda_{1}\mu_{0}+\lambda_{2}\mu_{1}+\lambda_{3}\mu_{2}=0,$
(81)

and that

$\lim_{x\to+\infty}(\sigma^{\prime}(x))^{2}e^{\frac{-x^{2}}{2}}=\lim_{x\to-\infty}(\sigma^{\prime}(x))^{2}e^{\frac{-x^{2}}{2}}=0.$
(82)

######  Remark 11.

Note that since (78) is a second order ODE, its solutions are parametrized by
two constants in addition to being parametrized by
$\lambda_{1},\lambda_{2},\lambda_{3}$. These five constants are set by our
three constraints (25) together with two boundary conditions from our
variational problem, which are
$\lim_{x\to\pm\infty}\sigma^{\prime}(x)e^{\frac{-x^{2}}{2}}=0$. These last two
we replace (see proof) by (82).

######  Remark 12.

The lemma implies that knowing $\mu_{0},\mu_{1},\mu_{2}$ and the objective
value $\mathbb{E}((\sigma^{\prime}(Z))^{2})$ is enough to determine
$\lambda_{1},\lambda_{2},\lambda_{3}$, even without solving the ODE.

###### Proof of Lemma D.1.

Derivation of equation (78): A Lagragian for (25)-(25) is

$\mathbb{E}((\sigma^{\prime}(Z))^{2}+\lambda_{1}(\mu_{0}-\sigma(Z))+\lambda_{2}(\mu_{1}-Z\sigma(Z))+\lambda_{3}((1/2)(\mu_{2}-\sigma(Z))^{2})),$
(83)

which can be written as
$\int^{\infty}_{-\infty}L(z,\sigma,\sigma^{\prime}){\rm d}z$ where, if we
define $p(z)=\frac{1}{\sqrt{2\pi}}e^{\frac{-z^{2}}{2}}$, the Lagrangian
density $L$ is

$\displaystyle
L(z,\sigma,{\sigma^{\prime}})=p(z)({\sigma^{\prime}}^{2}+\lambda_{1}(\mu_{0}-\sigma)+\lambda_{2}(\mu_{1}-z\sigma)+\lambda_{3}(1/2)(\mu_{2}-\sigma^{2})).$
(84)

We use the Euler-Lagrange equation with free boundary conditions [44] to get
the necessary condition (78). To do so, we compute in sequence

$\displaystyle\frac{\partial L}{\partial\sigma}$
$\displaystyle=-p(z)(\lambda_{1}+\lambda_{2}z+\lambda_{3}\sigma),$ (85)
$\displaystyle\frac{\partial L}{\partial\sigma^{\prime}}$
$\displaystyle=p(z)(2\sigma^{\prime}),$ (86) $\displaystyle\frac{\rm d}{{\rm
d}z}\frac{\partial L}{\partial\sigma^{\prime}}$
$\displaystyle=p^{\prime}(z)(2\sigma^{\prime})+p(z)(2\sigma^{\prime\prime})=p(z)(-2z\sigma^{\prime}+2\sigma^{\prime\prime}).$
(87)

These lead to

$\displaystyle 0=\frac{\rm d}{{\rm d}z}\frac{\partial
L}{\partial\sigma^{\prime}}-\frac{\partial
L}{\partial\sigma}=p(z)(-2z\sigma^{\prime}+2\sigma^{\prime\prime}+\lambda_{1}+\lambda_{2}z+\lambda_{3}\sigma)\text{
and }$ (88) $\displaystyle 0=\lim_{z\to\pm\infty}\frac{\partial
L}{\partial\sigma^{\prime}}=\lim_{z\to\pm\infty}p(z)\sigma^{\prime}(z),$ (89)

where the last condition follows from the fact that there is no boundary
condition on $\sigma$. Since $p(z)>0$, equation (88) implies (78).

Derivation of equations (82): Since we are working with necessary conditions,
we can choose not list (89) in our lemma. Rather, we include condition (82),
which a consequence of the fact that
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$ must be finite. Indeed,
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=\int^{\infty}_{-\infty}p(z)(\sigma^{\prime}(z))^{2}<\infty$
implies that $p(z)(\sigma^{\prime}(z))^{2}$ must vanish at $\pm\infty$.
Although not necessary for this proof, note that, since $p(z)$ goes to zero as
$z\to\pm\infty$, equations (82) imply equations (89).

Derivation of equations (79)-(81):

Equation (78) must hold for all $x$. Hence, we can replace in it $x$ by a
standard normal random variable $Z$ and compute the expected value of both of
its sides. This leads to

$-2\operatorname{\mathbb{E}}(Z\sigma^{\prime}(Z))+2\operatorname{\mathbb{E}}(\sigma^{\prime\prime}(Z))+\lambda_{1}+\lambda_{3}\mu_{0}=0.$
(90)

We can also multiply (78) by $x$, replace $x$ by a standard normal random
variable $Z$ and compute the expected value of both of its sides. This leads
to,

$-2\operatorname{\mathbb{E}}(Z^{2}\sigma^{\prime}(Z))+2\operatorname{\mathbb{E}}(Z\sigma^{\prime\prime}(Z))+\lambda_{2}+\lambda_{3}\mu_{1}=0.$
(91)

Finally, we multiply (78) by $\sigma(x)$, replace $x$ by a standard normal
random variable $Z$ and compute the expected value of both of its sides. This
leads to,

$-2\operatorname{\mathbb{E}}(Z\sigma(Z)\sigma^{\prime}(Z))+2\operatorname{\mathbb{E}}(\sigma(Z)\sigma^{\prime\prime}(Z))+\lambda_{1}\mu_{0}+\lambda_{2}\mu_{1}+\lambda_{3}\mu_{2}=0.$
(92)

Using integration by parts, and the fact that for
$p(z)=\frac{1}{\sqrt{2\pi}}e^{\frac{-z^{2}}{2}}$ we have that
$p^{\prime}(z)=-zp(z)$ and $p^{\prime\prime}(z)=(1-z^{2})p(z)$, we derive the
following useful relationships

$\displaystyle\operatorname{\mathbb{E}}(\sigma^{\prime\prime}(Z))=\operatorname{\mathbb{E}}(Z\sigma^{\prime}(Z)),$
(93)
$\displaystyle\operatorname{\mathbb{E}}(Z\sigma^{\prime\prime}(Z))=\operatorname{\mathbb{E}}(Z^{2}\sigma^{\prime}(Z))-\operatorname{\mathbb{E}}(\sigma^{\prime}(Z)),$
(94) $\displaystyle\operatorname{\mathbb{E}}(\sigma^{\prime}(Z))=\mu_{1},$
(95)
$\displaystyle\operatorname{\mathbb{E}}(\sigma(Z)\sigma^{\prime\prime}(Z))=\operatorname{\mathbb{E}}(Z\sigma(Z)\sigma^{\prime}(Z))-\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2}).$
(96)

To derive these relationships via integration by parts we made use of the
following relationships

$\displaystyle\lim_{z\to\pm\infty}\sigma^{\prime}(z)\sigma(z)p(z)=\lim_{z\to\pm\infty}\sigma^{\prime}(z)zp(z)=\lim_{z\to\pm\infty}\sigma^{\prime}(z)p(z)=\lim_{z\to\pm\infty}\sigma(z)p(z)=0,$
(97)

which can be proved from
$\operatorname{\mathbb{E}}((\sigma(Z))^{2}),\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$.
Also, from
$\operatorname{\mathbb{E}}((\sigma(Z))^{2}),\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$
and $\mu_{1}\in\mathbb{R}$, we can show that each of the expected values in
the right-hand-side of (93)-(96) is well-defined. Hence, the left-hand-side of
(93)-(96) is well defined.

To finish the proof we replace (93) in (90) to get
$\lambda_{1}+\lambda_{3}\mu_{0}=0$, which is equation (79). Then we replace
(94) and (95) in (91) to get $-2\mu_{1}+\lambda_{2}+\lambda_{3}\mu_{1}=0$,
which is equation (80). Lastly, we replace (96) in (92) to get
$-2\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})+\lambda_{1}\mu_{0}+\lambda_{2}\mu_{1}+\lambda_{3}\mu_{2}=0$,
which is equation (81). ∎

######  Lemma D.2.

The solutions of (78) are of the form

$\sigma(x)=\bar{\sigma}(x)+c_{1}H_{\frac{\lambda_{3}}{2}}\left(\frac{x}{\sqrt{2}}\right)+c_{2}F_{\\{\frac{-\lambda_{3}}{4}\\},\\{\frac{1}{2}\\}}\left(\frac{x^{2}}{2}\right),$
(98)

where

$\displaystyle\bar{\sigma}(x)=\frac{\lambda_{2}x}{2}-\frac{\lambda_{1}x^{2}}{4}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right),\text{
if }\lambda_{3}=0$ (99)
$\displaystyle\bar{\sigma}(x)=-\frac{\lambda_{1}}{2}-\frac{\lambda_{2}x}{2}+\frac{1}{2}\sqrt{\frac{\pi}{2}}\lambda_{2}e^{\frac{x^{2}}{2}}\text{erf}\left(\frac{x}{\sqrt{2}}\right)-\frac{1}{4}\lambda_{2}x^{3}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right),\text{
if }\lambda_{3}=2$ (100)
$\displaystyle\bar{\sigma}(x)=-\frac{\lambda_{1}}{\lambda_{3}}-\frac{\lambda_{2}x}{\lambda_{3}-2},\text{
if }\lambda_{3}\notin\\{0,2\\}$ (101)

where erf is the Guass error function , $F_{\\{a_{k}\\},\\{b_{k}\\}}$ is the
generalized hypergeometric function with parameters $\\{a_{k}\\},\\{b_{k}\\}$,
and $H_{n}$ is the Hermite polynomial, extended to a possibly non-integer $n$.

######  Remark 13.

By an extension of $H_{n}(x)$ to non-integer we mean that
$H_{n}(x)=F_{\\{-\frac{1}{2}(n-1)\\},\\{\frac{3}{2}\\}}(x^{2})$, which is
defined for non-integer $n$.

###### Proof of Lemma D.2.

Since (78) is a second order ODE, its solutions are spanned by any particular
solution $\bar{\sigma}(x)$ plus a linear combination of two solutions to its
associated homogeneous ODE. The homogenous ODE associate with (78) is
$-2x\sigma^{\prime}(x)+2\sigma^{\prime\prime}(x)+\lambda_{3}\sigma(x)=0$. The
change of variable $\sigma(x)=\tilde{\sigma}(x/\sqrt{2})$ allows us to get
$-2x\tilde{\sigma}^{\prime}(x)+\tilde{\sigma}^{\prime\prime}(x)+\lambda_{3}\tilde{\sigma}(x)=0$,
which is a well-known ODE called the _Hermite differential equation_. This
implies that the homogeneous solutions of our ODE are spanned by
$H_{\frac{\lambda_{3}}{2}}\left(\frac{x}{\sqrt{2}}\right)$ and
$F_{\\{\frac{-\lambda_{3}}{4}\\},\\{\frac{1}{2}\\}}\left(\frac{x^{2}}{2}\right)$.
The particular solutions $\bar{\sigma}(x)$ can be confirmed by direct
substitution. ∎

######  Lemma D.3.

Let $\sigma$ be of the form (98). If $\lambda_{3}=0$, then
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ implies that
$\sigma(x)$ is of the form

$\sigma(x)=c+\frac{x\lambda_{2}}{2},$ (102)

for some $c$, and that
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=\frac{\lambda^{2}_{2}}{4}$.

###### Proof of Lemma D.3.

If $\lambda_{3}=0$ then based on Lemma D.2 we can re-write that

$\displaystyle\sigma(x)=\sqrt{\frac{\pi}{2}}c_{1}\text{erfi}\left(\frac{x}{\sqrt{2}}\right)+c_{2}-\frac{1}{4}x\left(\lambda_{1}xF\left(\frac{x^{2}}{2}\right)-2\lambda_{2}\right),$
(103)

for some $c_{1},c_{2}$, where erfi is the inverse of erf. From this it follows
that,

$\sigma^{\prime}(x)=\frac{1}{4}\left(e^{\frac{x^{2}}{2}}\left(4c_{1}-\sqrt{2\pi}\lambda_{1}\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)+2\lambda_{2}\right).$
(104)

The objective $\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$
implies that
$\lim_{x\to\pm\infty}(\sigma^{\prime}(x))^{2}e^{-\frac{x^{2}}{2}}=0$. Since
$\lim_{x\to\pm\infty}\text{erf}(x)=\pm 1$, we have that
$\lim_{x\to\pm\infty}(\sigma^{\prime}(x))^{2}e^{-\frac{x^{2}}{2}}=0$ implies
that $4c_{1}=\pm\sqrt{2\pi}\lambda_{1}$ for both $\pm$ simultaneously. This
implies that $c_{1}=\lambda_{1}=0$.

Substituting $c_{1}=\lambda_{1}=0$ in (103) and simplifying we get
$\sigma(x)=c_{2}+\frac{\lambda_{2}x}{2}$. From this expression one can then
directly compute $\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$. ∎

######  Lemma D.4.

Let $\sigma$ be of the form (98). If $\lambda_{3}=2$, then
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ implies that
$\sigma(x)$ is of the form

$\sigma(x)=cx-\frac{\lambda_{1}}{2},$ (105)

for some $c$, and that
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=c^{2}$.

###### Proof of Lemma D.4.

If $\lambda_{3}=2$ then based on Lemma D.2 we can re-write that

$\displaystyle\sigma(x)=\frac{1}{4}\Bigg{(}-x\Big{(}\lambda_{2}\left(x^{2}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right)+2\right)$
$\displaystyle+2\sqrt{2}\left(\sqrt{\pi}c_{2}\text{erfi}\left(\frac{x}{\sqrt{2}}\right)-2c_{1}\right)\Big{)}+e^{\frac{x^{2}}{2}}\left(\sqrt{2\pi}\lambda_{2}\text{erf}\left(\frac{x}{\sqrt{2}}\right)+4c_{2}\right)-2\lambda_{1}\Bigg{)}$
(106)

for some $c_{1}$ and $c_{2}$, where erfi is the inverse of erf. From this it
follows that

$\sigma^{\prime}(x)=\frac{2c_{1}-\sqrt{\pi}c_{2}\text{erfi}\left(\frac{x}{\sqrt{2}}\right)}{\sqrt{2}}-\frac{1}{4}\lambda_{2}x^{2}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right).$
(107)

The objective $\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$
implies that
$\lim_{x\to\pm\infty}(\sigma^{\prime}(x))^{2}e^{-\frac{x^{2}}{2}}=0$. Since
$\lim_{x\to\pm\infty}\left(\text{erfi}\left(\frac{x}{\sqrt{2}}\right)\right)^{2}=\lim_{x\to\pm\infty}\left(x^{2}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right)\right)^{2}=\infty$,
and since

$\lim_{x\to\pm\infty}\frac{x^{2}F_{\\{1,1\\},\\{\frac{3}{2},2\\}}\left(\frac{x^{2}}{2}\right)}{\text{erfi}\left(\frac{x}{\sqrt{2}}\right)}=\pm\pi,$
(108)

it follows we only have a finite objective if
$\frac{-\sqrt{\pi}c_{2}}{\sqrt{2}}=\pm\frac{\lambda_{2}\pi}{4}$ for both $\pm$
simultaneously. But this implies that $c_{2}=\lambda_{2}=0$.

Substituting $c_{2}=\lambda_{2}=0$ in (D.3), and simplifying, leads to
$\sigma(x)=c_{1}\sqrt{2}x-\frac{\lambda_{1}}{2}$. The $\sqrt{2}$ factor can be
absorbed by $c_{1}$. From this expression one can compute
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$. ∎

######  Lemma D.5.

Let $\sigma$ be of the form (98). If $\lambda_{3}\notin\\{0,2\\}$,
$c_{1}=c_{2}=0$ then

$\sigma(x)=-\frac{\lambda_{1}}{\lambda_{3}}-\frac{\lambda_{2}x}{\lambda_{3}-2},$
(109)

and

$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=\left(\frac{\lambda_{2}}{\left(\lambda_{3}-2\right)}\right)^{2}.$
(110)

###### Proof of Lemma D.5.

This follows directly from Lemma D.2. ∎

######  Lemma D.6.

Let $\sigma$ be of the form (98). If $\lambda_{3}\notin\\{0,2\\}$, and either
$c_{1}$ or $c_{2}$ are non-zero, then
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ implies that
$\lambda_{3}=4k$ for $k\in\mathbb{Z}^{+}$.

###### Proof of Lemma D.6.

If $\lambda_{3}\notin\\{0,2\\}$, then by Lemma D.2 we have a formula for
$\sigma(x)$ from which we get that

$\sigma^{\prime}(x)=\frac{c_{1}\lambda_{3}H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)}{\sqrt{2}}-\frac{1}{2}c_{2}\lambda_{3}xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)-\frac{\lambda_{2}}{\lambda_{3}-2},$
(111)

where $F$ and $H$ are as defined in Lemma D.2.

The boundeness of $\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$ is
dependent on how fast
$H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)$ and
$xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)$
grow as $x\to\pm\infty$.

If $c_{1}=0$ and $c_{2}\neq 0$,
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ only if
$\operatorname{\mathbb{E}}\left(\left(xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)\right)^{2}\right)<\infty$,
which in turn is true only if
$\lim_{x\to\infty}\left(xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)\right)^{2}e^{\frac{-x^{2}}{2}}=0$.
This implies that $\lambda_{3}$ is even.

If $c_{1}\neq 0$ and $c_{2}=0$,
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ only if
$\operatorname{\mathbb{E}}\left(\left(H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)\right)^{2}\right)<\infty$,
which in turn is true only if
$\lim_{x\to-\infty}\left(H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)\right)^{2}e^{\frac{-x^{2}}{2}}=0$.
This implies that $\lambda_{3}$ is even.

If $c_{1},c_{2}\neq 0$,
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ only if
$\lim_{x\to\infty}\left(\sigma^{\prime}(x)\right)^{2}e^{\frac{-x^{2}}{2}}=0$.
But
$xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)$
grows much faster than
$H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)$ as $x\to\infty$,
hence it must be that
$\lim_{x\to\infty}\left(xF_{\\{1-\frac{\lambda_{3}}{4}\\},\\{\frac{3}{2}\\}}\left(\frac{x^{2}}{2}\right)\right)^{2}e^{\frac{-x^{2}}{2}}=0$.
This implies that $\lambda_{3}$ is even. ∎

######  Lemma D.7.

Let $\lambda_{3}=4k$ for $k\in\mathbb{Z}^{+}$, and let $\sigma(x)$ be a
solution of (78), then

$\sigma(x)=-\frac{\lambda_{1}}{\lambda_{3}}-\frac{\lambda_{2}x}{\lambda_{3}-2}+cH_{2k}\left(\frac{x}{\sqrt{2}}\right)$
(112)

for some $c$ and

$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=\frac{4^{k}((2k)!)^{2}}{(2k-1)!}c^{2}+\left(\frac{\lambda_{2}}{\lambda_{3}-2}\right)^{2}.$
(113)

###### Proof of Lemma D.7.

If $\lambda_{3}=4k$ then

$F_{\\{\frac{-\lambda_{3}}{4}\\},\\{\frac{1}{2}\\}}\left(x^{2}/2\right)=\frac{(-1)^{k}k!}{(2k)!}H_{\lambda_{3}/2}(x/\sqrt{2}),$
(114)

and hence the homogenous part of $\sigma(x)$ can be written as
$cH_{\lambda_{3}/2}(x)$, from which (112) follows. From this expression it
follows that

$\sigma^{\prime}(x)=\frac{c\lambda_{3}H_{\frac{\lambda_{3}}{2}-1}\left(\frac{x}{\sqrt{2}}\right)}{\sqrt{2}}-\frac{\lambda_{2}}{\lambda_{3}-2},$
(115)

and from this expression the value for
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$ follows. ∎

######  Lemma D.8.

Let $\sigma$ be of the form (98). If $\lambda_{3}\notin\\{0,2\\}$, then
$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})<\infty$ implies that
$\sigma(x)$ is of the form

$\sigma(x)=-\frac{\lambda_{1}}{\lambda_{3}}-\frac{\lambda_{2}x}{\lambda_{3}-2}+cH_{\lambda_{3}/2}\left(\frac{x}{\sqrt{2}}\right)$
(116)

for some $c$ that must be zero if $\lambda_{3}\neq 4k$, $k\in\mathbb{Z}^{+}$,
and

$\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})=\frac{4^{k}((2k)!)^{2}}{(2k-1)!}c^{2}+\left(\frac{\lambda_{2}}{\lambda_{3}-2}\right)^{2}.$
(117)

###### Proof of Lemma D.8.

This follows directly from Lemmas D.5-D.7. ∎

######  Lemma D.9.

If $\lambda_{1},\lambda_{2},\lambda_{3}$ satisfy (79)-(81), then

$\displaystyle\left(\frac{\lambda_{2}}{\lambda_{3}-2}\right)^{2}=\mu^{2}_{1}.$
(118)

###### Proof of Lemma D.9.

Defining $R=\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$ and solving
the linear system (79)-(81) leads to

$\lambda_{1}=\frac{2\mu_{0}\left(R-\mu_{1}^{2}\right)}{\mu_{0}^{2}+\mu_{1}^{2}-\mu_{2}},\lambda_{2}=\frac{2\mu_{1}\left(\mu_{0}^{2}-\mu_{2}+R\right)}{\mu_{0}^{2}+\mu_{1}^{2}-\mu_{2}},\lambda_{3}=-\frac{2\left(R-\mu_{1}^{2}\right)}{\mu_{0}^{2}+\mu_{1}^{2}-\mu_{2}}.$
(119)

Computing $\lambda_{2}/(\lambda_{3}-2)$ we get $-\mu_{1}$, from which the
first relation follows.

If we solve
$\lambda_{3}=-\frac{2\left(R-\mu_{1}^{2}\right)}{\mu_{0}^{2}+\mu_{1}^{2}-\mu_{2}}$
for $R$ and recall that $\mu^{2}_{\star}=\mu_{2}-\mu_{0}^{2}-\mu_{1}^{2}$, the
second relation follows.

∎

We are now ready to prove Theorem 7, which we restate here for convenience.

######  Theorem ( 7).

The minimizers of (25) are

$\displaystyle\sigma(x)=ax^{2}+bx+c,$ (120)

where

$a=\pm\frac{\mu_{\star}}{\sqrt{2}},b=\mu_{1},\text{ and }c=\mu_{0}-a.$ (121)

###### Proof of Theorem 7.

We will show that any minimizer of (25) must be a function
$\sigma(x)=ax^{2}+bx+c$ for some $a,b,c$. From this fact, the variational
problem can be reduced to a simple quadratic programming problem over $a,b,c$,
from which it is straighforward to derive (121) and (121).

If $\mu_{\star}=0$ then by Lemma 4.1, we know that the solution must be
linear, and hence we are done.

If $\mu_{\star}>0$ then by Lemma 4.1 we know that $\sigma(x)$ cannot be a
linear function. Hence, from Lemma D.3 and Lemma D.4 we know that
$\lambda_{3}$ cannot be $0$ or $2$. Therefore, its solution must be of the
form specified by Lemma D.8 with $c\neq 0$ and $\lambda_{3}=4k$ for some
$k\in\mathbb{Z}^{+}$.

Define $R=\operatorname{\mathbb{E}}((\sigma^{\prime}(Z))^{2})$. From the
constraints (25) we know that $\lambda_{1},\lambda_{2}$ and $R$ can be written
as a linear function of $\lambda_{3}=4k$. In particular,
$R=\mu^{2}_{1}+\frac{\lambda_{3}}{2}\mu^{2}_{\star}$. From (117) and Lemma D.9
we can write that $R=\frac{4^{k}((2k)!)^{2}}{(2k-1)!}c^{2}+\mu^{2}_{1}$, which
implies that
$c=\pm\left(\frac{R-\mu^{2}_{1}}{\frac{4^{k}((2k)!)^{2}}{(2k-1)!}}\right)$ is
also a function of $\lambda_{3}$. Therefore, the solution to is parametrized
by $\lambda_{3}$ alone which must be chosen to minimize
$R=\mu^{2}_{1}+\frac{\lambda_{3}}{2}\mu^{2}_{\star}$. Therefore, we must
choose $\lambda_{3}=4$, the smallest possible multiple of $4$. This implies
that $H_{\lambda_{3}/2}(x/\sqrt{2})$ is a quadratic function, from which it
follows that $\sigma(x)$ is also a quadratic function, and hence we are done.
∎

###  D.4 Proof of Theorem 8

Before we prove Theorem 8, we will state and prove a series of intermediary
results.

######  Lemma D.10.

A necessary condition for optimality of $\sigma(x)$ is that

$x+\lambda_{1}+\lambda_{2}x+\lambda_{3}\sigma(x)=0,\text{ for all
}x:\sigma^{\prime}(x)\neq 0,$ (122)

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ must be such that,

$\displaystyle\operatorname{\mathbb{E}}(\sigma(Z))=\mu_{0},$ (123)
$\displaystyle\operatorname{\mathbb{E}}(\sigma(Z)Z)=\mu_{1},$ (124)
$\displaystyle\operatorname{\mathbb{E}}((\sigma(Z))^{2})=\mu_{2}.$ (125)

###### Proof of Lemma D.10.

A Lagragian for (122) is

$\mathbb{E}(|\sigma^{\prime}(Z)|+\lambda_{1}(\mu_{0}-\sigma(Z))+\lambda_{2}(\mu_{1}-Z\sigma(Z))+\lambda_{3}((1/2)(\mu_{2}-\sigma(Z))^{2})),$
(126)

which can be written as
$\int^{\infty}_{-\infty}L(z,\sigma,\sigma^{\prime}){\rm d}z$ where, if we
define $p(z)=\frac{1}{\sqrt{2\pi}}e^{\frac{-z^{2}}{2}}$, the Lagrangian
density $L$ is

$\displaystyle
L(z,\sigma,\sigma^{\prime})=p(z)(|\sigma^{\prime}|+\lambda_{1}(\mu_{0}-\sigma)+\lambda_{2}(\mu_{1}-z\sigma)+\lambda_{3}(1/2)(\mu_{2}-\sigma^{2})).$
(127)

Any variation $\sigma(z)+\epsilon\eta(z)$ of an optimal $\sigma(z)$ must yield
$\operatorname{\mathbb{E}}(|\sigma^{\prime}(z)+\epsilon\eta^{\prime}(z)|)\geq\operatorname{\mathbb{E}}(|\sigma^{\prime}(z)|)$.
In particular, this must be the case for any variation such that $\eta(z)=0$
whenever $\sigma^{\prime}(z)=0$. If we focus on these variations, from
$\operatorname{\mathbb{E}}(|\sigma^{\prime}(z)+\epsilon\eta^{\prime}(z)|)\geq\operatorname{\mathbb{E}}(|\sigma^{\prime}(z)|)$
the Euler-Lagrange equation can be derived despite $|\cdot|$ not being
differentiable at $0$. To be specific, it must hold that

$\frac{\rm d}{{\rm d}z}\frac{\partial
L}{\partial\sigma^{\prime}}-\frac{\partial L}{\partial\sigma}=0\;\forall
z:\sigma^{\prime}(z)\neq 0.$ (128)

Since there are no fixed boundary conditions on our integration domain
$(-\infty,\infty)$, it also needs to hold that
$\lim_{z\to\pm\infty}\frac{\partial
L}{\partial\sigma^{\prime}}=\lim_{z\to\pm\infty}p(z)\sigma^{\prime}(z)=0$,
which we choose not to list in our necessary conditions.

Since $\frac{\rm d}{{\rm d}z}\frac{\partial L}{\partial\sigma^{\prime}}=0$ if
$\sigma^{\prime}(z)\neq 0$, equation (128) implies (122). First order
optimality conditions imply that the Lagrange multipliers must be choose such
that
$\operatorname{\mathbb{E}}(\sigma(Z))=\mu_{0},\operatorname{\mathbb{E}}(\sigma(Z)Z)=\mu_{1},\operatorname{\mathbb{E}}((\sigma(Z))^{2})=\mu_{2}$.
∎

######  Lemma D.11.

If $\lambda_{3}\neq 0$, the solutions of (122) are of the form

$\displaystyle\sigma(x)=-\frac{\lambda_{1}}{\lambda_{3}}-\frac{1+\lambda_{2}}{\lambda_{3}}\min\\{\max\\{x,b\\},c\\},$
(129)

for some constants $b,c$ where $b<c$ if $\frac{1+\lambda_{2}}{\lambda_{3}}\leq
0$ and $b>c$ otherwise.

###### Proof of Lemma D.11.

Since $\sigma$ is a one-dimensional function, the requirement of existence of
weak derivative implies that there exists $v$ absolute continuous that agrees
with $\sigma$ almost everywhere. We will work with these absolute continuous
representations of $\sigma$. Other solutions can differ from the absolute
continuous solutions only up to a set of measure zero with respect to the
Guassian measure.

From (122) we know that wherever $\sigma^{\prime}(x)\neq 0$, we have that
$\sigma(x)=ax+b$ for the same fixed $a$ and $b$. Hence, since $\sigma$ is
continuous, $\sigma$ must be an alternation of flat portions and portions with
the same slope $a$. Because of continuity, we cannot have a flat portion
interrupt portion of slope $a$ (unless $a=0$), as illustrated in Figure
2-right, and $\sigma$ must be as in the other two cases in Figure 2. These
have a form as in (129).

Figure 2: The first two functions are the only two possible types of
continuous functions that satisfy (122). The right-most function also
satisfies (122) but is not continuous.

∎

######  Lemma D.12.

Let $\sigma$ be of the form (129). Then,

$\displaystyle\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z)|)=\frac{1}{2}\left|\frac{1+\lambda_{2}}{\lambda_{3}}\right|{\mathbb{P}(Z\in[b,c])},$
(130) $\displaystyle\mu_{1}\triangleq$
$\displaystyle\operatorname{\mathbb{E}}(Z\sigma(Z))=-\frac{1}{2}\frac{1+\lambda_{2}}{\lambda_{3}}{\mathbb{P}(Z\in[b,c])},$
(131)

where $[b,c]$ should be interpreted as $[c,b]$ if $c<b$. In particular,
$\operatorname{\mathbb{E}}(|\sigma^{\prime}(Z)|)=|\mu_{1}|$.

###### Proof of Lemma D.12.

From Lemma D.11, we have explicit formulas for $\sigma$ and $\sigma^{\prime}$.
The proof boils down to a direct calculation of the expected values, which
themselves boil down to computing a few Guassian integrals. ∎

We are now ready to prove Theorem 8, which we restate below for convenience.

######  Theorem (8).

One minimizer of (3) is

$\displaystyle\sigma(x)=\mu_{0}+a\min\\{\max\\{x,-s\\},s\\}$ (132)

where $a=\frac{\mu_{1}}{\text{erf}(s/\sqrt{2})}$, erf is the Guass error
function, and $s\in\mathbb{R}$ is the unique solution to the equation

$\zeta^{2}\triangleq\frac{\mu^{2}_{1}}{\mu^{2}_{\star}}=\frac{e^{\frac{s^{2}}{2}}\text{erf}\left(\frac{s}{\sqrt{2}}\right)^{2}}{e^{\frac{s^{2}}{2}}\left(1-\text{erf}\left(\frac{s}{\sqrt{2}}\right)\right)\left(s^{2}+\text{erf}\left(\frac{s}{\sqrt{2}}\right)\right)-\sqrt{\frac{2}{\pi}}s},$
(133)

if $\mu_{\star}\neq 0$, and $s=\infty$ if $\mu_{\star}=0$.

###### Proof of Theorem 8.

From Lemma D.11, we know that if $\lambda_{3}\neq 0$, then any minimizer must
have the form (129). From Lemma D.12 we know that all of the functions of this
form have the same objective. Hence, if $\lambda_{3}\neq 0$, all of the
functions of the form (129) that satisfy constraints (25) are a global
minimizer .

We set $\lambda_{3}=1\neq 0$. To satisfy the three constraints
$\eqref{eq:var_problem}$ we have $4$ remaining values to play with, namely
$\lambda_{1},\lambda_{2},b,c$. Hence, we set $b=-c$. With a
reparameterization, this leads to $\sigma$ having the form
$\sigma(x)=b+a\min\\{\max\\{x,-s\\},s\\}$, where $b$ has a new meaning. That
is, $\sigma$ is flat outside of the interval $[-s/|a|,s/|a|]$, $s\geq 0$, and
inside of this interval it has slope $a$. The goal is to find $a,b,s$ from the
constraints $\eqref{eq:var_problem}$.

From $\sigma(x)=b+a\min\\{\max\\{x,-s\\},s\\}$ a direct computations leads to

$\displaystyle\mu_{0}$ $\displaystyle=b,$ (134) $\displaystyle\mu_{1}$
$\displaystyle=a\text{erf}\left(\frac{s}{\sqrt{2}}\right),$ (135)
$\displaystyle\mu_{2}$
$\displaystyle=a^{2}\left(s^{2}-1\right)\text{erfc}\left(\frac{s}{\sqrt{2}}\right)+a^{2}\left(1-\sqrt{\frac{2}{\pi}}se^{-\frac{s^{2}}{2}}\right)+b^{2},$
(136)

where $\text{erfc}=1-\text{erf}$. We can use the first two equations to write
$b$ and $a$ as a function of $\mu_{0},\mu_{1},s$. Substituting $a$ and $b$
with these functions in the third equation, and simplifying, leads to

$\mu_{2}=\mu_{0}^{2}-\frac{\mu_{1}^{2}\left(-\left(s^{2}-1\right)\text{erfc}\left(\frac{s}{\sqrt{2}}\right)+\sqrt{\frac{2}{\pi}}e^{-\frac{s^{2}}{2}}s-1\right)}{\text{erf}\left(\frac{s}{\sqrt{2}}\right)^{2}}.$
(137)

Recalling that $\mu_{2}=\mu^{2}_{\star}+\mu^{2}_{1}+\mu^{2}_{0}$, replacing
this definition into the above equation, and simplifying leads to

$\frac{\mu^{2}_{1}}{\mu^{2}_{\star}}\triangleq\zeta^{2}=-\frac{e^{\frac{s^{2}}{2}}\text{erf}\left(\frac{s}{\sqrt{2}}\right)^{2}}{e^{\frac{s^{2}}{2}}\left(\text{erf}\left(\frac{s}{\sqrt{2}}\right)-1\right)\left(s^{2}+\text{erf}\left(\frac{s}{\sqrt{2}}\right)\right)+\sqrt{\frac{2}{\pi}}s}.$
(138)

One can show that the function on the right hand side (138) is monotonic
increasing in $s\in[-\infty,\infty]$ with range $[0,\infty]$, which implies
that there is only one $s$ that solves (138).

∎

###  D.5 Proof of Theorem 9

###### Proof of Theorem 9.

Theorem 9 amounts a statement about the solutions of the optimization problem
(29) for regime $R_{1}$.

Its proof amounts to studying the local minima of the objective via its first
and second derivatives.

We will prove the theorem for $\psi_{1}<\psi_{2}$ and $\psi_{1}>\psi_{2}$
separately. For $\psi_{1}=\psi_{2}$ the objective is not defined.

In what follows, we let
$O=(1-\alpha)\mathcal{E}^{\infty}_{R_{1}}+\alpha\mathcal{S}^{\infty}_{R_{1}}$.
We will use the fact that $O$ is a one-dimensional function of
$\zeta^{2}\in[0,+\infty]$. We will use this and the fact that (14) defines a
monotonic function between $\chi$ and $\zeta^{2}$, to express $O$ as a
function of $\chi$ and study the solutions of the optimization problem (29)

in the variable $\chi$. Notice that (14) can be solved for $\zeta^{2}$ as
$\zeta^{2}=\frac{\mu^{2}_{1}}{\mu^{2}_{\star}}=\frac{\chi+\min\\{\psi_{1},\psi_{2}\\}}{\chi(-1+\chi+\min\\{\psi_{1},\psi_{2}\\})}$
and, calling $\chi$ by $x$, this can be arranged to get (30), which connects
optimal values of $\chi$ with optimal values of $\mu_{0},\mu_{1},\mu_{2}$.
Henceforth, we denote $\chi$ by $x$. We note that since
$\zeta^{2}\in[0,+\infty]$, from (14) it follows that
$x\in[x_{L},x_{R}]\triangleq[-\min\\{\psi_{1},\psi_{2}\\},\min\\{0,1-\min\\{\psi_{1},\psi_{2}\\}\\}]$.
Hence, we only need to study the function $O(x)$ in this interval.

Case 1, $\psi_{1}<\psi_{2}$:

For the sake of simplicity, and for the most part, we will assume that
$\psi_{1}\neq 1$ and omit the argument for $\psi_{1}=1$. The argument for
$\psi_{1}=1$ is almost identical to the argument for $\psi_{1}\neq 1$ if we
work with the extended reals $[-\infty,\infty]$. Furthermore, most conclusions
for $\psi_{1}=1$ can be obtained by taking the limit of $\psi_{1}\to 1$ with
$\psi_{1}\neq 1$. The only situation where this is not the case is that the
polynomial $p(x)$, referenced right after Table 1, has an expression when
$\psi_{1}=1$ that cannot be obtained as the limit of its expression for
$1\neq\psi_{1}$.

We start by studying the second derivative of $O$. A direct computation yields

$\displaystyle\frac{{\rm d}^{2}O}{{\rm d}x^{2}}=\tilde{O}(x)+C,\text{ where
}C=\frac{2F_{1}^{2}\alpha}{\psi_{1}-\psi_{2}}$ (139)

and $\tilde{O}(x)=-\frac{2\alpha
F_{1}^{2}\psi_{1}\left(-3x^{2}(\psi_{1}-1)-2x^{3}+(\psi_{1}-1)^{2}\psi_{1}\right)}{\left((x+\psi_{1})^{2}-\psi_{1}\right)^{3}}$.

A tedious calculation (omitted) shows that $\frac{{\rm d}^{2}\tilde{O}}{{\rm
d}x^{2}}\geq 0$ for $x\in[x_{L},x_{R}]$ (i.e. it is convex), that
$\tilde{O}(x_{L})<\tilde{O}(x_{R})$, and that $\frac{{\rm d}\tilde{O}}{{\rm
d}x}(x=x_{L})<0$. From this it follows that, depending on the value of
$\psi_{2}$, the concavity $\frac{{\rm d}^{2}O}{{\rm d}x^{2}}$ is positive or
negative, as illustrated in the following figure:

Figure 3: Depending on the value of $\psi_{2}$, the function $O$ is convex,
convex-concave-convex, concave-convex, or concave respectively. The points A,
B, and C will be referenced later in the proof. It is possible to compute
closed-form expressions for the $y$-coordinate of these points.

To be specific, starting from large $\psi_{2}$, i.e. small $-C$, and
decreasing its value, i.e. increasing $-C$, we obtain the following four
scenarios. While $\psi_{2}$ is large and $-C$ is bellow $A_{2}$, the function
is convex. After $\psi_{2}$ reaches a value $\beta_{1}$ at which $-C$ touches
$A_{2}$, the function is convex for small $x$, then concave, and then convex
for large $x$. As $\psi_{2}$ keeps decreasing, and after it reaches a value
$\beta_{2}$ at which $-C$ touches $A_{1}$, the function is concave for small
$x$, and then convex for large $x$. Finally, after $\psi_{2}$ reaches a value
$\beta_{3}$ at which $-C$ touches $A_{3}$ , the function is concave. Note that
by definition $0\leq\beta_{3}<\beta_{2}<\beta_{1}$.

It is possible to compute closed-form expressions for the $y$-coordinate of
the points $A_{1},A_{2},A_{3}$, which we denote by $A_{1},A_{2},$ and $A_{3}$.
Namely,
$A_{1}=F^{2}_{1}\alpha(2+\frac{2}{\psi_{1}}),A_{2}=F_{1}^{2}\alpha\left(\frac{\sqrt{1-\psi_{1}}+1}{\psi_{1}}-\frac{1}{2}\right)$,
if $\psi_{1}\leq 1$, and
$A_{2}=F_{1}^{2}\alpha\left(\sqrt{\frac{\psi_{1}-1}{\psi_{1}}}-\frac{1}{2\psi_{1}}+1\right)$,
if $\psi_{1}>1$, and
$A_{3}=F_{1}^{2}\alpha\max\left\\{\frac{2}{{\psi_{1}}-{\psi_{1}}^{2}},\frac{2{\psi_{1}}}{{\psi_{1}}-1}\right\\}$.
Using the closed-form expression for $-C$, see (139), we find closed-form
expressions for $\beta_{1},\beta_{2}$, and $\beta_{3}$ as
$\beta_{1}=\psi_{2}:A_{2}(\psi_{2})+C(\psi_{2})=0$,
$\beta_{2}=\psi_{2}:A_{1}(\psi_{2})+C(\psi_{2})=0$, and
$\beta_{3}=\psi_{2}:A_{3}(\psi_{2})+C(\psi_{2})=0$. By definition of
$\beta_{1},\beta_{2}$, and $\beta_{3}$, these equations have a unique solution
when $\psi_{2}>\psi_{1}>0$ and their explicit expressions are given in (31),
(32) and (33) respectively.

Now that we have characterized the curvature of $O(x)$, we are ready to locate
its global minimum. To do so, we will use the curvature of $O(x)$ together
with the first-order optimally condition $\frac{{\rm d}O}{{\rm d}x}=0$, and
the following three extra pieces of information: the sign of the derivative of
$O(x)$ at $x=x_{L}$; the sign of the derivative of $O(x)$ at $x=x_{R}$; and
the sign of $O(x_{L})-O(x_{R})$.

A direct computation yields

$\displaystyle\frac{{\rm d}O}{{\rm d}x}$
$\displaystyle=\frac{p_{0}+p_{1}x+p_{2}x^{2}+p_{3}x^{3}+p_{4}x^{4}+p_{5}x^{5}}{(\psi_{1}-(x+\psi_{1})^{2})^{2}(\psi_{1}-\psi_{2})},\text{
if }\psi_{1}\neq 1,\text{ and }$ (140) $\displaystyle\frac{{\rm d}O}{{\rm
d}x}$
$\displaystyle=\frac{q_{0}+q_{1}x+q_{2}x^{2}+q_{3}x^{3}}{(2+x)^{2}(-1+\psi_{2})},\text{
if }\psi_{1}=1.$ (141)

where the coefficients $p_{0},\dots,p_{5}$ and $q_{0}\dots,q_{3}$ are, apart
from a multiplying constant, given in (60)- (69).

The roots of the first denominator, i.e. $-\sqrt{\psi_{1}}-\psi_{1}$ and
$\sqrt{\psi_{1}}-\psi_{1}$, are not a root of the first numerator when
$\psi_{1}\neq 1$, and the roots of the second denominator, i.e. $-2$, are not
a root of the second numerator when $\psi_{2}>\psi_{1}=1$. Therefore, the
first-order optimality conditions are $p(x)=0$.

Not all solutions of $p(x)=0$ minimize $O(x)$. To locate the minimizer, we use
the sign of the derivative of $O(x)$ at $x=x_{L}$; the sign of the derivative
of $O(x)$ at $x=x_{R}$; and the sign of $O(x_{L})-O(x_{R})$. These signs can
be determined using Lemma D.13. Lemma D.13 also proves Remark 9.

######  Lemma D.13.

If $\psi_{1}<\psi_{2}$, the following relationships hold,

$\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{L}}=-\frac{{F_{1}}^{2}(\alpha+(\alpha-1){\psi_{2}})+\alpha\left({F_{\star}}^{2}+{\tau}^{2}\right)}{{\psi_{1}}-{\psi_{2}}},$
(142) $\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}=\frac{{F_{1}}^{2}{\psi_{2}}-\alpha\left(F_{1}^{2}(-4\psi_{1}+3\psi_{2}+1)+F_{\star}^{2}+\tau^{2}\right)}{\psi_{1}-\psi_{2}}\text{
if }\psi_{1}\leq 1,$ (143) $\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}=\frac{{F_{1}}^{2}(\alpha+2\alpha{\psi_{1}}-3\alpha{\psi_{2}}+{\psi_{2}})-\alpha\left({F_{\star}}^{2}+{\tau}^{2}\right)}{{\psi_{1}}-{\psi_{2}}}\text{
if }\psi_{1}>1,$ (144) $\displaystyle
O(x_{L})-O(x_{R})=\frac{\psi_{1}\left(F_{1}^{2}(\alpha-2\alpha\psi_{1}+(2\alpha-1)\psi_{2})+\alpha\left(F_{\star}^{2}+\tau^{2}\right)\right)}{\psi_{1}-\psi_{2}}\text{
if }\psi_{1}\leq 1,\text{ and }$ (145) $\displaystyle
O(x_{L})-O(x_{R})=\frac{\alpha\left({F_{\star}}^{2}+{\tau}^{2}\right)-{F_{1}}^{2}(\alpha{\psi_{1}}-2\alpha{\psi_{2}}+{\psi_{2}})}{{\psi_{1}}-{\psi_{2}}}\text{
if }\psi_{1}>1.$ (146)

Also,

$\displaystyle\frac{{\rm d}O}{{\rm d}x}\bigg{\rvert}_{x=x_{L}}<0\text{ if
}\alpha<\alpha_{L}\text{ and }\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{L}}\geq 0\text{ if }\alpha\geq\alpha_{L},$ (147)
$\displaystyle\frac{{\rm d}O}{{\rm d}x}\bigg{\rvert}_{x=x_{R}}<0\text{ if
}\alpha<\alpha_{R}\text{ and }\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}\geq 0\text{ if }\alpha\geq\alpha_{R},\text{ and }$
(148) $\displaystyle O(x_{L})-O(x_{R})>0\text{ if }\alpha<\alpha_{C}\text{ and
}O(x_{L})-O(x_{R})\leq 0\text{ if }\alpha\geq\alpha_{C},$ (149)

where $\alpha_{L},\alpha_{R}$ and $\alpha_{C}$ are given in (34), (36), and
(35) respectively.

Furthermore, the following are true and help determine from which row in Table
1 to read $x$. If $\psi_{2}<\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{L}<\alpha_{C}<\alpha_{R}$. If
$\psi_{2}>\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{R}<\alpha_{C}<\alpha_{L}$. If
$\psi_{2}=\min\\{2\psi_{1},\psi_{1}+1\\}$, then
$\alpha_{L}=\alpha_{C}=\alpha_{R}$. In particular, it follows from this that
if $\psi_{2}>\beta_{1}$, then $\alpha_{R}\leq\alpha_{C}\leq\alpha_{L}$ and if
$\psi_{2}\leq\beta_{3}$, then $\alpha_{L}\leq\alpha_{C}\leq\alpha_{R}$.

Finally, $\alpha_{L},\alpha_{R},\alpha_{C}\in[0,1]$.

###### Proof.

The derivation of (142)-(146) follows from direct substitution of $x=x_{L}$ or
$x=x_{R}$ into $O$ (for which we have expressions from Theorems 1 and 4) or
its derivative (in eq. (140)-(141)). The derivation of (147)-(149) follows
from the observation that the equations (142)-(146) are linear in $\alpha$,
and hence we can easily compute the values of $\alpha$ at which these
expressions change from a negative value to a positive value. Once an explicit
formula for $\alpha_{L},\alpha_{C}$, and $\alpha_{R}$ is obtained, it is easy
to find the criteria to decide their relative magnitude by comparing the term
under parenthesis in the denominators of (34), (36), and (35). It is also easy
to see from their formulas that their value is always in the range $[0,1]$. ∎

To finish the proof of Theorem 9 we consider the different scenarios in Table
1. In what follows, statements about the concavity of $O$ are justified via
the explanation accompanying Figure 3, and statements about the slope of $O$
are based on Lemma D.13.

Case 1.1,
$\psi_{1}<\psi_{2}\land\psi_{2}\geq\beta_{1}\land\alpha<\alpha_{L}\land\alpha<\alpha_{R}$:
The function $O$ is convex and is decreasing at $x=x_{L}$ and decreasing at
$x=x_{R}$. Hence its minimum is at $x=x_{R}$.

Case 1.2,
$\psi_{1}<\psi_{2}\land\psi_{2}\geq\beta_{1}\land\alpha_{R}<\alpha<\alpha_{L}$:
The function $O$ is convex and it is decreasing at $x=x_{L}$ and increasing at
$x=x_{R}$. Hence it has a unique minimizer at $x=x_{1}$.

When $\psi_{1}<\psi_{2}\land\psi_{2}\geq\beta_{1}$, we can use Lemma D.13 and
a tedious calculation (omitted) to show that $\alpha_{R}<\alpha_{L}$, which is
why the 4th and 5th rows of the first column of Table 1 are empty.

Case 1.3,
$\psi_{1}<\psi_{2}\land\psi_{2}\geq\beta_{1}\land\alpha>\alpha_{L}\land\alpha>\alpha_{R}$:
The function $O$ is convex and it is increasing at both $x=x_{L}$ and
$x=x_{R}$. Hence its minimum is at $x=x_{L}$.

Case 1.4,
$\psi_{1}<\psi_{2}\land\beta_{2}<\psi_{2}<\beta_{1}\land\alpha<\alpha_{L}\land\alpha<\alpha_{R}$:
The function $O$ is first convex, then concave and then convex. It is
decreasing at both $x=x_{L}$ and at $x=x_{R}$. Hence $O$ has at most one local
minimum in the interior of the domain, at $x=x_{1}$ if it exists, and a local
minimum at $x_{R}$, the domain being $[x_{L},x_{R}]$. Therefore, the minimum
can be expressed as $x_{R}\sqcup x_{1}$.

Case 1.5,
$\psi_{1}<\psi_{2}\land\beta_{2}<\psi_{2}<\beta_{1}\land\alpha_{R}<\alpha<\alpha_{L}$:
The function $O$ is first convex, then concave and then convex. It is
decreasing at $x=x_{L}$ and increasing at $x=x_{R}$. Hence $O$ has no local
minimum at the end points of the domain $[x_{L},x_{R}]$. Also, $O$ either has
exactly one local minimum in the interior of the domain, at $x=x_{1}$, or, $O$
has exactly two local minimum and one local maximum (resp.) in the interior of
the domain, at $x=x_{1}$, $x=x_{3}$ and $x=x_{2}$ respectively. Therefore, the
minimum can be expressed as $x_{1}\sqcup x_{3}$. Note that $x_{1}$ always
exists but $x_{3}$ might not.

Case 1.6,
$\psi_{1}<\psi_{2}\land\beta_{2}<\psi_{2}<\beta_{1}\land\alpha_{L}<\alpha<\alpha_{R}$:
The function $O$ is first convex, then concave and then convex. It is
increasing at $x=x_{L}$ and decreasing at $x=x_{R}$. Hence, the minimum is
either at the $x_{L}$ or $x_{R}$, depending whether $\alpha>\alpha_{C}$ or
$\alpha<\alpha_{C}$. This case is an example where it is clear that allowing
$\alpha=\alpha_{C}$ leads to non-uniqueness in the optimal $x$. See Remark 9.

Case 1.7,
$\psi_{1}<\psi_{2}\land\beta_{2}<\psi_{2}<\beta_{1}\land\alpha>\alpha_{L}\land\alpha>\alpha_{R}$:
The function $O$ is first convex, then concave, and then convex. It is
increasing at both $x=x_{R}$ and $x=x_{L}$. Hence $O$ either has no critical
point in the interior of the domain, or it has two critical point in the
interior of the domain, at $x=x_{1}$ (local maximum) and $x=x_{2}$ (local
minimum) if they exists, and always has a local minimum at $x_{L}$. Therefore,
the minimum can be expressed as $x_{L}\sqcup x_{2}$. Note that $x_{L}$ always
exists, but $x_{2}$ not always exists.

Case 1.8,
$\psi_{1}<\psi_{2}\land\beta_{3}<\psi_{2}\leq\beta_{2}\land\alpha<\alpha_{L}\land\alpha<\alpha_{R}$:
The function $O$ is first concave and convex. and is decreasing at both
$x=x_{L}$ and $x=x_{R}$. Hence its minimum is at $x=x_{R}$.

Case 1.9,
$\psi_{1}<\psi_{2}\land\beta_{3}<\psi_{2}\leq\beta_{2}\land\alpha_{R}<\alpha<\alpha_{L}$:
The function $O$ is first concave and convex. It is decreasing at $x=x_{L}$
and increasing at $x=x_{R}$. Hence, $O$ has exactly one local minimum in the
interior of the domain, which is at $x=x_{1}$ and always exists, and which is
also a global minimum.

Case 1.10,
$\psi_{1}<\psi_{2}\land\beta_{3}<\psi_{2}\leq\beta_{2}\land\alpha_{L}<\alpha<\alpha_{R}$:
The function $O$ is first concave and convex. It is increasing at $x=x_{L}$
and decreasing at $x=x_{R}$. Hence, the minimum is either at the $x_{L}$ or
$x_{R}$, depending whether $\alpha>\alpha_{C}$ or $\alpha<\alpha_{C}$.

Case 1.11,
$\psi_{1}<\psi_{2}\land\beta_{3}<\psi_{2}\leq\beta_{2}\land\alpha>\alpha_{L}\land\alpha>\alpha_{R}$:
The function $O$ is first concave and convex. It is increasing at both
$x=x_{L}$ and $x=x_{R}$. Hence $O$ either has no critical point in the
interior of the domain or it has two critical point in the interior of the
domain, at $x=x_{1}$ (local maximum) and $x=x_{2}$ (local minimum) if they
exists, and always has a local minimum at $x_{L}$. Therefore, the minimum can
be expressed as $x_{L}\sqcup x_{2}$. Note that $x_{L}$ always exists, but
$x_{2}$ not always exists.

Case 1.12,
$\psi_{1}<\psi_{2}\land\psi_{2}\leq\beta_{3}\land\alpha<\alpha_{L}\land\alpha<\alpha_{R}$:
The function $O$ is concave. It is decreasing at both $x=x_{L}$ and $x=x_{R}$.
Hence, its minimum is at $x=x_{R}$

When $\psi_{1}<\psi_{2}\land\psi_{2}\geq\beta_{1}$, Lemma D.13 and a tedious
calculation (omitted) shows that that $\alpha_{L}<\alpha_{R}$, which is why
the second and third rows of the last column of Table 1 are empty.

Case 1.13,
$\psi_{1}<\psi_{2}\land\psi_{2}\leq\beta_{3}\land\alpha_{L}<\alpha<\alpha_{R}$:
The function $O$ is concave. It is increasing at $x=x_{L}$ and decreasing at
$x=x_{R}$. Hence, the minimum is either at the $x_{L}$ or $x_{R}$, depending
whether $\alpha>\alpha_{C}$ or $\alpha<\alpha_{C}$.

Case 1.14,
$\psi_{1}<\psi_{2}\land\psi_{2}\leq\beta_{3}\land\alpha>\alpha_{L}\land\alpha>\alpha_{R}$:
The function $O$ is concave. It is increasing at $x=x_{L}$ and at $x=x_{R}$.
Hence, the minimum is at $x_{L}$.

Case 2, $\psi_{1}>\psi_{2}$:

We will deal with the case $\psi_{2}=1$ at the end. For now, we assume that
$\psi_{2}\neq 1$.

We first prove that the second derivative of $O$ is convex, just like in the
case when $\psi_{1}<\psi_{2}$. A direct computation yields

$\displaystyle\frac{{\rm d}^{2}O}{{\rm d}x^{2}}=\tilde{O}(x)+C,\text{ where
}C=-\frac{2F_{1}^{2}\alpha}{\psi_{1}-\psi_{2}}$ (150)

and
$\tilde{O}(x)=-\frac{2\psi_{2}\left(3x^{2}\left((F_{\star}^{2}+\tau^{2})-F_{1}^{2}(\psi_{2}-1)\right)-2x^{3}F_{1}^{2}+6x(F_{\star}^{2}+\tau^{2})\psi_{2}+\psi_{2}\left(F_{1}^{2}(\psi_{2}-1)^{2}+(F_{\star}^{2}+\tau^{2})(3\psi_{2}+1)\right)\right)}{\left((x+\psi_{2})^{2}-\psi_{2}\right)^{3}}$.

A tedious calculation (omitted) shows that $\frac{{\rm d}^{2}\tilde{O}}{{\rm
d}x^{2}}\geq 0$ for $x\in[x_{L},x_{R}]$ (i.e. it is convex), that
$\tilde{O}(x_{L})<\tilde{O}(x_{R})$, and that $\frac{{\rm d}\tilde{O}}{{\rm
d}x}(x=x_{L})<0$. To do this calculation, we recommend the following. First
break $\tilde{O}$ into two components. One component proportional to
$F^{2}_{1}$, called $\tilde{O}_{1}$, and one component proportional to
$(F_{\star}^{2}+\tau^{2})$, called $\tilde{O}_{\star}$. Then, show that both
components are convex, that
$\tilde{O}_{\star}(x_{L})\leq\tilde{O}_{\star}(x_{R})$, that
$\tilde{O}_{1}(x_{L})<\tilde{O}_{1}(x_{R})$, that $\frac{{\rm
d}\tilde{O}_{\star}}{{\rm d}x}(x=x_{L})=0$, and that $\frac{{\rm
d}\tilde{O}_{1}}{{\rm d}x}(x=x_{L})<0$.

From this it follows that, depending on the value of $\psi_{1}$, the concavity
$\frac{{\rm d}^{2}O}{{\rm d}x^{2}}$ is positive or negative. The situation is
exactly the same as in the Figure 3 but now the $x$ axis is $\psi_{1}$, the
points $A_{1}$, $A_{2}$ and $A_{3}$ are different, and so are the definitions
of $\beta_{1},\beta_{2}$ and $\beta_{3}$. We do however have that, by
definition, $0\leq\beta_{3}<\beta_{2}<\beta_{1}$.

With a slight abuse of notation we refer to the $y$-coordinate value of this
points by $A_{1},A_{2}$, and $A_{3}$. We now have
$A_{1}=\frac{2\left(F_{1}^{2}(\psi_{2}+1)+(F_{\star}^{2}+\tau^{2})\right)}{\psi_{2}}$
and
$A_{3}=-\frac{2\max\left\\{1,\psi_{2}\right\\}\left(F_{1}^{2}(\psi_{2}-1)^{2}+(F_{\star}^{2}+\tau^{2})(3\min\left\\{1,\psi_{2}\right\\}+\max\left\\{1,\psi_{2}\right\\})\right)}{(\psi_{2}-1)^{3}\min\left\\{1,\psi_{2}\right\\}}$.
Although we are not considering yet the case when $\psi_{2}=1$, notice that
when $\psi_{2}=1$ we have that $A_{3}=+\infty$, and in fact we also have
$O(x_{R})=+\infty$. Getting $A_{2}$ is a bit more complicated. From the first
order condition $\frac{{\rm d}\tilde{O}}{{\rm d}x}=0$ and the convexity of
$\tilde{O}$ – recall that $\tilde{O}$ a rational function – we can extract
that the $x$-coordinate of $A_{2}$ is the unique root of a 4th degree
polynomial $h(x)=h_{0}+h_{1}x+h_{2}x^{2}+h_{3}x^{3}+h_{4}x^{4}$ in the range
$x\in[x_{L},x_{R}]$, where
$h_{0}=\psi_{2}^{2}\left(\rho(\psi_{2}-1)^{2}+2(\psi_{2}+1)\right)$,
$h_{1}=2\psi_{2}\left(\rho(\psi_{2}-1)^{2}+(3\psi_{2}+1)\right)$,
$h_{2}=6\psi_{2}$, $h_{3}=2\left(1-\rho(\psi_{2}-1)\right)$, $h_{4}=-\rho$,
Call this root $x_{A_{2}}$. We then have $A_{2}=\tilde{O}(x_{A_{2}})$. It
turns out that we can write $A_{2}$ directly as the solution of
$g(A_{2}/((\tau^{2}+F^{2}_{\star})\psi_{2}))=0$, where
$g(x)=g_{0}+g_{1}x+g_{2}x^{2}+g_{3}x^{3}+g_{4}x^{4}$ with $g_{0}=\rho^{4}$,
$g_{1}=-4\left(\rho\psi_{2}+\rho+1\right)\left(\rho\psi_{2}\left(2\rho\psi_{2}-3\rho+4\right)+2(\rho+1)^{2}\right)$,
$g_{2}=-4\psi_{2}^{2}\left(\rho\psi_{2}\left(7\rho\psi_{2}-20\rho+14\right)+7(\rho+1)^{2}\right)$,
$g_{3}=-16\psi_{2}^{4}\left(\rho\psi_{2}+\rho+1\right)$, and
$g_{4}=16\psi_{2}^{6}$.

Using the expressions for $A_{1}$ and $A_{3}$, and the expression for $C$, see
(150), $\beta_{2}$ and $\beta_{3}$ are defined as
$\beta_{2}=\psi_{1}:A_{1}(\psi_{1})+C(\psi_{1})=0$, and
$\beta_{3}=\psi_{1}:A_{3}(\psi_{1})+C(\psi_{1})=0$. By definition of
$\beta_{2}$ and $\beta_{3}$, these equations have a unique solution when
$\psi_{1}>\psi_{2}>0$ and their expressions are given in (38) and (39)
respectively. We can also define $\beta_{1}$ as the solution of
$\beta_{1}=\psi_{1}:A_{2}(\psi_{1})+C(\psi_{1})=0$. By definition of
$\beta_{1}$, the solution is unique in the range $\psi_{1}>\psi_{2}>0$. We can
use the fact that $A_{2}=\tilde{O}(x_{A_{2}})$ to write that
$\beta_{1}=\psi_{2}+\frac{2F^{2}_{1}\alpha}{\tilde{O}(x_{A_{2}})}$. We can
also use the fact that $g(A_{2}/((\tau^{2}+F^{2}_{\star})\psi_{2}))=0$, which
implies that $g(-C(\psi_{1})\;/\;((\tau^{2}+F^{2}_{\star})\psi_{2}))=0$ when
$\psi_{1}=\beta_{1}$, to write $\beta_{1}$ as the root of a 4th degree
polynomial $r(x)$ that is specified in Appendix B.4, which is the way in which
we decide to state Theorem 9.

Now that we have characterized the curvature of $O(x)$, we are ready to locate
its global minimum. To do so, we will use the curvature of $O(x)$ together
with the first-order optimally condition $\frac{{\rm d}O}{{\rm d}x}=0$, and
the following three extra pieces of information: the sign of the derivative of
$O(x)$ at $x=x_{L}$; the sign of the derivative of $O(x)$ at $x=x_{R}$; and
the sign of $O(x_{L})-O(x_{R})$.

A direct computation yields

$\displaystyle\frac{{\rm d}O}{{\rm d}x}$
$\displaystyle=\frac{p_{0}+p_{1}x+p_{2}x^{2}+p_{3}x^{3}+p_{4}x^{4}+p_{5}x^{5}}{(\psi_{2}-(x+\psi_{2})^{2})^{2}(\psi_{1}-\psi_{2})},$
(151)

where the coefficients $p_{0},\dots,p_{5}$ are, apart from a multiplying
constant, given in (71)- (76). The roots of the denominator, i.e.
$-\sqrt{\psi_{2}}-\psi_{2}$ and $\sqrt{\psi_{2}}-\psi_{2}$, are not a root of
the numerator, therefore, the first order optimalit conditions are $p(x)=0$.

Not all solutions of $p(x)=0$ minimize $O(x)$. To locate the minimizer, we use
the sign of the derivative of $O(x)$ at $x=x_{L}$; the sign of the derivative
of $O(x)$ at $x=x_{R}$; and the sign of $O(x_{L})-O(x_{R})$. These signs can
be determined using Lemma D.14. Lemma D.14 also proves Remark 10.

######  Lemma D.14.

If $\psi_{1}>\psi_{2}$ then the following relationships hold.

$\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{L}}=\frac{\alpha\left(F_{\star}^{2}{+}\tau^{2}\right){+}F_{1}^{2}\left(\alpha\left(-\psi_{2}\right){+}2(\alpha{-}1)\psi_{1}{+}\alpha{+}\psi_{2}\right)}{\psi_{1}{-}\psi_{2}}.$
(152) $\displaystyle\text{If }\psi_{2}=1\text{ then }\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}=+\infty.$ (153) $\displaystyle\text{If
}\psi_{2}<1\text{ then }$ (154) $\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}=\frac{\left(\alpha\psi_{2}^{2}{-}2(\alpha{+}1)\psi_{2}{+}\alpha{+}2\psi_{1}\right)\left(F_{\star}^{2}{+}\tau^{2}\right){+}F_{1}^{2}\left(\psi_{2}{-}1\right){}^{2}\left(2\alpha\psi_{1}{-}(3\alpha{+}1)\psi_{2}{+}\alpha\right)}{\left(\psi_{1}{-}\psi_{2}\right)\left(\psi_{2}{-}1\right){}^{2}}.$
(155) $\displaystyle\text{If }\psi_{2}>1\text{ then }$ (156)
$\displaystyle\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{R}}=\frac{F_{1}^{2}\left(\psi_{2}{-}1\right){}^{2}\left(-2\alpha\psi_{1}{+}(\alpha{+}1)\psi_{2}{+}\alpha\right){-}\left(\psi_{2}\left((\alpha{-}2)\psi_{2}{-}2\alpha{+}2\psi_{1}\right){+}\alpha\right)\left(F_{\star}^{2}{+}\tau^{2}\right)}{\left(\psi_{2}{-}1\right){}^{2}\left(\psi_{2}{-}\psi_{1}\right)}.$
(157) $\displaystyle\text{If }\psi_{2}=1\text{ then
}O(x_{L})-O(x_{R})=-\infty.$ (158) $\displaystyle\text{If }\psi_{2}<1\text{
then }$ (159) $\displaystyle
O(x_{L})-O(x_{R})=\frac{\psi_{2}\left(F_{1}^{2}\left(\psi_{2}{-}1\right)\left((1{-}2\alpha)\psi_{1}{+}\alpha\left(2\psi_{2}{-}1\right)\right){-}\left((\alpha{+}1)\psi_{2}{-}\alpha{-}\psi_{1}\right)\left(F_{\star}^{2}{+}\tau^{2}\right)\right)}{\left(\psi_{1}{-}\psi_{2}\right)\left(\psi_{2}{-}1\right)}.$
(160) $\displaystyle\text{If }\psi_{2}>1\text{ then }$ (161) $\displaystyle
O(x_{L})-O(x_{R})=$ (162)
$\displaystyle\frac{\psi_{2}\left((\alpha{-}1)\left(F_{\star}^{2}{+}\tau^{2}\right){+}\alpha
F_{1}^{2}\right){+}\psi_{1}\left((2\alpha{-}1)F_{1}^{2}\left(\psi_{2}{-}1\right){+}F_{\star}^{2}{+}\tau^{2}\right){-}\alpha\left(F_{\star}^{2}{+}\tau^{2}\right){-}\alpha
F_{1}^{2}\psi_{2}^{2}}{\left(\psi_{2}{-}1\right)\left(\psi_{2}{-}\psi_{1}\right)}.$
(163)

Also, the following is true.

$\displaystyle\text{If }\alpha<\alpha_{L}\text{ then }\frac{{\rm d}O}{{\rm
d}x}\bigg{\rvert}_{x=x_{L}}<0\text{; and if }\alpha\geq\alpha_{L}\text{ then
}\frac{{\rm d}O}{{\rm d}x}\bigg{\rvert}_{x=x_{L}}\geq 0.$ (164)
$\displaystyle\text{Let }\psi_{2}\neq 1.\text{ If }E_{1}\text{ is true then
}\frac{{\rm d}O}{{\rm d}x}\bigg{\rvert}_{x=x_{R}}<0\text{; and if }E_{2}\text{
is true then }\frac{{\rm d}O}{{\rm d}x}\bigg{\rvert}_{x=x_{R}}>0.$ (165)

Conditions $E_{1}$ and $E_{2}$ are defined in Theorem 9.

$\displaystyle\text{Let }\psi_{2}\neq 1.\text{ If }\alpha<\alpha_{C}\text{
then }O(x_{L})-O(x_{R})>0\text{; and if }\alpha\geq\alpha_{C}\text{ then
}O(x_{L})-O(x_{R})\leq 0$ (166)

Constant $\alpha_{C}$ is defined in equation (41) in Theorem 9.

Furthermore, we always have that $\alpha_{L}\in[0,1]$. It also holds that
$\alpha_{C}\in[0,1]$ if and only if
$\psi_{1}\leq\psi_{2}/(1-\rho|1-\psi_{2}|)$. We have that $\alpha_{R}\in[0,1]$
if and only if

$\displaystyle\psi_{1}\geq\psi_{2}+\frac{1}{2}\rho(\psi_{2}-1)^{2}\min\\{\psi_{2},1\\}\text{
and }$ (167)
$\displaystyle\psi_{1}\leq\frac{-1+\rho(\psi_{2}-1)^{2}(2\psi_{2}+\min\\{2\psi_{2}-1,1\\})+\psi_{2}(2+\max\\{\psi_{2},2-\psi_{2}\\})}{2(\max\\{\psi_{2},1\\}+\rho(\psi_{2}-1)^{2})}.$
(168)

Finally, $A\geq B$ always, where $A$ and $B$ are given in (43) and (44).
Furthermore, $B<\psi_{2}$ if and only if
$1/\rho>\gamma\triangleq\min\\{1,\max\\{0,2\psi_{2}-1\\}\\}$.

###### Proof.

Except (158) and (153), the derivation of (152)-(163) follows from direct
substitution of $x=x_{L}$ or $x=x_{R}$ into $O$ – for which we have
expressions from Theorems 1 and 4 – or its derivative in equation 151.

Equation (158), for when $\psi_{2}=1$, is obtained by taking the limit of
(160) and (163) as $\psi_{2}\uparrow 1$ and $\psi_{2}\downarrow 1$
respectively. Equation (153) for when $\psi_{2}=1$ is obtained by taking the
limit of (155) and (157) as $\psi_{2}\uparrow 1$ and $\psi_{2}\downarrow 1$
respectively.

The derivation of condition (164) follows from the observation that the
equation (152) is linear in $\alpha$ and is always negative for $\alpha=0$ and
positive for $\alpha=1$. Hence, we can easily compute a value
$\alpha_{L}\in[0,1]$ at which the expression changes from a negative value to
a positive value. The expression for $\alpha_{L}$ we obtain is (40).

The derivation of condition (165) can be obtained through following the
observations. First notice that $\frac{{\rm d}O}{{\rm
d}x}\big{\rvert}_{x=x_{R}}$ is a linear increasing function of $\alpha$. Now
focus on the following three implications.

  1. 1.

If at $\alpha=1$ we have $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}<0$,
then $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}<0$ for all
$\alpha\in[0,1]$;

  2. 2.

If at $\alpha=0$ we have $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}>0$,
then $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}>0$ for all
$\alpha\in[0,1]$;

  3. 3.

If at $\alpha=0$ we have $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}<0$
and at $\alpha=1$ we have $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}>0$,
then there exists $\alpha_{R}\in[0,1]$ such that
$(\alpha<\alpha_{R}\Rightarrow\frac{{\rm d}O}{{\rm
d}x}\big{\rvert}_{x=x_{R}}<0)$ and $(\alpha>\alpha_{R}\Rightarrow\frac{{\rm
d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}>0)$;

A direct computation shows that the sufficient condition in the first
implication holds if and only if $\psi_{1}<B$, where $B$ is given in (44); the
sufficient condition in the second implication holds if and only if
$\psi_{1}>A$, where $A$ is given in (43); the sufficient condition in the the
third implication holds if and only if $B<\psi_{1}<A$. Therefore, if