all $x$. For every neuron $\nu\in N_{\ell}\backslash T$, since the outgoing
weights are unchanged, we get for every $x$

$y_{\nu}(\theta^{\prime},x)\boldsymbol{w}^{\prime}_{\nu\to\bullet}=y_{\nu}(\theta,x)\boldsymbol{w}_{\nu\to\bullet}.$

Moreover, since $y_{\nu_{2}}(\theta,x)=\lambda y_{\nu_{1}}(\theta,x)$ and
$\boldsymbol{y}_{\ell}(\theta,x)=\boldsymbol{y}_{\ell}(\theta^{\prime},x)$, we
obtain for every $x$

$\displaystyle
y_{\nu_{1}}(\theta^{\prime},x)\boldsymbol{w}^{\prime}_{\nu_{1}\to\bullet}+y_{\nu_{2}}(\theta^{\prime},x)\boldsymbol{w}^{\prime}_{\nu_{2}\to\bullet}$
$\displaystyle=y_{\nu_{1}}(\theta,x)\boldsymbol{v}^{\prime}_{1}+y_{\nu_{2}}(\theta,x)\boldsymbol{v}^{\prime}_{2}$
$\displaystyle=y_{\nu_{1}}(\theta,x)(\boldsymbol{v}_{1}+\lambda\epsilon\mathtt{1}_{N_{\ell+1}})+\lambda
y_{\nu_{1}}(\theta,x)(\boldsymbol{v}_{2}-\epsilon\mathtt{1}_{N_{\ell+1}})$
$\displaystyle=y_{\nu_{1}}(\theta,x)\boldsymbol{v}_{1}+\lambda
y_{\nu_{1}}(\theta,x)\boldsymbol{v}_{2}$
$\displaystyle=y_{\nu_{1}}(\theta,x)\boldsymbol{w}_{\nu_{1}\to\bullet}+y_{\nu_{2}}(\theta,x)\boldsymbol{w}_{\nu_{2}\to\bullet}.$

Summing over all hidden neurons we obtain
$\boldsymbol{z}_{\ell+1}(\theta,x)=\boldsymbol{z}_{\ell+1}(\theta^{\prime},x)$
for every $x$, and since all the next affine layers are unchanged, we obtain
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta}$. It is not
difficult to check that $\theta^{\prime}=\theta^{\prime}(\epsilon)$ is not
scaling equivalent to $\theta$ and can be made arbitrarily close to it. This
shows that $\theta$ is not locally S-identifiable from $\mathbb{R}^{N_{0}}$.
By contraposition, if $\theta$ is locally S-identifiable from
$\mathbb{R}^{N_{0}}$ then it has no positive twins.

In the case of negative twins we have $y_{\nu_{1}}(\theta,x)=\mathtt{ReLU}(t)$
and $y_{\nu_{2}}(\theta,x)=|\lambda|\mathtt{ReLU}(-t)$ with
$t=t(x):=z_{\nu_{1}}(\theta,x)=\langle\boldsymbol{w}_{1},y_{\ell-1}(\theta,x)\rangle+b_{1}$.
Since $\mathcal{X}$ is bounded there is some finite $M>0$ such that
$|z_{\nu_{1}}(\theta,x)|\leq M$ for every $x\in\mathcal{X}$. Consider
$\theta^{\prime}$ obtained by keeping all weights and biases unchanged from
$\theta$ except the incoming and outgoing weights of $\nu_{1},\nu_{2}$, their
biases, and the biases of the neurons of the next layer, $\eta\in N_{\ell+1}$,
which are set as:

  * •

$\boldsymbol{w}^{\prime}_{\bullet\to\nu_{1}}=\boldsymbol{w}^{\prime}_{\bullet\to\nu_{2}}=\boldsymbol{w}_{\bullet\to\nu_{1}}$;

  * •

$b^{\prime}_{\nu_{1}}=b_{\nu_{1}}$; $b^{\prime}_{\nu_{2}}=b_{\nu_{1}}+M$;

  * •

$\boldsymbol{w}^{\prime}_{\nu_{1}\to\bullet}=\boldsymbol{w}_{\nu_{1}\to\bullet}+|\lambda|\boldsymbol{w}_{\nu_{2}\to\bullet}$;
$\boldsymbol{w}^{\prime}_{\nu_{2}\to\bullet}=-|\lambda|\boldsymbol{w}_{\nu_{2}\to\bullet}$;

  * •

$b^{\prime}_{\eta}=b_{\eta}+M$

For each $\eta\in N_{\ell+1}$, using Fact 4 for $\alpha=w_{\nu_{1}\to\eta}$,
$\beta=|\lambda|w_{\nu_{2}\to\eta}$ we obtain

$\displaystyle
y_{\nu_{1}}(\theta,x)w_{\nu_{1}\to\eta}+y_{\nu_{2}}(\theta,x)w_{\nu_{2}\to\eta}$
$\displaystyle=\alpha\mathtt{ReLU}(t)+\beta\mathtt{ReLU}(-t)$
$\displaystyle=(\alpha+\beta)\mathtt{ReLU}(t)-\beta\mathtt{ReLU}(t+M)+M$
$\displaystyle=(w_{\nu_{1}\to\eta}+|\lambda|w_{\nu_{2}\to\eta})\mathtt{ReLU}(t)-|\lambda|w_{\nu_{2}\to\eta}\mathtt{ReLU}(t+M)+M$
$\displaystyle=w^{\prime}_{\nu_{1}\to\eta}\mathtt{ReLU}(t)+w^{\prime}_{\nu_{2}\to\eta}\mathtt{ReLU}(t+M)+M$
$\displaystyle=w^{\prime}_{\nu_{1}\to\eta}\mathtt{ReLU}(z_{\nu_{1}}(\theta^{\prime},x))+w^{\prime}_{\nu_{2}\to\eta}\mathtt{ReLU}(z_{\nu_{2}}(\theta^{\prime},x))+M$
$\displaystyle=w^{\prime}_{\nu_{1}\to\eta}y_{\nu_{1}}(\theta^{\prime},x)+w^{\prime}_{\nu_{2}\to\eta}y_{\nu_{2}}(\theta^{\prime},x)+M.$

Reasoning as in the case of positive twins we obtain
$\boldsymbol{z}_{\ell+1}(\theta^{\prime},x)=\boldsymbol{z}_{\ell+1}(\theta,x)$
and eventually
$\boldsymbol{R}_{\theta^{\prime}}(x)=\boldsymbol{R}_{\theta}(x)$ for every $x$
in the bounded set $\mathcal{X}$. Since the sign of
$\boldsymbol{w}_{\nu_{2}\to\bullet}$ has changed, $\theta^{\prime}$ is not PS-
equivalent to $\theta$. This shows that $\theta$ is not PS-identifiable from
$\mathcal{X}$ with respect to $\Theta=\mathbb{R}^{E\cup\bar{H}}$.

By contraposition, assuming that $\theta$ is PS-identifiable from a bounded
set $\mathcal{X}$ with respect to $\Theta=\mathbb{R}^{E\cup\bar{H}}$, there is
no negative twin. Besides, by Theorem 2, such a $\theta$ is also locally
S-identifiable from $\mathcal{X}$ with respect to
$\Theta=\mathbb{R}^{E\cup\bar{H}}$, hence it is locally S-identifiable from
$\mathbb{R}^{N_{0}}$ with respect to $\Theta=\mathbb{R}^{E\cup\bar{H}}$. By
the first part of the lemma, we conclude that $\theta$ has no positive twins
either. Hence, it has no twins.

##  Appendix D Proof of Lemma 4

We will use the following observation.

######  Fact 5.

for $\chi\in\\{0,1\\}$ and $e=(-1)^{\chi}$ we have $\mathtt{ReLU}(t)=\chi
t+e\mathtt{ReLU}(et)$ for every $t\in\mathbb{R}$.

Assume for the sake of contradiction that $\theta$ is not irreducible:
$\boldsymbol{W}_{\ell+1}\boldsymbol{I}_{T}\boldsymbol{W}_{\ell}=0$ for some
non-empty $T\subset N_{\ell}$ with some $1\leq\ell\leq L-1$. Denote
$\theta^{\prime}$ a network with the same weights and biases as $\theta$
except on layers $\ell$ and $\ell+1$, where
$\boldsymbol{W}^{\prime}_{\ell},\boldsymbol{W}^{\prime}_{\ell+1}$ and
$\boldsymbol{b}^{\prime}_{\ell},\boldsymbol{b}^{\prime}_{\ell+1}$ will soon be
described. By an easy induction we have
$\boldsymbol{y}_{\ell^{\prime}}(\theta^{\prime},x)=\boldsymbol{y}_{\ell^{\prime}}(\theta,x)$
for $0\leq\ell^{\prime}\leq\ell-1$.

Defining $\boldsymbol{J}_{T}=\mathtt{diag}(e_{\nu})_{\nu\in N_{\ell}}$ with
$e_{\nu}=-1$ if $\nu\in T$ and $e_{\nu}=1$ otherwise, we obtain from Fact 5
that for every vector $\boldsymbol{z}_{\ell}\in\mathbb{R}^{N_{\ell}}$,
$\mathtt{ReLU}(\boldsymbol{z}_{\ell})=\boldsymbol{I}_{T}\boldsymbol{z}_{\ell}+\boldsymbol{J}_{T}\
\mathtt{ReLU}(\boldsymbol{J}_{T}\boldsymbol{z}_{\ell})$. Define
$\boldsymbol{W}^{\prime}_{\ell}=\boldsymbol{J}_{T}\boldsymbol{W}_{\ell}$,
$\boldsymbol{W}^{\prime}_{\ell+1}=\boldsymbol{W}_{\ell+1}\boldsymbol{J}_{T}$,
$\boldsymbol{b}^{\prime}_{\ell}=\boldsymbol{J}_{T}\boldsymbol{b}_{\ell}$. For
each $x\in\mathbb{R}^{N_{0}}$, since
$\boldsymbol{y}_{\ell-1}(\theta,x)=\boldsymbol{y}_{\ell-1}(\theta^{\prime},x)$
and $\boldsymbol{J}_{T}^{2}=\boldsymbol{Id}_{\mathbb{R}^{N_{\ell}}}$ we get
using the shorthands $\boldsymbol{z}_{i}=\boldsymbol{z}_{i}(\theta,x)$,
$\boldsymbol{z}^{\prime}_{i}=\boldsymbol{z}_{i}(\theta^{\prime},x)$,
$i\in\\{\ell,\ell+1\\}$

$\displaystyle\boldsymbol{z}_{\ell}=$
$\displaystyle\boldsymbol{W}_{\ell}\boldsymbol{y}_{\ell-1}(\theta,x)+\boldsymbol{b}_{\ell}=\boldsymbol{J}_{T}\left(\boldsymbol{J}_{T}\boldsymbol{W}_{\ell}\boldsymbol{y}_{\ell-1}(\theta,x)+\boldsymbol{J}_{T}\boldsymbol{b}_{\ell}\right)=\boldsymbol{J}_{T}\boldsymbol{z}^{\prime}_{\ell}$
$\displaystyle\boldsymbol{z}_{\ell+1}=$ $\displaystyle\boldsymbol{W}_{\ell+1}\
\mathtt{ReLU}(\boldsymbol{z}_{\ell})+\boldsymbol{b}_{\ell+1}=\boldsymbol{W}_{\ell+1}\left(\boldsymbol{I}_{T}\boldsymbol{z}_{\ell}+\boldsymbol{J}_{T}\
\mathtt{ReLU}(\boldsymbol{J}_{T}\boldsymbol{z}_{\ell})\right)+\boldsymbol{b}_{\ell+1}$
$\displaystyle=$
$\displaystyle\underbrace{\boldsymbol{W}_{\ell+1}\boldsymbol{I}_{T}\boldsymbol{W}_{\ell}}_{=0}\boldsymbol{y}_{\ell-1}(\theta,x)+\boldsymbol{W}_{\ell+1}\boldsymbol{I}_{T}\boldsymbol{b}_{\ell}+\boldsymbol{W}^{\prime}_{\ell+1}\
\mathtt{ReLU}(\boldsymbol{z}^{\prime}_{\ell})+\boldsymbol{b}_{\ell+1}$
$\displaystyle=$ $\displaystyle\boldsymbol{W}^{\prime}_{\ell+1}\
\mathtt{ReLU}(\boldsymbol{z}^{\prime}_{\ell})+(\boldsymbol{W}_{\ell+1}\boldsymbol{I}_{T}\boldsymbol{b}_{\ell}+\boldsymbol{b}_{\ell+1}).$

Defining
$\boldsymbol{b}^{\prime}_{\ell+1}:=\boldsymbol{W}_{\ell+1}\boldsymbol{I}_{T}\boldsymbol{b}_{\ell}+\boldsymbol{b}_{\ell+1}$,
we get
$\boldsymbol{z}_{\ell+1}(\theta,x)=\boldsymbol{z}^{\prime}_{\ell+1}(\theta^{\prime},x)$
for all $x$. Since all other layers of $\theta$ and $\theta^{\prime}$ are
identical, an easy induction yields
$\boldsymbol{R}_{\theta}=\boldsymbol{R}_{\theta^{\prime}}$, where
$\theta^{\prime}\in\mathbb{R}^{E\cup\bar{H}}=\Theta$. To conclude, we prove
below that $\theta^{\prime}$ is not PS-equivalent to $\theta$: this
contradicts the assumption that $\theta$ is PS-identifiable and concludes the
proof.

For the sake of (yet another) contradiction, assume that
$\theta^{\prime}\sim_{PS}$, so that there exists diagonal matrices
$\boldsymbol{\Lambda}_{\ell^{\prime}}\in\mathbb{R}^{N_{\ell^{\prime}}\times
N_{\ell^{\prime}}}$ with positive entries and permutation matrices
$\boldsymbol{\Pi}_{\ell^{\prime}}\in\mathbb{R}^{N_{\ell^{\prime}}\times
N_{\ell^{\prime}}}$, $0\leq\ell^{\prime}\leq L$, such that
$\boldsymbol{\Lambda}_{0}=\boldsymbol{\Pi}_{0}=\boldsymbol{I}_{N_{0}}$,
$\boldsymbol{\Lambda}_{L}=\boldsymbol{\Pi}_{L}=\boldsymbol{I}_{N_{L}}$,
$\boldsymbol{W}^{\prime}_{\ell^{\prime}}=\boldsymbol{\Pi}_{\ell^{\prime}}\boldsymbol{\Lambda}_{\ell^{\prime}}\boldsymbol{W}_{\ell^{\prime}}\boldsymbol{\Lambda}^{-1}_{\ell^{\prime}-1}\boldsymbol{\Pi}_{\ell^{\prime}-1}^{-1}$,
and
$\boldsymbol{b}^{\prime}_{\ell^{\prime}}=\boldsymbol{\Pi}_{\ell^{\prime}}\boldsymbol{\Lambda}_{\ell^{\prime}}\boldsymbol{b}_{\ell^{\prime}}$
for every $1\leq\ell^{\prime}\leq L$. We show by induction that
$\boldsymbol{\Lambda}_{\ell^{\prime}}=\boldsymbol{\Pi}_{\ell^{\prime}}=\boldsymbol{I}_{N_{\ell^{\prime}}}$
for every $0\leq\ell^{\prime}<\ell$. This trivially holds for
$\ell^{\prime}=0$. If it holds for some $\ell^{\prime}<\ell-1$ then, as
$(\boldsymbol{W}^{\prime}_{\ell^{\prime}+1},\boldsymbol{b}^{\prime}_{\ell^{\prime}+1})=(\boldsymbol{W}_{\ell^{\prime}+1},\boldsymbol{b}_{\ell^{\prime}+1})$
by construction of $\theta^{\prime}$, we have

$\displaystyle(\boldsymbol{W}_{\ell^{\prime}+1},\boldsymbol{b}_{\ell^{\prime}+1})=(\boldsymbol{W}^{\prime}_{\ell^{\prime}+1},\boldsymbol{b}^{\prime}_{\ell^{\prime}+1})$
$\displaystyle=(\boldsymbol{\Pi}_{\ell^{\prime}+1}\boldsymbol{\Lambda}_{\ell^{\prime}+1}\boldsymbol{W}_{\ell^{\prime}+1}\boldsymbol{\Lambda}_{\ell^{\prime}}^{-1}\boldsymbol{\Pi}_{\ell^{\prime}}^{-1},\boldsymbol{\Pi}_{\ell^{\prime}+1}\boldsymbol{\Lambda}_{\ell^{\prime}+1}\boldsymbol{b}_{\ell^{\prime}+1})$
$\displaystyle=\boldsymbol{\Pi}_{\ell^{\prime}+1}\boldsymbol{\Lambda}_{\ell^{\prime}+1}(\boldsymbol{W}_{\ell^{\prime}+1},\boldsymbol{b}_{\ell^{\prime}+1}),$

i.e.,
$(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})=\lambda_{\pi(\nu)}(\boldsymbol{w}_{\bullet\to\pi(\nu)},b_{\pi(\nu)})$
for every $\nu\in N_{\ell^{\prime}+1}$, with $\pi$ the permutation of
$N_{\ell^{\prime}+1}$ associated to $\boldsymbol{\Pi}_{\ell^{\prime}+1}$ and
$\boldsymbol{\Lambda}_{\ell^{\prime}+1}=\mathtt{diag}(\lambda_{\nu})_{\nu\in
N_{\ell^{\prime}+1}}$. Since $\theta$ has no twin, it follows that $\pi$ is
the identity and $\lambda_{\nu}=1$ for every $\nu\in N_{\ell^{\prime}+1}$,
which concludes the induction. Now, since
$(\boldsymbol{W}^{\prime}_{\ell},\boldsymbol{b}^{\prime}_{\ell})=\boldsymbol{J}_{T}(\boldsymbol{W}_{\ell},\boldsymbol{b}_{\ell})$
by construction of $\theta^{\prime}$, we have

$\displaystyle\boldsymbol{J}_{T}(\boldsymbol{W}_{\ell},\boldsymbol{b}_{\ell})=(\boldsymbol{W}^{\prime}_{\ell},\boldsymbol{b}^{\prime}_{\ell})=(\boldsymbol{\Pi}_{\ell}\boldsymbol{\Lambda}_{\ell}\boldsymbol{W}_{\ell}\boldsymbol{\Lambda}_{\ell-1}^{-1}\boldsymbol{\Pi}_{\ell-1}^{-1},\boldsymbol{\Pi}_{\ell}\boldsymbol{\Lambda}_{\ell}\boldsymbol{b}_{\ell})=\boldsymbol{\Pi}_{\ell}\boldsymbol{\Lambda}_{\ell}(\boldsymbol{W}_{\ell},\boldsymbol{b}_{\ell}).$

As a result, for each $\nu\in T\neq\emptyset$ we have
$-(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})=\lambda_{\pi(\nu)}(\boldsymbol{w}_{\bullet\to\pi(\nu)},b_{\pi(\nu)})$
where $\pi$ is the permutation of $N_{\ell}$ associated to
$\boldsymbol{\Pi}_{\ell}$ and
$\boldsymbol{\Lambda}_{\ell}=\mathtt{diag}(\lambda_{\nu})_{\nu\in N_{\ell}}$.
However, since $\theta$ has no twin,
$(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})$ is not collinear to any
$(\boldsymbol{w}_{\bullet\to\nu^{\prime}},b_{\nu^{\prime}})$, $\nu^{\prime}\in
N_{\ell}$, $\nu^{\prime}\neq\nu$, hence $\pi(\nu)=\nu$. It follows the
$-(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})=\lambda_{\nu}(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})$,
and as $\lambda_{\nu}>0$ we obtain
$(\boldsymbol{w}_{\bullet\to\nu},b_{\nu})=0$, therefore $\theta$ is not
admissible. However, by Lemma 2, since $\theta$ is PS-identifiable with
respect to $\Theta=\mathbb{R}^{E\cup\bar{H}}$, it is admissible. Hence the
desired contradiction.

##  Appendix E Proof of Lemma 9 and Lemma 10

###### Proof of Lemma 9.

The proof is by induction on $L$. For $L=1$, since $\boldsymbol{I}_{0}$ is the
identity

$\displaystyle\boldsymbol{R}_{\theta}(x)=\boldsymbol{z}_{1}(\theta,x)=\boldsymbol{W}_{1}x+\boldsymbol{b}_{1}=\boldsymbol{W}_{1}\boldsymbol{I}_{0}x+\boldsymbol{b}_{1}.$

With the convention that a product of matrices over an empty index set is the
identity, this establishes (23) for $L=1$. Now, assuming that (23) holds for
every network of depth $L$, let us prove it for $\theta$ of depth $L+1$. For
this, observe that $\boldsymbol{z}_{L}(\theta,x)$ is the realization of a
network $\underline{\theta}$ of depth $L$ made of the first $L$ affine layers
of $\theta$, hence by the induction hypothesis we can use (23) to get

$\boldsymbol{z}_{L}(\theta,x)=\boldsymbol{R}_{\underline{\theta}}(x)=\left(\Pi_{\ell=1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)x+\sum_{\ell^{\prime}=1}^{L-1}\left(\Pi_{\ell=\ell^{\prime}+1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell}\right)\boldsymbol{b}_{\ell^{\prime}}$

Since
$\boldsymbol{y}_{L}(\theta,x)=\boldsymbol{a}_{L}(\theta,x)\odot\boldsymbol{z}_{L}(\theta,x)=\boldsymbol{I}_{L}\boldsymbol{z}_{L}(\theta,x)$
we get

$\displaystyle\boldsymbol{R}_{\theta}(x)$
$\displaystyle=\boldsymbol{z}_{L+1}(\theta,x)=\boldsymbol{W}_{L+1}\boldsymbol{y}_{L}(\theta,x)+\boldsymbol{b}_{L+1}$
$\displaystyle=\boldsymbol{W}_{L+1}\boldsymbol{I}_{L}\left(\left(\Pi_{\ell=1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)x+\sum_{\ell^{\prime}=1}^{L}\left(\Pi_{\ell=\ell^{\prime}+1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)\boldsymbol{b}_{\ell^{\prime}}\right)+\boldsymbol{b}_{L+1}.$

To conclude simply observe that

$\displaystyle\boldsymbol{W}_{L+1}\boldsymbol{I}_{L}\left(\Pi_{\ell=1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)$
$\displaystyle=\left(\Pi_{\ell=1}^{L+1}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right),$
$\displaystyle\boldsymbol{W}_{L+1}\boldsymbol{I}_{L}\left(\sum_{\ell^{\prime}=1}^{L}\left(\Pi_{\ell=\ell^{\prime}+1}^{L}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)\boldsymbol{b}_{\ell^{\prime}}\right)$
$\displaystyle=\sum_{\ell^{\prime}=1}^{L}\left(\Pi_{\ell=\ell^{\prime}+1}^{L+1}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)\boldsymbol{b}_{\ell^{\prime}}$
$\displaystyle\text{and that with}\
\ell^{\prime}=L+1,\quad\boldsymbol{b}_{L+1}$
$\displaystyle=\boldsymbol{b}_{\ell^{\prime}}=\left(\Pi_{\ell=\ell^{\prime}+1}^{L+1}\boldsymbol{W}_{\ell}\boldsymbol{I}_{\ell-1}\right)\boldsymbol{b}_{\ell^{\prime}}.\qed$

###### Proof of Lemma 10.

With $\mathcal{P}_{H}:=\cup_{\ell=1}^{L-1}\mathcal{P}_{\ell}$ the set of all
paths from a hidden neuron to an output neuron, for each output neuron
$\eta\in N_{L}$ we prove at the end of this section that

(50) $\displaystyle\boldsymbol{R}_{\theta}(x)_{\eta}$
$\displaystyle=\sum_{p\in\mathcal{P}_{0},p_{L}=\eta}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)x_{p_{0}}+\sum_{p\in\mathcal{P}_{H},p_{L}=\eta}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)+\theta_{\eta}$

Any $p\in\mathcal{P}_{0}$ is uniquely written $p=\mu\to q\to\eta$ with
$\mu=p_{0}$ its input neuron, $\eta\in N_{L}$ its output neuron, and
$q=(p_{1},\ldots,p_{L-1})\in\mathcal{Q}_{1}$ a partial path from the first
layer to the penultimate layer, and $\alpha_{\mu\to
q\to\eta}(\theta,x)=\alpha_{q}(\theta,x)$ for every $q\in\mathcal{Q}_{1}$ and
any $\mu\in N_{0}$,$\eta\in N_{L}$.
Similarly, for $1\leq\ell\leq L-1$, every partial path
$p\in\mathcal{P}_{\ell}$ starting from the $\ell$-th hidden layer and ending
at the output layer can be written as $p=q\to\eta$ where $\eta=p_{L}\in N_{L}$
and $q\in\mathcal{Q}_{\ell}$ starts from the $\ell$-th hidden layer and ends
at the penultimate layer, and we have
$\alpha_{q\to\eta}(\theta,x)=\alpha_{q}(\theta,x)$ for all $\theta,x$.
Therefore, (50) can be rewritten as

$\displaystyle\boldsymbol{R}_{\theta}(x)_{\eta}$
$\displaystyle=\sum_{q\in\mathcal{Q}_{1}}\alpha_{q}(\theta,x)\sum_{\mu\in
N_{0}}{\Phi}_{\mu\to
q\to\eta}(\theta)x_{\mu}+\sum_{q\in\mathcal{Q}}\alpha_{q}(\theta,x){\Phi}_{q\to\eta}(\theta)+\theta_{\eta}$
$\displaystyle=\sum_{q\in\mathcal{Q}_{1}}\alpha_{q}(\theta,x)[\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta)x]_{q}+\sum_{q\in\mathcal{Q}+1}[\bar{\boldsymbol{\alpha}}(\theta,x)]_{q}[\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta)]_{q}$
$\displaystyle=\langle\boldsymbol{Q}\bar{\boldsymbol{\alpha}}(\theta,x),\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta)x\rangle+\langle\bar{\boldsymbol{\alpha}}(\theta,x),\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta)\rangle.$

where we used that
$\bar{\boldsymbol{\alpha}}(\theta,x):=(\boldsymbol{\alpha}(\theta,x),1)$ with
$\boldsymbol{\alpha}(\theta,x):=(\alpha_{q}(\theta,x))_{q\in\mathcal{Q}}$, and
$\boldsymbol{Q}$ is the canonical restriction from $\mathcal{Q}+1$ to
$\mathcal{Q}_{1}$. ∎

###### Proof of Equation (50).

We prove the result by induction on the number of layers $L$.

For $L=1$ we have $\mathcal{P}_{0}=\\{(\mu,\eta)\\}_{\mu\in N_{0},\eta\in
N_{1}}$ and $\mathcal{P}_{1}=\\{(\eta)\\}_{\eta\in N_{1}}$. Since
$H=\emptyset$, $\alpha_{p}(\theta,x)=1$ for all $p\in\mathcal{P}$ and
$\mathcal{P}_{H}=\emptyset$. We have ${\Phi}_{p}(\theta)=w_{\mu\to\eta}$ for
all $p=(\mu,\eta)\in\mathcal{P}_{0}$ and ${\Phi}_{p}(\theta)=b_{\eta}$ for
each $p=(\eta)\in\mathcal{P}_{1}$. It follows that

$\displaystyle\sum_{\begin{subarray}{c}p\in\mathcal{P}_{0}\\\
p_{L}=\eta\end{subarray}}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)x_{p_{0}}+\sum_{\begin{subarray}{c}p\in\mathcal{P}_{H}\\\
p_{L}=\eta\end{subarray}}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)+\theta_{\eta}$
$\displaystyle=\sum_{\begin{subarray}{c}\mu\in
N_{0}\end{subarray}}w_{\mu\to\eta}x_{\mu}+b_{\eta}=\left(\boldsymbol{W}_{1}x+\boldsymbol{b}_{1}\right)_{\eta}=(\boldsymbol{R}_{\theta}(x))_{\eta}.$

This establishes (50) for $L=1$. Assume now that (50) holds for networks of
depth $L\geq 1$. With $\theta$ a network of depth $L+1$, observe that
$\boldsymbol{z}_{L}(\theta,x)=\boldsymbol{R}_{\tilde{\theta}}(x)$ with
$\tilde{\theta}$ the network made of the first $L$ affine layers of $\theta$.
Using the induction hypothesis, we get, for $\nu\in N_{L-1}$,

(51) $\displaystyle z_{\nu}(\theta,x)$
$\displaystyle=\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{0}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}\alpha_{\tilde{p}}(\tilde{\theta},x){\Phi}_{\tilde{p}}(\tilde{\theta})x_{\tilde{p}_{0}}+\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{H}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}\alpha_{\tilde{p}}(\tilde{\theta},x){\Phi}_{\tilde{p}}(\tilde{\theta})$

with $\widetilde{\mathcal{P}}_{0}=\\{(p_{0},\dots,p_{L-1})\mid
p\in\mathcal{P}_{0}\\}$,
$\widetilde{\mathcal{P}_{H}}=\\{(p_{\ell},\dots,p_{L-1})\mid
p=(p_{\ell},\ldots,p_{L})\in\mathcal{P}_{H}\\}$. Since
$\mathtt{ReLU}\left(z_{\nu}(\theta,x)\right)=a_{\nu}(\theta,x)z_{\nu}(\theta,x)$
we get

$\displaystyle(\boldsymbol{R}_{\theta}(x))_{\eta}$ $\displaystyle=\sum_{\nu\in
N_{L-1}}y_{\nu}(\theta,x)w_{\nu\to\eta}+b_{\eta}=\sum_{\nu\in
N_{L-1}}\mathtt{ReLU}\left(z_{\nu}(\theta,x)\right)w_{\nu\to\eta}+b_{\eta}$
$\displaystyle=\sum_{\nu\in
N_{L-1}}a_{\nu}(\theta,x)z_{\nu}(\theta,x)w_{\nu\to\eta}+b_{\eta}$
$\displaystyle=\sum_{\nu\in
N_{L-1}}\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{0}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}a_{\nu}(\theta,x)\alpha_{\tilde{p}}(\tilde{\theta},x){\Phi}_{\tilde{p}}(\tilde{\theta})w_{\nu\to\eta}x_{\tilde{p}_{0}}$
$\displaystyle+\sum_{\nu\in
N_{L-1}}\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{H}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}a_{\nu}(\theta,x)\alpha_{\tilde{p}}(\tilde{\theta},x){\Phi}_{\tilde{p}}(\tilde{\theta})w_{\nu\to\eta}$
$\displaystyle+b_{\eta}$

For each path such that $\tilde{p}_{L-1}=\nu\in N_{L-1}$ we have
$a_{\nu}(\theta,x)\alpha_{\tilde{p}}(\tilde{\theta},x){\Phi}_{\tilde{p}}(\tilde{\theta})w_{\nu\to\eta}=\alpha_{\tilde{p}\to\eta}(\theta,x){\Phi}_{\tilde{p}\to\eta}(\theta)$,
and $p:=\tilde{p}\to\eta$ belongs to $\mathcal{P}_{0}$ (resp. to
$\mathcal{P}_{H}$) if, and only if, $\tilde{p}\in\tilde{\mathcal{P}}_{0}$
(resp. $\tilde{p}\in\tilde{\mathcal{P}}_{H}$). Thus,

$\displaystyle(\boldsymbol{R}_{\theta}(x))_{\eta}$ $\displaystyle=\sum_{\nu\in
N_{L-1}}\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{0}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}\alpha_{\tilde{p}\to\eta}(\theta,x){\Phi}_{\tilde{p}\to\eta}(\theta)x_{\tilde{p}_{0}}+\sum_{\nu\in
N_{L-1}}\sum_{\begin{subarray}{c}\tilde{p}\in\widetilde{\mathcal{P}}_{H}\\\
\tilde{p}_{L-1}=\nu\end{subarray}}\alpha_{\tilde{p}\to\eta}(\theta,x){\Phi}_{\tilde{p}\to\eta}(\theta)+b_{\eta}$
$\displaystyle=\sum_{\begin{subarray}{c}p\in\mathcal{P}_{0}\\\
p_{L}=\eta\end{subarray}}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)x_{p_{0}}+\sum_{\begin{subarray}{c}p\in\mathcal{P}_{H}\\\
p_{L}=\eta\end{subarray}}\alpha_{p}(\theta,x){\Phi}_{p}(\theta)+b_{\eta}.\qed$

##  Appendix F Proof of Lemma 11

The result is proved by induction on the network’s depth. The case $L=1$ is
trivial with the convention that a union over an empty family is empty. For
any depth, since

$\left(\cup_{\nu\in H}\Gamma_{\nu}(\theta)\right)^{c}=\cap_{\nu\in
H}\Gamma_{\nu}^{c}(\theta)=\cap_{\ell=1}^{L-1}\left(\cap_{\nu\in
N_{\ell}}\Gamma_{\nu}^{c}(\theta)\right)=\cap_{\ell=1}^{L-1}\left(\cup_{\nu\in
N_{\ell}}\Gamma_{\nu}(\theta)\right)^{c}$

the result is equivalent to
$\mathcal{X}_{\theta}^{\prime}=\cap_{\ell=1}^{L-1}\left(\cup_{\nu\in
N_{\ell}}\Gamma_{\nu}(\theta)\right)^{c}$, which is the quantity manipulated
in the induction. Assume that the result is valid for all parameters of depth
$L$ and consider $\theta$ a parameter of depth $L+1\geq 2$. Denoting
$\underline{\theta}=g(\theta)$ its restriction to its first $L$ layers, we
will show that
$\mathcal{X}_{\theta}^{\prime}=\mathcal{X}_{\underline{\theta}}^{\prime}\cap\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}(\theta)\right)^{c}$. First we prove
$(\mathcal{X}_{\underline{\theta}}^{\prime})^{c}\cup\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}\right)\subset(\mathcal{X}_{\theta}^{\prime})^{c}$.

  * •

if $x\notin\mathcal{X}_{\underline{\theta}}^{\prime}$ then (by definition of
$\mathcal{X}_{\underline{\theta}}^{\prime}$) the function
$(\underline{\theta}^{\prime},x^{\prime})\mapsto\boldsymbol{a}(\underline{\theta}^{\prime},x^{\prime})$
is not locally constant around $(\underline{\theta},x)$ hence there exists
$1\leq\ell\leq L-1$ and $\nu\in N_{\ell}$ such that
$a_{\nu}(\underline{\theta}^{\prime},x^{\prime})$ is not locally constant
around $(\underline{\theta},x)$. Since $\ell\leq L-1$, for every
$\theta^{\prime},x^{\prime}$ we have
$a_{\nu}(\theta^{\prime},x^{\prime})=a_{\nu}(\underline{\theta^{\prime}},x^{\prime})$
with $\underline{\theta}^{\prime}=g(\theta^{\prime})$ the restriction of
$\theta^{\prime}$ to its first $L$ layers. We obtain that
$a_{\nu}(\theta^{\prime},x^{\prime})$ is not locally constant around
$(\theta,x)$, showing that $x\notin\mathcal{X}_{\theta}^{\prime}$.

  * •

If $x\in\cup_{\nu\in N_{L}}\Gamma_{\nu}(\theta)$, there exists $\nu\in N_{L}$
such that $x\in\Gamma_{\nu}(\theta)$ hence $z_{\nu}(\theta,x)=0$, the gradient
is well-defined, and $\nabla z_{\nu}(\theta,x)\neq 0$. This implies that the
sign of $z_{\nu}(\theta,x^{\prime})$ is not locally constant around $x$, hence
$x^{\prime}\mapsto a_{\nu}(\theta,x^{\prime})$ is not locally constant around
$x$, therefore
$(\theta^{\prime},x^{\prime})\mapsto\boldsymbol{a}(\theta^{\prime},x^{\prime})$
is not locally constant around $(\theta,x)$. Thus,
$x\notin\mathcal{X}_{\theta}^{\prime}$.

This establishes equivalently that
$\mathcal{X}_{\theta}^{\prime}\subset\mathcal{X}_{\underline{\theta}}^{\prime}\cap\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}\right)^{c}$.

Vice-versa, consider
$x\in\mathcal{X}_{\underline{\theta}}^{\prime}\cap\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}(\theta)\right)^{c}$. Since
$x\in\mathcal{X}_{\underline{\theta}}^{\prime}$,
$\boldsymbol{a}(\underline{\theta}^{\prime},x^{\prime})$ is locally constant
around $(\underline{\theta},x)$, hence
$(\theta^{\prime},x^{\prime})\mapsto\boldsymbol{a}_{\ell}(g(\theta^{\prime}),x^{\prime})=\boldsymbol{a}_{\ell}(\theta^{\prime},x^{\prime})$
is locally constant around $(\theta,x)$ for each $1\leq\ell\leq L-1$. There
remains to show that $\boldsymbol{a}_{L}(\theta^{\prime},x^{\prime})$ is
locally constant around $(\theta,x)$. Indeed, since $x\notin\cup_{\nu\in
N_{L}}\Gamma_{\nu}$, we have $z_{\nu}(\theta,x)\neq 0$ for every $\nu\in
N_{L}$. By continuity of
$(\theta^{\prime},x^{\prime})\mapsto\boldsymbol{z}_{L}(\theta^{\prime},x^{\prime})$,
there exists a neighborhood of $(\theta,x)$ on which
$\mathtt{sign}(z_{\nu}(\theta^{\prime},x^{\prime}))$ is constant for every
$\nu\in N_{L}$, hence $\boldsymbol{a}_{L}(\theta^{\prime},x^{\prime})$ is
locally constant around $(\theta,x)$. Overall, we get that
$(\theta,x)\mapsto\boldsymbol{a}(\theta^{\prime},x^{\prime})$ is locally
constant around $(\theta,x)$, i.e. $x\in\mathcal{X}_{\theta}^{\prime}$. This
concludes the proof that
$\mathcal{X}_{\underline{\theta}}^{\prime}\cap\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}(\theta)\right)^{c}\subset\mathcal{X}_{\theta}^{\prime}$,
hence the equality
$\mathcal{X}_{\theta}^{\prime}=\mathcal{X}_{\underline{\theta}}^{\prime}\cap\left(\cup_{\nu\in
N_{L}}\Gamma_{\nu}(\theta)\right)^{c}$.

##  Appendix G Proof of Lemma 13

First we prove that $\mathtt{s}_{c}\in\mathtt{A}(\theta)$ and
$(\mathtt{s}_{c},0)\in\mathtt{\bar{A}}(\theta)$ for each $c$. Since $\theta$
is admissible, one can check (cf Definition 8) that two hidden neurons
$\nu,\nu^{\prime}\in H$ of a shallow network are:

  * •

positive twins if, and only if, $a_{\nu}(\theta,x)=a_{\nu^{\prime}}(\theta,x)$
for all $x\in\mathcal{X}_{\theta}$;

  * •

negative twins if, and only if,
$a_{\nu}(\theta,x)=1-a_{\nu^{\prime}}(\theta,x)$ for all
$x\in\mathcal{X}_{\theta}$;

Since we are on a shallow architecture, we identify
$\mathcal{Q}=\mathcal{Q}_{1}$ with $H$ and $\boldsymbol{a}(\theta,x)$ with
$\boldsymbol{\alpha}(\theta,x)$. Considering the $c$-th equivalence class
$T_{c}$ of twins, it follows that for every $x$ there is
$\epsilon_{c}(x)\in\\{-1,+1\\}$ such that

(52) $\displaystyle
2\boldsymbol{\alpha}_{T_{c}}(\theta,x)=2\boldsymbol{a}_{T_{c}}(\theta,x)=\mathtt{1}_{T_{c}}+\epsilon_{c}(x)\cdot\mathtt{s}_{c}$

where for any $\boldsymbol{u}\in\mathbb{R}^{H}$,
$\boldsymbol{u}_{T}\in\mathbb{R}^{H}$ is its restriction to $T$ (which matches
$\boldsymbol{u}$ on its coordinates indexed by $T$ and is zero elsewhere), and
$\mathtt{1}_{H}\in\mathtt{R}^{H}$ is the vector with all entries equal to one,
while $\mathtt{1}_{T}$ is its restriction to $T$. To continue we use the
following result.

######  Lemma 19.

Consider a shallow network with parameter $\theta$, and $T\subset H$ an
equivalence class of twin neurons. There are
$x_{T}^{+},x_{T}^{-}\in\mathcal{X}_{\theta}$ such that

(53) $|a_{\nu}(\theta,x^{+}_{T})-a_{\nu}(\theta,x^{-}_{T})|=\begin{cases}1,\
\text{if}\ \nu\in T\\\ 0,\ \text{otherwise}\end{cases}$

###### Proof.

For each $\nu\in H$ denote
$\mathcal{V}_{\nu}=\\{x\in\mathbb{R}^{N_{0}}:\langle\boldsymbol{w}_{\bullet\to\nu},x\rangle+b_{\nu}=0\\}$.
Since $\theta$ is admissible, $\boldsymbol{w}_{\bullet\to\nu}\neq 0$ for each
$\nu\in H$, hence these linear spaces are hyperplanes. The hyperplanes
associated to two neurons coincide if, and only if, these neurons are twins.
Choose an arbitrary $\nu\in T$. Since $\mathcal{V}_{\nu}$ is distinct from
each of the (finitely many) $\mathcal{V}_{\nu^{\prime}}$, $\nu^{\prime}\notin
T$, there exists $x_{0}\in\mathcal{V}_{\nu}$ that belongs to the complement of
$\cup_{\nu^{\prime}\notin T}\mathcal{V}_{\nu^{\prime}}$. As this complement is
open, there exists $\epsilon>0$ such that
$B(x_{0},\epsilon\|\boldsymbol{w}_{\bullet\to\nu}\|_{2})$ does not intersect
any of the hyperplanes $\mathcal{V}_{\nu^{\prime}}$, $\nu^{\prime}\notin T$.
Since $x_{T}^{\pm}:=x_{0}\pm\boldsymbol{w}_{\bullet\to\nu}\epsilon/2\in
B(x_{0},\epsilon\|\boldsymbol{w}_{\bullet\to\nu}\|_{2})$ we obtain:
$a_{\nu^{\prime}}(\theta,x^{+}_{T})=a_{\nu^{\prime}}(\theta,x^{-}_{T})$ for
every $\nu^{\prime}\notin T$, and
$\mathtt{sign}(\langle\boldsymbol{w}_{\bullet\to\nu},x_{T}^{\pm}\rangle+b_{\nu})=\pm
1$ hence $a_{\nu}(\theta,x^{+}_{T})=1-a_{\nu}(\theta,x^{-}_{T})$. The latter
extends to each $\nu^{\prime}\in T$ by the twin property, and yields the
conclusion. ∎

By Lemma 19 there are $x^{+}_{c},x^{-}_{c}\in\mathcal{X}_{\theta}$ such that

(54) $|a_{\nu}(\theta,x^{+}_{c})-a_{\nu}(\theta,x^{-}_{c})|=\begin{cases}1,\
\text{if}\ \nu\in T_{c}\\\ 0,\ \text{if}\ \nu\in H\backslash T_{c}\end{cases}$

It follows that
$\boldsymbol{\alpha}(\theta,x^{+}_{c})-\boldsymbol{\alpha}(\theta,x^{-}_{c})=\boldsymbol{\alpha}_{T_{c}}(\theta,x^{+}_{c})-\boldsymbol{\alpha}_{T_{c}}(\theta,x^{-}_{c})=\pm\mathtt{s}_{c}$.
As a result

$\displaystyle\mathtt{s}_{c}$
$\displaystyle=\pm\left(\boldsymbol{\alpha}(\theta,x^{+}_{c})-\boldsymbol{\alpha}(\theta,x^{-}_{c})\right)\in\mathtt{A}(\theta)$
$\displaystyle(\mathtt{s}_{c},0)$
$\displaystyle=\pm\left(\bar{\boldsymbol{\alpha}}(\theta,x^{+}_{c})-\bar{\boldsymbol{\alpha}}(\theta,x^{-}_{c})\right)\in\mathtt{\bar{A}}(\theta)$

as claimed. Using (52) and the partition of $H$ into $T_{1},\ldots,T_{C}$ we
have for any $x\in\mathcal{X}_{\theta}$

(55) $\displaystyle 2\boldsymbol{\alpha}(\theta,x)$
$\displaystyle=\sum_{c}2\boldsymbol{\alpha}_{T_{c}}(\theta,x)=\sum_{c}(\mathtt{1}_{T_{c}}+\epsilon_{c}(x)\cdot\mathtt{s}_{c})=\mathtt{1}_{H}+\sum_{c}\epsilon_{c}(x)\mathtt{s}_{c}$

We obtain

(56)
$\displaystyle\mathtt{1}_{H}=2\boldsymbol{\alpha}(\theta,x)-\sum_{c}\epsilon_{c}(x)\mathtt{s}_{c},$

and since $\boldsymbol{\alpha}(\theta,x)\in\mathtt{A}(\theta)$ and
$\mathtt{s}_{c}\in\mathtt{A}(\theta)$ for all $c$, it follows that
$\mathtt{1}_{H}\in\mathtt{A}(\theta)$. This proves
$\mathtt{span}\left\\{\mathtt{1}_{H},\mathtt{s}_{c},1\leq c\leq
C\right\\}\subseteq\mathtt{A}(\theta)$. Vice-versa, (55) shows
$\boldsymbol{\alpha}(\theta,x)\in\mathtt{span}\left\\{\mathtt{1}_{H},\mathtt{s}_{c},1\leq
c\leq C\right\\}$ for every $x\in\mathcal{X}_{\theta}$, hence
$\mathtt{A}(\theta)\subseteq\mathtt{span}\left\\{\mathtt{1}_{H},\mathtt{s}_{c},1\leq
c\leq C\right\\}$. By (56) we also get

$(\mathtt{1}_{H},2)=2(\boldsymbol{\alpha}(\theta,x),1)-\sum_{c}\epsilon_{c}(x)(\mathtt{s}_{c},0),$

and since
$(\boldsymbol{\alpha}(\theta,x),1)=\bar{\boldsymbol{\alpha}}(\theta,x)\in\mathtt{\bar{A}}(\theta)$
and $(\mathtt{s}_{c},0)\in\mathtt{\bar{A}}(\theta)$, we get
$(\mathtt{1}_{H},2)\in\mathtt{\bar{A}}(\theta)$. This proves
$\mathtt{span}\left\\{(\mathtt{1}_{H},2),(\mathtt{s}_{c},0),1\leq c\leq
C\right\\}\subseteq\mathtt{\bar{A}}(\theta)$, and also implies

$2\bar{\boldsymbol{\alpha}}(\theta,x)=(\mathtt{1}_{H},2)+\sum_{c}\epsilon_{c}(x)(\mathtt{s}_{c},0)$

hence
$\mathtt{\bar{A}}(\theta)\subseteq\mathtt{span}\left\\{(\mathtt{1}_{H},2),(\mathtt{s}_{c},0),1\leq
c\leq C\right\\}$.

##  Appendix H Proof of Lemma 16

We use the shorthands $\boldsymbol{w}_{\nu}=\boldsymbol{w}_{\bullet\to\nu}$,
$\boldsymbol{v}_{\nu}=\boldsymbol{w}_{\nu\to\bullet}$.

Given the assumption there are $C=|H|-1$ classes of twin neurons, all being
trivial except one made of a pair of negative twins $\\{\nu,\nu^{\prime}\\}$.
Without loss of generality we enumerate the neurons and their classes such
that $T_{1}=\\{\nu_{1},\nu_{2}\\}=\\{\nu,\nu^{\prime}\\}$ and
$T_{c}=\\{\nu_{c+1}\\}$, $2\leq c\leq C=|H|-1$. First we establish that, with
this numbering,

(57)
$\mathtt{\bar{A}}^{\perp}(\theta)=\mathtt{span}\left\\{(1,1,0,\ldots,0,-1)\right\\}\
\text{and}\ \mathtt{A}^{\perp}(\theta)=\\{0\\}.$

The signatures of the classes are $\mathtt{s}_{1}=\delta_{1}-\delta_{2}$ and
$\mathtt{s}_{c}=\delta_{c+1}$, $2\leq c\leq C$. By Lemma 13 we have
$\mathtt{A}(\theta)=\mathtt{span}\left\\{\mathtt{1}_{H},\mathtt{s}_{c},1\leq
c\leq C\right\\}$. It is not difficult to check 111111If, instead of a single
pair of negative twins, we consider a single pair of positive twins, then
$\mathtt{s}_{1}=\delta_{1}+\delta_{2}$ and the spanning vectors of
$\mathtt{A}(\theta)$ become linearly _dependent_ , with
$\mathtt{A}^{\perp}(\theta)=\mathtt{span}\left\\{(1,-1,0,\ldots,0)\right\\}\neq\\{0\\}$.
that the $C+1=|H|$ spanning vectors are linearly independent, hence
$\mathtt{A}(\theta)=\mathbb{R}^{H}$ and $\mathtt{A}^{\perp}(\theta)=\\{0\\}$.
Now, consider
$\boldsymbol{v}=(v_{1},\ldots,v_{|H|+1})\in\mathtt{\bar{A}}^{\perp}(\theta)$.
By Lemma 13, this vector is orthogonal to each $(\mathtt{s}_{c},0)$, $1\leq
c\leq C$, and to $(\mathtt{1}_{H},2)$. For $2\leq c\leq C$, orthogonality to
$(\mathtt{s}_{c},0)=(\delta_{c+1},0)$ implies $v_{c+1}=0$, hence
$\boldsymbol{v}=(\alpha,\beta,0,\ldots,0,\gamma)$ for some
$\alpha,\beta,\gamma\in\mathbb{R}$. Orthogonality to
$(\mathtt{s}_{1},0)=(1,-1,0,\ldots,0)$ implies $\beta=\alpha$, and
orthogonality to $(\mathtt{1}_{H},2)$ implies $\gamma=-\alpha$, hence
$\boldsymbol{v}$ is proportional to $(1,1,0,\ldots,0,-1)$ as claimed. Since
$\mathtt{\bar{A}}(\theta)$ is spanned by $C+1=|H|$ vectors, its dimension is
at most $|H|$, hence the dimension of $\mathtt{\bar{A}}^{\perp}(\theta)$ is at
least one. This concludes the proof that
$\mathtt{\bar{A}}^{\perp}(\theta)=\mathtt{span}\left\\{(1,1,0,\ldots,0,-1)\right\\}$.

Since $\theta$ is admissible, there is an input neuron $\mu\in N_{0}$ such
that $w_{\mu\to\nu_{1}}\neq 0$. Since $\nu_{1}$,$\nu_{2}$ are twins, we also
have $w_{\mu\to\nu_{2}}\neq 0$. Let $\epsilon_{0}:=\min_{1\leq j\leq
2}|w_{\mu\to\nu_{j}}|/2$. Consider $\theta^{\prime}\in B(\theta,\epsilon_{0})$
such that
$\boldsymbol{\Phi}(\theta^{\prime})-\boldsymbol{\Phi}(\theta)\in\mathtt{V}(\theta)$.
First, observe that $w^{\prime}_{\mu\to\nu_{j}}\neq 0$ for $j=1,2$. Then, in
light of Lemma 8 and (57), for every $\eta\in N_{2}$, we have
$\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta^{\prime})=\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta)$
and

(58)
$\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta^{\prime})-\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta)\in\mathtt{span}\left\\{(1,1,0,\ldots,0,-1)\right\\},$

hence
$b^{\prime}_{\nu_{c+1}}w^{\prime}_{\nu_{c+1}\to\eta}=b_{\nu_{c+1}}w_{\nu_{c+1}\to\eta}$
for $2\leq c\leq C$, and there are scalars $\lambda_{\eta}\in\mathbb{R}$ such
that

(59) $\displaystyle
b^{\prime}_{\nu_{j}}w^{\prime}_{\nu_{j}\to\eta}-b_{\nu_{j}}w_{\nu_{j}\to\eta}$
$\displaystyle=\lambda_{\eta},\ \forall j\in\\{1,2\\}\qquad\text{and}\
b^{\prime}_{\eta}-b_{\eta}=-\lambda_{\eta}.$

When $\Theta$ is the set of parameters with zero output biases, the fact that
$\theta,\theta^{\prime}\in\Theta$ implies $b^{\prime}_{\eta}=b_{\eta}=0$,
hence $\lambda_{\eta}=0$ and
$\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta^{\prime})=\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta)$
for every $\eta\in N_{2}$. We show below that the same holds for arbitrary
$\Theta$ when $\boldsymbol{w}_{\nu_{1}\to\bullet}$ and
$\boldsymbol{w}_{\nu_{2}\to\bullet}$ are linearly independent. This implies
$\boldsymbol{\Phi}(\theta^{\prime})=\boldsymbol{\Phi}(\theta)$, hence $\theta$
is then $\epsilon$-non-degenerate with respect to $\Theta$.

Indeed, the equality
$\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta^{\prime})=\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta)$
for all $\eta\in N_{2}$ implies that for $1\leq j\leq 2$,

(60) $\displaystyle w^{\prime}_{\mu\to\nu_{j}}w^{\prime}_{\nu_{j}\to\eta}$
$\displaystyle=w_{\mu\to\nu_{j}}w_{\nu_{j}\to\eta},\ \forall\eta\in N_{2},$

and since $w^{\prime}_{\mu\to\nu_{j}}\neq 0$ for $j=1,2$, we obtain from (59)
and (60) that for each $\eta\in N_{2}$,

$\displaystyle\lambda_{\eta}$
$\displaystyle=b^{\prime}_{\nu_{j}}w^{\prime}_{\nu_{j}\to\eta}-b_{\nu_{j}}w_{\nu_{j}\to\eta}=b^{\prime}_{\nu_{j}}\frac{w^{\prime}_{\mu\to\nu_{j}}w^{\prime}_{\nu_{j}\to\eta}}{w^{\prime}_{\mu\to\nu_{j}}}-b_{\nu_{j}}w_{\nu_{j}\to\eta}$
$\displaystyle=b^{\prime}_{\nu_{j}}\frac{w_{\mu\to\nu_{j}}w_{\nu_{j}\to\eta}}{w^{\prime}_{\mu\to\nu_{j}}}-b_{\nu_{j}}w_{\nu_{j}\to\eta}=\left(b^{\prime}_{\nu_{j}}\frac{w_{\mu\to\nu_{j}}}{w^{\prime}_{\mu\to\nu_{j}}}-b_{\nu_{j}}\right)w_{\nu_{j}\to\eta}$

We obtain $\boldsymbol{\lambda}=x_{j}\boldsymbol{w}_{\nu_{j}\to\bullet}$,
$j=1,2$ where $\boldsymbol{\lambda}:=(\lambda_{\eta})_{\eta\in N_{2}}$ and
$x_{j}:=b^{\prime}_{\nu_{j}}\frac{w_{\mu\to\nu_{j}}}{w^{\prime}_{\mu\to\nu_{j}}}-b_{\nu_{j}}$.
Since $\boldsymbol{w}_{\nu_{1}\to\bullet}$ and
$\boldsymbol{w}_{\nu_{2}\to\bullet}$ are linearly independent, it follows that
$x_{1}=x_{2}=0$, hence $\boldsymbol{\lambda}=\boldsymbol{0}$.

Assume now that $\boldsymbol{w}_{\nu_{1}\to\bullet}$ and
$\boldsymbol{w}_{\nu_{2}\to\bullet}$ are linearly _dependent_ , and recall
that since $\theta$ is admissible they are both nonzero vectors, hence
$\boldsymbol{w}_{\nu_{2}\to\bullet}=\alpha\boldsymbol{w}_{\nu_{2}\to\bullet}$
for some $\alpha\neq 0$. Consider $0<\epsilon<\epsilon_{0}$ and set
$\theta^{\prime}$ as follows:

$\displaystyle\boldsymbol{W}^{\prime}_{\ell}$
$\displaystyle=\boldsymbol{W}_{\ell},\ 1\leq\ell\leq 2;$ $\displaystyle
b^{\prime}_{\nu}$ $\displaystyle=b_{\nu},\ \nu\in
H\backslash\\{\nu_{1},\nu_{2}\\};$ $\displaystyle b^{\prime}_{\nu_{1}}$
$\displaystyle=b_{\nu_{1}}+\gamma\epsilon;$ $\displaystyle
b^{\prime}_{\nu_{2}}$ $\displaystyle=b_{\nu_{2}}+\gamma\epsilon/\alpha,j=1,2;$
$\displaystyle b^{\prime}_{\eta}$
$\displaystyle=b_{\eta}-w_{\nu_{1}\to\eta}\gamma\epsilon,\eta\in N_{2},$

with
$0<\gamma<\min(1,|\alpha|,1/\|\boldsymbol{w}_{\nu_{1}\to\bullet}\|_{\infty})$
so that $\theta^{\prime}\in B(\theta,\epsilon)$. Since the weights of
$\theta^{\prime}$ and $\theta$ coincide we have
$\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta^{\prime})=\boldsymbol{\Phi}^{\mathtt{i}}_{\eta}(\theta)$
for every $\eta\in N_{2}$. It is not difficult to check that, with
$\lambda_{\eta}:=w_{\nu_{1}\to\eta}\epsilon$, we also have
$\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta^{\prime})-\boldsymbol{\Phi}^{\mathtt{h}}_{\eta}(\theta)=\lambda_{\eta}(1,1,0,\ldots,0,-1)$,
hence
$\boldsymbol{\Phi}(\theta^{\prime})-\boldsymbol{\Phi}(\theta)\in\mathtt{V}(\theta)$.
Yet, $\boldsymbol{\Phi}(\theta^{\prime})\neq\boldsymbol{\Phi}(\theta)$ since
$\boldsymbol{\lambda}:=(\lambda_{\eta})_{\eta\in
N_{2}}=\epsilon\boldsymbol{w}_{\nu_{1}\to\bullet}\neq\boldsymbol{0}$. Assuming
that $\theta$ belongs to the interior of $\Theta$, we have
$\theta^{\prime}\in\Theta\cap B(\theta,\epsilon)$ for small enough $\epsilon$.
It follows that $\theta$ is degenerate.

##  Appendix I Details on Example 4

$\theta$ is PS-identifiable from $\mathcal{X}=\mathbb{R}$. Consider an
arbitrary $\theta^{\prime}\in\Theta=\mathbb{R}^{E\cup\bar{H}}$. If
$\boldsymbol{R}_{\theta^{\prime}}(x)=\boldsymbol{R}_{\theta}(x)=|x|$ on
$\mathbb{R}$ then $\theta^{\prime}$ is admissible (otherwise its realization
would be, up to an additive constant, proportional to a single shifted version
of the ReLU, which would prevent it from being equal to
$\boldsymbol{R}_{\theta}=\mathtt{abs}$) hence $w^{\prime}_{\nu\to\nu_{i}}\neq
0$, $i=1,2$. Writing $\alpha_{i}=|w^{\prime}_{\mu\to\nu_{i}}|\
w^{\prime}_{\nu_{i}\to\eta}$ and
$\beta_{i}=-b^{\prime}_{\nu_{i}}/|w^{\prime}_{\mu\to\nu_{i}}|$, and
$s_{i}=\mathtt{sign}(w^{\prime}_{\mu\to\nu_{i}})\in\\{-1,+1\\}$ for $i=1,2$,
we have $\alpha_{i}\neq 0$ and

$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}\mathtt{ReLU}(s_{1}(x-s_{1}\beta_{1}))+\alpha_{2}\mathtt{ReLU}(s_{2}(x-s_{2}\beta_{2}))+b^{\prime}_{\eta},\
\forall x\in\mathbb{R}.$

If we had $s_{1}\beta_{1}\neq s_{2}\beta_{2}$,
$\boldsymbol{R}_{\theta^{\prime}}$ would be non-differentiable at two distinct
points $s_{1}\beta_{1},s_{2}\beta_{2}$. However
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta}=\mathtt{abs}$ is
differentiable on $\mathbb{R}\backslash\\{0\\}$, hence
$s_{1}\beta_{1}=s_{2}\beta_{2}$, and a similar reasoning yields
$s_{1}\beta_{1}=s_{2}\beta_{2}=0$. Since $|s_{i}|=1$, we get
$\beta_{1}=\beta_{2}=0$ and

$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}\mathtt{ReLU}(s_{1}x)+\alpha_{2}\mathtt{ReLU}(s_{2}x)+b^{\prime}_{\eta},\
\forall x\in\mathbb{R}.$

If we had $s_{1}=s_{2}$, the realization would be
$(\alpha_{1}+\alpha_{2})\mathtt{ReLU}(s_{1}x)+b^{\prime}_{\eta}$, which cannot
match $\mathtt{abs}$, hence $s_{2}=-s_{1}$. Without loss of generality (up to
a permutation of indices of the hidden layer) $s_{1}=1$, $s_{2}=-1$. Now, for
$x<0$ we have
$-x=|x|=\boldsymbol{R}_{\theta^{\prime}}(x)=-\alpha_{2}x+b^{\prime}_{\eta}$
while for $x>0$ we get
$x=|x|=\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}x+b_{\eta}$, hence
$\alpha_{1}=1$, $\alpha_{2}=1$, $b^{\prime}_{\eta}=0$. Overall, up to the
possible permutation of the hidden layer, we obtain
$\mathtt{sign}(\theta^{\prime})=\mathtt{sign}(\theta)$ and
$\alpha_{1}s_{1}=1$, $\alpha_{2}s_{2}=-1$,
$\alpha_{1}\beta_{1}=\alpha_{2}\beta_{2}=0$, $b^{\prime}_{\eta}=0$, hence
$\boldsymbol{\Phi}(\theta^{\prime})=\boldsymbol{\Phi}(\theta)$. Since $\theta$
is admissible, it follows by Theorem 1 that $\theta^{\prime}\sim_{PS}\theta$.
Since this holds for any $\theta^{\prime}$ such that
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta}$, this shows that
$\theta$ is PS-identifiable from $\mathcal{X}=\mathbb{R}$ with respect to
$\Theta=\mathbb{R}^{E\cup\bar{H}}$.

$\theta$ is locally S-identifiable from some finite set $F\subset\mathbb{R}$
(with $0\in F$)

With the same notations as above, observe that there is $\epsilon>0$ such that
for every $\theta^{\prime}\in B(\theta,\epsilon)$ we have
$s_{i}:=\mathtt{sign}(w^{\prime}_{\mu\to\nu_{i}})=\mathtt{sign}(w_{\mu\to\nu_{i}})$,
$i=1,2$, and
$\max(|\alpha_{1}-1|,|\alpha_{2}-1|,|\beta_{1}|,|\beta_{2}|,|b^{\prime}_{\eta}|)\leq
1/2$. Consider $\theta^{\prime}\in B(\theta,\epsilon)$ such that
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta^{\prime}}$ on
$F=\\{-3,-2,-1,0,1,2,3\\}$. We have $s_{1}=+1$, $s_{2}=-1$ hence

$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}\mathtt{ReLU}(x-\beta_{1})+\alpha_{2}\mathtt{ReLU}(-x-\beta_{2})+b^{\prime}_{\eta}.$

Since $|\beta_{i}|\leq 1/2$ for $i=1,2$, we have
$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}(x-\beta_{1})+b^{\prime}_{\eta}$
for every $x\geq 1/2$, hence
$\alpha_{1}(y-x)=\boldsymbol{R}_{\theta^{\prime}}(y)-\boldsymbol{R}_{\theta^{\prime}}(x)=\boldsymbol{R}_{\theta}(y)-\boldsymbol{R}_{\theta}(x)=|y|-|x|=y-x$
for $(x,y)=(1,2)$. Therefore $\alpha_{1}=1$. A similar reasoning with
$(x,y)=(-2,-1)$ shows that $\alpha_{2}=1$, hence

$\boldsymbol{R}_{\theta^{\prime}}(x)=\mathtt{ReLU}(x-\beta_{1})+\mathtt{ReLU}(-x-\beta_{2})+b^{\prime}_{\eta}$

Specializing to $x=1$, since $x-\beta_{1}>0$ and $-x-\beta_{2}<0$ we get
$1=|x|=\boldsymbol{R}_{\theta^{\prime}}(x)=x-\beta_{1}+b^{\prime}_{\eta}=1-\beta_{1}+b^{\prime}_{\eta}$
hence $b^{\prime}_{\eta}=\beta_{1}$. Similarly, with $x=-1$, we get
$b^{\prime}_{\eta}=\beta_{2}$ hence

$\boldsymbol{R}_{\theta^{\prime}}(x)=\mathtt{ReLU}(x-b^{\prime}_{\eta})+\mathtt{ReLU}(-x-b^{\prime}_{\eta})+b^{\prime}_{\eta}.$

Specializing to $x=0\in F$ yields

$0=|x|=\boldsymbol{R}_{\theta^{\prime}}(x)=b^{\prime}_{\eta}+2\mathtt{ReLU}(-b^{\prime}_{\eta})=\begin{cases}b^{\prime}_{\eta}&\text{if}\
b^{\prime}_{\eta}\geq 0\\\ -b^{\prime}_{\eta}&\text{if}\ b^{\prime}_{\eta}\leq
0\end{cases}=|b^{\prime}_{\eta}|$

hence $b^{\prime}_{\eta}=0$. Overall we have shown that for every
$\theta^{\prime}\in B(\theta,\epsilon)$ such that
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta}$ on
$F=\\{-2,-1,0,1,2\\}$ we have $\alpha_{1}=\alpha_{2}=1$, $s_{1}=1,s_{2}=-1$,
$\beta_{1}=\beta_{2}=b^{\prime}_{\eta}=0$. These imply
$\boldsymbol{\Phi}(\theta^{\prime})=\boldsymbol{\Phi}(\theta)$ and
$\mathtt{sign}(\theta^{\prime})=\mathtt{sign}(\theta)$ hence
$\theta^{\prime}\sim_{S}\theta$. In other words, $\theta$ is locally
S-identifiable from $F$.

##  Appendix J Details on Example 5

Here we show, as claimed in Example 5 that the parameter
$\theta_{0}\in\mathbb{R}^{E\cup\bar{H}}$ from Example 1 is PS-identifiable
from $\mathcal{X}=\mathbb{R}$ with respect to the set $\Theta_{0}$ of
parameters with zero output biases. Consider an arbitrary
$\theta^{\prime}\in\Theta_{0}$. If
$\boldsymbol{R}_{\theta^{\prime}}(x)=\boldsymbol{R}_{\theta_{0}}(x)=x$ on
$\mathcal{X}$ then $\theta^{\prime}$ is admissible (otherwise its realization
would be, up to an additive constant, proportional to a single shifted version
of the ReLU, which would prevent it from being equal to
$\boldsymbol{R}_{\theta}=\mathtt{id}$) hence $w^{\prime}_{\nu\to\nu_{i}}\neq
0$, $i=1,2$. Writing $\alpha_{i}=|w^{\prime}_{\mu\to\nu_{i}}|\
w^{\prime}_{\nu_{i}\to\eta}$ and
$\beta_{i}=-b^{\prime}_{\nu_{i}}/|w^{\prime}_{\mu\to\nu_{i}}|$, and
$s_{i}=\mathtt{sign}(w^{\prime}_{\mu\to\nu_{i}})\in\\{-1,+1\\}$ for $i=1,2$,
we have $\alpha_{i}\neq 0$ and since the output bias is zero

$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}\mathtt{ReLU}(s_{1}(x-s_{1}\beta_{1}))+\alpha_{2}\mathtt{ReLU}(s_{2}(x-s_{2}\beta_{2})),\
\forall x\in\mathcal{X}.$

If we had $s_{1}\beta_{1}\neq s_{2}\beta_{2}$,
$\boldsymbol{R}_{\theta^{\prime}}$ would be non-differentiable at two distinct
points $s_{1}\beta_{1},s_{2}\beta_{2}$. However
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta_{0}}=\mathtt{id}$ is
differentiable on $\mathbb{R}$, hence $s_{1}\beta_{1}=s_{2}\beta_{2}$. It
follows that
$s_{2}\beta_{1}=s_{1}\beta_{1}=\mathtt{id}(s_{1}\beta_{1})=\boldsymbol{R}_{\theta^{\prime}}(s_{1}\beta_{1})=0$.
Since $|s_{i}|=1$, we get $\beta_{1}=\beta_{2}=0$ and

$\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}\mathtt{ReLU}(s_{1}x)+\alpha_{2}\mathtt{ReLU}(s_{2}x),\
\forall x\in\mathcal{X}.$

If we had $s_{1}=s_{2}$, the realization would be
$(\alpha_{1}+\alpha_{2})\mathtt{ReLU}(s_{1}x)$, which cannot match
$\mathtt{id}$ on $\mathcal{X}$, hence $s_{2}=-s_{1}$. Without loss of
generality (up to a permutation of indices of the hidden layer) $s_{1}=1$,
$s_{2}=-1$. Now, for $x<0$ we have
$x=\mathtt{id}(x)=\boldsymbol{R}_{\theta^{\prime}}(x)=-\alpha_{2}x$ while for
$x>0$ we get
$x=\mathtt{id}(x)=\boldsymbol{R}_{\theta^{\prime}}(x)=\alpha_{1}x$, hence
$\alpha_{1}=1$, $\alpha_{2}=-1$ (and $b^{\prime}_{\eta}=0$ because
$\theta^{\prime}\in\Theta_{0}$). Overall, up to the possible permutation of
the hidden layer, we obtain
$\mathtt{sign}(\theta^{\prime})=\mathtt{sign}(\theta_{0})$ and
$\alpha_{1}s_{1}=1$, $\alpha_{2}s_{2}=-1$,
$\alpha_{1}\beta_{1}=\alpha_{2}\beta_{2}=0$, $b^{\prime}_{\eta}=0$, hence
$\boldsymbol{\Phi}(\theta^{\prime})=\boldsymbol{\Phi}(\theta_{0})$. Since
$\theta_{0}$ is admissible, it follows by Theorem 1 that
$\theta^{\prime}\sim_{PS}\theta_{0}$. Since this holds for any
$\theta^{\prime}\in\Theta_{0}$ such that
$\boldsymbol{R}_{\theta^{\prime}}=\boldsymbol{R}_{\theta_{0}}$, this shows
that $\theta$ is PS-identifiable from $\mathcal{X}=\mathbb{R}$ with respect to
$\Theta_{0}$.