*Old tables

This section will be removed in the submitted article.

06 May 2023

|  | task |  sl-MNIST $\uparrow$ |  SHD $\uparrow$ |  PTB $\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\textbf{29.38}\pm 6.65$ | $34.66\pm 8.30$ | $121.16\pm 3.85$
|  | $\rho$ | $0.492\pm 0.008$ | $0.496\pm 0.007$ | $0.492\pm 0.011$
| $\rho_{t}=1$ | metric |  $24.18\pm 6.86$ | $38.16\pm\textbf{2.71}$ |  $\textbf{113.31}\pm\textbf{1.71}$
|  | $\rho$ |  $0.971\pm 0.024$ | $1.019\pm 0.019$ |  $0.889\pm 0.070$
| none | metric | $29.02\pm\textbf{2.00}$ | $\textbf{46.40}\pm 12.12$ | $120.77\pm 2.67$
|  | $\rho$ | $0.354\pm 0.002$ | $0.363\pm 0.001$ | $0.353\pm 0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\textbf{28.12}\pm 3.41$ | $35.46\pm 10.64$ |  $237.47\pm 97.22$
|  | $\rho$ | $0.500\pm 0.003$ | $0.508\pm 0.003$ |  $0.414\pm 0.001$
| $\rho_{t}=1$ | metric | $24.80\pm 5.44$ |  $33.40\pm\textbf{5.63}$ |  $238.82\pm 96.21$
|  | $\rho$ | $0.997\pm 0.009$ |  $0.845\pm 0.177$ |  $0.414\pm 0.001$
| none | metric | $23.51\pm\textbf{2.50}$ | $\textbf{39.84}\pm 5.91$ | $\textbf{173.67}\pm\textbf{3.21}$
|  | $\rho$ | $1.317\pm 0.011$ | $1.349\pm 0.007$ | $0.760\pm 0.004$
GRU | $\rho_{t}=0.5$ | metric | $\textbf{95.14}\pm 0.48$ | $\textbf{85.58}\pm\textbf{1.35}$ | $\textbf{103.12}\pm\textbf{0.25}$
|  | $\rho$ | $0.488\pm 0.016$ | $0.491\pm 0.006$ | $0.499\pm 0.001$
| $\rho_{t}=1$ | metric | $95.04\pm 0.75$ | $84.10\pm 2.27$ | $103.41\pm 0.78$
|  | $\rho$ | $0.999\pm 0.004$ | $1.003\pm 0.005$ | $1.002\pm 0.004$
| none | metric | $94.79\pm\textbf{0.38}$ | $84.92\pm 2.05$ | $104.05\pm 0.53$
|  | $\rho$ | $0.710\pm 0.004$ | $0.719\pm 0.002$ | $0.694\pm 0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21\pm\textbf{0.24}$ |  $\textbf{84.48}\pm\textbf{0.97}$ | $\textbf{108.83}\pm 1.65$
|  | $\rho$ | $0.503\pm 0.006$ |  $0.536\pm 0.026$ | $0.511\pm 0.016$
| $\rho_{t}=1$ | metric |  $\textbf{95.53}\pm 0.42$ | $83.95\pm 2.44$ | $109.33\pm\textbf{0.71}$
|  | $\rho$ |  $0.955\pm 0.038$ | $0.997\pm 0.014$ | $1.000\pm 0.018$
| none | metric | $95.20\pm 0.95$ | $84.17\pm 2.19$ | $109.03\pm 3.03$
|  | $\rho$ | $0.876\pm 0.005$ | $0.887\pm 0.002$ | $0.847\pm 0.003$
ALIF+ | $\rho_{t}=1$ | metric |  $90.31\pm 1.22$ |  $\textbf{82.11}\pm 3.01$ |  $131.78\pm 1.68$
|  | $\rho$ |  $0.966\pm 0.009$ |  $0.950\pm 0.102$ |  $0.978\pm 0.208$
| none | metric | $\textbf{91.68}\pm\textbf{1.01}$ | $77.99\pm\textbf{2.68}$ | $\textbf{130.18}\pm\textbf{1.06}$
|  | $\rho$ | $2.225\pm 0.033$ | $1.748\pm 0.016$ | $1.690\pm 0.022$
ALIF± | $\rho_{t}=1$ | metric | $\textbf{87.17}\pm 2.25$ | $74.19\pm 3.71$ |  $\textbf{127.94}\pm\textbf{0.97}$
|  | $\rho$ | $0.999\pm 0.001$ | $1.000\pm 0.000$ |  $0.876\pm 0.176$
| none | metric | $82.07\pm\textbf{1.12}$ | $\textbf{74.76}\pm\textbf{1.34}$ | $130.31\pm 3.05$
|  | $\rho$ | $1.212\pm 0.007$ | $1.130\pm 0.003$ | $1.094\pm 0.002$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better, and perplexity for PTB, the lower the better. The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.02$, and those experiments are currently running again. |  | task |  sl-MNIST $\uparrow$ |  SHD $\uparrow$ |  PTB $\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.19\pm 0.30$ | $\textbf{14.71}\pm\textbf{3.50}$ |  $643.36\pm\textbf{0.63}$
|  | $\rho$ | $0.498\pm 0.003$ | $0.498\pm 0.002$ |  $0.202\pm 0.001$
| $\rho_{t}=1$ | metric | $\textbf{11.45}\pm\textbf{0.16}$ | $9.45\pm 4.17$ |  $\textbf{403.84}\pm 276.62$
|  | $\rho$ | $0.999\pm 0.019$ | $1.016\pm 0.019$ |  $1.053\pm 0.064$
| none | metric | $11.30\pm 0.40$ | $12.90\pm 3.82$ | $643.23\pm 0.94$
|  | $\rho$ | $0.357\pm 0.002$ | $0.361\pm 0.001$ | $0.351\pm 0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $17.15\pm 4.90$ | $\textbf{24.90}\pm 3.62$ |  $585.56\pm 99.79$
|  | $\rho$ | $0.505\pm 0.008$ | $0.511\pm 0.003$ |  $0.426\pm 0.002$
| $\rho_{t}=1$ | metric | $17.90\pm\textbf{2.04}$ |  $18.77\pm\textbf{1.69}$ | 0
|  | $\rho$ | $1.002\pm 0.006$ |  $0.859\pm 0.163$ | 0
| none | metric | $\textbf{19.63}\pm 6.60$ | $16.43\pm 3.51$ | $\textbf{500.04}\pm\textbf{94.49}$
|  | $\rho$ | $1.323\pm 0.009$ | $1.341\pm 0.006$ | $0.769\pm 0.001$
GRU | $\rho_{t}=0.5$ | metric | $\textbf{95.61}\pm\textbf{0.28}$ | $86.47\pm\textbf{0.60}$ |  $651.71\pm 8.00$
|  | $\rho$ | $0.502\pm 0.002$ | $0.500\pm 0.002$ |  $0.448\pm 0.002$
| $\rho_{t}=1$ | metric | $94.33\pm 1.31$ | $86.24\pm 2.38$ |  $\textbf{249.92}\pm 268.44$
|  | $\rho$ | $0.999\pm 0.008$ | $1.016\pm 0.017$ |  $0.822\pm 0.020$
| none | metric | $\textbf{95.61}\pm 0.38$ | $\textbf{87.81}\pm 1.56$ | $650.33\pm\textbf{1.17}$
|  | $\rho$ | $0.695\pm 0.003$ | $0.702\pm 0.002$ | $0.687\pm 0.002$
LSTM | $\rho_{t}=0.5$ | metric | $\textbf{95.65}\pm\textbf{0.35}$ | $82.04\pm\textbf{1.15}$ | $642.86\pm 1.05$
|  | $\rho$ | $0.501\pm 0.001$ | $0.506\pm 0.008$ | $0.509\pm 0.002$
| $\rho_{t}=1$ | metric | $53.70\pm 48.80$ | $\textbf{82.59}\pm 2.73$ |  $\textbf{141.58}\pm 4.04$
|  | $\rho$ | $0.996\pm 0.006$ | $1.010\pm 0.007$ |  $0.933\pm 0.035$
| none | metric | $74.92\pm 42.28$ | $67.71\pm 23.73$ | $644.18\pm\textbf{0.59}$
|  | $\rho$ | $0.858\pm 0.002$ | $0.871\pm 0.001$ | $0.854\pm 0.001$
ALIF+ | $\rho_{t}=1$ | metric |  $\textbf{29.14}\pm 29.95$ | $\textbf{18.93}\pm 12.85$ |  $\textbf{661.67}\pm 6.22$
|  | $\rho$ |  $0.799\pm 0.005$ | $0.995\pm 0.012$ |  $0.620\pm 0.003$
| none | metric | $11.05\pm\textbf{0.28}$ | $18.85\pm\textbf{10.20}$ | $665.07\pm\textbf{1.72}$
|  | $\rho$ | $2.281\pm 0.014$ | $1.663\pm 0.007$ | $1.650\pm 0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.76\pm\textbf{0.80}$ | $14.23\pm 5.72$ |  $662.17\pm 6.09$
|  | $\rho$ | $0.980\pm 0.014$ | $1.000\pm 0.001$ |  $0.619\pm 0.002$
| none | metric | $\textbf{18.90}\pm 15.85$ | $\textbf{15.26}\pm\textbf{5.53}$ | $\textbf{661.85}\pm\textbf{0.89}$
|  | $\rho$ | $1.090\pm 0.005$ | $0.979\pm 0.002$ | $0.973\pm 0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN $\sigma$ $\rho_{t}=0.5$ metric
$\textbf{41.29}\pm 3.82$ $30.38\pm\textbf{6.97}$
$\textbf{14.71}\pm\textbf{3.50}$ $\textbf{7.99}\pm 3.43$ $\rho$ $0.504\pm
0.005$ $0.500\pm 0.003$ $0.498\pm 0.002$ $0.500\pm 0.002$ $\rho_{t}=1$ metric
$37.64\pm\textbf{3.55}$ $24.52\pm 8.06$ $9.45\pm 4.17$ $4.94\pm 0.16$ $\rho$
$0.992\pm 0.016$ $1.019\pm 0.077$ $1.016\pm 0.019$ $1.003\pm 0.004$ none
metric $38.20\pm 6.84$ $\textbf{30.47}\pm 13.34$ $12.90\pm 3.82$
$5.06\pm\textbf{0.16}$ $\rho$ $0.366\pm 0.001$ $0.362\pm 0.001$ $0.361\pm
0.001$ $0.361\pm 0.000$ RNN $ReLU$ $\rho_{t}=0.5$ metric $37.48\pm 9.97$
$26.52\pm 6.23$ $\textbf{24.90}\pm 3.62$ $18.27\pm 2.88$ $\rho$ $0.512\pm
0.003$ $0.511\pm 0.003$ $0.511\pm 0.003$ $0.512\pm 0.001$ $\rho_{t}=1$ metric
$37.92\pm\textbf{4.14}$ $28.24\pm\textbf{5.18}$ $18.77\pm\textbf{1.69}$
$\textbf{18.32}\pm 5.13$ $\rho$ $0.821\pm 0.199$ $1.004\pm 0.010$ $0.859\pm
0.163$ $0.731\pm 0.007$ none metric $\textbf{39.06}\pm 10.85$
$\textbf{34.19}\pm 5.98$ $16.43\pm 3.51$ $15.59\pm\textbf{2.31}$ $\rho$
$1.355\pm 0.004$ $1.346\pm 0.003$ $1.341\pm 0.006$ $1.339\pm 0.005$ GRU
$\rho_{t}=0.5$ metric $\textbf{84.53}\pm 2.13$ $\textbf{86.05}\pm 2.98$
$86.47\pm\textbf{0.60}$ $\textbf{86.12}\pm 2.35$ $\rho$ $0.461\pm 0.006$
$0.495\pm 0.009$ $0.500\pm 0.002$ $0.500\pm 0.002$ $\rho_{t}=1$ metric
$76.01\pm 3.56$ $84.29\pm\textbf{1.79}$ $86.24\pm 2.38$
$85.87\pm\textbf{1.52}$ $\rho$ $0.993\pm 0.006$ $1.006\pm 0.005$ $1.016\pm
0.017$ $1.031\pm 0.021$ none metric $84.01\pm\textbf{0.91}$ $86.01\pm 2.48$
$\textbf{87.81}\pm 1.56$ $84.72\pm 3.89$ $\rho$ $0.751\pm 0.002$ $0.710\pm
0.002$ $0.702\pm 0.002$ $0.699\pm 0.002$ LSTM $\rho_{t}=0.5$ metric
$\textbf{83.67}\pm\textbf{1.24}$ $\textbf{86.75}\pm\textbf{0.44}$
$82.04\pm\textbf{1.15}$ $57.13\pm 35.78$ $\rho$ $0.528\pm 0.006$ $0.514\pm
0.014$ $0.506\pm 0.008$ $0.504\pm 0.003$ $\rho_{t}=1$ metric $67.26\pm 25.88$
$84.15\pm 1.81$ $\textbf{82.59}\pm 2.73$ $\textbf{70.54}\pm 25.93$ $\rho$
$0.978\pm 0.032$ $1.005\pm 0.007$ $1.010\pm 0.007$ $1.030\pm 0.028$ none
metric $76.69\pm 9.27$ $83.98\pm 2.20$ $67.71\pm 23.73$
$40.56\pm\textbf{19.39}$ $\rho$ $0.914\pm 0.003$ $0.879\pm 0.002$ $0.871\pm
0.001$ $0.869\pm 0.002$ ALIF+ $\rho_{t}=1$ metric
$\textbf{77.98}\pm\textbf{1.02}$ $53.58\pm 32.72$ $\textbf{18.93}\pm 12.85$
$\textbf{5.93}\pm 1.24$ $\rho$ $1.003\pm 0.005$ $1.012\pm 0.032$ $0.995\pm
0.012$ $0.913\pm 0.128$ none metric $76.94\pm 2.06$
$\textbf{62.43}\pm\textbf{4.44}$ $18.85\pm\textbf{10.20}$
$4.69\pm\textbf{0.58}$ $\rho$ $1.853\pm 0.023$ $1.701\pm 0.011$ $1.663\pm
0.007$ $1.656\pm 0.007$ ALIF± $\rho_{t}=1$ metric
$\textbf{67.74}\pm\textbf{1.52}$ $\textbf{66.96}\pm 9.64$ $14.23\pm 5.72$
$\textbf{7.88}\pm 3.94$ $\rho$ $1.000\pm 0.003$ $1.001\pm 0.002$ $1.000\pm
0.001$ $1.000\pm 0.001$ none metric $66.83\pm 4.23$ $64.05\pm\textbf{5.50}$
$\textbf{15.26}\pm\textbf{5.53}$ $4.90\pm\textbf{0.17}$ $\rho$ $1.460\pm
0.007$ $1.016\pm 0.001$ $0.979\pm 0.002$ $0.933\pm 0.001$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  1 09 May 2023

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN$\sigma$ | $\rho_{t}=0.5$ | metric | $\textbf{29.38}\pm 6.65$ | $34.66\pm 8.30$ | $117.33\pm 6.06$
|  | $\rho$ | $0.492\pm 0.008$ | $0.496\pm 0.007$ | $0.492\pm 0.011$
| $\rho_{t}=1$ | metric | $23.94\pm 3.82$ | $38.31\pm\textbf{2.73}$ |  $\textbf{112.51}\pm\textbf{1.66}$
|  | $\rho$ | $0.970\pm 0.029$ | $1.019\pm 0.019$ |  $0.889\pm 0.070$
| none | metric | $29.02\pm\textbf{2.00}$ | $\textbf{46.40}\pm 12.12$ | 0
|  | $\rho$ | $0.354\pm 0.002$ | $0.363\pm 0.001$ | 0
RNN$ReLU$ | $\rho_{t}=0.5$ | metric | $\textbf{28.12}\pm 3.41$ | $35.46\pm 10.64$ |  $247.21\pm 97.18$
|  | $\rho$ | $0.500\pm 0.003$ | $0.508\pm 0.003$ |  $0.414\pm 0.001$
| $\rho_{t}=1$ | metric | $24.80\pm 5.44$ |  $33.50\pm\textbf{5.65}$ |  $237.47\pm 97.22$
|  | $\rho$ | $0.997\pm 0.009$ |  $0.845\pm 0.177$ |  $0.414\pm 0.001$
| none | metric | $23.51\pm\textbf{2.50}$ | $\textbf{39.84}\pm 5.91$ | $\textbf{173.67}\pm\textbf{3.21}$
|  | $\rho$ | $1.317\pm 0.011$ | $1.349\pm 0.007$ | $0.760\pm 0.004$
GRU | $\rho_{t}=0.5$ | metric | $\textbf{95.10}\pm 0.55$ | $\textbf{85.58}\pm\textbf{1.35}$ | $\textbf{103.12}\pm 0.25$
|  | $\rho$ | $0.488\pm 0.016$ | $0.491\pm 0.006$ | $0.499\pm 0.001$
| $\rho_{t}=1$ | metric | $95.04\pm 0.75$ | $83.59\pm 2.70$ | $103.41\pm 0.78$
|  | $\rho$ | $0.999\pm 0.004$ | $1.003\pm 0.005$ | $1.002\pm 0.004$
| none | metric | $94.78\pm\textbf{0.46}$ | $84.92\pm 2.05$ | $103.78\pm\textbf{0.21}$
|  | $\rho$ | $0.713\pm 0.000$ | $0.719\pm 0.002$ | $0.694\pm 0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21\pm\textbf{0.24}$ | $\textbf{85.18}\pm\textbf{0.27}$ | $108.18\pm 0.73$
|  | $\rho$ | $0.503\pm 0.006$ | $0.536\pm 0.026$ | $0.511\pm 0.016$
| $\rho_{t}=1$ | metric | $\textbf{95.64}\pm 0.40$ | $83.41\pm 2.36$ | $109.33\pm 0.71$
|  | $\rho$ | $0.955\pm 0.038$ | $0.997\pm 0.014$ | $1.000\pm 0.018$
| none | metric | $95.09\pm 1.13$ | $83.69\pm 2.41$ | $\textbf{107.52}\pm\textbf{0.27}$
|  | $\rho$ | $0.877\pm 0.004$ | $0.887\pm 0.003$ | $0.847\pm 0.004$
ALIF | $\rho_{t}=1$ | metric | $92.59\pm 0.93$ |  $\textbf{82.01}\pm\textbf{3.21}$ | $\textbf{129.82}\pm\textbf{0.96}$
|  | $\rho$ | $0.966\pm 0.009$ |  $0.950\pm 0.102$ | $0.978\pm 0.208$
| none | metric | $\textbf{93.27}\pm\textbf{0.55}$ | 0 | $130.18\pm 1.06$
|  | $\rho$ | $2.248\pm 0.033$ | 0 | $1.690\pm 0.022$
ALIF± | $\rho_{t}=1$ | metric | $\textbf{89.20}\pm\textbf{0.74}$ | $\textbf{74.19}\pm\textbf{3.71}$ |  $\textbf{125.80}\pm\textbf{0.59}$
|  | $\rho$ | $0.999\pm 0.001$ | $1.000\pm 0.000$ |  $0.876\pm 0.176$
| none | metric | $88.26\pm 1.19$ | $71.73\pm nan$ | $128.19\pm~{}nan$
|  | $\rho$ | $1.213\pm 0.008$ | $1.129\pm~{}nan$ | $1.093\pm~{}nan$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better, and perplexity for PTB, the lower the better. The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.05$, and those experiments are currently running again. |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN$\sigma$ | $\rho_{t}=0.5$ | metric | $11.19\pm~{}0.30$ | $\textbf{14.71}\pm~{}3.50$ |  $643.26\pm~{}\textbf{0.70}$
|  | $\rho$ | $0.498\pm~{}0.003$ | $0.498\pm~{}0.002$ |  $0.202\pm~{}0.001$
| $\rho_{t}=1$ | metric | $11.45\pm~{}\textbf{0.20}$ | $11.01\pm~{}\textbf{3.38}$ |  $\textbf{395.55}\pm~{}286.50$
|  | $\rho$ | $1.006\pm~{}0.017$ | $1.012\pm~{}0.021$ |  $1.072\pm~{}0.112$
| none | metric | $\textbf{11.51}\pm~{}0.21$ | $12.90\pm~{}3.82$ | $643.23\pm~{}0.94$
|  | $\rho$ | $0.356\pm~{}0.001$ | $0.361\pm~{}0.001$ | $0.351\pm~{}0.001$
RNN$ReLU$ | $\rho_{t}=0.5$ | metric | $17.15\pm~{}4.90$ | $\textbf{24.90}\pm~{}3.62$ |  $515.00\pm~{}nan$
|  | $\rho$ | $0.505\pm~{}0.008$ | $0.511\pm~{}0.003$ |  $0.427\pm~{}nan$
| $\rho_{t}=1$ | metric | $17.90\pm~{}\textbf{2.04}$ |  $18.73\pm~{}\textbf{1.73}$ | 0
|  | $\rho$ | $1.002\pm~{}0.006$ |  $0.859\pm~{}0.163$ | 0
| none | metric | $\textbf{19.63}\pm~{}6.60$ | $16.43\pm~{}3.51$ | $\textbf{500.04}\pm~{}\textbf{94.49}$
|  | $\rho$ | $1.323\pm~{}0.009$ | $1.341\pm~{}0.006$ | $0.769\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\textbf{95.61}\pm~{}\textbf{0.28}$ | $86.47\pm~{}\textbf{0.60}$ |  $652.90\pm~{}6.50$
|  | $\rho$ | $0.502\pm~{}0.002$ | $0.500\pm~{}0.002$ |  $0.448\pm~{}0.002$
| $\rho_{t}=1$ | metric | $94.33\pm~{}1.31$ | $86.44\pm~{}1.48$ |  $\textbf{113.00}\pm~{}\textbf{0.66}$
|  | $\rho$ | $0.999\pm~{}0.008$ | $1.016\pm~{}0.017$ |  $0.919\pm~{}0.046$
| none | metric | $\textbf{95.61}\pm~{}0.38$ | $\textbf{88.17}\pm~{}1.70$ | $650.56\pm~{}1.31$
|  | $\rho$ | $0.695\pm~{}0.003$ | $0.702\pm~{}0.001$ | $0.686\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.90\pm~{}0.30$ | $80.18\pm~{}2.41$ | $642.86\pm~{}1.05$
|  | $\rho$ | $0.501\pm~{}0.001$ | $0.506\pm~{}0.010$ | $0.509\pm~{}0.002$
| $\rho_{t}=1$ | metric | $53.70\pm~{}48.80$ | $\textbf{84.53}\pm~{}\textbf{2.10}$ | $\textbf{262.52}\pm~{}253.12$
|  | $\rho$ | $0.996\pm~{}0.006$ | $1.010\pm~{}0.007$ | $0.961\pm~{}0.054$
| none | metric | $\textbf{96.07}\pm~{}\textbf{0.17}$ | $59.68\pm~{}38.46$ | $644.18\pm~{}\textbf{0.59}$
|  | $\rho$ | $0.858\pm~{}0.002$ | $0.872\pm~{}0.001$ | $0.854\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $\textbf{46.93}\pm~{}41.49$ | $\textbf{39.44}\pm~{}\textbf{17.85}$ |  $\textbf{658.06}\pm~{}5.53$
|  | $\rho$ |  $0.803\pm~{}0.008$ | $1.003\pm~{}0.005$ |  $0.620\pm~{}0.003$
| none | metric | $11.22\pm~{}\textbf{0.10}$ | 0 | $666.46\pm~{}\textbf{0.61}$
|  | $\rho$ | $2.272\pm~{}0.014$ | 0 | $1.657\pm~{}0.033$
ALIF± | $\rho_{t}=1$ | metric | $10.76\pm~{}0.80$ | $35.64\pm~{}\textbf{17.13}$ |  $664.09\pm~{}3.06$
|  | $\rho$ | $0.990\pm~{}0.011$ | $1.000\pm~{}0.001$ |  $0.619\pm~{}0.002$
| none | metric | $\textbf{10.82}\pm~{}\textbf{0.68}$ | $\textbf{37.21}\pm~{}nan$ | $\textbf{661.75}\pm~{}\textbf{0.75}$
|  | $\rho$ | $1.087\pm~{}0.007$ | $0.980\pm~{}nan$ | $0.973\pm~{}0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN$\sigma$ $\rho_{t}=0.5$ metric
$\textbf{41.29}\pm\textbf{3.82}$ $30.38\pm\textbf{6.97}$ $\textbf{14.71}\pm
3.50$ $\textbf{7.99}\pm 3.43$ $\rho$ $0.504\pm 0.005$ $0.500\pm 0.003$
$0.498\pm 0.002$ $0.500\pm 0.002$ $\rho_{t}=1$ metric $38.50\pm 4.71$
$27.99\pm 11.55$ $11.01\pm\textbf{3.38}$ $4.94\pm 0.16$ $\rho$ $0.992\pm
0.016$ $1.019\pm 0.077$ $1.012\pm 0.021$ $1.003\pm 0.004$ none metric
$38.20\pm 6.84$ $\textbf{30.47}\pm 13.34$ $12.90\pm 3.82$
$5.06\pm\textbf{0.16}$ $\rho$ $0.366\pm 0.001$ $0.362\pm 0.001$ $0.361\pm
0.001$ $0.361\pm 0.000$ RNN$ReLU$ $\rho_{t}=0.5$ metric $37.48\pm 9.97$
$26.52\pm 6.23$ $\textbf{24.90}\pm 3.62$ $\textbf{18.27}\pm 2.88$ $\rho$
$0.512\pm 0.003$ $0.511\pm 0.003$ $0.511\pm 0.003$ $0.512\pm 0.001$
$\rho_{t}=1$ metric $34.03\pm\textbf{4.43}$ $28.24\pm\textbf{5.18}$
$18.73\pm\textbf{1.73}$ $15.78\pm 5.05$ $\rho$ $0.821\pm 0.199$ $1.004\pm
0.010$ $0.859\pm 0.163$ $0.731\pm 0.007$ none metric $\textbf{39.06}\pm 10.85$
$\textbf{34.19}\pm 5.98$ $16.43\pm 3.51$ $15.59\pm\textbf{2.31}$ $\rho$
$1.355\pm 0.004$ $1.346\pm 0.003$ $1.341\pm 0.006$ $1.339\pm 0.005$ GRU
$\rho_{t}=0.5$ metric $\textbf{85.95}\pm\textbf{1.15}$ $\textbf{86.05}\pm
2.98$ $86.47\pm\textbf{0.60}$ $85.06\pm 2.41$ $\rho$ $0.461\pm 0.006$
$0.495\pm 0.009$ $0.500\pm 0.002$ $0.500\pm 0.002$ $\rho_{t}=1$ metric
$76.01\pm 3.56$ $84.29\pm\textbf{1.79}$ $86.44\pm 1.48$ $83.69\pm 2.88$ $\rho$
$0.993\pm 0.006$ $1.006\pm 0.005$ $1.016\pm 0.017$ $1.031\pm 0.021$ none
metric $84.55\pm 1.45$ $86.01\pm 2.48$ $\textbf{88.17}\pm 1.70$
$\textbf{85.10}\pm\textbf{1.39}$ $\rho$ $0.751\pm 0.002$ $0.710\pm 0.002$
$0.702\pm 0.001$ $0.699\pm 0.002$ LSTM $\rho_{t}=0.5$ metric
$\textbf{81.99}\pm\textbf{1.65}$ $\textbf{86.75}\pm\textbf{0.44}$ $80.18\pm
2.41$ $62.16\pm 38.41$ $\rho$ $0.528\pm 0.006$ $0.514\pm 0.014$ $0.506\pm
0.010$ $0.504\pm 0.003$ $\rho_{t}=1$ metric $66.73\pm 25.41$ $84.15\pm 1.81$
$\textbf{84.53}\pm\textbf{2.10}$ $\textbf{71.49}\pm 26.49$ $\rho$ $0.978\pm
0.032$ $1.005\pm 0.007$ $1.010\pm 0.007$ $1.030\pm 0.028$ none metric
$76.69\pm 9.27$ $84.98\pm 3.17$ $59.68\pm 38.46$ $24.28\pm\textbf{2.72}$
$\rho$ $0.914\pm 0.003$ $0.879\pm 0.002$ $0.872\pm 0.001$ $0.868\pm 0.002$
ALIF $\rho_{t}=1$ metric $\textbf{78.10}\pm\textbf{1.24}$ $\textbf{74.34}\pm
8.61$ $\textbf{39.44}\pm\textbf{17.85}$ $\textbf{5.55}\pm\textbf{1.21}$ $\rho$
$1.003\pm 0.005$ $1.008\pm 0.035$ $1.003\pm 0.005$ $0.969\pm 0.047$ none
metric $76.94\pm 2.06$ $74.01\pm\textbf{2.75}$ 0 $4.28\pmnan$ $\rho$ $1.853\pm
0.023$ $1.707\pm 0.016$ 0 $1.651\pmnan$ ALIF± $\rho_{t}=1$ metric
$\textbf{67.74}\pm\textbf{1.52}$ $\textbf{74.60}\pm\textbf{2.55}$
$35.64\pm\textbf{17.13}$ $\textbf{9.69}\pm 8.96$ $\rho$ $1.000\pm 0.003$
$1.001\pm 0.002$ $1.000\pm 0.001$ $1.000\pm 0.001$ none metric $66.83\pm 4.23$
0 37.21 $4.88\pm\textbf{0.10}$ $\rho$ $1.460\pm 0.007$ 0 $0.980\pmnan$
$0.933\pm 0.002$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  2  May 14

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN$\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{29.38}~{}\pm~{}6.65$ | $34.66~{}\pm~{}8.30$ | $\cellcolor{NiceGreen!25}\textbf{116.81}~{}\pm~{}7.31$
|  | $\rho$ | $0.492~{}\pm~{}0.008$ | $0.496~{}\pm~{}0.007$ | $0.497~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $27.30~{}\pm~{}2.57$ | $37.00~{}\pm~{}\textbf{3.80}$ |  $149.32~{}\pm~{}20.17$
|  | $\rho$ |  $0.878~{}\pm~{}0.166$ | $1.003~{}\pm~{}0.004$ |  $0.916~{}\pm~{}0.013$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN$ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{28.12}~{}\pm~{}3.41$ | $35.46~{}\pm~{}10.64$ |  $159.84~{}\pm~{}7.27$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.508~{}\pm~{}0.003$ |  $0.414~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $24.80~{}\pm~{}5.44$ |  $33.93~{}\pm~{}\textbf{4.96}$ |  $\cellcolor{NiceOrange!25}\textbf{159.65}~{}\pm~{}7.50$
|  | $\rho$ | $0.997~{}\pm~{}0.009$ |  $0.845~{}\pm~{}0.177$ |  $0.414~{}\pm~{}0.001$
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\textbf{39.84}~{}\pm~{}5.91$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.43}~{}\pm~{}\textbf{0.39}$ | $\cellcolor{NiceGreen!25}\textbf{85.58}~{}\pm~{}\textbf{1.35}$ | $\cellcolor{NiceGreen!25}\textbf{103.12}~{}\pm~{}0.25$
|  | $\rho$ | $0.485~{}\pm~{}0.021$ | $0.491~{}\pm~{}0.006$ | $0.499~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $95.04~{}\pm~{}0.75$ | $84.40~{}\pm~{}2.21$ | $103.41~{}\pm~{}0.78$
|  | $\rho$ | $0.999~{}\pm~{}0.004$ | $1.003~{}\pm~{}0.005$ | $1.002~{}\pm~{}0.004$
| none | metric | $94.85~{}\pm~{}0.40$ | $84.92~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21~{}\pm~{}\textbf{0.24}$ |  $59.66~{}\pm~{}18.29$ | $\cellcolor{NiceGreen!25}\textbf{108.09}~{}\pm~{}0.86$
|  | $\rho$ | $0.503~{}\pm~{}0.006$ |  $0.536~{}\pm~{}0.031$ | $0.503~{}\pm~{}0.005$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.31}~{}\pm~{}0.52$ | $83.59~{}\pm~{}2.86$ | $108.99~{}\pm~{}\textbf{0.29}$
|  | $\rho$ |  $0.955~{}\pm~{}0.038$ | $1.004~{}\pm~{}0.004$ | $1.009~{}\pm~{}0.004$
| none | metric | $95.21~{}\pm~{}0.95$ | $\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.93~{}\pm~{}2.83$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{92.77}~{}\pm~{}\textbf{0.57}$ | $\cellcolor{NiceOrange!25}\textbf{83.18}~{}\pm~{}\textbf{0.95}$ |  $141.39~{}\pm~{}12.27$
|  | $\rho$ |  $0.967~{}\pm~{}0.015$ | $1.001~{}\pm~{}0.001$ |  $0.918~{}\pm~{}0.208$
| none | metric | $86.38~{}\pm~{}5.16$ | $70.26~{}\pm~{}7.12$ | $\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.225~{}\pm~{}0.033$ | $1.748~{}\pm~{}0.016$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{82.05}~{}\pm~{}6.26$ | $\cellcolor{NiceOrange!25}\textbf{73.98}~{}\pm~{}\textbf{3.75}$ |  $158.54~{}\pm~{}26.89$
|  | $\rho$ | $0.999~{}\pm~{}0.001$ | $1.000~{}\pm~{}0.000$ |  $0.735~{}\pm~{}0.075$
| none | metric | $81.42~{}\pm~{}\textbf{1.17}$ | $68.20~{}\pm~{}6.85$ | $\textbf{133.18}~{}\pm~{}\textbf{7.05}$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.130~{}\pm~{}0.003$ | $1.094~{}\pm~{}0.001$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better, and perplexity for PTB, the lower the better. The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.05$, and those experiments are currently running again. |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN$\sigma$ | $\rho_{t}=0.5$ | metric | $11.19~{}\pm~{}0.30$ | $\textbf{14.71}~{}\pm~{}3.50$ |  $640.79~{}\pm~{}\textbf{0.14}$
|  | $\rho$ | $0.498~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ |  $0.202~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $\textbf{11.52}~{}\pm~{}\textbf{0.12}$ | $12.21~{}\pm~{}\textbf{1.32}$ |  $\textbf{431.34}~{}\pm~{}242.13$
|  | $\rho$ | $0.991~{}\pm~{}0.010$ | $1.009~{}\pm~{}0.017$ |  $0.938~{}\pm~{}0.122$
| none | metric | $11.41~{}\pm~{}0.20$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN$ReLU$ | $\rho_{t}=0.5$ | metric | $17.15~{}\pm~{}4.90$ | $24.90~{}\pm~{}3.62$ |  $\textbf{176.87}~{}\pm~{}\textbf{36.19}$
|  | $\rho$ | $0.505~{}\pm~{}0.008$ | $0.511~{}\pm~{}0.003$ |  $0.427~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $17.90~{}\pm~{}\textbf{2.04}$ |  $\textbf{25.87}~{}\pm~{}10.18$ |  $178.60~{}\pm~{}42.65$
|  | $\rho$ | $1.002~{}\pm~{}0.006$ |  $0.859~{}\pm~{}0.163$ |  $0.429~{}\pm~{}0.003$
| none | metric | $\textbf{19.63}~{}\pm~{}6.60$ | $16.43~{}\pm~{}\textbf{3.51}$ | $500.04~{}\pm~{}94.49$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\textbf{95.61}~{}\pm~{}\textbf{0.28}$ | $86.47~{}\pm~{}\textbf{0.60}$ |  $641.26~{}\pm~{}\textbf{0.11}$
|  | $\rho$ | $0.502~{}\pm~{}0.002$ | $0.500~{}\pm~{}0.002$ |  $0.447~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $94.33~{}\pm~{}1.31$ | $87.26~{}\pm~{}1.90$ |  $\textbf{134.87}~{}\pm~{}5.07$
|  | $\rho$ | $0.999~{}\pm~{}0.008$ | $1.003~{}\pm~{}0.001$ |  $0.953~{}\pm~{}0.024$
| none | metric | $\textbf{95.61}~{}\pm~{}0.38$ | $\textbf{88.12}~{}\pm~{}1.39$ | $648.25~{}\pm~{}4.75$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.002$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.92~{}\pm~{}0.31$ | $\textbf{82.12}~{}\pm~{}\textbf{2.70}$ | $642.86~{}\pm~{}1.05$
|  | $\rho$ | $0.501~{}\pm~{}0.001$ | $0.506~{}\pm~{}0.008$ | $0.509~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $53.70~{}\pm~{}48.80$ | $81.34~{}\pm~{}6.23$ | $\textbf{142.69}~{}\pm~{}11.78$
|  | $\rho$ | $0.996~{}\pm~{}0.006$ | $1.010~{}\pm~{}0.007$ | $0.985~{}\pm~{}0.083$
| none | metric | $\textbf{96.19}~{}\pm~{}\textbf{0.29}$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}\textbf{0.59}$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $11.32~{}\pm~{}\textbf{0.23}$ | $\textbf{6.17}~{}\pm~{}1.86$ |  $\textbf{644.53}~{}\pm~{}\textbf{0.24}$
|  | $\rho$ |  $0.891~{}\pm~{}0.066$ | $1.000~{}\pm~{}0.007$ |  $0.620~{}\pm~{}0.003$
| none | metric | $\textbf{11.75}~{}\pm~{}0.71$ | $4.87~{}\pm~{}\textbf{0.11}$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $1.671~{}\pm~{}0.002$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.76~{}\pm~{}\textbf{0.80}$ | $\textbf{40.96}~{}\pm~{}20.69$ |  $\textbf{643.48}~{}\pm~{}\textbf{0.11}$
|  | $\rho$ | $0.990~{}\pm~{}0.011$ | $1.000~{}\pm~{}0.001$ |  $0.619~{}\pm~{}0.002$
| none | metric | $\textbf{11.73}~{}\pm~{}1.54$ | $19.84~{}\pm~{}\textbf{6.56}$ | $661.75~{}\pm~{}0.75$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.003$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN$\sigma$ $\rho_{t}=0.5$ metric
$\textbf{41.29}~{}\pm~{}3.82$ $30.38~{}\pm~{}6.97$
$\textbf{14.71}~{}\pm~{}3.50$ $\textbf{7.99}~{}\pm~{}3.43$ $\rho$
$0.504~{}\pm~{}0.005$ $0.500~{}\pm~{}0.003$ $0.498~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $36.63~{}\pm~{}\textbf{2.97}$
$15.31~{}\pm~{}\textbf{1.05}$ $12.21~{}\pm~{}\textbf{1.32}$
$4.83~{}\pm~{}0.16$ $\rho$ $0.992~{}\pm~{}0.016$ $1.030~{}\pm~{}0.119$
$1.009~{}\pm~{}0.017$ $1.003~{}\pm~{}0.004$ none metric $38.20~{}\pm~{}6.84$
$\textbf{30.47}~{}\pm~{}13.34$ $12.90~{}\pm~{}3.82$
$5.06~{}\pm~{}\textbf{0.16}$ $\rho$ $0.366~{}\pm~{}0.001$
$0.362~{}\pm~{}0.001$ $0.361~{}\pm~{}0.001$ $0.361~{}\pm~{}0.000$ RNN$ReLU$
$\rho_{t}=0.5$ metric $37.48~{}\pm~{}9.97$ $26.52~{}\pm~{}6.23$
$24.90~{}\pm~{}3.62$ $18.27~{}\pm~{}2.88$ $\rho$ $0.512~{}\pm~{}0.003$
$0.511~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$ $0.512~{}\pm~{}0.001$ $\rho_{t}=1$
metric $36.79~{}\pm~{}\textbf{2.26}$ $28.24~{}\pm~{}\textbf{5.18}$
$\textbf{25.87}~{}\pm~{}10.18$ $\textbf{22.50}~{}\pm~{}4.38$ $\rho$
$0.821~{}\pm~{}0.199$ $1.004~{}\pm~{}0.010$ $0.859~{}\pm~{}0.163$
$0.731~{}\pm~{}0.007$ none metric $\textbf{39.06}~{}\pm~{}10.85$
$\textbf{34.19}~{}\pm~{}5.98$ $16.43~{}\pm~{}\textbf{3.51}$
$15.59~{}\pm~{}\textbf{2.31}$ $\rho$ $1.355~{}\pm~{}0.004$
$1.346~{}\pm~{}0.003$ $1.341~{}\pm~{}0.006$ $1.339~{}\pm~{}0.005$ GRU
$\rho_{t}=0.5$ metric $80.62~{}\pm~{}nan$ $\textbf{86.05}~{}\pm~{}2.98$
$86.47~{}\pm~{}\textbf{0.60}$ $85.03~{}\pm~{}2.38$ $\rho$ $0.453~{}\pm~{}nan$
$0.495~{}\pm~{}0.009$ $0.500~{}\pm~{}0.002$ $0.500~{}\pm~{}0.002$ $\rho_{t}=1$
metric $76.01~{}\pm~{}3.56$ $84.29~{}\pm~{}\textbf{1.79}$ $87.26~{}\pm~{}1.90$
$85.10~{}\pm~{}2.53$ $\rho$ $0.993~{}\pm~{}0.006$ $1.006~{}\pm~{}0.005$
$1.003~{}\pm~{}0.001$ $1.031~{}\pm~{}0.021$ none metric
$\textbf{84.55}~{}\pm~{}\textbf{1.45}$ $86.01~{}\pm~{}2.48$
$\textbf{88.12}~{}\pm~{}1.39$ $\textbf{87.40}~{}\pm~{}\textbf{2.29}$ $\rho$
$0.751~{}\pm~{}0.002$ $0.710~{}\pm~{}0.002$ $0.702~{}\pm~{}0.002$
$0.699~{}\pm~{}0.002$ LSTM $\rho_{t}=0.5$ metric $61.72~{}\pm~{}29.02$
$\textbf{86.86}~{}\pm~{}\textbf{0.48}$ $\textbf{82.12}~{}\pm~{}\textbf{2.70}$
$43.71~{}\pm~{}44.85$ $\rho$ $0.532~{}\pm~{}0.003$ $0.508~{}\pm~{}0.007$
$0.506~{}\pm~{}0.008$ $0.504~{}\pm~{}0.003$ $\rho_{t}=1$ metric
$60.34~{}\pm~{}24.50$ $84.15~{}\pm~{}1.81$ $81.34~{}\pm~{}6.23$
$\textbf{56.58}~{}\pm~{}24.22$ $\rho$ $0.978~{}\pm~{}0.032$
$1.005~{}\pm~{}0.007$ $1.010~{}\pm~{}0.007$ $1.030~{}\pm~{}0.028$ none metric
$\textbf{76.69}~{}\pm~{}\textbf{9.27}$ $83.74~{}\pm~{}2.81$
$62.98~{}\pm~{}27.77$ $27.90~{}\pm~{}\textbf{7.79}$ $\rho$
$0.914~{}\pm~{}0.003$ $0.879~{}\pm~{}0.002$ $0.871~{}\pm~{}0.001$
$0.869~{}\pm~{}0.002$ ALIF $\rho_{t}=1$ metric
$\textbf{78.10}~{}\pm~{}\textbf{1.24}$ $46.41~{}\pm~{}9.78$
$\textbf{6.17}~{}\pm~{}1.86$ $\textbf{5.59}~{}\pm~{}1.53$ $\rho$
$1.003~{}\pm~{}0.005$ $1.012~{}\pm~{}0.032$ $1.000~{}\pm~{}0.007$
$1.003~{}\pm~{}0.011$ none metric $77.18~{}\pm~{}1.59$
$\textbf{57.88}~{}\pm~{}\textbf{6.96}$ $4.87~{}\pm~{}\textbf{0.11}$
$5.03~{}\pm~{}\textbf{0.11}$ $\rho$ $1.853~{}\pm~{}0.023$
$1.701~{}\pm~{}0.011$ $1.671~{}\pm~{}0.002$ $1.654~{}\pm~{}0.004$ ALIF±
$\rho_{t}=1$ metric $\textbf{67.74}~{}\pm~{}\textbf{1.52}$
$\textbf{66.92}~{}\pm~{}\textbf{2.50}$ $\textbf{40.96}~{}\pm~{}20.69$
$\textbf{14.20}~{}\pm~{}6.21$ $\rho$ $1.000~{}\pm~{}0.003$
$1.001~{}\pm~{}0.002$ $1.000~{}\pm~{}0.001$ $1.000~{}\pm~{}0.000$ none metric
$66.83~{}\pm~{}4.23$ $60.47~{}\pm~{}7.05$ $19.84~{}\pm~{}\textbf{6.56}$
$5.06~{}\pm~{}\textbf{0.18}$ $\rho$ $1.460~{}\pm~{}0.007$
$1.016~{}\pm~{}0.001$ $0.979~{}\pm~{}0.003$ $0.933~{}\pm~{}0.001$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  3 May 19

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN$\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{29.38}~{}\pm~{}6.65$ | $34.66~{}\pm~{}8.30$ | $\cellcolor{NiceGreen!25}\textbf{127.97}~{}\pm~{}23.09$
|  | $\rho$ | $0.492~{}\pm~{}0.008$ | $0.496~{}\pm~{}0.007$ | $0.492~{}\pm~{}0.011$
| $\rho_{t}=1$ | metric |  $27.57~{}\pm~{}3.85$ | $32.75~{}\pm~{}\textbf{6.12}$ |  $145.06~{}\pm~{}nan$
|  | $\rho$ |  $0.900~{}\pm~{}0.142$ | $1.019~{}\pm~{}0.019$ |  $0.939~{}\pm~{}nan$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN$ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{28.12}~{}\pm~{}3.41$ | $35.46~{}\pm~{}10.64$ |  $\cellcolor{NiceGreen!25}\textbf{161.83}~{}\pm~{}8.83$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.508~{}\pm~{}0.003$ |  $0.414~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $24.80~{}\pm~{}5.44$ |  $33.30~{}\pm~{}\textbf{4.89}$ |  $162.02~{}\pm~{}8.94$
|  | $\rho$ | $0.997~{}\pm~{}0.009$ |  $0.845~{}\pm~{}0.177$ |  $0.414~{}\pm~{}0.001$
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\textbf{39.84}~{}\pm~{}5.91$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{95.42}~{}\pm~{}\textbf{0.32}$ | $\cellcolor{NiceGreen!25}\textbf{85.58}~{}\pm~{}\textbf{1.35}$ | $\cellcolor{NiceGreen!25}\textbf{103.12}~{}\pm~{}0.25$
|  | $\rho$ |  $0.480~{}\pm~{}0.020$ | $0.491~{}\pm~{}0.006$ | $0.499~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $95.04~{}\pm~{}0.75$ | $84.40~{}\pm~{}2.21$ | $103.41~{}\pm~{}0.78$
|  | $\rho$ | $0.999~{}\pm~{}0.004$ | $1.003~{}\pm~{}0.005$ | $1.002~{}\pm~{}0.004$
| none | metric | $94.85~{}\pm~{}0.40$ | $84.92~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21~{}\pm~{}\textbf{0.24}$ |  $64.29~{}\pm~{}24.40$ | $\cellcolor{NiceGreen!25}\textbf{108.09}~{}\pm~{}\textbf{0.86}$
|  | $\rho$ | $0.503~{}\pm~{}0.006$ |  $0.536~{}\pm~{}0.026$ | $0.503~{}\pm~{}0.005$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.46}~{}\pm~{}0.38$ | $82.87~{}\pm~{}2.75$ | $109.93~{}\pm~{}1.90$
|  | $\rho$ |  $0.955~{}\pm~{}0.038$ | $0.997~{}\pm~{}0.014$ | $1.000~{}\pm~{}0.018$
| none | metric | $95.21~{}\pm~{}0.95$ | $\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.93~{}\pm~{}2.83$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{91.25}~{}\pm~{}\textbf{1.48}$ | $\cellcolor{NiceOrange!25}\textbf{83.18}~{}\pm~{}\textbf{0.95}$ |  $133.64~{}\pm~{}5.25$
|  | $\rho$ |  $0.769~{}\pm~{}0.003$ | $1.001~{}\pm~{}0.001$ |  $0.846~{}\pm~{}0.222$
| none | metric | $86.38~{}\pm~{}5.16$ | $70.26~{}\pm~{}7.12$ | $\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.225~{}\pm~{}0.033$ | $1.748~{}\pm~{}0.016$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{82.05}~{}\pm~{}6.26$ | $\cellcolor{NiceOrange!25}\textbf{73.98}~{}\pm~{}\textbf{3.75}$ |  $134.29~{}\pm~{}\textbf{1.44}$
|  | $\rho$ | $0.999~{}\pm~{}0.001$ | $1.000~{}\pm~{}0.000$ |  $0.687~{}\pm~{}0.005$
| none | metric | $81.42~{}\pm~{}\textbf{1.17}$ | $68.20~{}\pm~{}6.85$ | $\textbf{130.54}~{}\pm~{}5.09$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.130~{}\pm~{}0.003$ | $1.094~{}\pm~{}0.002$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better ($\uparrow$), and perplexity for PTB, the lower the better ($\downarrow$). The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.05$, and those experiments are currently running again. |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.19~{}\pm~{}0.30$ | $\cellcolor{NiceGreen!25}\textbf{14.71}~{}\pm~{}3.50$ |  $640.79~{}\pm~{}\textbf{0.14}$
|  | $\rho$ | $0.498~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ |  $0.202~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $11.14~{}\pm~{}0.77$ | $11.72~{}\pm~{}\textbf{1.46}$ |  $\cellcolor{NiceOrange!25}\textbf{538.82}~{}\pm~{}202.60$
|  | $\rho$ | $0.999~{}\pm~{}0.019$ | $1.016~{}\pm~{}0.019$ |  $0.944~{}\pm~{}0.109$
| none | metric | $\textbf{11.41}~{}\pm~{}\textbf{0.20}$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $17.15~{}\pm~{}4.90$ | $\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ | $644.26~{}\pm~{}\textbf{1.71}$
|  | $\rho$ | $0.505~{}\pm~{}0.008$ | $0.511~{}\pm~{}0.003$ | $0.501~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric | $17.90~{}\pm~{}\textbf{2.04}$ |  $20.93~{}\pm~{}\textbf{1.56}$ | 0
|  | $\rho$ | $1.002~{}\pm~{}0.006$ |  $0.859~{}\pm~{}0.163$ | 0
| none | metric | $\textbf{19.63}~{}\pm~{}6.60$ | $16.43~{}\pm~{}3.51$ | $\textbf{500.04}~{}\pm~{}94.49$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.61}~{}\pm~{}\textbf{0.28}$ | $86.47~{}\pm~{}\textbf{0.60}$ |  $641.27~{}\pm~{}\textbf{0.10}$
|  | $\rho$ | $0.502~{}\pm~{}0.002$ | $0.500~{}\pm~{}0.002$ |  $0.448~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $94.33~{}\pm~{}1.31$ | $86.73~{}\pm~{}1.42$ |  $\cellcolor{NiceOrange!25}\textbf{126.16}~{}\pm~{}6.97$
|  | $\rho$ | $0.999~{}\pm~{}0.008$ | $1.016~{}\pm~{}0.017$ |  $0.962~{}\pm~{}0.020$
| none | metric | $\textbf{95.61}~{}\pm~{}0.38$ | $\textbf{88.12}~{}\pm~{}1.39$ | $648.25~{}\pm~{}4.75$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.002$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.92~{}\pm~{}0.31$ | $\cellcolor{NiceGreen!25}\textbf{82.12}~{}\pm~{}\textbf{2.70}$ | $642.86~{}\pm~{}1.05$
|  | $\rho$ | $0.501~{}\pm~{}0.001$ | $0.506~{}\pm~{}0.008$ | $0.509~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $53.70~{}\pm~{}48.80$ | $80.17~{}\pm~{}8.50$ |  $\cellcolor{NiceOrange!25}\textbf{263.29}~{}\pm~{}251.11$
|  | $\rho$ | $0.996~{}\pm~{}0.006$ | $1.010~{}\pm~{}0.007$ |  $0.954~{}\pm~{}0.060$
| none | metric | $\textbf{96.19}~{}\pm~{}\textbf{0.29}$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}\textbf{0.59}$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $11.30~{}\pm~{}\textbf{0.13}$ | $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$ |  $\cellcolor{NiceOrange!25}\textbf{644.53}~{}\pm~{}\textbf{0.26}$
|  | $\rho$ |  $0.753~{}\pm~{}0.001$ | $1.000~{}\pm~{}0.007$ |  $0.621~{}\pm~{}0.003$
| none | metric | $\textbf{11.75}~{}\pm~{}0.71$ | $4.93~{}\pm~{}\textbf{0.14}$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $1.667~{}\pm~{}0.007$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.76~{}\pm~{}\textbf{0.80}$ | $\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$ |  $\cellcolor{NiceOrange!25}\textbf{643.45}~{}\pm~{}\textbf{0.11}$
|  | $\rho$ | $0.990~{}\pm~{}0.011$ | $1.000~{}\pm~{}0.001$ |  $0.621~{}\pm~{}0.003$
| none | metric | $\textbf{11.73}~{}\pm~{}1.54$ | $18.89~{}\pm~{}\textbf{5.68}$ | $661.75~{}\pm~{}0.75$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN$\sigma$ $\rho_{t}=0.5$ metric
$\cellcolor{NiceGreen!25}\textbf{41.29}~{}\pm~{}\textbf{3.82}$
$30.38~{}\pm~{}\textbf{6.97}$
$\cellcolor{NiceGreen!25}\textbf{14.71}~{}\pm~{}3.50$
$\cellcolor{NiceGreen!25}\textbf{7.99}~{}\pm~{}3.43$ $\rho$
$0.504~{}\pm~{}0.005$ $0.500~{}\pm~{}0.003$ $0.498~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $37.94~{}\pm~{}3.91$
$22.72~{}\pm~{}8.04$ $11.72~{}\pm~{}\textbf{1.46}$ $4.83~{}\pm~{}0.16$ $\rho$
$0.992~{}\pm~{}0.016$ $1.019~{}\pm~{}0.077$ $1.016~{}\pm~{}0.019$
$1.003~{}\pm~{}0.004$ none metric $38.20~{}\pm~{}6.84$
$\textbf{30.47}~{}\pm~{}13.34$ $12.90~{}\pm~{}3.82$
$5.06~{}\pm~{}\textbf{0.16}$ $\rho$ $0.366~{}\pm~{}0.001$
$0.362~{}\pm~{}0.001$ $0.361~{}\pm~{}0.001$ $0.361~{}\pm~{}0.000$ RNN$ReLU$
$\rho_{t}=0.5$ metric $37.48~{}\pm~{}9.97$ $26.52~{}\pm~{}6.23$
$\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ $18.27~{}\pm~{}2.88$
$\rho$ $0.512~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$
$0.512~{}\pm~{}0.001$ $\rho_{t}=1$ metric $36.28~{}\pm~{}\textbf{3.28}$
$28.24~{}\pm~{}\textbf{5.18}$ $20.93~{}\pm~{}\textbf{1.56}$
$\cellcolor{NiceOrange!25}\textbf{21.88}~{}\pm~{}4.96$ $\rho$
$0.821~{}\pm~{}0.199$ $1.004~{}\pm~{}0.010$ $0.859~{}\pm~{}0.163$
$0.731~{}\pm~{}0.007$ none metric $\textbf{39.06}~{}\pm~{}10.85$
$\textbf{34.19}~{}\pm~{}5.98$ $16.43~{}\pm~{}3.51$
$15.59~{}\pm~{}\textbf{2.31}$ $\rho$ $1.355~{}\pm~{}0.004$
$1.346~{}\pm~{}0.003$ $1.341~{}\pm~{}0.006$ $1.339~{}\pm~{}0.005$ GRU
$\rho_{t}=0.5$ metric $81.27~{}\pm~{}1.81$
$\cellcolor{NiceGreen!25}\textbf{86.05}~{}\pm~{}2.98$
$86.47~{}\pm~{}\textbf{0.60}$ $85.03~{}\pm~{}2.38$ $\rho$
$0.461~{}\pm~{}0.006$ $0.495~{}\pm~{}0.009$ $0.500~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $76.01~{}\pm~{}3.56$
$84.29~{}\pm~{}\textbf{1.79}$ $86.73~{}\pm~{}1.42$
$85.73~{}\pm~{}\textbf{1.72}$ $\rho$ $0.993~{}\pm~{}0.006$
$1.006~{}\pm~{}0.005$ $1.016~{}\pm~{}0.017$ $1.031~{}\pm~{}0.021$ none metric
$\textbf{84.55}~{}\pm~{}\textbf{1.45}$ $86.01~{}\pm~{}2.48$
$\textbf{88.12}~{}\pm~{}1.39$ $\textbf{87.40}~{}\pm~{}2.29$ $\rho$
$0.751~{}\pm~{}0.002$ $0.710~{}\pm~{}0.002$ $0.702~{}\pm~{}0.002$
$0.699~{}\pm~{}0.002$ LSTM $\rho_{t}=0.5$ metric $42.52~{}\pm~{}\textbf{7.18}$
$\cellcolor{NiceGreen!25}\textbf{87.14}~{}\pm~{}\textbf{0.69}$
$\cellcolor{NiceGreen!25}\textbf{82.12}~{}\pm~{}\textbf{2.70}$
$43.71~{}\pm~{}44.85$ $\rho$ $0.528~{}\pm~{}0.006$ $0.514~{}\pm~{}0.014$
$0.506~{}\pm~{}0.008$ $0.504~{}\pm~{}0.003$ $\rho_{t}=1$ metric
$67.39~{}\pm~{}25.89$ $84.15~{}\pm~{}1.81$ $80.17~{}\pm~{}8.50$
$\cellcolor{NiceOrange!25}\textbf{56.80}~{}\pm~{}25.30$ $\rho$
$0.978~{}\pm~{}0.032$ $1.005~{}\pm~{}0.007$ $1.010~{}\pm~{}0.007$
$1.030~{}\pm~{}0.028$ none metric $\textbf{76.69}~{}\pm~{}9.27$
$83.74~{}\pm~{}2.81$ $62.98~{}\pm~{}27.77$ $27.90~{}\pm~{}\textbf{7.79}$
$\rho$ $0.914~{}\pm~{}0.003$ $0.879~{}\pm~{}0.002$ $0.871~{}\pm~{}0.001$
$0.869~{}\pm~{}0.002$ ALIF $\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{78.10}~{}\pm~{}\textbf{1.24}$
$52.33~{}\pm~{}18.36$ $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$
$\cellcolor{NiceOrange!25}\textbf{5.59}~{}\pm~{}1.53$ $\rho$
$1.003~{}\pm~{}0.005$ $0.932~{}\pm~{}0.129$ $1.000~{}\pm~{}0.007$
$1.003~{}\pm~{}0.011$ none metric $77.18~{}\pm~{}1.59$
$\textbf{57.88}~{}\pm~{}\textbf{6.96}$ $4.93~{}\pm~{}\textbf{0.14}$
$4.93~{}\pm~{}\textbf{0.23}$ $\rho$ $1.853~{}\pm~{}0.023$
$1.701~{}\pm~{}0.011$ $1.667~{}\pm~{}0.007$ $1.656~{}\pm~{}0.005$ ALIF±
$\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{67.74}~{}\pm~{}\textbf{1.52}$
$\cellcolor{NiceOrange!25}\textbf{66.92}~{}\pm~{}\textbf{2.50}$
$\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$
$\cellcolor{NiceOrange!25}\textbf{11.96}~{}\pm~{}6.76$ $\rho$
$1.000~{}\pm~{}0.003$ $1.001~{}\pm~{}0.002$ $1.000~{}\pm~{}0.001$
$0.917~{}\pm~{}0.166$ none metric $66.83~{}\pm~{}4.23$ $60.47~{}\pm~{}7.05$
$18.89~{}\pm~{}\textbf{5.68}$ $5.06~{}\pm~{}\textbf{0.18}$ $\rho$
$1.460~{}\pm~{}0.007$ $1.016~{}\pm~{}0.001$ $0.979~{}\pm~{}0.002$
$0.933~{}\pm~{}0.001$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  4 May 21

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{29.38}~{}\pm~{}6.65$ | $34.66~{}\pm~{}8.30$ | $\cellcolor{NiceGreen!25}\textbf{116.81}~{}\pm~{}7.31$
|  | $\rho$ | $0.492~{}\pm~{}0.008$ | $0.496~{}\pm~{}0.007$ | $0.497~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $26.39~{}\pm~{}\textbf{1.49}$ | $35.10~{}\pm~{}\textbf{7.45}$ |  $395.35~{}\pm~{}285.28$
|  | $\rho$ |  $0.900~{}\pm~{}0.142$ | $1.019~{}\pm~{}0.019$ |  $0.574~{}\pm~{}0.409$
| none | metric | $29.02~{}\pm~{}2.00$ | $\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{28.12}~{}\pm~{}3.41$ | $35.46~{}\pm~{}10.64$ |  $\cellcolor{NiceGreen!25}\textbf{156.57}~{}\pm~{}6.95$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.508~{}\pm~{}0.003$ |  $0.415~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $24.80~{}\pm~{}5.44$ |  $34.59~{}\pm~{}\textbf{5.35}$ |  $162.79~{}\pm~{}15.62$
|  | $\rho$ | $0.997~{}\pm~{}0.009$ |  $0.845~{}\pm~{}0.177$ |  $0.430~{}\pm~{}0.004$
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\textbf{39.84}~{}\pm~{}5.91$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{95.28}~{}\pm~{}\textbf{0.17}$ | $\cellcolor{NiceGreen!25}\textbf{85.58}~{}\pm~{}\textbf{1.35}$ | $\cellcolor{NiceGreen!25}\textbf{103.12}~{}\pm~{}0.25$
|  | $\rho$ |  $0.480~{}\pm~{}0.020$ | $0.491~{}\pm~{}0.006$ | $0.499~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $95.04~{}\pm~{}0.75$ | $84.40~{}\pm~{}2.21$ | $103.41~{}\pm~{}0.78$
|  | $\rho$ | $0.999~{}\pm~{}0.004$ | $1.003~{}\pm~{}0.005$ | $1.002~{}\pm~{}0.004$
| none | metric | $94.85~{}\pm~{}0.40$ | $84.92~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21~{}\pm~{}\textbf{0.24}$ |  $76.46~{}\pm~{}16.26$ | $109.01~{}\pm~{}\textbf{1.97}$
|  | $\rho$ | $0.503~{}\pm~{}0.006$ |  $0.536~{}\pm~{}0.026$ | $0.511~{}\pm~{}0.016$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.55}~{}\pm~{}0.45$ | $\cellcolor{NiceOrange!25}\textbf{83.77}~{}\pm~{}2.36$ | $110.08~{}\pm~{}2.19$
|  | $\rho$ |  $0.955~{}\pm~{}0.038$ | $0.997~{}\pm~{}0.014$ | $1.000~{}\pm~{}0.018$
| none | metric | $95.21~{}\pm~{}0.95$ | $83.61~{}\pm~{}\textbf{1.97}$ | $\textbf{108.93}~{}\pm~{}2.83$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{89.12}~{}\pm~{}5.42$ | $\cellcolor{NiceOrange!25}\textbf{83.18}~{}\pm~{}\textbf{0.95}$ |  $143.20~{}\pm~{}12.42$
|  | $\rho$ |  $0.776~{}\pm~{}0.009$ | $1.001~{}\pm~{}0.001$ |  $0.792~{}\pm~{}0.183$
| none | metric | $86.38~{}\pm~{}\textbf{5.16}$ | $70.26~{}\pm~{}7.12$ | $\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.225~{}\pm~{}0.033$ | $1.748~{}\pm~{}0.016$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{82.05}~{}\pm~{}6.26$ | $\cellcolor{NiceOrange!25}\textbf{73.98}~{}\pm~{}\textbf{3.75}$ |  $138.79~{}\pm~{}\textbf{1.02}$
|  | $\rho$ | $0.999~{}\pm~{}0.001$ | $1.000~{}\pm~{}0.000$ |  $0.686~{}\pm~{}0.004$
| none | metric | $81.42~{}\pm~{}\textbf{1.17}$ | $68.20~{}\pm~{}6.85$ | $\textbf{130.54}~{}\pm~{}5.09$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.130~{}\pm~{}0.003$ | $1.094~{}\pm~{}0.002$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better ($\uparrow$), and perplexity for PTB, the lower the better ($\downarrow$). The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.05$, and those experiments are currently running again. |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.19~{}\pm~{}0.30$ | $14.71~{}\pm~{}\textbf{3.50}$ |  $640.79~{}\pm~{}\textbf{0.14}$
|  | $\rho$ | $0.498~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ |  $0.201~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $11.14~{}\pm~{}0.77$ | $\cellcolor{NiceOrange!25}\textbf{15.36}~{}\pm~{}3.66$ | $\cellcolor{NiceOrange!25}\textbf{545.41}~{}\pm~{}189.42$
|  | $\rho$ | $0.999~{}\pm~{}0.019$ | $1.016~{}\pm~{}0.019$ | $0.998~{}\pm~{}0.073$
| none | metric | $\textbf{11.41}~{}\pm~{}\textbf{0.20}$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $17.15~{}\pm~{}4.90$ | $\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ | $644.26~{}\pm~{}\textbf{1.71}$
|  | $\rho$ | $0.505~{}\pm~{}0.008$ | $0.511~{}\pm~{}0.003$ | $0.501~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric | $17.90~{}\pm~{}\textbf{2.04}$ |  $21.82~{}\pm~{}\textbf{3.11}$ | 0
|  | $\rho$ | $1.002~{}\pm~{}0.006$ |  $0.859~{}\pm~{}0.163$ | 0
| none | metric | $\textbf{19.63}~{}\pm~{}6.60$ | $16.43~{}\pm~{}3.51$ | $\textbf{500.04}~{}\pm~{}94.49$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.61}~{}\pm~{}\textbf{0.28}$ | $86.47~{}\pm~{}\textbf{0.60}$ |  $641.27~{}\pm~{}\textbf{0.10}$
|  | $\rho$ | $0.502~{}\pm~{}0.002$ | $0.500~{}\pm~{}0.002$ |  $0.448~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $94.33~{}\pm~{}1.31$ | $87.10~{}\pm~{}1.12$ | $\cellcolor{NiceOrange!25}\textbf{130.28}~{}\pm~{}19.75$
|  | $\rho$ | $0.999~{}\pm~{}0.008$ | $1.016~{}\pm~{}0.017$ | $0.983~{}\pm~{}0.018$
| none | metric | $\textbf{95.61}~{}\pm~{}0.38$ | $\textbf{88.12}~{}\pm~{}1.39$ | $648.25~{}\pm~{}4.75$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.002$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.92~{}\pm~{}0.31$ | $82.12~{}\pm~{}2.70$ | $642.86~{}\pm~{}1.05$
|  | $\rho$ | $0.501~{}\pm~{}0.001$ | $0.506~{}\pm~{}0.008$ | $0.509~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $53.70~{}\pm~{}48.80$ | $\cellcolor{NiceOrange!25}\textbf{83.64}~{}\pm~{}\textbf{2.21}$ | $\cellcolor{NiceOrange!25}\textbf{264.89}~{}\pm~{}250.24$
|  | $\rho$ | $0.996~{}\pm~{}0.006$ | $1.010~{}\pm~{}0.007$ | $0.985~{}\pm~{}0.035$
| none | metric | $\textbf{96.19}~{}\pm~{}\textbf{0.29}$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}\textbf{0.59}$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $11.26~{}\pm~{}\textbf{0.10}$ | $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$ |  $\cellcolor{NiceOrange!25}\textbf{644.51}~{}\pm~{}\textbf{0.31}$
|  | $\rho$ |  $0.747~{}\pm~{}0.035$ | $1.000~{}\pm~{}0.007$ |  $0.621~{}\pm~{}0.003$
| none | metric | $\textbf{11.75}~{}\pm~{}0.71$ | $4.93~{}\pm~{}\textbf{0.14}$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $1.667~{}\pm~{}0.007$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.76~{}\pm~{}\textbf{0.80}$ | $\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$ |  $\cellcolor{NiceOrange!25}\textbf{643.48}~{}\pm~{}\textbf{0.11}$
|  | $\rho$ | $0.990~{}\pm~{}0.011$ | $1.000~{}\pm~{}0.001$ |  $0.620~{}\pm~{}0.003$
| none | metric | $\textbf{11.73}~{}\pm~{}1.54$ | $18.89~{}\pm~{}\textbf{5.68}$ | $661.75~{}\pm~{}0.75$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN $\sigma$ $\rho_{t}=0.5$ metric
$\cellcolor{NiceGreen!25}\textbf{41.29}~{}\pm~{}\textbf{3.82}$
$30.38~{}\pm~{}\textbf{6.97}$ $14.71~{}\pm~{}\textbf{3.50}$
$\cellcolor{NiceGreen!25}\textbf{7.99}~{}\pm~{}3.43$ $\rho$
$0.504~{}\pm~{}0.005$ $0.500~{}\pm~{}0.003$ $0.498~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $37.96~{}\pm~{}3.93$
$20.87~{}\pm~{}7.19$ $\cellcolor{NiceOrange!25}\textbf{15.36}~{}\pm~{}3.66$
$4.83~{}\pm~{}0.16$ $\rho$ $0.992~{}\pm~{}0.016$ $1.019~{}\pm~{}0.077$
$1.016~{}\pm~{}0.019$ $1.003~{}\pm~{}0.004$ none metric $38.20~{}\pm~{}6.84$
$\textbf{30.47}~{}\pm~{}13.34$ $12.90~{}\pm~{}3.82$
$5.06~{}\pm~{}\textbf{0.16}$ $\rho$ $0.366~{}\pm~{}0.001$
$0.362~{}\pm~{}0.001$ $0.361~{}\pm~{}0.001$ $0.361~{}\pm~{}0.000$ RNN $ReLU$
$\rho_{t}=0.5$ metric $37.48~{}\pm~{}9.97$ $26.52~{}\pm~{}6.23$
$\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ $18.27~{}\pm~{}2.88$
$\rho$ $0.512~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$
$0.512~{}\pm~{}0.001$ $\rho_{t}=1$ metric $36.79~{}\pm~{}\textbf{2.26}$
$28.24~{}\pm~{}\textbf{5.18}$ $21.82~{}\pm~{}\textbf{3.11}$
$\cellcolor{NiceOrange!25}\textbf{19.74}~{}\pm~{}7.30$ $\rho$
$0.821~{}\pm~{}0.199$ $1.004~{}\pm~{}0.010$ $0.859~{}\pm~{}0.163$
$0.731~{}\pm~{}0.007$ none metric $\textbf{39.06}~{}\pm~{}10.85$
$\textbf{34.19}~{}\pm~{}5.98$ $16.43~{}\pm~{}3.51$
$15.59~{}\pm~{}\textbf{2.31}$ $\rho$ $1.355~{}\pm~{}0.004$
$1.346~{}\pm~{}0.003$ $1.341~{}\pm~{}0.006$ $1.339~{}\pm~{}0.005$ GRU
$\rho_{t}=0.5$ metric $80.95~{}\pm~{}2.78$
$\cellcolor{NiceGreen!25}\textbf{86.05}~{}\pm~{}2.98$
$86.47~{}\pm~{}\textbf{0.60}$ $85.03~{}\pm~{}2.38$ $\rho$
$0.461~{}\pm~{}0.006$ $0.495~{}\pm~{}0.009$ $0.500~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $76.01~{}\pm~{}3.56$
$84.29~{}\pm~{}\textbf{1.79}$ $87.10~{}\pm~{}1.12$
$85.73~{}\pm~{}\textbf{1.72}$ $\rho$ $0.993~{}\pm~{}0.006$
$1.006~{}\pm~{}0.005$ $1.016~{}\pm~{}0.017$ $1.031~{}\pm~{}0.021$ none metric
$\textbf{84.55}~{}\pm~{}\textbf{1.45}$ $86.01~{}\pm~{}2.48$
$\textbf{88.12}~{}\pm~{}1.39$ $\textbf{87.40}~{}\pm~{}2.29$ $\rho$
$0.751~{}\pm~{}0.002$ $0.710~{}\pm~{}0.002$ $0.702~{}\pm~{}0.002$
$0.699~{}\pm~{}0.002$ LSTM $\rho_{t}=0.5$ metric $42.58~{}\pm~{}\textbf{1.69}$
$\cellcolor{NiceGreen!25}\textbf{87.14}~{}\pm~{}\textbf{0.69}$
$82.12~{}\pm~{}2.70$ $43.71~{}\pm~{}44.85$ $\rho$ $0.528~{}\pm~{}0.006$
$0.514~{}\pm~{}0.014$ $0.506~{}\pm~{}0.008$ $0.504~{}\pm~{}0.003$ $\rho_{t}=1$
metric $59.35~{}\pm~{}24.95$ $84.15~{}\pm~{}1.81$
$\cellcolor{NiceOrange!25}\textbf{83.64}~{}\pm~{}\textbf{2.21}$
$\cellcolor{NiceOrange!25}\textbf{57.91}~{}\pm~{}24.69$ $\rho$
$0.978~{}\pm~{}0.032$ $1.005~{}\pm~{}0.007$ $1.010~{}\pm~{}0.007$
$1.030~{}\pm~{}0.028$ none metric $\textbf{76.69}~{}\pm~{}9.27$
$83.74~{}\pm~{}2.81$ $62.98~{}\pm~{}27.77$ $27.90~{}\pm~{}\textbf{7.79}$
$\rho$ $0.914~{}\pm~{}0.003$ $0.879~{}\pm~{}0.002$ $0.871~{}\pm~{}0.001$
$0.869~{}\pm~{}0.002$ ALIF $\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{78.10}~{}\pm~{}\textbf{1.24}$
$52.38~{}\pm~{}18.44$ $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$
$\cellcolor{NiceOrange!25}\textbf{5.59}~{}\pm~{}1.53$ $\rho$
$1.003~{}\pm~{}0.005$ $0.932~{}\pm~{}0.129$ $1.000~{}\pm~{}0.007$
$1.003~{}\pm~{}0.011$ none metric $77.18~{}\pm~{}1.59$
$\textbf{57.88}~{}\pm~{}\textbf{6.96}$ $4.93~{}\pm~{}\textbf{0.14}$
$4.93~{}\pm~{}\textbf{0.23}$ $\rho$ $1.853~{}\pm~{}0.023$
$1.701~{}\pm~{}0.011$ $1.667~{}\pm~{}0.007$ $1.656~{}\pm~{}0.005$ ALIF±
$\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{67.74}~{}\pm~{}\textbf{1.52}$
$\cellcolor{NiceOrange!25}\textbf{66.92}~{}\pm~{}\textbf{2.50}$
$\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$
$\cellcolor{NiceOrange!25}\textbf{11.96}~{}\pm~{}6.76$ $\rho$
$1.000~{}\pm~{}0.003$ $1.001~{}\pm~{}0.002$ $1.000~{}\pm~{}0.001$
$0.917~{}\pm~{}0.166$ none metric $66.83~{}\pm~{}4.23$ $60.47~{}\pm~{}7.05$
$18.89~{}\pm~{}\textbf{5.68}$ $5.06~{}\pm~{}\textbf{0.18}$ $\rho$
$1.460~{}\pm~{}0.007$ $1.016~{}\pm~{}0.001$ $0.979~{}\pm~{}0.002$
$0.933~{}\pm~{}0.001$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  5 June 1

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{29.38}~{}\pm~{}6.65$ | $34.66~{}\pm~{}8.30$ | $\cellcolor{NiceGreen!25}\textbf{116.81}~{}\pm~{}7.31$
|  | $\rho$ | $0.492~{}\pm~{}0.008$ | $0.496~{}\pm~{}0.007$ | $0.497~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $26.98~{}\pm~{}3.29$ | $36.37~{}\pm~{}\textbf{5.23}$ |  $276.09~{}\pm~{}244.64$
|  | $\rho$ |  $0.900~{}\pm~{}0.142$ | $1.019~{}\pm~{}0.019$ |  $0.671~{}\pm~{}0.335$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{28.12}~{}\pm~{}3.41$ | $35.46~{}\pm~{}10.64$ |  $\cellcolor{NiceGreen!25}\textbf{158.79}~{}\pm~{}10.43$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.508~{}\pm~{}0.003$ |  $0.416~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $24.80~{}\pm~{}5.44$ |  $33.34~{}\pm~{}\textbf{4.90}$ |  $167.83~{}\pm~{}13.57$
|  | $\rho$ | $0.997~{}\pm~{}0.009$ |  $0.845~{}\pm~{}0.177$ |  $0.430~{}\pm~{}0.004$
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\textbf{39.84}~{}\pm~{}5.91$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{95.28}~{}\pm~{}\textbf{0.17}$ | $\cellcolor{NiceGreen!25}\textbf{85.58}~{}\pm~{}\textbf{1.35}$ | $\cellcolor{NiceGreen!25}\textbf{103.12}~{}\pm~{}0.25$
|  | $\rho$ |  $0.480~{}\pm~{}0.020$ | $0.491~{}\pm~{}0.006$ | $0.499~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $95.04~{}\pm~{}0.75$ | $84.40~{}\pm~{}2.21$ | $103.41~{}\pm~{}0.78$
|  | $\rho$ | $0.999~{}\pm~{}0.004$ | $1.003~{}\pm~{}0.005$ | $1.002~{}\pm~{}0.004$
| none | metric | $94.85~{}\pm~{}0.40$ | $84.92~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.21~{}\pm~{}\textbf{0.24}$ |  $66.83~{}\pm~{}20.71$ | $109.22~{}\pm~{}2.37$
|  | $\rho$ | $0.503~{}\pm~{}0.006$ |  $0.536~{}\pm~{}0.026$ | $0.511~{}\pm~{}0.016$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.60}~{}\pm~{}0.40$ | $82.87~{}\pm~{}2.75$ | $109.93~{}\pm~{}\textbf{1.90}$
|  | $\rho$ |  $0.955~{}\pm~{}0.038$ | $0.997~{}\pm~{}0.014$ | $1.000~{}\pm~{}0.018$
| none | metric | $95.21~{}\pm~{}0.95$ | $\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $\textbf{108.93}~{}\pm~{}2.83$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{86.45}~{}\pm~{}\textbf{3.30}$ | $\cellcolor{NiceOrange!25}\textbf{83.18}~{}\pm~{}\textbf{0.95}$ |  $139.21~{}\pm~{}13.13$
|  | $\rho$ |  $0.768~{}\pm~{}0.003$ | $1.001~{}\pm~{}0.001$ |  $0.899~{}\pm~{}0.148$
| none | metric | $86.38~{}\pm~{}5.16$ | $70.26~{}\pm~{}7.12$ | $\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.225~{}\pm~{}0.033$ | $1.748~{}\pm~{}0.016$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{82.05}~{}\pm~{}6.26$ | $\cellcolor{NiceOrange!25}\textbf{73.98}~{}\pm~{}\textbf{3.75}$ |  $138.80~{}\pm~{}nan$
|  | $\rho$ | $0.999~{}\pm~{}0.001$ | $1.000~{}\pm~{}0.000$ |  $0.691~{}\pm~{}nan$
| none | metric | $81.42~{}\pm~{}\textbf{1.17}$ | $68.20~{}\pm~{}6.85$ | $\textbf{130.54}~{}\pm~{}\textbf{5.09}$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.130~{}\pm~{}0.003$ | $1.094~{}\pm~{}0.002$
|  |  |  |  |
Table : Effect of stabilization across tasks and networks. The network preparation by pre-training to stability, generally helps performance, with $\rho_{t}=0.5$ generally being preferable over $\rho_{t}=1$ across tasks and networks, with networks of depth $d=2$. Here we report the test set results across three different tasks. Metrics are accuracy for sl-MNIST and SHD, the higher the better ($\uparrow$), and perplexity for PTB, the lower the better ($\downarrow$). The non-differentiable networks, were often unable to reach a target norm of $\rho_{t}=0.5$, but the preparation to $\rho_{t}=1$ was still often beneficial. The strike through the number means that it wasn’t close enough to the target radius, where close enough means at a distance of $0.05$, and those experiments are currently running again. |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.19~{}\pm~{}0.30$ | $\cellcolor{NiceGreen!25}\textbf{14.71}~{}\pm~{}\textbf{3.50}$ |  $640.79~{}\pm~{}\textbf{0.14}$
|  | $\rho$ | $0.498~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ |  $0.201~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $11.14~{}\pm~{}0.77$ | $14.21~{}\pm~{}4.50$ | $\cellcolor{NiceOrange!25}\textbf{536.77}~{}\pm~{}206.68$
|  | $\rho$ | $0.999~{}\pm~{}0.019$ | $1.016~{}\pm~{}0.019$ | $0.998~{}\pm~{}0.073$
| none | metric | $\textbf{11.41}~{}\pm~{}\textbf{0.20}$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $17.15~{}\pm~{}4.90$ | $\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ | $644.26~{}\pm~{}\textbf{1.71}$
|  | $\rho$ | $0.505~{}\pm~{}0.008$ | $0.511~{}\pm~{}0.003$ | $0.501~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric | $17.90~{}\pm~{}\textbf{2.04}$ |  $23.54~{}\pm~{}5.61$ | 0
|  | $\rho$ | $1.002~{}\pm~{}0.006$ |  $0.859~{}\pm~{}0.163$ | 0
| none | metric | $\textbf{19.63}~{}\pm~{}6.60$ | $16.43~{}\pm~{}\textbf{3.51}$ | $\textbf{500.04}~{}\pm~{}94.49$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.61}~{}\pm~{}\textbf{0.28}$ | $86.47~{}\pm~{}\textbf{0.60}$ |  $641.27~{}\pm~{}\textbf{0.10}$
|  | $\rho$ | $0.502~{}\pm~{}0.002$ | $0.500~{}\pm~{}0.002$ |  $0.448~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $94.33~{}\pm~{}1.31$ | $86.55~{}\pm~{}2.16$ | $\cellcolor{NiceOrange!25}\textbf{129.92}~{}\pm~{}13.24$
|  | $\rho$ | $0.999~{}\pm~{}0.008$ | $1.016~{}\pm~{}0.017$ | $0.983~{}\pm~{}0.018$
| none | metric | $\textbf{95.61}~{}\pm~{}0.38$ | $\textbf{88.12}~{}\pm~{}1.39$ | $648.25~{}\pm~{}4.75$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.002$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $95.92~{}\pm~{}0.31$ | $82.12~{}\pm~{}2.70$ | $642.86~{}\pm~{}1.05$
|  | $\rho$ | $0.501~{}\pm~{}0.001$ | $0.506~{}\pm~{}0.008$ | $0.509~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $53.70~{}\pm~{}48.80$ | $\cellcolor{NiceOrange!25}\textbf{84.51}~{}\pm~{}\textbf{1.77}$ | $\cellcolor{NiceOrange!25}\textbf{267.78}~{}\pm~{}248.18$
|  | $\rho$ | $0.996~{}\pm~{}0.006$ | $1.010~{}\pm~{}0.007$ | $0.985~{}\pm~{}0.035$
| none | metric | $\textbf{96.19}~{}\pm~{}\textbf{0.29}$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}\textbf{0.59}$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $11.43~{}\pm~{}\textbf{0.21}$ | $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$ |  $\cellcolor{NiceOrange!25}\textbf{645.99}~{}\pm~{}\textbf{0.82}$
|  | $\rho$ |  $0.754~{}\pm~{}0.002$ | $1.000~{}\pm~{}0.007$ |  $0.963~{}\pm~{}0.029$
| none | metric | $\textbf{11.75}~{}\pm~{}0.71$ | $4.93~{}\pm~{}\textbf{0.14}$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $1.667~{}\pm~{}0.007$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.76~{}\pm~{}\textbf{0.80}$ | $\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$ |  $\cellcolor{NiceOrange!25}\textbf{642.95}~{}\pm~{}\textbf{0.37}$
|  | $\rho$ | $0.990~{}\pm~{}0.011$ | $1.000~{}\pm~{}0.001$ |  $0.676~{}\pm~{}0.010$
| none | metric | $\textbf{11.73}~{}\pm~{}1.54$ | $18.89~{}\pm~{}\textbf{5.68}$ | $661.75~{}\pm~{}0.75$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : Depth 5

1.1!  depth 1 3 5 7 net LSC RNN $\sigma$ $\rho_{t}=0.5$ metric
$\cellcolor{NiceGreen!25}\textbf{41.29}~{}\pm~{}\textbf{3.82}$
$30.38~{}\pm~{}\textbf{6.97}$
$\cellcolor{NiceGreen!25}\textbf{14.71}~{}\pm~{}\textbf{3.50}$
$\cellcolor{NiceGreen!25}\textbf{7.99}~{}\pm~{}3.43$ $\rho$
$0.504~{}\pm~{}0.005$ $0.500~{}\pm~{}0.003$ $0.498~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $37.96~{}\pm~{}3.93$
$23.16~{}\pm~{}8.02$ $14.21~{}\pm~{}4.50$ $4.83~{}\pm~{}0.16$ $\rho$
$0.992~{}\pm~{}0.016$ $1.019~{}\pm~{}0.077$ $1.016~{}\pm~{}0.019$
$1.003~{}\pm~{}0.004$ none metric $38.20~{}\pm~{}6.84$
$\textbf{30.47}~{}\pm~{}13.34$ $12.90~{}\pm~{}3.82$
$5.06~{}\pm~{}\textbf{0.16}$ $\rho$ $0.366~{}\pm~{}0.001$
$0.362~{}\pm~{}0.001$ $0.361~{}\pm~{}0.001$ $0.361~{}\pm~{}0.000$ RNN $ReLU$
$\rho_{t}=0.5$ metric $37.48~{}\pm~{}9.97$ $26.52~{}\pm~{}6.23$
$\cellcolor{NiceGreen!25}\textbf{24.90}~{}\pm~{}3.62$ $18.27~{}\pm~{}2.88$
$\rho$ $0.512~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$ $0.511~{}\pm~{}0.003$
$0.512~{}\pm~{}0.001$ $\rho_{t}=1$ metric $35.68~{}\pm~{}\textbf{2.22}$
$28.24~{}\pm~{}\textbf{5.18}$ $23.54~{}\pm~{}5.61$
$\cellcolor{NiceOrange!25}\textbf{19.01}~{}\pm~{}\textbf{2.04}$ $\rho$
$0.821~{}\pm~{}0.199$ $1.004~{}\pm~{}0.010$ $0.859~{}\pm~{}0.163$
$0.731~{}\pm~{}0.007$ none metric $\textbf{39.06}~{}\pm~{}10.85$
$\textbf{34.19}~{}\pm~{}5.98$ $16.43~{}\pm~{}\textbf{3.51}$
$15.59~{}\pm~{}2.31$ $\rho$ $1.355~{}\pm~{}0.004$ $1.346~{}\pm~{}0.003$
$1.341~{}\pm~{}0.006$ $1.339~{}\pm~{}0.005$ GRU $\rho_{t}=0.5$ metric
$81.51~{}\pm~{}1.56$ $\cellcolor{NiceGreen!25}\textbf{86.05}~{}\pm~{}2.98$
$86.47~{}\pm~{}\textbf{0.60}$ $85.03~{}\pm~{}2.38$ $\rho$
$0.461~{}\pm~{}0.006$ $0.495~{}\pm~{}0.009$ $0.500~{}\pm~{}0.002$
$0.500~{}\pm~{}0.002$ $\rho_{t}=1$ metric $76.01~{}\pm~{}3.56$
$84.29~{}\pm~{}\textbf{1.79}$ $86.55~{}\pm~{}2.16$
$85.81~{}\pm~{}\textbf{1.74}$ $\rho$ $0.993~{}\pm~{}0.006$
$1.006~{}\pm~{}0.005$ $1.016~{}\pm~{}0.017$ $1.031~{}\pm~{}0.021$ none metric
$\textbf{84.55}~{}\pm~{}\textbf{1.45}$ $86.01~{}\pm~{}2.48$
$\textbf{88.12}~{}\pm~{}1.39$ $\textbf{87.40}~{}\pm~{}2.29$ $\rho$
$0.751~{}\pm~{}0.002$ $0.710~{}\pm~{}0.002$ $0.702~{}\pm~{}0.002$
$0.699~{}\pm~{}0.002$ LSTM $\rho_{t}=0.5$ metric $50.44~{}\pm~{}21.78$
$\cellcolor{NiceGreen!25}\textbf{86.75}~{}\pm~{}\textbf{0.44}$
$82.12~{}\pm~{}2.70$ $43.71~{}\pm~{}44.85$ $\rho$ $0.528~{}\pm~{}0.006$
$0.514~{}\pm~{}0.014$ $0.506~{}\pm~{}0.008$ $0.504~{}\pm~{}0.003$ $\rho_{t}=1$
metric $55.03~{}\pm~{}20.35$ $84.15~{}\pm~{}1.81$
$\cellcolor{NiceOrange!25}\textbf{84.51}~{}\pm~{}\textbf{1.77}$
$\cellcolor{NiceOrange!25}\textbf{66.24}~{}\pm~{}24.62$ $\rho$
$0.978~{}\pm~{}0.032$ $1.005~{}\pm~{}0.007$ $1.010~{}\pm~{}0.007$
$1.030~{}\pm~{}0.028$ none metric $\textbf{76.69}~{}\pm~{}\textbf{9.27}$
$83.74~{}\pm~{}2.81$ $62.98~{}\pm~{}27.77$ $27.90~{}\pm~{}\textbf{7.79}$
$\rho$ $0.914~{}\pm~{}0.003$ $0.879~{}\pm~{}0.002$ $0.871~{}\pm~{}0.001$
$0.869~{}\pm~{}0.002$ ALIF $\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{78.10}~{}\pm~{}\textbf{1.24}$
$45.72~{}\pm~{}9.27$ $\cellcolor{NiceOrange!25}\textbf{6.17}~{}\pm~{}1.86$
$\cellcolor{NiceOrange!25}\textbf{5.59}~{}\pm~{}1.53$ $\rho$
$1.003~{}\pm~{}0.005$ $0.932~{}\pm~{}0.129$ $1.000~{}\pm~{}0.007$
$1.003~{}\pm~{}0.011$ none metric $77.18~{}\pm~{}1.59$
$\textbf{57.88}~{}\pm~{}\textbf{6.96}$ $4.93~{}\pm~{}\textbf{0.14}$
$4.93~{}\pm~{}\textbf{0.23}$ $\rho$ $1.853~{}\pm~{}0.023$
$1.701~{}\pm~{}0.011$ $1.667~{}\pm~{}0.007$ $1.656~{}\pm~{}0.005$ ALIF±
$\rho_{t}=1$ metric
$\cellcolor{NiceOrange!25}\textbf{67.74}~{}\pm~{}\textbf{1.52}$
$\cellcolor{NiceOrange!25}\textbf{66.92}~{}\pm~{}\textbf{2.50}$
$\cellcolor{NiceOrange!25}\textbf{40.96}~{}\pm~{}20.69$
$\cellcolor{NiceOrange!25}\textbf{11.96}~{}\pm~{}6.76$ $\rho$
$1.000~{}\pm~{}0.003$ $1.001~{}\pm~{}0.002$ $1.000~{}\pm~{}0.001$
$0.917~{}\pm~{}0.166$ none metric $66.83~{}\pm~{}4.23$ $60.47~{}\pm~{}7.05$
$18.89~{}\pm~{}\textbf{5.68}$ $5.06~{}\pm~{}\textbf{0.18}$ $\rho$
$1.460~{}\pm~{}0.007$ $1.016~{}\pm~{}0.001$ $0.979~{}\pm~{}0.002$
$0.933~{}\pm~{}0.001$

Table : Effect of stabilization across depths and networks. Networks of
different depths are trained on the SHD task. All seem to benefit of the
stabilization technique used before training, and especially when the networks
are prepared to have a $\rho_{t}=0.5$ as the networks become deeper.

##  6 23 June 2023

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{35.15}~{}\pm~{}7.92$ |  $38.67~{}\pm~{}\textbf{5.19}$ |  $\cellcolor{NiceGreen!25}\textbf{137.09}~{}\pm~{}31.40$
|  | $\rho$ |  $0.530~{}\pm~{}0.017$ |  $0.422~{}\pm~{}0.068$ |  $0.348~{}\pm~{}0.119$
| $\rho_{t}=1$ | metric |  $21.75~{}\pm~{}8.29$ |  $18.49~{}\pm~{}13.33$ |  $49421654336997193357983744.00~{}\pm~{}nan$
|  | $\rho$ |  $1.090~{}\pm~{}0.136$ |  $1.585~{}\pm~{}0.816$ |  $1.305~{}\pm~{}nan$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\cellcolor{NiceBlue!25}\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{40.37}~{}\pm~{}8.51$ | $33.80~{}\pm~{}5.96$ |  $186.85~{}\pm~{}102.26$
|  | $\rho$ |  $0.541~{}\pm~{}0.020$ | $0.500~{}\pm~{}0.004$ |  $0.474~{}\pm~{}0.011$
| $\rho_{t}=1$ | metric |  $29.06~{}\pm~{}3.84$ |  $26.69~{}\pm~{}\textbf{5.69}$ |  $\cellcolor{NiceOrange!25}\textbf{125.23}~{}\pm~{}nan$
|  | $\rho$ |  $1.031~{}\pm~{}0.036$ |  $1.064~{}\pm~{}0.091$ |  $0.930~{}\pm~{}nan$
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\cellcolor{NiceBlue!25}\textbf{39.84}~{}\pm~{}5.91$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $95.22~{}\pm~{}\textbf{0.21}$ | $84.09~{}\pm~{}\textbf{0.97}$ | $\cellcolor{NiceGreen!25}\textbf{100.28}~{}\pm~{}nan$
|  | $\rho$ | $0.505~{}\pm~{}0.015$ | $0.505~{}\pm~{}0.012$ | $0.502~{}\pm~{}nan$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.40}~{}\pm~{}0.80$ |  $77.87~{}\pm~{}7.69$ |  $101.77~{}\pm~{}2.51$
|  | $\rho$ |  $1.382~{}\pm~{}0.436$ |  $1.071~{}\pm~{}0.040$ |  $0.978~{}\pm~{}0.047$
| none | metric | $94.85~{}\pm~{}0.40$ | $\cellcolor{NiceBlue!25}\textbf{84.92}~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric |  $74.67~{}\pm~{}42.45$ | $59.41~{}\pm~{}23.88$ | $\cellcolor{NiceGreen!25}\textbf{108.16}~{}\pm~{}\textbf{1.77}$
|  | $\rho$ |  $0.525~{}\pm~{}0.014$ | $0.499~{}\pm~{}0.001$ | $0.510~{}\pm~{}0.005$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.49}~{}\pm~{}\textbf{0.91}$ |  $60.54~{}\pm~{}31.37$ |  $109.63~{}\pm~{}2.00$
|  | $\rho$ |  $1.169~{}\pm~{}0.178$ |  $1.134~{}\pm~{}0.075$ |  $1.082~{}\pm~{}0.025$
| none | metric | $95.21~{}\pm~{}0.95$ | $\cellcolor{NiceBlue!25}\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.93~{}\pm~{}2.83$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $61.72~{}\pm~{}43.90$ |  $67.85~{}\pm~{}7.14$ | $647.18~{}\pm~{}\textbf{0.96}$
|  | $\rho$ |  $0.970~{}\pm~{}0.050$ |  $0.923~{}\pm~{}0.109$ | $0.994~{}\pm~{}0.056$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{86.38}~{}\pm~{}\textbf{5.16}$ | $\cellcolor{NiceBlue!25}\textbf{70.26}~{}\pm~{}\textbf{7.12}$ | $\cellcolor{NiceBlue!25}\textbf{130.18}~{}\pm~{}1.06$
|  | $\rho$ | $2.225~{}\pm~{}0.033$ | $1.748~{}\pm~{}0.016$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | 0 | $\cellcolor{NiceOrange!25}\textbf{75.55}~{}\pm~{}\textbf{0.50}$ |  $645.09~{}\pm~{}\textbf{1.94}$
|  | $\rho$ | 0 | $1.000~{}\pm~{}0.004$ |  $1.044~{}\pm~{}0.031$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{81.42}~{}\pm~{}\textbf{1.17}$ | $68.20~{}\pm~{}6.85$ | $\cellcolor{NiceBlue!25}\textbf{130.54}~{}\pm~{}5.09$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.130~{}\pm~{}0.003$ | $1.094~{}\pm~{}0.002$
|  |  |  |  |
|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.32~{}\pm~{}0.24$ | $\cellcolor{NiceGreen!25}\textbf{14.57}~{}\pm~{}6.05$ |  $\cellcolor{NiceGreen!25}\textbf{638.74}~{}\pm~{}\textbf{0.40}$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.491~{}\pm~{}0.021$ |  $0.532~{}\pm~{}0.051$
| $\rho_{t}=1$ | metric |  $11.18~{}\pm~{}1.22$ |  $8.41~{}\pm~{}4.46$ |  $659.35~{}\pm~{}36.27$
|  | $\rho$ |  $0.978~{}\pm~{}0.034$ |  $1.121~{}\pm~{}0.192$ |  $0.978~{}\pm~{}0.044$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.41}~{}\pm~{}\textbf{0.20}$ | $12.90~{}\pm~{}\textbf{3.82}$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{31.56}~{}\pm~{}7.05$ | $\cellcolor{NiceGreen!25}\textbf{20.96}~{}\pm~{}6.33$ |  $623.75~{}\pm~{}\textbf{75.35}$
|  | $\rho$ | $0.516~{}\pm~{}0.023$ | $0.516~{}\pm~{}0.014$ |  $0.480~{}\pm~{}0.012$
| $\rho_{t}=1$ | metric | 0 |  $20.34~{}\pm~{}nan$ | 0
|  | $\rho$ | 0 |  $1.026~{}\pm~{}nan$ | 0
| none | metric | $19.63~{}\pm~{}\textbf{6.60}$ | $16.43~{}\pm~{}\textbf{3.51}$ | $\cellcolor{NiceBlue!25}\textbf{500.04}~{}\pm~{}94.49$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{96.67}~{}\pm~{}\textbf{0.33}$ | $83.99~{}\pm~{}nan$ |  $\cellcolor{NiceGreen!25}\textbf{256.58}~{}\pm~{}256.28$
|  | $\rho$ | $0.506~{}\pm~{}0.007$ | $0.506~{}\pm~{}nan$ |  $0.523~{}\pm~{}0.034$
| $\rho_{t}=1$ | metric |  $80.69~{}\pm~{}32.30$ |  $70.63~{}\pm~{}nan$ |  $644.32~{}\pm~{}\textbf{0.40}$
|  | $\rho$ |  $1.179~{}\pm~{}0.138$ |  $1.092~{}\pm~{}nan$ |  $1.098~{}\pm~{}0.047$
| none | metric | $95.61~{}\pm~{}0.38$ | $\cellcolor{NiceBlue!25}\textbf{88.12}~{}\pm~{}\textbf{1.39}$ | $648.25~{}\pm~{}4.75$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.002$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $68.26~{}\pm~{}49.60$ |  $43.51~{}\pm~{}54.73$ |  $639.90~{}\pm~{}\textbf{0.22}$
|  | $\rho$ | $0.503~{}\pm~{}0.002$ |  $0.536~{}\pm~{}0.021$ |  $0.553~{}\pm~{}0.089$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{96.24}~{}\pm~{}\textbf{0.22}$ |  $62.68~{}\pm~{}\textbf{25.56}$ |  $\cellcolor{NiceOrange!25}\textbf{158.37}~{}\pm~{}nan$
|  | $\rho$ |  $1.173~{}\pm~{}0.165$ |  $1.129~{}\pm~{}0.081$ |  $1.152~{}\pm~{}nan$
| none | metric | $96.19~{}\pm~{}0.29$ | $\cellcolor{NiceBlue!25}\textbf{62.98}~{}\pm~{}27.77$ | $644.18~{}\pm~{}0.59$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $11.33~{}\pm~{}\textbf{0.25}$ | $\cellcolor{NiceOrange!25}\textbf{5.00}~{}\pm~{}0.17$ |  $\cellcolor{NiceOrange!25}\textbf{649.52}~{}\pm~{}\textbf{0.95}$
|  | $\rho$ |  $0.978~{}\pm~{}0.079$ | $0.993~{}\pm~{}0.027$ |  $0.958~{}\pm~{}0.057$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.75}~{}\pm~{}0.71$ | $4.93~{}\pm~{}\textbf{0.14}$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $1.667~{}\pm~{}0.007$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | $10.83~{}\pm~{}\textbf{0.63}$ | $\cellcolor{NiceOrange!25}\textbf{43.03}~{}\pm~{}12.65$ | $\cellcolor{NiceOrange!25}\textbf{650.16}~{}\pm~{}\textbf{0.39}$
|  | $\rho$ | $0.999~{}\pm~{}0.008$ | $0.996~{}\pm~{}0.001$ | $0.999~{}\pm~{}0.066$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.73}~{}\pm~{}1.54$ | $18.89~{}\pm~{}\textbf{5.68}$ | $661.75~{}\pm~{}0.75$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
|  | depth | 1 | 3 | 5 | 7
---|---|---|---|---|---|---
net | LSC |  |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{40.32}~{}\pm~{}\textbf{5.29}$ | $\cellcolor{NiceGreen!25}\textbf{39.18}~{}\pm~{}\textbf{4.30}$ | $\cellcolor{NiceGreen!25}\textbf{14.57}~{}\pm~{}6.05$ | $4.79~{}\pm~{}\textbf{0.15}$
|  | $\rho$ | $0.513~{}\pm~{}0.021$ | $0.506~{}\pm~{}0.019$ | $0.491~{}\pm~{}0.021$ | $0.483~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric |  $35.00~{}\pm~{}8.87$ |  $23.38~{}\pm~{}12.35$ |  $8.41~{}\pm~{}4.46$ |  $\cellcolor{NiceOrange!25}\textbf{5.31}~{}\pm~{}0.77$
|  | $\rho$ |  $1.297~{}\pm~{}0.122$ |  $1.230~{}\pm~{}0.389$ |  $1.121~{}\pm~{}0.192$ |  $1.044~{}\pm~{}0.009$
| none | metric | $38.20~{}\pm~{}6.84$ | $30.47~{}\pm~{}13.34$ | $12.90~{}\pm~{}\textbf{3.82}$ | $5.06~{}\pm~{}0.16$
|  | $\rho$ | $0.366~{}\pm~{}0.001$ | $0.362~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $29.60~{}\pm~{}\textbf{6.66}$ | $31.29~{}\pm~{}6.87$ | $\cellcolor{NiceGreen!25}\textbf{20.96}~{}\pm~{}6.33$ |  $\cellcolor{NiceGreen!25}\textbf{16.06}~{}\pm~{}\textbf{1.71}$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.499~{}\pm~{}0.004$ | $0.516~{}\pm~{}0.014$ |  $0.549~{}\pm~{}0.023$
| $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{39.17}~{}\pm~{}10.89$ |  $\cellcolor{NiceOrange!25}\textbf{36.60}~{}\pm~{}7.94$ |  $20.34~{}\pm~{}nan$ | 0
|  | $\rho$ | $0.991~{}\pm~{}0.018$ |  $1.025~{}\pm~{}0.044$ |  $1.026~{}\pm~{}nan$ | 0
| none | metric | $39.06~{}\pm~{}10.85$ | $34.19~{}\pm~{}\textbf{5.98}$ | $16.43~{}\pm~{}\textbf{3.51}$ | $15.59~{}\pm~{}2.31$
|  | $\rho$ | $1.355~{}\pm~{}0.004$ | $1.346~{}\pm~{}0.003$ | $1.341~{}\pm~{}0.006$ | $1.339~{}\pm~{}0.005$
GRU | $\rho_{t}=0.5$ | metric | $81.39~{}\pm~{}3.19$ | $85.81~{}\pm~{}\textbf{1.13}$ | $83.99~{}\pm~{}nan$ | $\cellcolor{NiceGreen!25}\textbf{87.48}~{}\pm~{}\textbf{0.78}$
|  | $\rho$ | $0.490~{}\pm~{}0.010$ | $0.497~{}\pm~{}0.003$ | $0.506~{}\pm~{}nan$ | $0.502~{}\pm~{}0.002$
| $\rho_{t}=1$ | metric | $66.53~{}\pm~{}6.34$ |  $79.63~{}\pm~{}6.00$ |  $70.63~{}\pm~{}nan$ |  $46.47~{}\pm~{}58.17$
|  | $\rho$ | $1.006~{}\pm~{}0.016$ |  $1.067~{}\pm~{}0.055$ |  $1.092~{}\pm~{}nan$ |  $1.167~{}\pm~{}0.132$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{1.45}$ | $\cellcolor{NiceBlue!25}\textbf{86.01}~{}\pm~{}2.48$ | $\cellcolor{NiceBlue!25}\textbf{88.12}~{}\pm~{}\textbf{1.39}$ | $87.40~{}\pm~{}2.29$
|  | $\rho$ | $0.751~{}\pm~{}0.002$ | $0.710~{}\pm~{}0.002$ | $0.702~{}\pm~{}0.002$ | $0.699~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric |  $70.84~{}\pm~{}17.19$ |  $24.40~{}\pm~{}14.58$ |  $43.51~{}\pm~{}54.73$ |  $21.70~{}\pm~{}28.71$
|  | $\rho$ |  $0.603~{}\pm~{}0.090$ |  $0.535~{}\pm~{}0.014$ |  $0.536~{}\pm~{}0.021$ |  $0.532~{}\pm~{}0.018$
| $\rho_{t}=1$ | metric | $44.44~{}\pm~{}\textbf{3.70}$ |  $82.38~{}\pm~{}\textbf{2.07}$ |  $62.68~{}\pm~{}\textbf{25.56}$ |  $\cellcolor{NiceOrange!25}\textbf{52.04}~{}\pm~{}41.83$
|  | $\rho$ | $1.013~{}\pm~{}0.026$ |  $1.282~{}\pm~{}0.007$ |  $1.129~{}\pm~{}0.081$ |  $1.083~{}\pm~{}0.128$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{76.69}~{}\pm~{}9.27$ | $\cellcolor{NiceBlue!25}\textbf{83.74}~{}\pm~{}2.81$ | $\cellcolor{NiceBlue!25}\textbf{62.98}~{}\pm~{}27.77$ | $27.90~{}\pm~{}\textbf{7.79}$
|  | $\rho$ | $0.914~{}\pm~{}0.003$ | $0.879~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.869~{}\pm~{}0.002$
ALIF | $\rho_{t}=1$ | metric | $76.53~{}\pm~{}\textbf{0.76}$ |  $51.50~{}\pm~{}23.97$ | $\cellcolor{NiceOrange!25}\textbf{5.00}~{}\pm~{}0.17$ |  $\cellcolor{NiceOrange!25}\textbf{5.05}~{}\pm~{}\textbf{0.14}$
|  | $\rho$ | $1.005~{}\pm~{}0.011$ |  $0.972~{}\pm~{}0.013$ | $0.993~{}\pm~{}0.027$ |  $0.980~{}\pm~{}0.017$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{77.18}~{}\pm~{}1.59$ | $\cellcolor{NiceBlue!25}\textbf{57.88}~{}\pm~{}\textbf{6.96}$ | $4.93~{}\pm~{}\textbf{0.14}$ | $4.93~{}\pm~{}0.23$
|  | $\rho$ | $1.853~{}\pm~{}0.023$ | $1.701~{}\pm~{}0.011$ | $1.667~{}\pm~{}0.007$ | $1.656~{}\pm~{}0.005$
ALIF± | $\rho_{t}=1$ | metric | $66.81~{}\pm~{}\textbf{3.82}$ |  $18.85~{}\pm~{}nan$ | $\cellcolor{NiceOrange!25}\textbf{43.03}~{}\pm~{}12.65$ | $4.92~{}\pm~{}\textbf{0.10}$
|  | $\rho$ | $1.002~{}\pm~{}0.003$ |  $1.038~{}\pm~{}nan$ | $0.996~{}\pm~{}0.001$ | $1.008~{}\pm~{}0.018$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{66.83}~{}\pm~{}4.23$ | $\cellcolor{NiceBlue!25}\textbf{60.47}~{}\pm~{}\textbf{7.05}$ | $18.89~{}\pm~{}\textbf{5.68}$ | $\cellcolor{NiceBlue!25}\textbf{5.06}~{}\pm~{}0.18$
|  | $\rho$ | $1.460~{}\pm~{}0.007$ | $1.016~{}\pm~{}0.001$ | $0.979~{}\pm~{}0.002$ | $0.933~{}\pm~{}0.001$
|  |  |  |  |  |

##  7 June 28

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{36.73}~{}\pm~{}8.86$ |  $37.39~{}\pm~{}\textbf{5.31}$ | 0
|  | $\rho$ |  $0.532~{}\pm~{}0.046$ |  $1.321~{}\pm~{}0.584$ | 0
| $\rho_{t}=1$ | metric |  $22.26~{}\pm~{}7.12$ |  $17.18~{}\pm~{}6.25$ | 0
|  | $\rho$ |  $2.203~{}\pm~{}1.072$ |  $3.753~{}\pm~{}2.642$ | 0
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\cellcolor{NiceBlue!25}\textbf{46.40}~{}\pm~{}12.12$ | $\cellcolor{NiceBlue!25}\textbf{641.65}~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{34.16}~{}\pm~{}7.12$ | $33.80~{}\pm~{}5.96$ | 0
|  | $\rho$ |  $0.534~{}\pm~{}0.027$ | $0.500~{}\pm~{}0.004$ | 0
| $\rho_{t}=1$ | metric |  $32.18~{}\pm~{}\textbf{1.59}$ | $30.71~{}\pm~{}nan$ | 0
|  | $\rho$ |  $1.039~{}\pm~{}0.062$ | $0.999~{}\pm~{}nan$ | 0
| none | metric | $23.51~{}\pm~{}2.50$ | $\cellcolor{NiceBlue!25}\textbf{39.84}~{}\pm~{}\textbf{5.91}$ | $\cellcolor{NiceBlue!25}\textbf{173.67}~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.35}~{}\pm~{}\textbf{0.23}$ | $84.29~{}\pm~{}\textbf{1.28}$ | $\cellcolor{NiceGreen!25}\textbf{100.28}~{}\pm~{}nan$
|  | $\rho$ | $0.516~{}\pm~{}0.019$ | $0.501~{}\pm~{}0.009$ | $0.502~{}\pm~{}nan$
| $\rho_{t}=1$ | metric |  $94.13~{}\pm~{}2.40$ |  $74.15~{}\pm~{}nan$ | 0
|  | $\rho$ |  $1.621~{}\pm~{}0.957$ |  $1.104~{}\pm~{}nan$ | 0
| none | metric | $94.85~{}\pm~{}0.40$ | $\cellcolor{NiceBlue!25}\textbf{84.92}~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric |  $67.24~{}\pm~{}48.71$ | $52.14~{}\pm~{}23.19$ | $110.56~{}\pm~{}nan$
|  | $\rho$ |  $0.530~{}\pm~{}0.018$ | $0.499~{}\pm~{}0.001$ | $0.505~{}\pm~{}nan$
| $\rho_{t}=1$ | metric |  $83.06~{}\pm~{}24.34$ |  $45.67~{}\pm~{}26.11$ | 0
|  | $\rho$ |  $1.302~{}\pm~{}0.198$ |  $1.071~{}\pm~{}0.041$ | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{95.13}~{}\pm~{}\textbf{0.93}$ | $\cellcolor{NiceBlue!25}\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $\cellcolor{NiceBlue!25}\textbf{107.52}~{}\pm~{}\textbf{0.27}$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.004$
ALIF | $\rho_{t}=1$ | metric | 0 | 0 |  $649.26~{}\pm~{}\textbf{0.32}$
|  | $\rho$ | 0 | 0 |  $2.921~{}\pm~{}0.336$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{92.88}~{}\pm~{}nan$ | $\cellcolor{NiceBlue!25}\textbf{74.89}~{}\pm~{}nan$ | $\cellcolor{NiceBlue!25}\textbf{130.18}~{}\pm~{}1.06$
|  | $\rho$ | $2.225~{}\pm~{}nan$ | $1.740~{}\pm~{}nan$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | 0 | $\cellcolor{NiceOrange!25}\textbf{75.28}~{}\pm~{}nan$ |  $645.23~{}\pm~{}\textbf{1.29}$
|  | $\rho$ | 0 | $0.996~{}\pm~{}nan$ |  $2.185~{}\pm~{}0.311$
| none | metric | 0 | 0 | $\cellcolor{NiceBlue!25}\textbf{128.19}~{}\pm~{}nan$
|  | $\rho$ | 0 | 0 | $1.093~{}\pm~{}nan$
|  |  |  |  |
|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.32~{}\pm~{}\textbf{0.24}$ |  $\cellcolor{NiceGreen!25}\textbf{12.98}~{}\pm~{}1.70$ |  $638.61~{}\pm~{}0.21$
|  | $\rho$ | $0.504~{}\pm~{}0.005$ |  $0.638~{}\pm~{}0.033$ |  $0.633~{}\pm~{}0.201$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{11.50}~{}\pm~{}2.17$ |  $4.69~{}\pm~{}\textbf{0.10}$ |  $\cellcolor{NiceOrange!25}\textbf{638.24}~{}\pm~{}\textbf{0.11}$
|  | $\rho$ |  $1.529~{}\pm~{}0.218$ |  $1.669~{}\pm~{}0.501$ |  $1.430~{}\pm~{}0.117$
| none | metric | $11.37~{}\pm~{}0.27$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{31.84}~{}\pm~{}6.68$ |  $\cellcolor{NiceGreen!25}\textbf{22.14}~{}\pm~{}12.37$ |  $518.76~{}\pm~{}132.43$
|  | $\rho$ |  $0.542~{}\pm~{}0.035$ |  $0.547~{}\pm~{}0.040$ |  $0.434~{}\pm~{}0.045$
| $\rho_{t}=1$ | metric |  $25.12~{}\pm~{}nan$ |  $17.69~{}\pm~{}\textbf{1.29}$ | 0
|  | $\rho$ |  $0.975~{}\pm~{}nan$ |  $1.041~{}\pm~{}0.014$ | 0
| none | metric | $19.63~{}\pm~{}\textbf{6.60}$ | $16.43~{}\pm~{}3.51$ | $\cellcolor{NiceBlue!25}\textbf{500.04}~{}\pm~{}\textbf{94.49}$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{96.53}~{}\pm~{}\textbf{0.21}$ | 0 | 0
|  | $\rho$ | $0.510~{}\pm~{}0.003$ | 0 | 0
| $\rho_{t}=1$ | metric |  $70.87~{}\pm~{}44.87$ | 0 | 0
|  | $\rho$ |  $1.156~{}\pm~{}0.187$ | 0 | 0
| none | metric | $95.61~{}\pm~{}0.38$ | $\cellcolor{NiceBlue!25}\textbf{88.17}~{}\pm~{}\textbf{1.70}$ | $\cellcolor{NiceBlue!25}\textbf{648.25}~{}\pm~{}\textbf{4.75}$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.702~{}\pm~{}0.001$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $75.12~{}\pm~{}42.76$ |  $4.81~{}\pm~{}nan$ | $\cellcolor{NiceGreen!25}\textbf{640.17}~{}\pm~{}\textbf{0.37}$
|  | $\rho$ | $0.509~{}\pm~{}0.009$ |  $0.535~{}\pm~{}nan$ | $0.517~{}\pm~{}0.017$
| $\rho_{t}=1$ | metric |  $95.83~{}\pm~{}\textbf{0.03}$ | 0 |  $640.22~{}\pm~{}nan$
|  | $\rho$ |  $1.029~{}\pm~{}0.079$ | 0 |  $1.873~{}\pm~{}nan$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{96.19}~{}\pm~{}0.29$ | $\cellcolor{NiceBlue!25}\textbf{62.98}~{}\pm~{}\textbf{27.77}$ | $644.18~{}\pm~{}0.59$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric | 0 |  $\cellcolor{NiceOrange!25}\textbf{5.05}~{}\pm~{}nan$ | 0
|  | $\rho$ | 0 |  $1.024~{}\pm~{}nan$ | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.22}~{}\pm~{}\textbf{0.10}$ | 0 | $\cellcolor{NiceBlue!25}\textbf{655.61}~{}\pm~{}\textbf{12.53}$
|  | $\rho$ | $2.272~{}\pm~{}0.014$ | 0 | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric | 0 | 0 | 0
|  | $\rho$ | 0 | 0 | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.73}~{}\pm~{}\textbf{1.54}$ | 0 | $\cellcolor{NiceBlue!25}\textbf{661.75}~{}\pm~{}\textbf{0.75}$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | 0 | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : task:5 |  | depth | 1 | 3 | 5 | 7
---|---|---|---|---|---|---
net | LSC |  |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{40.49}~{}\pm~{}5.91$ |  $\cellcolor{NiceGreen!25}\textbf{31.18}~{}\pm~{}\textbf{4.80}$ |  $\cellcolor{NiceGreen!25}\textbf{12.98}~{}\pm~{}1.70$ |  $4.76~{}\pm~{}0.20$
|  | $\rho$ |  $0.716~{}\pm~{}0.193$ |  $1.229~{}\pm~{}0.675$ |  $0.638~{}\pm~{}0.033$ |  $0.858~{}\pm~{}0.533$
| $\rho_{t}=1$ | metric |  $36.92~{}\pm~{}\textbf{4.81}$ |  $18.22~{}\pm~{}8.23$ |  $4.69~{}\pm~{}\textbf{0.10}$ |  $4.78~{}\pm~{}0.24$
|  | $\rho$ |  $5.843~{}\pm~{}4.956$ |  $1.717~{}\pm~{}0.405$ |  $1.669~{}\pm~{}0.501$ |  $1.443~{}\pm~{}0.052$
| none | metric | $38.20~{}\pm~{}6.84$ | $30.47~{}\pm~{}13.34$ | $12.90~{}\pm~{}3.82$ | $\cellcolor{NiceBlue!25}\textbf{5.06}~{}\pm~{}\textbf{0.16}$
|  | $\rho$ | $0.366~{}\pm~{}0.001$ | $0.362~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $35.00~{}\pm~{}\textbf{8.56}$ | $31.71~{}\pm~{}6.72$ |  $\cellcolor{NiceGreen!25}\textbf{22.14}~{}\pm~{}12.37$ |  $12.53~{}\pm~{}\textbf{2.23}$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.511~{}\pm~{}0.028$ |  $0.547~{}\pm~{}0.040$ |  $0.612~{}\pm~{}0.016$
| $\rho_{t}=1$ | metric | $34.83~{}\pm~{}11.24$ | $\cellcolor{NiceOrange!25}\textbf{41.15}~{}\pm~{}\textbf{1.36}$ |  $17.69~{}\pm~{}\textbf{1.29}$ | 0
|  | $\rho$ | $1.011~{}\pm~{}0.040$ | $0.999~{}\pm~{}0.002$ |  $1.041~{}\pm~{}0.014$ | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{39.06}~{}\pm~{}10.85$ | $34.19~{}\pm~{}5.98$ | $16.43~{}\pm~{}3.51$ | $\cellcolor{NiceBlue!25}\textbf{15.59}~{}\pm~{}2.31$
|  | $\rho$ | $1.355~{}\pm~{}0.004$ | $1.346~{}\pm~{}0.003$ | $1.341~{}\pm~{}0.006$ | $1.339~{}\pm~{}0.005$
GRU | $\rho_{t}=0.5$ | metric | $75.10~{}\pm~{}10.53$ | $\cellcolor{NiceGreen!25}\textbf{86.04}~{}\pm~{}\textbf{1.25}$ | 0 | $\cellcolor{NiceGreen!25}\textbf{87.55}~{}\pm~{}nan$
|  | $\rho$ | $0.489~{}\pm~{}0.015$ | $0.496~{}\pm~{}0.004$ | 0 | $0.502~{}\pm~{}nan$
| $\rho_{t}=1$ | metric |  $69.64~{}\pm~{}3.92$ | 0 | 0 | 0
|  | $\rho$ |  $1.043~{}\pm~{}0.077$ | 0 | 0 | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{1.45}$ | $86.01~{}\pm~{}2.48$ | $\cellcolor{NiceBlue!25}\textbf{88.17}~{}\pm~{}\textbf{1.70}$ | $85.96~{}\pm~{}\textbf{0.61}$
|  | $\rho$ | $0.751~{}\pm~{}0.002$ | $0.710~{}\pm~{}0.002$ | $0.702~{}\pm~{}0.001$ | $0.700~{}\pm~{}0.001$
LSTM | $\rho_{t}=0.5$ | metric |  $68.37~{}\pm~{}15.23$ | $46.01~{}\pm~{}nan$ |  $4.81~{}\pm~{}nan$ |  $5.29~{}\pm~{}nan$
|  | $\rho$ |  $0.551~{}\pm~{}0.059$ | $0.508~{}\pm~{}nan$ |  $0.535~{}\pm~{}nan$ |  $0.549~{}\pm~{}nan$
| $\rho_{t}=1$ | metric | $41.98~{}\pm~{}\textbf{4.08}$ |  $78.61~{}\pm~{}nan$ | 0 | $\cellcolor{NiceOrange!25}\textbf{28.51}~{}\pm~{}nan$
|  | $\rho$ | $1.014~{}\pm~{}0.028$ |  $1.090~{}\pm~{}nan$ | 0 | $0.983~{}\pm~{}nan$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{76.69}~{}\pm~{}9.27$ | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{2.81}$ | $\cellcolor{NiceBlue!25}\textbf{62.98}~{}\pm~{}\textbf{27.77}$ | $27.90~{}\pm~{}\textbf{7.79}$
|  | $\rho$ | $0.914~{}\pm~{}0.003$ | $0.880~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.869~{}\pm~{}0.002$
ALIF | $\rho_{t}=1$ | metric | $76.24~{}\pm~{}nan$ | 0 |  $\cellcolor{NiceOrange!25}\textbf{5.05}~{}\pm~{}nan$ | 0
|  | $\rho$ | $1.006~{}\pm~{}nan$ | 0 |  $1.024~{}\pm~{}nan$ | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{77.18}~{}\pm~{}\textbf{1.59}$ | 0 | 0 | 0
|  | $\rho$ | $1.853~{}\pm~{}0.023$ | 0 | 0 | 0
ALIF± | $\rho_{t}=1$ | metric | $63.80~{}\pm~{}nan$ | 0 | 0 | 0
|  | $\rho$ | $0.985~{}\pm~{}nan$ | 0 | 0 | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{66.83}~{}\pm~{}\textbf{4.23}$ | 0 | 0 | $\cellcolor{NiceBlue!25}\textbf{5.24}~{}\pm~{}nan$
|  | $\rho$ | $1.460~{}\pm~{}0.007$ | 0 | 0 | $0.932~{}\pm~{}nan$
|  |  |  |  |  |
Table : stack

##  8 29 June 2023

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{36.73}~{}\pm~{}8.86$ |  $37.39~{}\pm~{}\textbf{5.31}$ |  $146.94~{}\pm~{}39.29$
|  | $\rho$ |  $0.532~{}\pm~{}0.046$ |  $1.321~{}\pm~{}0.584$ |  $0.944~{}\pm~{}0.403$
| $\rho_{t}=1$ | metric |  $22.26~{}\pm~{}7.12$ |  $17.18~{}\pm~{}6.25$ |  $\cellcolor{NiceOrange!25}\textbf{138.16}~{}\pm~{}3.68$
|  | $\rho$ |  $2.203~{}\pm~{}1.072$ |  $3.753~{}\pm~{}2.642$ |  $1.635~{}\pm~{}0.588$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\cellcolor{NiceBlue!25}\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{34.16}~{}\pm~{}7.12$ | $33.80~{}\pm~{}5.96$ |  $\cellcolor{NiceGreen!25}\textbf{120.36}~{}\pm~{}nan$
|  | $\rho$ |  $0.534~{}\pm~{}0.027$ | $0.500~{}\pm~{}0.004$ |  $0.473~{}\pm~{}nan$
| $\rho_{t}=1$ | metric |  $32.18~{}\pm~{}\textbf{1.59}$ | $30.71~{}\pm~{}nan$ | 0
|  | $\rho$ |  $1.039~{}\pm~{}0.062$ | $0.999~{}\pm~{}nan$ | 0
| none | metric | $23.51~{}\pm~{}2.50$ | $\cellcolor{NiceBlue!25}\textbf{39.84}~{}\pm~{}\textbf{5.91}$ | $173.67~{}\pm~{}\textbf{3.21}$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.35}~{}\pm~{}\textbf{0.23}$ | $84.29~{}\pm~{}\textbf{1.28}$ | $\cellcolor{NiceGreen!25}\textbf{101.99}~{}\pm~{}2.89$
|  | $\rho$ | $0.516~{}\pm~{}0.019$ | $0.501~{}\pm~{}0.009$ | $0.503~{}\pm~{}0.014$
| $\rho_{t}=1$ | metric |  $94.13~{}\pm~{}2.40$ |  $74.15~{}\pm~{}nan$ |  $104.24~{}\pm~{}4.92$
|  | $\rho$ |  $1.621~{}\pm~{}0.957$ |  $1.104~{}\pm~{}nan$ |  $1.626~{}\pm~{}1.007$
| none | metric | $94.85~{}\pm~{}0.40$ | $\cellcolor{NiceBlue!25}\textbf{84.92}~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric |  $67.24~{}\pm~{}48.71$ | $52.14~{}\pm~{}23.19$ |  $\cellcolor{NiceGreen!25}\textbf{107.80}~{}\pm~{}3.13$
|  | $\rho$ |  $0.530~{}\pm~{}0.018$ | $0.499~{}\pm~{}0.001$ |  $0.612~{}\pm~{}0.168$
| $\rho_{t}=1$ | metric |  $83.06~{}\pm~{}24.34$ |  $45.67~{}\pm~{}26.11$ |  $108.97~{}\pm~{}3.45$
|  | $\rho$ |  $1.302~{}\pm~{}0.198$ |  $1.071~{}\pm~{}0.041$ |  $1.058~{}\pm~{}0.013$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{95.13}~{}\pm~{}\textbf{0.93}$ | $\cellcolor{NiceBlue!25}\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.80~{}\pm~{}\textbf{2.56}$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $81.39~{}\pm~{}\textbf{11.64}$ |  $62.60~{}\pm~{}18.23$ |  $649.26~{}\pm~{}\textbf{0.32}$
|  | $\rho$ |  $1.294~{}\pm~{}0.044$ |  $1.425~{}\pm~{}0.586$ |  $2.921~{}\pm~{}0.336$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{92.88}~{}\pm~{}nan$ | $\cellcolor{NiceBlue!25}\textbf{76.37}~{}\pm~{}\textbf{4.39}$ | $\cellcolor{NiceBlue!25}\textbf{130.18}~{}\pm~{}1.06$
|  | $\rho$ | $2.225~{}\pm~{}nan$ | $2.070~{}\pm~{}0.286$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{85.80}~{}\pm~{}4.78$ |  $\cellcolor{NiceOrange!25}\textbf{74.94}~{}\pm~{}\textbf{0.32}$ |  $518.65~{}\pm~{}254.39$
|  | $\rho$ |  $1.103~{}\pm~{}0.132$ |  $1.077~{}\pm~{}0.081$ |  $2.017~{}\pm~{}0.500$
| none | metric | $81.46~{}\pm~{}\textbf{1.41}$ | $74.52~{}\pm~{}1.22$ | $\cellcolor{NiceBlue!25}\textbf{130.88}~{}\pm~{}\textbf{4.40}$
|  | $\rho$ | $1.217~{}\pm~{}0.008$ | $1.211~{}\pm~{}0.004$ | $1.168~{}\pm~{}0.065$
|  |  |  |  |
Table : task |  | depth | 1 | 3 | 5 | 7
---|---|---|---|---|---|---
net | LSC |  |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{40.49}~{}\pm~{}5.91$ |  $\cellcolor{NiceGreen!25}\textbf{31.18}~{}\pm~{}\textbf{4.80}$ |  $12.37~{}\pm~{}\textbf{1.60}$ |  $4.76~{}\pm~{}0.20$
|  | $\rho$ |  $0.716~{}\pm~{}0.193$ |  $1.229~{}\pm~{}0.675$ |  $0.651~{}\pm~{}0.032$ |  $0.858~{}\pm~{}0.533$
| $\rho_{t}=1$ | metric |  $36.92~{}\pm~{}\textbf{4.81}$ |  $23.85~{}\pm~{}11.35$ |  $8.09~{}\pm~{}5.90$ |  $4.87~{}\pm~{}0.23$
|  | $\rho$ |  $5.843~{}\pm~{}4.956$ |  $1.612~{}\pm~{}0.339$ |  $1.536~{}\pm~{}0.422$ |  $1.467~{}\pm~{}0.056$
| none | metric | $38.20~{}\pm~{}6.84$ | $30.47~{}\pm~{}13.34$ | $\cellcolor{NiceBlue!25}\textbf{12.90}~{}\pm~{}3.82$ | $\cellcolor{NiceBlue!25}\textbf{5.06}~{}\pm~{}\textbf{0.16}$
|  | $\rho$ | $0.366~{}\pm~{}0.001$ | $0.362~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $35.00~{}\pm~{}\textbf{8.56}$ | $31.71~{}\pm~{}6.72$ |  $\cellcolor{NiceGreen!25}\textbf{22.14}~{}\pm~{}12.37$ |  $12.53~{}\pm~{}\textbf{2.23}$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.511~{}\pm~{}0.028$ |  $0.547~{}\pm~{}0.040$ |  $0.612~{}\pm~{}0.016$
| $\rho_{t}=1$ | metric | $34.83~{}\pm~{}11.24$ | $\cellcolor{NiceOrange!25}\textbf{41.15}~{}\pm~{}\textbf{1.36}$ |  $17.69~{}\pm~{}\textbf{1.29}$ | 0
|  | $\rho$ | $1.011~{}\pm~{}0.040$ | $0.999~{}\pm~{}0.002$ |  $1.041~{}\pm~{}0.014$ | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{39.06}~{}\pm~{}10.85$ | $34.19~{}\pm~{}5.98$ | $16.43~{}\pm~{}3.51$ | $\cellcolor{NiceBlue!25}\textbf{15.59}~{}\pm~{}2.31$
|  | $\rho$ | $1.355~{}\pm~{}0.004$ | $1.346~{}\pm~{}0.003$ | $1.341~{}\pm~{}0.006$ | $1.339~{}\pm~{}0.005$
GRU | $\rho_{t}=0.5$ | metric | $75.10~{}\pm~{}10.53$ | $85.76~{}\pm~{}\textbf{1.17}$ | $86.43~{}\pm~{}\textbf{1.55}$ | $\cellcolor{NiceGreen!25}\textbf{87.77}~{}\pm~{}\textbf{0.24}$
|  | $\rho$ | $0.489~{}\pm~{}0.015$ | $0.497~{}\pm~{}0.003$ | $0.504~{}\pm~{}0.002$ | $0.504~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $69.64~{}\pm~{}3.92$ |  $80.28~{}\pm~{}4.78$ |  $33.99~{}\pm~{}32.91$ |  $37.85~{}\pm~{}36.66$
|  | $\rho$ |  $1.043~{}\pm~{}0.077$ |  $1.132~{}\pm~{}0.030$ |  $1.182~{}\pm~{}0.050$ |  $1.180~{}\pm~{}0.122$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{1.45}$ | $\cellcolor{NiceBlue!25}\textbf{86.01}~{}\pm~{}2.48$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $85.96~{}\pm~{}0.61$
|  | $\rho$ | $0.751~{}\pm~{}0.002$ | $0.710~{}\pm~{}0.002$ | $0.701~{}\pm~{}0.003$ | $0.700~{}\pm~{}0.001$
LSTM | $\rho_{t}=0.5$ | metric |  $68.37~{}\pm~{}15.23$ | $73.25~{}\pm~{}18.28$ |  $51.66~{}\pm~{}40.59$ |  $5.08~{}\pm~{}\textbf{0.19}$
|  | $\rho$ |  $0.551~{}\pm~{}0.059$ | $0.518~{}\pm~{}0.029$ |  $0.543~{}\pm~{}0.018$ |  $0.542~{}\pm~{}0.021$
| $\rho_{t}=1$ | metric | $41.98~{}\pm~{}\textbf{4.08}$ |  $83.61~{}\pm~{}4.09$ |  $\cellcolor{NiceOrange!25}\textbf{71.89}~{}\pm~{}\textbf{22.71}$ | $\cellcolor{NiceOrange!25}\textbf{59.46}~{}\pm~{}28.38$
|  | $\rho$ | $1.014~{}\pm~{}0.028$ |  $1.117~{}\pm~{}0.117$ |  $1.073~{}\pm~{}0.125$ | $0.982~{}\pm~{}0.017$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{76.69}~{}\pm~{}9.27$ | $\cellcolor{NiceBlue!25}\textbf{84.82}~{}\pm~{}\textbf{2.35}$ | $62.98~{}\pm~{}27.77$ | $27.90~{}\pm~{}7.79$
|  | $\rho$ | $0.914~{}\pm~{}0.003$ | $0.879~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.869~{}\pm~{}0.002$
ALIF | $\rho_{t}=1$ | metric | $74.81~{}\pm~{}2.80$ |  $55.59~{}\pm~{}9.06$ |  $5.06~{}\pm~{}\textbf{0.20}$ |  $\cellcolor{NiceOrange!25}\textbf{5.07}~{}\pm~{}\textbf{0.13}$
|  | $\rho$ | $1.010~{}\pm~{}0.003$ |  $1.223~{}\pm~{}0.261$ |  $1.056~{}\pm~{}0.054$ |  $1.223~{}\pm~{}0.317$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{77.18}~{}\pm~{}\textbf{1.59}$ | $\cellcolor{NiceBlue!25}\textbf{63.11}~{}\pm~{}\textbf{5.52}$ | $\cellcolor{NiceBlue!25}\textbf{5.14}~{}\pm~{}0.29$ | $4.98~{}\pm~{}0.17$
|  | $\rho$ | $1.853~{}\pm~{}0.023$ | $2.164~{}\pm~{}0.020$ | $2.105~{}\pm~{}0.006$ | $2.075~{}\pm~{}0.005$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{68.17}~{}\pm~{}\textbf{3.62}$ |  $55.65~{}\pm~{}24.91$ |  $\cellcolor{NiceOrange!25}\textbf{29.64}~{}\pm~{}16.74$ |  $5.08~{}\pm~{}\textbf{0.11}$
|  | $\rho$ | $1.005~{}\pm~{}0.014$ |  $1.208~{}\pm~{}0.229$ |  $1.085~{}\pm~{}0.040$ |  $1.268~{}\pm~{}0.304$
| none | metric | $66.83~{}\pm~{}4.23$ | $\cellcolor{NiceBlue!25}\textbf{68.57}~{}\pm~{}\textbf{3.01}$ | $18.86~{}\pm~{}\textbf{4.26}$ | $\cellcolor{NiceBlue!25}\textbf{5.13}~{}\pm~{}0.19$
|  | $\rho$ | $1.460~{}\pm~{}0.007$ | $1.082~{}\pm~{}0.002$ | $0.979~{}\pm~{}0.002$ | $0.933~{}\pm~{}0.001$
|  |  |  |  |  |
Table : stack |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.32~{}\pm~{}\textbf{0.24}$ |  $12.37~{}\pm~{}\textbf{1.60}$ |  $\cellcolor{NiceGreen!25}\textbf{486.38}~{}\pm~{}263.67$
|  | $\rho$ | $0.504~{}\pm~{}0.005$ |  $0.651~{}\pm~{}0.032$ |  $0.897~{}\pm~{}0.478$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{11.50}~{}\pm~{}2.17$ |  $8.09~{}\pm~{}5.90$ |  $638.57~{}\pm~{}\textbf{0.51}$
|  | $\rho$ |  $1.529~{}\pm~{}0.218$ |  $1.536~{}\pm~{}0.422$ |  $1.291~{}\pm~{}0.260$
| none | metric | $11.37~{}\pm~{}0.27$ | $\cellcolor{NiceBlue!25}\textbf{12.90}~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{31.84}~{}\pm~{}6.68$ |  $\cellcolor{NiceGreen!25}\textbf{22.14}~{}\pm~{}12.37$ |  $518.76~{}\pm~{}132.43$
|  | $\rho$ |  $0.542~{}\pm~{}0.035$ |  $0.547~{}\pm~{}0.040$ |  $0.434~{}\pm~{}0.045$
| $\rho_{t}=1$ | metric |  $25.12~{}\pm~{}nan$ |  $17.69~{}\pm~{}\textbf{1.29}$ | 0
|  | $\rho$ |  $0.975~{}\pm~{}nan$ |  $1.041~{}\pm~{}0.014$ | 0
| none | metric | $19.63~{}\pm~{}\textbf{6.60}$ | $16.43~{}\pm~{}3.51$ | $\cellcolor{NiceBlue!25}\textbf{500.04}~{}\pm~{}\textbf{94.49}$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{96.45}~{}\pm~{}\textbf{0.24}$ | $86.43~{}\pm~{}\textbf{1.55}$ | $\cellcolor{NiceGreen!25}\textbf{122.36}~{}\pm~{}14.16$
|  | $\rho$ | $0.510~{}\pm~{}0.002$ | $0.504~{}\pm~{}0.002$ | $0.490~{}\pm~{}0.009$
| $\rho_{t}=1$ | metric |  $70.87~{}\pm~{}44.87$ |  $33.99~{}\pm~{}32.91$ |  $547.96~{}\pm~{}160.25$
|  | $\rho$ |  $1.156~{}\pm~{}0.187$ |  $1.182~{}\pm~{}0.050$ |  $1.329~{}\pm~{}0.398$
| none | metric | $95.61~{}\pm~{}0.38$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $648.25~{}\pm~{}\textbf{4.75}$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.701~{}\pm~{}0.003$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $75.12~{}\pm~{}42.76$ |  $51.66~{}\pm~{}40.59$ | $640.17~{}\pm~{}\textbf{0.37}$
|  | $\rho$ | $0.509~{}\pm~{}0.009$ |  $0.543~{}\pm~{}0.018$ | $0.517~{}\pm~{}0.017$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{96.21}~{}\pm~{}0.48$ |  $\cellcolor{NiceOrange!25}\textbf{71.89}~{}\pm~{}\textbf{22.71}$ |  $\cellcolor{NiceOrange!25}\textbf{473.95}~{}\pm~{}296.02$
|  | $\rho$ |  $0.964~{}\pm~{}0.106$ |  $1.073~{}\pm~{}0.125$ |  $1.512~{}\pm~{}0.446$
| none | metric | $96.19~{}\pm~{}\textbf{0.29}$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}0.59$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $10.99~{}\pm~{}0.84$ |  $5.06~{}\pm~{}\textbf{0.20}$ |  $\cellcolor{NiceOrange!25}\textbf{650.22}~{}\pm~{}\textbf{1.85}$
|  | $\rho$ |  $1.769~{}\pm~{}0.840$ |  $1.056~{}\pm~{}0.054$ |  $1.405~{}\pm~{}0.192$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.20}~{}\pm~{}\textbf{0.08}$ | $\cellcolor{NiceBlue!25}\textbf{5.14}~{}\pm~{}0.29$ | $655.61~{}\pm~{}12.53$
|  | $\rho$ | $2.277~{}\pm~{}0.013$ | $2.105~{}\pm~{}0.006$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric |  $11.33~{}\pm~{}\textbf{0.25}$ |  $\cellcolor{NiceOrange!25}\textbf{29.64}~{}\pm~{}16.74$ |  $\cellcolor{NiceOrange!25}\textbf{650.94}~{}\pm~{}1.37$
|  | $\rho$ |  $1.619~{}\pm~{}0.780$ |  $1.085~{}\pm~{}0.040$ |  $1.248~{}\pm~{}0.010$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.73}~{}\pm~{}1.54$ | $18.86~{}\pm~{}\textbf{4.26}$ | $661.75~{}\pm~{}\textbf{0.75}$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : task5

##  9 3 July 2023

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{32.83}~{}\pm~{}2.69$ |  $39.62~{}\pm~{}7.89$ | $\cellcolor{NiceGreen!25}\textbf{116.96}~{}\pm~{}5.36$
|  | $\rho$ | $0.512~{}\pm~{}0.008$ |  $0.464~{}\pm~{}0.055$ | $0.481~{}\pm~{}0.015$
| $\rho_{t}=1$ | metric |  $26.27~{}\pm~{}2.85$ |  $20.21~{}\pm~{}\textbf{6.92}$ |  $171.15~{}\pm~{}15.82$
|  | $\rho$ |  $1.109~{}\pm~{}0.057$ |  $0.850~{}\pm~{}0.296$ |  $1.052~{}\pm~{}0.019$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\cellcolor{NiceBlue!25}\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{32.39}~{}\pm~{}3.11$ | $33.80~{}\pm~{}5.96$ | $\cellcolor{NiceGreen!25}\textbf{120.78}~{}\pm~{}\textbf{0.49}$
|  | $\rho$ | $0.518~{}\pm~{}0.030$ | $0.500~{}\pm~{}0.004$ | $0.498~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $30.82~{}\pm~{}\textbf{2.30}$ | $30.71~{}\pm~{}nan$ | 0
|  | $\rho$ | $1.011~{}\pm~{}0.017$ | $0.999~{}\pm~{}nan$ | 0
| none | metric | $23.51~{}\pm~{}2.50$ | $\cellcolor{NiceBlue!25}\textbf{39.84}~{}\pm~{}\textbf{5.91}$ | $173.67~{}\pm~{}3.21$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{95.15}~{}\pm~{}\textbf{0.25}$ | $84.29~{}\pm~{}\textbf{1.28}$ | $\cellcolor{NiceGreen!25}\textbf{101.99}~{}\pm~{}2.89$
|  | $\rho$ | $0.502~{}\pm~{}0.010$ | $0.501~{}\pm~{}0.009$ | $0.503~{}\pm~{}0.014$
| $\rho_{t}=1$ | metric |  $93.48~{}\pm~{}3.15$ |  $75.68~{}\pm~{}10.57$ | $102.42~{}\pm~{}2.15$
|  | $\rho$ |  $1.040~{}\pm~{}0.040$ |  $1.065~{}\pm~{}0.037$ | $1.000~{}\pm~{}0.034$
| none | metric | $94.85~{}\pm~{}0.40$ | $\cellcolor{NiceBlue!25}\textbf{84.92}~{}\pm~{}2.05$ | $103.78~{}\pm~{}\textbf{0.21}$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $74.26~{}\pm~{}42.18$ | $52.03~{}\pm~{}18.94$ |  $108.18~{}\pm~{}\textbf{1.79}$
|  | $\rho$ | $0.514~{}\pm~{}0.013$ | $0.499~{}\pm~{}0.001$ |  $0.530~{}\pm~{}0.044$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.21}~{}\pm~{}\textbf{0.29}$ |  $63.46~{}\pm~{}26.71$ |  $\cellcolor{NiceOrange!25}\textbf{106.95}~{}\pm~{}2.41$
|  | $\rho$ |  $1.073~{}\pm~{}0.080$ |  $1.099~{}\pm~{}0.056$ |  $1.032~{}\pm~{}0.023$
| none | metric | $95.13~{}\pm~{}0.93$ | $\cellcolor{NiceBlue!25}\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.80~{}\pm~{}2.56$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $55.99~{}\pm~{}39.41$ | $61.87~{}\pm~{}11.08$ |  $649.13~{}\pm~{}1.74$
|  | $\rho$ |  $0.973~{}\pm~{}0.048$ | $1.001~{}\pm~{}0.007$ |  $2.436~{}\pm~{}1.663$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{85.94}~{}\pm~{}\textbf{4.89}$ | $\cellcolor{NiceBlue!25}\textbf{74.44}~{}\pm~{}\textbf{5.26}$ | $\cellcolor{NiceBlue!25}\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.226~{}\pm~{}0.032$ | $2.027~{}\pm~{}0.249$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric | $76.58~{}\pm~{}13.39$ | $73.93~{}\pm~{}5.52$ |  $647.78~{}\pm~{}\textbf{1.70}$
|  | $\rho$ | $1.005~{}\pm~{}0.033$ | $1.019~{}\pm~{}0.026$ |  $1.761~{}\pm~{}1.288$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{80.12}~{}\pm~{}\textbf{1.80}$ | $\cellcolor{NiceBlue!25}\textbf{74.77}~{}\pm~{}\textbf{1.12}$ | $\cellcolor{NiceBlue!25}\textbf{130.88}~{}\pm~{}4.40$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.192~{}\pm~{}0.040$ | $1.168~{}\pm~{}0.065$
|  |  |  |  |
Table : task |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.32~{}\pm~{}\textbf{0.24}$ |  $\cellcolor{NiceGreen!25}\textbf{21.74}~{}\pm~{}\textbf{2.81}$ |  $639.84~{}\pm~{}2.46$
|  | $\rho$ | $0.504~{}\pm~{}0.005$ |  $0.524~{}\pm~{}0.013$ |  $0.540~{}\pm~{}0.076$
| $\rho_{t}=1$ | metric |  $10.92~{}\pm~{}0.61$ | $7.99~{}\pm~{}3.07$ | $\cellcolor{NiceOrange!25}\textbf{638.51}~{}\pm~{}\textbf{0.59}$
|  | $\rho$ |  $1.049~{}\pm~{}0.075$ | $1.017~{}\pm~{}0.051$ | $1.014~{}\pm~{}0.058$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.37}~{}\pm~{}0.27$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}0.94$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{29.89}~{}\pm~{}8.57$ | $\cellcolor{NiceGreen!25}\textbf{27.00}~{}\pm~{}11.99$ | $\cellcolor{NiceGreen!25}\textbf{255.62}~{}\pm~{}284.36$
|  | $\rho$ | $0.513~{}\pm~{}0.025$ | $0.507~{}\pm~{}0.012$ | $0.495~{}\pm~{}0.005$
| $\rho_{t}=1$ | metric | 0 | $17.12~{}\pm~{}nan$ | 0
|  | $\rho$ | 0 | $1.016~{}\pm~{}nan$ | 0
| none | metric | $19.63~{}\pm~{}\textbf{6.60}$ | $16.43~{}\pm~{}\textbf{3.51}$ | $500.04~{}\pm~{}\textbf{94.49}$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{96.45}~{}\pm~{}\textbf{0.24}$ | $86.37~{}\pm~{}\textbf{1.27}$ | $\cellcolor{NiceGreen!25}\textbf{117.37}~{}\pm~{}10.88$
|  | $\rho$ | $0.510~{}\pm~{}0.002$ | $0.502~{}\pm~{}0.004$ | $0.493~{}\pm~{}0.006$
| $\rho_{t}=1$ | metric |  $95.83~{}\pm~{}0.99$ |  $64.15~{}\pm~{}28.64$ | $186.65~{}\pm~{}7.84$
|  | $\rho$ |  $1.057~{}\pm~{}0.116$ |  $1.025~{}\pm~{}0.039$ | $1.018~{}\pm~{}0.006$
| none | metric | $95.61~{}\pm~{}0.38$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $648.25~{}\pm~{}\textbf{4.75}$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.701~{}\pm~{}0.003$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $75.12~{}\pm~{}42.76$ | $66.67~{}\pm~{}\textbf{22.70}$ | $640.15~{}\pm~{}\textbf{0.26}$
|  | $\rho$ | $0.505~{}\pm~{}0.004$ | $0.505~{}\pm~{}0.015$ | $0.511~{}\pm~{}0.015$
| $\rho_{t}=1$ | metric |  $95.82~{}\pm~{}\textbf{0.06}$ |  $\cellcolor{NiceOrange!25}\textbf{68.80}~{}\pm~{}28.06$ |  $\cellcolor{NiceOrange!25}\textbf{399.32}~{}\pm~{}278.22$
|  | $\rho$ |  $1.158~{}\pm~{}0.085$ |  $1.022~{}\pm~{}0.036$ |  $1.180~{}\pm~{}0.120$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{96.19}~{}\pm~{}0.29$ | $62.98~{}\pm~{}27.77$ | $644.18~{}\pm~{}0.59$
|  | $\rho$ | $0.858~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.854~{}\pm~{}0.001$
ALIF | $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{11.62}~{}\pm~{}0.27$ |  $\cellcolor{NiceOrange!25}\textbf{5.14}~{}\pm~{}\textbf{0.13}$ | $\cellcolor{NiceOrange!25}\textbf{647.86}~{}\pm~{}13.26$
|  | $\rho$ |  $1.679~{}\pm~{}1.270$ |  $1.029~{}\pm~{}0.005$ | $0.994~{}\pm~{}0.049$
| none | metric | $11.08~{}\pm~{}\textbf{0.25}$ | $5.14~{}\pm~{}0.29$ | $655.61~{}\pm~{}\textbf{12.53}$
|  | $\rho$ | $2.281~{}\pm~{}0.014$ | $2.105~{}\pm~{}0.006$ | $1.650~{}\pm~{}0.023$
ALIF± | $\rho_{t}=1$ | metric |  $10.92~{}\pm~{}\textbf{0.51}$ | $\cellcolor{NiceOrange!25}\textbf{20.29}~{}\pm~{}17.66$ | $\cellcolor{NiceOrange!25}\textbf{651.48}~{}\pm~{}1.90$
|  | $\rho$ |  $1.663~{}\pm~{}1.332$ | $1.014~{}\pm~{}0.013$ | $0.993~{}\pm~{}0.047$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.73}~{}\pm~{}1.54$ | $17.68~{}\pm~{}\textbf{4.21}$ | $661.75~{}\pm~{}\textbf{0.75}$
|  | $\rho$ | $1.090~{}\pm~{}0.005$ | $0.979~{}\pm~{}0.002$ | $0.973~{}\pm~{}0.001$
|  |  |  |  |
Table : task5 |  | depth | 1 | 3 | 5 | 7
---|---|---|---|---|---|---
net | LSC |  |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{42.42}~{}\pm~{}\textbf{4.46}$ | $\cellcolor{NiceGreen!25}\textbf{39.15}~{}\pm~{}6.54$ |  $\cellcolor{NiceGreen!25}\textbf{21.74}~{}\pm~{}\textbf{2.81}$ |  $4.79~{}\pm~{}\textbf{0.15}$
|  | $\rho$ | $0.508~{}\pm~{}0.029$ | $0.515~{}\pm~{}0.038$ |  $0.524~{}\pm~{}0.013$ |  $0.473~{}\pm~{}0.017$
| $\rho_{t}=1$ | metric |  $33.44~{}\pm~{}11.06$ |  $8.15~{}\pm~{}\textbf{2.59}$ | $7.99~{}\pm~{}3.07$ |  $\cellcolor{NiceOrange!25}\textbf{5.43}~{}\pm~{}1.03$
|  | $\rho$ |  $1.221~{}\pm~{}0.049$ |  $1.088~{}\pm~{}0.035$ | $1.017~{}\pm~{}0.051$ |  $1.024~{}\pm~{}0.031$
| none | metric | $38.20~{}\pm~{}6.84$ | $30.47~{}\pm~{}13.34$ | $12.90~{}\pm~{}3.82$ | $5.06~{}\pm~{}0.16$
|  | $\rho$ | $0.366~{}\pm~{}0.001$ | $0.362~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $35.00~{}\pm~{}\textbf{8.56}$ | $31.67~{}\pm~{}6.73$ | $\cellcolor{NiceGreen!25}\textbf{27.00}~{}\pm~{}11.99$ | $\cellcolor{NiceGreen!25}\textbf{17.40}~{}\pm~{}6.75$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ | $0.507~{}\pm~{}0.012$ | $0.514~{}\pm~{}0.028$
| $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{40.82}~{}\pm~{}8.56$ | $\cellcolor{NiceOrange!25}\textbf{41.15}~{}\pm~{}\textbf{1.36}$ | $17.12~{}\pm~{}nan$ | 0
|  | $\rho$ | $1.008~{}\pm~{}0.006$ | $0.999~{}\pm~{}0.002$ | $1.016~{}\pm~{}nan$ | 0
| none | metric | $39.06~{}\pm~{}10.85$ | $34.19~{}\pm~{}5.98$ | $16.43~{}\pm~{}\textbf{3.51}$ | $15.59~{}\pm~{}\textbf{2.31}$
|  | $\rho$ | $1.355~{}\pm~{}0.004$ | $1.346~{}\pm~{}0.003$ | $1.341~{}\pm~{}0.006$ | $1.339~{}\pm~{}0.005$
GRU | $\rho_{t}=0.5$ | metric |  $81.95~{}\pm~{}2.79$ | $85.76~{}\pm~{}\textbf{1.17}$ | $86.37~{}\pm~{}\textbf{1.27}$ | $\cellcolor{NiceGreen!25}\textbf{87.88}~{}\pm~{}\textbf{0.30}$
|  | $\rho$ |  $0.528~{}\pm~{}0.038$ | $0.497~{}\pm~{}0.003$ | $0.502~{}\pm~{}0.004$ | $0.503~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $73.24~{}\pm~{}4.54$ | $81.21~{}\pm~{}2.92$ |  $64.15~{}\pm~{}28.64$ | $75.87~{}\pm~{}4.04$
|  | $\rho$ |  $1.038~{}\pm~{}0.066$ | $1.018~{}\pm~{}0.043$ |  $1.025~{}\pm~{}0.039$ | $0.991~{}\pm~{}0.006$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{1.45}$ | $\cellcolor{NiceBlue!25}\textbf{86.01}~{}\pm~{}2.48$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $86.79~{}\pm~{}1.15$
|  | $\rho$ | $0.751~{}\pm~{}0.002$ | $0.710~{}\pm~{}0.002$ | $0.701~{}\pm~{}0.003$ | $0.698~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $65.87~{}\pm~{}18.78$ | $74.57~{}\pm~{}19.14$ | $66.67~{}\pm~{}\textbf{22.70}$ |  $45.64~{}\pm~{}29.46$
|  | $\rho$ | $0.493~{}\pm~{}0.015$ | $0.503~{}\pm~{}0.008$ | $0.505~{}\pm~{}0.015$ |  $0.560~{}\pm~{}0.064$
| $\rho_{t}=1$ | metric | $48.37~{}\pm~{}\textbf{6.79}$ |  $\cellcolor{NiceOrange!25}\textbf{84.92}~{}\pm~{}\textbf{2.32}$ |  $\cellcolor{NiceOrange!25}\textbf{68.80}~{}\pm~{}28.06$ |  $\cellcolor{NiceOrange!25}\textbf{53.95}~{}\pm~{}25.71$
|  | $\rho$ | $1.007~{}\pm~{}0.050$ |  $1.096~{}\pm~{}0.036$ |  $1.022~{}\pm~{}0.036$ |  $1.043~{}\pm~{}0.073$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{76.69}~{}\pm~{}9.27$ | $84.82~{}\pm~{}2.35$ | $62.98~{}\pm~{}27.77$ | $27.90~{}\pm~{}\textbf{7.79}$
|  | $\rho$ | $0.914~{}\pm~{}0.003$ | $0.879~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.869~{}\pm~{}0.002$
ALIF | $\rho_{t}=1$ | metric | $75.14~{}\pm~{}2.38$ | $45.26~{}\pm~{}25.64$ |  $\cellcolor{NiceOrange!25}\textbf{5.14}~{}\pm~{}\textbf{0.13}$ | $4.87~{}\pm~{}\textbf{0.35}$
|  | $\rho$ | $1.007~{}\pm~{}0.005$ | $0.999~{}\pm~{}0.019$ |  $1.029~{}\pm~{}0.005$ | $0.989~{}\pm~{}0.005$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{77.18}~{}\pm~{}\textbf{1.59}$ | $\cellcolor{NiceBlue!25}\textbf{61.42}~{}\pm~{}\textbf{5.64}$ | $5.14~{}\pm~{}0.29$ | $\cellcolor{NiceBlue!25}\textbf{5.24}~{}\pm~{}0.47$
|  | $\rho$ | $1.853~{}\pm~{}0.023$ | $2.074~{}\pm~{}0.181$ | $2.105~{}\pm~{}0.006$ | $1.956~{}\pm~{}0.206$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{68.17}~{}\pm~{}\textbf{3.62}$ | $45.76~{}\pm~{}13.98$ | $\cellcolor{NiceOrange!25}\textbf{20.29}~{}\pm~{}17.66$ | $\cellcolor{NiceOrange!25}\textbf{5.38}~{}\pm~{}0.61$
|  | $\rho$ | $1.005~{}\pm~{}0.014$ | $1.011~{}\pm~{}0.022$ | $1.014~{}\pm~{}0.013$ | $1.003~{}\pm~{}0.003$
| none | metric | $66.83~{}\pm~{}4.23$ | $\cellcolor{NiceBlue!25}\textbf{68.57}~{}\pm~{}\textbf{3.01}$ | $17.68~{}\pm~{}\textbf{4.21}$ | $5.17~{}\pm~{}\textbf{0.18}$
|  | $\rho$ | $1.460~{}\pm~{}0.007$ | $1.082~{}\pm~{}0.002$ | $0.979~{}\pm~{}0.002$ | $0.933~{}\pm~{}0.001$
|  |  |  |  |  |
Table : stack

##  10 8 July 2023

|  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{31.96}~{}\pm~{}2.82$ |  $39.59~{}\pm~{}5.54$ |  $\cellcolor{NiceGreen!25}\textbf{116.32}~{}\pm~{}4.46$
|  | $\rho$ |  $0.533~{}\pm~{}0.049$ |  $0.539~{}\pm~{}0.064$ |  $0.542~{}\pm~{}0.101$
| $\rho_{t}=1$ | metric |  $29.87~{}\pm~{}3.72$ |  $17.19~{}\pm~{}\textbf{4.46}$ |  $166.02~{}\pm~{}8.47$
|  | $\rho$ |  $1.949~{}\pm~{}0.108$ |  $2.075~{}\pm~{}0.328$ |  $1.782~{}\pm~{}0.084$
| none | metric | $29.02~{}\pm~{}\textbf{2.00}$ | $\cellcolor{NiceBlue!25}\textbf{46.40}~{}\pm~{}12.12$ | $641.65~{}\pm~{}\textbf{1.76}$
|  | $\rho$ | $0.354~{}\pm~{}0.002$ | $0.363~{}\pm~{}0.001$ | $0.353~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{33.20}~{}\pm~{}2.91$ | $33.80~{}\pm~{}5.96$ | $\cellcolor{NiceGreen!25}\textbf{120.78}~{}\pm~{}\textbf{0.49}$
|  | $\rho$ | $0.504~{}\pm~{}0.007$ | $0.500~{}\pm~{}0.004$ | $0.498~{}\pm~{}0.001$
| $\rho_{t}=1$ | metric | $30.39~{}\pm~{}nan$ | $30.71~{}\pm~{}nan$ | 0
|  | $\rho$ | $1.015~{}\pm~{}nan$ | $0.999~{}\pm~{}nan$ | 0
| none | metric | $23.51~{}\pm~{}\textbf{2.50}$ | $\cellcolor{NiceBlue!25}\textbf{39.84}~{}\pm~{}\textbf{5.91}$ | $173.67~{}\pm~{}3.21$
|  | $\rho$ | $1.317~{}\pm~{}0.011$ | $1.349~{}\pm~{}0.007$ | $0.760~{}\pm~{}0.004$
GRU | $\rho_{t}=0.5$ | metric | $95.15~{}\pm~{}0.25$ | $84.29~{}\pm~{}\textbf{1.28}$ | $\cellcolor{NiceGreen!25}\textbf{101.99}~{}\pm~{}2.89$
|  | $\rho$ | $0.502~{}\pm~{}0.010$ | $0.501~{}\pm~{}0.009$ | $0.503~{}\pm~{}0.014$
| $\rho_{t}=1$ | metric |  $\cellcolor{NiceOrange!25}\textbf{95.32}~{}\pm~{}\textbf{0.17}$ |  $79.83~{}\pm~{}3.39$ | $104.09~{}\pm~{}\textbf{0.16}$
|  | $\rho$ |  $1.076~{}\pm~{}0.120$ |  $1.055~{}\pm~{}0.048$ | $1.001~{}\pm~{}0.007$
| none | metric | $94.85~{}\pm~{}0.40$ | $\cellcolor{NiceBlue!25}\textbf{84.92}~{}\pm~{}2.05$ | $103.78~{}\pm~{}0.21$
|  | $\rho$ | $0.710~{}\pm~{}0.004$ | $0.719~{}\pm~{}0.002$ | $0.694~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric |  $74.26~{}\pm~{}42.18$ | $52.03~{}\pm~{}18.94$ |  $\cellcolor{NiceGreen!25}\textbf{108.13}~{}\pm~{}1.83$
|  | $\rho$ |  $0.521~{}\pm~{}0.023$ | $0.499~{}\pm~{}0.001$ |  $0.528~{}\pm~{}0.041$
| $\rho_{t}=1$ | metric |  $95.07~{}\pm~{}\textbf{0.40}$ |  $68.50~{}\pm~{}27.12$ |  $110.32~{}\pm~{}\textbf{1.77}$
|  | $\rho$ |  $1.121~{}\pm~{}0.216$ |  $1.090~{}\pm~{}0.076$ |  $1.022~{}\pm~{}0.017$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{95.13}~{}\pm~{}0.93$ | $\cellcolor{NiceBlue!25}\textbf{83.61}~{}\pm~{}\textbf{1.97}$ | $108.80~{}\pm~{}2.56$
|  | $\rho$ | $0.876~{}\pm~{}0.005$ | $0.887~{}\pm~{}0.002$ | $0.847~{}\pm~{}0.003$
ALIF | $\rho_{t}=1$ | metric |  $47.72~{}\pm~{}35.20$ |  $60.62~{}\pm~{}11.42$ |  $649.12~{}\pm~{}1.48$
|  | $\rho$ |  $1.900~{}\pm~{}0.885$ |  $1.556~{}\pm~{}0.741$ |  $3.318~{}\pm~{}0.579$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{85.94}~{}\pm~{}\textbf{4.89}$ | $\cellcolor{NiceBlue!25}\textbf{74.44}~{}\pm~{}\textbf{5.26}$ | $\cellcolor{NiceBlue!25}\textbf{130.18}~{}\pm~{}\textbf{1.06}$
|  | $\rho$ | $2.226~{}\pm~{}0.032$ | $2.027~{}\pm~{}0.249$ | $1.690~{}\pm~{}0.022$
ALIF± | $\rho_{t}=1$ | metric |  $77.23~{}\pm~{}12.63$ |  $72.89~{}\pm~{}6.44$ |  $647.75~{}\pm~{}\textbf{0.47}$
|  | $\rho$ |  $1.064~{}\pm~{}0.137$ |  $1.049~{}\pm~{}0.043$ |  $3.444~{}\pm~{}0.157$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{80.12}~{}\pm~{}\textbf{1.80}$ | $\cellcolor{NiceBlue!25}\textbf{74.77}~{}\pm~{}\textbf{1.12}$ | $\cellcolor{NiceBlue!25}\textbf{133.78}~{}\pm~{}6.82$
|  | $\rho$ | $1.212~{}\pm~{}0.007$ | $1.192~{}\pm~{}0.040$ | $1.178~{}\pm~{}0.057$
|  |  |  |  |
Table : task |  | depth | 1 | 3 | 5 | 7
---|---|---|---|---|---|---
net | LSC |  |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric |  $\cellcolor{NiceGreen!25}\textbf{44.50}~{}\pm~{}\textbf{6.15}$ |  $\cellcolor{NiceGreen!25}\textbf{37.20}~{}\pm~{}5.50$ |  $\cellcolor{NiceGreen!25}\textbf{18.85}~{}\pm~{}\textbf{1.75}$ |  $6.11~{}\pm~{}2.53$
|  | $\rho$ |  $0.603~{}\pm~{}0.078$ |  $0.559~{}\pm~{}0.068$ |  $0.600~{}\pm~{}0.015$ |  $0.705~{}\pm~{}0.196$
| $\rho_{t}=1$ | metric |  $31.10~{}\pm~{}13.86$ |  $9.28~{}\pm~{}\textbf{2.46}$ |  $9.36~{}\pm~{}3.25$ |  $\cellcolor{NiceOrange!25}\textbf{7.57}~{}\pm~{}1.83$
|  | $\rho$ |  $2.630~{}\pm~{}0.500$ |  $1.640~{}\pm~{}0.063$ |  $1.406~{}\pm~{}0.052$ |  $1.352~{}\pm~{}0.027$
| none | metric | $38.20~{}\pm~{}6.84$ | $30.47~{}\pm~{}13.34$ | $12.90~{}\pm~{}3.82$ | $5.06~{}\pm~{}\textbf{0.16}$
|  | $\rho$ | $0.366~{}\pm~{}0.001$ | $0.362~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.001$ | $0.361~{}\pm~{}0.000$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $35.00~{}\pm~{}\textbf{8.56}$ | $31.67~{}\pm~{}6.73$ | $\cellcolor{NiceGreen!25}\textbf{27.00}~{}\pm~{}11.99$ | $\cellcolor{NiceGreen!25}\textbf{20.01}~{}\pm~{}2.82$
|  | $\rho$ | $0.500~{}\pm~{}0.003$ | $0.498~{}\pm~{}0.002$ | $0.507~{}\pm~{}0.012$ | $0.500~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $38.29~{}\pm~{}12.14$ | $\cellcolor{NiceOrange!25}\textbf{41.15}~{}\pm~{}\textbf{1.36}$ | 0 | 0
|  | $\rho$ |  $1.063~{}\pm~{}0.083$ | $0.999~{}\pm~{}0.002$ | 0 | 0
| none | metric | $\cellcolor{NiceBlue!25}\textbf{39.06}~{}\pm~{}10.85$ | $34.19~{}\pm~{}5.98$ | $16.43~{}\pm~{}\textbf{3.51}$ | $15.59~{}\pm~{}\textbf{2.31}$
|  | $\rho$ | $1.355~{}\pm~{}0.004$ | $1.346~{}\pm~{}0.003$ | $1.341~{}\pm~{}0.006$ | $1.339~{}\pm~{}0.005$
GRU | $\rho_{t}=0.5$ | metric | $82.26~{}\pm~{}2.98$ | $85.76~{}\pm~{}\textbf{1.17}$ | $86.37~{}\pm~{}\textbf{1.27}$ | $\cellcolor{NiceGreen!25}\textbf{87.88}~{}\pm~{}\textbf{0.30}$
|  | $\rho$ | $0.506~{}\pm~{}0.015$ | $0.497~{}\pm~{}0.003$ | $0.502~{}\pm~{}0.004$ | $0.503~{}\pm~{}0.003$
| $\rho_{t}=1$ | metric |  $71.84~{}\pm~{}2.51$ | $80.82~{}\pm~{}2.81$ | $78.09~{}\pm~{}1.68$ | $75.87~{}\pm~{}4.04$
|  | $\rho$ |  $1.045~{}\pm~{}0.081$ | $1.014~{}\pm~{}0.042$ | $1.019~{}\pm~{}0.028$ | $0.991~{}\pm~{}0.006$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{84.55}~{}\pm~{}\textbf{1.45}$ | $\cellcolor{NiceBlue!25}\textbf{86.01}~{}\pm~{}2.48$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $86.79~{}\pm~{}1.15$
|  | $\rho$ | $0.751~{}\pm~{}0.002$ | $0.710~{}\pm~{}0.002$ | $0.701~{}\pm~{}0.003$ | $0.698~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $74.73~{}\pm~{}15.83$ | $74.57~{}\pm~{}19.14$ | $67.84~{}\pm~{}23.91$ |  $32.87~{}\pm~{}33.20$
|  | $\rho$ | $0.497~{}\pm~{}0.008$ | $0.503~{}\pm~{}0.008$ | $0.515~{}\pm~{}0.033$ |  $0.545~{}\pm~{}0.044$
| $\rho_{t}=1$ | metric |  $44.83~{}\pm~{}\textbf{4.38}$ |  $\cellcolor{NiceOrange!25}\textbf{85.26}~{}\pm~{}\textbf{1.02}$ | $\cellcolor{NiceOrange!25}\textbf{70.59}~{}\pm~{}\textbf{22.26}$ |  $\cellcolor{NiceOrange!25}\textbf{56.51}~{}\pm~{}18.77$
|  | $\rho$ |  $1.052~{}\pm~{}0.037$ |  $1.159~{}\pm~{}0.049$ | $1.017~{}\pm~{}0.035$ |  $1.105~{}\pm~{}0.194$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{76.69}~{}\pm~{}9.27$ | $84.82~{}\pm~{}2.35$ | $62.98~{}\pm~{}27.77$ | $27.90~{}\pm~{}\textbf{7.79}$
|  | $\rho$ | $0.914~{}\pm~{}0.003$ | $0.879~{}\pm~{}0.002$ | $0.871~{}\pm~{}0.001$ | $0.869~{}\pm~{}0.002$
ALIF | $\rho_{t}=1$ | metric | $75.14~{}\pm~{}2.38$ |  $56.79~{}\pm~{}20.71$ |  $5.06~{}\pm~{}\textbf{0.20}$ | $4.82~{}\pm~{}\textbf{0.26}$
|  | $\rho$ | $1.007~{}\pm~{}0.005$ |  $1.151~{}\pm~{}0.217$ |  $1.026~{}\pm~{}0.006$ | $1.010~{}\pm~{}0.025$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{77.18}~{}\pm~{}\textbf{1.59}$ | $\cellcolor{NiceBlue!25}\textbf{61.42}~{}\pm~{}\textbf{5.64}$ | $\cellcolor{NiceBlue!25}\textbf{5.13}~{}\pm~{}0.24$ | $\cellcolor{NiceBlue!25}\textbf{5.22}~{}\pm~{}0.39$
|  | $\rho$ | $1.853~{}\pm~{}0.023$ | $2.074~{}\pm~{}0.181$ | $2.103~{}\pm~{}0.006$ | $1.988~{}\pm~{}0.180$
ALIF± | $\rho_{t}=1$ | metric | $\cellcolor{NiceOrange!25}\textbf{68.17}~{}\pm~{}\textbf{3.62}$ |  $42.39~{}\pm~{}20.98$ | $16.37~{}\pm~{}16.41$ | $5.13~{}\pm~{}0.62$
|  | $\rho$ | $1.005~{}\pm~{}0.014$ |  $1.120~{}\pm~{}0.182$ | $1.018~{}\pm~{}0.014$ | $1.014~{}\pm~{}0.020$
| none | metric | $66.83~{}\pm~{}4.23$ | $\cellcolor{NiceBlue!25}\textbf{68.57}~{}\pm~{}\textbf{3.01}$ | $\cellcolor{NiceBlue!25}\textbf{17.68}~{}\pm~{}\textbf{4.21}$ | $\cellcolor{NiceBlue!25}\textbf{5.17}~{}\pm~{}\textbf{0.18}$
|  | $\rho$ | $1.460~{}\pm~{}0.007$ | $1.082~{}\pm~{}0.002$ | $0.979~{}\pm~{}0.002$ | $0.933~{}\pm~{}0.001$
|  |  |  |  |  |
Table : stack |  | task | sl-MNIST$\uparrow$ | SHD$\uparrow$ | PTB$\downarrow$
---|---|---|---|---|---
net | LSC |  |  |  |
RNN $\sigma$ | $\rho_{t}=0.5$ | metric | $11.32~{}\pm~{}\textbf{0.24}$ |  $\cellcolor{NiceGreen!25}\textbf{18.85}~{}\pm~{}\textbf{1.75}$ |  $640.81~{}\pm~{}4.24$
|  | $\rho$ | $0.504~{}\pm~{}0.005$ |  $0.600~{}\pm~{}0.015$ |  $0.624~{}\pm~{}0.167$
| $\rho_{t}=1$ | metric |  $10.78~{}\pm~{}0.82$ |  $9.36~{}\pm~{}3.25$ |  $\cellcolor{NiceOrange!25}\textbf{638.84}~{}\pm~{}0.98$
|  | $\rho$ |  $1.429~{}\pm~{}0.083$ |  $1.406~{}\pm~{}0.052$ |  $1.232~{}\pm~{}0.211$
| none | metric | $\cellcolor{NiceBlue!25}\textbf{11.37}~{}\pm~{}0.27$ | $12.90~{}\pm~{}3.82$ | $643.23~{}\pm~{}\textbf{0.94}$
|  | $\rho$ | $0.357~{}\pm~{}0.002$ | $0.361~{}\pm~{}0.001$ | $0.351~{}\pm~{}0.001$
RNN $ReLU$ | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{31.69}~{}\pm~{}9.48$ | $\cellcolor{NiceGreen!25}\textbf{27.00}~{}\pm~{}11.99$ | $\cellcolor{NiceGreen!25}\textbf{255.62}~{}\pm~{}284.36$
|  | $\rho$ | $0.505~{}\pm~{}0.009$ | $0.507~{}\pm~{}0.012$ | $0.495~{}\pm~{}0.005$
| $\rho_{t}=1$ | metric | 0 | 0 | 0
|  | $\rho$ | 0 | 0 | 0
| none | metric | $19.63~{}\pm~{}\textbf{6.60}$ | $16.43~{}\pm~{}\textbf{3.51}$ | $500.04~{}\pm~{}\textbf{94.49}$
|  | $\rho$ | $1.323~{}\pm~{}0.009$ | $1.341~{}\pm~{}0.006$ | $0.769~{}\pm~{}0.001$
GRU | $\rho_{t}=0.5$ | metric | $\cellcolor{NiceGreen!25}\textbf{96.45}~{}\pm~{}\textbf{0.24}$ | $86.37~{}\pm~{}\textbf{1.27}$ | $\cellcolor{NiceGreen!25}\textbf{117.37}~{}\pm~{}10.88$
|  | $\rho$ | $0.510~{}\pm~{}0.002$ | $0.502~{}\pm~{}0.004$ | $0.493~{}\pm~{}0.006$
| $\rho_{t}=1$ | metric | $96.11~{}\pm~{}0.68$ | $78.09~{}\pm~{}1.68$ | $186.65~{}\pm~{}7.84$
|  | $\rho$ | $0.996~{}\pm~{}0.064$ | $1.019~{}\pm~{}0.028$ | $1.018~{}\pm~{}0.006$
| none | metric | $95.61~{}\pm~{}0.38$ | $\cellcolor{NiceBlue!25}\textbf{86.81}~{}\pm~{}3.05$ | $648.25~{}\pm~{}\textbf{4.75}$
|  | $\rho$ | $0.695~{}\pm~{}0.003$ | $0.701~{}\pm~{}0.003$ | $0.687~{}\pm~{}0.002$
LSTM | $\rho_{t}=0.5$ | metric | $75.12~{}\pm~{}42.76$ | $67.84~{}\pm~{}23.91$ | $640.08~{}\pm~{}\textbf{0.26}$
|  | $\rho$ | $0.507~{}\pm~{}0.006$ | $0.515~{}\pm~{}0.033$ | $0.511~{}\pm~{}0.012$