different $\lambda$’s: $10^{-2}$ and $10^{-4}$.

Tables 11 and 12 show the search spaces of hyperparameters. For each tuning
trial, we train EARLIEST and evaluate its averaged per-class accuracy per
every 500 training iterations. After all training iterations, we save the best
averaged per-class accuracy. After all tuning trials, we select the best
combination of the hyperparameters, which is shown in Tables 11 and 12 in cyan
letters. The approximated runtime of a single tuning trial is 14 hours, and
the GPU consumption is 2.0 GBs.

Table 11:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-2}$ in Figure 5: NMNIST-100f. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 200
Table 12:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-4}$ in Figure 5: NMNIST-100f. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 200

###  I.4 Figure 5: UCF101

##### MSPRT-TANDEM and NP test

The approximation formula is the M-TANDEM formula. The loss function consists
of the LSEL and the multiplet loss. The temporal integrator is Peephole LSTM
[17] with the hidden state dimensions set to 256 followed by a fully-connected
layer to output logits for classification. The temporal integrator has
2,387,456 trainable parameters. The batch size is fixed to 256. The number of
training iterations is 10,000. We use the effective number [9] as the cost
matrix of the LSEL, instead of $1/M_{k}$, to avoid over-emphasizing the
minority class and to simplify the parameter tuning (only one extra parameter
$\beta$ is introduced).

Table 13 shows the search space of hyperparameters. For each tuning trial, we
train the temporal integrator and evaluate its mean averaged per-class
accuracy per every 200 training iterations. After all training iterations, we
save the best mean averaged per-class accuracy. After all tuning trials, we
select the best combination of the hyperparameters, which is shown in Table 13
in cyan letters. The approximated runtime of one statistics trial is 8 hours,
and the GPU consumption is 16–32 GBs.

Table 13:  Hyperparameter Search Space of MSPRT-TANDEM in Figure 5: UCF101. The best hyperparameter combination is highlighted in cyan. $\gamma$ is defined as $L_{\mathrm{total}}=L_{\mathrm{mult}}+\gamma L_{\mathrm{LSEL}}$. $\beta$ controls the cost matrix [9].  Order |  { 0, 10, 25, 40, 49 }
---|---
Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { Adam, RMSprop }
$\beta$ |  { 0.99, 0.999, 0.9999, 0.99999, 1. }
# tuning trials | 100

##### LSTM-s/m

The backbone model is Peephole LSTM with the hidden state dimensions set to
256 followed by a fully-connected layer to output logits for classification.
LSTM has 2,387,456 trainable parameters. The batch size is fixed to 256. The
number of training iterations is 5,000.

Tables 14 and 15 show the search spaces of hyperparameters. For each tuning
trial, we train LSTM and evaluate its mean averaged per-class accuracy per
every 200 training iterations. After all training iterations, we save the best
mean averaged per-class accuracy. After all tuning trials, we select the best
combination of the hyperparameters, which is shown in Tables 14 and 15 in cyan
letters. The approximated runtime of one statistics trial is 3 hours, and the
GPU consumption is 2.6 GBs.

Table 14:  Hyperparameter Search Space of LSTM-s in Figure 5: UCF101. The best hyperparameter combination is highlighted in cyan. $\gamma$ controls the strength of monotonicity and is defined as $L_{\mathrm{total}}=L_{\mathrm{cross\mbox{-}entropy}}+\gamma L_{\mathrm{ranking}}$ [47].  Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-2}$, $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100
Table 15:  Hyperparameter Search Space of LSTM-m in Figure 5: UCF101. The best hyperparameter combination is highlighted in cyan. $\gamma$ controls the strength of monotonicity and is defined as $L_{\mathrm{total}}=L_{\mathrm{cross\mbox{-}entropy}}+\gamma L_{\mathrm{ranking}}$ [47].  Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-2}$, $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100

##### EARLIEST

The main backbone is LSTM [32] with the hidden state dimensions set to 256.
The whole architecture has 2,387,817 trainable parameters. The batch size is
256. The number of training iterations is 5,000. We train EARLIEST with two
different $\lambda$’s: $10^{-1}$ and $10^{-10}$.

Tables 16 and 17 show the search spaces of hyperparameters. For each tuning
trial, we train EARLIEST and evaluate its averaged per-class accuracy per
every 500 training iterations. After all training iterations, we save the best
averaged per-class accuracy. After all tuning trials, we select the best
combination of the hyperparameters, which is shown in Tables 16 and 17 in cyan
letters. The approximated runtime of a single tuning trial is 0.5 hours, and
the GPU consumption is 2.0 GBs.

Table 16:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-1}$ in Figure 5: UCF101. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100
Table 17:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-10}$ in Figure 5: UCF101. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100

###  I.5 Figure 5: HMDB51

##### MSPRT-TANDEM and NP test

The approximation formula is the M-TANDEM formula. The loss function consists
of the LSEL and the multiplet loss. The temporal integrator is Peephole LSTM
[17] with the hidden state dimensions set to 256 followed by a fully-connected
layer to output logits for classification. The temporal integrator has
2,374,656 trainable parameters. The batch size is fixed to 128. The number of
training iterations is 10,000. We use the effective number [9] as the cost
matrix of the LSEL, instead of $1/M_{k}$, to avoid over-emphasising the
minority class and to simplify the parameter tuning (only one extra parameter
$\beta$ is introduced).

Table 18 shows the search space of hyperparameters. For each tuning trial, we
train the temporal integrator and evaluate its mean averaged per-class
accuracy per every 200 training iterations. After all training iterations, we
save the best mean averaged per-class accuracy. After all tuning trials, we
select the best combination of the hyperparameters, which is shown in Table 18
in cyan letters. The approximated runtime of one statistics trial is 8 hours,
and the GPU consumption is 16–32 GBs.

Table 18:  Hyperparameter Search Space of MSPRT-TANDEM in Figure 5: HMDB51. The best hyperparameter combination is highlighted in cyan. $\gamma$ is defined as $L_{\mathrm{total}}=L_{\mathrm{mult}}+\gamma L_{\mathrm{LSEL}}$. $\beta$ controls the cost matrix [9].  Order |  { 0, 10, 40, 60, 78 }
---|---
Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { Adam, RMSprop }
$\beta$ |  { 0.99, 0.999, 0.9999, 0.99999, 1. }
# tuning trials | 100

##### LSTM-s/m

The backbone model is Peephole LSTM with the hidden state dimensions set to
256 followed by a fully-connected layer to output logits for classification.
LSTM has 2,374,656 trainable parameters. The batch size is fixed to 128. The
number of training iterations is 10,000.

Tables 19 and 20 show the search spaces of hyperparameters. For each tuning
trial, we train LSTM and evaluate its mean averaged per-class accuracy per
every 200 training iterations. After all training iterations, we save the best
mean averaged per-class accuracy. After all tuning trials, we select the best
combination of the hyperparameters, which is shown in Tables 19 and 20 in cyan
letters. The approximated runtime of one statistics trial is 3 hours, and the
GPU consumption is 2.6 GBs.

Table 19:  Hyperparameter Search Space of LSTM-s in Figure 5: HMDB51. The best hyperparameter combination is highlighted in cyan. $\gamma$ controls the strength of monotonicity and is defined as $L_{\mathrm{total}}=L_{\mathrm{cross\mbox{-}entropy}}+\gamma L_{\mathrm{ranking}}$ [47].  Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-2}$, $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { RMSprop }
# tuning trials | 100
Table 20:  Hyperparameter Search Space of LSTM-m in Figure 5: HMDB51. The best hyperparameter combination is highlighted in cyan. $\gamma$ controls the strength of monotonicity and is defined as $L_{\mathrm{total}}=L_{\mathrm{cross\mbox{-}entropy}}+\gamma L_{\mathrm{ranking}}$ [47].  Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-2}$, $10^{-1}$, $1$, $10$, $10^{2}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100

##### EARLIEST

The main backbone is LSTM [32] with the hidden state dimensions set to 256.
The whole architecture has 2,374,967 trainable parameters. The batch size is
256. The number of training iterations is 5,000. We train EARLIEST with two
different $\lambda$’s: $10^{-1}$ and $10^{-10}$.

Tables 21 and 22 show the search spaces of hyperparameters. For each tuning
trial, we train EARLIEST and evaluate its averaged per-class accuracy per
every 500 training iterations. After all training iterations, we save the best
averaged per-class accuracy. After all tuning trials, we select the best
combination of the hyperparameters, which is shown in Tables 21 and 22 in cyan
letters. The approximated runtime of a single tuning trial is 0.5 hours, and
the GPU consumption is 2.0 GBs.

Table 21:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-1}$ in Figure 5: HMDB51. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100
Table 22:  Hyperparameter Search Space of EARLIEST with $\lambda=10^{-10}$ in Figure 5: HMDB51. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-1}$, $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
---|---
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop }
# tuning trials | 100

###  I.6 Figure 1: LLR Trajectories

We plot ten different $i$’s randomly selected from the validation set of
NMNIST-100f. The base temporal integrator is selected from the models used for
NMNIST-100f in Figure 5. More example trajectories are shown in Figure 10 in
Appendix J.

###  I.7 Figure 9: Ablation Study of LSEL and Multiplet Loss

The training procedure is totally similar to that of Figure 5. Tables 23 and
24 show the search spaces of hyperparameters. “TANDEM LSEL+Mult” is the same
model as “MSPRT-TANDEM” in Figure 5 (NMNIST-100f).

Table 23:  Hyperparameter Search Space of “TANDEM Mult” in Figure 9. The best hyperparameter combination is highlighted in cyan.  Order |  { 0, 25, 50, 75, 99 }
---|---
Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ | N/A
Optimizer |  { Adam, RMSprop }
# tuning trials | 200
Table 24:  Hyperparameter Search Space of “TANDEM LSEL” in Figure 9. The best hyperparameter combination is highlighted in cyan.  Order |  { 0, 25, 50, 75, 99 }
---|---
Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-1}$, $1$, $10$, $10^{2}$, $10^{3}$}
Optimizer |  { Adam, RMSprop }
# tuning trials | 200

###  I.8 Figure 2 Left: LSEL v.s. Conventional DRE Losses

We define the loss functions used in Figure 2. Conventional DRE losses are
often biased (LLLR), unbounded (LSIF and DSKL), or numerically unstable (LSIF,
LSIFwC, and BARR), especially when applied to multiclass classification,
leading to suboptimal performances (Figure 2).) Because conventional DRE
losses are restricted to binary DRE, we modify two LSIF-based and three KLIEP-
based loss functions for DRME. The logistic loss we use is introduced in
Appendix D.

The original LSIF [35] is based on a kernel method and estimates density
ratios via minimizing the mean squared error between $p$ and $\hat{r}q\,$ (
$\hat{r}:=\hat{p}/\hat{q}$ ). We define a variant of LSIF for DRME as

$\displaystyle\hat{L}_{\rm
LSIF}:=\sum_{t\in[T]}\sum_{\begin{subarray}{c}k,l\in[K]\\\ (k\neq
l)\end{subarray}}\left[\frac{1}{M_{l}}\sum_{i\in
I_{l}}\hat{\Lambda}^{2}_{kl}(X_{i}^{(1,t)})-\frac{1}{M_{k}}\sum_{i\in
I_{k}}\hat{\Lambda}_{kl}(X_{i}^{(1,t)})\right]\,,$ (61)

where the likelihood ratio is denoted by
$\hat{\Lambda}(X):=e^{\hat{\lambda}(X)}=\hat{p}(X|k)/\hat{p}(X|l)$. Because of
the $k,l$-summation, $\hat{L}_{\rm LSIF}$ is symmetric with respect to the
denominator and numerator, unlike the original LSIF. Therefore, we can expect
that $\hat{L}_{\rm LSIF}$ is more stable than the original one. However,
$\hat{L}_{\rm LSIF}$ inherits the problems of the original LSIF; it is
unbounded and numerically unstable. The latter is due to dealing with
$\hat{\Lambda}$ directly, which easily explodes when the denominator is small.
This instability is not negligible, especially when LSIF is used with DNNs.
The following LSIF with Constraint (LSIFwC)999Do not confuse this with cLSIF
[35]. avoids the explosion by adding a constraint:

$\displaystyle\hat{L}_{\rm
LSIFwC}:=\sum_{t\in[T]}\sum_{\begin{subarray}{c}k,l\in[K]\\\ (k\neq
l)\end{subarray}}\bigg{[}$ $\displaystyle\frac{1}{M_{l}}\sum_{i\in
I_{l}}\hat{\Lambda}^{2}_{kl}(X_{i}^{(1,t)})-\frac{1}{M_{k}}\sum_{i\in
I_{k}}\hat{\Lambda}_{kl}(X_{i}^{(1,t)})$
$\displaystyle+\gamma\bigg{|}\frac{1}{M_{l}}\sum_{i\in
I_{l}}\hat{\Lambda}_{kl}(X_{i}^{(1,t)})-1\bigg{|}\bigg{]}\,,$ (62)

where $\gamma>0$ is a hyperparameter. $\hat{L}_{LSIFwC}$ is symmetric and
bounded for sufficiently large $\gamma$. However, it is still numerically
unstable due to $\hat{\Lambda}$. Note that the constraint term is equivalent
to the probability normalization $\int
dx\,p(x|l)\,(\hat{p}(x|k)/\hat{p}(x|l))=1$.

DSKL [38], BARR [38], and LLLR [13] are based on KLIEP [58], which estimates
density ratios via minimizing the Kullback-Leibler divergence [41] between $p$
and $\hat{r}q\,$ ( $\hat{r}:=\hat{p}/\hat{q}$ ). We define a variant of DSKL
for DRME as

$\displaystyle\hat{L}_{\rm
DSKL}:=\sum_{t\in[T]}\sum_{\begin{subarray}{c}k,l\in[K]\\\ (k\neq
l)\end{subarray}}\left[\frac{1}{M_{l}}\sum_{i\in
I_{l}}\hat{\lambda}_{kl}(X_{i}^{(1,t)})-\frac{1}{M_{k}}\sum_{i\in
I_{k}}\hat{\lambda}_{kl}(X_{i}^{(1,t)})\right]$ (63)

The original DSKL is symmetric, and the same is true for $\hat{L}_{\rm BARR}$,
while the original KLIEP is not. $\hat{L}_{\rm BARR}$ is relatively stable
compared with $\hat{L}_{\rm LSIF}$ and $\hat{L}_{\rm LSIFwC}$ because it does
not include $\hat{\Lambda}$ but $\hat{\lambda}$; still, $\hat{L}_{\rm DSKL}$
is unbounded and can diverge. BARR for DRME is

$\displaystyle\hat{L}_{\rm
BARR}:=\sum_{t\in[T]}\sum_{\begin{subarray}{c}k,l\in[K]\\\ (k\neq
l)\end{subarray}}\left[-\frac{1}{M_{k}}\sum_{i\in
I_{k}}\hat{\lambda}_{kl}(X_{i}^{(1,t)})+\gamma\left|\frac{1}{M_{l}}\sum_{i\in
I_{l}}\hat{\Lambda}_{kl}(X_{i}^{(1,t)})-1\right|\right]\,,$ (64)

where $\gamma>0$ is a hyperparameter. $\hat{L}_{\rm BARR}$ is symmetric and
bounded because of the second term but is numerically unstable because of
$\hat{\Lambda}$. LLLR for DRME is

$\displaystyle\hat{L}_{\rm
LLLR}:=\sum_{t\in[T]}\sum_{\begin{subarray}{c}k,l\in[K]\\\ (k\neq
l)\end{subarray}}\frac{1}{M_{k}+M_{l}}\left[\sum_{i\in
I_{l}}\sigma(\hat{\lambda}_{kl}(X_{i}^{(1,t)}))+\sum_{i\in
I_{k}}(1-\sigma(\hat{\lambda}_{kl}(X_{i}^{(1,t)})))\right]\,,$ (65)

where $\sigma$ is the sigmoid function. LLLR is symmetric, bounded, and
numerically stable but is biased in the sense that it does not necessarily
converge to the optimal solution $\lambda$; i.e., the probability
normalization constraint, which is explicitly included in the original KLIEP,
cannot be exactly satisfied. Finally, the logistic loss is defined as (52).

All the models share the same feature extractor (Appendix I.1) and temporal
integrator (Appendix I.2) without the M-TANDEM(wO) approximation or multiplet
loss. The search spaces of hyperparameters are given in Tables 25–31. The
other setting follows Appendix I.2.

Table 25:  Hyperparameter Search Space of LSIF in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 26:  Hyperparameter Search Space of LSIFwC in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-4}$, $10^{-3}$, $10^{-2}$, $1$, $10$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 27:  Hyperparameter Search Space of DSKL in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 28:  Hyperparameter Search Space of BARR in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-4}$, $10^{-3}$, $10^{-2}$, $1$, $10$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 29:  Hyperparameter Search Space of LLLR in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 30:  Hyperparameter Search Space of Logistic Loss in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150
Table 31:  Hyperparameter Search Space of LSEL in Figure 2 Left. The best hyperparameter combination is highlighted in cyan.  Learning rate |  { $10^{-3}$, $10^{-4}$, $10^{-5}$, $10^{-6}$ }
---|---
Weight decay |  { $10^{-2}$, $10^{-3}$, $10^{-4}$, $10^{-5}$ }
Optimizer |  { Adam, RMSprop, Momentum }
# tuning trials | 150

###  I.9 Figure 2 Right: LSEL v.s. Logistic Loss

The experimental condition follows that of Appendix I.3. The logistic loss is
defined as (52). The hyperparameters are given in Tables 32 and 33.

Table 32:  Hyperparameter Search Space of Logistic Loss in Figure 2 Right. The best hyperparameter combination is highlighted in cyan.  Order |  { 0, 25, 50, 75, 99 }
---|---
Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-1}$, $1$, $10$, $10^{2}$, $10^{3}$}
Optimizer |  { Adam, RMSprop }
# tuning trials | 300
Table 33:  Hyperparameter Search Space of LSEL in Figure 2 Right. The best hyperparameter combination is highlighted in cyan.  Order |  { 0, 25, 50, 75, 99 }
---|---
Learning rate |  { $10^{-2}$, $10^{-3}$, $10^{-4}$ }
Weight decay |  { $10^{-3}$, $10^{-4}$, $10^{-5}$ }
$\gamma$ |  { $10^{-1}$, $1$, $10$, $10^{2}$, $10^{3}$}
Optimizer |  { Adam, RMSprop }
# tuning trials | 300

###  I.10 Exact Error Rates in Figures 5 and 9

Tables 34–41 show the averaged per-class error rates plotted in the figures in
Section 4.

Table 34:  Averaged Per-Class Error Rates (%) and SEM of Figure 2 (Left:
NMNIST-H). Blanks mean N/A.

Time | LSIF | LSIFwC | DSKL | BARR
---|---|---|---|---
1.00 |  83.822 $\pm$ 2.142 |  81.585 $\pm$ 2.908 |  |  66.579 $\pm$ 0.108
2.00 |  73.330 $\pm$ 2.142 |  67.749 $\pm$ 2.908 |  45.145 $\pm$ 0.170 |  45.902 $\pm$ 0.108
3.00 |  70.492 $\pm$ 2.142 |  63.237 $\pm$ 2.908 |  30.719 $\pm$ 0.170 |  30.698 $\pm$ 0.108
4.00 |  68.258 $\pm$ 2.142 |  60.061 $\pm$ 2.908 |  20.409 $\pm$ 0.170 |  20.172 $\pm$ 0.108
5.00 |  66.485 $\pm$ 2.142 |  57.860 $\pm$ 2.908 |  18.102 $\pm$ 0.170 |  12.990 $\pm$ 0.108
6.00 |  64.975 $\pm$ 2.142 |  55.939 $\pm$ 2.908 |  17.899 $\pm$ 0.170 |  8.374 $\pm$ 0.108
7.00 |  63.549 $\pm$ 2.142 |  54.317 $\pm$ 2.908 |  14.442 $\pm$ 0.170 |  5.529 $\pm$ 0.108
8.00 |  62.137 $\pm$ 2.142 |  52.635 $\pm$ 2.908 |  12.942 $\pm$ 0.170 |  3.993 $\pm$ 0.108
9.00 |  60.551 $\pm$ 2.142 |  50.886 $\pm$ 2.908 |  11.681 $\pm$ 0.170 |  3.202 $\pm$ 0.108
10.00 |  58.657 $\pm$ 2.142 |  48.947 $\pm$ 2.908 |  10.368 $\pm$ 0.170 |  2.787 $\pm$ 0.108
11.00 |  56.716 $\pm$ 2.142 |  46.501 $\pm$ 2.908 |  9.858 $\pm$ 0.170 |  2.563 $\pm$ 0.108
12.00 |  54.976 $\pm$ 2.142 |  44.114 $\pm$ 2.908 |  8.493 $\pm$ 0.170 |  2.457 $\pm$ 0.108
13.00 |  52.800 $\pm$ 2.142 |  41.901 $\pm$ 2.908 |  7.909 $\pm$ 0.170 |  2.404 $\pm$ 0.108
14.00 |  49.940 $\pm$ 2.142 |  40.073 $\pm$ 2.908 |  7.151 $\pm$ 0.170 |  2.376 $\pm$ 0.108
15.00 |  47.293 $\pm$ 2.142 |  38.990 $\pm$ 2.908 |  6.216 $\pm$ 0.170 |  2.359 $\pm$ 0.108
16.00 |  45.203 $\pm$ 2.142 |  38.013 $\pm$ 2.908 |  5.821 $\pm$ 0.170 |  2.352 $\pm$ 0.108
17.00 |  42.664 $\pm$ 2.142 |  36.799 $\pm$ 2.908 |  4.632 $\pm$ 0.170 |  2.348 $\pm$ 0.108
18.00 |  38.406 $\pm$ 2.142 |  34.540 $\pm$ 2.908 |  3.971 $\pm$ 0.170 |  2.345 $\pm$ 0.108
19.00 |  32.666 $\pm$ 2.142 |  30.737 $\pm$ 2.908 |  3.484 $\pm$ 0.170 |  2.343 $\pm$ 0.108
20.00 |  30.511 $\pm$ 2.142 |  29.880 $\pm$ 2.908 |  2.088 $\pm$ 0.170 |  2.343 $\pm$ 0.108
Trials | 40 | 40 | 60 | 40

Time | LLLR | Logistic | LSEL
---|---|---|---
1.00 |  63.930 $\pm$ 0.026 |  |  63.956 $\pm$ 0.020
2.00 |  44.250 $\pm$ 0.026 |  43.873 $\pm$ 0.097 |  43.918 $\pm$ 0.020
3.00 |  30.693 $\pm$ 0.026 |  29.058 $\pm$ 0.097 |  28.957 $\pm$ 0.020
4.00 |  21.370 $\pm$ 0.026 |  18.973 $\pm$ 0.097 |  18.865 $\pm$ 0.020
5.00 |  14.834 $\pm$ 0.026 |  11.992 $\pm$ 0.097 |  11.910 $\pm$ 0.020
6.00 |  10.386 $\pm$ 0.026 |  7.522 $\pm$ 0.097 |  7.444 $\pm$ 0.020
7.00 |  7.516 $\pm$ 0.026 |  4.970 $\pm$ 0.097 |  4.877 $\pm$ 0.020
8.00 |  5.693 $\pm$ 0.026 |  3.485 $\pm$ 0.097 |  3.403 $\pm$ 0.020
9.00 |  4.563 $\pm$ 0.026 |  2.702 $\pm$ 0.097 |  2.647 $\pm$ 0.020
10.00 |  3.818 $\pm$ 0.026 |  2.326 $\pm$ 0.097 |  2.288 $\pm$ 0.020
11.00 |  3.316 $\pm$ 0.026 |  2.133 $\pm$ 0.097 |  2.103 $\pm$ 0.020
12.00 |  2.961 $\pm$ 0.026 |  2.030 $\pm$ 0.097 |  2.010 $\pm$ 0.020
13.00 |  2.705 $\pm$ 0.026 |  1.985 $\pm$ 0.097 |  1.977 $\pm$ 0.020
14.00 |  2.516 $\pm$ 0.026 |  1.964 $\pm$ 0.097 |  1.956 $\pm$ 0.020
15.00 |  2.365 $\pm$ 0.026 |  1.951 $\pm$ 0.097 |  1.941 $\pm$ 0.020
16.00 |  2.254 $\pm$ 0.026 |  1.943 $\pm$ 0.097 |  1.934 $\pm$ 0.020
17.00 |  2.163 $\pm$ 0.026 |  1.938 $\pm$ 0.097 |  1.932 $\pm$ 0.020
18.00 |  2.094 $\pm$ 0.026 |  1.937 $\pm$ 0.097 |  1.932 $\pm$ 0.020
19.00 |  2.036 $\pm$ 0.026 |  1.937 $\pm$ 0.097 |  1.932 $\pm$ 0.020
20.00 |  1.968 $\pm$ 0.026 |  1.937 $\pm$ 0.097 |  1.932 $\pm$ 0.020
Trials | 80 | 80 | 80

Table 35:  Averaged Per-Class Error Rates (%) and SEM of Figure 2 (Right: NMNIST-100f). Frames 1–50. Blanks mean N/A.  Time | Logistic (with M-TANDEM) | LSEL (with M-TANDEM)
---|---|---
1.00 |  56.273 $\pm$ 0.045 |  56.196 $\pm$ 0.044
2.00 |  37.772 $\pm$ 0.045 |  37.492 $\pm$ 0.044
3.00 |  27.922 $\pm$ 0.045 |  27.520 $\pm$ 0.044
4.00 |  22.282 $\pm$ 0.045 |  21.889 $\pm$ 0.044
5.00 |  18.725 $\pm$ 0.045 |  18.369 $\pm$ 0.044
6.00 |  16.251 $\pm$ 0.045 |  15.901 $\pm$ 0.044
7.00 |  14.383 $\pm$ 0.045 |  14.035 $\pm$ 0.044
8.00 |  12.898 $\pm$ 0.045 |  12.590 $\pm$ 0.044
9.00 |  11.730 $\pm$ 0.045 |  11.421 $\pm$ 0.044
10.00 |  10.758 $\pm$ 0.045 |  10.468 $\pm$ 0.044
11.00 |  9.943 $\pm$ 0.045 |  9.659 $\pm$ 0.044
12.00 |  9.245 $\pm$ 0.045 |  8.992 $\pm$ 0.044
13.00 |  8.663 $\pm$ 0.045 |  8.410 $\pm$ 0.044
14.00 |  8.158 $\pm$ 0.045 |  7.907 $\pm$ 0.044
15.00 |  7.721 $\pm$ 0.045 |  7.472 $\pm$ 0.044
16.00 |  7.341 $\pm$ 0.045 |  7.096 $\pm$ 0.044
17.00 |  7.006 $\pm$ 0.045 |  6.776 $\pm$ 0.044
18.00 |  6.717 $\pm$ 0.045 |  6.498 $\pm$ 0.044
19.00 |  6.476 $\pm$ 0.045 |  6.257 $\pm$ 0.044
20.00 |  6.258 $\pm$ 0.045 |  6.041 $\pm$ 0.044
21.00 |  6.069 $\pm$ 0.045 |  5.851 $\pm$ 0.044
22.00 |  5.900 $\pm$ 0.045 |  5.686 $\pm$ 0.044
23.00 |  5.748 $\pm$ 0.045 |  5.545 $\pm$ 0.044
24.00 |  5.615 $\pm$ 0.045 |  5.423 $\pm$ 0.044
25.00 |  5.502 $\pm$ 0.045 |  5.317 $\pm$ 0.044
26.00 |  5.406 $\pm$ 0.045 |  5.221 $\pm$ 0.044
27.00 |  5.318 $\pm$ 0.045 |  5.136 $\pm$ 0.044
28.00 |  5.246 $\pm$ 0.045 |  5.064 $\pm$ 0.044
29.00 |  5.175 $\pm$ 0.045 |  5.001 $\pm$ 0.044
30.00 |  5.114 $\pm$ 0.045 |  4.947 $\pm$ 0.044
31.00 |  5.057 $\pm$ 0.045 |  4.898 $\pm$ 0.044
32.00 |  5.011 $\pm$ 0.045 |  4.854 $\pm$ 0.044
33.00 |  4.974 $\pm$ 0.045 |  4.816 $\pm$ 0.044
34.00 |  4.939 $\pm$ 0.045 |  4.784 $\pm$ 0.044
35.00 |  4.908 $\pm$ 0.045 |  4.757 $\pm$ 0.044
36.00 |  4.882 $\pm$ 0.045 |  4.733 $\pm$ 0.044
37.00 |  4.858 $\pm$ 0.045 |  4.713 $\pm$ 0.044
38.00 |  4.837 $\pm$ 0.045 |  4.694 $\pm$ 0.044
39.00 |  4.818 $\pm$ 0.045 |  4.676 $\pm$ 0.044
40.00 |  4.801 $\pm$ 0.045 |  4.661 $\pm$ 0.044
41.00 |  4.786 $\pm$ 0.045 |  4.649 $\pm$ 0.044
42.00 |  4.773 $\pm$ 0.045 |  4.639 $\pm$ 0.044
43.00 |  4.761 $\pm$ 0.045 |  4.630 $\pm$ 0.044
44.00 |  4.751 $\pm$ 0.045 |  4.623 $\pm$ 0.044
45.00 |  4.742 $\pm$ 0.045 |  4.615 $\pm$ 0.044
46.00 |  4.735 $\pm$ 0.045 |  4.608 $\pm$ 0.044
47.00 |  4.727 $\pm$ 0.045 |  4.604 $\pm$ 0.044
48.00 |  4.722 $\pm$ 0.045 |  4.598 $\pm$ 0.044
49.00 |  4.716 $\pm$ 0.045 |  4.594 $\pm$ 0.044
50.00 |  4.711 $\pm$ 0.045 |  4.590 $\pm$ 0.044
Trials | 150 | 150
Table 36:  Averaged Per-Class Error Rates (%) and SEM of Figure 2 (Right: NMNIST-100f). Frames 51–100. Blanks mean N/A.  Time | Logistic (with M-TANDEM) | LSEL (with M-TANDEM)
---|---|---
51.00 |  4.706 $\pm$ 0.045 |  4.587 $\pm$ 0.044
52.00 |  4.703 $\pm$ 0.045 |  4.586 $\pm$ 0.044
53.00 |  4.700 $\pm$ 0.045 |  4.584 $\pm$ 0.044
54.00 |  4.697 $\pm$ 0.045 |  4.580 $\pm$ 0.044
55.00 |  4.694 $\pm$ 0.045 |  4.577 $\pm$ 0.044
56.00 |  4.692 $\pm$ 0.045 |  4.576 $\pm$ 0.044
57.00 |  4.690 $\pm$ 0.045 |  4.575 $\pm$ 0.044
58.00 |  4.688 $\pm$ 0.045 |  4.573 $\pm$ 0.044
59.00 |  4.687 $\pm$ 0.045 |  4.572 $\pm$ 0.044
60.00 |  4.686 $\pm$ 0.045 |  4.572 $\pm$ 0.044
61.00 |  4.685 $\pm$ 0.045 |  4.571 $\pm$ 0.044
62.00 |  4.684 $\pm$ 0.045 |  4.570 $\pm$ 0.044
63.00 |  4.683 $\pm$ 0.045 |  4.569 $\pm$ 0.044
64.00 |  4.683 $\pm$ 0.045 |  4.569 $\pm$ 0.044
65.00 |  4.682 $\pm$ 0.045 |  4.569 $\pm$ 0.044
66.00 |  4.681 $\pm$ 0.045 |  4.568 $\pm$ 0.044
67.00 |  4.681 $\pm$ 0.045 |  4.568 $\pm$ 0.044
68.00 |  4.680 $\pm$ 0.045 |  4.567 $\pm$ 0.044
69.00 |  4.679 $\pm$ 0.045 |  4.567 $\pm$ 0.044
70.00 |  4.678 $\pm$ 0.045 |  4.567 $\pm$ 0.044
71.00 |  4.678 $\pm$ 0.045 |  4.566 $\pm$ 0.044
72.00 |  4.678 $\pm$ 0.045 |  4.566 $\pm$ 0.044
73.00 |  4.678 $\pm$ 0.045 |  4.565 $\pm$ 0.044
74.00 |  4.678 $\pm$ 0.045 |  4.565 $\pm$ 0.044
75.00 |  4.678 $\pm$ 0.045 |  4.565 $\pm$ 0.044
76.00 |  4.678 $\pm$ 0.045 |  4.565 $\pm$ 0.044
77.00 |  4.678 $\pm$ 0.045 |  4.565 $\pm$ 0.044
78.00 |  4.677 $\pm$ 0.045 |  4.565 $\pm$ 0.044
79.00 |  4.677 $\pm$ 0.045 |  4.564 $\pm$ 0.044
80.00 |  4.676 $\pm$ 0.045 |  4.565 $\pm$ 0.044
81.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
82.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
83.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
84.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
85.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
86.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
87.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
88.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
89.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
90.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
91.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
92.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
93.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
94.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
95.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
96.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
97.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
98.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
99.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
100.00 |  4.676 $\pm$ 0.045 |  4.564 $\pm$ 0.044
Trials | 150 | 150
Table 37:  Averaged Per-Class Error Rates (%) and SEM of Figure 5 (NMNIST-H). Blanks mean N/A.  Time | MSPRT-TANDEM | LSTM-s | LSTM-m |  EARLIEST $10^{-2}$ |  EARLIEST $10^{2}$
---|---|---|---|---|---
1.00 |  |  64.003 $\pm$ 0.010 |  64.009 $\pm$ 0.011 |  |
1.34 |  |  |  |  |  57.973 $\pm$ 0.066
2.00 |  44.172 $\pm$ 0.046 |  46.676 $\pm$ 0.008 |  46.697 $\pm$ 0.008 |  |
3.00 |  29.190 $\pm$ 0.046 |  35.517 $\pm$ 0.007 |  35.536 $\pm$ 0.008 |  |
4.00 |  18.964 $\pm$ 0.046 |  25.936 $\pm$ 0.007 |  25.940 $\pm$ 0.007 |  |
5.00 |  11.994 $\pm$ 0.046 |  20.510 $\pm$ 0.007 |  20.512 $\pm$ 0.006 |  |
6.00 |  7.449 $\pm$ 0.046 |  15.393 $\pm$ 0.007 |  15.404 $\pm$ 0.006 |  |
7.00 |  4.870 $\pm$ 0.046 |  12.603 $\pm$ 0.005 |  12.614 $\pm$ 0.005 |  |
8.00 |  3.383 $\pm$ 0.046 |  10.025 $\pm$ 0.005 |  10.018 $\pm$ 0.005 |  |
9.00 |  2.639 $\pm$ 0.046 |  8.036 $\pm$ 0.005 |  8.033 $\pm$ 0.005 |  |
10.00 |  2.281 $\pm$ 0.046 |  6.963 $\pm$ 0.005 |  6.958 $\pm$ 0.005 |  |
11.00 |  2.087 $\pm$ 0.046 |  5.788 $\pm$ 0.005 |  5.801 $\pm$ 0.005 |  |
12.00 |  1.996 $\pm$ 0.046 |  4.886 $\pm$ 0.003 |  4.892 $\pm$ 0.004 |  |
13.00 |  1.963 $\pm$ 0.046 |  4.398 $\pm$ 0.003 |  4.400 $\pm$ 0.003 |  |
13.41 |  |  |  |  3.162 $\pm$ 0.027 |
14.00 |  1.942 $\pm$ 0.046 |  3.735 $\pm$ 0.003 |  3.737 $\pm$ 0.003 |  |
15.00 |  1.926 $\pm$ 0.046 |  3.190 $\pm$ 0.003 |  3.198 $\pm$ 0.003 |  |
16.00 |  1.919 $\pm$ 0.046 |  2.841 $\pm$ 0.003 |  2.850 $\pm$ 0.003 |  |
17.00 |  1.917 $\pm$ 0.046 |  2.576 $\pm$ 0.003 |  2.576 $\pm$ 0.003 |  |
18.00 |  1.916 $\pm$ 0.046 |  2.376 $\pm$ 0.003 |  2.374 $\pm$ 0.003 |  |
19.00 |  1.916 $\pm$ 0.046 |  2.118 $\pm$ 0.003 |  2.119 $\pm$ 0.003 |  |
20.00 |  1.916 $\pm$ 0.046 |  1.923 $\pm$ 0.003 |  1.921 $\pm$ 0.003 |  |
Trials | 300 | 300 | 300 | 50 | 45
Table 38:  Averaged Per-Class Error Rates (%) and SEM of Figure 5 (NMNIST-100f). Frames 1–50. Blanks mean N/A.  Time | MSPRT-TANDEM | LSTM-s | LSTM-m | EARLIEST Lam1e-2 | EARLIEST Lam1e-4
---|---|---|---|---|---
1.00 |  |  54.677 $\pm$ 0.016 |  54.696 $\pm$ 0.016 |  |
2.00 |  35.942 $\pm$ 0.148 |  38.748 $\pm$ 0.014 |  38.677 $\pm$ 0.015 |  |
3.00 |  25.203 $\pm$ 0.148 |  29.348 $\pm$ 0.014 |  29.350 $\pm$ 0.014 |  |
4.00 |  19.167 $\pm$ 0.148 |  24.048 $\pm$ 0.014 |  23.981 $\pm$ 0.015 |  |
5.00 |  15.450 $\pm$ 0.148 |  20.490 $\pm$ 0.014 |  20.422 $\pm$ 0.014 |  |
6.00 |  12.997 $\pm$ 0.148 |  18.065 $\pm$ 0.014 |  17.959 $\pm$ 0.015 |  |
7.00 |  11.224 $\pm$ 0.148 |  15.715 $\pm$ 0.014 |  15.606 $\pm$ 0.015 |  |
8.00 |  9.912 $\pm$ 0.148 |  14.306 $\pm$ 0.014 |  14.215 $\pm$ 0.014 |  |
9.00 |  8.880 $\pm$ 0.148 |  13.124 $\pm$ 0.014 |  13.046 $\pm$ 0.013 |  |
10.00 |  8.068 $\pm$ 0.148 |  12.129 $\pm$ 0.012 |  12.038 $\pm$ 0.012 |  |
11.00 |  7.405 $\pm$ 0.148 |  11.386 $\pm$ 0.013 |  11.302 $\pm$ 0.013 |  |
12.00 |  6.862 $\pm$ 0.148 |  10.838 $\pm$ 0.012 |  10.767 $\pm$ 0.013 |  |
13.00 |  6.409 $\pm$ 0.148 |  10.121 $\pm$ 0.012 |  10.034 $\pm$ 0.012 |  |
14.00 |  6.032 $\pm$ 0.148 |  9.626 $\pm$ 0.013 |  9.555 $\pm$ 0.013 |  |
15.00 |  5.715 $\pm$ 0.148 |  9.102 $\pm$ 0.012 |  9.007 $\pm$ 0.012 |  |
16.00 |  5.444 $\pm$ 0.148 |  8.710 $\pm$ 0.013 |  8.625 $\pm$ 0.013 |  |
17.00 |  5.212 $\pm$ 0.148 |  8.268 $\pm$ 0.011 |  8.235 $\pm$ 0.012 |  |
18.00 |  5.025 $\pm$ 0.148 |  7.903 $\pm$ 0.012 |  7.852 $\pm$ 0.012 |  |
19.00 |  4.865 $\pm$ 0.148 |  7.613 $\pm$ 0.012 |  7.586 $\pm$ 0.011 |  |
19.46 |  |  |  |  20.044 $\pm$ 0.229 |
20.00 |  4.730 $\pm$ 0.148 |  7.366 $\pm$ 0.012 |  7.354 $\pm$ 0.011 |  |
21.00 |  4.623 $\pm$ 0.148 |  7.150 $\pm$ 0.012 |  7.150 $\pm$ 0.011 |  |
22.00 |  4.536 $\pm$ 0.148 |  6.998 $\pm$ 0.011 |  7.006 $\pm$ 0.011 |  |
23.00 |  4.464 $\pm$ 0.148 |  6.816 $\pm$ 0.011 |  6.821 $\pm$ 0.011 |  |
24.00 |  4.401 $\pm$ 0.148 |  6.674 $\pm$ 0.011 |  6.655 $\pm$ 0.011 |  |
25.00 |  4.353 $\pm$ 0.148 |  6.491 $\pm$ 0.012 |  6.502 $\pm$ 0.013 |  |
26.00 |  4.310 $\pm$ 0.148 |  6.351 $\pm$ 0.011 |  6.373 $\pm$ 0.012 |  |
27.00 |  4.275 $\pm$ 0.148 |  6.200 $\pm$ 0.011 |  6.200 $\pm$ 0.013 |  |
28.00 |  4.246 $\pm$ 0.148 |  6.061 $\pm$ 0.011 |  6.077 $\pm$ 0.013 |  |
29.00 |  4.222 $\pm$ 0.148 |  5.935 $\pm$ 0.011 |  5.948 $\pm$ 0.012 |  |
30.00 |  4.203 $\pm$ 0.148 |  5.876 $\pm$ 0.012 |  5.870 $\pm$ 0.011 |  |
31.00 |  4.186 $\pm$ 0.148 |  5.811 $\pm$ 0.011 |  5.823 $\pm$ 0.012 |  |
32.00 |  4.171 $\pm$ 0.148 |  5.658 $\pm$ 0.011 |  5.676 $\pm$ 0.012 |  |
33.00 |  4.160 $\pm$ 0.148 |  5.552 $\pm$ 0.011 |  5.566 $\pm$ 0.012 |  |
34.00 |  4.147 $\pm$ 0.148 |  5.456 $\pm$ 0.012 |  5.474 $\pm$ 0.012 |  |
35.00 |  4.139 $\pm$ 0.148 |  5.395 $\pm$ 0.011 |  5.412 $\pm$ 0.012 |  |
36.00 |  4.133 $\pm$ 0.148 |  5.307 $\pm$ 0.011 |  5.319 $\pm$ 0.012 |  |
37.00 |  4.126 $\pm$ 0.148 |  5.231 $\pm$ 0.011 |  5.249 $\pm$ 0.012 |  |
38.00 |  4.121 $\pm$ 0.148 |  5.183 $\pm$ 0.011 |  5.211 $\pm$ 0.012 |  |
39.00 |  4.118 $\pm$ 0.148 |  5.148 $\pm$ 0.011 |  5.182 $\pm$ 0.011 |  |
40.00 |  4.116 $\pm$ 0.148 |  5.121 $\pm$ 0.011 |  5.168 $\pm$ 0.012 |  |
41.00 |  4.113 $\pm$ 0.148 |  5.055 $\pm$ 0.011 |  5.098 $\pm$ 0.012 |  |
42.00 |  4.112 $\pm$ 0.148 |  5.057 $\pm$ 0.012 |  5.091 $\pm$ 0.013 |  |
43.00 |  4.110 $\pm$ 0.148 |  4.980 $\pm$ 0.012 |  5.017 $\pm$ 0.012 |  |
44.00 |  4.109 $\pm$ 0.148 |  4.940 $\pm$ 0.011 |  4.976 $\pm$ 0.013 |  |
45.00 |  4.109 $\pm$ 0.148 |  4.892 $\pm$ 0.012 |  4.921 $\pm$ 0.013 |  |
46.00 |  4.108 $\pm$ 0.148 |  4.825 $\pm$ 0.012 |  4.865 $\pm$ 0.013 |  |
47.00 |  4.108 $\pm$ 0.148 |  4.780 $\pm$ 0.012 |  4.795 $\pm$ 0.013 |  |
48.00 |  4.108 $\pm$ 0.148 |  4.730 $\pm$ 0.012 |  4.767 $\pm$ 0.013 |  |
49.00 |  4.107 $\pm$ 0.148 |  4.709 $\pm$ 0.012 |  4.742 $\pm$ 0.013 |  |
50.00 |  4.107 $\pm$ 0.148 |  4.690 $\pm$ 0.012 |  4.731 $\pm$ 0.013 |  |
Trials | 150 | 150 | 150 | 50 | 60
Table 39:  Averaged Per-Class Error Rates (%) and SEM of Figure 5 (NMNIST-100f). Frames 51–100. Blanks mean N/A.  Time | MSPRT-TANDEM | LSTM-s | LSTM-m |  EARLIEST $10^{-2}$ |  EARLIEST $10^{-4}$
---|---|---|---|---|---
51.00 |  4.107 $\pm$ 0.148 |  4.620 $\pm$ 0.012 |  4.661 $\pm$ 0.014 |  |
52.00 |  4.107 $\pm$ 0.148 |  4.588 $\pm$ 0.012 |  4.637 $\pm$ 0.014 |  |
53.00 |  4.107 $\pm$ 0.148 |  4.595 $\pm$ 0.011 |  4.634 $\pm$ 0.013 |  |
54.00 |  4.106 $\pm$ 0.148 |  4.538 $\pm$ 0.011 |  4.580 $\pm$ 0.013 |  |
55.00 |  4.107 $\pm$ 0.148 |  4.526 $\pm$ 0.010 |  4.566 $\pm$ 0.013 |  |
56.00 |  4.107 $\pm$ 0.148 |  4.531 $\pm$ 0.012 |  4.555 $\pm$ 0.013 |  |
57.00 |  4.107 $\pm$ 0.148 |  4.470 $\pm$ 0.011 |  4.509 $\pm$ 0.012 |  |
58.00 |  4.107 $\pm$ 0.148 |  4.444 $\pm$ 0.011 |  4.489 $\pm$ 0.012 |  |
59.00 |  4.108 $\pm$ 0.148 |  4.449 $\pm$ 0.011 |  4.496 $\pm$ 0.012 |  |
60.00 |  4.108 $\pm$ 0.148 |  4.447 $\pm$ 0.011 |  4.494 $\pm$ 0.013 |  |
61.00 |  4.108 $\pm$ 0.148 |  4.459 $\pm$ 0.011 |  4.484 $\pm$ 0.012 |  |
62.00 |  4.108 $\pm$ 0.148 |  4.408 $\pm$ 0.011 |  4.447 $\pm$ 0.012 |  |
63.00 |  4.108 $\pm$ 0.148 |  4.391 $\pm$ 0.011 |  4.429 $\pm$ 0.013 |  |
64.00 |  4.108 $\pm$ 0.148 |  4.377 $\pm$ 0.011 |  4.412 $\pm$ 0.013 |  |
65.00 |  4.108 $\pm$ 0.148 |  4.370 $\pm$ 0.011 |  4.397 $\pm$ 0.013 |  |
66.00 |  4.109 $\pm$ 0.148 |  4.378 $\pm$ 0.011 |  4.405 $\pm$ 0.013 |  |
67.00 |  4.109 $\pm$ 0.148 |  4.394 $\pm$ 0.011 |  4.422 $\pm$ 0.012 |  |
68.00 |  4.109 $\pm$ 0.148 |  4.360 $\pm$ 0.011 |  4.386 $\pm$ 0.012 |  |
69.00 |  4.109 $\pm$ 0.148 |  4.307 $\pm$ 0.012 |  4.324 $\pm$ 0.012 |  |
70.00 |  4.109 $\pm$ 0.148 |  4.277 $\pm$ 0.011 |  4.315 $\pm$ 0.012 |  |
71.00 |  4.109 $\pm$ 0.148 |  4.268 $\pm$ 0.012 |  4.312 $\pm$ 0.012 |  |
72.00 |  4.109 $\pm$ 0.148 |  4.252 $\pm$ 0.012 |  4.286 $\pm$ 0.011 |  |
73.00 |  4.109 $\pm$ 0.148 |  4.247 $\pm$ 0.011 |  4.278 $\pm$ 0.012 |  |
74.00 |  4.109 $\pm$ 0.148 |  4.203 $\pm$ 0.011 |  4.221 $\pm$ 0.012 |  |
75.00 |  4.109 $\pm$ 0.148 |  4.207 $\pm$ 0.011 |  4.234 $\pm$ 0.012 |  |
76.00 |  4.109 $\pm$ 0.148 |  4.203 $\pm$ 0.011 |  4.217 $\pm$ 0.011 |  |
77.00 |  4.109 $\pm$ 0.148 |  4.171 $\pm$ 0.011 |  4.191 $\pm$ 0.012 |  |
78.00 |  4.109 $\pm$ 0.148 |  4.154 $\pm$ 0.011 |  4.182 $\pm$ 0.012 |  |
79.00 |  4.109 $\pm$ 0.148 |  4.155 $\pm$ 0.010 |  4.179 $\pm$ 0.012 |  |
80.00 |  4.109 $\pm$ 0.148 |  4.129 $\pm$ 0.010 |  4.144 $\pm$ 0.012 |  |
81.00 |  4.109 $\pm$ 0.148 |  4.134 $\pm$ 0.011 |  4.138 $\pm$ 0.012 |  |
82.00 |  4.109 $\pm$ 0.148 |  4.113 $\pm$ 0.011 |  4.126 $\pm$ 0.013 |  |
83.00 |  4.109 $\pm$ 0.148 |  4.117 $\pm$ 0.011 |  4.135 $\pm$ 0.013 |  |
84.00 |  4.109 $\pm$ 0.148 |  4.093 $\pm$ 0.011 |  4.105 $\pm$ 0.012 |  |
85.00 |  4.109 $\pm$ 0.148 |  4.096 $\pm$ 0.012 |  4.113 $\pm$ 0.012 |  |
86.00 |  4.109 $\pm$ 0.148 |  4.106 $\pm$ 0.012 |  4.112 $\pm$ 0.012 |  |
87.00 |  4.109 $\pm$ 0.148 |  4.099 $\pm$ 0.011 |  4.110 $\pm$ 0.013 |  |
88.00 |  4.109 $\pm$ 0.148 |  4.093 $\pm$ 0.011 |  4.101 $\pm$ 0.012 |  |
89.00 |  4.109 $\pm$ 0.148 |  4.072 $\pm$ 0.011 |  4.094 $\pm$ 0.013 |  |
90.00 |  4.109 $\pm$ 0.148 |  4.077 $\pm$ 0.011 |  4.095 $\pm$ 0.013 |  |
91.00 |  4.109 $\pm$ 0.148 |  4.063 $\pm$ 0.011 |  4.086 $\pm$ 0.013 |  |
92.00 |  4.109 $\pm$ 0.148 |  4.069 $\pm$ 0.011 |  4.083 $\pm$ 0.013 |  |
93.00 |  4.109 $\pm$ 0.148 |  4.073 $\pm$ 0.011 |  4.096 $\pm$ 0.013 |  |
94.00 |  4.109 $\pm$ 0.148 |  4.069 $\pm$ 0.011 |  4.105 $\pm$ 0.013 |  |
95.00 |  4.109 $\pm$ 0.148 |  4.062 $\pm$ 0.011 |  4.101 $\pm$ 0.013 |  |
96.00 |  4.109 $\pm$ 0.148 |  4.076 $\pm$ 0.012 |  4.109 $\pm$ 0.013 |  |
97.00 |  4.109 $\pm$ 0.148 |  4.056 $\pm$ 0.011 |  4.098 $\pm$ 0.012 |  |
98.00 |  4.109 $\pm$ 0.148 |  4.063 $\pm$ 0.011 |  4.104 $\pm$ 0.012 |  |
99.00 |  4.109 $\pm$ 0.148 |  4.066 $\pm$ 0.011 |  4.100 $\pm$ 0.012 |  |
99.99 |  |  |  |  |  4.586 $\pm$ 0.020
100.00 |  4.109 $\pm$ 0.148 |  4.057 $\pm$ 0.011 |  4.097 $\pm$ 0.012 |  |
Trials | 150 | 150 | 150 | 50 | 60
Table 40:  Averaged Per-Class Error Rates (%) and SEM of Figure 9. Frames 1–50. Blanks mean N/A.  Time | TANDEM LSEL | TANDEM Mult | TANDEM LSEL+Mult
---|---|---|---
1.00 |  56.196 $\pm$ 0.044 |  55.705 $\pm$ 0.036 |
2.00 |  37.742 $\pm$ 0.044 |  36.650 $\pm$ 0.036 |  35.942 $\pm$ 0.148
3.00 |  27.721 $\pm$ 0.044 |  26.072 $\pm$ 0.036 |  25.203 $\pm$ 0.148
4.00 |  22.037 $\pm$ 0.044 |  20.019 $\pm$ 0.036 |  19.167 $\pm$ 0.148
5.00 |  18.486 $\pm$ 0.044 |  16.320 $\pm$ 0.036 |  15.450 $\pm$ 0.148
6.00 |  16.015 $\pm$ 0.044 |  13.842 $\pm$ 0.036 |  12.997 $\pm$ 0.148
7.00 |  14.127 $\pm$ 0.044 |  12.028 $\pm$ 0.036 |  11.224 $\pm$ 0.148
8.00 |  12.664 $\pm$ 0.044 |  10.636 $\pm$ 0.036 |  9.912 $\pm$ 0.148
9.00 |  11.487 $\pm$ 0.044 |  9.564 $\pm$ 0.036 |  8.880 $\pm$ 0.148
10.00 |  10.526 $\pm$ 0.044 |  8.692 $\pm$ 0.036 |  8.068 $\pm$ 0.148
11.00 |  9.711 $\pm$ 0.044 |  7.989 $\pm$ 0.036 |  7.405 $\pm$ 0.148
12.00 |  9.042 $\pm$ 0.044 |  7.393 $\pm$ 0.036 |  6.862 $\pm$ 0.148
13.00 |  8.454 $\pm$ 0.044 |  6.897 $\pm$ 0.036 |  6.409 $\pm$ 0.148
14.00 |  7.945 $\pm$ 0.044 |  6.477 $\pm$ 0.036 |  6.032 $\pm$ 0.148
15.00 |  7.510 $\pm$ 0.044 |  6.119 $\pm$ 0.036 |  5.715 $\pm$ 0.148
16.00 |  7.124 $\pm$ 0.044 |  5.830 $\pm$ 0.036 |  5.444 $\pm$ 0.148
17.00 |  6.803 $\pm$ 0.044 |  5.579 $\pm$ 0.036 |  5.212 $\pm$ 0.148
18.00 |  6.519 $\pm$ 0.044 |  5.370 $\pm$ 0.036 |  5.025 $\pm$ 0.148
19.00 |  6.276 $\pm$ 0.044 |  5.188 $\pm$ 0.036 |  4.865 $\pm$ 0.148
20.00 |  6.060 $\pm$ 0.044 |  5.032 $\pm$ 0.036 |  4.730 $\pm$ 0.148
21.00 |  5.869 $\pm$ 0.044 |  4.899 $\pm$ 0.036 |  4.623 $\pm$ 0.148
22.00 |  5.700 $\pm$ 0.044 |  4.785 $\pm$ 0.036 |  4.536 $\pm$ 0.148
23.00 |  5.556 $\pm$ 0.044 |  4.687 $\pm$ 0.036 |  4.464 $\pm$ 0.148
24.00 |  5.434 $\pm$ 0.044 |  4.604 $\pm$ 0.036 |  4.401 $\pm$ 0.148
25.00 |  5.327 $\pm$ 0.044 |  4.537 $\pm$ 0.036 |  4.353 $\pm$ 0.148
26.00 |  5.229 $\pm$ 0.044 |  4.477 $\pm$ 0.036 |  4.310 $\pm$ 0.148
27.00 |  5.143 $\pm$ 0.044 |  4.426 $\pm$ 0.036 |  4.275 $\pm$ 0.148
28.00 |  5.070 $\pm$ 0.044 |  4.383 $\pm$ 0.036 |  4.246 $\pm$ 0.148
29.00 |  5.006 $\pm$ 0.044 |  4.353 $\pm$ 0.036 |  4.222 $\pm$ 0.148
30.00 |  4.952 $\pm$ 0.044 |  4.325 $\pm$ 0.036 |  4.203 $\pm$ 0.148
31.00 |  4.903 $\pm$ 0.044 |  4.299 $\pm$ 0.036 |  4.186 $\pm$ 0.148
32.00 |  4.858 $\pm$ 0.044 |  4.279 $\pm$ 0.036 |  4.171 $\pm$ 0.148
33.00 |  4.820 $\pm$ 0.044 |  4.261 $\pm$ 0.036 |  4.160 $\pm$ 0.148
34.00 |  4.787 $\pm$ 0.044 |  4.246 $\pm$ 0.036 |  4.147 $\pm$ 0.148
35.00 |  4.760 $\pm$ 0.044 |  4.234 $\pm$ 0.036 |  4.139 $\pm$ 0.148
36.00 |  4.736 $\pm$ 0.044 |  4.222 $\pm$ 0.036 |  4.133 $\pm$ 0.148
37.00 |  4.715 $\pm$ 0.044 |  4.211 $\pm$ 0.036 |  4.126 $\pm$ 0.148
38.00 |  4.697 $\pm$ 0.044 |  4.204 $\pm$ 0.036 |  4.121 $\pm$ 0.148
39.00 |  4.679 $\pm$ 0.044 |  4.197 $\pm$ 0.036 |  4.118 $\pm$ 0.148
40.00 |  4.662 $\pm$ 0.044 |  4.193 $\pm$ 0.036 |  4.116 $\pm$ 0.148
41.00 |  4.651 $\pm$ 0.044 |  4.188 $\pm$ 0.036 |  4.113 $\pm$ 0.148
42.00 |  4.640 $\pm$ 0.044 |  4.183 $\pm$ 0.036 |  4.112 $\pm$ 0.148
43.00 |  4.631 $\pm$ 0.044 |  4.180 $\pm$ 0.036 |  4.110 $\pm$ 0.148
44.00 |  4.624 $\pm$ 0.044 |  4.176 $\pm$ 0.036 |  4.109 $\pm$ 0.148
45.00 |  4.615 $\pm$ 0.044 |  4.174 $\pm$ 0.036 |  4.109 $\pm$ 0.148
46.00 |  4.609 $\pm$ 0.044 |  4.172 $\pm$ 0.036 |  4.108 $\pm$ 0.148
47.00 |  4.604 $\pm$ 0.044 |  4.170 $\pm$ 0.036 |  4.108 $\pm$ 0.148
48.00 |  4.599 $\pm$ 0.044 |  4.168 $\pm$ 0.036 |  4.108 $\pm$ 0.148
49.00 |  4.594 $\pm$ 0.044 |  4.167 $\pm$ 0.036 |  4.107 $\pm$ 0.148
50.00 |  4.591 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.107 $\pm$ 0.148
Trials | 150 | 150 | 150
Table 41:  Averaged Per-Class Error Rates (%) and SEM of Figure 9. Frames 51–100. Blanks mean N/A.  Time | TANDEM LSEL | TANDEM Mult | TANDEM LSEL+Mult
---|---|---|---
51.00 |  4.588 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.107 $\pm$ 0.148
52.00 |  4.586 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.107 $\pm$ 0.148
53.00 |  4.584 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.107 $\pm$ 0.148
54.00 |  4.580 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.106 $\pm$ 0.148
55.00 |  4.577 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.107 $\pm$ 0.148
56.00 |  4.576 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.107 $\pm$ 0.148
57.00 |  4.575 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.107 $\pm$ 0.148
58.00 |  4.573 $\pm$ 0.044 |  4.162 $\pm$ 0.036 |  4.107 $\pm$ 0.148
59.00 |  4.572 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
60.00 |  4.572 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
61.00 |  4.571 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
62.00 |  4.570 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.108 $\pm$ 0.148
63.00 |  4.570 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
64.00 |  4.569 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
65.00 |  4.569 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.108 $\pm$ 0.148
66.00 |  4.568 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
67.00 |  4.568 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
68.00 |  4.567 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
69.00 |  4.567 $\pm$ 0.044 |  4.163 $\pm$ 0.036 |  4.109 $\pm$ 0.148
70.00 |  4.567 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
71.00 |  4.566 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
72.00 |  4.566 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
73.00 |  4.565 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
74.00 |  4.565 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
75.00 |  4.565 $\pm$ 0.044 |  4.164 $\pm$ 0.036 |  4.109 $\pm$ 0.148
76.00 |  4.565 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
77.00 |  4.565 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
78.00 |  4.565 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
79.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
80.00 |  4.565 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
81.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
82.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
83.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
84.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
85.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
86.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
87.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
88.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
89.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
90.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
91.00 |  4.564 $\pm$ 0.044 |  4.165 $\pm$ 0.036 |  4.109 $\pm$ 0.148
92.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
93.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
94.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
95.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
96.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
97.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
98.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
99.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
100.00 |  4.564 $\pm$ 0.044 |  4.166 $\pm$ 0.036 |  4.109 $\pm$ 0.148
Trials | 150 | 150 | 150

##  Appendix J LLR Trajectories

Figures 10 and 11 show the LLR trajectories of NMNIST-H, NMNIST-100f, UCF101,
and HMDB51. The base models are the same as those in Figure 5.

Figure 10: Examples of LLR trajectories. Top: NMNIST-H. Bottom: NMNIST-100f.
The red curves represent $\min_{l}\\{\hat{\lambda}_{y_{i}l}\\}$, while the
blue curves represent $\min_{l}\\{\hat{\lambda}_{kl}\\}$ ($k\neq y_{i}$). If
the red curve reaches the threshold (necessarily positive), then the
prediction is correct, while if the blue curve reaches the threshold, then the
prediction is wrong. We plot ten different $i$’s randomly selected from the
validation set. The red and blue curves are gradually separated as more frames
are observed: evidence accumulation.

Figure 11: Examples of LLR trajectories. Top: UCF101. Bottom: HMDB51. All
curves are $\min_{l}\\{\hat{\lambda}_{y_{i}l}\\}$, and the negative rows
$\min_{l}\\{\hat{\lambda}_{kl}\\}$ ($k\neq y_{i}$) are omitted for clarity.
Therefore, all the curves should be in the upper half-plane (positive LLRs).
We plot 20 and 30 different $i$’s randomly selected from the validation sets
of UCF101 and HMDB51, respectively. Several curves gradually go upwards as
more frames are observed: evidence accumulation.

##  Appendix K NMNIST-H and NMNIST-100f

[13] propose an MNIST-based sequential dataset for early classification of
time series: NMNIST; however, NMNIST is so simple that accuracy tends to
saturate immediately and the performance comparison of models is difficult,
especially in the early stage of sequential predictions. We thus create a more
complex dataset, NMNIST-H, with higher noise density than NMNIST. Figure 12 is
an example video of NMNIST-H. It is hard, if not impossible, for humans to
classify the video within 10 frames.

NMNIST-100f is a more challenging dataset than NMNIST-H. Each video consists
of 100 frames, which is 5 times longer than in NMNIST and NMNSIT-H. Figure 13
is an example video of NMNIST-100f. Because of the dense noise, classification
is unrealistic for humans, while MSPRT-TANDEM attains approximately 90 %
accuracy with only 8 frames (see Figure 5).

Figure 12:  NMNIST-H. The first frame is at the top left, and the last frame
is at the bottom right. The original MNIST image (28$\times$28 pixels) is
gradually revealed (10 pixels per frame). The label of this example is 6. The
mean image is given in Figure 14 (left).

Figure 13:  NMNIST-100f. The first frame is at the top left, and the last
frame is at the bottom right. An MNIST image (28$\times$28 pixels) is filled
with white pixels except for 15 randomly selected pixels. Unlike NMNIST-H, the
number of original pixels (15) is fixed throughout all frames. The label of
this example is 3. The mean image is given in Figure 14 (right).

Figure 14: Mean images of Figures 12 (left) and 13 (right).

##  Appendix L Details of Statistical Tests

###  L.1 Model Comparison: Figure 5

For an objective comparison, we conduct statistical tests: two-way ANOVA [15]
followed by Tukey-Kramer multi-comparison test [65, 40]. In the tests, a small
number of trials reduces test statistics, making it difficult to claim
significance because the test statistic of the Tukey-Kramer test is
proportional to $1/\sqrt{(1/n+1/m)}$, where $n$ and $m$ are trial numbers of
two models to be compared. These statistical tests are standard, e.g., in
biological science, in which variable trial numbers are inevitable in
experiments. All the statistical tests are executed with a customized [49]
script.

In the two-way ANOVA, the two factors are defined as the phase (early and late
stages of the SAT curve) and model. The actual numbers of frames shown in
Table 42 are chosen so that the compared models can use as similar frames as
possible and thus depend only on the dataset. Note that EARLIEST cannot
flexibly change the mean hitting time; thus, we include the results of
EARLIEST to the groups with as close to the number of frames as possible.

The $p$-values are summarized in Tables 43 to 47. The $p$-values with
asterisks are statistically significant: one, two and three asterisks show
$p<0.05$, $p<0.01$, and $p<0.001$, respectively. Our results, especially in
the late phase, are statistically significantly better than those in the early
phase, confirming that accumulating evidence leads to better performance.

Table 42: Definition of the phases (in the number of frames). | Early phase | Late phase
---|---|---
| EARLIEST | All but EARLIEST | EARLIEST | All but EARLIEST
NMNIST-H | 1.34 | 2 | 13.41 | 13
NMNIST-100f | 13.41 | 19 | 99.99 | 100
UCF101 | 1.36 | 1 | 49.93 | 50
HMDB51 | 1.43 | 1 | 36.20 | 36
Table 43: $p$-values from the Tukey-Kramer multi-comparison test conducted on NMNIST-100f.(Figure 2) |  | Logistic | LSEL
---|---|---|---
|  | early | late | early
Logistic | late | ***1E-07 |  |
LSEL | early | ***1E-09 | ***1E-09 |
| late | ***1E-09 | *2E-3 | ***1E-09
Table 44: $p$-values from the Tukey-Kramer multi-comparison test conducted on NMNIST-H (Figure 5). |  | MSPRT-TANDEM | NP test | LSTM-s | LSTM-m | EARLIEST
---|---|---|---|---|---|---
|  | early | late | early | late | early | late | early | late | early
MSPRT-TANDEM | late | ***1E-07 |  |  |  |  |  |  |  |
NP test | early | ***1E-07 | ***1E-07 |  |  |  |  |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 |  |  |  |  |  |
LSTM-s | early | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 |  |  |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 |  |  |  |
LSTM-m | early | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |
EARLIEST | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07
Table 45: $p$-values from the Tukey-Kramer multi-comparison test conducted on NMNIST-100f (Figure 5). |  | MSPRT-TANDEM | NP test | LSTM-s | LSTM-m | EARLIEST
---|---|---|---|---|---|---
|  | early | late | early | late | early | late | early | late | early
MSPRT-TANDEM | late | ***1E-07 |  |  |  |  |  |  |  |
NP test | early | ***1E-07 | ***1E-07 |  |  |  |  |  |  |
| late | ***1E-07 | 1.0 | ***1E-07 |  |  |  |  |  |
LSTM-s | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 |  |  |  |  |
| late | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |  |  |
LSTM-m | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 |  |  |
| late | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |
EARLIEST | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 |
| late | ***7E-05 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07
Table 46: $p$-values from the Tukey-Kramer multi-comparison test conducted on UCF101 (Figure 5). |  | MSPRT-TANDEM | NP test | LSTM-s | LSTM-m | EARLIEST
---|---|---|---|---|---|---
|  | early | late | early | late | early | late | early | late | early
MSPRT-TANDEM | late | ***1E-07 |  |  |  |  |  |  |  |
NP test | early | 1.0 | ***1E-07 |  |  |  |  |  |  |
| late | ***1E-07 | 1.0 | ***1E-07 |  |  |  |  |  |
LSTM-s | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 |  |  |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 |  |  |  |
LSTM-m | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | 1.0 | ***1E-07 |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | *3E-03 | ***1E-07 |  |
EARLIEST | early | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | 1E-01 | ***1E-07 | *6E-03 | ***1E-07 |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07
Table 47: $p$-values from the Tukey-Kramer multi-comparison test conducted on HMDB51 (Figure 5). |  | MSPRT-TANDEM | NP test | LSTM-s | LSTM-m | EARLIEST
---|---|---|---|---|---|---
|  | early | late | early | late | early | late | early | late | early
MSPRT-TANDEM | late | ***1E-07 |  |  |  |  |  |  |  |
NP test | early | 1.0 | ***1E-07 |  |  |  |  |  |  |
| late | ***1E-07 | 0.2 | ***1E-07 |  |  |  |  |  |
LSTM-s | early | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |  |  |  |
| late | ***1E-07 | ***2E-07 | ***1E-07 | ***2E-02 | ***1E-07 |  |  |  |
LSTM-m | early | 1.0 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |  |  |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-04 | ***1E-07 | 1.0 | ***1E-07 |  |
EARLIEST | early | 1.0 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 | 1.0 | ***1E-07 |
| late | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***1E-07 | ***7E-07 | ***1E-07

###  L.2 Ablation Study: Figure 9

We also test the three conditions in the ablation study. We select one phase
at the 20th frame to conduct the one-way ANOVA with one model factor: LSEL +
Multiplet loss, LSEL only, and Multiplet only. The $p$-values are summarized
in Table 48. The result shows that using both the LSEL and the multiplet loss
is statistically significantly better than using either of the two losses.

Table 48: Figure 9: $p$-values from the Tukey-Kramer multi-comparison test conducted on the ablation test. | MSPRT-TANDEM
---|---
| LSEL+Multiplet | LSEL only
LSEL only | ***1E-09 |
Multiplet only | ***2E-07 | ***1E-09

##  Appendix M Supplementary Discussion

##### Interpretability.

Interpretability of classification results is one of the important interests
in early classification of time series [73, 18, 19, 20, 36]. MSPRT-TANDEM can
use the LLR trajectory to visualize the prediction process (see Figures 2
(Right) and 10); a large gap of LLRs between two timestamps means that these
timestamps are decisive.

##### Threshold matrix.

In our experiment, we use single-valued threshold matrices for simplicity.
General threshold matrices may enhance performance, especially when the
dataset is class-imbalanced [43, 4, 33]. Tuning the threshold after training
is referred to as thresholding, or threshold-moving [54, 7]. MSPRT-TANDEM has
multiple thresholds in the matrix form, and thus it is an interesting future
work to exploit such a matrix structure to attain higher accuracy.

##### How to determine threshold.

A user can choose a threshold by evaluating the mean hitting time and accuracy
on a dataset at hand (possibly the validation dataset, training dataset, or
both). As mentioned in Section 3.4, we do not have to retrain the model
(mentioned in Section 3.4). We can modify the threshold even after deployment,
if necessary, to address domain shift. This flexibility is a huge advantage
compared with other models that require additional training every time the
user wants to control the speed-accuracy tradeoff [8].

## Supplementary references

  * [1] Microsoft Vision Model ResNet-50.  https://pypi.org/project/microsoftvision/.  Accessed: May 14, 2021. License: Unknown.
  * [2] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  License: Apache License 2.0. Software available from tensorflow.org.
  * [3] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’19, page 2623–2631, New York, NY, USA, 2019. Association for Computing Machinery.  License: MIT License.
  * [4] A. Ali, S. M. Shamsuddin, A. L. Ralescu, et al.  Classification with class imbalance problem: a review.  Int. J. Advance Soft Compu. Appl, 7(3):176–204, 2015.
  * [5] K. J. Arrow, D. Blackwell, and M. A. Girshick.  Bayes and minimax solutions of sequential decision problems.  Econometrica, Journal of the Econometric Society, pages 213–244, 1949.
  * [6] J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl.  Algorithms for hyper-parameter optimization.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24, pages 2546–2554. Curran Associates, Inc., 2011.
  * [7] M. Buda, A. Maki, and M. A. Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106:249–259, 2018.
  * [8] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han.  Once-for-all: Train one network and specialize it for efficient deployment.  In International Conference on Learning Representations, 2020.
  * [9] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie.  Class-balanced loss based on effective number of samples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [10] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser.  Universal transformers.  In International Conference on Learning Representations, 2019.
  * [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [12] V. P. Dragalin, A. G. Tartakovsky, and V. V. Veeravalli.  Multihypothesis sequential probability ratio tests. i. asymptotic optimality.  IEEE Transactions on Information Theory, 45(7):2448–2461, November 1999.
  * [13] A. F. Ebihara, T. Miyagawa, K. Sakurai, and H. Imaoka.  Sequential density ratio estimation for simultaneous optimization of speed and accuracy.  In International Conference on Learning Representations, 2021.
  * [14] T. S. Ferguson.  Mathematical statistics: A decision theoretic approach, volume 1.  Academic press, 2014.
  * [15] R. Fisher.  Statistical methods for research workers.  Edinburgh Oliver & Boyd, 1925.
  * [16] T. Gangopadhyay, V. Ramanan, A. Akintayo, P. K. Boor, S. Sarkar, S. R. Chakravarthy, and S. Sarkar.  3d convolutional selective autoencoder for instability detection in combustion systems, 2021.
  * [17] F. A. Gers and J. Schmidhuber.  Recurrent nets that time and count.  Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, 3:189–194 vol.3, 2000.
  * [18] M. F. Ghalwash and Z. Obradovic.  Early classification of multivariate temporal observations by extraction of interpretable shapelets.  BMC bioinformatics, 13(1):195, 2012.
  * [19] M. F. Ghalwash, V. Radosavljevic, and Z. Obradovic.  Extraction of interpretable multivariate patterns for early diagnostics.  In 2013 IEEE 13th International Conference on Data Mining, pages 201–210, 2013.
  * [20] M. F. Ghalwash, V. Radosavljevic, and Z. Obradovic.  Utilizing temporal patterns for estimating uncertainty in interpretable early decision making.  In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, page 402–411, New York, NY, USA, 2014. Association for Computing Machinery.
  * [21] A. Ghodrati, B. E. Bejnordi, and A. Habibian.  FrameExit: Conditional early exiting for efficient video recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
  * [22] G. Golubev and R. Khas’minskii.  Sequential testing for several signals in gaussian white noise.  Theory of Probability & Its Applications, 28(3):573–584, 1984\.
  * [23] A. Graves.  Generating sequences with recurrent neural networks.  arXiv preprint arXiv:1308.0850, 2013.
  * [24] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.  On calibration of modern neural networks.  In International Conference on Machine Learning, pages 1321–1330. PMLR, 2017.
  * [25] A. Gupta, R. Pal, R. Mishra, H. P. Gupta, T. Dutta, and P. Hirani.  Game theory based early classification of rivers using time series data.  In 2019 IEEE 5th World Forum on Internet of Things (WF-IoT), pages 686–691, 2019.
  * [26] M. U. Gutmann and A. Hyvärinen.  Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.  The journal of machine learning research, 13(1):307–361, 2012.
  * [27] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. Del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, 09 2020.  License: BSD 3-Clause "New" or "Revised" License.
  * [28] T. Hartvigsen, C. Sen, X. Kong, and E. Rundensteiner.  Adaptive-halting policy network for early classification.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 101–110, New York, NY, USA, 2019. ACM.
  * [29] N. Hatami and C. Chira.  Classifiers with a reject option for early time-series classification.  2013 IEEE Symposium on Computational Intelligence and Ensemble Learning (CIEL), pages 9–16, 2013.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun.  Identity mappings in deep residual networks.  In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 630–645, 2016.
  * [32] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Comput., 9(8):1735–1780, Nov. 1997.
  * [33] C. Hong, R. Ghosh, and S. Srinivasan.  Dealing with class imbalance using thresholding.  arXiv preprint arXiv:1607.02705, 2016.
  * [34] Z. Huang, Z. Ye, S. Li, and R. Pan.  Length adaptive recurrent model for text classification.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, page 1019–1027, New York, NY, USA, 2017. Association for Computing Machinery.
  * [35] T. Kanamori, S. Hido, and M. Sugiyama.  A least-squares approach to direct importance estimation.  Journal of Machine Learning Research, 10(Jul):1391–1445, 2009.
  * [36] F. Karim, H. Darabi, S. Harford, S. Chen, and A. Sharabiani.  A framework for accurate time series classification based on partial observation.  In 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pages 634–639, 2019.
  * [37] Y. Kaya, S. Hong, and T. Dumitras.  Shallow-Deep Networks: Understanding and mitigating network overthinking.  In International Conference on Machine Learning, pages 3301–3310. PMLR, 2019.
  * [38] H. Khan, L. Marcuse, and B. Yener.  Deep density ratio estimation for change point detection.  arXiv preprint arXiv:1905.09876, 2019.
  * [39] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [40] C. Y. Kramer.  Extension of multiple range tests to group means with unequal numbers of replications.  Biometrics, 12(3):307–310, 1956.
  * [41] S. Kullback and R. A. Leibler.  On information and sufficiency.  Ann. Math. Statist., 22(1):79–86, 03 1951.
  * [42] Y. LeCun, C. Cortes, and C. Burges.  Mnist handwritten digit database.  ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2, 2010.  License: Creative Commons Attribution-Share Alike 3.0 license.
  * [43] R. Longadge and S. Dongre.  Class imbalance problem in data mining review.  arXiv preprint arXiv:1305.1707, 2013.
  * [44] A. P. López-Monroy, F. A. González, M. Montes, H. J. Escalante, and T. Solorio.  Early text classification using multi-resolution concept representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1216–1225, 2018.
  * [45] G. Lorden.  Nearly-optimal sequential tests for finitely many parameter values.  Annals of Statistics, 5:1–21, 01 1977.
  * [46] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [47] S. Ma, L. Sigal, and S. Sclaroff.  Learning activity progression in lstms for activity detection and early detection.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1942–1950, 2016.
  * [48] C. Martinez, E. Ramasso, G. Perrin, and M. Rombaut.  Adaptive early classification of temporal sequences using deep reinforcement learning.  Knowledge-Based Systems, 190:105290, February 2020.
  * [49] MATLAB.  version 9.3.0 (R2017b).  The MathWorks Inc., Natick, Massachusetts, 2017.
  * [50] A. McGovern, D. H. Rosendahl, R. A. Brown, and K. K. Droegemeier.  Identifying predictive multi-dimensional time series motifs: an application to severe weather prediction.  Data Mining and Knowledge Discovery, 22(1-2):232–258, 2011.
  * [51] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar.  Long-tail learning via logit adjustment.  In International Conference on Learning Representations, 2021.
  * [52] U. Mori, A. Mendiburu, E. J. Keogh, and J. A. Lozano.  Reliable early classification of time series based on discriminating the classes over time.  Data Mining and Knowledge Discovery, 31:233–263, 04 2016.
  * [53] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.  License: https://github.com/pytorch/pytorch/blob/master/LICENSE.
  * [54] M. D. Richard and R. P. Lippmann.  Neural network classifiers estimate bayesian a posteriori probabilities.  Neural computation, 3(4):461–483, 1991.
  * [55] D. E. Rumelhart, G. E. Hinton, and R. J. Williams.  Learning representations by back-propagating errors.  Nature, 323(6088):533–536, 1986.
  * [56] M. Rußwurm, S. Lefèvre, N. Courty, R. Emonet, M. Körner, and R. Tavenard.  End-to-end learning for early classification of time series.  CoRR, abs/1901.10681, 2019.
  * [57] P. Schäfer and U. Leser.  TEASER: Early and accurate time series classification.  Data Mining and Knowledge Discovery, 34(5):1336–1362, 2020.
  * [58] M. Sugiyama, T. Suzuki, S. Nakajima, H. Kashima, P. von Bünau, and M. Kawanabe.  Direct importance estimation for covariate shift adaptation.  Annals of the Institute of Statistical Mathematics, 60(4):699–746, 2008.
  * [59] T. Suzuki, H. Kataoka, Y. Aoki, and Y. Satoh.  Anticipating traffic accidents with adaptive loss and large-scale incident db.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3521–3529, 2018.
  * [60] A. G. Tartakovskij.  Sequential composite hypothesis testing with dependent nonstationary observations.  Probl. Inf. Transm., 17:18–28, 1981.
  * [61] A. Tartakovsky.  Sequential methods in the theory of information systems, 1991.
  * [62] A. Tartakovsky.  Asymptotically optimal sequential tests for nonhomogeneous processes.  Sequential analysis, 17(1):33–61, 1998.
  * [63] A. Tartakovsky, I. Nikiforov, and M. Basseville.  Sequential Analysis: Hypothesis Testing and Changepoint Detection.  Chapman & Hall/CRC, 1st edition, 2014.
  * [64] A. G. Tartakovsky.  Asymptotic optimality of certain multihypothesis sequential tests: Non-i.i.d. case.  Statistical Inference for Stochastic Processes, 1(3):265–295, 1998\.
  * [65] J. W. Tukey.  Comparing individual means in the analysis of variance.  Biometrics, 5 2:99–114, 1949.
  * [66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017\.
  * [67] N. Verdenskaya and A. Tartakovskii.  Asymptotically optimal sequential testing of multiple hypotheses for nonhomogeneous gaussian processes in asymmetric case.  Theory of Probability & Its Applications, 36(3):536–547, 1992\.
  * [68] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. Jarrod Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. Carey, İ. Polat, Y. Feng, E. W. Moore, J. Vand erPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and S. . . Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17:261–272, 2020.  License: BSD 3-Clause "New" or "Revised" License.
  * [69] A. Wald.  Sequential tests of statistical hypotheses.  Ann. Math. Statist., 16(2):117–186, 06 1945.
  * [70] B. Wang, L. Huang, and M. Hoai.  Active vision for early recognition of human actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [71] W. Wang, C. Chen, W. Wang, P. Rai, and L. Carin.  Earliness-aware deep convolutional networks for early time series classification.  arXiv preprint arXiv:1611.04578, 2016.
  * [72] Z. Xing, J. Pei, and P. S. Yu.  Early classification on time series.  Knowledge and Information Systems, 31(1):105–127, Apr. 2012.
  * [73] Z. Xing, J. Pei, P. S. Yu, and K. Wang.  Extracting interpretable features for early classification on time series.  In Proceedings of the 11th SIAM International Conference on Data Mining, SDM 2011, pages 247–258. SIAM, 2011.
  * [74] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei.  BERT loses patience: Fast and robust inference with early exit.  In Advances in Neural Information Processing Systems, 2020.