# From Multilayer Perceptron to GPT: A Reflection on Deep Learning Research
for
Wireless Physical Layer

Mohamed Akrout, Amine Mezghani, , Ekram Hossain, ,
Faouzi Bellili, , Robert W. Heath

###### Abstract

Most research studies on deep learning (DL) applied to the physical layer of
wireless communication do not put forward the critical role of the accuracy-
generalization trade-off in developing and evaluating practical algorithms. To
highlight the disadvantage of this common practice, we revisit a data decoding
example from one of the first papers introducing DL-based end-to-end wireless
communication systems to the research community and promoting the use of
artificial intelligence (AI)/DL for the wireless physical layer. We then put
forward two key trade-offs in designing DL models for communication, namely,
accuracy versus generalization and compression versus latency. We discuss
their relevance in the context of wireless communications use cases using
emerging DL models including large language models (LLMs). Finally, we
summarize our proposed evaluation guidelines to enhance the research impact of
DL on wireless communications. These guidelines are an attempt to reconcile
the empirical nature of DL research with the rigorous requirement metrics of
wireless communications systems.

##  I Introduction

Researchers are developing use cases where DL can potentially enhance the
system performance and reduce complexity/overhead compared to classical
methods (see envisioned examples for standardization in the 3GPP Release 18
[1, Section 9.2]). Since the introduction of the AlexNet network in 2012, the
use of deep neural networks (DNNs) has skyrocketed within the communications
community by substituting conventional optimization solvers with generative
and/or discriminative DL techniques. Fig. 1 summarizes some of the key DL
models applied to communications problems, starting from the multilayer
perception (MLP) model [2] to the diffusion model [3]. We refer the reader to
[4] for a comprehensive survey on the applications of DL for wireless physical
layer design. In Fig. 1, we also include the recent generative pre-trained
transformer (GPT) models [5, 6] as they are currently initiating many
discussions within the communications research community about the data
compression properties of large language models (LLMs) and their role in
replicating digital twins (cf. Section III-B).

Because the sixth-generation (6G) networks are envisaged as multi-band,
decentralized, fully autonomous, and hyper-flexible user-centric systems
encompassing satellite, aerial, terrestrial, underwater, and underground
communications, DL techniques are expected to partially or fully substitute
classical methods, their assessment metrics should comply with rigorous
evaluation guidelines that equally address latency, complexity,
generalization, and accuracy. In this paper, we reflect on the last decade of
research on DL for wireless communications with a focus on the physical and
link layers. We start by highlighting the limitations of state-of-the-art DL
methods for wireless communications. We do so by revisiting one of the first
published papers [7] introducing DL-based end-to-end communications systems
over additive white Gaussian noise (AWGN) channels. By doing so, we pinpoint
how [7], as many of the subsequent DL papers for wireless, turn the spotlight
on the accuracy of DL methods at the cost of sacrificing their generalization
capabilities. We also highlight how the open literature does not draw enough
attention to the important practical considerations for designing DL systems
for wireless communications, such as data acquirement and adaptation to new
system dimensions. We then describe the two key trade-offs in designing DL-
aided wireless communications systems, namely, accuracy versus generalization
and compression versus latency. These two trade-offs offer new evaluation
guidelines to assess future DL research directions for wireless
communications.

Figure 1: Publication timeline of the key DL models used and/or being
investigated by the communications research community.

##  II Limitations of the State-of-the-Art
DL Techniques for Wireless Communications

In this section, we describe the limitations of neglecting the data
distribution shift when evaluating DL models for wireless communications at
the physical layer. Specifically, we revisit an example from [7] to illustrate
the drawbacks of blindly applying black-box DL models in a plug-and-play
manner when only the model accuracy is assessed. We also describe other
challenges arising from the use of DL techniques that are usually not examined
in the open literature.

###  II-A Evaluation of a Single Metric

It is always possible to beat a classical method that solves a non-closed form
problem using DL techniques based on deep neural networks (DNNs) given a known
model by generating training and test datasets on which DNNs are both trained
and evaluated. For this reason, it is critical to use both the accuracy and
generalization of DNNs to assess their performance on a variety of:

  * •

in-distribution (ID) scenarios where the training and testing datasets are
generated from the same distribution (e.g., the same user speed is assumed to
generate mobility data to be used for training and testing).

  * •

out-of-distribution (OOD) scenarios where a distribution shift occurs between
training and testing datasets (e.g., different user speeds are assumed to
generate mobility data to be used for training and testing).

In other words, DL models that are either accurate and non-robust or highly
biased and robust are equally worthless for real-time physical/link layer
applications.

To illustrate this idea, we revisit the study [7], which is one of the first
papers promising the design of communications systems as an autoencoder for
reconstruction tasks that jointly optimize transmitter and receiver components
in a single end-to-end process. As shown in Fig. 2, the physical communication
chain of the transmitter and the receiver are substituted by an encoder and a
decoder, respectively. The channel is represented as a noisy non-parameterized
layer that injects an additive Gaussian white noise at a specific energy per
bit to noise power spectral density ratio, $E_{b}/N_{0}$.

Figure 2: An end-to-end communication system over an AWGN channel as an
autoencoder: the input is the one-hot encoded transmit symbol $\bm{s}$ while
the estimated one-hot symbol $\widehat{\bm{s}}$ is argmax of the softmax
probability distribution over all possible messages.

In [7], the training of the autoencoder was performed on a dataset generated
at $E_{b}/N_{0}=7\,\textrm{dB}$. However, the evaluation was conducted over a
range of $E_{b}/N_{0}$ between in $[-4\,\textrm{dB},\,8\,\textrm{dB}]$. By
doing so, the authors obtained a lower block error rate (BLER) than the
Hamming code with rate $R=4/7$ and concluded that the autoencoder “has learned
some joint coding and modulation scheme, such that a coding gain is achieved”.

From a machine learning theory perspective, this conclusion is questionable.
For this reason, we train multiple autoencoders, each with a training
$E_{b}/N_{0}\in\\{-4,0,5,7,8\\}$ dB. We then evaluate each autoencoder on the
testing range $E_{b}/N_{0}\in[-4\,\textrm{dB},\,8\,\textrm{dB}]$ as shown in
Fig. 3. There, all autoencoders in Fig. 3 exhibit a decreasing BLER over the
entire test $E_{b}/N_{0}$ range even though they were trained on one single
$E_{b}/N_{0}$ value. While this fact suggests that autoencoders over AWGN
channels generalize well to out-of-distribution decoding scenarios, it can
also indicate that the problem at hand is easy to solve because the
distribution shift between the training data distribution associated with
$E_{b}/N_{0}=7\,\textrm{dB}$ and the test data distributions with
$E_{b}/N_{0}\in[-4\,\textrm{dB},\,8\,\textrm{dB}]$ is minimal. To confirm this
fact, we report in Table I the percentage of the area overlap between the data
distribution of the received signal
$\bm{y}\sim\mathcal{N}\left(\bm{x},\frac{1}{2\,R\,E_{b}/N_{0}}\,\mathbf{I}\right)$
for training and testing $E_{b}/N_{0}$ values 111To quantify the shift between
two distributions, their area overlap is more informative than their KL-
divergence because the latter only accounts for the region where both
distributions are non-zero.. We observe the fact that lower testing values for
$E_{b}/N_{0}$ compared to the training value $E_{b}/N_{0}=7\,\textrm{dB}$
yields a lower area overlap, or equivalently, a higher OOD shift. However, it
is interesting to note the significantly high overall overlap. This suggests
that the AWGN channel is a simplistic model to assess the generalization
performance of DL models. One can perceive the AWGN channel model in
communication as the MNIST dataset in computer vision on which any great
performance is considered obsolete by the computer science community due to
the intrinsic simplicity of the handwritten digit classification task.

TABLE I: The percentage of area overlap between the training ($E_{b}/N_{0}=7\,\textrm{dB}$) and testing received signal distributions ($E_{b}/N_{0}\in\\{-4,0,5,8\\}$ dB). Test $\mathbf{E_{b}/N_{0}}$ |  $-4$ dB |  $0$ dB |  $5$ dB |  $8$ dB
---|---|---|---|---
Area overlap with | 45.70% | 62.97% | 88.91% | 94.43%
train $\mathbf{E_{b}/N_{0}=7}$ dB
Figure 3: BLER of the autoencoder versus test $E_{b}/N_{0}$ for multiple
training $E_{b}/N_{0}$.

It is also unclear whether the chosen value $E_{b}/N_{0}=7\,\textrm{dB}$ is
the best one to select. It is seen that training autoencoders on lower
$E_{b}/N_{0}$ values in $[-4\,\textrm{dB},\,8\,\textrm{dB}]$ (i.e., with
higher noise levels) leads to a smaller BLER across the entire test
$E_{b}/N_{0}$ interval. This result not only shows that the train
$E_{b}/N_{0}=7\,\textrm{dB}$ selected in [7] is not the best choice, but also
demonstrates how noisy training can be more beneficial for better
generalization.

Given all the aforementioned reasons, the fact that the autoencoder
outperformed the Hamming code in [7] is more justified by the simplicity of
the AWGN channel model which does not shift significantly the received signal
in the testing $E_{b}/N_{0}$ interval. As a matter of fact, adding a noise
correlation to the AWGN channel or accounting for the fading effects by
changing the AWGN channel to a Rayleigh one break down the decoding
performance of the autoencoder. In summary, the analysis of the results as a
function of the trade-off between generalization and accuracy metrics opens
the door to future rigorous investigations and reveals more insights about
better data generation and model training choices.

###  II-B Is Meta-learning Sufficient for Generalization?

To sidestep the need for large data samples to train DL models, meta-learning
optimizes a general model using samples from multiple tasks (a.k.a., meta-
tasks) in order to adapt to new unseen tasks. By designing meta-tasks
associated with specific communication scenarios using the model-agnostic
meta-learning (MAML) framework [8], prior work reported better performance for
meta-learning solutions compared to standard DL methods. A few attempts bypass
the black-box nature of DL models and connect communication systems models
with meta-learning. This enables learning a subset of the model-based
parameters, thereby minimizing the search space induced by black-box DL models
[9]. However, the large body of research about generalization in wireless is
limited to creating multiple meta-tasks with different communication
conditions following the MAML-like framework. While enumerating and generating
meta-tasks do not scale for some complex communication scenarios,
understanding which feature is invariant across different domains becomes
critical for scaling DL techniques. By doing so, one relies on the domain
knowledge in addition to the standard two-step optimization of meta-learning.
For instance, variations of complex signals related to the phase generalize
better than those related to the amplitude. Overall, the study of the features
in the context of meta-learning (a.k.a., meta-features) for wireless problems
has not been explored by the wireless communications community. Existing
studies rely on back-propagation to extract suitable correlations
characterizing specific meta-tasks. This is different from those in the ML
community which investigate data properties that affect the learning
performance, and measure similarities between datasets and meta-features using
multiple criteria such as mutual information and density skewness [10].

###  II-C Unquestioned Sources of Dataset

In conventional communications protocols, a significant portion of a
transmission interval (e.g., time slot) is usually used to send training
sequences for channel estimation. The real-time computation involved during
these training time slots defines the computational complexity of many
communication methods. For possible future AI-aided communication protocols,
it is still unclear whether they will be designed based on fully offline
training procedures or by additionally relying on extra finetuning steps. The
latter scenario raises the question about real-time collection of data for
adaptation purposes, and the related overhead to determine the complexity of
DL techniques. The existing research in the open literature disregards these
practical challenges and focuses entirely on fully offline-trained DL methods,
which, at the current state, cannot fulfill the adaption capabilities of
wireless systems envisioned for 6G.

When accuracy is the only metric of evaluation of DL models, ignoring these
practical challenges seems acceptable because offline training is enough to
judge the DL performance. However, when the assessment of DL techniques
accounts for their generalization issues, the availability of data sources and
their properties becomes a central component of the analysis.

###  II-D Use of Reinforcement Learning for Optimization

In the pre-DL era (i.e., before 2012), optimizing non-convex problems using
gradient descent (GD) algorithms were not very popular in the wireless
research community. Instead of using GD-based optimization, researchers
convexified the non-convex problems in order to solve them. DL research has
introduced a plethora of optimizers to train DNNs which has made GD a popular
technique. This is partly due to the unique properties of flat minima
characterizing DNNs’ loss functions [11]. Consequently, DL techniques have
been used to solve optimization problems such as channel and power allocation,
beamforming/precoding, user association, and trajectory planning for unmanned
aerial vehicles (UAVs) in the physical/link layers in various contexts and
system models.

A common practice to tackle non-convex communication problems in the DL era is
by resorting to the reinforcement learning (RL) paradigm. RL agents are
trained to learn the optimal policy to act on an environment in order to
maximize the sum of the instantaneous reward signals received from an
environment. We refer the reader to [12] for a rigorous treatment of RL
formulation in terms of Markov decision processes (MDPs). By substituting the
reward signal within the RL formalism with a convex or non-convex cost
function to be optimized, the RL agent can find the best policy to optimize
it. For instance, one can associate the beamforming vector to the action of
the RL agent and the achievable rate of the communication system to be the
reward signal [13].

Because wireless communications problems must satisfy multiple constraints
simultaneously (e.g., power, latency, signal-to-interference-plus-noise ratio
[SINR]), prior work made use of clipping strategies to enforce constraints on
the RL agent’s output. While this strategy provides good results in some
scenarios, it does not guarantee an optimal solution. A better choice would be
to cast the communication problem within a constrained MDP formalism [14].
However, little effort has been dedicated to properly incorporating the
constraints, and the unconstrained MDP approach remains a popular data-driven
approach to optimize non-convex communication problems. In addition, the
discussion of RL challenges in terms of sample efficiency and generalization
is usually neglected despite the fact that it is an active research area
within the ML research community.

(a)

(b)

Figure 4: Plots of training and test losses characterized by: (a) a U-shaped
loss curve justified by the bias-variance trade-off, and (b) the double-
descent loss curve combining the U-shaped loss curve for the under-
parameterized regime and a decreasing loss for the over-parameterized regime.
The latter is not a phenomenon exhibited by all deep learning models and hence
is not well understood yet.

###  II-E Fixed Structures of Deep Neural Networks

DNNs have fixed structures and it is not possible to change them anymore after
initialization. This inflexibility is considered a drawback because many
wireless communications problems may require different input/output sizes over
time. As one example, consider the problem of channel estimation at the base
station based on the SINR vector with each component being associated with a
specific user. In this case, the DNN input is the SINR vector while the
estimated channel matrix represents the DNN output. During multiple
transmission blocks, the number of users changes and so does the size of the
SINR vector. Using the dropout method [15], it is possible to randomly drop
the contribution of some neurons during the DNN training. While this provides
flexibility in the structure of hidden layers, the input and output layers
still have a fixed size, and hence limited flexibility in practice. Actively
altering the network structure by adding neurons is both a promising and
challenging direction. Because DL research for wireless communications
generally considers DNNs as black-box modules, this field has not attracted
the attention of the wireless research community and opted for constant-
padding the input and output layers to the expected maximum size. Another
unexplored area in this direction is the mapping of different vector sizes to
the same latent space, thereby unifying the DNN input into the same feature
space. While feature extractors for computer vision and natural language
processing (NLP) tasks are abundant, wireless communications problems require
signal-based feature extractors, which have not been well examined by our
research community yet. It is worth noting that the recent wave of NLP models
has allowed DNNs to have an apparent flexibility in the input and output sizes
since LLMs are trained to sequentially predict the next token given the
current token. This is to be opposed to the application of DL for wireless
where the output of DNNs is obtained with a one-shot inference run.

##  III Two Key Trade-offs for DL
Applied to Wireless Physical Layer

In this section, we highlight the importance of two fundamental learning
trade-offs in the assessment of DL models, namely, accuracy versus
generalization, and compression versus latency. In particular, we discuss how
accuracy-generalization should guide the evaluation of smaller DL models and
highlight the importance of the compression-latency trade-off for practical
use cases of LLMs. Through this section, generalization does not refer to the
stability of the model’s performance under noise and adversarial examples
(i.e., adversarial robustness), but rather to the ability of the model to
generalize to unseen scenarios.

Figure 5: The key trade-offs in assessing the relevance of small DL models and
LLMs for communication use cases.

###  III-A Accuracy Versus Generalization

(a)

(b)

Figure 6: Image transfer using (a) conventional communication, and (b) using a
LLM as an image compressor to a prompt.

In classical machine learning, the trade-off between the accuracy and the
generalization pertains to the fundamental bias–variance trade-off which
characterizes the generalization capabilities of predictive models [16]. The
bias–variance trade-off stipulates that a model must have enough parameters to
capture the underlying structure of the dataset without over-fitting spurious
patterns. Fig. 4(a) depicts the expected U-shaped variation of the test loss
whose minimum represents the sweet spot in terms of the number of parameters
between under-fitting and over-fitting. In DL practice, however, deeper
networks with a large number of parameters are trained to interpolate between
the training samples and do maintain a lower test loss accuracy on test
datasets as depicted by the “double descent” curve in Fig. 4(b). While this
over-parameterized regime depicted might look contradictory to classical
(i.e., under-parameterized) regime, this over-parameterized regime is not well
understood and is being actively investigated within the theoretical DL
community [17]. This is because the double descent behavior does not
consistently occur for every DNN and some of them, even very deep ones, still
empirically obey the bias-variance trade-off [18]. Whether the DNN follows the
variance-bias trade-off or the double descent behavior depends on the DNN’s
parameters such as the number of input data points, the number of layers, and
the overall number of parameters.

For the above reasons, evaluating the performance of DL models must be
entirely tied to the employed DNN architecture. For example, one cannot claim
that DL decoding is superior to classical methods because the mean-square
error of one specific DNN tested on a few evaluation scenarios is lower than
the one of a classical method (cf. Section II-A). Recent efforts from the
communication community initiated the application of different learning
paradigms including transfer learning, meta-learning, and continual learning
to investigate the generalization of DNNs [19]. Some studies also showed that
even the incorporation of domain knowledge in supervised learning without the
explicit use of other learning paradigms can improve the DNN’s generalization.
For instance, by alternating between the time-domain and frequency-domain
representations of signals and using the idea of successive-estimation and
cancellation, it is possible to design DNNs to handle multiple sinusoid waves
and improve their estimation performance on out-of-distribution samples [20].

In summary, by blinding applying DNNs without accounting for their accuracy-
generalization trade-off, most research results become biased and less
impactful in the long term.

At the time of writing, potential leaks about the unknown GPT-4 architecture
on Reddit [21] revealed that GPT-4 is not one giant monolithic lossy dataset
compressor but rather an ensemble of eight 220B-parameter LLMs, each trained
with different data/task distributions. This suggests that GPT-4 is a mixture
of experts (unlike GPT-3.5 and GPT-3) and operates at a much smaller over-
parameterized regime that challenges the myth of very large and fully end-to-
end DNNs.

###  III-B Compression Versus Latency

Figure 7: The implementation of digital twins depends on finding the trade-off
between latency and compression that satisfies the QoS.

The study of the trade-off between lossless/lossy compression and latency has
historically evolved around reaching the source coding limit established in
1948 by Shannon’s seminal work [22]. In a nutshell, it stipulates that higher
compression ratios require longer block lengths and thus higher encoding and
decoding time complexity. Specifically, Shannon showed that the codeword
length of an optimal prefix-free code is approximately the negative logarithm
of the codeword’s probability. He also proved that the expected message length
of an optimal prefix-free code is close to the entropy of the message. Shannon
also explored the information-theoretic relationship between compression and
(next-letter) prediction by estimating the entropy of the English language
[23]. Similar to the connection between next-letter predictors and data
compressors, the emerging LLMs represent lossy data compressors in a data-
driven fashion which exploit the inherent redundancy within the human language
to embed massive datasets into a significantly smaller DNN model [24]. They
are giant DNNs with billions of parameters compressing a large number of
datasets by learning to predict the next token from the previous context
composed of one or multiple tokens.

Given the tight information-theoretic relationship between the compression
ratio and the decoding latency, it is therefore natural at this time to review
some of the envisioned communication use cases of LLMs. The fascination of the
general public with the quality of the output text of LLMs suggests that their
compression quality will be at the cost of their decoding latency. Indeed, an
LLM with a few billion parameters requires around 100 GB of RAM. Moreover, the
current optimized deployment of LLMs for efficient inference comes at the
expense of high backend infrastructure costs and significant latency. Aware of
these issues, important research efforts from the LLM community are actively
examining the effect of quantization on the inference time of LLMs without
significantly affecting their accuracies. For these reasons, the discussed
applications of LLMs by the communications research community focus on
latency-tolerant use cases at the application layer.

The first use case of LLMs for communication is image transfer. As shown in
Fig. 6(b), a user equipped with an LLM obtains the text prompt of the image to
transfer. Only the prompt is sent through the channel as a bit stream. The
prompt is then recovered from the received bits to probe the LLM, which in
turn outputs the image to the receiver. This is unlike the conventional data
transfer where the entire bit stream of the image is transmitted through the
channel as in Fig. 6(a). Here, the LLM-aided image transfer can provide a
significant data compression rate under two conditions. The first one is that
the parameters of the transmit and receive LLMs must be equal, or that their
embeddings yield the encoded and decoded image. The second one is related to
the compression-latency trade-off where both the encoding and decoding time of
LLMs must provide a significant advantage over the improvement of the
compression ratio between an image and a text prompt.

Another use case of LLMs is to minimize the need for communication between two
users by learning a personalized LLM as a digital twin for each one of them as
shown in Fig. 7. This means that an LLM must be able to reliably mimic the
user interaction. From a probabilistic perspective, this corresponds to
reliably learning the joint distribution of reactions, preferences, and
thoughts of any user. The realizability of this scenario is still far from
being a reality in the near future due to multiple reasons. This includes the
data privacy and security concern, as well as the hallucination effects
exhibited by LLMs as they tend to output text that appears to be correct but
is actually false or not based on the input given. In summary, short-cutting
the cost of communication must come with the benefit of highly accurate
digital twins. Another important limitation to implementing these use cases is
the lack of rigorous metrics to assess the performance of LLMs. In fact,
existing evaluation protocols are either automatic (e.g., the ROUGE metric for
text summarization) exhibiting poor correlation to human judgments, or manual
yielding noisy and potentially biased annotations. In summary, the adaptation
of LLMs to the unique signal-based nature of communication datasets is vague
and uncertain and is still in its infancy.

##  IV Conclusion

The ever-increasing requirements for future 6G wireless applications call for
the urgent need to go beyond the accuracy metric to assess DL methods for
communication. We have described the importance of the accuracy-generalization
and compression-latency trade-offs in shaping the future evaluation guidelines
of DL techniques for wireless problems as summarized in Fig. 8.

Figure 8: The proposed evaluation components.

These evaluation criteria should be continuously updated in light of a new
understanding of the specific challenges facing the applications of deep
learning models for wireless communication problems. We also have discussed
how these metrics are critical in evaluating the relevance of emerging deep
learning models including large language models. We believe these trade-offs
bridge the gap between the empirical nature of deep learning models applied to
communication problems and the challenging technical requirements of future
communication systems.

## Acknowledgments

The work of M. Akrout was supported by the doctoral scholarship of the Natural
Sciences and Engineering Research Council of Canada (NSERC). The work of A.
Mezghani, E. Hossain, and F. Bellili were supported by Discovery Grants from
NSERC. The work of R. Heath was supported by the National Science Foundation
(grant nos. NSF-ECCS-2153698, NSF-CCF-2225555, NSF-CNS-2147955) and in part by
funds from federal agency and industry partners as specified in the Resilient
& Intelligent NextG Systems (RINGS) program. The authors also acknowledge the
insightful comments and suggestions from Osvaldo Simeone.

## References

  * [1] 3GPP Release 18, “Study on Artificial Intelligence (AI)/Machine Learning (ML) for NR Air Interface RAN,” Meeting #112, Athens, Greece, Tech. Rep., 27th February – 3rd March 2023.
  * [2] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _nature_ , vol. 323, no. 6088, pp. 533–536, 1986.
  * [3] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 6840–6851, 2020.
  * [4] J. Tanveer, A. Haider, R. Ali, and A. Kim, “Machine learning for physical layer in 5g and beyond wireless networks: A survey,” _Electronics_ , vol. 11, no. 1, p. 121, 2022.
  * [5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [6] OpenAI, “Gpt-4 technical report,” 2023.
  * [7] T. O’shea and J. Hoydis, “An introduction to deep learning for the physical layer,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 3, no. 4, pp. 563–575, 2017.
  * [8] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [9] T. Raviv, S. Park, O. Simeone, Y. C. Eldar, and N. Shlezinger, “Online meta-learning for hybrid model-based deep receivers,” _IEEE Transactions on Wireless Communications_ , 2023.
  * [10] A. Rivolli, L. P. Garcia, C. Soares, J. Vanschoren, and A. C. de Carvalho, “Characterizing classification datasets: a study of meta-features for meta-learning,” _arXiv preprint arXiv:1808.10406_ , 2018.
  * [11] R. Mulayoff and T. Michaeli, “Unique properties of flat minima in deep networks,” in _International conference on machine learning_. PMLR, 2020, pp. 7108–7118.
  * [12] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [13] Y. Al-Eryani, M. Akrout, and E. Hossain, “Multiple access in cell-free networks: Outage performance, dynamic clustering, and deep reinforcement learning-based design,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 4, pp. 1028–1042, 2020.
  * [14] E. Altman, _Constrained Markov decision processes_. CRC press, 1999, vol. 7.
  * [15] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [16] T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman, _The elements of statistical learning: data mining, inference, and prediction_. Springer, 2009, vol. 2.
  * [17] M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off,” _Proceedings of the National Academy of Sciences_ , vol. 116, no. 32, pp. 15 849–15 854, 2019\.
  * [18] S. Bubeck and M. Sellke, “A universal law of robustness via isoperimetry,” _Journal of the ACM_ , vol. 70, no. 2, pp. 1–18, 2023.
  * [19] M. Akrout, A. Feriani, F. Bellili, A. Mezghani, and E. Hossain, “Domain generalization in machine learning models for wireless communications: Concepts, state-of-the-art, and open issues,” _arXiv preprint arXiv:2303.08106_ , 2023.
  * [20] R. M. Dreifuerst and R. W. Heath, “Signalnet: A low resolution sinusoid decomposition and estimation network,” _IEEE Transactions on Signal Processing_ , vol. 70, pp. 4454–4467, 2022.
  * [21] “Potential GPT-4 architecture description,” https://www.reddit.com/r/LocalLLaMA/comments/14eoh4f/rumor_potential_gpt4_architecture_description/, accessed: 2023-06-25.
  * [22] C. E. Shannon, “A mathematical theory of communication,” _The Bell system technical journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [23] C. E. Shannon, “Prediction and entropy of printed english,” _Bell system technical journal_ , vol. 30, no. 1, pp. 50–64, 1951.
  * [24] Z. Jiang, M. Yang, M. Tsirlin, R. Tang, Y. Dai, and J. Lin, ““low-resource” text classification: A parameter-free classification method with compressors,” in _Findings of the Association for Computational Linguistics: ACL 2023_ , 2023, pp. 6810–6828.


# Corner Case Data Description and Detection

Tinghui Ouyang1,Vicent Sant Marco1, Yoshinao Isobe2, Hideki Asoh1, Yutaka
Oiwa2, Yoshiki Seo1
1Artificial Intelligence Research Center (AIRC),
National Institute of Advanced Industrial Science and Technology (AIST),
Tokyo, Japan 2Cyber Physical Security Research Center,
National Institute of Advanced Industrial Science and Technology (AIST),
Tsukuba, Japan<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As the major factors affecting the safety of deep learning models, corner
cases and related detection are crucial in AI quality assurance for
constructing safety- and security-critical systems. The generic corner case
researches involve two interesting topics. One is to enhance DL models’
robustness to corner case data via the adjustment on parameters/structure. The
other is to generate new corner cases for model retraining and improvement.
However, the complex architecture and the huge amount of parameters make the
robust adjustment of DL models not easy, meanwhile it is not possible to
generate all real-world corner cases for DL training. Therefore, this paper
proposes a simple and novel approach aiming at corner case data detection via
a specific metric. This metric is developed on surprise adequacy (SA) which
has advantages on capture data behaviors. Furthermore, targeting at
characteristics of corner case data, three modifications on distanced-based SA
are developed for classification applications in this paper. Consequently,
through the experiment analysis on MNIST data and industrial data, the
feasibility and usefulness of the proposed method on corner case data
detection are verified.

###### Index Terms:

corner case data detection, surprise adequacy, modified distanced-based SA, AI
quality testing,

##  I Introduction

As the fast development of machine learning technologies in the last decade,
especially deep learning (DL) based techniques, this makes Artificial
Intelligent (AI) a widely-known term in both industries and our daily life.
However, this fast development also makes us concerning AI applications, and
one of these concerns is the safety [1]. As more complex architectures of deep
neural networks are applied to obtain high accuracy in applications, the
safety of DL-based system becomes more critical than ever before, especially
in life- and property-related fields, e.g. medical diagnosis [2], malware
detection [3], autonomous driving [4] and so on. For example, autonomous
driving is being developed in many big companies, like Tesla, Ford,
Waymo/Google, but several serious accidents have happened in real-world due to
DLs’ safety issues [5]-[6]. Therefore, it is significantly important in the DL
system testing to find factors affecting safety and to take measurements to
eliminate them.

In both traditional software and DL-based systems, incorrect or unexpected
corner case behaviors are always regarded to play an important part of factors
affecting systems’ safety [7]. For example, in autonomous driving, many
reported real-world collision cases are related to those rare or previously
unseen corner cases. In [8], a collision between a Tesla car and a trailer was
reported since the Tesla’s DL system failed to deal with the corner cases of
“white color against a brightly lit sky” and the “high ride height”. Another
collision was reported on a Google self-driving car crashed to a bus since its
DL system made wrong decision on rare conditions (corner cases) [9]. Besides
the given two examples, many other wrong decisions made by AI systems are more
or less related to different corner cases. Therefore, to detect and fix those
potential flaws or undesired corner case behaviors, corner case detection is
surely crucial for system designers who expect to construct safety- and
security-critical DL systems, and it should also be a necessary part of
systematical AI testing.

Until now, there have many scholars studying on corner cases and related AI
testing techniques [10]-[11]. Firstly, from the perspective of concepts,
corner cases are namely those erroneous behaviors in the DL-based software,
which are analogous to bugs in traditional software [12]. While, different
with traditional software, corner cases of DL systems pay more attention on
testing data instead of system architecture. Hence, traditional software
usually aims at patch fixing after bug detection, DL-based systems need to
consider these detected erroneous corner case data into the training data set,
then subsequently retrain the AI model to improve its structure and
parameters. Secondly, researches about corner cases study mainly include the
following two parts. One kind of study is to test a given DL model’s
robustness and stability when facing with corner case data. For example, some
adversarial attack strategies, such as in [13]-[14], are applied on the
original testing data to generate adversarial samples which can reflect the
feature of corner cases to some extent, then DL model’s accuracy and
robustness are tested. The DeepFool proposed in [15] can also realize the
generation of adversarial corner cases and robustness testing. The
DeepMutation proposed in [16] aimed at mutating DL models’ structure and
parameters directly, and testing the modified model’s performance on the
original testing data. Moreover, through this mutation and testing, some
undetected corner cases can be further detected. For example, if a data is
found erroneous after a small model mutation, it is possible to be regarded as
a potential corner case due to its high risk of causing wrong decision.
However, considering current industrial DL systems always having thousands of
neurons and millions of parameters, it is extremely challenging to detect
corner cases via introducing small mutations or perturbations on models. The
other kind of study is to generate corner case data for retraining and
improving DL systems’ performance. For instance, the DeepMutation technique
also proposed some mutation operators on data, and generated some corner cases
for DL model testing. In [17], biased corner cases were also generated from
MNIST dataset based on the metamorphic testing technique. Moreover, the
DeepTest method proposed in [7] aimed at the autonomous driving scenario, and
generated lots of corner cases by leveraging image transformations techniques
to change the driving conditions, like rain, fog and lighting. Furthermore,
the DeepXplore method [18] proposed an idea of neuron coverage, and made use
of it to iteratively learn erroneous corner cases. There were thousands of
incorrect corner case behaviors generated in this paper. However, even though
companies like Tesla and Google have developed many effective techniques in
corner case study, it is still hardly possible to generate and consider all
kinds of corner cases in the training process. The only feasible way would be
studying and detecting corner cases as possible in AI testing.

Concerning the mentioned techniques and problems on corner case study, this
paper proposes to develop a novel corner case data detection method by using
modified _distance-based surprise adequacy_ (DSA). The idea of surprised
adequacy (SA) was initially proposed by Kim et al [19], which can be used as a
test adequacy tool for DL system testing. SA’s initial property is to describe
the surprising of testing data with respect to the training data, namely to
describe difference/similarity between testing and training data. In [19], two
kinds of SA are developed, one based on the probability density distribution,
the other based on distance similarity, and they were furthermore verified
useful in industrial applications [20]-[21]. While, through the review on SA-
related literature, two useful points can be obtained. One is that DSA is
specifically effective for classification applications. The other is that SA
was verified useful to capture data’s behaviors in DL testing, namely models’
neuron activation behaviors instead of plain data distribution, for example,
in [20] SA was validated to correlate with the correctness answers in NLP
study. With consideration of these two points, this paper is inspired to
leverage DSA for corner case detection in classification problems, since SA’s
capability on data description can be used not only on normal data but also on
incorrect/erroneous data (corner case data). Moreover, to enhance DSA’s
ability on capturing behaviors of corner case data, this paper also proposes
three kinds of modification on the DSA definition. Furthermore, based on the
proposed DSA and the corner case detection method, experiments based on a
benchmark classification data (MNIST) [22] and industrial application data are
implemented and studied in this paper. Consequently, the novelty and
contributions of this paper can be summarized as follows:

  * •

SA is applied for data description, especially for corner case data
description. Instead of describe data via the general plain data distribution
directly, SA adapts neurons activation behaviors with respect to a DL model to
describe data’s characteristics, which can capture profound data behaviors
responding to DL models in quality testing;

  * •

Three kinds of modification on DSA definitions are developed. First, the
capability of DSA on data behaviors description is inherited. Then, according
to the idea that erroneous corner cases are tightly related to the classifier
boundary, so modification on the DSA definition are proposed;

  * •

Based on DSA, a novel corner case data detection method is proposed. Different
with most of corner case study, the proposed method can be utilized as a tool
in recognition of corner case data. In this way, it is possible to detect any
corner case data with no need to generate/learn all possible corner case data;

  * •

Experiments on MNIST data and industrial data validate the feasibility of
using DSA to describe corner case data behaviors, and that it is useful to use
DSA on corner case data detection. Moreover, the proposed DSA3 achieve a
relatively better performance than others from results analysis.

The rest of the paper is organized as follows. Section 2 briefly describes the
idea of surprise adequacy and its related calculation process. Section 3 gives
out three kinds of modification based on corner case definition and their
characteristics. Section 4 implements experiments on the benchmark MNIST data
and an industrial classification data, and analyze the performance of using
DSA on corner case data detection. Finally, Section 5 concludes the
contributions of this paper, and think about the future work.

##  II Surprise Adequacy

To evaluate the quality of testing data in AI assurance, a good way is to
study neurons behavior in terms of a given deep learning (DL) model.
Generally, the more the diversity of neuron behaviors, the better the quality
of testing data. There are various methods to describe neuron behaviors in DL
models. For example, neuron coverage was proposed in [18], and its activation
status was taken as the behavior of a testing data point responding to the DL
model. Similarly, values and signs of neuron outputs can both be used to
describe neurons’ activation behaviors in [23].

While, compared with those metrics reflecting independent behaviors of testing
and training sets, in [19] an interesting idea was proposed to describe the
difference between the testing set’s behaviors and that of the whole training
set. To realize this idea, the activation status of neurons in DL models are
still firstly denoted. Assuming a set of inputs $X=\\{x_{1},x_{2},...\\}$ and
a trained DL model M consisted of a set of neurons
$\boldsymbol{N}=\\{n_{1},n_{2},...\\}$. For a given testing data $x\in X$ and
an ordered (sub)set of neurons $N\subseteq\boldsymbol{N}$, the activation
behavior (namely _activation trace_) of $x$ on $N$ is expressed by the vector
of activation values and it denoted as

$\alpha_{N}(x)=[a_{1}(x),a_{2}(x),...,a_{|N|}(x)]^{T}$ (1)

where each element $a_{n}(x)$ corresponds to the activation value of $x$ with
respect to an individual neuron $n$ in $N$. Hence, the set of activation
traces for $X$ is denoted as $A_{N}(X)=\\{\alpha_{N}(x)~{}|~{}x\in X\\}$.

Then, $A_{N}(Tr)$ is calculated based on the training dataset $Tr$, which
records neurons’ activation behaviors on all samples in $Tr$. Subsequently,
the activation behavior of testing data $Te$ is also obtained as $A_{N}(Te)$.
Finally, combining $A_{N}(Tr)$ and $A_{N}(Te)$, surprise adequacy (SA) was
defined to describe the relative novelty of testing inputs with respect to the
training data. It is actually denoted as the quantitative similarity measure
between $A_{N}(Tr)$ and $A_{N}(Te)$ in the following form.

$SA=SimilarilryMeasure(A_{N}(Te),A_{N}(Tr))$ (2)

In [19], there are two kinds of similarity measurement proposed to formalize
SA. One is the likelihood-based SA (LSA), and the other one is distance-based
SA (DSA). These two SAs are validated feasible to capture the relative
surprise of testing data in DL systems, and describe testing data’s behaviors,
which implies SAs are useful to evaluate the quality of testing data.

##  III Corner cases detection based on SA

###  III-A Denotation of corner cases

As well as the execution of traditional software [24], a dangerous condition
in AI system testing is processing data of corner cases which generally cause
incorrect and unexpected behaviors. For example, when DL-based autonomous
driving system processes corner cases of rainy weather or strong reflection,
an incorrect decision may be made to lead to crash bringing the loss of life
and property. Therefore, detecting corner-case samples is important in AI
testing. According to the above description of corner case, we can define the
corner case set as the following form

$Corner\ case\ set:\\{x|\ DL(x+pertubation)\neq label(x)\\}$ (3)

where, $x$ is denoted as a sample in corner case; its true label is denoted as
$label(x)$; $DL(*)$ is the output class based on a given DL model. Through
this definition, we see when a small perturbation is added into a corner-case
data $x$, where $perturbation$ is a small value such that
$0<{|perturbation|}\leq\varepsilon$ for a constant $\varepsilon$ , the class
recognized by DL system will be different with its true label. In this way, a
corner case set can include data samples with both incorrect and unexpected
behaviors, e.g. boundary adversarial data and incorrectly classified data
(outliers), as shown in Fig. 1

Figure 1: Diagram of corner case. Two class of data are colored as blue and
red. Data of corner case is colored as green, which includes incorrectly
classified data as well as data close to classification boundary which is
sensitive to cause unexpected recognition.

###  III-B Distance-based SA and its modification

Considering SA can capture the behavior of testing data, therefore we may
think about the possibility of using SA to describe behaviors of data in
corner case. Here, we mainly consider the distance-based SA (DSA), since it
describes the relation of testing data respect to the classification boundary.
According to the DSA definition in [19], firstly assuming a DL model M
consisting of a set of neurons $N$, it is learned based on a training dataset
$Tr$ with $C$ classes. According to the denotation of SA, the activation
traces of Tr can be calculated as $A_{N}(Tr)$ based on the trained M. Then,
for a new testing input $x$ in the class $c_{x}\in C$, its nearest neighbor in
$c_{x}$ is denoted as $x_{a}$. Finally, the surprise of $x$ respect to class
$c_{x}$ can be calculated as the distance between their activation trace
$\alpha_{N}(*)$, as the following form

$x_{a}=\underset{x_{i}\in X~{}\mbox{\small
s.t.}~{}c_{x_{i}}=c_{x}}{\operatorname{argmin}}\|\alpha_{N}(x)-\alpha_{N}(x_{i})\|$
(4) $dist_{a}=\|\alpha_{N}(x)-\alpha_{N}(x_{a})\|$ (5)

where, $\|*\|$ is the Euclidean distance. Subsequently, taking $x_{a}$ as the
reference point and finding its nearest neighbor $x_{b}$ in a class different
with $c_{x}$, then the surprise between $x_{a}$ and $x_{b}$ is also calculated

$x_{b}=\underset{x_{i}\in X~{}\mbox{\small s.t.}~{}c_{x_{i}}\neq
c_{x}}{\operatorname{argmin}}\|\alpha_{N}(x_{a})-\alpha_{N}(x_{i})\|$ (6)
$dist_{b}=\|\alpha_{N}(x_{a})-\alpha_{N}(x_{b})\|$ (7)

Combining the above definitions, the SA of testing data $x$ to the training
data is consequently defined as the ratio between $dist_{a}$ and $dist_{b}$,
as below

$DSA(x)=\frac{dist_{a}}{dist_{b}}$ (8)

According this definition, it is seen if the numerator is larger and the
denominator is smaller, the value of DSA will be larger, implying the testing
data $x$ is surprise to data of class $c_{x}$ in the training set, as shown in
Fig. 2(a). Therefore, we can see that DSA shows a way of describing data’s
surprise to training data by distance, and it is also useful to describe
data’s activation behaviors respect to the given DL model. While, the above
definition of DSA is simple and has some drawbacks to deal with independent
data points in extreme cases. Moreover, to evaluate its capability on
detecting data of corner case, several additional modifications are also
proposed in this paper to formalize DSA for corner case detection.

(1) Modification 1: novelty calculation via testing data itself

In the original DSA definition, we see that the denominator is actually to
calculate the surprise of $x_{a}$ (the nearest neighbor of the testing data
$x$ in the same class) respect to data of other different classes. Therefore,
DSA can be regarded as the comparison between $x$’s novelty in its belonging
class and its class novelty to other classes. This definition may be useful to
describe the surprise of a testing data to the whole training data. However,
to evaluate if a data sample belongs to corner case, its own novelty respect
to all classes seems more important. Hence, we modify the original DSA by just
adjusting the definition of $dist_{b}$ as the following form

$x_{b}=\underset{x_{i}\in X~{}\mbox{\small s.t.}~{}c_{x_{i}}\neq
c_{x}}{\operatorname{argmin}}\|\alpha_{N}(x)-\alpha_{N}(x_{i})\|$ (9)
$dist_{b}=\|\alpha_{N}(x)-\alpha_{N}(x_{b})\|$ (10)

The calculation of DSA keeps unchanged. This modification makes DSA consider
surprises of testing data $x$ to all classes independently, which may be
helpful to reflect the behaviors of data in corner case, as shown in Fig.
2(b).

(2) Modification 2: novelty calculation via global data descriptors

Compared with the original DSA, the first modification considers surprise of
testing data $x$ to data of all classes. While, these two DSA definitions may
have a common shortage on processing pair-wise rare data points, especially on
describing behaviors of corner case. For example, assuming the testing data
$x$ is the corner-case data, there happens to exist a neighbor $x_{a}$ very
close to $x$, namely $x\approx x_{a}$, since the above DSA are calculated
based on the distance of point-to-point, this may lead DSA to be a low value
due to $dist_{a}\rightarrow 0$. In this case, the novelty of $x$ respect to
training data is regarded as small based on SA definition. However, this
result causes the inconsistence of behavior novelty for data in corner case.
Furthermore, this result will be meaningless for corner case study as well.
Therefore, a possible modification on the basis of the first modification is
to select a more general class descriptor for surprise calculation. In [25],
the class centers were commonly used as feasible global descriptors. Hence, we
can replace the nearest neighbor with the class center in surprise
calculation, and the new equations can be written as below

$dist_{a}=\|\alpha_{N}(x)-\alpha_{N}(m_{a})\|$ (11)
$dist_{b}=\|\alpha_{N}(x)-\alpha_{N}(m_{b})\|$ (12)

where $m_{a}$ is the center of the class which $x$ belongs to; $m_{b}$ is the
center of a different class $c_{b}\ (c_{b}\neq c_{x})$ which the closest point
$x_{b}$ to $x$ belongs to. For each $s\in\\{a,b\\}$, the center point $m_{s}$
is calculated by

$m_{s}=\frac{1}{|X_{s}|}\sum_{x_{i}\in
X_{s}}{x_{i}},~{}\mbox{where}~{}X_{s}=\\{x_{i}\in X|\ c_{x_{i}}=c_{s}\\}.$
(13)

Then, the newly modified DSA can be calculated as $dist_{a}/dist_{b}$. Its
diagram is shown in Fig. 2(c).

(3) Modification 3: novelty calculation via local data descriptors

In the above modification of DSA, the characteristic of a class is simply
described by class center. However, corner-case data usually have some obvious
characteristics of closing to boundary or outlier location. Class centers are
good global descriptors of classes, but not effective to capture local
characteristics, e.g. in reflecting the characteristics of corner case data.
In this paper, to modify DSA for corner case data detection, we propose to
replace the global descriptors with local descriptors, e.g. the center of a
nearest neighborhood shown in Fig. 2(d). The DSA of testing data $x$ can be
still calculated via (11), (12) and (8), while the calculation of the center
$m_{s,\delta}$ for each $s\in\\{a,b\\}$ should be modified as

$m_{s,\delta}=\frac{1}{|X_{s,\delta}|}\sum_{x_{i}\in X_{s,\delta}}{x_{i}}$
(14)

where $X_{s,\delta}$ ($\subseteq X_{s}$) represents the neighborhood set of
inputs included in the hypersphere whose center is $x_{s}$ and radious is
$\delta$, namely several nearest neighbors of $x_{s}$. Here, two methods
determining the neighborhood set are provided. One kind of neighborhood is
denoted by a given size of Eculidean round (i.e. the same radius $\delta$ for
all inputs is used), as follows

$X_{s,\delta}=\\{x_{i}\in X_{s}~{}|~{}\|x_{i}-x_{s}\|<\delta\\}$ (15)

where, $x_{a}$ ($s=a$) and $x_{b}$ ($s=b$) are inputs explained in the
previous Modification 2 and $X_{s}$ is the set defined in (13). An alternative
definition of the nearest neighborhood is determined by $k$-nearest points
[26]. In this case, the radious $\delta$ is given for each $x_{s}$ such that
$X_{s}$ includes only $k$ nearest neighbors of $x_{s}$ (i.e.
$|X_{s,\delta}|=k$) and therefore

$X_{s,\delta}=\\{x_{i_{1}},x_{i_{2}},...,x_{i_{k}}\\}$ (16)

where, $x_{i_{j}}\in X_{s}$ is the $j_{th}$ nearest point to $x_{s}$.

Figure 2: Diagram of four types of DSA. (a) the original DSA. (b) modification
1. (c) modification 2. (d) modification 3.

###  III-C Corner case data coverage

Based on the above definitions of DSA, including the original one and
modifications, the analysis on studying DSA’s behaviors responding to corner
case data can be implemented. To quantitatively evaluate the capability of DSA
in corner case data detection, here we propose a metric as corner-case data
coverage as the following form

$cov(v_{th})=\frac{card(\\{d|\ d\in CD,\ DSA(d)>v_{th}\\})}{|CD|}*100\%$ (17)

where, $v_{th}$ is a given threshold; $CD$ represents the dataset of corner
case data; $|CD|$ is the cardinality; $\\{d\\}$ represents the set consisting
of all detected corner case data which have DSA values larger than the given
threshold. While, considering not all of data in corner case are detectable,
here we only take the detectable corner-case data into account, namely those
data wrongly recognized by DL. In this way, the proposed corner case data
coverage is actually to evaluate the percentage of erroneous behaviors of DL
systems.

##  IV Experiments and Evaluation

###  IV-A Dataset and the DL system description

To study behaviors of the proposed DSA on corner case data detection, in this
paper we take the MNIST data [22] as a reference in experiments and analysis.
MNIST is a widely-used dataset in machine learning research, which contains
ten classes of images, and dividing as 60,000/10,000 training/testing samples.
For recognizing different numbers in MNIST, we adopt a common-used five layer
Convolutional Neural Network (CNN), including convolutional, max-pooling,
dropout and full-connection layers. On the other hand, experiments in this
paper are implemented on a machine equipped with Intel i7-9750 CPU, 32GB RAM,
NVIDIA GeForce RTX 2060, running in Windows 10 Pro 64-bit OS. Code of this
paper is released [27], which is developed from SADL [28] on Keras, and
contains the proposed three kinds of modification of DSA are realized.

###  IV-B Qualitative analysis on corner case data detection

Firstly, through training the given CNN architecture above on MNIST data, we
can achieve $99.17\%$ accuracy on the original testing set. Then, based on the
proposed DSA definition and the trained DL model, we can calculate the values
of DSA on each testing sample, and subsequently study DSA’s behaviors
responding to corner case data detection. By sorting all testing samples as
the DSA descending trend, we can first study the influence of DSA variation on
DL model’s performance. In order to do comparative analysis, four kinds of DSA
are calculated on outputs of three selected layers, such as the convolutional
layer 1 (Layer1), convolutional layer 2 (Layer2) and the final full-connection
layer (Layer3) of the given CNN. Results are shown in Fig. 3.

Figure 3: Testing accuracy variation as the descending DSA. (a) Layer1. (b)
Layer2. (c) Layer3.

Fig. 3 shows the curves of accuracy vs. DSA variation. By assuming the first
$k_{n}$ samples are taken from the sorted dataset for testing, then an
accuracy value is calculated. As the value of $k_{n}$ ascends from a small
value to 10,000, the curve of accuracy can be plotted. Here, the beginning
point is $k_{n}=100$, where we can see the accuracy is low due to a certain
percentage of erroneous data (corner case data). This phenomenon can be found
in all curves during the beginning period where testing samples have large DSA
values. This implies that large values of DSA can capture corner case data to
some extent. Then, the ending point is $k_{n}=10,000$ meaning the whole
testing dataset, all curves reach the final testing accuracy $99.17\%$.
Moreover, to vividly analyze this foundation, we can further draw out images
of these samples with the largest DSA values, as shown in Fig. 4.

Figure 4: Images with the largest DSA values. (a) DSA0. (b) DSA1. (c) DSA2.
(d) DSA3.

In Fig. 4, images are drawn separately based on four DSA definitions, such as
DSA0- DSA3 representing the original DSA and other three modifications
respectively, and DSA3 adopts the second definition in (16) which is more
convenient in computation (e.g. $k$=20). From these images, we can see there
are indeed many corner case data, which are caused by various factors, like
unusual writing, thick chirography, large slant, structure disequilibrium and
so on. Some of samples can be unrecognized by not only AI but also human
beings, however they can have large value of DSA. Therefore, it illustrates
that DSA might be useful to detect corner case data.

###  IV-C Quantitative analysis of DSA behaviors

While, besides the conclusion that DSA can capture activation behaviors of
data from Fig. 3, there discovers the other interesting phenomenon, namely
DSA3 has relatively lower accuracy in all these three layers, especially at
the beginning period of curves. This implies DSA3 can capture more wrongly
recognized images to lower the accuracy when DSA values are selected as a
large value, which is also to say DSA3 has a better capability on corner case
data detection. To verify this conjecture and to quantitatively study
performance of different DSAs on corner case data detection, the proposed
corner-case data coverage is firstly calculated. Results of corner-case data
coverage based on four DSAs are shown in Fig. 5.

Figure 5: Corner case data coverage of testing data sorted as the descending
DSA. (a) Layer1. (b) Layer2. (c) Layer3.

In Fig. 5, the testing data is also sorted by the descending direction of DSA
values. For a given DSA threshold $v_{th}$, the number of data satisfying
$DSA(x)>v_{th}$ is counted. Meanwhile, under this condition the number of
wrongly classified data is also counted for the calculation of corner-case
data coverage. It is seen from these results, that DSA3 has the largest
corner-case data coverage on the analysis of all three given layers. On the
other hand, we can ignore the selection of threshold as a concert value, and
utilize the measurement of AUC-ROC [29] on corner case data detection to
evaluate different DSAs’ performance. By assuming to utilize DSA values
directly for distinguish normal data and corner case data, the AUC value of
this binary classification process can be calculated. AUC-ROC values of four
DSAs in three given layers are presented in Table I.

TABLE I: AUC-ROC of corner case data detection based on DSA | Layer1 | Layer2 | Layer3
---|---|---|---
DSA0 | 0.9422 | 0.9418 | 0.9759
DSA1 | 0.9657 | 0.9722 | 0.9833
DSA2 | 0.9114 | 0.9662 | 0.9875
DSA3 | 0.9814 | 0.9884 | 0.9966

From result of Table I, there are three conclusions can be obtained. First,
for the constructed CNN model on MNIST, it indeed achieves good performances
by using DSA to guide corner case data detection since all AUC values are
larger than 0.9. Second, it is found that the deeper the layer is, the better
DSA performs on corner case data detection, seeing
$AUC(Layer3)>AUC(Layer2)>AUC(Layer1)$. Third, DSA3 can have a relatively
better performance when compared to the other DSA definitions. Therefore,
based on these results, we can consider to choose the proposed DSA3 for corner
case data detection in the further study.

###  IV-D Adversarial testing analysis

To further analyze the performance of DSA, as an important AI testing
technique adversarial testing can also be applied here. Generally, based on
the original testing data, small perturbations imperceptible to humans are
added to generate adversarial samples which will lead the DL model to make
wrong behaviors (outputs). There are many different attack strategies applied
to generate adversarial testing set in literature. In this paper, we take four
widely-studied adversarial attacks to generate new testing set on MNIST, such
as Fast Gradient Sign Method (FGSM) [14], Basic Iterative Method (BIM) [30],
Carlini $\&$ Wagner $(C\&W)$ [31], and Projected Gradient Descent (PGD) [32].
Then, based on these four adversarial testing datasets, the proposed DSAs are
applied to detect the corner case data, namely misclassified data. Similar as
the above analysis, we do not care about the exact selection of DSA threshold,
and just utilize the AUC-ROC to evaluate the performance of using DSA on
corner case data detection. The results of ROC curves of four DSAs on
different adversarial testing sets are depicted in Fig. 6.

Figure 6: ROC of using DSAs on corner case data detection on four adversarial
testing data. (a) FGSM. (b) BIM. (c) CW. (d) PGD.

Fig. 6 shows the ROC curves on corner case data detection, where X-axis
represents the value of FPR, and Y-axis represents the value of TPR
respectively. In details [33], TPR means true positive rate, also called as
sensitivity or recall, which can be used to reflect the test’s ability on
correctly detecting corner case data here; FPR means false positive ratio,
also known as false alarm ratio, which is the probability of falsely rejecting
the null hypothesis for a particular test. Their calculation are presented as
below

$TPR=\frac{TP}{TP+FN}$ (18) $FPR=\frac{FP}{FP+TN}$ (19)

where, TP (True Positive), FP (False Positive), FN (False Negative), TN (True
Negative) are four events in confusion matrix of corner case data detection.
Moreover, based on these ROC curves, we can quantitatively calculate the
values of AUC as a KPI metric, as Table II

TABLE II: AUC-ROC of DSA-based corner case data detection on adversarial data testing. | FGSM | BIM | CW | PGD
---|---|---|---|---
DSA0 | 0.8520 | 0.9510 | 0.8390 | 0.9593
DSA1 | 0.8801 | 0.9702 | 0.8931 | 0.9722
DSA2 | 0.8812 | 0.9659 | 0.8510 | 0.9696
DSA3 | 0.9226 | 0.9915 | 0.9666 | 0.9921

According to the results of Table II, the same foundation is obtained that
DSA3 performs relatively better than the other DSA definitions on all these
four adversarial testing sets. Therefore, we can make use of DSA3 in the
following study related to corner case data and AI quality assurance.

###  IV-E Real-world data analysis

To further study the feasibility and generalization ability of the proposed
method, we also propose to apply the proposed DSA method to detect corner case
data in real-world application. Here, the metal casting product image data is
taken for case study, which is applied for industrial quality inspection in
real world [34]. It has totally 6,633 training data images, and 715 testing
images. The objective based on this data is an identical binary classification
problem which mainly aims at automatically recognizing defective metal casting
products and normal ones. Therefore, a general CNN model consisting of 2
convolutional layers and 2 full-connection layers are constructed for this
issue. After modeling and training, the final testing accuracy can reach
$97.90\%$ on this casting data. Then, according the proposed method on DSA
calculation and corner case data detection, we can also implement related
experiments, such as calculating four kinds of DSA values on given model
layers, e.g. two convolutional layers and the output layer are considered here
and also named as Layer1, Layer2, Layer3, corner case data coverage analysis,
and performance analysis about using DSA on corner case data detection.
Results of related experiments are presented in Fig. 7.

Figure 7: Corner case data coverage variance as the descending DSA. (a)
Layer1. (b) Layer2. (c) Layer3.

In Fig. 7, the corner case data coverage is calculated and plotted as the
descending values of four given DSA. We can see the proposed DSA3 has
relatively larger corner case data coverage, namely implying better capability
on corner case capture.

Figure 8: ROC curves of using DSAs on corner case data detection. (a) Layer1.
(b) Layer2. (c) Layer3.

Fig. 8 shows the ROC curves of using DSA on corner case data detection. It is
also found that DSA3 has larger values of TPR under the same value of FPR than
other DSAs, the similar phenomenon is shown in all three given layers
analysis. For the further quantitative analysis, based on the results of ROC
curves, we also calculate the values of AUC-ROC, as presented in Table III.

TABLE III: AUC-ROC of DSA-based corner case data detection on casting defect data | Layer1 | Layer2 | Layer3
---|---|---|---
DSA0 | 0.7565 | 0.7717 | 0.7990
DSA1 | 0.7520 | 0.7612 | 0.8002
DSA2 | 0.7584 | 0.7815 | 0.9952
DSA3 | 0.9437 | 0.9587 | 0.9908

From results of Table III, we find the similar conclusion as that obtained on
MNIST data that the proposed DSA modifications can improve the capability of
the original DSA on describing corner case data’s behaviors, meanwhile the
DSA3 performs relatively better than others. Moreover, as the layer becomes
deeper, the calculated DSA has better performance on corner case data
detection, seeing that $AUC(Layer3)>AUC(Layer2)>AUC(Layer1)$.

##  V Conclusions

In this paper, a novel method based on DSA has been proposed for corner case
data detection which would be useful in AI quality assurance, such as safety
analysis. While, to improve the capability of the original DSA on capture
behaviors of corner case data, three DSA modifications have been also
developed with the consideration of boundary features, and validated useful in
corner case data detection. Then, based on the experiments on MNIST which has
been regarded as the benchmark classification application, it is seen that
large values of DSA indeed capture many images with abnormal characteristics
with respect to standard numbers, which also implies DSA’s capability on
capturing corner cases. Subsequently, through the quantitative analysis on
erroneous corner case data detection, the modified DSAs have been found to
have relatively better performance than the original one. Meanwhile, the
proposed DSA3 performs the best on MNIST to some extent. Moreover, four
adversarial testing datasets of MNIST have been also generated to further
verify the feasibility of using DSA on corner case data detection. On the
other hand, the industrial data based on casting defect application has also
been studied in experiments, and its results further verify the general
feasibility of using the proposed idea on classification problems. While,
besides the obtained conclusions above, more issues would be still
meaningfully developed in future work. For example, the proposed DSA
modifications in this paper are not perfect on capturing corner case data,
other possible strategies are meaningful to be considered. Moreover, based on
the detected corner case data, further study on AI quality assurance could be
implemented, for example the AI robustness analysis and so on.

## Acknowledgment

This research is based on results obtained from a project ’JPNP20006’,
commissioned by the New Energy and Industrial Technology Development
Organization (NEDO)

## References

  * [1] M. Borg, C. Englund, and B. Duran, “Traceability and Deep Learning-Safety-critical Systems with Traces Ending in Deep Neural Networks,” In Proc. of the Grand Challenges of Traceability: The Next Ten Years, 2017, pp. 48-49.
  * [2] S. Ellahham, N. Ellahham, and M. C. E. Simsekler, “Application of artificial intelligence in the health care safety context: opportunities and challenges,” American Journal of Medical Quality, vol. 35, no. 4, 2020, pp. 341-348.
  * [3] Z. Yuan, Y. Lu, Z. Wang, and Y. Xue, “Droid-sec: deep learning in android malware detection,” In Proceedings of the 2014 ACM conference on SIGCOMM, pp. 371-372, August 2014.
  * [4] S. Liu, L. Liu, J. Tang, B. Yu, Y. Wang, and W. Shi, “Edge computing for autonomous driving: Opportunities and challenges,” Proceedings of the IEEE, vol. 107, no. 8, 2019, pp. 1697-1716.
  * [5] D. Rice, “The Driverless Car and the Legal System: Hopes and Fears as the Courts, Regulatory Agencies, Waymo, Tesla, and Uber Deal with this Exciting and Terrifying New Technology,” Journal of Strategic Innovation and Sustainability, vol. 14, no. 1, 2019.
  * [6] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, “A survey of deep learning techniques for autonomous driving,” Journal of Field Robotics, vol. 37, no. 3, 2020, pp. 362-386.
  * [7] Y. Tian, K. Pei, S. Jana, and B. Ray, “Deeptest: Automated testing of deep-neural-network-driven autonomous cars,” In Proceedings of the 40th international conference on software engineering, pp. 303-314, May 2018.
  * [8] tesla-accident 2016. “Understanding the fatal Tesla accident on Autopilot and the NHTSA probe,” https://electrek.co/2016/07/01/
  * [9] google-accident 2016. “A Google self-driving car caused a crash for the first time,” http://www.theverge.com/2016/2/29/11134344/google-selfdriving-car-crash-report.
  * [10] W. Wu, H. Xu, S. Zhong, M. R. Lyu, and I. King, “Deep validation: Toward detecting real-world corner cases for deep neural networks,” In 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE, pp. 125-137, June 2019.
  * [11] J. A. Bolte, A. Bar, D. Lipinski, and T. Fingscheidt, “Towards corner case detection for autonomous driving,” In 2019 IEEE Intelligent Vehicles Symposium (IV), IEEE, pp. 438-445, June 2019.
  * [12] C. J. Clemente, F. Jaafar, and Y. Malik, “Is predicting software security bugs using deep learning better than the traditional machine learning algorithms? ,” In 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS), IEEE, pp. 95-102, July 2018.
  * [13] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
  * [14] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and Harnessing Adversarial Examples,” In Proceedings of the 3rd International Conference on Learning Representations, 2015. http://arxiv.org/abs/1412.6572
  * [15] S. M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
  * [16] L. Ma, F. Zhang, J. Sun, M. Xue, B. Li, F. Juefei-Xu, and Y. Wang, “Deepmutation: Mutation testing of deep learning systems,” In 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE), IEEE, pp. 100-111, October 2018.
  * [17] S. Nakajima, and T. Y. Chen, “Generating biased dataset for metamorphic testing of machine learning programs,” In IFIP International Conference on Testing Software and Systems Springer, Cham, pp. 56-64, October 2019.
  * [18] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” In proceedings of the 26th Symposium on Operating Systems Principles, pp. 1-18, October 2017.
  * [19] J. Kim, R. Feldt, and S. Yoo, “Guiding deep learning system testing using surprise adequacy,” In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), IEEE, pp. 1039-1049, May 2019.
  * [20] S. Kim, and S. Yoo, “Evaluating Surprise Adequacy for Question Answering,” In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pp. 197-202, June 2020.
  * [21] J. Kim, J. Ju, R. Feldt, and S. Yoo, “Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving,” 2020. arXiv preprint arXiv:2006.00894.
  * [22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. ”Gradient-based learning applied to document recognition.” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, November 1998.
  * [23] Y. Sun, X. Huang, D. Kroening, J. Sharp, M. Hill, and R. Ashmore, “Testing deep neural networks,” 2018. arXiv preprint http://arXiv:1803.04792.
  * [24] R. Banabic, “Techniques for identifying elusive corner-case bugs in systems software,” (No. THESIS). EPFL. 2015.
  * [25] T. Ouyang, W. Pedrycz, O. F. Reyes-Galaviz, and N. J. Pizzi, “Granular description of data structures: a two-phase design,” IEEE Transactions on Cybernetics, 2019, Doi: 10.1109/TCYB.2018.2887115
  * [26] Y. He, A. Kusiak, T. Ouyang, and W. Teng, “Data-driven modeling of truck engine exhaust valve failures: a case study,” Journal of Mechanical Science and Technology, vol. 31, no. 6, 2017, pp. 2747-2757.
  * [27] https://github.com/thouyang/ccd-dsa
  * [28] https://github.com/SemanticApplicationDesignLanguage/sadl
  * [29] S. Narkhede, “Understanding AUC-ROC Curve,” Towards Data Science, 2018, pp. 26.
  * [30] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” 2016. arXiv preprint http://arXiv:1607.02533.
  * [31] N. Carlini, and D. Wagner, “Towards evaluating the robustness of neural networks,” In 2017 ieee symposium on security and privacy (sp), IEEE, pp. 39-57, May 2017.
  * [32] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” 2017. arXiv preprint http://arXiv:1706.06083.
  * [33] D. M. Powers, “Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation,” 2020. arXiv preprint https://arXiv:2010.16061.
  * [34] F. Riaz, K. Kamal, T. Zafar, and R. Qayyum, “An inspection approach for casting defects detection using image segmentation,” In 2017 International Conference on Mechanical, System and Control Engineering (ICMSC), IEEE, pp. 101-105, May 2017.