# STARNet: Sensor Trustworthiness and Anomaly Recognition via Approximated
Likelihood Regret for Robust Edge Autonomy

Nastaran Darabi∗, Sina Tayebati∗, Sureshkumar S., Sathya Ravi, Theja
Tulabandhula, and Amit R. Trivedi ∗ Both authors contributed equally to this
work. Authors are with the University of Illinois Chicago (UIC), Chicago, IL,
Email<EMAIL_ADDRESS>
###### Abstract

Complex sensors such as LiDAR, RADAR, and event cameras have proliferated in
autonomous robotics to enhance perception and understanding of the
environment. Meanwhile, these sensors are also vulnerable to diverse failure
mechanisms that can intricately interact with their operation environment. In
parallel, the limited availability of training data on complex sensors also
affects the reliability of their deep learning-based prediction flow, where
their prediction models can fail to generalize to environments not adequately
captured in the training set. To address these reliability concerns, this
paper introduces STARNet, a Sensor Trustworthiness and Anomaly Recognition
Network designed to detect untrustworthy sensor streams that may arise from
sensor malfunctions and/or challenging environments. We specifically benchmark
STARNet on LiDAR and camera data. STARNet employs the concept of approximated
likelihood regret, a gradient-free framework tailored for low-complexity
hardware, especially those with only fixed-point precision capabilities.
Through extensive simulations, we demonstrate the efficacy of STARNet in
detecting untrustworthy sensor streams in unimodal and multimodal settings. In
particular, the network shows superior performance in addressing internal
sensor failures, such as cross-sensor interference and crosstalk. In diverse
test scenarios involving adverse weather and sensor malfunctions, we show that
STARNet enhances prediction accuracy by approximately 10% by filtering out
untrustworthy sensor streams. STARNet is publicly available at
_https://github.com/sinatayebati/STARNet_.

###### Index Terms:

OOD Detection; LiDAR point clouds; Multimodal inference.

##  I Introduction

The demand for improved perception and a deeper understanding of the
environment in autonomous robotics has led to an increased reliance on complex
sensors. These sensors, such as those capable of capturing data beyond the
visible spectrum, provide robots with a more detailed view of their
surroundings. For instance, LiDAR sensors offer precise depth perception and
spatial resolution, enabling detailed 3D mapping, which is valuable for object
detection, mapping, and localization in autonomous navigation. LiDAR is also
effective in various lighting conditions, including nighttime and overcast
situations, where cameras struggle [1]. Similarly, RADAR performs well in
adverse weather conditions, reliably detecting objects in fog, rain, or snow
and accurately measuring velocities. Consequently, there is a growing interest
in utilizing combinations of these complex sensors to enhance object
recognition, material differentiation, environmental monitoring, and related
applications for robust autonomy in diverse scenarios.

Meanwhile, acquiring sufficient training data for advanced sensors is
challenging due to several factors. For instance, with the evolution of LiDAR
technology, new sensor designs, such as optical phased array [2], yield data
with distinct characteristics. Keeping training data up-to-date with these
advancements is intricate. Similarly, reflections from stationary objects like
buildings, trees, or parked vehicles can influence RADAR sensor readings,
making target differentiation in cluttered environments a significant labeling
challenge. Additionally, active sensors require more diverse training data due
to the unique influences of various environments and lighting conditions on
return signals. These factors complicate developing exhaustive training
datasets that cover all potential scenarios a robot might face in its
operational life.

Figure 1: Motivation for STARNet: (a) Predictions for edge autonomy can be
impacted by a lack of reliability in sensing, learning, and computing. Various
failures may impact sensors, such as a beam missing in LiDAR. Hard-to-
generalize environments impact learning reliability. Limited computing
resources at the edge impact the reliability of computations such as those
demanding high-precision operations. (b) In this work, we present STARNet,
which feeds upon intermediate representations of the sensor streams, a
function of sensor reliability and environment, to detect when the prediction
failures are likely. STARNet is designed to operate under low precision, so
edge computing constraints less impact it.

Additionally, complex sensors are also susceptible to various failures. The
demand for compact, energy-efficient sensors necessitates innovative
fabrication techniques. As sensor complexity increases, the manufacturing
process must achieve higher precision, which also raises the risk of internal
component failures. For example, on-chip integration of edge-emitting lasers
with nanometer node transistor technologies can result in complex failure
mechanisms due to thermal overstress [3], electrostatic discharge [4],
fabrication variabilities [5], aging [6], etc. Likewise, in emerging multi-
pixel LiDAR and RADAR sensors [7], interference from nearby pixels can result
in cross-talk, thus corrupting their sensor readings.

With the growing integration of complex sensors and advanced deep learning in
next-generation robotics, ensuring the continuous reliability of sensor data
streams becomes crucial. To address the challenges of robust autonomy, we
introduce STARNet: a Sensor Trustworthiness and Anomaly Recognition Network.
STARNet detects untrustworthy sensor features, which may arise from sensor
malfunctions or difficult-to-generalize environments, to alert downstream
decision-making processes of potential inaccuracies.

Within STARNet, we employ the concept of approximated likelihood regret which
is tailored for low-complexity hardware, especially those with only fixed-
point precision capabilities, thus ensuring minimal impact from edge computing
constraints. STARNet utilizes the generative capabilities of variational auto-
encoders (VAE) to learn the distribution of trustworthy sensor data streams,
which jointly depend on sensor reliability and the generalizability of the
environment based on the training set. The likelihood of a sensor stream is
computed using pre-trained VAE $L_{\text{VAE}}$ and then again by optimizing
the VAE against the input sample,i.e., $L_{\text{OPT}}$ using the proposed
gradient-free processing. The difference in likelihoods,
$L_{\text{OPT}}-L_{\text{VAE}}$, termed likelihood regret, effectively
differentiates between trustworthy and untrustworthy streams: trustworthy
streams display lower likelihood regret as they align closely with the learned
distribution. In contrast, untrustworthy streams exhibit a pronounced regret.
By continuously monitoring and filtering out untrustworthy sensor streams, we
demonstrate that bSTARNet improves the prediction accuracy by $\sim$10% across
various test cases on adverse weather and sensor malfunctioning.

##  II Background

###  II-A Sensor Failures

In autonomous navigation systems, sensor reliability directly influences the
efficacy of deep learning-based decision-making. While complex sensors are
becoming prevalent in autonomous robotics, they complicate reliability
verification due to their intricate failure mechanisms. For instance, LiDAR
sensors can experience ghost readings from multi-path interference, reduced
accuracy during adverse weather like fog, or even false negatives due to the
absorption of signals by raindrops [8]. Infrared sensors might be impacted by
ambient temperature fluctuations or direct exposure to bright sources, leading
to noisy or even incorrect readings [9]. The emerging event-based cameras are
impacted by rapid scene changes, causing a temporal information overload [10].
Modern solid-state RADARs can suffer from false reflections or clutter from
non-moving objects [11]. Acoustic sensors, used for subterranean or underwater
navigation, can face signal attenuation or reflection complications due to
environmental inconsistencies [12]. MEMS-based inertial sensors can drift over
time or get affected by sudden temperature shifts [13]. While precise modeling
of these sensor faults is challenging, especially due to their complex
interaction with the environment and autonomy task, even minor inaccuracies in
the input data can lead to disproportionately large errors in the output due
to the non-linearity of deep learning models.

###  II-B Out-of-Distribution Detection (OOD)

In STARNet, we utilize out-of-distribution (OOD) detection techniques to
identify sensor streams that significantly deviate from the training data
distribution. Various OOD detection methods exist. For instance, WarningNet
[14] detects sensor anomalies due to alterations in image input patterns via
deep learning, while the PAAD network [15] proactively signals robots about
anomalies in unpredictable environments. Generative models, such as
Variational Autoencoders (VAEs) [16] and Generative Adversarial Networks
(GANs) [17], are employed for OOD detection, using VAEs’ reconstruction errors
and GANs’ discriminator scores as OOD indicators. Recently, there has been a
surge of self-supervised learning methods [18, 19] for OOD detection as well.

However, many of these established methods, including Softmax probability
scores [20] and feature representations from the network’s last layer [21],
often fail to detect seemingly straightforward OOD scenarios, such as
discriminating the distribution of CIFAR-10 dataset images from MNIST images.
A key drawback of these methods is the potential overlap in the feature space
of in-distribution and OOD data. If an OOD sample is proximate to the decision
boundary in this space, it may be incorrectly identified as an in-distribution
sample. Unlike earlier approaches, the likelihood regret (LR) measure [22] has
shown a much-improved ability to detect OOD data by training a distribution
learner (e.g., VAE) on incoming sample to ascertain the discrepancy between
the predicted likelihood from the primary model and that from a model
retrained on this new sample. The LR values for OOD samples are typically
substantially different than those for in-distribution samples, aiding
discrimination. However, the computational cost of LR–owing to the need for
retraining on every sample–constrains its use in real-time edge devices. We
address this using gradient-free computation techniques, discussed next.

Figure 2: STARNet on Multimodal Data Streams: We demonstrate STARNet using
multimodal sensor streams from cameras and LiDAR point clouds. However,
STARNet’s framework is versatile and can be adapted to other multimodal
scenarios. In the current configuration, STARNet feeds upon sensor feature
representations generated for the primary task, specifically from PointNet and
PointFusion. A VAE ingests these features to learn their typical distribution.
During inference, STARNet calculates the likelihood regret through gradient-
free optimization to identify the discrepancy between the sensed feature and
learned feature distribution to alert the primary prediction mechanism of
potential inaccuracies when discrepancies are excessive.

###  II-C Gradient-Free Optimization

In STARNet, we use gradient-free optimization suitable for fixed-point
precision hardware. Traditional gradient-based methods rely on floating-point
precision hardware to accurately represent a broad spectrum of gradient
values. This increases energy and footprint due to extra circuitry for the
mantissa, exponent processing, and error correction. On the other hand, fixed-
point precision hardware, which uses a simpler arithmetic logic unit, is more
common in edge devices because deep learning inference often works well with
fixed-point computation. Therefore, by tailoring STARNet’s optimization
problems to be gradient-free, we present a more resource-efficient and widely
applicable framework.

Among the prominent gradient-free optimization methods, ZO-SGD [23] and ZO-SCD
[24] stand out for optimizing unconstrained stochastic optimization with
convergence rates of $O(\sqrt{d}/\sqrt{T})$, where $T$ represents the
iteration count. However, their convergence efficiency is hampered by the
variable dimension $d$. The ZO stochastic mirror descent method (ZO-SMD) was
introduced to mitigate this, establishing a tighter bound with dimension-
dependent factors [25]. Additionally, variance reduction techniques have
bolstered ZO-SGD and ZO-SCD, leading to stochastic variance-reduced algorithms
that boast enhanced convergence rates and iteration complexities [26]. Recent
ZO optimization algorithms, such as ZO proximal SGD (ZO-ProxSGD) [27], ZO via
conditional gradient (ZO-CG) [28], and online alternating direction method for
multipliers (ZO-ADMM) [29], emphasize gradient-free constrained optimization.
Another significant method in this domain is Simultaneous Perturbation
Stochastic Approximation (SPSA), which efficiently estimates gradients by
perturbing all dimensions simultaneously, allowing for scalability [30].

##  III Framework of STARNet

Fig. 2 presents the overview of STARNet. STARNet operates on the intermediate
sensor stream representation generated for downstream tasks, such as object
detection. These representations are fed to a VAE to learn their distribution
from the training set. Notably, the generated distributions are a function of
both the sensor reliability and the operating environment. By leveraging the
same representations for the downstream task, STARNet presents minimal
overheads. The reliability of an incoming sensor stream is verified by
computing its likelihood regret (LR) using the proposed gradient approximation
methods to determine if the sensor features emanate from the learned
distribution or are out-of-distribution, in which case, the downstream
controller is alerted for possible misprediction.

We characterize STARNet in two settings: LiDAR-only processing and
LiDAR+Camera processing. For the LiDAR-only case, the PointNet architecture
[31] is employed. Instead of relying on data representations like volumetric
grids or multiple 2D projections, PointNet processes raw point coordinates
directly, ensuring invariance under permutations of the input set. The network
structure in PointNet employs a combination of shared multi-layer perceptrons
(MLPs) and max-pooling layers to capture local features of individual points
and global contextual information of the entire point set. For LiDAR+camera
processing, PointFusion [32] is employed for feature extraction and fusion. By
aligning and integrating these heterogeneous data types, PointFusion exploits
the spatial depth information from point clouds and the rich texture and color
information from images. Even though we present our results on these two
specific test cases, the framework of STARNet can be generalized to other
extractors.

Figure 3: Gradient-free Optimization: A comparative evaluation of three
gradient-free optimization methods—Zero Order Sign, Zero Order Stochastic
Gradient Descent, and Simultaneous Perturbation Stochastic Approximation
(SPSA)—under two conditions: (a) heavy snow and (b) light snow. The graph
shows noticeable fluctuations in the performance of Zero Order Sign and Zero
Order Stochastic Gradient Descent as the number of iterations increases. At
the same time, SPSA is highlighted for its enhanced robustness and stability.

Figure 4: Feature Extraction Analysis: (a) High-level architecture of the
PointNet version used for feature extraction, detailing its three main
components: point level, feature level, and encoder level. (b) Comparative
analysis of AUC values derived from features extracted at each level,
highlighting the superior performance of encoder features under heavy and
light snow conditions.  Figure 5: Visualization of Corruption sets: Our
multimodal test-set contains the clean KITTI dataset and four different
failure cases on both camera images and bird-view LiDAR point clouds.

###  III-A Unsupervised Learning of Sensor Feature Distribution

We utilize a variational autoencoder (VAE) to learn the distribution of
extracted sensor features. This VAE models the observed variable $x$ using a
latent random variable $z$. The generative model is represented as:

$p(x)=\int p(x|z)p(z)dz$ (1)

For a computationally efficient training of VAE, we focus on maximizing the
evidence lower bound (ELBO), given by:

$\log(p_{\theta}(x))\geq\mathbb{E}_{q_{\phi}(z|x)}\left[\log
p_{\theta}(x|z)\right]-D_{KL}\left[q_{\phi}(z|x)||p(z)\right]$ (2)

Here, $q_{\phi}(z|x)$ is the variational approximation to the true posterior
$p_{\theta}(z|x)$, with global parameters $\phi$ (encoder) and $\theta$
(decoder). The Kullback-Leibler divergence, $D_{KL}$, measures the difference
between the prior and the approximating distribution. The main goal is to
train the VAE to maximize the ELBO across training data, ensuring the VAE
captures the data’s inherent distribution and offers a useful representation
in the latent space.

###  III-B Likelihood Regret under Gradient Approximation

LR captures the discrepancy between an input’s likelihood from its optimized
posterior distribution and the likelihood predicted by the VAE, defined as
$\text{LR}=L_{\text{OPT}}-L_{\text{VAE}}$, where $L_{\text{OPT}}$ is the
empirical dataset (or sample) optimum and $L_{\text{VAE}}$ is the optimum
corresponding to the input example. In-distribution samples should have lower
LR values than out-of-distribution samples. We explored gradient approximation
techniques for the computations of LR, differentiating it from the original
method in [22]. We specifically investigated several gradient approximation
methods, including zero-order optimization with stochastic gradient descent
(ZO_SGD) [33, 34], zero-order optimization with sign [35, 34], and
simultaneous perturbation stochastic approximation (SPSA) [30].

Fig. 3 shows the comparison among methods using AUC (Area Under the ROC, i.e.,
Receiver Operating Characteristic curve) as the evaluation metric, a
conventional metric for assessing the performance of OOD classification.
Compared to other methods, SPSA shows improved performance on a more
challenging light snow case and also consistently benefits with more
iterations of gradient-free model updates, making it our preferred choice.
Additionally, the method is also lightweight because it only requires
evaluating the system at two perturbed points, regardless of the
dimensionality of the problem. Also, considering the feature extraction flow
of PointNet in Fig. 4(a), we also analyzed the optimal abstraction level that
results in the best performance. PointNet comprises three components: point,
feature, and encoder levels. Upon extracting features from each component, our
analysis showed that encoder features yielded the highest AUC values, as
presented in Fig. 4(b). These findings are utilized in our simulation results
discussed next.

##  IV Simulation Results

###  IV-A Benchmark Testsets

To characterize the efficacy of STARNet, we developed benchmark testing
datasets featuring eight distinct corruption types, each presented in two
different severity levels (Heavy and moderate). These datasets are divided
into the following categories: (i) Natural Corruptions: Environmental factors
like fog, rain, and snow, (ii) External Disruptions: Issues such as motion
blur and beam missing (absence of laser beams), and (iii) Internal Sensor
Failures: Sensor-specific problems including crosstalk, incomplete echo
responses, and cross-sensor interference. Though our approach utilizes LiDAR-
Camera fusion, all corruption scenarios affected both data types. However, the
internal sensor failures pertained solely to the LiDAR sensor. We evaluated
our method using the KITTI-C dataset, which provides corruption sets for LiDAR
and camera data [36]. Figure 5 showcases dataset samples.

Figure 6: Unimodal results: Performance evaluation of STARNet on LiDAR point
clouds for eight different failure cases. The figure highlights the superior
performance of our method in addressing internal sensor failures, specifically
cross-sensor interference and crosstalk. Notably, the AUC values for heavy
crosstalk and cross-sensor interference are 0.9658 and 0.9938, respectively.

We evaluated STARNet’s efficacy in unimodal and multimodal settings
considering these corruptions. In the unimodal setting, only the LiDAR point
cloud was processed. The point cloud was combined with its corresponding image
in the multimodal setting. In the following, we discuss our simulation results
for both cases.

###  IV-B Results for LiDAR-only Processing

We examined eight failure scenarios for LiDAR point clouds, each presented in
heavy and moderate intensities. As shown in Fig. 6, our method performs well,
especially when dealing with internal sensor issues like cross-sensor
interference and crosstalk. Specifically, our method achieved an AUC value of
0.9658 in cases with intense crosstalk. For cross-sensor interference, the AUC
value was 0.9938.

Table I provides a detailed comparison of our method against various Out-of-
Distribution (OOD) detection techniques, including the original Likelihood
Regret[22], Log-Likelihood, Likelihood Ratio[37], and the Input Complexity
method[38]. The results in the table show that our approach, which uses an
approximate gradient-free likelihood regret with SPSA, is almost as effective
as the original method that relies on gradient descent. Additionally, these
methods perform better than other techniques, such as the likelihood ratio and
input complexity.

TABLE I: Comparison Results for Unimodal case (Metric: AUC)

| Ours (LR+SPSA) | Likelihood Regret[22] | Likelihood | Likelihood Ratio[37] | Input Complexity[38]
---|---|---|---|---|---
Fog | 0.7786 | 0.7804 | 0.4294 | 0.7612 | 0.4009
Snow | 0.8363 | 0.8077 | 0.3613 | 0.7884 | 0.3587
Rain | 0.9058 | 0.9138 | 0.4248 | 0.9216 | 0.3910
Motion Blur | 0.8174 | 0.7839 | 0.3972 | 0.7407 | 0.3914
Beam Missing | 0.7807 | 0.7421 | 0.4551 | 0.7531 | 0.4407
Incomplete Echo | 0.7385 | 0.7384 | 0.3839 | 0.7079 | 0.3635
Cross-sensor | 0.9938 | 0.9939 | 0.4180 | 0.8743 | 0.3734
Crosstalk | 0.9658 | 0.969 | 0.3087 | 0.9235 | 0.2780

The pre-trained model is only on the clean KITTI point cloud. Thus, the
results are obtained under unsupervised learning.

###  IV-C Results for Camera + LiDAR Processing

In the context of multimodal scenarios, we undertook an analysis of four
distinct failure situations concerning the fusion of LiDAR point clouds and
images. Two levels of intensity characterized each of these scenarios. As
illustrated in Fig. 7, our method’s performance closely aligns with the
original LR’s. A comprehensive summary of our findings is presented in Table
II. The table incorporates accuracy values based on the R2 score. This metric
underscores the notion that, while there might be minor discrepancies in OOD
detection, the overall task can still be executed with high precision. Our
results indicate that integrating images and LiDAR data enhances the AUC
values. For instance, in the presence of snow, the AUC increases by 0.0801
under heavy snow. Similarly, AUC increases by 0.0939 under moderate snow
conditions.

Figure 7: Multimodal results: Performance evaluation of STARNet on LiDAR point
clouds and images for four different failure cases.  TABLE II: Fusion Results
(Metric: AUC)

| LR+SPSA | LR | Log-LL | ACC
---|---|---|---|---
H Fog | 0.9548 | 0.9502 | 0.2860 | 0.9663
M Fog | 0.9479 | 0.922 | 0.2846 | 0.9667
H Snow | 0.9164 | 0.905 | 0.3283 | 0.9756
M Snow | 0.8707 | 0.8674 | 0.3974 | 0.97
H Rain | 0.8745 | 0.8547 | 0.351 | 0.9798
M Rain | 0.8590 | 0.8352 | 0.4042 | 0.9799
H Motion Blur | 0.8902 | 0.8529 | 0.6058 | 0.9750
M Motion Blur | 0.8428 | 0.7829 | 0.6238 | 0.9763

Accuracy (ACC) is reported as the R2 score.

The pre-trained model is on the Clean KITTI point cloud.

Figure 8: Object Detection Accuracy for KITTI dataset: We employed a VAE to
process LiDAR point clouds and corresponding labels of various objects,
including cars, pedestrians, and cyclists. The VAE then outputs bounding boxes
for these objects. The network was tested on challenging conditions, including
heavy, medium, and low snow and other corruption cases.

##  V Conclusion

Although deep learning models can be enhanced for robustness by augmenting
training data with noisy sets and diverse operating environments, solely
relying on augmentation is insufficient since it is hard to anticipate and
gather enough training data to capture the complexity and diversity of
environments encountered in practical scenarios. Comparatively, STARNet
enables risk-conscious decision-making by continuously monitoring the
trustworthiness of incoming sensor streams. In Fig. 8, we summarize the net
significance of STARNet in improving robust decision-making. Utilizing a VAE
trained solely on clean data for object detection, we observed its
susceptibility to several environmental and sensor-based corruptions,
including snow, cross-sensor noise, and motion blur. These disturbances result
in a decline from its original prediction accuracy of approximately 87.32%.
With STARNet’s integration to detect and mitigate untrustworthy sensor data,
the prediction accuracy in these challenging scenarios improved by $\sim$10%,
effectively reverting to the level observed with clean data. Moreover,
STARNet’s reliance on computationally efficient gradient approximation
techniques ensures its viability for continuous monitoring, even on resource-
constrained edge devices.

## References

  * [1] K. Arikumar, A. Deepak Kumar, T. R. Gadekallu, S. B. Prathiba, and K. Tamilarasi, “Real-time 3d object detection and classification in autonomous driving environment using 3d lidar and camera sensors,” _Electronics_ , vol. 11, no. 24, p. 4203, 2022.
  * [2] C. V. Poulton, M. J. Byrd, P. Russo, E. Timurdogan, M. Khandaker, D. Vermeulen, and M. R. Watts, “Long-range lidar and free-space data communication with high-performance optical phased arrays,” _IEEE Journal of Selected Topics in Quantum Electronics_ , vol. 25, no. 5, pp. 1–8, 2019.
  * [3] Y. Zhao and Y. Qu, “Impact of self-heating effect on transistor characterization and reliability issues in sub-10 nm technology nodes,” _IEEE Journal of the Electron Devices Society_ , vol. 7, pp. 829–836, 2019.
  * [4] Y. Yan, W. Lan, Y. Chen, D. Yang, Y. Zhou, Z. Zhu, and J. J. Liou, “Reliability issues of thin film transistors subject to electrostatic discharge stresses: an overview,” _Advanced Electronic Materials_ , vol. 8, no. 2, p. 2100886, 2022.
  * [5] B. M. Kalasapati and S. L. Tripathi, “Robustness evaluation of electrical characteristics of sub-22 nm finfets affected by physical variability,” _Materials Today: Proceedings_ , vol. 49, pp. 2245–2252, 2022.
  * [6] S. Tan, M. Tahoori, T. Kim, S. Wang, Z. Sun, and S. Kiamehr, “Long-term reliability of nanometer vlsi systems,” _Cham: Springer_ , 2019.
  * [7] A. L. Diehm, M. Hammer, M. Hebel, and M. Arens, “Mitigation of crosstalk effects in multi-lidar configurations,” in _Electro-Optical Remote Sensing XII_ , vol. 10796. SPIE, 2018, pp. 13–24.
  * [8] R. Heinzler, P. Schindler, J. Seekircher, W. Ritter, and W. Stork, “Weather influence and classification with automotive lidar sensors,” in _2019 IEEE intelligent vehicles symposium (IV)_. IEEE, 2019, pp. 1527–1534.
  * [9] T. Gandhi and M. M. Trivedi, “Pedestrian protection systems: Issues, survey, and challenges,” _IEEE Transactions on intelligent Transportation systems_ , vol. 8, no. 3, pp. 413–430, 2007.
  * [10] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis _et al._ , “Event-based vision: A survey,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 1, pp. 154–180, 2020.
  * [11] A. Sole, O. Mano, G. P. Stein, H. Kumon, Y. Tamatsu, and A. Shashua, “Solid or not solid: Vision for radar target validation,” in _IEEE Intelligent Vehicles Symposium, 2004_. IEEE, 2004, pp. 819–824.
  * [12] D. Li, S. Cao, S. I. Lee, and J. Xiong, “Experience: practical problems for acoustic sensing,” in _Proceedings of the 28th Annual International Conference on Mobile Computing And Networking_ , 2022, pp. 381–390.
  * [13] M. Kok, J. D. Hol, and T. B. Schön, “Using inertial sensors for position and orientation estimation,” _arXiv preprint arXiv:1704.06053_ , 2017.
  * [14] M. Lee, B. Mudassar, T. Na, and S. Mukhopadhyay, “Warningnet: A deep learning platform for early warning of task failures under input perturbation for reliable autonomous platforms,” in _2020 57th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 2020, pp. 1–6.
  * [15] T. Ji, A. N. Sivakumar, G. Chowdhary, and K. Driggs-Campbell, “Proactive anomaly detection for robot navigation with multi-sensor fusion,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4975–4982, 2022.
  * [16] M. Masuda, R. Hachiuma, R. Fujii, H. Saito, and Y. Sekikawa, “Toward unsupervised 3d point cloud anomaly detection using variational autoencoder,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 3118–3122.
  * [17] P. Bergmann and D. Sattlegger, “Anomaly detection in 3d point clouds using deep geometric descriptors,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 2613–2623.
  * [18] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _arXiv preprint arXiv:1903.12261_ , 2019.
  * [19] M. Afham, I. Dissanayake, D. Dissanayake, A. Dharmasiri, K. Thilakarathna, and R. Rodrigo, “Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9902–9912.
  * [20] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _arXiv preprint arXiv:1610.02136_ , 2016.
  * [21] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [22] Z. Xiao, Q. Yan, and Y. Amit, “Likelihood regret: An out-of-distribution detection score for variational auto-encoder,” _Advances in neural information processing systems_ , vol. 33, pp. 20 685–20 696, 2020.
  * [23] S. Ghadimi and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic programming,” _SIAM Journal on Optimization_ , vol. 23, no. 4, pp. 2341–2368, 2013.
  * [24] X. Lian, H. Zhang, C.-J. Hsieh, Y. Huang, and J. Liu, “A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order,” _Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * [25] T. G. Phillips, N. Guenther, and P. R. McAree, “When the dust settles: The four behaviors of lidar in the presence of fine airborne particulates,” _Journal of field robotics_ , vol. 34, no. 5, pp. 985–1009, 2017.
  * [26] S. Liu, B. Kailkhura, P.-Y. Chen, P. Ting, S. Chang, and L. Amini, “Zeroth-order stochastic variance reduction for nonconvex optimization,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [27] S. Ghadimi, G. Lan, and H. Zhang, “Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization,” _Mathematical Programming_ , vol. 155, no. 1-2, pp. 267–305, 2016.
  * [28] K. Balasubramanian and S. Ghadimi, “Zeroth-order (non)-convex stochastic optimization via conditional gradient and gradient updates,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [29] X. Gao, B. Jiang, and S. Zhang, “On the information-adaptive variants of the admm: an iteration complexity perspective,” _Journal of Scientific Computing_ , vol. 76, pp. 327–363, 2018.
  * [30] S. Bhatnagar and L. Prashanth, “Generalized simultaneous perturbation stochastic approximation with reduced estimator bias,” in _2023 57th Annual Conference on Information Sciences and Systems (CISS)_. IEEE, 2023, pp. 1–6.
  * [31] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [32] D. Xu, D. Anguelov, and A. Jain, “Pointfusion: Deep sensor fusion for 3d bounding box estimation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 244–253.
  * [33] P. Zhao, P.-Y. Chen, S. Wang, and X. Lin, “Towards query-efficient black-box adversary with zeroth-order natural gradient descent,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 6909–6916.
  * [34] S. Liu, P.-Y. Chen, B. Kailkhura, G. Zhang, A. O. Hero III, and P. K. Varshney, “A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications,” _IEEE Signal Processing Magazine_ , vol. 37, no. 5, pp. 43–54, 2020.
  * [35] M. Cheng, S. Singh, P. Chen, P.-Y. Chen, S. Liu, and C.-J. Hsieh, “Sign-opt: A query-efficient hard-label adversarial attack,” _arXiv preprint arXiv:1909.10773_ , 2019.
  * [36] L. Kong, Y. Liu, X. Li, R. Chen, W. Zhang, J. Ren, L. Pan, K. Chen, and Z. Liu, “Robo3d: Towards robust and reliable 3d perception against corruptions,” _arXiv preprint arXiv:2303.17597_ , 2023.
  * [37] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, “Likelihood ratios for out-of-distribution detection,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [38] J. Serrà, D. Álvarez, V. Gómez, O. Slizovskaia, J. F. Núñez, and J. Luque, “Input complexity and out-of-distribution detection with likelihood-based generative models,” _arXiv preprint arXiv:1909.11480_ , 2019.


# Non-Linear Self Augmentation Deep Pipeline for Cancer Treatment outcome
Prediction

Francesco Rundo  ADG Central R&D
STMicroelectronics
Catania, Italy
<EMAIL_ADDRESS>Concetto Spampinato  PerCeiVe Lab
University of Catania
Catania, Italy
<EMAIL_ADDRESS>Michael Rundo  PerCeiVe Lab
University of Catania
Catania, Italy
<EMAIL_ADDRESS>
###### Abstract

Immunotherapy emerges as promising approach for treating cancer. Encouraging
findings have validated the efficacy of immunotherapy medications in
addressing tumors, resulting in prolonged survival rates and notable
reductions in toxicity compared to conventional chemotherapy methods. However,
the pool of eligible patients for immunotherapy remains relatively small,
indicating a lack of comprehensive understanding regarding the physiological
mechanisms responsible for favorable treatment response in certain individuals
while others experience limited benefits. To tackle this issue, the authors
present an innovative strategy that harnesses a non-linear cellular
architecture in conjunction with a deep downstream classifier. This approach
aims to carefully select and enhance 2D features extracted from chest-abdomen
CT images, thereby improving the prediction of treatment outcomes. The
proposed pipeline has been meticulously designed to seamlessly integrate with
an advanced embedded Point of Care system. In this context, the authors
present a compelling case study focused on Metastatic Urothelial Carcinoma
(mUC), a particularly aggressive form of cancer. Performance evaluation of the
proposed approach underscores its effectiveness, with an impressive overall
accuracy of approximately 93%

###### Index Terms:

Cellular Non-Linear Network, Deep Convolutional Network, Immunotherapy,
Radiomics

##  I Introduction

Immunotherapy has emerged as the forefront in the battle against cancer,
revolutionizing the approach by treating tumors as if they were infectious
agents. The concept involves empowering the patient’s immune system to
recognize and eliminate cancer cells, akin to arming the body’s defenses [1,
2]. While the immune system naturally recognizes cancer cells and activates T
lymphocytes to attack them, cancer cells possess mechanisms to evade this
response, exploiting the immune system’s self-regulation process through
various proteins acting as ”accelerators” or ”brakes” on T cells [1, 2, 3, 4].
A promising strategy in immunotherapy focuses on inhibiting ”Immunological
Checkpoints” (ICIs) by employing specific antibodies. These ICIs counteract
the disabling effects imposed by cancer cells, thereby boosting the efficacy
of T lymphocytes in combating tumors [1, 2, 3]. In this context, our
discussion will revolve around immunotherapy treatments targeting the PD-1
receptor, which represent a significant advancement in the field [3]. Research
studies have revealed a crucial mechanism employed by cancer cells to evade
the immune system’s response. They utilize a molecule called PD-L1, which is
present on their membrane and binds to the PD-1 receptor on T lymphocytes,
effectively disabling their protective actions. In the context of
immunotherapy, ICIs targeting the PD-1/PD-L1 interaction aim to inhibit these
receptors, enabling T lymphocytes to recognize and eliminate cancer cells [3,
4]. In this study, we investigate this mechanism specifically in the case of
metastatic bladder cancer, focusing on metastatic Urothelial Carcinoma (mUC)
[4]. Urothelial carcinoma is known to be one of the most aggressive forms of
cancer affecting the urinary system, leading to a significant number of deaths
annually [1, 5]. Platinum-based chemotherapy is currently considered the
standard treatment for mUC, as demonstrated by various clinical trials [4].
However, understanding the dynamics of long-term survival rates in patients
with bladder cancer undergoing chemotherapy or immunotherapy is crucial, and
it is assessed through parameters such as Progression-Free Survival (PFS) rate
and Overall Survival (OS) rate, which provide quantitative measures of
treatment effectiveness [6]. In the case of high-dose chemotherapy treatment,
the median Overall Survival (OS) ranges from 12 to 15 months for a cisplatin-
based regimen and approximately 9 months for a carboplatin-based regimen [7].
However, these chemotherapy treatments are associated with significant
toxicity. In contrast, immunotherapy has emerged as a new standard of care due
to its potential benefits. Immune checkpoint inhibitors (ICIs) such as
Atezolizumab and Pembrolizumab have shown a median OS of more than 10 months
in both cases, with a notable reduction in side effects compared to
chemotherapy treatments [8, 9]. Despite the effectiveness of immunotherapy,
only a subset of patients, approximately 20% to 30%, experience a positive
response [10, 11, 12]. Therefore, the development of discriminative biomarkers
capable of identifying patients who are likely to benefit from immunotherapy
is a primary focus of current research activities. In recent years, there has
been a growing interest in using visual features from medical images and
biomarkers such as PD-L1 expression level or blood indexes like the neutrophil
to lymphocyte ratio as predictive immunotherapy outcome biomarkers [13]. The
field of Radiomics, which involves the quantitative analysis of large sets of
biomedical multi-modal data, has played a significant role in linking clinical
trials to imaging research [14, 15, 16]. In this study, we propose an
innovative and less invasive pipeline for bladder cancer diagnosis based on
deep learning algorithms. Specifically, we focus on classifying visual
features extracted from chest-abdomen CT-scan images using a specially
configured Cellular Non-Linear Network (CNN). This pipeline aims to accurately
identify and augment CT cancer lesions. Furthermore, our designed pipeline is
intended to be implemented on an embedded hardware accelerated Point of Care
system, ensuring its practicality and efficiency.

##  II Related Works

Accurate prediction of medical treatment outcomes for metastatic diseases
requires the development of robust quantitative data processing pipelines.
Machine and Deep Learning approaches have been extensively explored in the
scientific literature for this purpose [15, 16, 17]. In a study by Wang et al.
[15], the performance of various Machine Learning (ML) algorithms was
evaluated to predict mortality after radical cystectomy in bladder cancer
patients. The Regularized Extreme Learning Machine demonstrated superior
accuracy compared to other methods. Similarly, Garapati et al. [16] applied
common ML algorithms to analyze CT-scan urography in a clinical study, with
Support Vector Machine (SVM) achieving impressive results. Hasnain et al. [17]
proposed ML methods, including K-Nearest Neighbors, AdaBoost, and SVM, for
estimating bladder cancer disease. Their approach focused on predicting cancer
recurrence and survival using multi-modal data analysis. The results showed
specificity and sensitivity higher than 70%, highlighting the effectiveness of
Radiomics in medical oncology. The advent of large-scale multi-source medical
data has transformed the learning model paradigm, leading to more efficient
data processing approaches. Various solutions for cancer image-lesion
segmentation, including 2D and 3D convolutional networks, have been proposed
[17, 18, 19, 20, 21]. These methods have shown promising performance in
segmenting metastatic lesions from CT imaging, particularly when dealing with
lesions of the same type (visceral or lymphatic).

Additional deep pipelines for estimating the response to cancer treatments
based on quantitative data analysis have been proposed in recent research [22,
23, 24]. In [22], a modified version of the AlexNet backbone [23] was employed
to develop a Deep Learning architecture for assessing the response to
chemotherapeutic treatment. The proposed deep network was trained on segmented
CT slices to learn visual features and achieved promising results in
predicting treatment outcomes. Cha et al. [24] presented a Deep Learning
pipeline for predicting immunotherapy outcomes in bladder cancer patients. The
pipeline consisted of two stages: primary cancer-lesion segmentation in CT
scans and treatment response prediction using a deep architecture. The
performance of the pipeline was evaluated on a dataset of 82 bladder cancer
patients, achieving a specificity of 81% (DL-CNN) and a sensitivity of 66.7%
(RF-ROI). In [25], the authors introduced a novel deep pipeline for detecting
immunotherapy outcomes in bladder cancer patients using artificial
intelligence technology. The pipeline involved a deep classifier trained on
visual features generated by a stack of encoders. The results showed improved
performance compared to the authors’ previous solution, with an accuracy of
86.05%, specificity of 89.29%, and sensitivity of 80.00%. These advanced deep
pipelines demonstrate the progress made in using deep learning and artificial
intelligence techniques to predict treatment outcomes in bladder cancer
patients, offering promising results and potential for embedded systems in
clinical practice [22, 24, 25].

Figure 1: The proposed immunotherapy treatment outcome prediction pipeline.

##  III The proposed Deep Network Framework

In Fig. 1, we reported the comprehensive architecture of the proposed deep
pipeline, designed to predict the immunotherapy treatment outcome for patients
diagnosed with metastatic bladder cancer. The pipeline leverages radiomics
techniques to quantitatively analyze selected chest-abdomen CT-scan imaging
cancer lesions. The identification of CT-scan lesions is carried out by
experienced oncologists/radiologists in accordance with the RECIST 1.1
guideline [26]. Unlike many existing pipelines [24], the proposed approach
takes into account both primary lesion images (bladder) and metastatic lesion
images (visceral or lymph nodes). This unique feature enhances the method’s
robustness and efficiency in the medical domain. Initially, the semi-automatic
pipeline performs a bounding box segmentation of the CT lesion identified by
skilled physicians. This segmentation defines the Region of Interest (ROI) to
which the predictive pipeline is applied. Those ROI images have different
biologically visual features as they have been extracted from various body-
sites of mUC metastasis (lungs, abdominal organs, bladder, lymph nodes, etc
..). The diverse nature of the dataset poses a challenge in constructing a
fully automated semantic segmentation pipeline. Consequently, the segmented
ROIs, representing the CT lesions, undergo further processing using a
customized and expanded 2D Cellular Non-Linear Network (2D-CNN). This network
generates a set of augmented domain-agnostic features, which are then fed into
a downstream 2D Deep Classifier (2D-DNN). The purpose of the 2D-DNN is to
accurately classify the augmented visual features and predict the treatment
response of patients. Specifically, it distinguishes patients who may exhibit
a favorable response to immunotherapy treatment (CR: Complete Response / PR:
Partial Response / SD: Stable Disease) from those who experience disease
progression (PD: Progressive Disease). Several 2D-DNN backbones have been
tested preferring deep architectures with medium-low complexity to make the
overall pipeline portable on embedded platforms based on STA1295 ASIL-B
certified hardware architecture111https://www.st.com/en/automotive-
infotainment-and-telematics/sta1295.html. The following sections provide a
detailed description of the proposed solution.

###  III-A The Bounding Box Segmentation block

The proposed pipeline incorporates a semi-automatic CT-scan lesion
segmentation process guided by experienced oncologists/radiologists.
Initially, the entire chest-abdomen CT scan of each patient is examined, and
the oncologist/radiologist manually identifies a specific cancerous lesion
based on the criteria outlined in the RECIST revision 1.1 guideline [26]. Most
modern CT imaging software includes automated ROI selection based on spatial,
dimensional, or morphological criteria. Following the lesion selection, our
segmentation software automatically extracts a bounding box ROI of size
$M\times N$ centered around the lesion. The specific dimensions (M, N) can
vary, and the overall performance of the pipeline remains unaffected as the
segmented lesion will be subsequently rescaled to match the input size of the
downstream 2D-DNN classifier. For this particular implementation, we opted for
a bounding box ROI size of $64\times 64$. Additional details regarding the
RECIST 1.1 guideline can be found in [26]. The term RECIST refers to an
official standardized methodology for evaluating treatment outcomes in solid
tumors [26]. The RECIST criteria, as of interest in this research work,
includes the following items:

  * •

Measurable lesion (target lesion): a lesion that can be accurately measured
with longest diameter (in one dimension) $\geq$ 20 mm (CT scan imaging);

  * •

A sum of the longest diameter (LD) for all CT image target lesions will be
computed and reported as the baseline for the follow-up evaluations.

The RECIST 1.1 guideline consequently specifies how to classify the patient’s
response to a specific cancer therapy:

  * •

A patient shows a complete response (CR) to the medical treatment if all
identified target lesions disappear at the end-treatment CT imaging.

  * •

A patient shows a partial response (PR) if the target lesions (LD sum) are
reduced by at least 30 %.

  * •

A patient shows a progressive disease (PD) if the LD sum increases by, at
least, 20% ;

  * •

A patient instead reports stable disease (SD) if no significant increase or
decrease is observed on the target lesions.

This work as well as the clinical trial to which it refers, provide results
based on RECIST 1.1 guideline[26].

###  III-B The 2D-CNN Features Generative Model

This block involves further processing of the collected ROIs to extract
informative visual features. The proposed generative model utilizes an
extended and appropriately configured version of 2D Cellular Nonlinear
Networks (CNNs). The theory of Cellular Neural (or Nonlinear) Networks (CNNs)
is briefly summarized.

The initial architecture of CNNs was proposed by L.O. Chua and L. Yang [27].
CNNs can be described as a high-speed array of analog processors, referred to
as ”cells” [27]. The fundamental unit of a CNN is the cell, and the processing
of CNNs is governed by instructions provided by cloning templates [27]. Each
cell within a CNN can be viewed as a dynamical system arranged in a 2D or 3D
topological structure. Interactions between CNN cells occur within a
neighborhood defined by a heuristically determined radius [27]. Each CNN cell
possesses an input, a state, and an output, with the state typically being
mapped through a PieceWise Linear function. Stability analysis and
considerations regarding the dynamics of CNNs have been documented in [27,
28].

It is worth noting that the utilized 2D-CNN belongs to the category of
transient-response CNNs. These architectures perform the transformation of
input data within a limited time frame, specifically during the transient
stage. During this phase, each individual cell of the CNN dynamically evolves
from its initial state along a trajectory that converges to the CNN’s steady-
state [27, 28]. This allows us to extract intermediate 2D-CNN transformation
features from the input data, which will be utilized as augmented generated
features. In the CNN paradigm, the relationship between state, input, output,
and neighborhood can be hypothesized in various ways, considering that CNNs
are considered ”Universal Machines” [29, 30]. Thus, we proceed with an
extended mathematical representation of a generative model based on CNNs. The
CNN can be considered as a system of cells (or non-linear neurons) defined on
a normed-space $S_{N}$ (grid structure), which is a discrete subset of $R^{n}$
(generally $n\leq 3$) with distance function d: $S_{N}\rightarrow N$ (N is the
set of positive integer numbers). Cells are indexed in a space-set $L_{i}$.
Neighborhood function $N_{r}(.)$ of a k-th cell can be defined as follows:

$\begin{split}&N_{r}:L_{i}\rightarrow L_{i}^{\beta}\\\
&N_{r}(k)=\\{l|d(i,j)\leq r\\}\end{split}$ (1)

where $\beta$ depends on r (neighborhood radius) and on space geometry of the
grid. The CNNs can be implemented as a single layer or multi-layers so that
the cell grid can be e.g., a planar array (with rectangular, square, octagonal
geometry) or a k-dimensional array (usually $k\geq 3$), generally considered
and realized as a stack of k-dimensional arrays (layers). Therefore, CNN can
be represented as a time-continuous - space-discrete system whose dynamic is
well defined by the spatio-temporal evolution of the cells. The following
mathematical model defines the dynamic of an extended CNN cell:

$\begin{split}\partial x_{j}/\partial t&=g[x_{j}\ (t)]+\\\
&+\sum_{\begin{subarray}{c}\gamma\in
N_{r}(i)\end{subarray}}\aleph_{\vartheta_{j}}\left(x_{j}|_{\left(t-\tau,t\right]},y_{\gamma}{\
|}_{\left(t-\tau,t\right]};p_{\ j}^{A}\right)+\\\ &+\sum_{\gamma\in
N_{r}\left(j\right)}{\mathcal{B}_{\varphi_{j}}\left(x_{j}|_{\left(t-\tau,t\right]},u_{\gamma}{\
|}_{\left(t-\tau,t\right]};p_{\ j}^{B}\right)}+\\\ &+\sum_{\gamma\in
N_{r}\left(j\right)}{\mathbb{C}_{\rho_{j}}\left(x_{j}|_{\left(t-\tau,t\right]},x_{\gamma}{\
|}_{\left(t-\tau,t\right]};p_{\ j}^{C}\right)}+\\\ &+I_{j}\left(t\right)\\\
\end{split}$ (2) $\begin{split}y_{j}(t)=\psi(x_{j}|_{t-\tau,t})\end{split}$
(3)

In Equation 2, the variables $x$, $y$, $u$, and $I_{j}$ represent the cell
state, output, input, and bias, respectively. The indices $j$ and $\gamma$
refer to specific cells, while $g$ represents the local instantaneous feedback
function. The function $N_{r}$ defines the neighborhood function, and $p^{A}$,
$p^{B}$, and $p^{C}$ are arrays of configurable parameters specific to each
cell. The notation $z|T$ denotes the restriction of function $z(\cdot)$ to the
interval $T$ of its argument. The symbol $\aleph_{\vartheta_{j}}$ represents
the neighborhood feedback functional, which is one of several applicable
functions identified by the index $\vartheta_{j}$. Similarly,
$\mathcal{B}{\varphi_{j}}$ denotes the input functional, and
$\mathbb{C}{\rho_{j}}$ represents the cell-state functional. The term $I_{j}$
corresponds to an ad-hoc defined bias. The function $\psi$ describes the
mathematical correlation between the cell state and its output. Applying a
linear re-mapping and extension of the model reported in Eqs. (2)-(3), the
following CNN cell generative dynamical model is proposed:

$\begin{split}C\frac{dx_{ij}\left(t\right)}{dt}&=-\frac{1}{R_{x}}x_{ij}+\\\
&+\sum_{C\left(k,l\right)\in
N_{r}\left(i,j\right)}{A\left(i,j;k,l\right)}y_{kl}\left(t\right)+\\\
&+\sum_{C\left(k,l\right)\in
N_{r}\left(i,j\right)}{B\left(i,j;k,l\right)}u_{kl}\left(t\right)+\\\
&+\sum_{C\left(k,l\right)\in
N_{r}\left(i,j\right)}{C\left(i,j;k,l\right)}x_{kl}\left(t\right)+\\\
&+\sum_{C\left(k,l\right)\in
N_{r}\left(i,j\right)}{D\left(i,j;k,l\right)}(y_{ij}\left(t\right),y_{kl}\left(t\right))+\\\
&+I\\\ &1\leq i\leq M,1\leq j\leq N\end{split}$ (4)

$\begin{split}y_{ij}(t)&=\frac{1}{2}(|x_{ij}(t)+1|-|x_{ij}(t)-1|)\\\
\end{split}$ (5)
$\begin{split}N_{r}(i,j)&=\\{C_{r}(k,l);(max{(|k-i|,|i-j|)}\leq r)\\}\\\
&(1\leq k\leq M,1\leq l\leq N))\end{split}$ (6)

In Equations ($4$)-($6$), the term $Nr(i,j)$ represents the neighborhood of
each CNN cell $C(i,j)$ with a radius of $r$. The variables $x_{ij}$, $y_{ij}$,
$u_{ij}$, and $I$ denote the state, output, input, and bias of cell $C(i,j)$,
respectively. The cloning templates $A(i,j;k,l)$, $B(i,j;k,l)$, $C(i,j;k,l)$,
and $D(i,j;k,l)$ define the CNN processing, with $D(i,j;k,l)$ representing the
extended non-linear space-invariant cloning template used in the proposed
2D-CNN model [31]. The configuration of the cloning templates determines the
specific features generated by the CNN, as discussed in [29, 30, 32]. The
spatial dimensions of the CNN are defined by the setup (M, N) in Equations
($4$)-($6$), with $R_{x}$ representing an ad-hoc defined coefficient related
to the hardware implementation of the CNN cell, set to 1. In the implemented
CNN generative features model, we input the segmented ROI lesion described in
the previous section to both the input and state of the 2D-CNN. To match the
input size of the downstream 2D-DNN ($M_{D}\times N_{D}$), we perform a
preliminary bi-cubic rescaling of the segmented ROI to $M_{D}\times N_{D}$
size. Subsequently, an $M_{D}\times N_{D}$ 2D grid CNN is implemented. Each of
the resized $M_{D}\times N_{D}$ segmented ROIs is fed into the input $u_{ij}$
and state $x_{ij}$ of the $M_{D}\times N_{D}$ 2D grid CNN, with each input and
state of the CNN cells corresponding to the pixel values of the resized input
visual lesion. In this case, the gray level intensity is used as the input for
the CT ROI lesion, as it is a single-layer gray-level image. The workflow of
the implemented CNN generative features model is illustrated in Figure 2. Let
$L_{j}(x,y)$ denote the j-th visual segmented resized gray-level ROI image,
which is fed as input ($u_{ij}$) and state ($x_{ij}$) to the $M_{D}\times
N_{D}$ CNN grid. The image processing task, guided by the configured
generative model according to Equations ($4$)-($6$), is performed. For each
setup $S_{m}$ of the cloning templates $A(i,j;k,l)$, $B(i,j;k,l)$,
$C(i,j;k,l)$, $D(i,j;k,l)$, and bias $I$ (generative model), a specific visual
feature $f_{Sm}(x,y)$ of the input ROI $L_{j}(x,y)$ is generated. After
conducting several tests, we have chosen to use $m=97$ different generative
models, meaning 97 different configurations of $3\times 3$ cloning template
matrices and biases of size $1\times 1$. This allows us to generate 97
augmented visual features for each segmented resized ROI $L_{j}(x,y)$.

Through an unsupervised approach that minimizes the loss of the downstream
2D-DNN classifier (maximizing the discrimination accuracy), we have selected
the generative models. During the training process, a random-driven searching
algorithm was employed to select the $3\times 3$ cloning template and bias
configurations $S_{m}$ that produced an improvement in the discriminatory
performance of the 2D-DNN classifier. The designed generative models (cloning
templates and biases) as well as the adopted transient configuration of the
used 2D-CNN can be downloaded from ad-hoc designed web
page222https://iplab.dmi.unict.it/immunotherapy/. In Fig. 3 a collection of
such 2D-CNN generated features is reported. The reported features are
associated to different CT cancer native lesion types i.e. visceral, lymph-
nodes, etc..

Figure 2: The proposed 2D-CNN generative model

Figure 3: A collection of the 2D-CNN generated features

###  III-C The 2D-DNN Classifier with Decision System

This block focuses on learning the 2D-CNN generated augmented features to
accurately classify patients eligible for immunotherapy treatment. Fig. 4
illustrates the underlying architecture of the proposed deep model. Different
2D deep classifier backbones were investigated to assess their computational
complexity on the implemented embedded hardware Point of Care and determine
the ones that deliver the best predictive performance. The following 2D deep
classifier backbones were validated: ResNet-18, VGG-19, Xception, MobileNetV2,
GoogleNet, and AlexNet [33]. Additionally, a NasNetMobile-based 2D Deep
Classifier was tested [34]. For each patient, the target CT lesions (ROIs)
that comply with the RECIST 1.1 guidelines are selected using the introduced
semi-automatic segmentation block. The resized ROIs are then augmented using
the proposed 2D-CNN generative engine. The generated augmented feature maps
are subsequently classified by the 2D-DNN, which provides a probability
estimation of belonging to class 1 (CR / PR: complete or partial response to
immunotherapy treatment or SD: stable disease) or class 2 (PD: progressive
disease). In the last block, there is a Decision System depicted in Fig. 4.
This layer aims to collect the classifications of the individual augmented
visual 2D-CNN generated features. As mentioned earlier, for each patient, a
set of $m=97$ visual features is generated by the 2D-CNN generative model.
Each of these feature images is then classified by the 2D-DNN as belonging to
either class 1 or class 2. It is necessary to determine the main
(statistically more representative) classification among the generated
features associated with the patient whose CT visual lesion they refer to. The
task of the Decision System, for each patient, is to determine the main 2D-DNN
classification rate of the CNNs generated features. This predominant
classification will be the definitive classification of the patient.
Analytically, the Decision Layer will produce the following output:

$D_{k}=\xi_{c}(C_{DNN}(f_{Sm}));\ \ C_{DNN}(f_{Sm})=\\{1,2\\}$ (7)

Eq. (7) shows the mathematical model of the Decision Layer. The function
$\xi_{c}$ will analyze the 2D-DNN classification $C_{DNN}$ rate of the
$f_{Sm}$ generated features ($m=97$) related to the k-th patient, determining
the class (1 or 2) that statistically is most represented. The output class of
the Decision System ($\xi_{c}$) becomes the definitive classification
(immunotherapy outcome prediction) related to the analyzed patient.

Figure 4: The proposed immunotherapy outcome deep predictor.

##  IV Experimental Results

We conducted a retrospective analysis using a dataset of 106 mUC cancer
lesions extracted from chest-abdomen CT scans of patients with histologically
confirmed bladder cancer who participated in clinical trials. These patients
had bladder cancer that progressed after platinum-based chemotherapy and were
subsequently treated with a PD-L1 ICIs immunotherapy agent in the second-line
setting. All patients provided written informed consent, and the clinical
trials were approved by the IRB ”Catania 1 Ethical Committee” (Nr. D4191C00068
and MO29983). For each patient, a chest-abdomen CT-scan was performed during
the diagnostic phase for disease staging. The CT imaging was conducted using a
GE multi-slice (64 slices) CT scanner, with a slice thickness of 2.5 mm. The
clinical and personal history data of each patient complemented the CT scans.
The aim of our proposed pipeline is to investigate the use of augmented CT
image features at the time of bladder cancer diagnosis to predict the outcome
of immunotherapy treatment. Statistical information about the recruited
dataset is as follows: approximately 30% of the patients were under the age of
60, and 91% of the patients were male, with the remaining 9% being female.
Among the subjects, about 33% had lymph node metastases, while the remaining
67% had various visceral metastatic lesions. Out of the 106 cases (target
lesions), 43 showed a complete/partial response or disease stabilization
following immunotherapy treatment (CR/PR/SD: Class 1), while 63 cases
experienced disease progression despite anti-PD-L1 drug treatment (PD: Class
2). The dataset was split as follows: 76 target lesions (28 of Class 1 and 48
of Class 2) were used for training and validation, and the remaining 30 CT
images (15 of Class 1 and 15 of Class 2) were used as the test set. These
lesions were augmented using the proposed generative model based on the
2D-CNN. Considering that for each CT image lesion, a generative engine of m=97
CNN models setup (3x3 cloning templates and 1x1 biases) was defined, the
augmented dataset consists of a total of 7372 (76x97) image features (2716
images of Class 1 and 4656 images of Class 2) for the training and validation
set. The test set comprises 2910 image features (1455 images for each class).
The proposed pipeline was developed using MATLAB rev. 2019b with full
toolboxes. The implementation was carried out on a server with an Intel
16-Core processor and an NVIDIA GeForce RTX 2080 GPU. We utilized the Deep
Learning and custom matCNN MATLAB toolboxes for implementing the 2D-DNN and
2D-CNN generative models, respectively.

For all the tested 2D-DNN model backbones, we used the following learning
parameters: a mini-batch size of 10, an initial learning rate of 3e-4, a
maximum of 900 epochs, and the stochastic gradient descent with momentum
(SGDM) algorithm as the learning optimizer.

We configured the 2D-CNN as an $M_{d}\times N_{d}$ grid according to the input
image size of the tested 2D-DNN classifier, as specified in Table I. The
2D-CNN was set up with 3x3 cloning templates and 1x1 biases, which were made
available for download on the web-page mentioned earliernote2. Table I
presents the performance evaluation of the tested 2D-DNN architectures with
2D-CNN augmentation. The performance of the proposed pipeline was validated
using classical metrics such as Accuracy, Sensitivity, and Specificity.

To evaluate the pipeline’s performance, we considered ”True Positive” as the
correct classification of patients who showed a positive response to
immunotherapy treatment (CR, PR, or SD) and were classified as belonging to
Class 1. ”True Negative” referred to patients who were classified as belonging
to Class 2 and did not show a response to the immunotherapy drug (PD). ”False
Negative” and ”False Positive” values were also computed accordingly.

Table I shows the performance results (on the test set) in terms of accuracy,
sensitivity, and specificity using each selected 2D-DNN classifier with the
augmented CT image dataset through the proposed 2D-CNN generative model. While
some architectures achieved 100% sensitivity, deep architectures demonstrated
acceptable performance even in terms of specificity, which is preferred.

ResNet-18 and VGG-19 exhibited an interesting trade-off in classification
performance. ResNet-18 achieved an accuracy of 93.33%, sensitivity of 93.33%,
and specificity of 93.33%. VGG-19 achieved an accuracy of 93.33%, sensitivity
of 100.00%, and specificity of 86.66%. All performance results are reported in
Table I. The accuracy, sensitivity, and specificity values refer to the
overall classification of the generated features performed by the Decision
System as described in Eq. (7) ($\xi_{c}$ output).

The analysis of Table I confirms that despite the limited architectural
complexity of the tested classifier backbones (due to hardware limitations of
the embedded Point of Care), the overall prediction performances are highly
promising.

TABLE I: 2D-DNN Performance Benchmark - 2D-CNN Dataset Augmentation Model

2D-Deep Classifier Backbone | Metrics
---|---
| Accuracy | Sensitivity | Specificity | Input Size
ResNet-18 | $93.33\%$ | $93.33\%$ | $93.33\%$ | $224x224$
VGG-19 | $93.33\%$ | $100.00\%$ | $86.66\%$ | $224x224$
XCeption | $86.66\%$ | $93.33\%$ | $80.00\%$ | $299x299$
MobileNetV2 | $83.33\%$ | $100.00\%$ | $66.66\%$ | $224x224$
GoogleNet | $86.66\%$ | $86.66\%$ | $86.66\%$ | $224x224$
AlexNet | $83.33\%$ | $80.00\%$ | $86.66\%$ | $227x227$
NasNetMobile | $76.66\%$ | $80.00\%$ | $73.33\%$ | $224x224$
Previous[25] | $86.05\%$ | $80.00\%$ | $89.29\%$ | $40x40$

TABLE II: 2D-DNN Performance Benchmark - Classical Dataset Augmentation Method

Deep Classifier Backbone | Metrics
---|---
| Accuracy | Sensitivity | Specificity
ResNet-18 | $86,66\%$ | $100.00\%$ | $78.66\%$
VGG-19 | $73.33\%$ | $86.66\%$ | $60.00\%$
3D-DenseNet | $83.33\%$ | $86.66\%$ | $80.00\%$
ResNet-101 | $76.66\%$ | $80.00\%$ | $73.33\%$

Table II provides a comparison of the performance of the most effective
architectures from Table I (ResNet-18 and VGG-19) without the 2D-CNN
generative model, using traditional input augmentation approaches. The results
demonstrate that the performance, in terms of overall performance, was
significantly lower compared to the same architectures trained with a dataset
augmented through 2D-CNN.

ResNet-18, without the 2D-CNN generative model, showed an improvement in
sensitivity (100% compared to 93.33% with 2D-CNN) but a considerable decrease
in specificity (78.66% compared to 93.33% with 2D-CNN), limiting the overall
performance of the pipeline.

Similarly, VGG-19 without the 2D-CNN generative model exhibited significantly
lower performance compared to when it was used with the upstream 2D-CNN
generative model.

Even when compared to more complex architectures such as 3D-DenseNet (with a
classification stage and 16 CT slices as input temporal depth) and 2D
ResNet-101, which were tested without the use of the 2D-CNN generative model
(as shown in Table II), the valid contribution of the 2D-CNN generative model
in enhancing performance was evident.

These results highlight the effectiveness of the 2D-CNN generative model in
improving the overall performance of the pipeline, surpassing the traditional
input augmentation approaches and even outperforming more complex
architectures.

##  V Conclusion and Discussion

This study introduces an innovative pipeline for predicting immunotherapy
treatment outcome in patients diagnosed with bladder cancer using augmented CT
image features. Traditional approaches for predicting treatment response have
limitations and invasiveness, making the development of non-invasive image-
based biomarkers crucial. The pipeline focuses on learning image features from
chest-abdomen CT scans of metastatic bladder cancer patients, leveraging the
unique response patterns observed in ICIs treatment.

To overcome the challenge of limited labeled clinical data, an extended 2D-CNN
generative model was employed and validated. The experimental results
presented in Tables I and II demonstrate the promising performance of the
proposed pipeline in terms of accuracy, sensitivity, and specificity. This
improvement is attributed to the combination of a deep high-capacity
classifier and the 2D-CNN augmented training set.

The pipeline successfully analyzes all RECIST 1.1 compliant lesions,
indicating that it is not strongly influenced by lesion selection and is
robust across different lesion types. The implementation of the pipeline is
designed for the embedded STA1295 platform with OpenCV and YOCTO Linux OS.

Future studies aim to enhance the pipeline by incorporating a fully automatic
segmentation block. An enhanced GradCAM-driven algorithm is being
investigated, which integrates the RECIST 1.1 guideline to enable automatic CT
ROI segmentation based on explainable visual features. Additionally, an
adaptive Deep GAN architecture is being explored as a generative model.

Overall, the proposed pipeline shows promising potential for predicting
immunotherapy treatment outcome in bladder cancer patients, and ongoing
research aims to further enhance its capabilities.

## References

  * [1] H. O. Alsaab, S. Sau, R. Alzhrani, K. Tatiparti, K. Bhise, S. K. Kashaw, and A. K. Iyer, “Pd-1 and pd-l1 checkpoint signaling inhibition for cancer immunotherapy: mechanism, combinations, and clinical outcome,” _Frontiers in pharmacology_ , vol. 8, p. 561, 2017.
  * [2] K. R. Spencer, J. Wang, A. W. Silk, S. Ganesan, H. L. Kaufman, and J. M. Mehnert, “Biomarkers for immunotherapy: current developments and challenges,” _American Society of Clinical Oncology educational book_ , vol. 36, pp. e493–e503, 2016.
  * [3] X. Ding, Q. Chen, Z. Yang, J. Li, H. Zhan, N. Lu, M. Chen, Y. Yang, J. Wang, and D. Yang, “Clinicopathological and prognostic value of pd-l1 in urothelial carcinoma: a meta-analysis,” _Cancer management and research_ , vol. 11, p. 4171, 2019.
  * [4] T. C. Zhou, A. I. Sankin, S. A. Porcelli, D. S. Perlin, M. P. Schoenberg, and X. Zang, “A review of the pd-1/pd-l1 checkpoint in bladder cancer: From mediator of immune escape to target for treatment,” in _Urologic Oncology: Seminars and Original Investigations_ , vol. 35, no. 1. Elsevier, 2017, pp. 14–20.
  * [5] J. Ferlay, I. Soerjomataram, R. Dikshit, S. Eser, C. Mathers, M. Rebelo, D. M. Parkin, D. Forman, and F. Bray, “Cancer incidence and mortality worldwide: sources, methods and major patterns in globocan 2012,” _International journal of cancer_ , vol. 136, no. 5, pp. E359–E386, 2015.
  * [6] M. De Santis, J. Bellmunt, G. Mead, J. M. Kerst, M. Leahy, P. Maroto, T. Gil, S. Marreaud, G. Daugaard, I. Skoneczna _et al._ , “Randomized phase ii/iii trial assessing gemcitabine/carboplatin and methotrexate/carboplatin/vinblastine in patients with advanced urothelial cancer who are unfit for cisplatin-based chemotherapy: Eortc study 30986,” _Journal of clinical oncology_ , vol. 30, no. 2, p. 191, 2012.
  * [7] E. Seront and J.-P. Machiels, “Molecular biology and targeted therapies for urothelial carcinoma,” _Cancer treatment reviews_ , vol. 41, no. 4, pp. 341–353, 2015.
  * [8] T. Powles, I. Durán, M. S. Van Der Heijden, Y. Loriot, N. J. Vogelzang, U. De Giorgi, S. Oudard, M. M. Retz, D. Castellano, A. Bamias _et al._ , “Atezolizumab versus chemotherapy in patients with platinum-treated locally advanced or metastatic urothelial carcinoma (imvigor211): a multicentre, open-label, phase 3 randomised controlled trial,” _The Lancet_ , vol. 391, no. 10122, pp. 748–757, 2018.
  * [9] J. Bellmunt, R. De Wit, D. J. Vaughn, Y. Fradet, J.-L. Lee, L. Fong, N. J. Vogelzang, M. A. Climent, D. P. Petrylak, T. K. Choueiri _et al._ , “Pembrolizumab as second-line therapy for advanced urothelial carcinoma,” _New England Journal of Medicine_ , vol. 376, no. 11, pp. 1015–1026, 2017\.
  * [10] P. Sharma, P. Bono, J. W. Kim, P. Spiliopoulou, E. Calvo, R. N. Pillai, P. A. Ott, F. G. De Braud, M. A. Morse, D. T. Le _et al._ , “Efficacy and safety of nivolumab monotherapy in metastatic urothelial cancer (muc): Results from the phase i/ii checkmate 032 study.” 2016.
  * [11] C. Massard, M. S. Gordon, S. Sharma, S. Rafii, Z. A. Wainberg, J. Luke, T. J. Curiel, G. Colon-Otero, O. Hamid, R. E. Sanborn _et al._ , “Safety and efficacy of durvalumab (medi4736), an anti–programmed cell death ligand-1 immune checkpoint inhibitor, in patients with advanced urothelial bladder cancer,” _Journal of Clinical Oncology_ , vol. 34, no. 26, p. 3119, 2016\.
  * [12] A. B. Apolo, J. R. Infante, A. Balmanoukian, M. R. Patel, D. Wang, K. Kelly, A. E. Mega, C. D. Britten, A. Ravaud, A. C. Mita _et al._ , “Avelumab, an anti–programmed death-ligand 1 antibody, in patients with refractory metastatic urothelial carcinoma: results from a multicenter, phase ib study,” _Journal of Clinical Oncology_ , vol. 35, no. 19, p. 2117, 2017.
  * [13] G. L. Banna, T. Oliver, F. Rundo, U. Malapelle, F. Fraggetta, M. L. Libra, and A. Addeo, “The promise of digital biopsy for the prediction of tumor molecular features and clinical outcomes associated with immunotherapy,” _Frontiers in medicine_ , vol. 6, p. 172, 2019.
  * [14] P. Lambin, E. Rios-Velazquez, R. Leijenaar, S. Carvalho, R. G. van Stiphout, P. Granton, C. M. Zegers, R. Gillies, R. Boellard, A. Dekker _et al._ , “Radiomics: Extracting more information from medical images using advanced feature analysis,” _European Journal of Cancer_ , vol. 48, no. 4, pp. 441–446, 2012.
  * [15] G. Wang, K.-M. Lam, Z. Deng, and K.-S. Choi, “Prediction of mortality after radical cystectomy for bladder cancer by machine learning techniques,” _Computers in biology and medicine_ , vol. 63, pp. 124–132, 2015.
  * [16] S. S. Garapati, L. Hadjiiski, K. H. Cha, H.-P. Chan, E. M. Caoili, R. H. Cohan, A. Weizer, A. Alva, C. Paramagul, J. Wei _et al._ , “Urinary bladder cancer staging in ct urography using machine learning,” _Medical physics_ , vol. 44, no. 11, pp. 5814–5823, 2017.
  * [17] Z. Hasnain, J. Mason, K. Gill, G. Miranda, I. S. Gill, P. Kuhn, and P. K. Newton, “Machine learning models for predicting post-cystectomy recurrence and survival in bladder cancer patients,” _PloS one_ , vol. 14, no. 2, 2019\.
  * [18] K. H. Cha, L. Hadjiiski, R. K. Samala, H.-P. Chan, E. M. Caoili, and R. H. Cohan, “Urinary bladder segmentation in ct urography using deep-learning convolutional neural network and level sets,” _Medical physics_ , vol. 43, no. 4, pp. 1882–1896, 2016.
  * [19] M. Gordon, L. Hadjiiski, K. Cha, H.-P. Chan, R. Samala, R. H. Cohan, and E. M. Caoili, “Segmentation of inner and outer bladder wall using deep-learning convolutional neural network in ct urography,” in _Medical Imaging 2017: Computer-Aided Diagnosis_ , vol. 10134. International Society for Optics and Photonics, 2017, p. 1013402.
  * [20] X. Ma, L. Hadjiiski, J. Wei, H.-P. Chan, K. Cha, R. H. Cohan, E. M. Caoili, R. Samala, C. Zhou, and Y. Lu, “2d and 3d bladder segmentation using u-net-based deep-learning,” in _Medical Imaging 2019: Computer-Aided Diagnosis_ , vol. 10950. International Society for Optics and Photonics, 2019, p. 109500Y.
  * [21] E. Shkolyar, X. Jia, T. C. Chang, D. Trivedi, K. E. Mach, M. Q.-H. Meng, L. Xing, and J. C. Liao, “Augmented bladder tumor detection using deep learning,” _European urology_ , vol. 76, no. 6, pp. 714–718, 2019.
  * [22] E. Wu, L. M. Hadjiiski, R. K. Samala, H.-P. Chan, K. H. Cha, C. Richter, R. H. Cohan, E. M. Caoili, C. Paramagul, A. Alva _et al._ , “Deep learning approach for assessment of bladder cancer treatment response,” _Tomography_ , vol. 5, no. 1, p. 201, 2019.
  * [23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [24] K. H. Cha, L. Hadjiiski, H.-P. Chan, A. Z. Weizer, A. Alva, R. H. Cohan, E. M. Caoili, C. Paramagul, and R. K. Samala, “Bladder cancer treatment response assessment in ct using radiomics with deep-learning,” _Scientific reports_ , vol. 7, no. 1, pp. 1–12, 2017.
  * [25] F. Rundo, C. Spampinato, G. L. Banna, and S. Conoci, “Advanced deep learning embedded motion radiomics pipeline for predicting anti-pd-1/pd-l1 immunotherapy response in the treatment of bladder cancer: Preliminary results,” _Electronics_ , vol. 8, no. 10, p. 1134, 2019.
  * [26] E. A. Eisenhauer, P. Therasse, J. Bogaerts, L. H. Schwartz, D. Sargent, R. Ford, J. Dancey, S. Arbuck, S. Gwyther, M. Mooney _et al._ , “New response evaluation criteria in solid tumours: revised recist guideline (version 1.1),” _European journal of cancer_ , vol. 45, no. 2, pp. 228–247, 2009.
  * [27] L. O. Chua and L. Yang, “Cellular neural networks: Theory,” _IEEE Transactions on circuits and systems_ , vol. 35, no. 10, pp. 1257–1272, 1988.
  * [28] P. Arena, S. Baglio, L. Fortuna, and G. Manganaro, “Dynamics of state controlled cnns,” in _1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96_ , vol. 3. IEEE, 1996, pp. 56–59.
  * [29] L. O. Chua and T. Roska, “The cnn universal machine. i. the architecture,” in _CNNA’92 Proceedings Second International Workshop on Cellular Neural Networks and Their Applications_. IEEE, 1992, pp. 1–10.
  * [30] L. Fortuna, P. Arena, D. Balya, and A. Zarandy, “Cellular neural networks: a paradigm for nonlinear spatio-temporal processing,” _IEEE Circuits and Systems magazine_ , vol. 1, no. 4, pp. 6–21, 2001.
  * [31] T. Roska and L. O. Chua, “Cellular neural networks with nonlinear and delay-type template elements,” in _IEEE International Workshop on Cellular Neural Networks and their Applications_ , 1990, pp. 12–25.
  * [32] C.-C. Lee and J. P. de Gyvez, “Color image processing in a cellular neural-network environment,” _IEEE Transactions on neural networks_ , vol. 7, no. 5, pp. 1086–1098, 1996.
  * [33] W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. E. Alsaadi, “A survey of deep neural network architectures and their applications,” _Neurocomputing_ , vol. 234, pp. 11 – 26, 2017. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0925231216315533
  * [34] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” 2017.