# Just a Second – Scheduling Thousands of Time-Triggered Streams in Large-
Scale Networks

Heiko Geppert
IPVS
University of Stuttgart
Germany
<EMAIL_ADDRESS>
&Frank Dürr
IPVS
University of Stuttgart
Germany
<EMAIL_ADDRESS>
&Sukanya Bhowmik
IPVS
University of Stuttgart
Germany
<EMAIL_ADDRESS>
&Kurt Rothermel
IPVS
University of Stuttgart
Germany
<EMAIL_ADDRESS>
0000-0002-5146-0184 0000-0002-3470-7712 0000-0001-5045-4252
0000-0001-8986-8241

###### Abstract

Deterministic real-time communication with bounded delay is an essential
requirement for many safety-critical cyber-physical systems, and has received
much attention from major standardization bodies such as IEEE and IETF. In
particular, Ethernet technology has been extended by time-triggered scheduling
mechanisms in standards like TTEthernet and Time-Sensitive Networking.
Although the scheduling mechanisms have become part of standards, the traffic
planning algorithms to create time-triggered schedules are still an open and
challenging research question due to the problem’s high complexity. In
particular, so-called plug-and-produce scenarios require the ability to extend
schedules on the fly within seconds. The need for scalable scheduling and
routing algorithms is further supported by large-scale distributed real-time
systems like smart energy grids with tight communication requirements. In this
paper, we tackle this challenge by proposing two novel algorithms called
Hierarchical Heuristic Scheduling (H2S) and Cost-Efficient Lazy Forwarding
Scheduling (CELF) to calculate time-triggered schedules for TTEthernet. H2S
and CELF are highly efficient and scalable, calculating schedules for more
than 45,000 streams on random networks with 1,000 bridges as well as a
realistic energy grid network within sub-seconds to seconds.

_Keywords_ deterministic networks $\cdot$ stream scheduling $\cdot$ TTEthernet

## References

  * [1] B. Varga, J. Farks, D. Fedyk, L. Berger, and D. Brungard, “The quick and the dead: The rise of deterministic networks,” ComSoc Technology News (blog), 2021\. [Online]. Available: https://www.comsoc.org/publications/ctn/quick-and-dead-rise-deterministic-networks
  * [2] IEC SC65C/WG18 and IEEE 802.1 TSN TG. (2021) IEC/IEEE 60802 TSN profile for industrial automation. [Online]. Available: https://1.ieee802.org/tsn/iec-ieee-60802/
  * [3] S. Abbasloo, Y. Xu, and H. J. Chao, “Hyline: a simple and practical flow scheduling for commodity datacenters,” in _2018 IFIP Networking Conference (IFIP Networking) and Workshops_ , 2018, pp. 1–9.
  * [4] L. Behera, “A fault-tolerant time-triggered scheduling algorithm of mixed-criticality systems,” _Computing_ , vol. 104, no. 3, pp. 577–599, 2022\.
  * [5] A. Finzi and R. Serna Oliver, “General Framework for Routing, Scheduling and Formal Timing Analysis in Deterministic Time-Aware Networks,” in _34th Euromicro Conference on Real-Time Systems (ECRTS 2022)_ , ser. Leibniz International Proceedings in Informatics (LIPIcs), M. Maggio, Ed., vol. 231\. Dagstuhl, Germany: Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2022, pp. 8:1–8:23. [Online]. Available: https://drops.dagstuhl.de/opus/volltexte/2022/16325
  * [6] M. Pahlevan and R. Obermaisser, “Genetic algorithm for scheduling time-triggered traffic in time-sensitive networks,” in _2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1, 2018, pp. 337–344.
  * [7] M. Pahlevan, N. Tabassam, and R. Obermaisser, “Heuristic list scheduler for time triggered traffic in time sensitive networks,” _SIGBED Rev._ , vol. 16, no. 1, p. 15–20, feb 2019. [Online]. Available: https://doi.org/10.1145/3314206.3314208
  * [8] M. Vlk, K. Brejchová, Z. Hanzálek, and S. Tang, “Large-scale periodic scheduling in time-sensitive networks,” _Computers & Operations Research_, vol. 137, p. 105512, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0305054821002549
  * [9] L. Wisniewski, M. Schumacher, J. Jasperneite, and C. Diedrich, “Increasing flexibility of time triggered ethernet based systems by optimal greedy scheduling approach,” in _2015 IEEE 20th Conference on Emerging Technologies Factory Automation (ETFA)_ , 2015, pp. 1–6.
  * [10] F. Dürr and N. G. Nayak, “No-wait packet scheduling for IEEE time-sensitive networks (TSN),” in _Proceedings of the 24th International Conference on Real-Time Networks and Systems_ , ser. RTNS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 203–212. [Online]. Available: https://doi.org/10.1145/2997465.2997494
  * [11] E. Schweissguth, P. Danielis, D. Timmermann, H. Parzyjegla, and G. Mühl, “ILP-based joint routing and scheduling for time-triggered networks,” in _Proceedings of the 25th International Conference on Real-Time Networks and Systems_ , ser. RTNS ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 8–17. [Online]. Available: https://doi.org/10.1145/3139258.3139289
  * [12] D. Hellmanns, A. Glavackij, J. Falk, R. Hummen, S. Kehrer, and F. Dürr, “Scaling TSN scheduling for factory automation networks,” in _2020 16th IEEE International Conference on Factory Communication Systems (WFCS)_ , 2020, pp. 1–8.
  * [13] N. G. Nayak, F. Dürr, and K. Rothermel, “Time-sensitive software-defined network (TSSDN) for real-time applications,” in _Proceedings of the 24th International Conference on Real-Time Networks and Systems_ , ser. RTNS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 193–202. [Online]. Available: https://doi.org/10.1145/2997465.2997487
  * [14] P. Pop, M. L. Raagaard, S. S. Craciunas, and W. Steiner, “Design optimisation of cyber-physical distributed systems using IEEE time-sensitive networks,” _IET Cyber-Physical Systems: Theory & Applications_, vol. 1, no. 1, pp. 86–94, 2016.
  * [15] A. C. T. d. Santos, B. Schneider, and V. Nigam, “Tsnsched: Automated schedule generation for time sensitive networking,” in _2019 Formal Methods in Computer Aided Design (FMCAD)_ , 2019, pp. 69–77.
  * [16] W. Steiner, “An evaluation of smt-based schedule synthesis for time-triggered multi-hop networks,” in _2010 31st IEEE Real-Time Systems Symposium_ , 2010, pp. 375–384.
  * [17] KUKA Aktiengesellschaft, “Hello industry 4.0, we go digital,” _Augsburg: Eberl Print GmbH_ , pp. 1–63, 2016.
  * [18] X. Tang, Q. Li, Y. Zuo, G. Lu, and H. Xiong, “Online schedule of sporadic life-critical traffic in ttethernet,” in _2019 IEEE 5th International Conference on Computer and Communications (ICCC)_ , 2019, pp. 1312–1316.
  * [19] Y. Huang, S. Wang, T. Huang, B. Wu, Y. Wu, and Y. Liu, “Online routing and scheduling for time-sensitive networks,” in _2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)_ , 2021, pp. 272–281.
  * [20] J. Falk, H. Geppert, F. Dürr, S. Bhowmik, and K. Rothermel, “Dynamic qos-aware traffic planning for time-triggered flows in the real-time data plane,” _IEEE Transactions on Network and Service Management_ , vol. 19, no. 2, pp. 1807–1825, 2022.
  * [21] M. L. Raagaard, P. Pop, M. Gutiérrez, and W. Steiner, “Runtime reconfiguration of time-sensitive networking (TSN) schedules for fog computing,” in _2017 IEEE Fog World Congress (FWC)_ , 2017, pp. 1–6.
  * [22] J. Krolikowski, S. Martin, P. Medagliani, J. Leguay, S. Chen, X. Chang, and X. Geng, “Joint routing and scheduling for large-scale deterministic ip networks,” _Computer Communications_ , vol. 165, pp. 33–42, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0140366420319642
  * [23] F. Heilmann and G. Fohler, “Size-based queuing: An approach to improve bandwidth utilization in TSN networks,” _SIGBED Rev._ , vol. 16, no. 1, p. 9–14, feb 2019. [Online]. Available: https://doi.org/10.1145/3314206.3314207
  * [24] C. Zhang, Y. Wang, R. Yao, B. Zhou, L. Cheng, Y. Xu, X. Li, J. Cheng, and B. Liu, “Packet-size aware scheduling algorithms in guard band for time sensitive networking,” _CCF Transactions on Networking_ , vol. 3, no. 1, pp. 4–20, 2020.
  * [25] S. Vitturi and F. Tramarin, “Energy efficient ethernet for real-time industrial networks,” _IEEE Transactions on Automation Science and Engineering_ , vol. 12, no. 1, pp. 228–237, 2015.
  * [26] F. Prinz, M. Schoeffler, A. Lechler, and A. Verl, “Dynamic real-time orchestration of i4.0 components based on time-sensitive networking,” _Procedia CIRP_ , vol. 72, pp. 910–915, 2018, 51st CIRP Conference on Manufacturing Systems. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2212827118303329
  * [27] D. Bujosa, M. Ashjaei, A. V. Papadopoulos, T. Nolte, and J. Proenza, “Hermes: Heuristic multi-queue scheduler for tsn time-triggered traffic with zero reception jitter capabilities,” in _Proceedings of the 30th International Conference on Real-Time Networks and Systems_ , ser. RTNS 2022\. New York, NY, USA: Association for Computing Machinery, 2022, p. 70–80. [Online]. Available: https://doi.org/10.1145/3534879.3534906
  * [28] S. S. Craciunas, R. S. Oliver, M. Chmelík, and W. Steiner, “Scheduling real-time communication in IEEE 802.1Qbv time sensitive networks,” in _Proceedings of the 24th International Conference on Real-Time Networks and Systems_ , ser. RTNS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 183–192. [Online]. Available: https://doi.org/10.1145/2997465.2997470
  * [29] R. Serna Oliver, S. S. Craciunas, and W. Steiner, “IEEE 802.1Qbv gate control list synthesis using array theory encoding,” in _2018 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS)_ , 2018, pp. 13–24.
  * [30] J. Falk, F. Dürr, and K. Rothermel, “Exploring practical limitations of joint routing and scheduling for TSN with ILP,” in _2018 IEEE 24th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA)_ , 2018, pp. 136–146.
  * [31] M. Vlk, Z. Hanzálek, and S. Tang, “Constraint programming approaches to joint routing and scheduling in time-sensitive networks,” _Computers & Industrial Engineering_, vol. 157, p. 107317, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0360835221002217
  * [32] A. Minaeva, D. Roy, B. Akesson, Z. Hanzálek, and S. Chakraborty, “Control performance optimization for application integration on automotive architectures,” _IEEE Transactions on Computers_ , vol. 70, no. 7, pp. 1059–1073, 2021.
  * [33] E. Schweissguth, D. Timmermann, H. Parzyjegla, P. Danielis, and G. Mühl, “ILP-based routing and scheduling of multicast realtime traffic in time-sensitive networks,” in _2020 IEEE 26th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA)_ , 2020, pp. 1–11.
  * [34] D. Tamas-Selicean, P. Pop, and W. Steiner, “Synthesis of communication schedules for ttethernet-based mixed-criticality systems,” in _Proceedings of the Eighth IEEE/ACM/IFIP International Conference on Hardware/Software Codesign and System Synthesis_ , ser. CODES+ISSS ’12. New York, NY, USA: Association for Computing Machinery, 2012, p. 473–482. [Online]. Available: https://doi.org/10.1145/2380445.2380518
  * [35] M. L. Raagaard and P. Pop, “Optimization algorithms for the scheduling of IEEE 802.1 time-sensitive networking (TSN),” _Tech. Univ. Denmark, Lyngby, Denmark, Tech. Rep_ , 2017.
  * [36] J. Falk, F. Dürr, and K. Rothermel, “Time-triggered traffic planning for data networks with conflict graphs,” in _2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS)_ , 2020, pp. 124–136.
  * [37] C. Gärtner, A. Rizk, B. Koldehofe, R. Hark, R. Guillaume, and R. Steinmetz, “Leveraging flexibility of time-sensitive networks for dynamic reconfigurability,” in _2021 IFIP Networking Conference (IFIP Networking)_. IEEE, 2021, pp. 1–6.
  * [38] C. Gärtner, A. Rizk, B. Koldehofe, R. Guillaume, R. Kundel, and R. Steinmetz, “On the incremental reconfiguration of time-sensitive networks at runtime,” in _2022 IFIP Networking Conference (IFIP Networking)_. IEEE, 2022, pp. 1–9.
  * [39] “IEEE Standard for Local and Metropolitan Area Networks–Bridges and Bridged Networks – Amendment 31: Stream Reservation Protocol (SRP) Enhancements and Performance Improvements,” _IEEE Std 802.1Qcc-2018 (Amendment to IEEE Std 802.1Q-2018 as amended by IEEE Std 802.1Qcp-2018)_ , pp. 1–208, 2018. [Online]. Available: https://ieeexplore.ieee.org/document/8514112
  * [40] J. Leskovec, A. Krause, C. Guestrin, C. Faloutsos, J. VanBriesen, and N. Glance, “Cost-effective outbreak detection in networks,” in _Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , ser. KDD ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 420–429. [Online]. Available: https://doi.org/10.1145/1281192.1281239
  * [41] J. Kim, S. Kim, and H. Yu, “Scalable and parallelizable processing of influence maximization for large-scale social networks?” in _2013 IEEE 29th International Conference on Data Engineering (ICDE)_ , 2013, pp. 266–277.
  * [42] J. Tang, X. Tang, and J. Yuan, “Influence maximization meets efficiency and effectiveness: A hop-based approach,” in _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017_ , ser. ASONAM ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 64–71. [Online]. Available: https://doi.org/10.1145/3110025.3110041
  * [43] H. Geppert, S. Bhowmik, and K. Rothermel, “Large-scale influence maximization with the influence maximization benchmarker suite,” in _Proceedings of the 4th ACM SIGMOD Joint International Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA)_, ser. GRADES-NDA ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3461837.3464510
  * [44] N. G. Nayak, F. Dürr, and K. Rothermel, “Routing algorithms for ieee802.1qbv networks,” _SIGBED Rev._ , vol. 15, no. 3, p. 13–18, aug 2018\. [Online]. Available: https://doi.org/10.1145/3267419.3267421
  * [45] A. A. Syed, S. Ayaz, T. Leinmüller, and M. Chandra, “Dynamic scheduling and routing for TSN based in-vehicle networks,” in _2021 IEEE International Conference on Communications Workshops (ICC Workshops)_ , 2021, pp. 1–6.
  * [46] A. B. Birchfield, T. Xu, K. M. Gegner, K. S. Shetye, and T. J. Overbye, “Grid structural characteristics as validation criteria for synthetic networks,” _IEEE Transactions on Power Systems_ , vol. 32, no. 4, pp. 3258–3265, 2017\.
  * [47] X. Zhu, M. H. F. Wen, V. O. K. Li, and K.-C. Leung, “Optimal pmu-communication link placement for smart grid wide-area measurement systems,” _IEEE Transactions on Smart Grid_ , vol. 10, no. 4, pp. 4446–4456, 2019.
  * [48] V. C. Gungor, D. Sahin, T. Kocak, S. Ergut, C. Buccella, C. Cecati, and G. P. Hancke, “A survey on smart grid potential applications and communication requirements,” _IEEE Transactions on Industrial Informatics_ , vol. 9, no. 1, pp. 28–42, 2013.
  * [49] Y. Yan, Y. Qian, H. Sharif, and D. Tipper, “A survey on smart grid communication infrastructures: Motivations, requirements and challenges,” _IEEE Communications Surveys & Tutorials_, vol. 15, no. 1, pp. 5–20, 2013\.
  * [50] G. López, J. Moreno, H. Amarís, and F. Salazar, “Paving the road toward smart grids through large-scale advanced metering infrastructures,” _Electric Power Systems Research_ , vol. 120, pp. 194–205, 2015, smart Grids: World’s Actual Implementations. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0378779614001862
  * [51] M. Kerai, “Smart meter statistics in great britain: Quarterly report to end september 2022,” online, 2022. [Online]. Available: https://www.gov.uk/government/statistics/smart-meters-in-great-britain-quarterly-update-september-2022
  * [52] J. Y. Yen, “Finding the K Shortest Loopless Paths in a Network,” _Management Science_ , vol. 17, no. 11, pp. 712–716, 1971.
  * [53] H. Kopetz and W. Steiner, _Real-time systems: design principles for distributed embedded applications_. Springer Nature, 2022.


captionUnsupported document class

# MEDS-Net: Multi-Encoder based Self-Distilled Network with Bidirectional
Maximum Intensity Projections for Lung Nodule Detection

Muhammad Usman  Center for Artificial Intelligence in Medicine and Imaging,
HealthHub Co. Ltd., Seoul, 06524, South Korea  Department of Computer Science
and Engineering, Seoul National University, Seoul, 08826, Republic of Korea
Azka Rehman  Center for Artificial Intelligence in Medicine and Imaging,
HealthHub Co. Ltd., Seoul, 06524, South Korea  Corresponding Author:
<EMAIL_ADDRESS>Abdullah Shahid  Center for Artificial Intelligence in
Medicine and Imaging, HealthHub Co. Ltd., Seoul, 06524, South Korea  Siddique
Latif  University of Southern Queensland, Springfield, Queensland 4300,
Australia  Shi‑Sub Byon  Center for Artificial Intelligence in Medicine and
Imaging, HealthHub Co. Ltd., Seoul, 06524, South Korea  Byoung-Dai Lee  School
of Computer Science and Engineering, Kyonggi University, Suwon, 16227, South
Korea  Sung‑Hyun Kim  Center for Artificial Intelligence in Medicine and
Imaging, HealthHub Co. Ltd., Seoul, 06524, South Korea  Byung‑il Lee  Center
for Artificial Intelligence in Medicine and Imaging, HealthHub Co. Ltd.,
Seoul, 06524, South Korea  Yeong‑Gil Shin  Department of Computer Science and
Engineering, Seoul National University, Seoul, 08826, Republic of Korea

###### Abstract

Early detection of lung cancer is an effective way to improve the cure rate of
patients. Therefore, it is crucial to accurately detect lung nodules in
computed tomography (CT) images for early diagnosis of lung cancer. However,
developing an accurate computer-aided detection (CADe) system for lung nodule
detection is a challenging task owing to the complex lung’s structure and the
heterogeneity of the lung nodules. In this study, we propose a lung nodule
detection scheme motivated by the clinical workflow of radiologists, which
combines CT scan sub-volume with bidirectional maximum intensity projection
(MIP) images in a single architecture. For this purpose, we designed a novel
multi-encoder based network (MEDS-Net) that exploits self-distillation to
effectively learn from three different types of inputs, including 3D sub-
volume, forward, and backward MIP images. MEDS-Net exploits three different
encoders to effectively extract the distinctive insights which are passed to
the decoder block. Further, the decoder block leverages the self-distillation
mechanism by connecting the distillation block, which contains four auxiliary
lung nodule detectors. It helps to expedite the convergence and improves the
learning ability of the proposed architecture. Finally, the proposed scheme
reduces the false positives by complementing the main detector with auxiliary
detectors. The proposed scheme has been rigorously evaluated on 888 scans of
the LIDC-IDRI dataset and obtained a competition performance metric (CPM)
score of 93.6%. The results demonstrate that incorporating bidirectional MIP
images enables MEDS-Net to accurately detect lung nodules and reduce the
number of false positives using auxiliary detectors.

##  I Introduction

Lung cancer is the most frequent cancer type, with the highest mortality rates
globally [2]. The survival rate of patients highly depends upon the early
detection of pulmonary nodules [3]. For diagnosing pulmonary nodules, computed
tomography (CT) has been extensively employed and proven effective [4].
However, it is often a time-consuming and tedious job to manually identify
nodules in CT scans. Radiologists need to read the CT scans slice by slice,
and depending upon the slice thickness, a chest CT may contain hundreds of
slices. Additionally, manual lung nodule detection is error-prone [5]. To
alleviate this problem, computer-aided detection (CADe) systems play an
important role in aiding radiologists in accurately diagnosing lung cancer.

Numerous efforts have been made in designing CADe systems; however, developing
a highly accurate and robust CADe system is still challenging due to the
heterogeneity of lung nodules and the high degree of similarity between the
lung nodules and their surrounding tissues. Deep learning (DL) based
techniques have made vast inroads in various medical imaging, including lung
nodule detection [6, 7]. In DL-based CADe systems, convolutional neural
networks (CNNs) have been extensively employed in 2D and 3D to achieve
significantly improved performance [8]. Such CADe system typically consists of
two stages: screening and false positives (FPs) reduction. The former is
typically applied to identify suspicious regions (i.e., nodule candidates) in
a patient’s exam. The latter stage is to reduce the FPs by applying the
additional CNNs[9, 10, 11, 12]. Although such CADe systems demonstrated
promising performance for lung nodule detection, these techniques add
computational complexity by increasing the number of CNNs to train and are
more prone to error as nodules missed at the first stage cannot be detected at
the second stage. Subsequently, the sensitivity at the detection stage is kept
high, significantly increasing the FPs and the load for the FPs reduction
stage.

To overcome the challenges associated with dual-stage based CADe systems, a
few studies proposed single-stage frameworks to detect the lung nodules [13,
14, 15]. Such techniques considerably improve the computational complexities
and ease the training and optimization process; however, they suffer from a
low performance that limits their real-time clinical employment. The
disconnectivity of existing CADe systems with clinical workflow is another
reason for not being contentedly opted for by clinical practitioners. In the
real-time clinical workflow, mostly, radiologists first examine the maximum
intensity projection (MIP) images [16] of CT scan to roughly screen the nodule
candidates and use the raw CT scan slices to finalize the decision. Only a few
studies [17, 18] utilized the MIP images to improve lung nodule detection
performance and comply with the clinical workflow. For instance, Masood et al.
[17] leveraged MIP to extract meaningful insights of lung nodules from 2D
views (i.e., axial, coronal, and sagittal) and used multidimensional region-
based fully convolutional neural network based CAD system for lung nodule
detection and classification, respectively. Similarly, Zheng et al. [18]
attempted to follow the clinical workflow by MIP with different slab
thicknesses (i.e., 5 mm, 10 mm, 15 mm) to train four different UNet
architectures and merged their outputs for nodule candidates detection.
Additionally, they also trained two different-sized DNNs for false positive
reduction. Overall, they improve the performance, however, the ensemble of six
different networks makes the CADe system very complex. In contrast, we propose
a single-stage system that exploits the 3D patch of CT scan along with MIP
images to detect the lung nodule. We highlight the difference between our work
with the existing work in Table LABEL:litraturer. Below is a summary of the
main contributions of this study in contrast to the existing literature.

  1. 1.

None of the studies have used self-distillation for lung nodule detection. We
design a novel single-stage multi-encoder-based CNN that utilizes the self-
distillation mechanism and improves the performance compared to state-of-the-
art single-stage as well as most dual-stage frameworks on the public LIDC-IDRI
dataset,

  2. 2.

This study is the first to exploit the outputs from auxiliary branches of the
same architecture to complement the primary network for false positive
reduction.

  3. 3.

While some studies used MIP with CT scan for lung nodule detection, however,
no studies use bidirectional MIP. We propose to utilize bidirectional MIP
images of three slab thicknesses, i.e., 3, 5, and 10 mm, for lung nodule
detection, which improves the network’s ability to distinguish nodules from
non-nodular structures in the lung region.

Figure 1: The illustration of our proposed computer-aided detection system.
(a) Multi-encoder based self-distilled network (MEDS-Net) which incorporates
three different inputs, i.e., 3D sub-volume of CT scan, forward and backward
MIP images of three slab thickness (i.e., 3, 5 and 10 mm). (b) The
architecture of 3D dense block which transforms the 3D sub-volume of
normalized thoracic CT scans into three-channel latent representation. (c) The
architecture of dense unit utilized in the dense block. (d) Describing the
architecture of attention unit used in residual connections from encoders to
decoder block. Gating signal and feature maps come from the previous De-Conv
block and the concatenated encoders features, respectively. (e) The
architecture of convolutional block. (f) The architecture of de-convolutional
block. (g) The architecture of detector block.

##  II Proposed Methodology

In this section, we describe the proposed framework for lung nodule detection
in abdominal CT scans, which emulate the clinical practices of radiologists
who utilize maximum intensity projection (MIP) images [16]. Our proposed
multi-encoder based self-distilled network (MEDS-Net) is designed to fully
exploit the MIP images along with a 3D patch of CT scan as shown in Fig. 1(a).
To further extend the ability of MEDS-Net to detect the lung nodules, we
enrich the inputs by extracting the MIP images from both forward and backward
directions. Most importantly, the proposed MEDS-Net leverages the self-
distillation mechanism and auxiliary outputs to optimize the learning process
and reduce false positives at the inference stage, respectively. The details
of each component of the proposed framework have been described in the
following subsections.

Figure 2: Demonstration of bidirectional MIP images. (a) Process of generating
the forward and backward MIP images. (b) Example of forward and backward MIP
of thickness 3 mm, 5 mm, 10 mm (left to right).

###  II-A Bidirectional Maximum Intensity Projection

Maximum intensity projection (MIP) image consists of maximum values, which
come across the plane of projection, i.e., the z-axis. MIP images of thoracic
CT scans have proven extremely effective for lung nodule detection [19];
therefore, it has been utilized by radiologists to quickly screen lung
nodules. MIP images can be generated of various slab thicknesses; however,
research has shown that slab thicknesses up to 10 mm are the most effective
ones, and increasing the slab thickness further reduces the detection
sensitivity [20]. Subsequently, in this work, we utilize MIP images with three
slab thicknesses, i.e., 3 mm, 5 mm, and 10 mm. The generation of a MIP image
can be described as follows:

$I_{x,y}=\max_{\Delta z}D_{x,y,\Delta z}$ (1)

where, $I_{x,y}$ and $D_{x,y,\Delta z}$ represent the generated MIP image and
sub-volume or slab of CT scan, respectively. Here, $\Delta z$ denotes the slab
thickness, which in our case is set to 3, 5, and 10 mm. Traditionally, MIP is
taken in one direction, which helps to distinguish nodules from the vessels
penetrating in one direction only. In this work, we take it one step further
to improve the visibility of nodules; bidirectional MIP images are extracted
to improve the nodules’ visibility. The process of generating the
bidirectional MIP images, i.e., forward and backward directions, with the slab
thicknesses of 3 mm, 5 mm, and 10 mm, has been demonstrated in Fig. 2. We also
normalize the slice thickness along the z-axis of the raw CT scan to 1 mm
while keeping the original pixel spacing in xy-plane for 3D patch-based input.

###  II-B Multi-Encoder based Self Distilled Network

The proposed multi-encoder based self distilled network (MEDS-Net) is an end-
to-end lung nodule detection framework which is demonstrated in Fig. 1(a) and
summarized as follows: (1) first 3D slab consisting of a total of 11 slices
(i.e., $11\times 256\times 256$) of 1 mm thickness is fed to dense block which
squeeze the representation into three channels by using dense units; (2) the
compressed features from dense block along with forward and backward MIP
images are inputted into encoder block which contains three deep encoders to
extract the meaningful features from each three-channel input; (3) the
extracted features at various levels of three encoders are concatenated to
combine with the corresponding decoder layers via attention units to decoder
block; (4) decoder block utilizes the deconvolutional layers to up-sample the
latent features at each stage that are utilized as gating signals to attention
units. At various levels of the decoder, the features are reshaped to the
higher dimensions to make same size as the input image to acquire the
segmentation result; (5) finally, outputs at four different levels have been
up-sampled by employing deconvolutional layers in a deep supervision block.

####  II-B1 Dense block

To incorporate the spatial, sequential information crucial to detect the small
nodules, we input a patch of the scan to the proposed architecture.
Concretely, the five adjacent slices from both sides of the central slice,
i.e., the 3D patch of depth $11\times 256\times 256$, is extracted and
inputted into 3D dense block, which is shown in Fig. 1(b). 3D dense block
takes 11 slices of 1 mm thickness as input and transforms into the meaningful
latent representative of the same dimension as three MIP images, i.e.,
$3\times 256\times 256$. 3D dense block is composed of four sets of dense
units followed by the max-pooling layers, and the fifth dense unit is followed
by the reshaping layer, which squeezes the information to desired dimensions.
Fig. 1(c) shows the architecture of each dense unit which is composed of five
sets of 3D convolution, Relu activation, and batch normalization. The output
of each set is propagated to all the next sets using skip connections. After
each dense unit features are down-sampled along the depth, subsequently, at
the end of the dense block, we have three channels of feature maps. These
feature maps are further fed into the encoder block to further refine the
features pertaining to nodule detection.

####  II-B2 Encoder block

In Fig. 1(a), the overall structure of encoder block has been demonstrated,
which includes three coding paths, namely, 3D patch and forward and backward
MIPs encoders. All three encoders are kept identical and inspired by
conventional UNet encoder [21]. Nevertheless, our encoders are deeper than the
encoder in the standard UNet to effectively learn meaningful insights from
lung structure which are crucial to distinguish the lung nodules. Each encoder
consists of six Conv-Blocks which down-sample the input to lower dimensions
while extracting the meaningful features. As shown in Fig. 1(f), Conv-Block
has two stages; the first stage has two 2D convolutional layers followed by a
batch normalization layer, then a skip connection is used to combine the input
with the output features of the first convolution stage. This type of short
skip connection tends to stabilize gradient updates. In the second stage, the
combined features are passed to the same set of layers as the first stage,
followed by an additional downsample layer. The width of each of our branches
is kept small to avoid overfitting. Our first Conv-Block contains eight
feature maps that get doubled after every downsampling layer, and eventually,
our final Conv-Block has 256 features. Finally, the output features from the
second to sixth Conv-Blocks of all three encoders are concatenated to input
into decoder block.

####  II-B3 Decoder block

The proposed architecture leverages rich information extracted from three
types of inputs, i.e., 3D sub-volume consisting of 1 mm thick CT slices,
forward, and backward MIP images. This information is extracted from encoders
at multiple scales by skip connections to feed into decoder block. In decoder
block, we exploit the attention gates [22] to enable the network to focus on
the expected nodule area in naive features coming from encoder block while
discarding the redundant ones. The attention units achieve this goal by
performing element-wise multiplication of input feature-maps and attention
coefficients to highlight important features as described in Fig. 1(d).
Overall, the decoder block consists of six deconvolution (DeConv) block which
up-samples the input features while extracting useful information at different
levels. Fig. 1(e) describes the architecture of DeConv block, which consists
of two stages. In the first stage, two convolution blocks are followed by a
batch normalization layer. Input and output features of stage one are combined
using a skip connection with the attention unit and forwarded to stage two.
The second stage is similar to the first one, followed by the up-sampling
layer. The decoder increases the dimension of features step by step till the
final deconvolution block, which has the same feature dimension as the ground
truth mask. Further, we extract the outputs of each DeConv-Block to feed
forward to the distillation block, which utilizes these features to
reconstruct the nodule masks.

###  II-C Self-Distillation Mechanism

The proposed framework employs a self-distillation mechanism to improve the
training process by using a self-distillation block as shown in Fig. 1(a),
which has been proven effective in optimizing the network’s performance for
classification tasks [23]. We extract the features from five DeConv-Blocks
within the decoder block to pass into the self-distillation blocks, which
consist of four auxiliary detectors and the main detector block. The main
detector is connected with the deepest DeConv-Block in the decoder block,
therefore, having access to enrich insights for lung nodule detection. Each
detector is responsible for immediately up-scaling the features from decoder
block to generate the lung nodule candidates. Each detector-block shares the
same architecture demonstrated in Fig 1(g). The first layer of the detector
block is an up-sampling layer that directly up-scale the features to the same
dimensions as inputs’ height and width, i.e., $256\times 256$. The second
layer is a 2D convolution layer with Relu activation, and the final layer is a
convolution layer of single filter and softmax activation.

During the training process, all the auxiliary detectors are trained as
student models via distillation from the main detector, which can be
conceptually regarded as the teacher model. To improve the learning ability of
student detectors in the proposed framework, we utilize three types of losses
during the training processes:
Loss 1: It is the dice loss that is calculated with labels of training data to
all the detector networks outputs. It provides equal opportunity to each
detector network to leverage the supervision from actual labels for optimizing
their weights. Loss 2: KL (Kullback-Leibler) divergence loss under the main
detector’s supervision, which acts as a teacher network. We utilize the
outputs of each auxiliary detector, i.e., softmax output, and output of the
main detector to compute the KL divergence. This loss helps distill the
knowledge from the main detector to the auxiliary detectors, forcing them to
learn to produce similar results as the deepest detector.
Loss 3: It is an L2 loss that is computed between the feature maps of the main
detector network and each auxiliary detector. The loss enables the auxiliary
detectors to distill the implicit knowledge in feature maps from the main
detector, which induces the auxiliary detectors’ feature maps to fit the
feature maps of the main detector.

###  II-D False Positive Reduction using Auxiliary Detectors

In contrast to previous studies, the false positive reduction is performed by
a separate network; our framework exploits the auxiliary detectors to reduce
the false positives at the time of inference. All the nodules detected by the
main detector are considered nodule candidates, and auxiliary detectors
complement the detection of the main detector to validate the nodule
candidate. Firstly, our main detector provides all the nodule candidates, and
we determine the 3D bounding box, which acts as region of the proposal ($RoP$)
for each nodule candidate while incorporating all the nodular voxels. The
similar $RoPs$ of the same dimensions and positions are extracted from the 3D
outputs of all four auxiliary detectors to detect the false positives. Given
that $n$ is the number of voxels in $RoP_{i}$ in the $ith$ detector, we can
define true positive determination criteria as follows:

$\displaystyle
isTP=thr\left(\frac{1}{nk}\sum_{i=1}^{k}\sum_{j=1}^{n}RoP_{i}(j),\tau\right)$
(2)

$\displaystyle thr(\theta,\tau)$
$\displaystyle=\left\\{\begin{array}[]{l}1,\theta>\tau\\\ 0,\text{ otherwise
}\end{array}\right.$ (3)

In the above equations, $k$ is the number of auxiliary detectors, which in our
case are four, and $\tau$ is the threshold of the nodule and non-nodule
probabilities. While $thr(.)$ is a binarized function to binarize the
possibility value $\theta$, which is in the range of 0 to 1. We normalize the
whole submissions of probabilities by $nk$ to limit the accumulation to 1. We
apply the same criteria to all the nodule candidates after the inference.

Figure 3: Description of the steps involved in the lung parenchyma
segmentation. (a) Raw thoracic CT image. (b) Mask of the air regions. (c) Mask
after the removal of irrelevant objects other than the lung region. (d)
Dilated mask of the lung parenchyma. (e) Image of segmented the lung
parenchyma.

##  III Experimental Setup

###  III-A Datasets

In this work, the lung image database consortium and image database resource
initiative (LIDC/IDRI) dataset [24] has been utilized for all the
experimentation. LIDC/IDRI dataset consists of 1018 thoracic computed
tomography (CT) scans acquired from 1010 different patients, which are
annotated to facilitate computer-aided systems on the assessment of lung
nodule detection, classification and quantification. In the LIDC/IDRI dataset,
slice thickness varies from 0.6 mm to 5.0 mm; however, we remove the scans
with slice thickness greater than 2.5 mm as such low-resolution scans are not
recommended for nodule screening [25]. LIDC/IDRI dataset is annotated in two
steps by four radiologists. Firstly, each radiologist annotated the nodules
individually, and later all the radiologists jointly analyzed each annotation
to reassess their annotations. For this study, we only utilized nodules having
diameters equal to or greater than 3 mm due to their clinical significance
[26]. In order to avoid possible outliers, we only considered the nodules
accepted by at least 3 out of 4 radiologists as the reference standard. After
applying the aforementioned criteria, 888 scans were left, which were utilized
for all the experimentation.

###  III-B Data Preprocessing

We preprocess all the scans prior to feeding them into the proposed MEDS-Net.
Our preprocessing consists of two stages; lung parenchyma segmentation and
scan normalization.

In the first stage, we automatically segment the lung parenchyma to exclude
irrelevant regions such as clothes, machine objects, tissues, spines, or ribs
in the scans. Fig. 3 demonstrates the steps followed to perform lung
segmentation. Initially, an air mask is created, and watershed algorithm [27]
is applied to remove the region other than the lung as shown in Fig. 3(b) and
3(c), respectively. In order to avoid missing wall-attached lesions that might
be nodules, we keep more boundary information by binary morphology operations,
i.e., closing and dilation. Finally, the binarized scan was masked with the
original scan.

In the second stage, we perform the scan normalization. Since the data set is
collected from various CT scanners in the LIDC/IDRI dataset, we set the window
level from -1000 HU to 400 HU and normalized images to the range between 0 and
1. We also normalized the slice thickness to 1 mm and cropped each slice
equally from all sides while keeping the lung at the center for normalizing to
$256\times 256$ dimensions.

###  III-C Training Strategy

We divide the whole dataset scans into eight subsets to perform 8-fold cross-
validation. For each fold, training, validation, and test set split are set to
62.5%, 12.5%, and 25% of the total scans, respectively. The batch size is set
to 3 due to the memory limitations of the GPU. We used adam optimizer with an
initial learning rate of 0.001 and first and second momentum of 0.9 for the
decay of the learning rate. We use early stopping with patience of 10 epochs
to avoid overfitting.

This study exploits three types of losses to train the model as described in
Section II-C, which can be written as

$Loss_{Total}=Loss_{1}+Loss_{2}+Loss_{3}$ (4)

To formulate these losses, we denote our main and auxiliary detectors by
$\theta_{m}$ and $\theta_{i/k}$ detectors, respectively. Where,
$i\in[\\![1,k]\\!]$ and $k$ is the number of auxiliary detectors.

$Loss_{1}=(1-\alpha)L_{DSC}(\theta_{m},y)+\sum_{i=1}^{k}L_{DSC}(\theta_{i/k},y)$
(5)

The first loss utilizes dice loss which is denoted as $L_{DSC}$ and can be
defined as:

$L_{DSC}=\frac{2.\mid x\cap y\mid}{\mid x\>\cup\>y\mid}$ (6)

The second loss Kullback-Leibler ($KL$) divergence between each auxiliary and
main detector that can be written as:

$Loss_{2}=\alpha.\sum_{i=1}^{k}KL(\theta_{i/k},\theta_{m})$ (7)

The third supervision comes from the features of the main deepest detector to
each auxiliary detector. Due to the difference in the size of features
inputted to each detector, additional convolution layers in detector blocks
are employed to align the dimensions prior to the extraction of feature maps
to calculate the feature loss. Feature loss can be formulated as:

$Loss_{3}=\lambda.\sum_{i=1}^{k}\mid F_{i/k},F_{m}\mid$ (8)

where $F_{i/k}$ and $F_{m}$ represents the features from $\theta_{i/k}$ and
$\theta_{m}$ detectors. While two hyper-parameters $\alpha$ and $\lambda$ are
used to moderate the role of each loss.

TABLE I: PERFORMANCE COMPARISON WITH OTHER COMPUTER-AIDED DETECTION SYSTEMS ON
THE LIDC/IDRI DATASET AT NODULE CANDIDATE DETECTION STAGE

CADe systems |  | Sensitivity
---
(%)
| Total
---
number of
candidates
| Average
---
number of
candidates
per scan
Zheng et al. [18] | 95.4 | 18,116 | 20.4
Setio et al. [28] | 31.8 | 258,075 | 290.6
Yuan et al. [29] | 94 | 44,627 | 50.25
Pereira et al. [30] | 98.15 | 44,111 | 49.67
Wang et al. [31] | 96.8 | 53, 484 | 60.2
Zhang et al. [32] | 100 | 45,939 | 51.7
Setio et al. [33] | 98.3 | 754,975 | 850.2
Tarres et al. [34] | 76.8 | 19,687 | 22.2
Our Method | 97.8 | 19,190 | 21.61

TABLE II: COMPLETE PIPELINE PERFORMANCE COMPARISON WITH OTHER COMPUTER-AIDED
DETECTION SYSTEMS ON THE LIDC/IDRI DATASET

CADe system | Year | Type of scheme | False positive per scan | CPM
---|---|---|---|---
0.125 | 0.25 | 0.5 | 1 | 2 | 4 | 8
Setio et al. [33] | 2017 | Dual-stage | 0.859 | 0.937 | 0.958 | 0.969 | 0.976 | 0.982 | 0.982 | 0.952
Wang et al. [31] | 2018 | Dual-stage | 0.788 | 0.847 | 0.895 | 0.934 | 0.952 | 0.959 | 0.963 | 0.903
Zhang et al. [32] | 2018 | Dual-stage | 0.89 | 0.931 | 0.944 | 0.949 | 0.965 | 0.972 | 0.976 | 0.947
Zheng et al. [18] | 2019 | Dual-stage | 0.876 | 0.899 | 0.912 | 0.927 | 0.942 | 0.948 | 0.953 | 0.922
Cao et al. [9] | 2020 | Dual-stage | 0.848 | 0.899 | 0.925 | 0.936 | 0.949 | 0.957 | 0.96 | 0.925
Li et al. [13] | 2020 | Single-stage | 0.6 | 0.674 | 0.751 | 0.824 | 0.85 | 0.853 | 0.859 | 0.773
Zhou et al. [12] | 2022 | Dual-stage | 0.742 | 0.84 | 0.8989 | 0.925 | 0.944 | 0.954 | 0.959 | 0.895
Zhao et al. [11] | 2022 | Dual-stage | 0.656 | 0.754 | 0.833 | 0.917 | 0.951 | 0.97 | 0.977 | 0.865
Agnes et al. [35] | 2022 | Dual-stage | 0.803 | 0.89 | 0.93 | 0.959 | 0.967 | 0.979 | 0.981 | 0.93
Zhu et al. [14] | 2022 | Single-stage | 0.782 | 0.834 | 0.893 | 0.917 | 0.932 | 0.952 | 0.956 | 0.895
Mei et al. [36] | 2022 | Dual-stage | 0.712 | 0.802 | 0.865 | 0.901 | 0.937 | 0.946 | 0.955 | 0.874
Guo et al. [37] | 2022 | Single-stage | 0.832 | 0.886 | 0.928 | 0.937 | 0.946 | 0.952 | 0.959 | 0.92
Our Framework | 2022 | Single-stage | 0.883 | 0.915 | 0.928 | 0.941 | 0.953 | 0.962 | 0.968 | 0.936

##  IV Experimental Results and Discussion

###  IV-A Performance Analysis

Typical CADe systems consist of two stages, i.e., candidate detection and
false positive reduction, and mostly two separate networks are trained to
achieve this goal. Nevertheless, in this study, the proposed framework
combines both stages by employing self-distillation during the training and
utilizing the auxiliary detectors for false positive reduction. For a
comprehensive analysis of the proposed CADe system, we compare our performance
with previously published methods for the candidate detection stage as well as
after the false positive reduction stage. Below we compare the performance of
the proposed MEDS-Net with multiple state-of-the-art studies.

####  IV-A1 Performance at nodule candidates detection stage

All the nodules detected by the main, the deepest detector, of the proposed
MEDS-Net are considered nodule candidates. The capability of nodule detection
of the proposed framework has been compared with the recently published
individual candidate detection systems. Table I summarizes the results of
candidate detection stages of previously published works. Setio et al. [28]
and Tarres et al. [34], employed classical algorithms based on machine
learning while Zheng et al. [18], Yuan et al. [29], Pereira et al. [30], Wang
et al. [31], Setio et al. [33], and our proposed framework utilizes the deep
learning based methods. Zhang et al. [32] developed a hybrid method by combing
classical techniques with the deep learning-based algorithm. Overall the
performance of deep learning-based methods is better than the conventional
algorithms. Zhang et al. [32], Setio et al. [28] and Pereira et al. [30]
achieved better detection rate. However, these studies had improved the true
positives at the cost of several-fold higher FPs. Similar to our work, Zheng
et al. [18] exploited the MIP images of various thicknesses to achieve 95%
sensitivity with 20.1 candidates per scan. This shows the effeteness of MIP
image for lung nodule detection. Although Zheng et al. [18] trained four
individual 2D networks with unidirectional MIP images, our framework with a
single network outperforms in terms of sensitivity by exploiting bidirectional
MIP images. The results demonstrate that incorporating bidirectional MIP
images in a self-distillation network significantly improves the proposed
architecture’s nodule detection capability.

####  IV-A2 Performance after false positives reduction

To provide a comparative analysis of the complete pipeline of the proposed
framework, we chose the published approaches which used the competition
performance metric (CPM) [38] for evaluation on the LIDC/IDRI dataset. Table
II summarizes the performance of these techniques. Among all the listed
methods, Setio et al. [33] demonstrated the best performance, which is
attributed to the combination of seven different nodule detection systems and
five false positive reduction systems with varied architectures to detect the
different types of nodules, such as subsolid, juxta-vascular, juxta-pleural
nodules, etc. Nevertheless, the proposed scheme is designed for all types of
nodules and consists of a single architecture. Our scheme exhibits comparable
performance and even better sensitivity for the small value of FPs per scan
(i.e., 0.125). Similarly, Zhang et al. [32] achieved better performance by
maintaining 100% sensitivity at the candidate detection stage with multi-scale
LoG filters, which cost more FPs compared to our scheme. Our scheme exploits
bidirectional MIPs with different levels of thickness to achieve comparable
results. For the smaller value of FPs per scan, our sensitivity is only 0.7%
lesser, demonstrating the effectiveness of utilizing the auxiliary detectors
for false positive reduction. Although Agnes et al. [35] has a lower CPM
score, their sensitivities are higher when the false positive rate is bigger
than one. A possible explanation is that they had utilized a two-stage scheme
in which they first exploited UNet+ architecture for nodule detection and
employed LSTM-based architecture for false positive reduction, making the
technique more complex. Similarly, Cao et al. [9] has shown better performance
for larger values of false positive rate by using a two-stage strategy. Guo et
al. [37] has obtained lesser CPM, and its performance degrades for a smaller
value of false positives. Similarly, Li et al. [13], Zhao et al. [11], and Mei
et al. [36] obtained lower performance than the proposed scheme. Most
importantly, our CADe system significantly outperforms single-stage systems,
which proves that incorporating self-distillation and further utilizing the
auxiliary detectors can significantly improve the lung detection rate
efficiently.

###  IV-B Ablation Study

To demonstrate the effectiveness of each component incorporated into the
proposed MEDS-Net architecture, we implemented various versions of proposed
framework. We analyzed results after the nodule detection stage as well as
after false positives reduction stage. The following subsections describe the
details of these experiments and the corresponding results.

####  IV-B1 Analysis of candidate detection stage

We initiate our evaluation by analyzing the contribution of each component of
MEDS-Net for the lung nodule detection stage. For this purpose, we implemented
eight downgraded versions of MEDS-Net, having different configurations and
architecture, to evaluate their lung nodules detection performance. Table
LABEL:ablationTab summarizes the results obtained from these models for lung
nodules detection stage. First, to fairly analyze the effect of each input
individually, models 1 to 3 utilize single encoder-based architecture which
takes only one input among 3D sub-volume, forward, and backward MIP images,
respectively. These networks are equipped with attention units and self-
distillation mechanisms for lung nodule detection tasks. The results
demonstrate that models 2 and 3 with MIP images of various thicknesses
achieved better sensitivity with a lower number of candidates than the 3D
input-based model. This proves that the incorporation of MIP images
significantly improves the learning ability of networks which helps to better
distinguish between nodule and non-nodular structures. Whereas, models 2 and 3
depict quite similar performance which shows that forward and backward MIP
images provide the same level of information, i.e., unidirectional insights,
to the network and their individual effect is the same. Models 4 and 5 are
trained with two types of inputs by using dual-encoder based architecture
having attention units and a self-distillation mechanism. Particularly, we
combined 3D sub-volume with forward MIP images and forward MIP images with
backward MIP images as inputs to models 4 and 5, respectively. It can be
observed that combining the two types of inputs improves the sensitivity and
also reduces the number of candidates per scan. However, model 4 performed
slightly better than model 5 which can be attributed to the diverse type of
inputs that enable the network to leverage the exceptional abilities of MIP
images to provide deeper insights and 3D patch input that improves the
localization of nodules. From models 6 to 9, we analyze the effectiveness of
incorporating the attention units and self-distillation in our proposed
architecture. For this purpose, we fixed our input pipeline and utilized
multi-encoder-based architecture incorporating the 3D sub-volume and
bidirectional (forward and backward) MIP images. Among the four versions, in
model 6, we observe severe degradation in the sensitivity when attention units
and self-distillation are removed. It also immensely increased the number of
nodule candidates, which shows that the attention and self-distillation
mechanism significantly contributes to reducing the number of false positives
at the candidate detection stage. Concretely, the individual incorporation of
attention units and self-distillation mechanism improve 3.94% and 5.78%
sensitivity, respectively. Most importantly, the self-distillation mechanism
greatly reduces the number of candidates, improving the architecture’s
efficiency. Finally, the best performance is shown by model 9, which
represents the proposed MEDS-Net architecture, having attention and self-
distillation mechanism with three types of inputs. The results evidence that
the combination of attention and self-distillation mechanism greatly increases
our architecture’s learning ability, enabling it to distinguish between the
nodules and non-nodular structures in the complex lung region.

To analyze the impact of self-distillation, we implemented the variant of the
proposed architecture without self-distillation. Since MEDS-Net is a deeper
network, it is critical to ensure that network is optimally trained without
encountering vanishing gradient problem [39]. Subsequently, we analyzed the
effect of self-distillation during the training process. The training and
validation curves from the deepest detector of both variants have been shown
in Fig. 4. Conceptually, self-distillation is an extended version of deep
supervision that is proven to be effective in addressing the vanishing
gradient problem for segmentation networks [40]. The results show that the
self-distillation mechanism in the proposed framework assists the training
process to optimize network learning. It improves the convergence rate and
helps optimize the learning by achieving lower training and validation losses.
These improvements can be attributed to the supervision of the intermediate
layers at multiple levels, which encourages the earlier layers to learn more
meaningful and representative features for improved detection of lung nodules.

We further analyze the performance of each detector, including main and
auxiliary detectors, in the proposed MEDS-Net. Table III summarized the
results obtained from all the detectors, including the baseline implementation
of multi-encoder based architecture without self-distillation. An ensemble
result is obtained by including all the nodule candidates detected by each
detector. The results of those detectors having performance lesser then the
baseline model have been highlighted with red color. The results demonstrate
that (i) all the networks have acquired significant performance improvement by
exploiting the self-distillation mechanism. (ii) The proposed MEDS-Net has
effectively detected almost all the nodules and achieved the sensitivity
$99.41\%$ with its five detectors. (iii) The overall performance of deeper
detectors is improved; however, several nodules missed by deeper networks have
been detected by the less deep network, indicating that each detector has
detected nodules independently. (iv) The self-distillation plays a crucial
role in improving the deepest network, the main detector. The introduction of
four auxiliary detectors improves the $4.08\%$ sensitivity of the main
detector and significantly reduces the number of false positives per scan.

TABLE III: PERFORMANCE COMPARISON OF EACH DETECTOR OF PROPOSED MEDS-NET FOR
LUNG NODULE DETECTION ON DIFFERENT SIZE OF NODULES

Total number of scans: 888
---
Total number of nodules: 1186 (3-10 mm: 905, 10-20 mm: 231, $\geq$20 mm: 50)
Feature Level |  | No. of detected
---
nodules
(3-10mm)
| No. of detected
---
nodules
(10-20mm)
| No. of detected
---
nodules
($\geq$20mm)
| Total No. of
---
detected
nodules
| Sensitivity
---
(%)
| False Positives
---
(FPs)
FPs per scan
| Multi-Encoder Net
---
(without Self-Distillation)
862 | 202 | 50 | 1114 | 93.93 | 28910 | 32.56
Auxiliary Detector (1/4) | 841 | 195 | 46 | 1082 | 91.23 | 30226 | 34.04
Auxiliary Detector (2/4) | 857 | 198 | 48 | 1103 | 93 | 26954 | 30.35
Auxiliary Detector (3/4) | 856 | 205 | 50 | 1111 | 93.68 | 22047 | 24.83
Auxiliary Detector (4/4) | 877 | 212 | 50 | 1139 | 96.04 | 19338 | 21.78
Main Detector | 889 | 221 | 50 | 1160 | 97.81 | 18004 | 20.27
Ensemble | 901 | 228 | 50 | 1179 | 99.41 | 36780 | 41.42

Figure 4: Comparison of learning curves of the training and validation curves
of proposed multi-encoder based networks with and without self-distillation
mechanism has been demonstrated.

####  IV-B2 Analysis after false positive reduction

In this section, we analyze the effect of each input incorporated in the
proposed MEDS-Net architecture on complete pipeline. Concretely, the proposed
MEDS-Net exploits bidirectional MIP images along with the 3D patch to detect
the lung nodules; therefore, it is important to analyze the impact of
bidirectional MIP images after false positive reduction. We implemented
single-encoder and dual-encoder versions of the proposed architecture with a
complete pipeline that includes the false positive reduction stage. We
implemented three single-encoder-based variants that took one input at a time
and trained those models with 3D patch, forward, and backward image inputs
individually. Similarly, we implemented two dual-encoder based variants, i.e.,
one with bidirectional MIP images inputs and another with 3D patch and forward
MIP images. All the variants are implemented with self-distillation mechanism
and followed by the same pipeline. To compare the performances of these
variants models with proposed MEDS-Net for nodule detection, we analyze the
free-response receiver operating characteristic (FROC) [41] curves of each
variant shown in Fig. 5. The single-encoder-based architecture with 3D patch
input has the lowest performance, and it becomes worse when we reduce the
number of false positives per scan. Since the 3D patch of scan consists of 11
slices of 1 mm thickness (i.e., of depth 10 mm) which is insufficient to
extract detailed insights of the nodule in the complex surroundings within the
lung, it is challenging to detect nodules without false positives. The nodule
bigger than 10 mm appear similar to the vessels, and small vessels have quite
similar structural appearance to the nodule, which confuses the network and
results in poor performance. Since conceptually, both directions of MIP images
contain the same amount of information, subsequently, the single-encoder
versions with forward and backward MIP images demonstrate similar; however,
better performance compared to its 3D patch variant. MIP images based models
are able to achieve almost 80% sensitivity even with 0.125 FPs per scan which
demonstrates the effectiveness of MIP images for lung nodule detection.

We also compared dual-encoder based architectures, for which we implemented
two combinations. At first, we utilized bidirectional MIP images and achieved
significantly improved performance, which can be attributed to the fact that
MIP images enhance the visibility of lung nodules, making it easier for the
network to distinguish them from other anatomical structures. Most
importantly, bidirectional MIP images cover a reasonable depth of scan, i.e.,
30 mm, which enables the network to better explore the insights of each
suspected nodule region. However, MIP images have certain limitations while
representing the information as they rely on maximum intensities, suppressing
the information in the intermediate intensities. Therefore, radiologists
examine the nodules in the raw CT slices after the initial screen in MIP
images. To this end, we also implemented a dual-encoder based variant with a
3D patch and forward MIP images. The results demonstrate that the combination
of 1 mm thick slices with unidirectional MIP images are more effective than
bidirectional MIPs as it equips the network with rich information which
contains high level 3D aspect in the form of MIP as well as low level, 2D
slice level information that helps to distinguish and precisely locate the
small nodules. Finally, combining raw 3D patch of scan with bidirectional MIPs
images in MEDS-Net depicts the optimal and consistent performance for lung
nodule detection.

Figure 5: Free-response receiver operating characteristic (FROC) curves of
various versions of proposed MEDS-Net architecture along with MEDS-Net results
on the LIDC/IDRI database.

##  V Conclusion

This work proposes a novel multi-encoder based self-distillation network
(MEDS-Net) that leverages bidirectional maximum intensity projection (MIP)
images for lung nodule detection system. Particularly, along with a 3D patch
of the scan, the proposed framework utilizes forward and backward MIP images
of 3, 5, and 10 mm thickness as input to improve the network learning. Most
importantly, unlike conventional computer-aided detection (CADe) systems, the
study exploits the auxiliary detectors to reduce the false positives, which
avoids the extra computational costs caused by the separate false positive
reduction networks. The key highlights are as follows:

  1. 1.

The framework has been comprehensively evaluated using the widely used
LIDC/IDRI public dataset. The results using the proposed MEDS-Net showed that
this network has excellent detection performance for the diverse nodule
morphology and has the ability to distinguish false-positive nodules
accurately. The CPM scores of the proposed method can be as high as 0.936,
superior to the most advanced competitive networks.

  2. 2.

MEDS-Net exploits multi-scale features learning by using intermediate
auxiliary outputs branches originating from various levels of decoder block to
minimize the false positives.

  3. 3.

The experiments have shown that employment of self-distillation significantly
improves the learning process by supervising the intermediate layers of the
decoder block. Ablation experiments have also demonstrated that all the
components in the proposed MEDS-Net are chosen carefully for effective lung
nodule detection.

Future works include extending the proposed CADe system while performing the
accurate segmentation of lung nodules and their classification in terms of
texture and malignancy levels.

## References

  * [1]
  * [2] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, “Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: A Cancer Journal for Clinicians_ , vol. 71, no. 3, pp. 209–249, 2021. [Online]. Available: https://acsjournals.onlinelibrary.wiley.com/doi/abs/10.3322/caac.21660
  * [3] D. R. Baldwin, “Prediction of risk of lung cancer in populations and in pulmonary nodules: significant progress to drive changes in paradigms,” pp. 1–3, 2015.
  * [4] I. Sluimer, A. Schilham, M. Prokop, and B. Van Ginneken, “Computer analysis of computed tomography scans of the lung: a survey,” _IEEE transactions on medical imaging_ , vol. 25, no. 4, pp. 385–405, 2006.
  * [5] M. Seemann, A. Staebler, T. Beinert, H. Dienemann, B. Obst, M. Matzko, C. Pistitsch, and M. Reiser, “Usefulness of morphological characteristics for the differentiation of benign from malignant solitary pulmonary lesions using hrct,” _European radiology_ , vol. 9, no. 3, pp. 409–417, 1999.
  * [6] J. Zhang, Y. Xia, H. Cui, and Y. Zhang, “Pulmonary nodule detection in medical images: A survey,” _Biomedical Signal Processing and Control_ , vol. 43, pp. 138–147, 2018.
  * [7] A. Halder, D. Dey, and A. K. Sadhu, “Lung nodule detection from feature engineering to deep learning in thoracic ct images: a comprehensive review,” _Journal of Digital Imaging_ , vol. 33, no. 3, pp. 655–677, 2020.
  * [8] P. Monkam, S. Qi, H. Ma, W. Gao, Y. Yao, and W. Qian, “Detection and classification of pulmonary nodules using convolutional neural networks: a survey,” _Ieee Access_ , vol. 7, pp. 78 075–78 091, 2019.
  * [9] H. Cao, H. Liu, E. Song, G. Ma, X. Xu, R. Jin, T. Liu, and C.-C. Hung, “A two-stage convolutional neural networks for lung nodule detection,” _IEEE journal of biomedical and health informatics_ , vol. 24, no. 7, pp. 2006–2015, 2020.
  * [10] A. Pezeshk, S. Hamidian, N. Petrick, and B. Sahiner, “3-d convolutional neural networks for automatic detection of pulmonary nodules in chest ct,” _IEEE journal of biomedical and health informatics_ , vol. 23, no. 5, pp. 2080–2090, 2018.
  * [11] D. Zhao, Y. Liu, H. Yin, and Z. Wang, “An attentive and adaptive 3d cnn for automatic pulmonary nodule detection in ct image,” _Expert Systems with Applications_ , p. 118672, 2022.
  * [12] Z. Zhou, F. Gou, Y. Tan, and J. Wu, “A cascaded multi-stage framework for automatic detection and segmentation of pulmonary nodules in developing countries,” _IEEE Journal of Biomedical and Health Informatics_ , 2022.
  * [13] Y. Li and Y. Fan, “Deepseed: 3d squeeze-and-excitation encoder-decoder convolutional neural networks for pulmonary nodule detection,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 1866–1869.
  * [14] X. Zhu, X. Wang, Y. Shi, S. Ren, and W. Wang, “Channel-wise attention mechanism in the 3d convolutional network for lung nodule detection,” _Electronics_ , vol. 11, no. 10, p. 1600, 2022.
  * [15] X. Luo, T. Song, G. Wang, J. Chen, Y. Chen, K. Li, D. N. Metaxas, and S. Zhang, “Scpm-net: An anchor-free 3d lung nodule detection network using sphere representation and center points matching,” _Medical Image Analysis_ , vol. 75, p. 102287, 2022.
  * [16] J. W. Wallis, T. R. Miller, C. A. Lerner, and E. C. Kleerup, “Three-dimensional display in nuclear medicine,” _IEEE transactions on medical imaging_ , vol. 8, no. 4, pp. 297–230, 1989.
  * [17] A. Masood, B. Sheng, P. Yang, P. Li, H. Li, J. Kim, and D. D. Feng, “Automated decision support system for lung cancer detection and classification via enhanced rfcn with multilayer fusion rpn,” _IEEE Transactions on Industrial Informatics_ , vol. 16, no. 12, pp. 7791–7801, 2020.
  * [18] S. Zheng, J. Guo, X. Cui, R. N. Veldhuis, M. Oudkerk, and P. M. Van Ooijen, “Automatic pulmonary nodule detection in ct scans using convolutional neural networks based on maximum intensity projection,” _IEEE transactions on medical imaging_ , vol. 39, no. 3, pp. 797–805, 2019.
  * [19] J. F. Gruden, S. Ouanounou, S. Tigges, S. D. Norris, and T. S. Klausner, “Incremental benefit of maximum-intensity-projection images on observer detection of small pulmonary nodules revealed by multidetector ct,” _American Journal of Roentgenology_ , vol. 179, no. 1, pp. 149–157, 2002\.
  * [20] S. Zheng, X. Cui, M. Vonder, R. N. Veldhuis, Z. Ye, R. Vliegenthart, M. Oudkerk, and P. M. van Ooijen, “Deep learning-based pulmonary nodule detection: Effect of slab thickness in maximum intensity projections at the nodule candidate detection stage,” _Computer methods and programs in biomedicine_ , vol. 196, p. 105620, 2020.
  * [21] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [22] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, “Attention gated networks: Learning to leverage salient regions in medical images,” _Medical Image Analysis_ , vol. 53, pp. 197–207, 2019\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841518306133
  * [23] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3713–3722.
  * [24] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman _et al._ , “The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans,” _Medical physics_ , vol. 38, no. 2, pp. 915–931, 2011.
  * [25] E. A. Kazerooni, J. H. Austin, W. C. Black, D. S. Dyer, T. R. Hazelton, A. N. Leung, M. F. McNitt-Gray, R. F. Munden, and S. Pipavath, “Acr–str practice parameter for the performance and reporting of lung cancer screening thoracic computed tomography (ct): 2014 (resolution 4),” _Journal of thoracic imaging_ , vol. 29, no. 5, pp. 310–316, 2014.
  * [26] N. L. S. T. R. Team, “Reduced lung-cancer mortality with low-dose computed tomographic screening,” _New England Journal of Medicine_ , vol. 365, no. 5, pp. 395–409, 2011.
  * [27] J. Angulo and D. Jeulin, “Stochastic watershed segmentation.” in _ISMM (1)_ , 2007, pp. 265–276.
  * [28] A. A. Setio, C. Jacobs, J. Gelderblom, and B. van Ginneken, “Automatic detection of large pulmonary solid nodules in thoracic ct images,” _Medical physics_ , vol. 42, no. 10, pp. 5642–5653, 2015.
  * [29] H. Yuan, Y. Wu, J. Cheng, Z. Fan, and Z. Zeng, “Pulmonary nodule detection using 3-d residual u-net oriented context-guided attention and multi-branch classification network,” _Ieee Access_ , vol. 10, pp. 82–98, 2021.
  * [30] F. R. Pereira, J. M. C. De Andrade, D. L. Escuissato, and L. F. De Oliveira, “Classifier ensemble based on computed tomography attenuation patterns for computer-aided detection system,” _IEEE Access_ , vol. 9, pp. 123 134–123 145, 2021.
  * [31] J. Wang, J. Wang, Y. Wen, H. Lu, T. Niu, J. Pan, and D. Qian, “Pulmonary nodule detection in volumetric chest ct scans using cnns-based nodule-size-adaptive detection and classification,” _IEEE access_ , vol. 7, pp. 46 033–46 044, 2019.
  * [32] J. Zhang, Y. Xia, H. Zeng, and Y. Zhang, “Nodule: Combining constrained multi-scale log filters with densely dilated 3d deep convolutional neural network for pulmonary nodule detection,” _Neurocomputing_ , vol. 317, pp. 159–167, 2018.
  * [33] A. A. A. Setio, A. Traverso, T. De Bel, M. S. Berens, C. Van Den Bogaard, P. Cerello, H. Chen, Q. Dou, M. E. Fantacci, B. Geurts _et al._ , “Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge,” _Medical image analysis_ , vol. 42, pp. 1–13, 2017.
  * [34] E. Lopez Torres, E. Fiorina, F. Pennazio, C. Peroni, M. Saletta, N. Camarlinghi, M. E. Fantacci, and P. Cerello, “Large scale validation of the m5l lung cad on heterogeneous ct datasets,” _Medical physics_ , vol. 42, no. 4, pp. 1477–1489, 2015.
  * [35] S. A. Agnes, J. Anitha, and A. A. Solomon, “Two-stage lung nodule detection framework using enhanced unet and convolutional lstm networks in ct images,” _Computers in Biology and Medicine_ , vol. 149, p. 106059, 2022.
  * [36] J. Mei, M.-M. Cheng, G. Xu, L.-R. Wan, and H. Zhang, “Sanet: A slice-aware network for pulmonary nodule detection,” _IEEE transactions on pattern analysis and machine intelligence_ , 2021.
  * [37] Z. Guo, L. Zhao, J. Yuan, and H. Yu, “Msanet: Multiscale aggregation network integrating spatial and channel information for lung nodule detection,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 26, no. 6, pp. 2547–2558, 2021.
  * [38] M. Niemeijer, M. Loog, M. D. Abramoff, M. A. Viergever, M. Prokop, and B. van Ginneken, “On combining computer-aided detection systems,” _IEEE Transactions on Medical Imaging_ , vol. 30, no. 2, pp. 215–223, 2010.
  * [39] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.
  * [40] Q. Dou, H. Chen, Y. Jin, L. Yu, J. Qin, and P.-A. Heng, “3d deeply supervised network for automatic liver segmentation from ct volumes,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 149–157.
  * [41] A. I. Bandos, H. E. Rockette, T. Song, and D. Gur, “Area under the free-response roc curve (froc) and a related summary index,” _Biometrics_ , vol. 65, no. 1, pp. 247–256, 2009.