2023

[3]Yogesh Kumar Meena

1]Department of Computer Science and Engineering, ABV-Indian Institute of
Information Technology & Management, Gwalior, India

2]Department of Computer Science and Engineering, ABV-Indian Institute of
Information Technology & Management, Gwalior, India

3]Department of Computer Science and Engineering, Indian Institute of
Technology Gandhinagar, India

# MultiFusionNet: Multilayer Multimodal Fusion of Deep Neural Networks for
Chest X‑Ray Image Classification

Saurabh Agarwal<EMAIL_ADDRESS>K. V. Arya<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

Chest X-ray imaging is a critical diagnostic tool for identifying pulmonary
diseases. However, manual interpretation of these images is time-consuming and
error-prone. Automated systems utilizing convolutional neural networks (CNNs)
have shown promise in improving the accuracy and efficiency of chest X-ray
image classification. While previous work has mainly focused on using feature
maps from the final convolution layer, there is a need to explore the benefits
of leveraging additional layers for improved disease classification.
Extracting robust features from limited medical image datasets remains a
critical challenge. In this paper, we propose a novel deep learning-based
multilayer multimodal fusion model that emphasizes extracting features from
different layers and fusing them. Our disease detection model considers the
discriminatory information captured by each layer. Furthermore, we propose the
fusion of different-sized feature maps (FDSFM) module to effectively merge
feature maps from diverse layers. The proposed model achieves a significantly
higher accuracy of 97.21% and 99.60% for both three-class and two-class
classifications, respectively. The proposed multilayer multimodal fusion
model, along with the FDSFM module, holds promise for accurate disease
classification and can also be extended to other disease classifications in
chest X-ray images.

###### keywords:

Medical Image Processing, Convolutional Neural Network (CNN), Multilayer
fusion Model, Multimodal Fusion Model, Disease Classifications, Chest X-Ray
Image

##  1 Introduction

Respiratory infections and pneumonia share significant similarities in their
causes. Specifically, pneumonia accounts for 14% of all deaths among children
under five years old, resulting in the loss of 740,180 young lives in 2019 WHO
(2022). Given the resemblance of respiratory infections, early-stage diagnosis
becomes challenging. Accurate diagnosis plays a crucial role in ensuring
appropriate treatment. The real-time reverse-transcriptase-polymerase chain
reaction (rRT-PCR) test is currently employed for COVID-19 detection. However,
this method can be time-consuming, taking hours to days, depending on
available resources. Thus, developing a detection method that offers immediate
and accurate results at an affordable cost is essential. Fundamental
diagnostic techniques such as Chest X-Ray (CXR) and Computed Tomography (CT)
contribute to the detection of various diseases through Computer-Aided
Diagnosis (CAD) systems. Compared to CT scans, conducting X-rays is more
accessible and cost-effective. Therefore, CXR has been chosen as the
diagnostic mechanism for detecting COVID-19 and pneumonia Jacobi .
(2020),Hayden Wrenn (2009). A convolutional neural network (CNN) has been
utilized for image vision tasks such as classification and recognition. In
medical image processing, CNNs can take Chest X-rays (CXRs) as input, assign
learnable weights and biases to different aspects of the image, and classify
diseases. Generally, CNNs require a large amount of data for training and
testing deep learning models Narin . (2021), but medical images often need
more availability. In recent studies, Transfer Learning (TL) has been used to
train and test deep learning models with limited datasets Pan Yang (2009). TL
involves transferring the knowledge of a model trained on millions of images
to another model with fewer images, such as medical images. Therefore, the
proposed model combines TL with ResNetV50 He . (2016) and Inceptionv3 Szegedy
. (2017). These two models are pre-trained on a large dataset (ImageNet), and
their knowledge is transferred to the proposed model.

In deep learning-based CNNs, each layer of the model contains distinct
information in terms of feature maps. The model’s final (dense) layer consists
of dense features used for classification and segmentation tasks. However, as
we progress from the initial to the dense layer, some features may be lost. To
overcome this, fusion methods Muhammad Hossain (2021), Fang . (2021) have
gained popularity in research, enabling the utilization of all learned
information to enhance model performance. Nonetheless, fusing feature maps
becomes challenging when they have different sizes in width and height.
Additionally, a model-wise fusion of different architectural types presents
further difficulties. In this paper, we address these issues by proposing a
novel approach that fuses TL with pre-trained models ResNetV50 and
Inceptionv3, leveraging their knowledge to improve diagnostic accuracy. We
provide three novel contributions in this paper, by:

  1. 1.

Introducing a novel multilayer multimodal fusion model to enhance the
classification accuracy of lung diseases and empirical evaluation is conducted
with potential subsidiary proposed models.

  2. 2.

Developing an independent feature map transformation module (FDSFM) to address
the issue of variable-sized feature maps generated across multiple layers.

  3. 3.

Providing a larger dataset (Cov-Pneum) for X-ray images by processing and
merging multiple publicly available datasets and hence, evaluation of the
performance of multiple state-of-art research models on this dataset.

The rest of the paper is organized as follows: Section 2 discusses a deep-
learning approach for chest X-ray image classification. In Section 3, the
proposed multilayer multimodal is discussed along with the experiment details.
The discussion about the dataset, preprocessing and training procedure is
given in Section 4. Section 5 considers the empirical evaluation of the model,
including subsidiary proposed models and extensive experimentation. Section 6
discusses the quantitative and qualitative analysis of the outcome of the
proposed model. Discussion on results generated by the model for the different
scenarios and future directions of the proposed research is presented in
Section 7, and we conclude the paper in Section 8.

##  2 Related Work

Deep learning algorithms have shown a significant role in image processing and
classification of lung diseases such as tuberculosis (TB), bacterial Pneumonia
(BP), viral Pneumonia (VP) and COVID-19, from CXR images Subramanian . (2022);
Tang . (2021). However, designing an effective lung disease detection model at
both feature selection and classification levels is challenging. This section
summarised and discussed multiple CNN architectures, parameters, performances,
advantages, and limitations.

Classical CNN architecture: LeCun et al. LeCun . (1989) introduce the LeNet
model, which is one of the earliest pre-trained models of simple CNN with 5
layers. Further, Krizhevsky et al. Krizhevsky . (2012) propose an AlexNet
model with 8 layers of CNN architectures. However, these deep learning models
suffer from extracting deep features because of the shallow network. Simonyan
and Zisserman Simonyan Zisserman (2014) introduce a new type of CNN
architecture (known as the VGG model) which consists of a fixed size
3$\times$3 layer with stride 1 (10 layers). Further, multiple variants of VGG
models, such as VGG-11, VGG-16, and VGG-19 with more layers, have been
utilised with a small-size convolution filter to achieve high performance.

Transfer learning and fine-tuning based CNN architecture: Gour and Jain Gour
Jain (2020) proposed a 30-layer CovNet30 deep learning model by stacking sub-
models of the pre-trained network VGG19 using logistic regression. The model
runs on the COVID-19 CXR dataset of 2764 chest X-ray images and achieves 93%
accuracy for class-3 classification. Rahimzadeh and Attar Rahimzadeh Attar
(2020) developed a CNN model by concatenating the two pre-trained models
(i.e., Xception and ResNet50) for 3-class classification (i.e., COVID- 19,
Pneumonia and general). This network was trained and tested on 11302 chest
X-ray images and achieved 91.4% accuracy. However, these models have high
false detection rates because of the loss of features when traversing from
shallow to deep layers, a significant concern in such neural networks.

A deep learning-based ensemble multimodal was implemented by Deb et al. Deb .
(2022) where 4 pre-trained networks are concatenated to form the model. This
model achieved an accuracy of 88.98% for 3-class classification and an
accuracy of 98.58% for 2-class classification. In addition, the first graph-
based COVID-19 classification ResGNet-C model is presented by Xiang et al. Yu
. (2021), integrating graph knowledge into the classification task. They also
gave a ResNet101-C model based on a pre-trained network ResNet101 that
extracts the deep features of lung diseases. The graph is constructed using
the Euclidean distance from these features and generating the prediction
result. A time-efficient generalized model with residual separable convolution
block improves the performance of the basic MobileNet model in Tangudu .
(2022). In addition, Kaya and Gürsoy Kaya Gürsoy (2023) classified CXR images
into COVID-19, Normal, and Pneumonia using transfer learning on the pre-
trained network MobileNetV2. They Developed an enhanced fine-tuning approach
that minimized the data loss and increased the number of transfer features.

New CNN architectures: Khan et al. A.I. Khan . (2020) proposed a CoroNet model
based on Xception pre-trained model with one flattened layer, one dropout
layer, and two fully connected layers. This model achieved 89.6% accuracy for
4-class cases (COVID vs Pneumonia bacterial vs Pneumonia viral vs Normal) and
95% accuracy for 3-class classification (COVID vs Pneumonia vs Normal). Ozturk
et al. Ozturk . (2020) proposed DarkCovidNet CNN architecture of 17 different
filter-size convolutional layers. This model provides 98.08% accuracy for
binary and 87.02% for multiple-class classification. A Covidnet model is
developed by Wang et al. L. Wang . (2020) that presents a lightweight
projection-expansion-projection-extension (PEPX) design to increase prediction
accuracy while reducing computation complexity. A human-machine collaborative
design strategy inspires this architecture. However, the efficacy of this
model cannot be generalized due to validation on the small dataset.

Moreover, some of the models developed the model related to other lung
diseases such as tuberculosis (TB), bacterial Pneumonia (BP), and viral
Pneumonia (VP). In Xu . (2021), Yujia et al. presented a two-stage segmented
based classification model MANet to classify lung diseases into five classes
(COVID-19, Normal, tuberculosis (TB), bacterial Pneumonia (BP), and viral
Pneumonia (VP)). They also presented the mask attention mechanism (MA) by
defining the attention maps for all features before feature extraction in CNNs
via a segmentation model trained at the first stage to reduce the
computational cost and distribute the attention of CNNs in the segmented lung
regions.

Despite CNN, some researchers used other techniques to develop the lung
disease classifier. Pathak et al. Pathak . (2020) developed a deep
bidirectional extended short-term memory network with a mixture density
network model for classifying COVID-19 disease. They also developed a
technique to optimize the hyperparameter of the model. In addition, Li and Xu
developed a FedFocus model using a federated learning framework with a dynamic
focus for COVID-19 detection on CXR images Li . (2022). They set parameter
aggregation weights by training loss to enhance the training efficiency. This
work aimed to improve the precision and stability of the COVID-19 detection
model.

In contrast, Mohagheghi et al. Mohagheghi . (2021) presented a three-module
deep learning model for classifying COVID-19, healthy, and Pneumonia cases.
CNN and content-based medical image retrieval algorithms provided the
percentage of lung infection and displayed the GGO locations. Here, diagnosis
of lung diseases is performed by CNN, which is the combination of VGG and hash
function. If the patient is positive for COVID-19, segmentation of the
abNormalities and lung lobes of the corresponding CT image are performed with
the CT involvement score calculated. Similarly, Chaudhary et al. Choudhary .
(2022) developed an only weight transfer method using CT images to identify
COVID-19 in which they transfer the knowledge of VGG16 and ResNet34 pre-
trained models. It contributed to deploying heavy ML programs in lightweight
devices. In addition, Wang et al. SH. Wang . (2023) developed a mobile app
ELUCNN to diagnose COVID-19. They succeeded tremendously with a ten-layer CNN
model with the exponential linear unit (ELU). Similarly, the MDA strategy was
used in data preprocessing.

A different approach presented by Ieracitano et al. Ieracitano . (2022) based
on fuzzy logic deep learning CovNNet model to classify the CXR images of
patients with Covid-19 Pneumonia and with interstitial Pneumonia not related
to Covid-19. CovNNet extracts the deep features and combines them with fuzzy
images generated by the fuzzy edge detection algorithm. It improved the
accuracy of the model up to 81%. Zhou et al. Zhou . (2021) enlightened the
issue in fine-tuning the COVID detection CNN model, in which the small dataset
and the significant domain shift are the main issues. They developed a novel
domain adaptation method semi-supervised open set domain adversarial network
for these. It aligns a data distribution to general data space. In the
presented work, results are localized by gradient-weighted class activation
mapping. Dhere and Sivaswamy Dhere Sivaswamy (2022) presented a multi-scale
attention residual learning architecture to classify chest x-ray images into
COVID-19, Pneumonia and healthy cases. With the new loss function, the
Conicity loss and hierarchical method model achieved 93%, 96.28%, and 84.51%
accuracy for the three datasets. This work used the DenseNet model to separate
pneumonia cases from normal cases and then used MARL architecture to
discriminate between COVID and non-COVID Pneumonia cases.

Fusion-based CNN architecture: The performance of a CNN model relies on the
features extracted by its layers. However, these features can be lost when
transitioning from the initial to the dense layer. To address this limitation,
several models have been proposed to fuse these features by combining multiple
models. For instance, Muhammad and Hossain Muhammad Hossain (2021) introduced
a CNN model that applied layer-wise fusion for detecting COVID-19 and
pneumonia. This fusion technique improved accuracy by 5.9% compared to the
model without fusion. Similarly, Ilhan et al. Ilhan . (2022) fused seven CNN
architectures at the feature level and classified CXR images into COVID-19,
None, and Pneumonia categories using a majority voting scheme. Waisy et al.
Al-Waisy . (2020) fused the features of two pre-trained models, ResNet34 and
HRNet, using CXR image scores to identify COVID-19 disease. Khan et al. A.
Khan . (2021) presented an image scene geometry recognition model that
utilized feature fusion and score-level fusion strategies to combine low-level
handcrafted features with deep CNN multi-stage features, resulting in a 12.21%
improvement in accuracy. Moreover, Fang et al. Fang . (2021) proposed a multi-
stage feature fusion network for COVID-19 detection by enhancing low-level
feature maps. They employed ResNet-18 with a multi-stage feature enhancement
module to extract and enhance these features. The development of an effective
layer-wise fusion model could address the problem of feature loss and
facilitate feature reusability.

Tabik et al., proposed a COVID-SDNet methodology Tabik . (2020) that includes
a fusion of segmentation, data transformation, data augmentation, and a
suitable CNN model to classify the CXR images into Normal, mild, moderate, and
severe COVID-19. The segmentation-based cropping method enhanced the
preprocessed images by eliminating irrelevant information, improving the
classification results. However, this work is limited to becoming a
generalized model due to the small dataset size. Shi et al. Shi . (2021)
presented an explainable attention-transfer classification model based on the
knowledge distillation network structure for diagnosing COVID-19. The
attention transfer direction goes from the teacher network to the student
network, in which the teacher network extracts global features and generates
attention maps for infection regions. The student network extracts the
irregular-shaped lesion regions to learn the discriminative features.
Moreover, image fusion is also performed to combine the attentive knowledge
and essential information of the original input. Guest editorial contribution
by Meena and Arya Meena Arya (2023) enlightened the various challenges and
future aspects in the multimodal integration of inputs/outputs. They provided
comprehensive studies of different multimodal approaches in lightweight
environments, enhancing the interpretability and accuracy of classification
systems. The limitation of this work is the restricted subjective dataset of
humans on which the model interpretability accuracy can be checked.

CNN architecture for other scenarios: Another work done on COVID-19 is
detecting the spread of COVID-19. In this direction, Alassafi et al. Alassafi
. (2022) presented a model for predicting the trend of the COVID-19 outbreak.
This model analyzes data from three countries Malaysia, Morocco, and Saudi
Arabia. The given model uses recurrent neural network (RNN) and long short-
term memory (LSTM) networks. In contrast of this, Quilodrán-Casas et al.
Quilodrán-Casas . (2022) introduce the SEIRS model based on LSTM and GAN with
a twin digital network to predict the COVID-19 outbreak in the UK. Other than
this, Wu et al. Wu . (2021) used Extreme Learning Machine (ELM) at the place
of the classification layer in the CNN model and used a sine-cosine algorithm
to tune the ELM’s parameter. In Tan . (2022), Tan et al. introduce a multi-
model model trained on CXR and CT images. The Functional AI model test CXR
images and generate classification result in terms of AUC (from 0.89 to 0.93
for binary classification).

A deeper analysis of the above state-of-the-art models shows that deep
learning models suffer from high false detection rates and low accuracy. The
model also suffered from overfitting due to the limited availability of
labelled medical images. Therefore, facing these limitations, we introduced a
significantly sized dataset (Cov-Pneum) to overcome data sparsity issues.
Similarly, the proposed Multilayer multimodal fusion model enhanced the dense
features by fusing feature maps of different layers sizes, reducing the false
detection rate and enhancing the classification accuracy.

##  3 Proposed Multilayer Multimodal Fusion Model

Figure 1: The architecture model of the multilayer multimodal fusion
incorporates the InceptionV3 and ResNet50V2 models. In the InceptionV3 model,
blocks 1 to 11 consist of layers with different-sized filters (1$\times$1,
3$\times$3, and 5$\times$5) in parallel mode. Similarly, the ResNet50V2 model
includes layers from blocks 1 to 5 with different-sized filters (1$\times$1
and 3$\times$3). The FM box represents the extracted feature maps generated
from different layers of their respective networks.

Figure 2: The layered diagram illustrates the novel module called Fusion of
Different-Sized Feature Maps (FDSFM). The size transformation is achieved
through pooling sampling and the Conv 1$\times$1 layer, enabling effective
fusion of feature maps.

The proposed approach incorporates layer-wise fusion of multiple layers and
model-wise fusion of different models. We have employed the early fusion
method, which combines modalities before the classification task. Our model is
built on the ResNet50V3 and InceptionV3 deep learning models, renowned for
their speed and accuracy in various lung disease detection tasks Fu . (2023),
Cannata . (2022), Srivastava . (2022). These models excel at extracting deep
features through various combinations of skip and residual connections. While
ResNet50V3 excels at handling deep networks and addressing the vanishing
gradient problem, InceptionV3’s modules facilitate multi-scale feature
extraction. The synergy of these models forms a robust framework for precisely
identifying intricate patterns and subtle anomalies in CXR images. Therefore,
we employ ResNet50V3 and InceptionV3 in this work. Detailed explanations of
our model architecture, multilayer fusion, FDSFM module, and multi-modal
fusion are provided in the subsequent sections.

###  3.1 Model architecture

The proposed architecture for multilayer multimodal fusion is illustrated in
Fig. 1. Two modules, namely ResNet and Inception, represent the pre-trained
ResNet50V2 and InceptionV3 models, respectively. They are utilized for feature
extraction in the form of feature maps. These feature maps are then resized
using the fusion of different-sized feature maps (FDSFM) module is illustrated
in Fig. 2. The FDSFM module consists of a max pooling layer for each feature
map, a concatenation layer, and a 1$\times$1 convolutional (Conv) layer. This
approach enhances the model’s ability to learn discriminative features from
CXR images.

After the multilayer fusion, a BatchNormalization layer, a 1$\times$1 Conv
layer, and a Global Average Pooling 2D (GAP2D) layer are applied. These
additional layers contribute to improving the generalization, convergence, and
computational efficiency of the proposed model. Furthermore, multimodal fusion
is performed using the addition operation to capture the distinctive
characteristics of CXR images, such as ground-glass opacity and pleural
effusion. To mitigate overfitting, a dropout of 30% is employed, and the
flattened output is passed through a feed-forward neural network consisting of
two dense layers. The final layer of the network employs softmax activation
for classification. Table 1 presents the detailed architecture of the proposed
model, including the number and types of layers, as well as the output size
and parameters of the model.

Table 1: The proposed Multilayer multimodal fusion model consists of layers with different types, kernel sizes, output sizes, and parameters. Layer | O/p / Kernal size | #Param
---|---|---
Pre-trained ResNet | - | -
Pre-trained Inception | - | -
ResNet_Layer_FM1 | (None, 28, 28, 128) | -
ResNet_Layer_FM2 | (None, 7, 7, 1024) | -
ResNet_Layer_FM3 | (None, 14, 14, 512) | -
ResNet_Layer_FM4 | (None, 28, 28, 256) | -
Inception_Layer_FM1 | (None, 5, 5, 384) | -
Inception_Layer_FM2 | (None, 5, 5, 448) | -
Inception_Layer_FM3 | (None, 5, 5, 384) | -
Inception_Layer_FM4 | (None, 5, 5, 448) | -
max_pooling2d_36 | K : 4 $\times$ 4 | 0
max_pooling2d_39 | K : 2 $\times$ 2 | 0
max_pooling2d_37 | K : 4 $\times$ 4 | 0
concatenate_18 | (None, 7, 7, 1920) | 0
concatenate_24 | (None, 5, 5, 1664) | 0
concatenate_19 | (None, 7, 7, 3968) | 0
concatenate_25 | (None, 5, 5, 3584) | 0
batch_norm_112 | (None, 7, 7, 3968) | 15872
batch_norm_115 | (None, 5, 5, 3584) | 14336
conv2d_102 | K : 2 $\times$ 2 | 8128512
conv2d_105 | K : 2 $\times$ 2 | 7342080
global_avg_pool2d_4 | (None, 2048) | 0
global_avg_pool2d_7 | (None, 2048) | 0
lambda (Add_func.) | (None, 2048) | 0
dense_2 | (None, 256) | 524544
dense_3 | (None, 3) | 771
Total params: 61,393,699
Trainable params: 16,011,011
Non-trainable params: 45,382,688

###  3.2 Multilayer fusion

We have designed a multilayer fusion model by performing fusion on the layers
of the ResNet50V2 and InceptionV3 models. The ResNet50V2 model utilizes a
residual architecture and introduces two $3\times 3$ convolutional layers with
16, 32, 64, and 128 filter depths. On the other hand, the InceptionV3 model is
based on the inception architecture, where parallel layers extract deep
features. Specifically, different-sized filters ($1\times 1$, $3\times 3$, and
$5\times 5$) are applied in parallel mode to achieve optimal performance.
Initially, a Conv $1\times 1$ layer is used before the Conv $3\times 3$ and
Conv $5\times 5$ layers to reduce the number of parameters. The outputs of
these filters are concatenated and passed through a pooling layer with a size
of $3\times 3$. To fuse dense features and prevent feature loss, we consider
the Conv layer from the middle module of the models.

The size of a feature map is determined by its length, width, and channel, and
only feature maps with the same dimensions can be fused. To minimize the
transformation overhead and information loss, we have considered a selection
layer that can choose feature maps of the same length and width generated from
a specific layer.However, feature maps of the same size cannot be obtained
from layers. Therefore, Section 3.3 proposes a novel fusion model that
appropriately resizes the feature maps. Subsequently, the resulting feature
maps from each model are fused at the concatenation layer. At the intermediate
layer, we combine these feature maps using Equation (1).

$X^{\prime}_{Concat}=x^{\prime}_{1}\cup x^{\prime}_{2}\cup
x^{\prime}_{3}\cup\ldots\cup x^{\prime}{{}_{i}}\cup\ldots\cup
x^{\prime}{{}_{n}}$ (1)

Where $x^{\prime}_{n}$ is the features maps of $n^{th}$ layer and
$X^{\prime}_{Concat}$ is the outcome of the fused features map.

###  3.3 Fusion of different sized feature maps

The fusion of multiple layers generated feature maps is still challenging due
to their variable sizes. To encounter this issue, we designed a novel fusion
of different-sized feature maps (FDSFM) method as illustrated in Fig. 2. It
first reduces the size of the feature map to the optimal size where all
feature maps can be fused efficiently. We applied MaxPooling2D layer of filter
size f and strides of s to downsample the input. We used Eq. (2) to reduce the
features map of $X^{(W\mathrm{x}L\mathrm{x}C)}$ into the appropriate size.
Where L, W and C represent length, width and number of channels, respectively.
Afterwards, for effective feature mapping, we applied pooling sampling Conv
($1\times 1$) for channel size adjustment.

$X^{(W^{\prime}\times L^{\prime}\times
C^{\prime})}=X^{\Big{(}\big{(}(W-f+1)/s\big{)}\>\times\>\big{(}(L-f+1)/s\big{)}\>\times\>C\Big{)}}$
(2)

###  3.4 Multimodal fusion

Similar multilayer fusion model, we designed a multimodal fusion model by
performing the fusion between ResNet50V2 and InceptionV3 models at the model
level. From the multilayer fusion model, we received two different feature
maps at the concatenate layer of each model. To remove the feature map size
restrictions, each feature map passes through the Global Average Pooling 2D
(GAP2D) layer. We designed and performed model-wise fusion by using Eq.( 3).

$X^{\prime}_{Add}=X^{\prime}_{Concat1}\odot X^{\prime}_{Concat2}$ (3)

Where $X^{\prime}_{Concat1}$ and $X^{\prime}_{Concat2}$ are the outcome of
layer-wise feature maps of the individual model. We performed element-wise
addition $(\odot)$ on these features maps to generate final fused features
maps. We noticed that addition fusion performed better as compared to
concatenate fusion.

When the model was developed, we trained this proposed model on a given
dataset of Ns training samples, where Ns=(X,$\mathfrak{L}$).We denote input
CXR images by X = $\\{x_{1},x_{2},x_{3},\ldots,x_{n}\\}$ and their
corresponding true labels by $\mathfrak{L}$
=$\\{t_{1},t_{2},t_{3},\ldots,t_{n}\\}$. We denote the total number of classes
that are available any time in the model by $t_{i}$, therefore
$t_{i}\in\\{0,1,2\\}$ where 0, 1, and 2 represent COVID-19, Pneumonia, and
Normal class labels, respectively. The total number of layers in model given
as L=$\\{l_{1},l_{2},l_{3},\ldots,l_{n}\\}$ and corresponding features maps
are represented as $X^{\prime}=$
{$x_{1}^{\prime},x_{2}^{\prime},x_{3}^{\prime},\ldots,x_{i}^{\prime}\ldots,x_{n}^{\prime}\\}$,
where each features map is given by $x_{i}^{\prime}\in\Re^{W,L,C}$ with W
(width), L (length), and C (number of channels).

The working functionality of the proposed model can be shown by the function
$f^{w}:X\rightarrow\mathfrak{L}$ that is parameterized by weight w. We
performed an intermediate process before generating of predicted class label
($\mathfrak{L}$). The predicted class score is calculated by $f(X_{i}\mathchar
24635\relax\;w)$ and the outcome of this is a real number from $-\infty$ to
$\infty)$. Now to restrict this output score, we convert it into prediction
probability and applied the logistic function softmax as given in Eq. (4).

$p\left(l_{i}\mid\left(f\left(X^{j}\mathchar
24635\relax\;w\right)\right)\right)=\frac{e^{f_{i}\left(X^{j}\mathchar
24635\relax\;w\right)}}{\sum_{i=1}^{n}e^{f_{i}\left(X^{j}\mathchar
24635\relax\;w\right)}}$ (4)

Further, we trained the proposed network model to minimize the loss and the
desired output was generated by assigning the highest probability to the
correct output labels as:

$w^{*}=\arg\min_{w}\sum_{j=1}^{n}\operatorname{loss}\left(t_{i},p\left(l_{i}\mid\left(f_{i}\left(X^{j},w\right)\right)\right)\right)$
(5)

Where loss is considered as categorical-cross-entropy by Eq. (6).

$\begin{split}\operatorname{loss}\left(t_{i},p\left(l_{i}\mid\left(f_{i}\left(X^{j}\mathchar
24635\relax\;w\right)\right)\right)\right)=\\\ \sum_{i=1}^{n}-\log
p\left(l_{i}\mid\left(f_{i}\left(X^{j}\mathchar
24635\relax\;w\right)\right)\right)\cdot t_{i}\end{split}$ (6)

Table 2: Size of the dataset for all classes (COVID-19 (C), Pneumonia (P), Normal (N)) that are used to validate the state-of-art CNN models and their dataset comparison with our proposed Cov-Pneum dataset. Dataset | Size of CXR images
---|---
C | P | N | Sum
DarkCovid-Net Ozturk . (2020) | 125 | 500 | 500 | 1125
CoroNet A.I. Khan . (2020) | 3180 | - | 10120 | 13800
CovidNet L. Wang . (2020) | - | - | - | 13975
CovidGAN Waheed . (2020) | 403 | - | 721 | 1124
Rahimzadeh et al. Rahimzadeh Attar (2020) | 180 | 6054 | 8851 | 15085
Shanjiang et al. Tang . (2021) | 473 | 5459 | 7966 | 13898
Cov-Pneum dataset | 4296 | 5824 | 11152 | 21272

The proposed model is then trained on the Cov-Pneum CXR dataset for training,
validation and testing.

##  4 Dataset, Preprocessing and Training Procedure

In this section, we present a step-by-step description of the data
preparation, preprocessing, and training and optimization processes.

###  4.1 Dataset description

We prepared a new dataset called Cov-Pneum by processing and merging three
well-known publicly available datasets from Kaggle Chowdhury . (2020); Rahman
. (2021); Kaggle (20211, 20212). The Cov-Pneum dataset consists of a total of
21,272 CXR images categorized into COVID-19 (lung infected with COVID-19
virus), Pneumonia (viral Pneumonia, non-COVID-19 Pneumonia, and COVID-19
Pneumonia infected lung), and Normal (clear lung) classes. The dataset
contains 4,296 COVID-19 images, 5,824 Pneumonia images, and 11,152 Normal
images. To improve the quality of the CXR images, we applied image scaling and
pre-processing operations.

Table 2 presents a comparison of the proposed Cov-Pneum dataset with other
state-of-the-art datasets used in multiple studies. We will publish the Cov-
Pneum dataset online and provide a link upon final approval. This will enable
fellow researchers to validate their models on a significant-sized dataset and
address data sparsity issues.

###  4.2 Data preprocessing

In the preprocessing stage, the CXR images were resized using bilinear
interpolation to ensure a uniform size and standardized dimensions. The pixel
intensities of the images were normalized within the range of (-1, 1). This
ensures that the input data is appropriately scaled, leading to more effective
and stable training.

Furthermore, in the preprocessing pipeline, we applied geometric
transformations using shear range and zoom parameters. A shear range of 0.1
allowed us to perform affine transformations on the images, horizontally
skewing them by a maximum of 10% in either direction. Similarly, a zoom factor
of 0.1 enabled us to randomly zoom the images in or out, with a maximum
variation of 10%. These transformations add variability to the dataset,
enhancing the model’s ability to learn and generalize from different
orientations, scales, and geometries present in the CXR images.

###  4.3 Training and optimization

All experiments were conducted using pre-trained models on the ImageNet
dataset Deng . (2009). Subsequently, these models were trained on the Cov-
Pneum dataset, which consists of CXR images with a size of (224, 224, 3). The
supervised learning approach was employed to train the proposed model and sub-
models on a total of 12,157 labeled CXR images categorized as COVID-19,
Pneumonia, and Normal. Similarly, 3,219 CXR images were used for validation,
and the remaining 5,896 CXR images were allocated for testing.

The proposed network performed a series of convolutional, pooling, and
activation operations on the data to extract meaningful features. The output
of the network was then compared to the true labels using a categorical cross-
entropy loss function given in Equation (6). The primary objective of using
this loss function was to minimize the error. To optimize the learning weights
of the model, the Adam optimizer Kingma Ba (2014) was employed due to its
effectiveness in other classification models Bera Shrivastava (2020).

The performance of the CNN model depends on several hyperparameters, including
the batch size, number of epochs, and learning rate. A range of values was
explored for each hyperparameter to evaluate their impact on the model’s
accuracy. For the learning rate, values ranging from 0.0001 to 0.1 were tested
with increments of 0.01. The batch size was varied from 16 to 128, increasing
by a factor of 8. Additionally, epoch values ranging from 10 to 100, with
increments of 10, were experimented with. Through this systematic exploration,
it was observed that a learning rate of 0.001, a batch size of 32, and 100
epochs yielded the highest accuracy for the proposed model. Various evaluation
metrics, such as confusion matrix, accuracy, precision, F-1 score, recall, and
ROC curve, were employed to assess and compare the performance of the model.

##  5 Empirical Evaluation

We propose and implement additional potential models for in-depth validation
and to find the efficacy of the proposed multilayer multimodal fusion model at
multiple levels. Likewise, these subsidiary models also address the loss of
layer features when features move from shallow layers to deeper layers.

###  5.1 Subsidiary proposed models

In addition to the multilayer multimodal fusion model, this work proposes the
three subsidiary models for in-depth comparison and evaluation.

1\. Multilayer fusion of ResNet50V2 model: This model is based on the
ResNet50V2 architecture (Fig. 1), where we employ layer-wise fusion. We
extract deep features from the Block 3, Block 4, and Block 5 layers of the
ResNet50V2 model. These extracted feature maps may have different sizes
(length and width), which poses a challenge for fusion. To address this, we
utilize the FDSFM module to transform and fuse these feature maps using Eq.
(1). Furthermore, the extracted features and the output of the ResNet50V2
model (before the dense layer) are fused at the concatenate layer. A dropout
layer with a rejection rate of 0.3, followed by a dense layer and a fully
connected layer with softmax activation, is applied at the top of the model.

2\. Multilayer fusion of InceptionV3 model: The same experiment is also
conducted with the InceptionV3 model, employing layer-wise fusion. Deep
feature maps are generated by extracting features from block 4 to block 11
layers. Fusion and size transformation of the feature maps are performed in
the FDSFM module. All generated feature maps are concatenated with the last
layer (before the dense layer) of the InceptionV3 model. A dropout layer with
a rejection rate of 0.3, followed by a dense layer and a fully connected layer
with softmax activation, is applied at the top of the model. The concatenation
of deep features significantly improves the performance of the proposed
multilayer fusion of the InceptionV3 model.

3\. Singlelayer multimodal fusion model: We implemented this model using
singlelayer multimodal fusion to demonstrate the difference between multilayer
fusion and singlelayer fusion, as well as multimodal fusion and singlemodal
fusion. Model-wise fusion was performed between the outputs of the last layers
of the ResNet50V2 and InceptionV3 models. Fusion was accomplished using a
point-wise addition operation, which effectively reduced the number of
parameters. This operation is represented by Eq. 3. The concatenated features
obtained from the fusion were then passed through a dropout layer with a
rejection rate of 0.3, followed by a dense layer and a fully connected layer
with softmax activation at the top of the model. As a result, the proposed
model achieved remarkable success in accurately classifying the input CXRs
based on the generated deep features.

Figure 3: Comparison of the learning behavior of baseline models (TL with
ResNet50V2 He . (2016), TL with InceptionV3 Szegedy . (2017)) and proposed
models in terms of training accuracy (a) and training loss curves at per epoch
time.

###  5.2 Experiments

Extensive experimentation has been conducted to evaluate the performance of
the proposed model, which involves fusing features from different layers of
the models. We implemented all four proposed models along with other state-of-
the-art models on the Cov-Pneum dataset to assess the effectiveness of the
proposed approach.

Experiment 1: Singlemodal fusion v/s multimodal fusion
In this experiment, we evaluate and compare the performance of singlemodal
fusion (i.e., multilayer fusion of ResNet50V2, multilayer fusion of
InceptionV3 model) and multimodal fusion (i.e., singlelayer multimodal fusion
model, multilayer multimodal fusion model). We implemented the models
discussed in Section III to demonstrate these evaluations. The experiment with
layer-wise fusion is conducted on both the ResNet50V2 model and the
InceptionV3 model to implement the singlemodal fusion. Similarly, the same
experiment is performed on the results of these separated models to build the
multimodal fusion. Furthermore, the multimodal fusion approach is applied to
the output of the last layers of the ResNet50V2 and InceptionV3 models to
create additional cases of multimodal fusion (singlelayer multimodal fusion
model).

Experiment 2: Singlelayer fusion v/s multilayer fusion
This experiment is conducted to evaluate the performance of the singlelayer
fusion model (i.e., singlelayer multimodal fusion model) with the multilayer
fusion model (i.e., multilayer fusion of ResNet50V2, multilayer fusion of
InceptionV3 model, and multilayer multimodal fusion model). In the singlelayer
fusion approach, the output layer of each respective model is fused, while in
the multilayer fusion, multiple layers of each model are combined.
Consequently, we performed a substantial number of experiments on different
models, ultimately achieving success by fusing the output layer of the
ResNet50V2 and InceptionV3 models. Firstly, layer-wise fusion is implemented
on the ResNet50V2 model to perform the multilayer fusion. Subsequently, the
same experiment is also conducted on the InceptionV3 model. Furthermore,
model-wise fusion is implemented on the output of these subsequent results.

Experiment 3: State-of-art models evaluation on Cov-Pneum data-set
The performance of the newly implemented model can be evaluated by comparing
it with the other state-of-art methods. The system environment and dataset are
the key parameters in validating the results. Therefore, we implemented the
state-of-art COVID-19 and Pneumonia detection methods such as CoroNet A.I.
Khan . (2020), DarkCovid-Net Ozturk . (2020), CovidNET L. Wang . (2020), Fangg
et al. Fang . (2021), Shanjiang et al. Tang . (2021), CovidGAN Waheed .
(2020), TL with VGG19 Apostolopoulos Mpesiana (2020), TL with ResNet50V2 He .
(2016), TL with Xception Chollet (2017), and TL with InceptionV3 Szegedy .
(2017) on proposed dataset Cov-Pneum. This experiment was performed in the
Jupyter Notebook environment using the Tensorflow Keras library on an Intel
Xeon Z440 with 8 GB RAM and a processing speed of 3.5 GHz.

Experiment 4: Proposed model evaluation on state-of-art datasets
We conducted this experiment to evaluate the proposed model performance on
state-of-art datasets. Here we considered the dataset Dark-CovidNet Ozturk .
(2020) to have 1125 CXR images (COVID-19: 125, Pneumonia: 500, and Normal:
500) and Rahimzadeh et al. Rahimzadeh Attar (2020) dataset has 15085 CXR
images (COVID-19: 180, Pneumonia: 6054 and Normal: 8851). Our proposed
Multilayer multimodal fusion model is implemented on this dataset for further
evaluation and validation.

##  6 Results

We present a comprehensive analysis of the complete experiments to demonstrate
the performance of our proposed work. To validate our results, we compare the
performance of the proposed model with several sub-models and state-of-the-art
COVID-19 and Pneumonia detection methods. It includes the proposed sub-models,
such as the ResNet50V2 model’s multilayer fusion of the InceptionV3 model,
singlelayer multimodal fusion model, and multilayer multimodal fusion model.
Whereas it includes the various state-of-the-art methods, such as CoroNet A.I.
Khan . (2020), DarkCovid-Net Ozturk . (2020), CovidNET L. Wang . (2020), Fangg
et al. Fang . (2021), Shanjiang et al. Tang . (2021), CovidGAN Waheed .
(2020), TL with VGG19 Apostolopoulos Mpesiana (2020), TL with ResNet50V2 He .
(2016), TL with Xception Chollet (2017), and TL with InceptionV3 Szegedy .
(2017).

###  6.1 Training performance

The learning behavior of the models in terms of accuracy and loss curves per
epoch is demonstrated in Fig. 3. It also compares the learning performance of
the proposed model with different sub-models and various baseline models. As
the number of epochs increases, the model shows a notable trend of achieving
the highest accuracy while substantially reducing the loss. This indicates the
good learning behavior of the proposed model during the training and
validation phases. Interestingly, the proposed model exhibits an accuracy
curve approaching one and a loss curve approaching zero, in contrast to other
state-of-the-art models. This exceptional performance serves as evidence of
the model’s excellent training behavior.

Table 3: Performance evaluation of state-of-art and our models on proposed Conv-Pneum dataset. Computed classification accuracy (%) for both 3-class (3-C) and binary (2-C) class models. Evaluate individual class performance P: Precision, R: Recall, and F-1 : F-1 Score. Here, M1 is multilayer fusion of ResNet50V2 model, M2 is multilayer fusion of InceptionV3 model, M3 is singlelayer multimodal fusion model, and M4 is multilayer multimodal fusion model. Existing vs proposed models | Covid-19 | Pneumonia | Normal |  Acc (3-C) |  Acc (2-C)
---|---|---|---|---|---
P | R | F-1 | P | R | F-1 | P | R | F-1
DarkCovid-Net Ozturk . (2020) | 0.97 | 0.90 | 0.94 | - | - | - | 0.98 | 0.99 | 0.98 | 87.02 | 98.08
CoroNet A.I. Khan . (2020) | 0.93 | 0.98 | 0.95 | 0.84 | 0.82 | 0.83 | 0.95 | 0.93 | 0.94 | 92.45 | -
CovidNET L. Wang . (2020) | 0.98 | 0.91 | 0.94 | 0.91 | 0.94 | 0.92 | 0.90 | 0.95 | 0.92 | 92.60 | 97.54
CovidGAN Waheed . (2020) | 0.96 | 0.90 | 0.93 | - | - | - | 0.94 | 0.97 | 0.95 | - | 94.78
TL with VGG19 Apostolopoulos Mpesiana (2020) | 0.86 | 0.99 | 0.92 | 0.94 | 0.98 | 0.96 | 0.99 | 0.92 | 0.96 | 90.25 | 94.62
TL with ResNet50V2 He . (2016) | 0.93 | 0.91 | 0.92 | 0.95 | 0.92 | 0.93 | 0.91 | 0.94 | 0.92 | 91.74 | 94.78
TL with Xception Chollet (2017) | 0.95 | 0.90 | 0.92 | 0.89 | 0.91 | 0.90 | 0.87 | 0.92 | 0.89 | 92.24 | 95.23
TL with InceptionV3 Szegedy . (2017) | 0.92 | 0.94 | 0.93 | 0.98 | 0.93 | 0.96 | 0.91 | 0.98 | 0.95 | 93.87 | 95.74
Fangg et al. Fang . (2021) | 0.99 | 0.94 | 0.96 | 0.96 | 0.94 | 0.95 | 0.90 | 0.98 | 0.94 | 95.29 | 97.47
Shanjiang et al. Tang . (2021) | 0.94 | 0.96 | 0.95 | 0.93 | 0.94 | 0.93 | 0.96 | 0.95 | 0.95 | 95.69 | 98.09
M1 | 0.90 | 0.92 | 0.91 | 0.99 | 0.93 | 0.96 | 0.84 | 0.97 | 0.90 | 94.01 | 97.82
M2 | 0.92 | 0.91 | 0.92 | 0.99 | 0.94 | 0.96 | 0.83 | 0.97 | 0.90 | 94.18 | 97.47
M3 | 0.97 | 0.93 | 0.95 | 0.99 | 0.98 | 0.99 | 0.95 | 0.93 | 0.94 | 95.95 | 98.05
M4 | 0.96 | 0.98 | 0.97 | 0.99 | 0.97 | 0.98 | 0.94 | 0.96 | 0.95 | 97.21 | 99.60

Figure 4: Performance of proposed models on the Conv-Pneum dataset. The
confusion matrix represents the detection rate of COVID-19 (C), Pneumonia (P),
and Normal (N) (3-Class) in (a-d). (a) multilayer fusion of ResNet50V2 model,
(b) multilayer fusion of InceptionV3 model, (c) singlelayer multimodal fusion
model, and (d) multilayer multimodal fusion model. The detection rate of
COVID-19 and Normal (2-Class) in (e-h). (e) multilayer fusion of ResNet50V2
model, (f) multilayer fusion of InceptionV3 model, (g) singlelayer multimodal
fusion model, and (h) multilayer multimodal fusion model.

###  6.2 Quantitative analysis

The comparison of the proposed model with various models in terms of class-2
(COVID-19 vs Normal) and class-3 (COVID-19 vs Pneumonia vs Normal)
classification accuracy is given in Table 3. The proposed model outperformed
the state-of-the-art algorithms and other sub-models by achieving class-2 and
class-3 classification accuracy of 99.6% and 97.21%, respectively, and
outperformed all existing models (p $<$ 0.05, Wilcoxon signed-rank test). Some
enhancement is also achieved by the sub-model (proposed model without layer-
wise fusion) compared to the second-best model Shanjiang et al. Tang . (2021),
where the sub-model achieved class-3 accuracy of 95.95% in comparison with an
accuracy of 95.69%.

The evaluation of system performance always depends on each class
classification result. Majorly, it is observed that some class classification
results may be far away from the overall classification results, which is why
we measure precision, recall, and F1 scores for each class of the proposed
model. The F1-score, which is the harmonic mean of precision and recall,
achieved by the proposed model for the class of COVID-19 is 0.94, for
Pneumonia is 0.92, and for Normal is 0.98. Furthermore, the model obtained
precision and recall values of 0.96, 0.94, 0.99, and 0.98, 0.96, 0.97,
respectively for the classes of COVID-19, Pneumonia, and Normal. Table 3 shows
that the proposed model surpassed the other state-of-the-art models and sub-
models across all evaluation metrics. Therefore, it is evident that the system
can accurately detect any class.

The confusion matrices of all four proposed models and sub-models in two
categories of class-2 and class-3 are shown in Fig. 4. In each matrix, the
rows represent the actual class, and the columns represent the class predicted
by each model. It was analyzed on the Conv-Pneum dataset, including 680
COVID-19, 1341 Normal, and 3875 Pneumonia CXR test images. The proposed
multilayer multimodal model fusion model correctly identified 630 COVID-19
cases, and only 25 cases were misclassified out of 3875 Pneumonia cases, while
in another instance, the singlelayer fusion model misclassified 257 cases. It
indicates that the multilayer fusion model’s error ratio is minimal compared
to the other sub-models.

The performance of the proposed model is also evaluated on two other state-of-
the-art datasets for validation purposes. The first dataset is the Dark-
CovidNet Ozturk . (2020) dataset, on which the proposed model achieved a
class-2 and class-3 classification accuracy of 98.25% and 95.96%,
respectively, compared to Dark-CovidNet Ozturk . (2020) accuracy of 98.08% and
87.02%. The second dataset is the Rahimzadeh et al. Rahimzadeh Attar (2020)
dataset, on which the proposed model achieved a class-2 accuracy of 98.25% and
class-3 accuracy of 95.96%, respectively.

As shown in Table 3, the proposed models outperformed the existing state-of-
the-art models. Furthermore, we compared the accuracy reported by state-of-
the-art works and found that our achieved accuracies are significantly higher
than previous works. The comparison between reported accuracy and accuracy on
the Cov-Pneum dataset of state-of-art models shows the changes. For instance,
CoroNet A.I. Khan . (2020) accuracy enhanced by around 3% on our dataset,
whereas CovidNet L. Wang . (2020) detection accuracy degraded by about 1%.
Furthermore, Shanjiang et al. Tang . (2021) performance is increased in terms
of accuracy (95.69%) on the Cov-Pneum dataset compared to the reported
accuracy of 94.53%. However, our proposed model surpassed all the other models
by achieving an accuracy of 99.60% for class-2 classification and 97.21% for
class-3 classification, which presents our model as the most suitable
classifier for CXR for the detection of COVID-19 and Pneumonia diseases.

###  6.3 Qualitative analysis

The performance of the proposed model is measured through a receiver operating
characteristic (ROC) curve, as shown in Fig. 5. It demonstrates the comparison
of True Positive Rate (TPR) vs False Positive Rate (FPR) between the proposed
model and the baseline and sub-models. Generally, the larger area covered by
the curve represents the highest accuracy model. It is visible that the
proposed multilayer multimodal fusion model has the highest area compared to
other comparable models. Moreover, to enhance the interpretability our
proposed model, we utilized Gradient-weighted Class Activation Mapping (Grad-
CAM), which effectively elucidates the model’s decision-making process. This
class activation map is generated by channeling gradient information from CXR
images back into the final convolutional layer, allowing us to discern the
significance of each neuron in classifying images into different disease
classes. Fig. 6 showcases the class activation maps for CXR images of
COVID-19, Pneumonia, and Normal cases. In these maps, light colors (such as
blue) represent clear lungs, while hot colors (like red and yellow) indicate
areas of infection in the lungs. Therefore, it is evident from the class
activation maps of COVID-19 and Pneumonia that infected lungs display a hot
color, while normal lungs appear in a lighter color (blue). These activation
maps serve as a valuable tool for healthcare professionals to visualize the
lungs and highlighting the infected areas. These results indicate that the
proposed model is a better classifier for COVID-19 and Pneumonia diseases.

Figure 5: The ROC curve demonstrates the comparison between baseline models (TL with ResNet50V2 He . (2016), TL with InceptionV3 Szegedy . (2017)) and proposed models in terms of AUC rate. Figure 6: Grad-CAM visualization for CXR images: Original images (1st column) and overlaid activation maps (2nd column) for (a) Normal, (b) Pneumonia, and (c) COVID-19 cases. Table 4: Computational performance evaluation of state-of-art and proposed model. Methods |  | #Param
---
(Million)
DarkCovid-Net Ozturk . (2020) | 11.64
CoroNet A.I. Khan . (2020) | 33.96
CovidNET L. Wang . (2020) | 11.75
CovidGAN Waheed . (2020) | 14
ResNet50V2 | 24.97
InceptionV3 | 23.8
Multilayer multimodal fusion model | 16.01

##  7 Discussion

This study introduces a novel fusion model that leverages the strengths of the
ResNet50V2 and InceptionV3 architectures, resulting in enhanced effectiveness
and improved classification reliability and accuracy. The main contribution of
this research lies in exploring various fusion scenarios, including
singlemodal vs multimodal and singlelayer vs multilayer, with significant
performance gains observed in the multilayer multimodal fusion approach.
Additionally, the proposed methodology successfully addresses the challenge of
fusing feature maps of different sizes, improving the model’s adaptability to
diverse data.

One significant accomplishment of this research is the creation of the Cov-
Pneum dataset, which surpasses the size of existing state-of-the-art datasets
by more than ($>$16-95%). This larger dataset mitigates data sparsity issues
and strengthens the robustness of our findings.

In the evaluation of the proposed model’s performance with varying dropout
values (0.2, 0.25, 0.3, 0.35, and 0.4), it was observed that the model
achieved its highest class-3 accuracy of 97.21% when a dropout rate of 0.3 was
applied. This result suggests that a dropout rate of 0.3 struck a balance
between preventing overfitting and preserving valuable features in the
network, leading to the most optimal performance. Lower dropout rates, such as
0.2 and 0.25, were associated with increased overfitting, resulting in lower
accuracy of 94.2% and 95.58%, respectively, while higher dropout rates of 0.35
and 0.4 overly regularized the model, causing a drop in accuracy due to
excessive feature dropout, resulting in lower accuracy of 96.25% and 94.15%,
respectively. Therefore, the optimized value of dropout rate is 0.3 taken in
this study leading to the best performance in lung disease classifications.

In-depth experiments are conducted to evaluate the performance of the fusion
approach, and it is observed that the four-layer fusion yields outstanding
results compared to singlelayer or three-layer fusion. However, fusing the
fifth layer leads to degraded results due to redundant feature maps and
increased computational overhead. Hence, we adopt the four-layer fusion for
our experiments.

An essential advantage of the proposed model is its computational efficiency,
outperforming existing methods in the number of trainable parameters. It is
evident from Table 4 that the number of trainable parameters used in the
proposed model is the least as compared to base models (ResNet and Inception).
We have successfully preserved the computational complexity after integrating
the two deep models. The proposed model has 16.01 million parameters,
approximately 67% of the parameters used in the computationally best available
base model. This substantial reduction allows our architecture to balance
computational efficiency and performance, making it an attractive solution for
resource-constrained scenarios where fast and efficient processing is
critical.

While deep learning-based models have been used for detecting COVID-19 and
Pneumonia lung diseases, our proposed model extends its applicability to the
healthcare industry. It can be deployed as an automated disease classifier at
the primary stage, allowing doctors to save time initiating treatment.
Moreover, it shows potential for use in other disease classification tasks and
may serve as an alternative in areas with limited access to radiologists.

###  7.1 Future directions of the proposed research

The quantitative and qualitative analysis suggests that our model is robust
and has the potential to be deployed in the healthcare system for accurate
disease classification and reducing the workload of radiologists. To further
extend the applicability of this research in the medical field, future
directions can explore the generalization capability of the model on other
lung diseases, such as Lung Cancer, Tuberculosis, and Pneumothorax. Moreover,
incorporating multimodal data, such as combining blood parameters with
radiological imaging, holds promise in improving detection accuracy and
providing a more comprehensive analysis of lung diseases. Exploring these
possibilities can lead to advancements in disease diagnosis and management,
ultimately benefiting patient care and healthcare systems.

##  8 Conclusions

In this study, we present a novel CNN-based fusion model tailored for the
classification of chest X-ray images. The primary goal of our research was to
enhance the accuracy and reliability of automated classification systems used
in diagnosing lung diseases, including COVID-19, Pneumonia, and Normal cases.
The proposed approach involves the multilayer fusion of ResNet50V2 and
InceptionV3 models, combined with multimodal fusion. Unlike previous studies
that focused on individual models or fusion techniques, our research
emphasizes the synergistic effects achieved through multilayer multimodal
fusion using state-of-the-art CNN architectures. Additionally, we have
introduced an independent feature map transform module to effectively fuse
feature maps of different sizes, resolving the variable-sized problem
encountered in generating multiple layers of feature maps. The experimental
results have demonstrated the promising potential of our approach in advancing
CXR classification, yielding improved accuracy and reliability. This work
opens new possibilities for computer-aided diagnosis (CAD) systems,
revolutionizing the field of medical image analysis.

## Compliance with Ethical Standards

The work is original and not submitted elsewhere. The authors do not have any
conflict of interest. This article does not contain any studies with human
participants or animals performed by any of the authors.

## Data availability statement

The original contributions presented in this study are publicly available.
This data and source code are available on https://github.com/MIILab-
IITGN/MultiFusionNet.

## Acknowledgment

YKM was supported by Indian Institute of Technology Gandhinagar startup grant
IP/IITGN/CSE/YM/2324/05. SA and KVA are supported by the ABV-Indian Institute
of Information Technology and Management, Gwalior, India

## Author contributions

SA, KVA, and YKM conceptualized and designed the methodology. SA prepared the
dataset and conducted the analyses. YKM and KVA supervised the study. SA and
YKM validated the results. YKM wrote the first draft of the manuscript. All
authors interpreted the data and revised the manuscript.

## References

  *   * Alassafi . (2022) ALASSAFI2022335Alassafi, M.O., Jarrah, M. Alotaibi, R.  2022\.  Time series predicting of COVID-19 based on deep learning Time series predicting of covid-19 based on deep learning.  Neurocomputing468335-344.
  * Al-Waisy . (2020) al2020covidAl-Waisy, A.S., Al-Fahdawi, S., Mohammed, M.A., Abdulkareem, K.H., Mostafa, S.A., Maashi, M.S.Garcia-Zapirain, B.  2020\.  COVID-CheXNet: hybrid deep learning framework for identifying COVID-19 virus in chest X-rays images Covid-chexnet: hybrid deep learning framework for identifying covid-19 virus in chest x-rays images.  Soft computing1–16.
  * Apostolopoulos Mpesiana (2020) TL-with-VGG19Apostolopoulos, I.D. Mpesiana, T.A.  2020\.  Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks.  Physical and engineering sciences in medicine432635–640.
  * Bera Shrivastava (2020) bera2020analysisBera, S. Shrivastava, V.K.  2020\.  Analysis of various optimizers on deep convolutional neural network model in the application of hyperspectral remote sensing image classification Analysis of various optimizers on deep convolutional neural network model in the application of hyperspectral remote sensing image classification.  International Journal of Remote Sensing4172664–2683.
  * Cannata . (2022) cannata2022deepCannata, S., Paviglianiti, A., Pasero, E., Cirrincione, G. Cirrincione, M.  2022\.  Deep Learning Algorithms for Automatic COVID-19 Detection on Chest X-Ray Images Deep learning algorithms for automatic covid-19 detection on chest x-ray images.  IEEE Access10119905–119913.
  * Chollet (2017) xceptionChollet, F.  2017\.  Xception: Deep learning with depthwise separable convolutions Xception: Deep learning with depthwise separable convolutions.  Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 1251–1258).
  * Choudhary . (2022) choudhary2022deepChoudhary, T., Gujar, S., Goswami, A., Mishra, V. Badal, T.  2022\.  Deep learning-based important weights-only transfer learning approach for COVID-19 CT-scan classification Deep learning-based important weights-only transfer learning approach for covid-19 ct-scan classification.  Applied Intelligence1–15.
  * Chowdhury . (2020) 24Chowdhury, M.E., Rahman, T., Khandakar, A., Mazhar, R., Kadir, M.A., Mahbub, Z.B.others  2020\.  Can AI help in screening viral and COVID-19 pneumonia? Can ai help in screening viral and covid-19 pneumonia?  IEEE Access8132665–132676.
  * Deb . (2022) sagarDeb, S.D., Jha, R.K., Jha, K. Tripathi, P.S.  2022\.  A multi model ensemble based deep convolution neural network structure for detection of COVID19 A multi model ensemble based deep convolution neural network structure for detection of covid19.  Biomedical Signal Processing and Control71103126.
  * Deng . (2009) deng2009imagenetDeng, J., Dong, W., Socher, R., Li, LJ., Li, K. Fei-Fei, L.  2009\.  Imagenet: A large-scale hierarchical image database Imagenet: A large-scale hierarchical image database.  2009 IEEE conference on computer vision and pattern recognition 2009 ieee conference on computer vision and pattern recognition ( 248–255).
  * Dhere Sivaswamy (2022) DhereDhere, A. Sivaswamy, J.  2022\.  COVID detection from Chest X-Ray Images using multi-scale attention Covid detection from chest x-ray images using multi-scale attention.  IEEE Journal of Biomedical and Health Informatics2641496–1505.
  * Fang . (2021) fang2021novelFang, Z., Ren, J., MacLellan, C., Li, H., Zhao, H., Hussain, A. Fortino, G.  2021\.  A novel multi-stage residual feature fusion network for detection of COVID-19 in chest X-ray images A novel multi-stage residual feature fusion network for detection of covid-19 in chest x-ray images.  IEEE Transactions on Molecular, Biological and Multi-Scale Communications8117–27.
  * Fu . (2023) fu2023pkaFu, Y., Xue, P., Zhang, Z. Dong, E.  2023\.  PKA 2-Net: Prior Knowledge-based Active Attention Network for Accurate Pneumonia Diagnosis on Chest X-ray Images Pka 2-net: Prior knowledge-based active attention network for accurate pneumonia diagnosis on chest x-ray images.  IEEE Journal of Biomedical and Health Informatics.
  * Gour Jain (2020) maheshGour, M. Jain, S.  2020\.  Stacked convolutional neural network for diagnosis of covid-19 disease from x-ray images Stacked convolutional neural network for diagnosis of covid-19 disease from x-ray images.  arXiv preprint arXiv:2006.13817.
  * Hayden Wrenn (2009) cxrHayden, G.E. Wrenn, K.W.  2009\.  Chest radiograph vs. computed tomography scan in the evaluation for pneumonia Chest radiograph vs. computed tomography scan in the evaluation for pneumonia.  The Journal of emergency medicine363266–270.
  * He . (2016) ResNetHe, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep residual learning for image recognition Deep residual learning for image recognition.  Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 770–778).
  * Ieracitano . (2022) IERACITANO2022202Ieracitano, C., Mammone, N., Versaci, M., Varone, G., Ali, AR., Armentano, A.others  2022\.  A fuzzy-enhanced deep learning approach for early detection of Covid-19 pneumonia from portable chest X-ray images A fuzzy-enhanced deep learning approach for early detection of covid-19 pneumonia from portable chest x-ray images.  Neurocomputing481202–215.
  * Ilhan . (2022) ilhan2022decisionIlhan, H.O., Serbes, G. Aydin, N.  2022\.  Decision and feature level fusion of deep features extracted from public COVID-19 data-sets Decision and feature level fusion of deep features extracted from public covid-19 data-sets.  Applied Intelligence5288551–8571.
  * Jacobi . (2020) jacobi2020portableJacobi, A., Chung, M., Bernheim, A. Eber, C.  2020\.  Portable chest X-ray in coronavirus disease-19 (COVID-19): A pictorial review Portable chest x-ray in coronavirus disease-19 (covid-19): A pictorial review.  Clinical imaging6435–42.
  * Kaggle (20211) 26Kaggle  20211\.  Chest X-Ray Images (Pneumonia). Chest x-ray images (pneumonia).  https://v.ht/WwR25
  * Kaggle (20212) 27Kaggle  20212\.  COVID-Net Open Source Initiative - COVIDx CXR-3 Dataset. Covid-net open source initiative - covidx cxr-3 dataset.  https://www.kaggle.com/andyczhao/COVIDx-cxr2?select= test
  * Kaya Gürsoy (2023) kaya2023mobilenetKaya, Y. Gürsoy, E.  2023\.  A MobileNet-based CNN model with a novel fine-tuning mechanism for COVID-19 infection detection A mobilenet-based cnn model with a novel fine-tuning mechanism for covid-19 infection detection.  Soft Computing1–15.
  * A. Khan . (2021) KHAN2021111Khan, A., Chefranov, A. Demirel, H.  2021\.  Image scene geometry recognition using low-level features fusion at multi-layer deep CNN Image scene geometry recognition using low-level features fusion at multi-layer deep cnn.  Neurocomputing440111–126.
  * A.I. Khan . (2020) CoroNetKhan, A.I., Shah, J.L. Bhat, M.M.  2020\.  CoroNet: A deep neural network for detection and diagnosis of COVID-19 from chest x-ray images Coronet: A deep neural network for detection and diagnosis of covid-19 from chest x-ray images.  Computer Methods and Programs in Biomedicine196105581.
  * Kingma Ba (2014) kingma2014adamKingma, D.P. Ba, J.  2014\.  Adam: A method for stochastic optimization Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * Krizhevsky . (2012) AlexNetKrizhevsky, A., Sutskever, I. Hinton, G.E.  2012\.  Imagenet classification with deep convolutional neural networks Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems25.
  * LeCun . (1989) LeNetLeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W. Jackel, L.D.  1989\.  Backpropagation applied to handwritten zip code recognition Backpropagation applied to handwritten zip code recognition.  Neural computation14541–551.
  * Li . (2022) zhengLi, Z., Xu, X., Cao, X., Liu, W., Zhang, Y., Chen, D. Dai, H.  2022\.  Integrated CNN and Federated Learning for COVID-19 Detection on Chest X-Ray Images Integrated cnn and federated learning for covid-19 detection on chest x-ray images.  IEEE/ACM Transactions on Computational Biology and Bioinformatics.
  * Meena Arya (2023) meena2023multimodalMeena, Y.K. Arya, K.  2023\.  Multimodal interaction and IoT applications Multimodal interaction and iot applications.  Multimedia Tools and Applications8244781–4785.
  * Mohagheghi . (2021) MohagheghiMohagheghi, S., Alizadeh, M., Safavi, S.M., Foruzan, A.H. Chen, YW.  2021\.  Integration of CNN, CBMIR, and visualization techniques for diagnosis and quantification of covid-19 disease Integration of cnn, cbmir, and visualization techniques for diagnosis and quantification of covid-19 disease.  IEEE Journal of Biomedical and Health Informatics2561873–1880.
  * Muhammad Hossain (2021) ultra-soundMuhammad, G. Hossain, M.S.  2021\.  COVID-19 and non-COVID-19 classification using multi-layers fusion from lung ultrasound images Covid-19 and non-covid-19 classification using multi-layers fusion from lung ultrasound images.  Information Fusion7280–88.
  * Narin . (2021) Narin_2021Narin, A., Kaya, C. Pamuk, Z.  2021\.  Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks Automatic detection of coronavirus disease (covid-19) using x-ray images and deep convolutional neural networks.  Pattern Analysis and Applications2431207–1220.
  * Ozturk . (2020) DarkCovidNetOzturk, T., Talo, M., Yildirim, E.A., Baloglu, U.B., Yildirim, O. Acharya, U.R.  2020\.  Automated detection of COVID-19 cases using deep neural networks with X-ray images Automated detection of covid-19 cases using deep neural networks with x-ray images.  Computers in biology and medicine121103792.
  * Pan Yang (2009) 6Pan, S.J. Yang, Q.  2009\.  A survey on transfer learning A survey on transfer learning.  IEEE Transactions on knowledge and data engineering22101345–1359.
  * Pathak . (2020) pathakPathak, Y., Shukla, P.K. Arya, K.  2020\.  Deep bidirectional classification model for COVID-19 disease infected patients Deep bidirectional classification model for covid-19 disease infected patients.  IEEE/ACM Transactions on Computational Biology and Bioinformatics1841234–1241.
  * Quilodrán-Casas . (2022) QUILODRANCASAS202211Quilodrán-Casas, C., Silva, V.L., Arcucci, R., Heaney, C.E., Guo, Y. Pain, C.C.  2022\.  Digital twins based on bidirectional LSTM and GAN for modelling the COVID-19 pandemic Digital twins based on bidirectional lstm and gan for modelling the covid-19 pandemic.  Neurocomputing47011–28.
  * Rahimzadeh Attar (2020) RAHIMZADEH2020100360Rahimzadeh, M. Attar, A.  2020\.  A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2 A modified deep convolutional neural network for detecting covid-19 and pneumonia from chest x-ray images based on the concatenation of xception and resnet50v2.  Informatics in medicine unlocked19100360.
  * Rahman . (2021) 25Rahman, T., Khandakar, A., Qiblawey, Y., Tahir, A., Kiranyaz, S., Kashem, S.B.A.others  2021\.  Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images.  Computers in biology and medicine132104319.
  * Shi . (2021) shiShi, W., Tong, L., Zhu, Y. Wang, M.D.  2021\.  COVID-19 automatic diagnosis with radiographic imaging: Explainable attention transfer deep neural networks Covid-19 automatic diagnosis with radiographic imaging: Explainable attention transfer deep neural networks.  IEEE Journal of Biomedical and Health Informatics2572376–2387.
  * Simonyan Zisserman (2014) VGGSimonyan, K. Zisserman, A.  2014\.  Very deep convolutional networks for large-scale image recognition Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556.
  * Srivastava . (2022) srivastava2022covixnetSrivastava, G., Chauhan, A., Jangid, M. Chaurasia, S.  2022\.  CoviXNet: A novel and efficient deep learning model for detection of COVID-19 using chest X-Ray images Covixnet: A novel and efficient deep learning model for detection of covid-19 using chest x-ray images.  Biomedical Signal Processing and Control78103848.
  * Subramanian . (2022) subramanian2022reviewSubramanian, N., Elharrouss, O., Al-Maadeed, S. Chowdhury, M.  2022\.  A review of deep learning-based detection methods for COVID-19 A review of deep learning-based detection methods for covid-19.  Computers in Biology and Medicine105233.
  * Szegedy . (2017) inceptionSzegedy, C., Ioffe, S., Vanhoucke, V. Alemi, A.A.  2017\.  Inception-v4, inception-resnet and the impact of residual connections on learning Inception-v4, inception-resnet and the impact of residual connections on learning.  Thirty-first AAAI conference on artificial intelligence. Thirty-first aaai conference on artificial intelligence.
  * Tabik . (2020) TabikTabik, S., Gómez-Ríos, A., Martín-Rodríguez, J.L., Sevillano-García, I., Rey-Area, M., Charte, D.others  2020\.  COVIDGR dataset and COVID-SDNet methodology for predicting COVID-19 based on chest X-ray images Covidgr dataset and covid-sdnet methodology for predicting covid-19 based on chest x-ray images.  IEEE journal of biomedical and health informatics24123595–3605.
  * Tan . (2022) TAN202236Tan, T., Das, B., Soni, R., Fejes, M., Yang, H., Ranjan, S.others  2022\.  Multi-modal trained artificial intelligence solution to triage chest X-ray for COVID-19 using pristine ground-truth, versus radiologists Multi-modal trained artificial intelligence solution to triage chest x-ray for covid-19 using pristine ground-truth, versus radiologists.  Neurocomputing48536–46.
  * Tang . (2021) Trans-edlTang, S., Wang, C., Nie, J., Kumar, N., Zhang, Y., Xiong, Z. Barnawi, A.  2021\.  EDL-COVID: Ensemble deep learning for COVID-19 case detection from chest X-ray images Edl-covid: Ensemble deep learning for covid-19 case detection from chest x-ray images.  IEEE Transactions on Industrial Informatics1796539–6549.
  * Tangudu . (2022) tangudu2022covidTangudu, V., Kakarla, J. Venkateswarlu, I.B.  2022\.  COVID-19 detection from chest x-ray using MobileNet and residual separable convolution block Covid-19 detection from chest x-ray using mobilenet and residual separable convolution block.  Soft Computing2652197–2208.
  * Waheed . (2020) CovidganWaheed, A., Goyal, M., Gupta, D., Khanna, A., Al-Turjman, F. Pinheiro, P.R.  2020\.  Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection Covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection.  Ieee Access891916–91923.
  * L. Wang . (2020) CovidnetWang, L., Lin, Z.Q. Wong, A.  2020\.  Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images Covid-net: A tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images.  Scientific Reports1011–12.
  * SH. Wang . (2023) wang2023elucnnWang, SH., Satapathy, S.C., Xie, MX. Zhang, YD.  2023\.  ELUCNN for explainable COVID-19 diagnosis Elucnn for explainable covid-19 diagnosis.  Soft Computing1–17.
  * WHO (2022) PneumoniaDashboardWHO  2022\.  Pneumonia. Pneumonia.  https://v.ht/X2oOi
  * Wu . (2021) wu2021evolvingWu, C., Khishe, M., Mohammadi, M., Taher Karim, S.H. Rashid, T.A.  2021\.  Evolving deep convolutional neutral network by hybrid sine–cosine and extreme learning machine for real-time COVID19 diagnosis from X-ray images Evolving deep convolutional neutral network by hybrid sine–cosine and extreme learning machine for real-time covid19 diagnosis from x-ray images.  Soft Computing1–20.
  * Xu . (2021) XU202196Xu, Y., Lam, HK. Jia, G.  2021\.  MANet: A two-stage deep learning method for classification of COVID-19 from Chest X-ray images Manet: A two-stage deep learning method for classification of covid-19 from chest x-ray images.  Neurocomputing44396–105.
  * Yu . (2021) YU2021592Yu, X., Lu, S., Guo, L., Wang, SH. Zhang, YD.  2021\.  ResGNet-C: A graph convolutional neural network for detection of COVID-19 Resgnet-c: A graph convolutional neural network for detection of covid-19.  Neurocomputing452592–605.
  * Zhou . (2021) zhouZhou, J., Jing, B., Wang, Z., Xin, H. Tong, H.  2021\.  Soda: Detecting covid-19 in chest x-rays with semi-supervised open set domain adaptation Soda: Detecting covid-19 in chest x-rays with semi-supervised open set domain adaptation.  IEEE/ACM Transactions on Computational Biology and Bioinformatics.