# MV-Swin-T: Mammogram Classification with Multi-View Swin Transformer

###### Abstract

Traditional deep learning approaches for breast cancer classification has
predominantly concentrated on single-view analysis. In clinical practice,
however, radiologists concurrently examine all views within a mammography
exam, leveraging the inherent correlations in these views to effectively
detect tumors. Acknowledging the significance of multi-view analysis, some
studies have introduced methods that independently process mammogram views,
either through distinct convolutional branches or simple fusion strategies,
inadvertently leading to a loss of crucial inter-view correlations. In this
paper, we propose an innovative multi-view network exclusively based on
transformers to address challenges in mammographic image classification. Our
approach introduces a novel shifted window-based dynamic attention block,
facilitating the effective integration of multi-view information and promoting
the coherent transfer of this information between views at the spatial feature
map level. Furthermore, we conduct a comprehensive comparative analysis of the
performance and effectiveness of transformer-based models under diverse
settings, employing the CBIS-DDSM and Vin-Dr Mammo datasets. Our code is
publicly available at https://github.com/prithuls/MV-Swin-T

Index Terms—  Mammogram, Multi-view, Breast Mass Classification, Transformer.

##  1 Introduction

Breast cancer ranks as the second leading cause of cancer-related deaths among
women globally, making it the most prevalent cancer affecting them [1]. Early
detection primarily relies on screening mammography, encompassing four
images—two from each breast taken from different angles: mediolateral-oblique
(MLO) from the side and Cranial-Caudal (CC) from above. While conventional
deep learning methods for breast cancer classification have focused on single-
view analysis, radiologists concurrently assess all views in mammography
exams, recognizing valuable correlations that provide crucial tumor
information. This highlights the importance of cross-view data analysis in
healthcare for identifying anomalies and making diagnoses as well as the
significance of multi-view or multi-image-based Computer-Aided Diagnosis (CAD)
schemes over single image-based CAD schemes. Recent research in breast cancer
classification and detection has applied deep learning techniques with
promising outcomes. Many current studies [2, 3, 4] aim to incorporate multi-
view architectures inspired by radiologists’ multi-view analysis, contributing
to more robust and high-performing models.

Efforts to integrate multiple views into breast cancer screening through the
utilization of Deep Neural Networks (DNNs) have roots in the work of Carneiro
et al. [5]. They trained models individually on the MLO and CC views, then
employed the features from the final fully connected layer to train a
multinomial logistic regression model. In a parallel work, Sridevi et al. in
[6] put forth a classification strategy utilizing CC and MLO views. Their
methodology encompassed initial image normalization, pectoral muscle removal,
and subsequent feature extraction through convolution and pooling layers, with
the extracted features being concatenated. In [7], Khan et al. introduced a
two-stage classification strategy involving the use of extracted ROIs from
four mammogram views. Various CNNs were employed as feature extractors, and
the extracted features from all views were concatenated using an early fusion
strategy, culminating in the final output through the classifier layer.

Recently, attention has become a core concept in deep learning due to its
widespread application. The objective of an attention mechanism in this
context is to enable the model to focus selectively on relevant local input
regions and feature channels, thereby avoiding equal treatment of all
locations and features. In medical image analysis, where diagnoses often hinge
on specific isolated areas of concern, recent research has explored the
integration of attention mechanisms into multi-view mammography analysis.
However, these endeavors still partially rely on convolutional methods and do
not fully exploit the inherent potential of attention mechanisms. The self-
attention mechanism, intrinsic to transformers, empowers models to dynamically
discern where and what to focus on, utilizing pertinent image regions or
features to augment task performance. With self-attention, vision transformers
[8] excel at capturing long-range dependencies within input sequences.

Despite the evident promise of transformers in modeling long-range
dependencies, their application in multi-view mammogram analysis remains
relatively uncharted territory. Some studies, such as [9], have embraced
hybrid models combining transformers and CNNs, introducing global cross-view
transformer blocks to amalgamate intermediate feature maps from CC and MLO
views. Another noteworthy work is [10], which employed a transformer-based
model for breast cancer segment detection. However, they processed multi-views
at a later stage of the network, missing opportunities to capture local
correlations between views and lacked results on publicly available datasets,
thereby constraining comparability with existing literature.

To fully exploit multi-view insights, we present a novel transformer-based
multi-view network, MV-Swin-T, built upon the Swin Transformer [11]
architecture for mammographic image classification. Our contributions include:

  1. 1.

Designing a novel multi-view network entirely based on the transformer
architecture, capitalizing on the benefits of transformer operations for
enhanced performance.

  2. 2.

A novel ”Multi-headed Dynamic Attention Block (MDA)” with fixed and shifted
window features to enable self and cross-view information fusion from both CC
and MLO views of the same breast.

  3. 3.

Addressing the challenge of effectively combining data from multiple views or
modalities, especially when images may not align correctly.

  4. 4.

We present results using the publicly available CBIS-DDSM And VinDr-Mammo
dataset.

Moreover, to comprehend the impact of transformers and different associated
modules, we introduce various architectural changes throughout the training
process, analyzing their overall effects on the entire network. Detailed
insights into these analyses are provided in the 3.2 section.

##  2 Methodology

Our approach focuses on developing a specialized network based on the Swin
Transformer architecture, specifically designed for classifying unregistered
multi-view mammogram pairs. To enhance the model’s capacity, we introduce the
novel Omni-Attention transformer block with advanced multi-head dynamic-
attention mechanisms and both regular and shifted window configurations. It is
imperative to underscore that our experimental focus is exclusively on
ipsilateral-views, deliberately excluding bilateral views. Ipsilateral
analysis involves diagnosis based on the CC and MLO views of the same breast,
while bilateral analysis integrates findings from corresponding views of both
breasts. This deliberate choice aligns with clinical norms, where radiologists
typically employ ipsilateral analysis to classify breast tumor masses, and
leverage symmetry information from bilateral analysis to detect asymmetries,
as highlighted in [12]. As our architecture lacks a standard backbone
structure, we avoided using pre-training and opted for a distinctive approach.

Omni-Attention Transformer Block: The Omni-Attention transformer block was
constructed by substituting the window and shifted window multi-head self-
attention (MSA) module, as presented in [11], with the regular and shifted
window multi-head dynamic attention (W-MDA and SW-MDA) module, while
maintaining consistency in other layers. The architectural modification is
shown in Figure 2. Each multi-head dynamic attention (MDA) module is succeeded
by a 2-layer Multi-Layer Perceptron (MLP) with Rectified Linear Unit (ReLU)
nonlinearity in between. It is noteworthy that, contrary to the Swin
Transformer paper, we have opted for ReLU nonlinearity instead of GELU
nonlinearity. To enhance stability and facilitate convergence, a LayerNorm
(LN) layer is applied before each MDA module and each MLP, with a residual
connection applied after each module.

Fig. 1: Illustration of self and cross attention operations within the
proposed Multi-head Dynamic Attention (MDA) block for ipsilateral views. The
term ’matmul’ denotes matrix multiplication.

Multi-Head Dynamic Attention: In each W-MDA block, we integrate both self-
attention and cross-attention mechanisms for both CC and MLO views, calculated
within the local windows. These windows are strategically arranged to evenly
partition the image in a non-overlapping manner, with a fixed size of 7 for
images with dimensions 224 $\times$ 224 and 12 for images with dimensions 384
$\times$ 384\. The calculation process for both self and cross-attention is
identical, with the key distinction being that for self-attention, the $Q$,
$K$, $V$ matrices are obtained from the same view. In contrast, for cross-view
attention, the $K$ matrix is derived from the other view, while the $Q$ and
$V$ matrices are derived from the origin view as showed in Figure 1.

$\begin{aligned}
DA_{CC}=[\text{Linear}(\sigma(\frac{Q_{CC}K_{CC}^{T}}{\text{scale}})\>\oplus\sigma(\frac{Q_{CC}K_{MLO}^{T}}{\text{scale}}))]V_{CC}\end{aligned}$

(1)

$\begin{aligned} W\text{-}MDA/SW\text{-}MDA=\langle
DA_{CC},DA_{MLO}\rangle\end{aligned}$

(2)

Equation 1 demonstrates the dynamic attention from input CC view. The $Q$,
$K$, $V$ represents the query, key and value of the corresponding inputs
respectively, and $\sigma$ represents softmax operation. We denote the fully
connected layer as linear in the equation. $W$-$MDA$ and $SW$-$MDA$ can be
constructed with corresponding dynamic attention of $DA_{CC},DA_{MLO}$
respectively. In equation 3, we depict the first sub-block of the Omni-
Attention transformer, comprising a LayerNorm (LN) layer, a multi-head dynamic
attention module (W-MDA), and a 2-layer MLP with ReLU non-linearity. Here, $l$
denotes the layer number, and $Z$ represents the feature map.

$\displaystyle\hat{Z}^{l}$ $\displaystyle=W\text{-}MDA(LN(Z^{l-1}))+Z^{l-1},$
(3) $\displaystyle Z^{l}$ $\displaystyle=MLP(LN(\hat{Z}^{l}))+\hat{Z}^{l}.$
$\displaystyle\hat{Z}^{l+1}$ $\displaystyle=SW\text{-}MDA(LN(Z^{l}))+Z^{l},$
(4) $\displaystyle Z^{l+1}$
$\displaystyle=MLP(LN(\hat{Z}^{l+1}))+\hat{Z}^{l+1}.$

Shifted Window Multi-head Dynamic Attention: The dynamic-attention module
based on fixed windows lacks inter-window connections, thereby limiting its
modeling capabilities. To incorporate cross-window connections while
preserving the computational efficiency of non-overlapping windows, we adopted
a shifted window partitioning approach similar to [11]. Equation 4 illustrates
the second sub-block of the Swin Transformer, comprising an LN layer, a
shifted window multi-head dynamic attention module (SW-MDA), and an MLP with
ReLU activation.

The proposed MV-Swin-T architecture represents Omni-Attention blocks
integrated into the initial two stages of the network. Following the second
stage, the outputs from different views are concatenated and channeled through
a fully connected layer to maintain consistent dimensions with a single-view
mammogram. The processed output is then directed through Swin Transformer
blocks in the third and fourth stages, ultimately resulting in the final
output.

Fig. 2: (a) Our proposed multi-view architecture. (b) Two successive Omni-
Attention Transformer Blocks featuring W-MDA and SW-MDA components for Multi-
Head Dynamic-Attention with regular and shifted window configurations. While
presented as a single-input diagram for simplicity, here, Z represents the
combined representations of CC and MLO, $Z=\langle Z_{CC},Z_{MLO}\rangle.$

##  3 Experiments and Results

###  3.1 Dataset & Implementation Setting

We evaluated our proposed architecture using the Curated Breast Imaging Subset
of the Digital Database for Screening Mammography (CBIS-DDSM) [13] and VinDr-
Mammo datasets [14]. The CBIS dataset comprises 2462 mammography images from
1231 women, providing both CC and MLO views for most cases. After data
preparation, we used 509 patient data records for training and 151 for
testing, excluding cases lacking both views. Additionally, we conducted
experiments incorporating calcification data from the DDSM dataset, resulting
in a total of 1062 images for training and 253 for testing. The VinDr-Mammo
dataset comprises 5000 four-view exams, yielding 8000 images for training and
2000 for testing when considering CC and MLO views of the same breast.

To enhance model robustness, we applied data augmentation techniques,
including random horizontal and vertical flipping, as well as random rotation,
to all images. Additionally, we resized all images to 224$\times$224 and
384$\times$384 before inputting them into the models. For preprocessing, we
followed a similar process as discussed in [15] for the CBIS dataset, while
for the VinDr-Mammo dataset, we employed the OpenCV Python package for image
normalization and enhancement. We categorized cases with BIRADS values of 1 to
3 as benign and those with values of 4 to 6 as malignant for the VinDr-Mammo
Dataset.

We developed the model using PyTorch and conducted training using the Adam
optimizer [16], initializing with learning rate annealing of 0.0001 and a
weight decay of 0.001. For both the dataset, we used binary cross-entropy loss
and threshold value of 0.5 to make decisions about class assignments. We
conducted our training for 100 epochs, employing a ReduceLROnPlateau learning
rate scheduler and an early stopping mechanism. Notably, the optimal result
was consistently achieved in fewer than 50 epochs across all experiments.

###  3.2 Results and Analysis

Table 1: Comparison of the proposed architecture on different architectural
settings on the CBIS-DDSM and VinDr-Mammo dataset. Here, Param.: No of
parameters (in million), AUC: Area under the curve, Acc.: Accuracy, WA:
Weighted Addition, Con.: Concatenation

Dataset |  Fusion Stages |  Image Size |  Attention Fusion | Param. | AUC | Acc.
---|---|---|---|---|---|---
| (%) | (%)
CBIS (Mass) | 4 | 2242 | WA | 55 | 66.16 | 64.58
4 | 3842 | Con. | 55 | 67.44 | 66.26
3 | 3842 | Con. | 40 | 69.28 | 67.21
2 | 2242 | Con. | 27 | 69.75 | 66.82
2 | 3842 | Con. | 29 | 71.37 | 68.63
CBIS (Mass & Calc) | 2 | 2242 | Con. | 27 | 65.08 | 63.74
2 | 3842 | Con. | 29 | 66.43 | 65.37
2 | 4482 | Con. | 30 | 64.66 | 63.19
VinDr- Mammo | 2 | 2242 | Con. | 27 | 95.88 | 95.00
2 | 3842 | Con. | 29 | 96.08 | 95.50

We explored various configurations, and the optimal model, featuring cross-
view attention, is illustrated in Figure 2. Table 1 presents a comparative
analysis of different architectural settings, specifically focusing on the
’Fusion Stage’ column, which signifies CC and MLO view fusion at the stage
level. Our experiments revealed that implementing fusion after the second
stage consistently yielded the best results across different datasets. This
reduction in parameters significantly enhances model efficiency, preventing
overfitting and reducing computational complexity, particularly beneficial
when handling a limited number of training images. Also, early integration
after the second stage ensures a comprehensive representation, considering
both views, thereby improving the model’s contextual understanding and overall
performance.

The ”Attention Fusion” column indicates how attention from CC and MLO views is
combined in the MDA modules. For this, we conducted experiments with weighted
addition and concatenation. In the weighted addition method, we tested various
weight values ranging from 0.9 to 0.5. However, we only presented results for
the value set of 0.9 and 0.1, where 0.9 is multiplied with the same view, and
0.1 is multiplied with the cross view, as this setting yielded the best result
among all the value sets. Nevertheless, the overall best result was achieved
with the concatenation method. Additionally, we examined two image sizes
(224$\times$224 and 384$\times$384) and diversified the training datasets to
include mass and mass-calc. The model delivers better result for image size
384$\times$384\. We conclude that as the success of the classification task
relies not only on accurate mass localization but also on effectively
attending to the mass boundary, increasing the image size augments the
network’s ability to comprehend mass boundaries.

Table 2: Comparing our MV-Swin-T to baseline Swin Transformer (Swin-T) on
CBIS-DDSM and VinDr-Mammo datasets, using multi-view input versus single
views. Results presented for an image size of 384$\times$384.

Dataset |  Architecture Name |  Number of View | AUC | Acc.
---|---|---|---|---
(%) | (%)
CBIS (Mass) | Swin-T | Single | 60.26 | 59.64
MV-Swin-T (ours) | Two | 71.37 | 68.63
CBIS (Mass & Calc) | Swin-T | Single | 61.28 | 60.46
MV-Swin-T (ours) | Two | 66.43 | 65.37
VinDr- Mammo | Swin-T | Single | 95.88 | 95.50
MV-Swin-T (ours) | Two | 96.08 | 95.50

Table 2 presents a comparative analysis of our proposed MV-Swin-T architecture
and the baseline Swin-T architecture, operating on single views. The results
distinctly showcase the superior performance of our proposed model across all
datasets. This further bloster the notion that assessment based on multi-view
fusion can effectively integrate relevant information from both views,
mitigating noise interference from single views and yielding more accurate
predictions. The model particularly showed optimum results for VinDr-Mammo
dataset. We attribute this performance gap to the varying image quality
between the two datasets. Additionally, the larger size of the VinDr dataset
could contribute to the superior performance, as deep learning models often
benefit from larger datasets for optimal results.

##  4 Discussion and Conclusion

Transformers have gained prominence as effective architectures for capturing
long-range dependencies in recent years. However, their potential in the field
of multi-view mammogram analysis remains significantly untapped. In this
research, we present a pure transformer-based multi-view network for breast
cancer classification, designed to emulate the way radiologists interpret
multi-view mammograms and utilize the information present in ipsilateral
views. We introduce a novel Multi-Headed Dynamic Attention (MDA) block that
implicitly learns to emphasize cancer-related local anomalies and accentuates
essential features by exploring cross-view information between two mammogram
views. This study involved extensive experimentation, exploring various
configurations across multiple datasets. In the concluding section, we
showcased the performance of our multi-view approach in contrast to single-
view methods. The results underscored the superiority of the multi-view
approach, highlighting its effectiveness in enhancing overall performance. We
believe that our approach represents a significant breakthrough in this
research field, opening the door to novel techniques that can process medical
imaging exams more similar to how radiologists analyze. Exploring strategies
to investigate the scalability of the multi-view approach to even larger
datasets could provide additional insights and improvements to the model’s
generalization capabilities.

##  5 Acknowledgments

This work has received support from the following grants and awards.

  * •

National Science Foundation under Grant No. OIA- 2148788.

  * •

Nevada DRIVE program (Doctoral Research in Innovation, Vision and Excellence)

##  6 Compliance with Ethical Standards

This research study was conducted retrospectively using human subject data
made available in open access by [13, 14]. Ethical approval was not required
as confirmed by the license attached with the open access data.

## References

  * [1] American Cancer Society,  “Breast cancer facts & figures 2019–2020,”  Am Cancer Soc, pp. 1–44, 2019.
  * [2] Eleonora Lopez, Eleonora Grassucci, Martina Valleriani, and Danilo Comminiello,  “Hypercomplex neural architectures for multi-view breast cancer classification,”  arXiv preprint arXiv:2204.05798, 2022.
  * [3] Lilei Sun, Junqian Wang, Zhijun Hu, Yong Xu, and Zhongwei Cui,  “Multi-view convolutional neural networks for mammographic image classification,”  IEEE Access, vol. 7, pp. 126273–126282, 2019.
  * [4] Trent Kyono, Fiona J Gilbert, and Mihaela Van Der Schaar,  “Triage of 2d mammographic images using multi-view multi-task convolutional neural networks,”  ACM Transactions on Computing for Healthcare, vol. 2, no. 3, pp. 1–24, 2021.
  * [5] Gustavo Carneiro, Jacinto Nascimento, and Andrew P Bradley,  “Deep learning models for classifying mammogram exams containing unregistered multi-view images and segmentation maps of lesions,”  Deep learning for medical image analysis, pp. 321–339, 2017.
  * [6] V Sridevi and J Abdul Samath,  “A combined deep cnn-lasso regression feature fusion and classification of mlo and cc view mammogram image,”  International Journal of System Assurance Engineering and Management, pp. 1–11, 2023.
  * [7] Hasan Nasir Khan, Ahmad Raza Shahid, Basit Raza, Amir Hanif Dar, and Hani Alquhayz,  “Multi-view feature fusion based four views model for mammogram classification using convolutional neural network,”  IEEE Access, vol. 7, pp. 165724–165733, 2019.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Gijs van Tulder, Yao Tong, and Elena Marchiori,  “Multi-view analysis of unregistered medical images using cross-view transformers,”  in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24. Springer, 2021, pp. 104–113.
  * [10] Xuxin Chen, Ke Zhang, Neman Abdoli, Patrik W Gilley, Ximin Wang, Hong Liu, Bin Zheng, and Yuchen Qiu,  “Transformers improve breast cancer diagnosis from unregistered multi-view mammograms,”  Diagnostics, vol. 12, no. 7, pp. 1549, 2022.
  * [11] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10012–10022.
  * [12] Zhicheng Yang, Zhenjie Cao, Yanbo Zhang, Yuxing Tang, Xiaohui Lin, Rushan Ouyang, Mingxiang Wu, Mei Han, Jing Xiao, Lingyun Huang, et al.,  “Momminet-v2: Mammographic multi-view mass identification networks,”  Medical Image Analysis, vol. 73, pp. 102204, 2021.
  * [13] Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin,  “A curated mammography data set for use in computer-aided detection and diagnosis research,”  Scientific data, vol. 4, no. 1, pp. 1–9, 2017.
  * [14] Hieu T Nguyen, Ha Q Nguyen, Hieu H Pham, Khanh Lam, Linh T Le, Minh Dao, and Van Vu,  “Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography,”  Scientific Data, vol. 10, no. 1, pp. 277, 2023.
  * [15] Prithul Sarker, Sushmita Sarker, George Bebis, and Alireza Tavakkoli,  “Connectedunets++: Mass segmentation from whole mammographic images,”  in International Symposium on Visual Computing. Springer, 2022, pp. 419–430.
  * [16] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.