energetic framework. Moreover note that, since the energy based framework is
predicated on ignoring the vicinity of the singularity, we do not obtain any
equilibrium condition at the point of singularity and hence forces can not be
prescribed at the tip of the cone. In our framework, a point force and a point
force dipole can be prescribed at the singular point through the integral
balance law (38b).

Finally, we briefly discuss the problem of constrained deformation where the
sheet is unilaterally constrained to adhere to an externally imposed rigid
shape. An example is afforded by the problem of squeezing a sheet of paper
inside a hollow sphere (or inside our fist). The sheet responds by crumpling
and developing a distribution of kinks and creases [23]. A simpler example is
to try to fit a sheet of paper inside a hollow cone [3]. This yields a
developable cone (conical singularity with $s=0$) such that there are portions
of the of the sheet directly in contact with the outer rigid cone and portions
which are folded away. A conical deformation in such a scenario would satisfy
the equilibrium of the form $D\Delta^{2}W-[\Phi,W]=B$ where $B$ is the
distributional contact force supported over the region where the sheet is in
contact with the rigid cone. If the constrained deformation has been achieved
by applying a point force then the singular force should be incorporated
within $B$.

###  4.2 Folds

In this section, we assume the domain $\Omega$ to be without any point $O$ of
singularity and having a singular interface $S$ which is a smooth curve such
that $\partial S-\partial\Omega=\emptyset$. We assume both the bulk and the
interfacial fields, and their derivatives, to remain bounded in the entire
domain. We restrict our attention to elastically inextensible plates (i.e.,
$\boldsymbol{E}^{e}=\boldsymbol{0}$ in $\Omega$). We define a fold as an
interfacial concentration in plastic bending strain. For the present
discussion we will assume that there is no bulk contribution to the plastic
bending strain, i.e.,
$\boldsymbol{\Lambda}^{p}\in\mathcal{C}(\Omega,\operatorname{Sym})$ with a
smooth line density $\boldsymbol{\gamma}$ such that
$\boldsymbol{\Lambda}^{p}(\boldsymbol{\psi})=\int_{S}\langle\boldsymbol{\gamma},\boldsymbol{\psi}\rangle\operatorname{dl}$
for all $\boldsymbol{\psi}\in\mathcal{D}(\Omega,\operatorname{Lin})$. The
compatibility condition
$\boldsymbol{\gamma}\times\boldsymbol{\nu}=\boldsymbol{0}$ implies that
$\boldsymbol{\gamma}$ is of the form
$\boldsymbol{\gamma}=\gamma_{0}\boldsymbol{\nu}\otimes\boldsymbol{\nu}$. The
fold is therefore prescribed in terms of a smooth scalar field $\gamma_{0}$
(the strength of the fold) and the geometry of the curve $S$. Moreover

$\llbracket\nabla
w\rrbracket=-\gamma_{0}\boldsymbol{\nu}\otimes\boldsymbol{\nu}.$ (58)

With the above considerations, and under the assumption of elastic
inextensibility, the first von Kármán equation (42) reduces to

$\displaystyle\frac{1}{2}[w,w]=-\operatorname{curl}\operatorname{curl}\boldsymbol{e}^{p}~{}\text{in}~{}\Omega-S,$
(59a) $\displaystyle\gamma_{0}\langle\\{\nabla\nabla
w\\},\boldsymbol{t}\otimes\boldsymbol{t}\rangle=\langle\llbracket\nabla\boldsymbol{e}^{p}\rrbracket,\boldsymbol{r}\rangle+k\langle\llbracket\boldsymbol{e}^{p}\rrbracket,\boldsymbol{\nu}\otimes\boldsymbol{\nu}\rangle~{}\text{on}~{}S,~{}\text{and}$
(59b)
$\displaystyle\langle\llbracket\boldsymbol{e}^{p}\rrbracket,\boldsymbol{t}\otimes\boldsymbol{t}\rangle=0~{}\text{on}~{}S,$
(59c)

while the second von Kármán equation (44), in the absence of external forces,
takes the form

$\displaystyle D\Delta^{2}w-[\phi,w]=0~{}\text{in}~{}\Omega-S,$ (60a)
$\displaystyle-D(1-\nu)\frac{d^{2}\gamma_{0}}{ds^{2}}+D\langle\llbracket\operatorname{div}\nabla\nabla
w\rrbracket,\boldsymbol{\nu}\rangle+\gamma_{0}\langle\nabla\nabla\phi,\boldsymbol{t}\otimes\boldsymbol{t}\rangle=0~{}\text{on}~{}S,~{}\text{and}$
(60b) $\displaystyle\llbracket\Delta
w\rrbracket=(1-\nu)k\gamma_{0}~{}\text{on}~{}S.$ (60c)

(a) A linear fold.

(b) A circular fold.

Figure 2: The transverse deformation of a flat, elastically inextensible,
sheet in the presence of a linear and a circular fold. In the former,
curvature on one side of the fold develops in response to the moment and
transverse force applied at the edge. In the latter, a distribution of
transverse force couples at the fold location is required to maintain the
desired shape of the deformation. The fold $S$ is indicated with a bold black
line.

We discuss the nature of these two sets of equations in the context of two
problems of folding an elastic sheet along a singular curve. First, we
consider a fold along a straight line $S$ (hence $\boldsymbol{\nu}$ is fixed
along $S$) such that the deformation on either side of the fold is
cylindrical, i.e.,
$\boldsymbol{\lambda}=g(\langle\boldsymbol{x},\boldsymbol{c}\rangle)\boldsymbol{c}\otimes\boldsymbol{c}$,
where $\boldsymbol{c}$ is a constant vector; see Figure 2(a). A transverse
deformation which satisfies $\boldsymbol{\lambda}=\nabla\nabla w$ (outside
$S$) is of the form $w=f(\langle\boldsymbol{x},\boldsymbol{c}\rangle)$ such
that ${d^{2}f}/{d{q}^{2}}=g$, where
$q=\langle\boldsymbol{x},\boldsymbol{c}\rangle$. Let $q=0$ represent $S$. At
the fold $S$, (58) implies
$\llbracket{df}/{d{q}}\rrbracket\boldsymbol{c}=-\gamma_{0}\boldsymbol{\nu}$,
from which we conclude that the interfacial line is necessarily orthogonal to
$\boldsymbol{c}$, i.e., $\langle\boldsymbol{t},\boldsymbol{c}\rangle=0$ and
$\llbracket{df}/{d{q}}\rrbracket=-\gamma_{0}$. Accordingly,
${d\gamma_{0}}/{d{s}}=0$, i.e., $\gamma_{0}$ is necessarily constant. We also
assume a vanishing plastic stretching strain (i.e.,
$\boldsymbol{e}^{p}=\boldsymbol{0}$). The assumed cylindrical form of the
deformation then satisfies the bulk compatibility conditions (59). We can use
the coordinates $(q,s)$ to describe $\Omega$ (even outside $S$). The
equilibrium equations (60) reduce to

$\displaystyle
D\frac{d^{4}f}{d{q}^{4}}-\frac{\partial^{2}\phi}{\partial{s}^{2}}\frac{d^{2}f}{d{q}^{2}}=0~{}\text{in}~{}\Omega-S,$
(61a) $\displaystyle
D\left\llbracket\frac{d^{3}f}{d{q}^{3}}\right\rrbracket+\gamma_{0}\frac{\partial^{2}\phi}{\partial
s^{2}}=0~{}\text{on}~{}S,~{}\text{and}$ (61b)
$\displaystyle\left\llbracket\frac{d^{2}f}{d{q}^{2}}\right\rrbracket=0~{}\text{on}~{}S.$
(61c)

The deformation $f(q)=\gamma_{0}q$, if $q\leq 0$, and $f(q)=0$, otherwise,
satisfies all the above equations with $\phi=0$. Such a deformation,
representing two flat regions connected by a plastic fold with no stress and
moment in $\Omega$, is a solution to our problem in the absence of any
external forcing in the bulk or at the boundary. We now obtain the solutions
when moment and transverse force are prescribed at one edge of the domain. We
consider a rectangular domain $\Omega$ to be defined by $a_{0}\leq q\leq
a_{1}$ such that $a_{0}\leq 0\leq a_{1}$. All the boundaries of $\Omega$,
except at $x_{c}=a_{1}$, are assumed to be free of forces and moments. At
$x_{c}=a_{1}$, we take $\phi=0$, ${\partial\phi}/{\partial q}=0$,
$D{d^{2}w}/{d{q}^{2}}=b_{0}$, and $D{d^{3}w}/{d{q}^{3}}=b_{1}$. The first two
conditions impose that there are no in plane forces applied at the edge. The
constants $b_{0}$ and $b_{1}$ represent the applied moment and the applied
transverse force at the edge. The deformation $f(q)=0$, if $q\leq 0$, and
$f(q)=-\gamma_{0}q+k_{1}{q}^{2}+k_{2}{q}^{3}$, otherwise, satisfies all the
governing equations and the boundary conditions with $\phi=0$,
$k_{1}=b_{0}/2D-b_{1}a_{1}/2D$, and $k_{2}=b_{1}/6D$.

As the second problem, we consider a circular interface $S$ given by
$|\boldsymbol{x}|=r_{0}$, with normal $\boldsymbol{\nu}=\boldsymbol{e}_{r}$
and curvature $k={1}/{r_{0}}$, see Figure 2(b). The fold is taken as
$\boldsymbol{\gamma}=\gamma_{0}\boldsymbol{e}_{r}\otimes\boldsymbol{e}_{r}$,
where $\gamma_{0}$ is constant. We consider a transverse displacement,
compatible with the fold, as $w=\gamma_{0}r_{0}$, if $r\leq r_{0}$, and
$w=\gamma_{0}r$, if $r>r_{0}$, i.e., the fold deforms the sheet into a planar
disc-like region (of radius $r_{0}$) and a cone for $r>r_{0}$, see Figure
2(b). We assume a plastic stretching strain of the form
$\boldsymbol{e}^{p}=\boldsymbol{0}$, if $r\leq r_{0}$, and
$\boldsymbol{e}^{p}={\gamma_{0}}^{2}\boldsymbol{e}_{r}\otimes\boldsymbol{e}_{r}$,
if $r>r_{0}$. The assumed form of $w$ and $\boldsymbol{e}^{p}$ satisfy the
conditions (59) identically. Note that we need a non trivial plastic
stretching strain to satisfy the interfacial compatibility condition (59b) for
the given deformation under the assumption of elastic inextensibility. On the
other hand, the equilibrium conditions (60a) and (60b) are satisfied for
$\phi=-\gamma_{0}\ln r$. The interfacial equilibrium condition (60c) however
can not be satisfied in the present form. In order to satisfy (60c) for the
given $w$ we would require a distribution of transverse force couple field
over $S$ in the form $f_{2}=-D\gamma_{0}/r_{0}$. Therefore a sheet of paper
can be folded about a circular curve, with a flat disc like region in the
interior and a perfect cone shape on the other side, if we superpose a
distribution of transverse force couples along the fold.

###  4.3 Fold terminating at an internal point

We consider a linear fold $S$ terminating in the interior of the domain
$\Omega$ at $O$, see Figure 3(a). In terms of the polar coordinates $r\geq 0$,
$-\pi<\theta\leq\pi$, with origin at $O$, the fold lies on the $\theta=\pi$
line. The constant tangent $\boldsymbol{t}$ to $S$ is along
$\boldsymbol{e}_{1}$ while the constant normal $\boldsymbol{\nu}$ to $S$ is
along $\boldsymbol{e}_{2}$. We assume the stretching plastic strain to vanish
identically over $\Omega$ and the bending plastic strain to concentrate on
$S$, i.e., $\boldsymbol{E}^{p}=\boldsymbol{0}$ and
$\boldsymbol{\Lambda}^{p}\in\mathcal{C}(\Omega,\operatorname{Sym})$, such that
$\boldsymbol{\Lambda}^{p}(\boldsymbol{\psi})=\gamma_{0}\int_{S}\langle\boldsymbol{e}_{2}\otimes\boldsymbol{e}_{2},\boldsymbol{\psi}\rangle\operatorname{dl}$
for all $\boldsymbol{\psi}\in\mathcal{D}(\Omega,\operatorname{Lin})$, where
$\gamma_{0}$ is constant. Such a plastic strain distribution yields a point
supported incompatibility,
$\operatorname{Curl}\boldsymbol{\Lambda}^{p}=-\gamma_{0}\boldsymbol{e}_{2}\delta_{O}$.
In other words, there is no interfacial incompatibility and the
incompatibility concentrates at the end point of the fold in the interior of
the domain. In comparison, the conical singularity (in Section 4.1) allows for
an incompatibility through a non-trivial Gaussian curvature at the defect
point while the circular fold (in Section 4.2) has an incompatibility
concentrating on the fold.

(a) The plate domain.

(b) The transverse displacement.

Figure 3: The plate domain showing a straight fold $S$ terminating at $O$ and
the corresponding transverse displacement.

For the considered form of $\boldsymbol{\Lambda}^{p}$, and recalling that
there is no concentration in elastic bending strain, we obtain
$\llbracket\nabla w\rrbracket=-\gamma_{0}\boldsymbol{\nu}$. Consequently, we
use elastic inextensibility (i.e., $\boldsymbol{E}^{e}=\boldsymbol{0}$ in
$\Omega$) and $k=0$ to simplify the first von Kármán equation (42) as

$\displaystyle[w,w]=0~{}\text{in }~{}\Omega-\\{S\cup\\{O\\}\\},$ (62a)
$\displaystyle\langle\\{\nabla\nabla w\\},\boldsymbol{e}_{1}\otimes\
\boldsymbol{e}_{1}\rangle\gamma_{0}=0~{}\text{on }~{}S-\\{O\\},~{}\text{and}$
(62b) $\displaystyle\nabla W\otimes\nabla W(\mathbb{A}\nabla\nabla
v^{\alpha})=0~{}\text{at}~{}O,$ (62c)

for all multi indices $\alpha\in\mathbb{N}^{2}$ such that
$|\alpha|\leq\operatorname{deg}(\nabla W\otimes\nabla W)+2$. On the other
hand, the second von Kármán equation (44), with a vanishing body force field,
simplifies to

$\displaystyle
D\Delta^{2}w-[\phi,w]=0~{}\text{in}~{}\Omega-\\{S\cup\\{O\\}\\},$ (63a)
$\displaystyle\begin{aligned}
D\left\langle\llbracket\operatorname{div}\nabla\nabla
w\rrbracket,\boldsymbol{e}_{2}\right\rangle+\gamma_{0}\left\langle\nabla\nabla\phi,\boldsymbol{e}_{1}\otimes\boldsymbol{e}_{1}\right\rangle=0~{}\text{on}~{}S-\\{O\\},\end{aligned}$
(63b) $\displaystyle\llbracket\Delta
w\rrbracket=0~{}\text{on}~{}S-\\{O\\},~{}\text{and}$ (63c) $\displaystyle
DW(\Delta^{2}v^{\alpha})+\nabla\Phi\otimes\nabla W(\mathbb{A}\nabla\nabla
v^{\alpha})=-D\left((1-\nu)\boldsymbol{\Lambda}^{p}(\nabla\nabla
v^{\alpha})+\nu(\operatorname{tr}(\boldsymbol{\Lambda}^{p})(\Delta
v^{\alpha}))\right)~{}\text{at}~{}O,$ (63d)

for all multi indices $\alpha\in\mathbb{N}^{2}$ such that $|\alpha|\leq q$
where $q$ is the maximum of $\operatorname{deg}(\nabla\Phi\otimes\nabla W)+2$,
$\operatorname{deg}(W)+4$, and
$\operatorname{deg}(\boldsymbol{\Lambda}^{p})+2$.

We consider a transverse deformation of the form $w=rg(\theta)$, where $g$ is
continuous but piecewise smooth across $S$. The condition $\llbracket\nabla
w\rrbracket=-\gamma_{0}\boldsymbol{\nu}$ then reduces to

$\frac{dg}{d\theta}(\pi)-\frac{dg}{d\theta}(-\pi)=\gamma_{0}.$ (64)

The considered $w$ satisfies (62a) identically. On substituting it in (62c) we
obtain the condition

$\int_{-\pi}^{\pi}g\left(g+\frac{d^{2}g}{d\theta^{2}}\right)d\theta+g(\pi)\left(\frac{dg}{d\theta}(\pi)-\frac{dg}{d\theta}(-\pi)\right)=0.$
(65)

On using the stress function $\phi=D\lambda\ln r$, (63a) reduce to

$\frac{d^{4}g}{d\theta^{4}}+(\lambda+2)\frac{d^{2}g}{d\theta^{2}}+(\lambda+1)g=0.$
(66)

This equation has a solution $g(\theta)=a\cos\mu\theta$, where
$\mu^{2}=\lambda+1$. It should be noted that here, unlike the solution in
Section 4.1, $\mu$ is not necessarily an integer. The constant coefficient $a$
is obtained by substituting this solution in (64) as
$a={\gamma_{0}}/({2\mu\sin(\pi\mu)})$. To calculate $\mu$, we substitute the
solution for $g$ in (65) to obtain

$(1+\mu^{2})\sin(2\pi\mu)+2\pi(1-\mu^{2})\mu=0.$ (67)

The nonlinear equation can be solved (numerically) to obtain $\mu=0.92$ (the
other roots are imaginary). Altogether, we have a transverse displacement of
the form [9]

$w=\frac{\gamma_{0}r}{2\mu\sin(\pi\mu)}\cos\mu\theta,$ (68)

where $\mu=0.92$, see Figure 3(b). The considered solutions for $w$ and $\phi$
identically satisfy (62b), (63b) and (63c). However Equation (63d), which
enforces equilibrium of transverse forces at $O$, is not satisfied. Whereas
the left hand side terms reduce to a Dirac, the right hand side terms reduce
to a gradient of the Dirac. The latter is proportional to $\nu$ while the
former is not. This inconsistency of the considered solution in its failure to
satisfy the equilibrium equation at $O$ indicates that the assumption of
elastic inextensibility needs to be perhaps relaxed in the neighborhood of the
singular point.

###  4.4 Folding into a tetrahedron

(a) The plate domain.

(b) The transverse displacement.

Figure 4: The plate domain showing several straight folds $S$ terminating at
$O$ on one end and at $\partial\Omega$ on the other. The transverse
displacement for a given prescription of folds in terms of plastic bending
strain field.

We consider a finite number ($n$) of linear folds each terminating at a point
$O$ in the interior of the domain, see Figure 4(a). Let
$S=\cup_{i=1}^{n}S_{i}$, where $S_{i}$ represents the $i$-th fold. Each
$S_{i}$ is a straight line with one end point at $\partial\Omega$ and the
other end point at $O$. The multitude of folds terminating in the interior of
the domain are modelled through a plastic bending strain concentrated on $S$
such that
$\boldsymbol{\Lambda}^{p}(\boldsymbol{\psi})=\sum_{i=0}^{n-1}\gamma_{i}\int_{S_{i}}\langle\boldsymbol{\nu}_{i}\otimes\boldsymbol{\nu}_{i},\boldsymbol{\psi}\rangle\operatorname{dl}$
where $\gamma_{i}$ and $\boldsymbol{\nu}_{i}$ represent the (constant)
strength and the (constant) normal associated with the $i$-th fold. We assume
that the incompatibility $N_{1}$ is identically zero and the plastic bending
strain is compatible, i.e.,
$\operatorname{Curl}\boldsymbol{\Lambda}^{p}=\boldsymbol{0}$, which is
equivalent to $\sum_{i=0}^{n-1}\gamma_{i}\boldsymbol{\nu}_{i}=\boldsymbol{0}$.
Due to a compatible plastic strain overall, we look for a solution such that
there are no elastic strain fields necessary to accommodate the prescribed
plastic strains. There are then no stretching strains and the total bending
strain is identical to the plastic bending strain. The transverse displacement
field $W$ satisfies $\boldsymbol{\Lambda}^{p}=\nabla\nabla W$. All the
governing equations, including the compatibility conditions and equilibrium
conditions, are trivially satisfied. In the following lemma, we calculate the
Gaussian curvature that appears due to such a bending strain distribution. It
is point supported at $O$ with only a Dirac, similar to the case of isolated
disclinations. The source of Gaussian curvature presently, however, is in the
plastic folds imposed on the domain.

######  Lemma 4.3.

Given $\boldsymbol{\Lambda}^{p}\in\mathcal{C}(\Omega,\operatorname{Sym})$ such
that
$\boldsymbol{\Lambda}^{p}(\boldsymbol{\psi})=\sum_{i=0}^{n-1}\gamma_{i}\int_{S_{i}}\langle\boldsymbol{\nu}_{i}\otimes\boldsymbol{\nu}_{i},\boldsymbol{\psi}\rangle\operatorname{dl}$,
where $S_{i}$ represents a straight line satisfying $\partial
S_{i}-\partial\Omega=\\{O\\}$ and $\boldsymbol{\nu}_{i}$ is the normal to
$S_{i},$ and $\operatorname{Curl}\boldsymbol{\Lambda}^{p}=\boldsymbol{0}$, we
obtain

$\displaystyle\sum_{i=0}^{n-1}\gamma_{i}\boldsymbol{\nu}_{i}=\boldsymbol{0}~{}\text{and}$
(69a)
$\displaystyle\operatorname{Det}(\boldsymbol{\Lambda}^{p})=-\frac{1}{2}\sum_{j=1}^{n-1}\sum_{i=0}^{j-1}\gamma_{i}\gamma_{j}\langle\boldsymbol{\nu}_{i},\boldsymbol{t}_{j}\rangle\delta_{O},$
(69b)

where $\boldsymbol{t}_{j}$ is the tangent to $S_{j}.$

###### Proof.

The relation (69a) follows immediately from [16, Id. 2.2]. Let $\Omega_{i}$ be
the angular sector bounded by $S_{i}$ and $S_{i+1}.$ Let
$W\in\mathcal{B}(\Omega)$ such that $\nabla
W\in\mathcal{B}(\Omega,\mathbb{R}^{2})$ with bulk density $\nabla
w=-\sum_{j=0}^{i}\gamma_{j}\boldsymbol{\nu}_{j}$ in $\Omega_{i}.$ We use [16,
Id. 2.2] to obtain

$\operatorname{Curl}\operatorname{Curl}(\nabla W\otimes\nabla
W)=\sum_{j=1}^{n-1}\sum_{i=0}^{j-1}\gamma_{i}\gamma_{j}\langle\boldsymbol{\nu}_{i},\boldsymbol{t}_{j}\rangle\delta_{O},$
(70)

which establishes the desired result. ∎

The transverse displacement $w$ is illustrated in Figure 4(b).

##  5 Conclusion

In this work we have extended the classical von Kármán plate equations, given
in terms of smooth fields, to a distributional form while allowing the fields
to be piecewise smooth across curves, to concentrate on the curves, and to be
singular at points in the domain. In writing the distributional form, the
central challenge was to unambiguously define the notions of distributional
determinant, distributional inner product, and distributional Monge-Ampère
bracket. The sources of inhomogeneities were taken in the form of plastic
strains, incompatibility fields (which in turn are related to the defect
densities), and body force fields, all allowed to be singular over points and
curves in the plate domain. The distributional forms of the von Kármán plate
equations were localized to obtain pointwise local forms of the equations in
the bulk, away from the singular points and curves, on the singular curves,
and at the singular points. These local equations are the main contribution of
this paper. The applicability of the developed framework was illustrated
through several examples associated with conical deformations (arising due to
defects and constrained deformations), folds, and folds terminating inside the
plate domain at a singular point. Our work provides a complete and rigorous
framework where a large variety of singular problems associated with von
Kármán plates can be discussed and systematically analyzed.

## Ackowledgement

AG acknowledges the financial support from SERB (DST) Grant No.
CRG/2018/002873 titled “Micromechanics of Defects in Thin Elastic Structures”.

## References

  * [1] M. Ben Amar and Y. Pomeau.  Crumpled paper.  Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 453:729–755, 1997.
  * [2] R. Brunetti and K. Fredenhagen.  Microlocal analysis and interacting quantum field theories: Renormalization on physical backgrounds.  Communications in Mathematical Physics, 208:623–661, 2000.
  * [3] E. Cerda and L. Mahadevan.  Conical surfaces and crescent singularities in crumpled sheets.  Physical Review Letters, 80:2358, 1998.
  * [4] P. G. Ciarlet, G. Geymonat, and F. Krasucki.  Nonlinear Donati compatibility conditions for the nonlinear Kirchhoff–von Kármán–Love plate theory.  Comptes Rendus Mathematique, 351:405–409, 2013.
  * [5] P. G. Ciarlet and S. Mardare.  Nonlinear Saint–Venant compatibility conditions and the intrinsic approach for nonlinearly elastic plates.  Mathematical Models and Methods in Applied Sciences, 23:2293–2321, 2013.
  * [6] E. Efrati, L. Pocivavsek, R. Meza, K. Y. C. Lee, and T. A. Witten.  Confined disclinations: Exterior versus material constraints in developable thin elastic sheets.  Physical Review E, 91:022404, 2015.
  * [7] L. C. Evans.  Weak convergence methods for nonlinear partial differential equations.  American Mathematical Society, Providence, Rhode Island, 1990.
  * [8] F. G. Friedlander and M. S. Joshi.  Introduction to the Theory of Distributions.  Cambridge University Press, Cambridge, 1998.
  * [9] F. Lechenault and M. Adda-Bedia.  Generic bistability in creased conical surfaces.  Physical Review Letters, 115:235501, 2015.
  * [10] A. E. Lobkovsky and T. A. Witten.  Properties of ridges in elastic membranes.  Physical Review E, 55:1577, 1997.
  * [11] S. Mardare.  On Poincaré and de Rham’s theorems.  Revue Roumaine de Mathématiques Pures et Appliquées, 53:523–541, 2008.
  * [12] M. M. Müller, M. B. Amar, and J. Guven.  Conical defects in growing sheets.  Physical Review Letters, 101:156104, 2008.
  * [13] S. Müller.  Det= det. A remark on the distributional determinant.  Comptes Rendus de l’Académie des Sciences - Series I - Mathematics, 311:13–17, 1990.
  * [14] S. Müller.  On the singular support of the distributional determinant.  Annales de l’Institut Henri Poincaré C, Analyse non linéaire, 10:657–696, 1993.
  * [15] H. Olbermann.  Energy scaling law for a single disclination in a thin elastic sheet.  Archive for Rational Mechanics and Analysis, 224:985–1019, 2017\.
  * [16] A. Pandey and A. Gupta.  Topological defects and metric anomalies as sources of incompatibility for piecewise smooth strain fields.  Journal of Elasticity, 139:237–267, 2020.
  * [17] A. Pandey and A. Gupta.  Point singularities in incompatible elasticity.  Journal of Elasticity, 147:229–256, 2021.
  * [18] A. Pandey, M. Singh, and A. Gupta.  Positive disclination in a thin elastic sheet with boundary.  Physical Review E, 104:065002, 2021.
  * [19] P. Podio-Guidugli and A. Favata.  Elasticity for Geotechnicians.  Springer, Switzerland, 2014.
  * [20] H. S. Seung and D. R. Nelson.  Defects in flexible membranes with crystalline order.  Physical Review A, 38:1005–1018, 1988.
  * [21] M. Singh, A. Pandey, and A. Gupta.  Interaction of a defect with the reference curvature of an elastic surface.  Soft Matter, 18:2979–2991, 2022.
  * [22] M. Singh, A. Roychowdhury, and A. Gupta.  Defects and metric anomalies in Föppl-von Kármán surfaces.  Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 478:20210829, 2022.
  * [23] T. A. Witten.  Stress focusing in elastic sheets.  Reviews of Modern Physics, 79:643–675, 2007.


captionUnsupported document class

# MESAHA-Net: Multi-Encoders based Self-Adaptive Hard Attention Network with
Maximum Intensity Projections for Lung Nodule Segmentation in CT Scan

Muhammad Usman  Azka Rehman  Abdullah Shahid  Siddique Latif  Shi‑Sub Byon
Sung‑Hyun Kim  Tariq Mahmood Khan  and Yeong‑Gil Shin Manuscript received
March 31, 2023; (Corresponding author: Muhammad Usman.)Muhammad Usman and
Yeong-Gil Shin are with the Department of Computer Science and Engineering,
Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, Republic of
Korea, (e-mail<EMAIL_ADDRESS>yshin@snu.ac.kr).Muhammad Usman, Azka
Rehman, Abdullah Shahid, Shi-Sub Byon and Sung‑Hyun Kim is with Center for
Artificial Intelligence in Medicine and Imaging, HealthHub Co. Ltd., Seoul
06524, Republic of Korea, (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>).Siddique Latif is with Queensland University of Technology, Brisbane, 4000,
QLD, Australia, (e-mail: siddique.latif@qut.edu.au).Tariq Mahmood Khan is with
Department of Electrical Engineering, COMSATS University Islamabad, Islamabad,
Pakistan, (e-mail<EMAIL_ADDRESS>
###### Abstract

Accurate lung nodule segmentation is crucial for early-stage lung cancer
diagnosis, as it can substantially enhance patient survival rates. Computed
tomography (CT) images are widely employed for early diagnosis in lung nodule
analysis. However, the heterogeneity of lung nodules, size diversity, and the
complexity of the surrounding environment pose challenges for developing
robust nodule segmentation methods. In this study, we propose an efficient
end-to-end framework, the multi-encoder-based self-adaptive hard attention
network (MESAHA-Net), for precise lung nodule segmentation in CT scans.
MESAHA-Net comprises three encoding paths, an attention block, and a decoder
block, facilitating the integration of three types of inputs: CT slice
patches, forward and backward maximum intensity projection (MIP) images, and
region of interest (ROI) masks encompassing the nodule. By employing a novel
adaptive hard attention mechanism, MESAHA-Net iteratively performs slice-by-
slice 2D segmentation of lung nodules, focusing on the nodule region in each
slice to generate 3D volumetric segmentation of lung nodules. The proposed
framework has been comprehensively evaluated on the LIDC-IDRI dataset, the
largest publicly available dataset for lung nodule segmentation. The results
demonstrate that our approach is highly robust for various lung nodule types,
outperforming previous state-of-the-art techniques in terms of segmentation
accuracy and computational complexity, rendering it suitable for real-time
clinical implementation.

Lung nodule segmentation, Adaptive hard-attention, Multi-encoder-based
architecture, maximum intensity projection, bidirectional MIP.

## Introduction

Lung cancer is the leading cause of death worldwide, accounting for
approximately 1.8 million deaths annually [1]. Early detection and diagnosis
of lung cancer are crucial for improving patient outcomes, including prognosis
and survival rates [2]. Computed tomography (CT) is the preferred imaging
modality for monitoring treatment response and analyzing pulmonary nodules
[3]. While most lung nodules are benign, approximately 20-30% of lung nodules
detected on CT scans can be malignant and indicative of early-stage lung
cancer [4]. The accurate segmentation of lung nodules is crucial in assisting
the early diagnosis of lung cancer. However, the task of segmenting lung
nodules is challenging due to the heterogeneity of nodule types, size
variations, and the complexity of the surrounding lung anatomy [5]. CT scans
are extensively utilized for the diagnosis of lung nodule has emerged as it
has proven the most effective imaging modality for early diagnosis and
analysis [2].

Despite the advances in CT imaging and its role in early lung cancer
detection, manual segmentation of lung nodules remains a time-consuming and
labor-intensive process, with several disadvantages and limitations. Manual
segmentation is subject to intra- and inter-observer variability, leading to
inconsistent and unreliable results [6]. Additionally, the large number of CT
images generated in lung cancer screening programs presents a significant
workload for radiologists, increasing the likelihood of human error and missed
or misinterpreted nodules [7]. Furthermore, the subtle and diverse nature of
lung nodules can also pose challenges for manual segmentation, particularly in
the presence of complex anatomical structures, such as blood vessels and
bronchi [5] as shown in Fig. 1. These limitations of manual segmentation
highlight the need for the development of computer-aided diagnosis (CAD)
systems for the segmentation and analysis of lung nodules.

A multitude of efforts have been dedicated to the development of computer-
aided diagnosis (CAD) systems to aid radiologists in identifying and analyzing
lung nodules from CT scans [8]. These CAD systems aim to enhance the
diagnostic process by automating the identification and segmentation of lung
nodules, thereby reducing the workload of radiologists and minimizing the
potential for human error [9]. Various techniques have been employed in the
development of computer-aided diagnosis (CAD) systems for lung nodule
segmentation, ranging from traditional image processing approaches [10, 11,
12] and machine learning algorithms [13, 14], to deep learning (DL)-based
methods [15, 16]. Among these techniques, DL-based methods have demonstrated
superior performance compared to conventional approaches, exhibiting enhanced
segmentation accuracy and robustness against diverse lung nodule
characteristics, such as type, texture, and size [17].

Deep learning (DL)-based studies for lung nodule segmentation can be broadly
categorized into three groups: voxel-level classification (VLC) [18, 19, 20],
3D segmentation [21, 22, 23], and 2D patch-wise segmentation [24, 25, 26]. The
VLC methodology involves classifying each voxel within the volume of interest
(VOI) as either nodular or non-nodular. However, this approach fails to
exploit global semantic information, leading to an increased number of false
positives stemming from inter-class similarities at nodule boundaries.
Moreover, the substantial computational resources demanded by VLC methods
render them ill-suited for real-time clinical applications.Several studies
have approached lung nodule segmentation as a 3D segmentation problem, wherein
the entire volume of interest (VOI) is input to the network for generating a
3D nodule segmentation [21, 22, 23]. However, these methodologies are plagued
by suboptimal segmentation performance due to rescaling-induced error (RIE),
which arises from significant rescaling during pre- and post-processing stages
Given the variability in nodule sizes, determining the optimal input dimension
for 3D networks proves challenging, consequently leading to a substantial
decline in segmentation outcomes. Furthermore, 3D models necessitate
considerable computational resources for both training and deployment in
clinical environments.

To address the limitations associated with VLC and 3D segmentation, 2D patch-
wise segmentation strategies are proposed in [24, 25, 26]. These techniques
perform 3D segmentation of nodules using 2D regions of interest (ROIs) as
input, encompassing the nodule in each slice. To estimate ROIs in adjacent
slices, two types of strategies have been employed: constant/fixed ROIs [20]
and dynamic ROIs methods [24]. In constant or fixed ROI-based methods, the
same ROI is applied to all surrounding slices, introducing redundant regions
in the ROIs and posing challenges for the network during nodule segmentation.
In contrast, dynamic ROI-based approaches determine ROIs for adjacent slices
dynamically, based on the segmentation of the nodule in the current slice,
effectively eliminating redundant regions and enabling the network to
concentrate on nodular areas.

Although these techniques have significantly improved performance through
multi-view analysis, they remain susceptible to RIE, limiting the robustness
of such approaches. To tackle these challenges, in this study, we introduce a
novel multi-encoder-based self-adaptive hard attention network (MESAHA-Net)
for volumetric segmentation of pulmonary nodules. The proposed framework
utilizes only the 2D ROI as input to yield 3D segmentation of a given lung
nodule. MESAHA-Net comprises three encoding paths, an attention block, and a
single decoding block. The proposed architecture incorporates three inputs:
the cropped slice, as well as forward and backward Maximum Intensity
Projection (MIP) images with a thickness of 3 mm, which are employed to
segment the nodule and estimate the ROI for forward and backward slices,
respectively. Moreover, ROI information is supplied as hard attention to the
attention block, allowing the network to concentrate solely on the ROI within
the given slice patch. This enables the network to tackle the challenges posed
by nodule size diversity. The network also leverages spatial and contextual
information to segment the nodule in the provided slice and estimate ROIs for
adjacent slices, which are subsequently used as input for nodule segmentation
in later iterations.

Figure 1: Multiple visual appearances of the pulmonary nodule are shown. The
variations within a nodule can be seen from column (a) to (e), and the inter-
nodule difference is presented from rows (i) to (iv).

## Related Work

Various efforts have been devoted to the development of highly accurate and
robust automatic nodule segmentation systems to aid radiologists [27]. These
approaches are typically designed to identify the nodular voxels in the given
volume of interest (VOI) containing the nodule. The most commonly employed
traditional techniques in this context include morphological method [12],
region growing method [11], and energy optimization method [10]. However,
several studies have compared traditional algorithms with advanced deep
learning-based techniques for lung nodule segmentation, demonstrating the
superior performance of the latter in terms of accuracy and robustness [28,
18].

Deep learning-based methodologies have addressed the lung nodule segmentation
challenge through various approaches, including voxel-level classification
(VLC), 2D patch-wise segmentation, and 3D segmentation techniques. For
example, Wang et al. [18] introduced Central Focused Convolutional Neural
Networks (CF-CNN) with central pooling, employing both 2D and 3D inputs
surrounding a specific voxel for nodule and non-nodule classification.
Likewise, Cao et al. [19] proposed a central intensity pooling layer (CIP) to
extract intensity features from the center voxel of a given volume of interest
(VOI), subsequently utilizing a CNN to obtain convolutional features for voxel
classification. Similarly, Liu et al. [20] suggested a dual-pathway
architecture, comprising ResBlocks, to integrate 2D and 3D inputs for voxel
classification. However, voxel-level classification techniques often fail to
optimally exploit global semantic information, resulting in a high false
positive rate. Additionally, the limited generalization capabilities of these
models, coupled with the substantial inference time required given VOI, render
them unsuitable for clinical applications.

Several studies have proposed 3D segmentation-based approaches to address
issues related to VLC. In such approaches, the entire VOI is inputted to
generate a 3D segmentation of the given lung nodule. However, this approach
faces challenges due to the varied size of lung nodules, which make it
difficult to select the optimal VOI dimensions. To overcome this problem, Zhou
et al. [21] developed a lung nodule segmentation system that utilized a 3D
UNet architecture. They added the Atrous Spatial Pyramid Pooling (ASPP) module
and convolutional block attention module (CBAM) to fuse high-level features in
the multi-scale and supplement global features, respectively. Similarly, Zhang
et al. [22] trained four 3D UNet networks of different input dimensions to
overcome the scale variant issue of lung nodules. They ensemble the output of
all four networks using multi-scale dense CRF to produce the 3D volumetric
lung nodule mask. Likewise, Tyagi et al. [23] employed a 3D conditional
generative adversarial network, consisting of a 3D UNet and a classification
network, with a spatial squeeze and channel excitation modules, that acted as
a generator and discriminator, respectively. These approaches achieved
significantly better performance, but they incur significant computational
complexities, making them impractical for real-time clinical utilization.

To overcome the issues pertaining to the computational complexities, 2D patch-
wise approaches were followed by a number of studies. For instance, Usman et
al. [24] proposed a novel method that utilized 2D ROI input to generate 3D
lung nodule segmentation. They employed a residual U-Net architecture for
nodule segmentation in multi-view analysis (MVA) and introduced an innovative
adaptive ROI (A-ROI) algorithm to determine the ROI for each successive slice
based on the current slice’s nodule segmentation mask. Wu et al. [25] proposed
a patch-wise segmentation scheme comprising two U-Nets to accommodate both 2D
and 3D patch inputs. The outputs of both networks were combined to generate
nodule segmentation for the given slice. Chen et al. [26] introduced a novel
Mask and Texture-driven Generative Adversarial Network (MTGAN) with a joint
multi-scale L1 loss for lung nodule segmentation. The MTGAN employed a U-Net
as the generator and a five-layer fully convolutional network (FCN) as the
discriminator. Similarly, Usman et al. [29] proposed a dual-encoder-based
architecture that eliminated the need for rescaling during pre- and post-
processing stages. They introduced hard attention to counteract issues caused
by rescaling and utilized the A-ROI algorithm for examining the nodule in its
surroundings. These techniques achieved significant improvements and
robustness in lung nodule segmentation performance. However, due to the
utilization of complex network architectures, the implementation of MVA, and
the use of handicraft algorithms, the overall complexity of these solution was
substantially increased.

The primary differences between the proposed MESAHA-Net and previous
approaches are three-fold: 1) we introduced a lightweight multi-encoder-based
architecture capable of leveraging the ROI mask as hard attention to produce
highly accurate lung nodule segmentation efficiently; 2) we incorporated
forward and backward MIP images to effectively estimate the ROI for adjacent
slices without employing any handcrafted algorithm; 3) we eliminated the need
for rescaling at pre- or post-processing stages by utilizing a novel adaptive
hard attention mechanism which eliminates the rescaling-induced error.

## Proposed Method

The proposed multi-encoder based self-adaptive hard attention network (MESAHA-
Net) is

Figure 2: The overall architecture of proposed multi-encoder self-adaptive
hard attention network (MESAHA-Net) for Lung Nodule segmentation is
demonstrated.

Figure 3: Illustration of blocks utilized in MESAHA-Net: (a) Convolutional
Block (Conv-Block), (b) Deconvolutional Block (Deconv-Block), (c) Attention
Generator Block (AGen Block), and (d) Attention Unit (AU).

equipped with three encoding path blocks, a hard-attention block, and a
decoder block. The three encoding paths allow the network to integrate slice
patch, forward, and backward MIP images, providing spatial and 3D contextual
information to enhance nodule segmentation performance and ROI estimation for
adjacent slices. The attention block facilitates the incorporation of the ROI
mask as hard attention, directing the network’s focus toward the nodular
region. Lastly, the decoder block processes the refined information to
generate the nodule segmentation mask and ROI masks for adjacent slices.
Detailed descriptions of each block are provided in the following sections.

###  0.1 Encoder block

The encoder block comprises three encoding branches, with each accepting a
single 2D image input. Specifically, the raw slice patch, and 3 mm thick
forward and backward MIP images are fed into each encoder. The raw slice patch
provides low-level information and 2D context, crucial for accurately
segmenting the nodule boundary within the given ROI. Forward and backward MIP
images compensate for the 3D spatial aspect by offering insights into the
nodule from both sides of the given slice.

All three encoders share an identical architecture. Each encoder is composed
of four Convolutional Blocks (Conv-blocks), which consist of two sets of 2D
convolutions as shown in Fig. 3 (a). Each set includes a convolution layer
followed by ReLU activation and batch normalization layers, with max pooling
applied afterward. Conv-blocks downsample the input while extracting coarse
features related to the nodule. These features are fed to the attention block
using skip connections, and the attention block forwards the refined features
to the decoder block for further processing.

###  0.2 Self-Adaptive Hard Attention Mechanism

The proposed MESAHA-Net employs a hard attention mechanism, utilizing a ROI
mask for initial segmentation. This network generates analogous ROI masks for
adjacent slices, which are subsequently fed back into the system for iterative
nodule segmentation in successive slices. A detailed description of the self-
hard attention mechanism’s algorithmic steps can be found in Algorithm 1.

Algorithm 1 : Self-Adoptive Hard Attention Mechanism

1:$n=n_{i}$, $R_{n}=R_{n_{i}}$

2:while $\sum{R_{n}}>0$ do

3: $\mathcal{I}_{\text{n}}=iCrop(I_{\text{n}},R_{n})$

4:
$\mathcal{M}^{\text{-}}=max(\mathcal{I}_{\text{n-2}},\mathcal{I}_{\text{n-1}},\mathcal{I}_{\text{n}})$

5:
$\mathcal{M}^{\text{+}}=max(\mathcal{I}_{\text{n}},\mathcal{I}_{\text{n+1}},\mathcal{I}_{\text{n+2}})$

6:
$R_{n-1},\mathcal{S}_{n},R_{n+1}=\mathcal{\psi}(\mathcal{I}_{n},\mathcal{M}^{\text{-}},\mathcal{M}^{\text{+}},R_{n})$

7: $n\leftarrow n\pm 1$

8: $R_{n}\leftarrow R_{n\pm 1}$

9:end while

In Algorithm 1, $n_{i}$ and $R_{n_{i}}$ represent the inputted slice number
and its corresponding ROI mask, respectively. The proposed algorithm initially
extracts a $96\times 96$ patch surrounding the given ROI, denoted by
$\mathcal{I}_{n}$. Subsequently, this cropped patch is employed to generate
forward and backward MIP images, represented as $\mathcal{M}^{\text{-}}$ and
$\mathcal{M}^{\text{+}}$, respectively. These MIP images are produced with a 3
mm slab thickness by utilizing three adjacent slices.

The MESAHA-Net, denoted by $\mathcal{\psi}$, accepts four inputs: the cropped
patch of the slice, forward and backward MIP images, and the ROI mask. It
predicts the lung nodule segmentation mask, $S_{n}$, for the $n^{th}$ slice,
along with ROI masks for both adjacent slices, i.e., $R_{n-1}$ and $R_{n+1}$.
To examine the presence of nodules in neighboring slices, the algorithm
employs the predicted ROI masks as input for the next iteration. Segmentation
is executed in both directions to obtain the volumetric segmentation of the
given nodule.

###  0.3 Attention Block

In the encoder block, convolutional layers systematically extract meaningful
features ($\lambda^{i}$) from high-dimensional inputs, such as raw slice
patch, forward and backward 3 mm MIP images. This sequential process enables
the model to capture a substantial receptive field and the semantic contextual
information vital for segmenting lung nodules in the given slice. However,
features derived from a large receptive field contain an abundance of
redundant information that does not correspond to the actual nodule’s region
of interest. To address this issue, an attention block is incorporated into
MESAHA-Net to refine the coarse features extracted from each encoder branch
after every conv-block. This attention block receives coarse features and
filters them to suppress redundant features by assigning lower weights. These
refined features are then forwarded to the decoder block for further
processing.

The architecture of attention block has been demonstrated in Figure 2. The
block consists of hard attention generator (HA-Gen) units and attention gating
(AG) units. The role of HA-Gen units is to exploit the inputted ROI mask to
generate the gating vector $g_{j}$ for AG units. The AG unit consists of 2D
convolution layer followed by batch normalization, relu and max-pooling layers
as shown in Fig. 3 (c). Whereas the overall architecture of AG unit is
inspired from [30], however, in contrast to [30], we employed multiplicative
attention [31] owing to its computational efficiency over additive attention
[32]. The output of AG ($\hat{\lambda}^{i}_{j,c}$) units is the element-wise
multiplication of input features-maps ($\lambda^{i}_{j}$) extracted from $ith$
convolution block and hard-attention coefficients $\mathcal{H}^{i}_{j}$ and
can be defined as follows:

$\hat{\lambda}^{i}_{j,c}=\lambda^{i}_{j,c}.\mathcal{H}^{i}_{j}$ (1)

Here hard attention coefficients, $\mathcal{H}^{i}_{j}\in[0,1]$, recognize the
salient image region which is essentially in our case is the ROI containing
the lung nodule. Where $j$ and $c$ represent the spatial and channel
dimensions, respectively. $\mathcal{H}^{i}_{j}$ can be defined as follows:

$\mathcal{H}^{i}_{j}=\sigma(\rho^{i}_{HA}(\lambda^{i}_{j,c},g_{j};\theta_{att}))$
(2)

$\rho^{i}_{HA}=\eta^{\tau}(\xi(W^{\tau}_{\lambda}\lambda^{i}_{j}.W_{g}^{\tau}g_{j}+\beta_{g}))+\beta_{\eta}$
(3)

$\sigma(\lambda^{i}_{j,c})=\frac{1}{1+e^{-\lambda^{i}_{j,c}}}$ (4)

In other words, attention unit is characterised by a set of parameters
$\theta_{att}$ containing; linear transformations
$W_{\lambda}\in\mathbb{R}^{F_{l}\times F_{\text{int
}}},W_{g}\in\mathbb{R}^{F_{g}\times F_{\text{int
}}},\eta\in\mathbb{R}^{F_{\text{int }}\times 1}$ and bias terms
$\beta_{g}\in\mathbb{R},\beta_{\eta}\in\mathbb{R}^{F_{\text{int }}}$. The
linear transformations are computed using channel-wise $1\times 1\times 1$
convolutions for the input tensors. This can be regarded as the vector
concatenation-based attention, where the concatenated features $\lambda^{i}$
and $g$ are linearly mapped to a $\mathbb{R}^{F_{\text{int }}}$ dimensional
intermediate space.

###  0.4 Decoder block

The decoder block is designed to process refined features from four levels of
attention blocks, as illustrated in Fig. 2. This block primarily upsamples
inputs and sequentially structures feature maps for nodule segmentation and
ROI masks. Its architecture comprises Deconv blocks, as depicted in Fig.
3(b).Each Deconv block first upsamples the input, followed by two sets of 2D
convolutions with ReLU activation. Afterward, a batch normalization layer is
applied, followed by a second CNN layer with ReLU activation. The feature map
at this stage is combined with the input from the ConvBlock, and an upsampling
layer is applied subsequently. Every ConvBlock from the first three encoders
is concatenated and passed to the attention units, with a feature map from the
attention-generating units serving as the gating signal. These attention
units’ outputs are concatenated with the DeConv block outputs of the decoder.

###  0.5 Loss Function

The total loss we used to train the model is combination of two losses, i.e.,
attention and segmentation loss.

$\mathcal{L}_{Total}=\alpha*\mathcal{L}_{\text{Seg}}+(1-\alpha)\mathcal{L}_{Att},$
(5)

Here $\alpha$ is dynamic hyper parameter which is used to control the
contribution of each loss and can defined as follows:

$\displaystyle\alpha$ $\displaystyle=\left\\{\begin{array}[]{l}1-\tau,\sum
R_{n\pm 1}=0\\\ \tau,\text{ otherwise }\end{array}\right.$ (6)

Since the proposed MESAHA-Net is 2D segmentation model and performs slice-by-
slice nodule segmentation in axial view, it is crucial for network to
accurately determine the ends of nodule along z-axis. Therefore, we use lower
values of $\alpha$ for end slices from both slides and higher for the
intermediate slices. End slices are defined as those slices for which any of
hard attention, ROI mask, becomes completely blank as there is no further
nodule present in the next coming slice. The purpose is to enforce the network
to learn the absence of nodule in the upcoming slice to stop the further
investigation in corresponding direction. We set $\tau$ to 0.3 for training
the proposed MESAHA-Net.

Attention loss is computed by calculating the dice loss between predicted ROI
masks ($R_{n\pm 1}^{p}$) by MESAHA-Net and ideal ROI masks ($R_{n\pm 1}^{g}$)
for $n\pm 1th$ slice. Ideal ROI mask is generated by using the reference
nodule masks while keeping the margin 20% from all fourth sides. Attention
loss can defined as:

$\mathcal{L}_{\text{Att}}=\left[1-\frac{R_{n+1}^{p}\cap
R_{n+1}^{g}}{R_{n+1}^{p}+R_{n+1}^{g}}+\frac{R_{n-1}^{p}\cap
R_{n-1}^{g}}{R_{n-1}^{p}+RoI_{n-1}}\right]$ (7)

The goal of nodule segmentation is to precisely detect the boundary/edges of
nodule and to achieve this goal we employed the boundary loss [33] which is
computed by integral framework that approximates the distance by computing the
integrals over the interface between mismatch regions of the two boundaries.
To calculate the distance Dist$(\partial G,\partial S)$ between two boundaries
of ground truth mask ($\partial G$) and predicted mask ($\partial S$), they
used the integral framework to formulate the boundary loss by formulating as
follows :

$\displaystyle\operatorname{Dist}(\partial G,\partial S)$
$\displaystyle=\int_{\partial G}\left\|m_{\partial S}(p)-p\right\|^{2}dp$
$\displaystyle=2\int_{\Delta S}D_{G}(p)dp$
$\displaystyle\left.=2\left(\int_{\Omega}\phi_{G}(p)s(p)dp-\int_{\Omega}\phi_{G}(p)g(p)dp\right)\right)$
(8)

Let $p$ be a point on the boundary $\partial G$ and $m_{\partial S}(p)$ be the
corresponding point on the boundary $\partial S$. The distance map of point
$q$ concerning the boundary $\partial G$ is represented by $D_{G}(p)$. In this
case, $\Omega$ denotes the spatial domain, and binary indicator functions are
used. The binary indicator function of region $S$, denoted by
$s:\Omega\rightarrow{{0,1}}$, is defined as $s(q)=$ 1 if $q\in S$ belongs to
the target and 0 otherwise. The level set representation of boundary $\partial
G$ is represented by $\phi_{G}:\Omega\rightarrow\mathbb{R}$, such that
$\phi_{G}(q)=-D_{G}(q)$ if $q\in G$ and $\phi_{G}(q)=D_{G}(q)$ otherwise. By
considering $S=S_{\theta}$ and replacing the binary variables $s(q)$ in Eq.
(4) with the softmax probability outputs of the network $s_{\theta}(q)$, we
derive the following boundary loss. This loss, up to a constant independent of
$\theta$, approximates the boundary distance
$\operatorname{Dist}\left(\partial G,\partial S_{\theta}\right)$:

$\mathcal{L}_{Seg}(\theta)=\int_{\Omega}\phi_{G}(q)s_{\theta}(q)dq$

Figure 4: The preprocessing stage is illustrated as follows: (a) Patch
position selection on the slices using the provided ROI, with the selected
patch and ROI highlighted in green and red, respectively. (b) Cropped patches.
(c) Forward and backward MIP images along with the ROI mask.

##  1 Experimental Setup

###  1.1 Dataset and Preprocessing

We evaluated the performance of our proposed MESAHA-Net using the Lung Image
Database Consortium and Image Database Resource Initiative (LIDC/IDRI) dataset
[5, 8]. This dataset contains 1,018 thoracic CT scans with marked-up annotated
lesions from 1,010 patients, sourced from seven universities and eight medical
imaging companies globally. It encompasses 2,610 lung nodules with diameters
ranging from 2.03 mm to 38.12 mm, slice thicknesses varying between 0.45 mm
and 5.0 mm, and axial plane resolutions (pixel spacing) from 0.46 mm $\times$
0.46 mm to 0.98 mm $\times$ 0.98 mm. Four specialized radiologists provided
the nodule annotations.

For our study, we selected nodules annotated by all four experts, totaling 893
nodules. We employed a 50% consensus criterion to establish ground-truth
boundaries, addressing the inter-observer variability among the radiologists
[34]. These expert radiologists evaluated all 893 nodules and provided
annotations for nine characteristics: Subtlety, Internal Structure,
Calcification, Sphericity, Margin, Lobulation, Speculation, Texture, and
Malignancy [5]. Following previous studies [18, 24], we partitioned the 893
nodules into training, validation, and testing sets, containing 356 (40%), 45
(5%), and 492 (55%) nodules, respectively. Table 1 demonstrates that the
distribution of clinical characteristics among the three subsets is
statistically comparable. We trained the proposed MESAHA-Net exclusively on
the training set by utilizing the validation set to determine the number of
training epochs. Lastly, we employed the test set to assess the performance of
our framework.

!ht

Table 1: Characteristic distributions of the LIDC training, validation and
testing sets. Values are shown in mean ± standard deviation.

Characteristics |  | Training set
---
(n = 356)
| Validation set
---
(n = 45)
| Testing set
---
(n = 492)
Diameter (mm) | 9.54 ± 4.92 | 8.95 ± 5.11 | 9.36 ± 4.84
Subtlety | 3.96 ± 1.10 | 3.881 ± 1.074 | 3.989 ± 1.096
InternalStructure | 1.05 ± 0.20 | 1.00 ± 0.07 | 1.020 ± 0.237
Calcification | 5.65 ± 0.96 | 5.60 ± 1.032 | 5.673 ± 0.934
Sphericity | 3.76 ± 0.99 | 3.75 ± 0.988 | 3.770 ± 0.989
Margin | 4.00 ± 1.12 | 3.95 ± 1.192 | 4.014 ± 1.091
Lobulation | 1.75 ± 1.07 | 1.66 ± 1.008 | 1.787 ± 1.092
Spiculation | 1.64 ± 1.05 | 1.56 ± 0.943 | 1.671 ± 1.089
Texture | 4.58 ± 0.97 | 4.50 ± 1.063 | 4.614 ± 0.936
Malignancy | 2.99 ± 1.25 | 2.99 ± 1.246 | 2.991 ± 1.246

Note: All attributes, except diameter and calcification, are measured on an
ordinal scale from 1 to 5, whereas calcification values span from 2 to 6.
Spiculation and lobulation indicate the quantity of these formations present
within a single nodule. The probabilities of sphericity, calcification, and
malignancy in a single nodule are also represented. There is no significant
statistical disparity among the characteristics in the three groups.

The overall workflow of preprocessing stage has been demonstrated in Fig. 4.
The proposed system initiates the automatic segmentation of lung nodules by
taking a 2D ROI as input. Our framework first employs the 2D ROI to identify
the area to be cropped, with dimensions of $96\times 96$, without any
resizing. This dimension of patch is selected after statistical analysis of
nodule area/appearance on axial axis. Later, the cropped slice patch is
transformed into a 2D ROI mask. Lastly, we generate Maximum Intensity
Projection (MIP) images of 3 mm in both directions (i.e., forward and
backwards). Radiologists commonly use MIP images to identify pulmonary nodules
within the lung region. MIP images are generated by overlaying the maximum
voxel values from a series of consecutive slices at each coordinate. This
process can be mathematically formulated as:

$I_{x,y}=\max{\Delta z}D{x,y,\Delta z}$ (9)

Here, $I_{x,y}$ and $D_{x,y,\Delta z}$ represent the generated MIP image and
the sub-volume or slab of the CT scan, respectively. $\Delta z$ denotes the
slab thickness, which in our case is set to 3 mm. To generate the forward and
backward MIP images, the range of $\Delta z$ is defined differently and can be
formulated as:

$\displaystyle\Delta z$
$\displaystyle=\left\\{\begin{array}[]{l}\left[\phi_{c},\phi_{c}+\phi_{m}\right],\text{
For forward MIP image}\\\ \left[\phi_{c}-\phi_{m},\phi_{c}\right],\text{ For
backward MIP image}\end{array}\right.$ (10)

In this equation, $\phi_{c}$ and $\phi_{m}$ represent the depth of the current
slice in the scan and the thickness of the MIP to be generated, i.e., 3 mm,
respectively. Incorporating MIP images serves to include 3D aspects in both
directions, aiding the proposed network in determining bounding boxes for
surrounding slices in both directions.

Figure 5: This figure illustrates the successive iterations necessary for
complete 3D segmentation of the nodule. The blue region represents the initial
provided ROI, while the red and green regions correspond to the outcomes of
first and second iterations, respectively.

###  1.2 Training and Inference Strategy

The proposed model estimates the nodule segmentation mask for the current
slice, in addition to the ROI masks for both adjacent slices. The inference
process is depicted in the Fig. 5. For generating the ROI masks during
training, we leveraged the lung nodule segmentation masks, maintaining the ROI
area 30% larger than the nodule area, which has been proven optimal [24]. From
the 356 training nodules, we treated each slice containing a nodule as a
training sample, subsequently generating 3,000 samples with nodules of
dimensions $96\times 96$. To enhance the model’s ability to recognize the
absence of a nodule, we incorporated 500 random patches of non-nodular regions
with the same dimensions. We adopted a similar strategy for the validation
set, producing 800 samples with nodules and 100 samples without nodules. For
testing, we provided a single 2D ROI for each nodule to predict the 3D mask,
comparing each prediction with the 3D ground truth mask to calculate the
overall performance.

We implemented the proposed network using the TensorFlow 2.0 platform. For
this experiment, we set the batch size to 15 and optimized the proposed
architecture using the Adam optimizer, with an initial learning rate of
$10^{-3}$. The network was trained for 200 epochs; despite the large dataset
and complex problem, our model parameters were small, resulting in fast
convergence of our training curve. We employed binary cross-entropy as the
loss function.

###  1.3 Evaluation Parameters

We utilize six widely-recognized metrics in volume segmentation for
evaluation: dice similarity coefficient (DSC, % ), positive predictive value
(PPV, % ), sensitivity (SEN, % ), average symmetric surface distance (ASD,
$\mathrm{mm}$ ), residual mean square distance (rms, mm), and Hausdorff
distance (HFD, mm). The complete definitions are provided in the equations
below:

$\small\mathrm{DSC}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$
(11)

and

$\small\mathrm{PPV}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\quad\mathrm{SEN}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
(12)

Here, TP represents the number of accurately segmented lung nodule voxels, FP
is the count of falsely segmented lung nodule voxels, and FN stands for the
number of falsely segmented background voxels. Additionally,

$\displaystyle\mathrm{ASD}$ $\displaystyle=\frac{\left(\sum_{z\in
SEG}\mathrm{~{}d}(\mathrm{z},\mathrm{GT})+\sum_{z\in\mathrm{GT}}\mathrm{d}(\mathrm{z},\mathrm{SEG})\right)}{|\mathrm{SEG}|+|\mathrm{GT}|}$
(13) $\displaystyle\small\mathrm{rms}$
$\displaystyle=\sqrt{\frac{\left(\sum_{z\in\mathrm{SEG}}\mathrm{d}(\mathrm{z},\mathrm{GT})^{2}+\sum_{\mathrm{z}\in\mathrm{GT}}\mathrm{d}(\mathrm{z},\mathrm{SEG})^{2}\right)}{|\mathrm{SEG}|+|\mathrm{GT}|}}$
(14)

and

$\small\mathrm{HFD}=\max[\mathrm{d}(\mathrm{SEG},\mathrm{GT}),\mathrm{d}(\mathrm{GT},\mathrm{SEG})]$
(15)

In these equations, GT and SEG denote the sets of voxels on the ground truth
surface and the automatically segmented lung nodule surface, respectively. The
symbol $\mathrm{d}(\mathrm{x},\mathrm{Y})$ represents the minimum Euclidean
distance from voxel $\mathrm{x}\in\mathrm{X}$ to $\mathrm{Y}$.

Higher DSC, PPV, and SEN values signify improved performance, while lower ASD,
rms, and HFD values indicate better performance. DSC, PPV, and SEN are
commonly employed to gauge the performance of lung nodule segmentation
techniques. Furthermore, to thoroughly assess the model’s segmentation
capability, we also employed ASD, rms, and HFD for detailed analysis.

## Results and Discussion

We conducted several experiments on the LIDC-IDRI dataset to assess the
effectiveness of our method. The outcomes were examined in terms of overall
performance, computational time analysis, robustness, and visual performance.
The subsequent subsections discuss our findings in detail.

Table 2: Average DSC percentages from pairwise comparisons between each of the four radiologists (R1 to R4), ground truth generated with 50% consensus, and MESAHA-Net. | R1 | R2 | R3 | R4 | Average
---|---|---|---|---|---
R1 | – | 77.16 | 77.86 | 77.44 | 77.48 ± 15.08
R2 | 77.16 | – | 79.25 | 76.60 | 77.67 ± 14.18
R3 | 77.86 | 79.25 | – | 77.78 | 78.29 ± 14.60
R4 | 77.44 | 76.60 | 77.78 | – | 77.27 ± 15.62
| Consensus (50%)
---
Ground Truth
86.47 | 84.63 | 86.68 | 87.48 | 86.31 ± 14.03
MESAHA-Net | 85.15 | 86.02 | 84.93 | 85.68 | 85.44 ± 17.21

Figure 6: The Dice Similarity Coefficient (DSC), Positive Predictive Value
(PPV), Sensitivity (SEN), Average Symmetric Surface Distance (ASD), root mean
square (rms), and Hausdorff Fractal Dimension (HFD) values obtained by Res-
Unet [24], DEHA-Net [29], and our proposed MESAHA-Net, both without and with
multi-view analysis (MVA), are presented alongside their respective standard
deviations.

### Overall Performance

We analyzed the performance of the proposed framework quantitatively while
comparing the results with previously established techniques with respect to
DSC, SEN and PPV as these parameters are utilized by most of the studies.
Table 3 shows the comparison of between our proposed schemes and other state-
of-the-art methods. The results demonstrate proposed MESAHA-Net outperforms
all the previous studies in term of DSC and SEN. However, DEHA-Net [29]
obtained slightly better PPV by utilizing the full slice as input with
handicraft AROI algorithm. In comparison with other systems, Res-UNet [24] and
DEHA-Net [29] achieved significantly improved performance which can be
attributed to the multi-view, i.e., axial, sagittal and coronal, analysis with
handicraft AROI algorithm which adds additional computational expense.
Similarly, CMSF [21] achieved comparative results by employing multiple
networks in cascade manner which greatly increases the computational cost and
makes it difficult to utilize the solution for the real-time clinical
applications. Whereas the proposed scheme only performs axial view analysis
with unified architecture which eliminates the need of additional handicraft
algorithm to estimate the ROIs for the adjacent slices. Most importantly,
proposed MESAHA-Net eliminates the issues pertaining to image scale
normalization by exploiting the adaptive hard attention mechanism.

Although the proposed MESAHA-Net performs patch-wise segmentation along the
axial axis, we also explored the possibility of further improving performance
by utilizing multi-view analysis (MVA). To this end, we performed segmentation
along the sagittal and coronal axes using the proposed MESAHA-Net, and the
final segmentation results were produced by employing the max consensus
criterion [35]. For the sagittal and coronal views, we utilized fixed ROIs
similar to previous methods, such as Res-UNet [24] and DEHA-Net [29]. We
compared both versions of our proposed technique, with and without MVA, to
previously published studies [24, 29] using all evaluation metrics defined in
Section 1.3. In particular, the DSC, PPV, SEN, ASD, rms, and HFD values
obtained by Res-UNet [24], DEHA-Net [29], and our proposed MESAHA-Net without
and with MVA are presented in Fig. 6, respectively.

The results indicate that both previous methods perform better with MVA.
However, MVA slightly degrades the performance of the proposed MESAHA-Net,
which can be attributed to the poor performance of the proposed architecture
along the sagittal and coronal axes due to the incorporation of multiple
inputs. In particular, the higher slice thickness diminishes the effectiveness
of forward and backward MIP images when performing segmentation along sagittal
and coronal views. This finding suggests that the proposed architecture does
not require MVA, as the incorporation of MIP images compensates for 3D
contextual information, which assists in accurate nodule segmentation.

The LIDC/IDRI dataset, annotated by four radiologists, employs a 50% consensus
criterion to determine the ground truth, as illustrated in Table 3. To perform
an insightful analysis concerning radiologist agreement, we conducted a
pairwise DSC assessment among all radiologists, the consensus ground truth,
and the output of MESAHA-Net on the test set. The results are illustrated in
Table 2. The results indicate that the average DSC between the consensus
ground truth and each radiologist is 86.31%, which is considerably greater
than the average inter-radiologist variability of 78.29%. Since the proposed
MESAHA-Net is trained using ground truth, it exhibits consistency when
compared to the four individual radiologists. The average DSC between MESAHA-
Net and each radiologist remains stable within the range of 84.93%–85.68%.

Table 3: The average $\pm$ standard deviation for the quantitative outcomes of
different segmentation techniques, with the best performance highlighted in
bold.

Methodology, year | DSC (%) | SEN (%) | PPV (%)
---|---|---|---
CF-CNN [18], 2017 | 82.15$\pm$10.76 | 92.75$\pm$12.83 | 75.84$\pm$13.14
CDP-ResNet + IWS [20], 2019 | 81.58$\pm$11.05 | 87.30$\pm$14.30 | 79.71$\pm$13.59
Res-UNET + AROI [24], 2020 | 87.55$\pm$10.58 | 91.62$\pm$8.47 | 88.24$\pm$9.52
DB-ResNet [19], 2020 | 82.74$\pm$10.19 | 89.35$\pm$11.79 | 79.64$\pm$13.34
DB UNET + LLIE [25], 2021 | 81.97$\pm$10.16 | 84.57$\pm$14.79 | 79.33$\pm$14.08
MTGAN [26], 2021 | 85.24$\pm$9.01 | – | –
M-SegSEUNet-CRF [22], 2022 | 80.50$\pm$7.6 | 79.80$\pm$14.30 | 84.6$\pm$10.9
CSE-GAN [23], 2022 | 80.74$\pm$ – | 85.46$\pm$ – | 80.56$\pm$ –
CMSF [21], 2022 | 86.75$\pm$ 9.25 | 89.07$\pm$ 8.31 | 83.26$\pm$ 10.21
FMGA [36], 2022 | 81.32$\pm$ – | 92.33$\pm$ – | 74.78$\pm$ –
DEHA-Net + AROI [29], 2023 | 87.91$\pm$6.27 | 90.84$\pm$8.22 | 89.56$\pm$10.07
Our Method | 88.27$\pm$7.42 | 92.88$\pm$9.54 | 86.95$\pm$11.29

Table 4: DSCs on different nodule groups of the LIDC testing set. | Characteristic scores
---|---
Characteristics | 1 | 2 | 3 | 4 | 5 | 6
Calcification | 87.38 | $-$ | 88.15 | 81.24 | 80.44 | 88.16
[1] | $-$ | [49] | [4] | [1] | [438]
Internal structure | 88.08 | 86.49 | $-$ | 89.61 | $-$ | $-$
| [489] | [1] | $-$ | [3] | $-$ | $-$
Lobulation | 91.23 | 80.84 | 92.12 | 80.95 | 91.44 | $-$
| [272] | [122] | [49] | [32] | [18] | $-$
Malignancy | 83.07 | 94.13 | 86.78 | 83.4 | 92.9 | $-$
| [70] | [100] | [162] | [86] | [75] | $-$
Margin | 89.42 | 81.86 | 85.01 | 93.56 | 85.66 | $-$
| [18] | [39] | [65] | [167] | [204] | $-$
Sphericity | 87.71 | 93.45 | 81.85 | 90 | 91.26 | $-$
| [4] | [44] | [155] | [148] | [142] | $-$
Speculation | 86.72 | 94.62 | 85.29 | 78.38 | 92.6 | $-$
| [312] | [96] | [43] | [19] | [23] | $-$
Subtlety | 85.03 | 92.36 | 93.15 | 89.71 | 84.2 | $-$
| [14] | [39] | [98] | [129] | [213] | $-$
Texture | 93.54 | 94.13 | 93.6 | 83.91 | 87.93 | $-$
| [20] | [5] | [23] | [49] | [396] | $-$

Figure 7: Distribution of DSC for the LIDC-IDRI test set.

### Robustness Analysis

The LIDC-IDRI dataset also contains annotations about several nodule features,
including subtlety, internal structure, calcification, sphericity, margin,
lobulation, spiculation, texture, and malignancy. These characteristics
reflect the varying difficulty levels in defining nodule boundaries. To
evaluate the robustness of our method, we separated the test data according to
the different levels of each feature and extracted the results for each level,
as displayed in Table 4. The findings imply that the dice scores attained for
each group are similar, indicating the robustness of our approach.

Additionally, we depicted the distribution of dice scores achieved on the
LIDC-IDRI dataset’s test set, as shown in Fig. 7. The outcomes reveal that
most of the test instances have scores above 80

Figure 8: Relationship between nodule size and the computation time taken by three lung nodule segmentation schemes. Table 5: Comparison of different networks  Network |  | Number of
---
parameters
| Inference time
---
per slice (ms)
VNet (3D) [37] | 9,448,849 | 127.95
UNet (3D) [38] | 8,822,065 | 119.46
ResUNet (2D) [24] | 6,272,916 | 96.61
DEHA-Net (2D) [29] | 536,447 | 72.84
Proposed MESAHA-Net (2D) | 436,707 | 48.34

### Computational Time Analysis

In real-time clinical settings, the time required for lung nodule segmentation
is of paramount importance. Consequently, the computation time of automatic
segmentation solutions plays a critical role in evaluating the efficiency of
lung nodule segmentation systems. In Table 3, we compared previously published
studies, among which Res-UNet with A-ROI[24] and DEHA-Net [29] achieved
comparable performance. Both techniques employ a similar patch-wise
investigation strategy, taking a single ROI as input to generate 3D
segmentation of lung nodules. Subsequently, in this section, we present a
comparison of the computational time of the proposed MESAHA-Net with Res-UNet
with A-ROI[24] and DEHA-Net [29]. To ensure a fair analysis, we implemented
all three schemes on the same hardware, i.e., an NVIDIA TITAN RTX GPU with a
32-core Intel Asus CPU. To perform the computational time analysis, we first
divided the entire test set into seven groups based on lung nodule diameter
(i.e., 3–5, 5–10, 10–15, 15–20, 20–25, 25–30, and greater than 30 mm).

We utilized the same ROI for the initialization of segmentation by each
technique and recorded the total computational time taken by each technique to
produce 3D segmentation of each lung nodule. Fig. 8 demonstrates the average
times recorded for each group of nodules with all three techniques. Firstly,
all three techniques follow an increasing trend with respect to the increase
in diameter, which is due to the patch-wise (2D) nature of these systems.
Larger nodules usually cover more slices, and for performing slice-by-slice
segmentation of nodules, more inferences are required. However, the increase
in computational time is significantly sharp for previously published studies,
whereas the proposed MESAHA-Net demonstrates an incremental change in
computational time and performs 3D segmentation of lung nodules more
efficiently.

We also compared the number of parameters for each network in Table 5, where
we included two 3D models as baselines, i.e., VNet [37] and UNet [38]. We
utilized all the techniques in the table to make inferences on the test set
and calculate the inference time per slice. The results indicate that the
proposed MESAHA-Net achieved the shortest inference time, making it the most
efficient technique for lung nodule segmentation. This optimal performance can
be attributed to the following factors: (1) The proposed MESAHA-Net is a
lightweight model with fewer parameters compared to Res-UNet[24] and DEHA-
Net[29]. (2) The proposed technique only performs axial view analysis, whereas
the other two methods necessitate multi-view analysis to achieve similar
performance. (3) The proposed solution employs a unified single model, i.e.,
MESAHA-Net, eliminating the need for additional handcrafted algorithms, such
as A-ROI, to estimate the ROIs for preceding and subsequent slices. (4) Unlike
DEHA-Net [29], which takes a full slice as input to circumvent rescaling-
induced errors, the proposed MESAHA-Net only takes a patch of a slice, i.e.,
$96\times 96$, as input.

### Qualitative Analysis

In this section, we present a qualitative analysis of the results obtained
from various methods for different types and sizes of lung nodules. Fig. 10
illustrates the 2D segmentation results from five distinct techniques,
including the proposed MESAHA-Net, on five diverse types and sizes of lung
nodules, i.e., L1–L5. Overall, it can be observed that 2D patch-wise
approaches yield better results than 3D network-based methods, primarily due
to the significant variations in lung nodule sizes. The results also
demonstrate that the proposed technique surpasses other methods while
maintaining consistent performance across different types and sizes of
nodules. This suggests that the proposed MESAHA-Net effectively improves lung
nodule segmentation by leveraging self-adaptive hard attention to mitigate
rescaling-induced errors.

To further evaluate the performance of the proposed scheme in detecting the
presence of nodules in each slice, we conducted a 3D rendering of the final 3D
segmentation for various nodules. Fig. 9 presents the results obtained for
five lung nodules with different diameters, showcasing multiple segmented
slices and the 3D rendered outcomes of the proposed MESAHA-Net alongside the
ground truth. These results demonstrate that the proposed scheme effectively
identifies the extent of lung nodules in preceding and subsequent slices,
enabling accurate 3D segmentation of lung nodules.

Figure 9: The segmentation results of the proposed MESAHA-Net on various
nodules of differing sizes are displayed. Each row represents slices from a
single nodule within the test set.

Figure 10: A visual comparison of the results obtained from 3D UNet[38],
VNet[37], 2D ResUNet[24], 2D DEHA-Net [29] with the AROI algorithm, and our
proposed MESAHA-Net is presented. Each row depicts the central slice of
different nodule from test set.

## Conclusion

In this study, we presented a novel multi-encoder-based self-adaptive hard
attention network (MESAHA-Net) that achieves accurate and efficient
segmentation of pulmonary nodules. Our proposed framework perform patch-wise
segmentation of lung nodules iteratively by focusing on the nodule region in
each slice to generate 3D volumetric segmentation of lung nodules. The MESAHA-
Net utilizes three distinct inputs: raw slice patch, bidirectional maximum
intensity projection (MIP) images, and hard attention in the form of regions
of interest (ROI) masks. This incorporation enables the framework to harness
both 2D and 3D spatial and contextual information, essential for accurately
segmenting lung nodules and estimating ROIs for neighboring slices.
Additionally, we have introduced an adaptive hard-attention mechanism that
enforces the network to focus on the nodule region while mitigating rescaling-
induced errors. We rigorously evaluated our proposed method on the LIDC-IDRI
dataset, the most comprehensive benchmark dataset available. The findings
indicate that our framework surpasses previously published lung nodule
segmentation approaches in terms of performance and computational complexity,
rendering it suitable for real-time clinical applications.

## References

  * [1] F. Bray _et al._ , “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: A Cancer Journal for Clinicians_ , vol. 68, no. 6, pp. 394–424, 2018\.
  * [2] C. I. Henschke _et al._ , “Survival of patients with stage i lung cancer detected on ct screening,” _New England Journal of Medicine_ , vol. 355, no. 17, pp. 1763–1771, 2006.
  * [3] H. J. Aerts, E. R. Velazquez, R. T. Leijenaar, C. Parmar, P. Grossmann, S. Carvalho, J. Bussink, R. Monshouwer, B. Haibe-Kains, D. Rietveld _et al._ , “Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach,” _Nature communications_ , vol. 5, no. 1, pp. 1–9, 2014.
  * [4] M. K. Gould _et al._ , “Evaluating test strategies for colorectal cancer screening: a decision analysis for the us preventive services task force,” _Annals of Internal Medicine_ , vol. 146, no. 5, pp. 358–369, 2007.
  * [5] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman _et al._ , “The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans,” _Medical physics_ , vol. 38, no. 2, pp. 915–931, 2011.
  * [6] M. F. McNitt-Gray, J. Wyckoff, J. W. Sayre, J. G. Goldin, and D. R. Aberle, “Standardizing ct lung density measure across scanner manufacturers,” _Medical Physics_ , vol. 36, no. 9, pp. 4124–4134, 2009.
  * [7] M. L. Giger, “Computer-aided diagnosis in medical imaging: historical review, current status and future potential,” _Computerized Medical Imaging and Graphics_ , vol. 37, no. 5, pp. 343–356, 2013.
  * [8] A. A. A. Setio, F. Ciompi, G. Litjens, P. Gerke, C. Jacobs, S. J. Van Riel, M. M. W. Wille, M. Naqibullah, C. I. Sánchez, and B. Van Ginneken, “Pulmonary nodule detection in ct images: false positive reduction using multi-view convolutional networks,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, pp. 1160–1169, 2016.
  * [9] A. McWilliams _et al._ , “Probability of cancer in pulmonary nodules detected on first screening ct,” _The New England Journal of Medicine_ , vol. 369, no. 10, pp. 910–919, 2013.
  * [10] B. Lassen, C. Jacobs, J. Kuhnigk, B. Van Ginneken, and E. Van Rikxoort, “Robust semi-automatic segmentation of pulmonary subsolid nodules in chest computed tomography scans,” _Physics in Medicine & Biology_, vol. 60, no. 3, p. 1307, 2015.
  * [11] S. Diciotti, S. Lombardo, M. Falchini, G. Picozzi, and M. Mascalchi, “Automated segmentation refinement of small lung nodules in ct scans by local shape analysis,” _IEEE Transactions on Biomedical Engineering_ , vol. 58, no. 12, pp. 3418–3428, 2011.
  * [12] J. Song, C. Yang, L. Fan, K. Wang, F. Yang, S. Liu, and J. Tian, “Lung lesion extraction using a toboggan based growing automatic segmentation approach,” _IEEE transactions on medical imaging_ , vol. 35, no. 1, pp. 337–353, 2015\.
  * [13] L. Lu, P. Devarakota, S. Vikal, D. Wu, Y. Zheng, and M. Wolf, “Computer aided diagnosis using multilevel image features on large-scale evaluation,” in _Medical Computer Vision. Large Data in Medical Imaging: Third International MICCAI Workshop, MCV 2013, Nagoya, Japan, September 26, 2013, Revised Selected Papers 3_. Springer, 2014, pp. 161–174.
  * [14] W. Gao and Z.-H. Zhou, “Dropout rademacher complexity of deep neural networks,” _Science China Information Sciences_ , vol. 59, pp. 1–12, 2016\.
  * [15] S. Hamidian, B. Sahiner, N. Petrick, and A. Pezeshk, “3d convolutional neural network for automatic detection of lung nodules in chest ct,” in _Medical Imaging 2017: Computer-Aided Diagnosis_ , vol. 10134. SPIE, 2017, pp. 54–59.
  * [16] L. Fu, J. Ma, Y. Ren, Y. S. Han, and J. Zhao, “Automatic detection of lung nodules: false positive reduction using convolution neural networks and handcrafted features,” in _Medical Imaging 2017: Computer-Aided Diagnosis_ , vol. 10134. SPIE, 2017, pp. 60–67.
  * [17] M. M. Warrier and L. Abraham, “A review on early diagnosis of lung cancer from ct images using deep learning,” in _Proceedings of the International Conference on Paradigms of Computing, Communication and Data Sciences: PCCDS 2022_. Springer, 2023, pp. 653–670.
  * [18] S. Wang, M. Zhou, Z. Liu, Z. Liu, D. Gu, Y. Zang, D. Dong, O. Gevaert, and J. Tian, “Central focused convolutional neural networks: Developing a data-driven model for lung nodule segmentation,” _Medical image analysis_ , vol. 40, pp. 172–183, 2017.
  * [19] H. Cao, H. Liu, E. Song, C.-C. Hung, G. Ma, X. Xu, R. Jin, and J. Lu, “Dual-branch residual network for lung nodule segmentation,” _Applied Soft Computing_ , vol. 86, p. 105934, 2020.
  * [20] H. Liu, H. Cao, E. Song, G. Ma, X. Xu, R. Jin, Y. Jin, and C.-C. Hung, “A cascaded dual-pathway residual network for lung nodule segmentation in ct images,” _Physica Medica_ , vol. 63, pp. 112–121, 2019.
  * [21] Z. Zhou, F. Gou, Y. Tan, and J. Wu, “A cascaded multi-stage framework for automatic detection and segmentation of pulmonary nodules in developing countries,” _IEEE Journal of Biomedical and Health Informatics_ , 2022.
  * [22] B. Zhang, S. Qi, Y. Wu, X. Pan, Y. Yao, W. Qian, and Y. Guan, “Multi-scale segmentation squeeze-and-excitation unet with conditional random field for segmenting lung tumor from ct images,” _Computer Methods and Programs in Biomedicine_ , p. 106946, 2022.
  * [23] S. Tyagi and S. N. Talbar, “Cse-gan: A 3d conditional generative adversarial network with concurrent squeeze-and-excitation blocks for lung nodule segmentation,” _Computers in Biology and Medicine_ , vol. 147, p. 105781, 2022.
  * [24] M. Usman, B.-D. Lee, S.-S. Byon, S.-H. Kim, B.-i. Lee, and Y.-G. Shin, “Volumetric lung nodule segmentation using adaptive roi with multi-view residual learning,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–15, 2020\.
  * [25] Z. Wu, Q. Zhou, and F. Wang, “Coarse-to-fine lung nodule segmentation in ct images with image enhancement and dual-branch network,” _IEEE Access_ , vol. 9, pp. 7255–7262, 2021.
  * [26] W. Chen, Q. Wang, K. Wang, D. Yang, X. Zhang, C. Liu, and Y. Li, “Mtgan: mask and texture-driven generative adversarial network for lung nodule segmentation,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 1029–1035.
  * [27] Y. Gu, J. Chi, J. Liu, L. Yang, B. Zhang, D. Yu, Y. Zhao, and X. Lu, “A survey of computer-aided diagnosis of lung nodules from ct scans using deep learning,” _Computers in biology and medicine_ , vol. 137, p. 104806, 2021\.
  * [28] P. Cao, X. Liu, J. Yang, D. Zhao, and J. Huang, “Improving deep convolutional neural networks for lung nodule segmentation in ct images,” _Physics in Medicine & Biology_, vol. 62, no. 23, p. 9174, 2017.
  * [29] M. Usman and Y. G. Shin, “Deha-net: Dual-encoder based hard attention network with adaptive roi mechanism for lung nodule segmentation,” 2022.
  * [30] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, B. Glocker, and D. Rueckert, “Attention u-net: Learning where to look for the pancreas,” in _Medical Imaging with Deep Learning_ , 2018. [Online]. Available: https://openreview.net/forum?id=Skft7cijM
  * [31] V. Mnih, N. Heess, A. Graves _et al._ , “Recurrent models of visual attention,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [32] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _arXiv preprint arXiv:1409.0473_ , 2014\.
  * [33] H. Kervadec, J. Bouchtiba, C. Desrosiers, E. Granger, J. Dolz, and I. Ben Ayed, “Boundary loss for highly unbalanced segmentation,” _Medical Image Analysis_ , vol. 67, p. 101851, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1361841520302152
  * [34] T. Kubota, A. K. Jerebko, M. Dewan, M. Salganicoff, and A. Krishnan, “Segmentation of pulmonary nodules of various densities with morphological approaches and convexity models,” _Medical Image Analysis_ , vol. 15, no. 1, pp. 133–154, 2011.
  * [35] S. K. Warfield, K. H. Zou, and W. M. Wells, “Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation,” _IEEE transactions on medical imaging_ , vol. 23, no. 7, pp. 903–921, 2004.
  * [36] Q. Chen, W. Xie, P. Zhou, C. Zheng, and D. Wu, “Multi-crop convolutional neural networks for fast lung nodule segmentation,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2021.
  * [37] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 fourth international conference on 3D vision (3DV)_. Ieee, 2016, pp. 565–571.
  * [38] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19_. Springer, 2016, pp. 424–432.