# Emergent helicity in free-standing semiflexible, charged polymers

Debarshi Mitra and Apratim Chatterji<EMAIL_ADDRESS>Dept. of
Physics, IISER-Pune, India-411008.

(August 28, 2024)

###### Abstract

Helical motifs are ubiquitious in macromolecular systems. The mechanism of
spontaneous emergence of helicity is unknown, especially in cases where
torsional interactions are absent. Emergence of helical order needs
coordinated organization over long distances in polymeric macromolecules. We
establish a very generic mechanism to obtain spontaneous helicity by inducing
screened Coulomb interactions between monomers in a semiflexible
heteropolymer. Due to changes in solvent conditions, different segments
(monomers) of a polymeric chain can get locally charged with charges of
differing polarities and magnitudes along the chain contour. This in turn
leads to spontaneous emergence of transient helical structures along the chain
contour for a wide range of Debye-lengths. We have avoided using torsional
potentials to obtain helical structures and rely only on radially symmetric
interactions. Lastly, transient helices can be made long-lived when they are
subjected to geometric confinement, which can emerge in experimental
realizations through a variety of conditions.

##  I Introduction

The helical motif is a recurrent feature of several macro-molecular systems
and is found in disparate systems in nature. These systems may be as varied as
the chromosome of bacterial cells kleckner_helix  and the secondary structure
of proteins or actin filaments or in collagen molecules actin ; collagen ;
Banavar2004 ; ecoli_nucleoid . Thus, the helical structure is ubiquitous in
the macro-molecular world. This has spurred a number of studies that
investigate their emergence in several natural systems Maritan2000 ;
Pokroy2009 ; Sabeur2008 ; Li2018 ; Vega2019 ; Snir2005 ; Banavar2004 ;
Gerbode2012 ; Forterre2011 ; Biswas2021 ; Chaudhuri2012 . Furthermore, helical
motifs in synthetic molecules have been explored for their key technological
ramifications Srivastava2010  such as the synthesis of NEMs devices Nakano2001
; Zhong2008 ; Jiang2013 ; Singh2014 ; synthetic_helix ; kotov , piezoelectric
devices Kong2003  and other optical materials Stratford2014 ; Lei2018 ;
feng2017assembly . For these applications, helical molecules are typically
designed using techniques such as vapour deposition which rely on specific
interactions between the constituent monomers. Alternatively, helical
templates are also often employed Wang2013 ; Wang2008 ; Zhang2002 ; Zhang2003
; Liu2011 ; Rapaport2002 .

It would be of interest to unearth mechanisms by which one may induce chains
to spontaneously self-organize to obtain helical structures at the $nm$ to
$\mu m$ length scales. If these mechanisms are generic in nature and
independent of the chemical details of the monomers, then one can have an
alternative strategy of designing synthetic helical macro-molecules in the
laboratory. More significantly, such studies would also shed light on the
mechanisms governing the formation of helical structures in nature, both
inside and outside the living cell.

Previous reports have demonstrated the formation of extremely short-lived
helical polymers in bad solvents undergoing collapse Sabeur2008 . The packing
of tubular filaments in confined cylinders also lead to helix formation for
particular ratios of pitch and radius Maritan2000 . Yet, another study shows
that the ground state of a self-attracting chain shows a variety of o
structural motifs including the helix, depending on the nature of the
stiffness present in the chain (energetic or entropic) krbi2019 ; Snir2005 .
In our previous work we showed that a bead-spring model of a semiflexible
polymer chain with monomers all having the same value and polarity of charges
and long range Coulomb interaction between monomers attains a transient
helical conformation, when repulsive interactions are switched on between the
monomers Mitra2020 . The monomers may become charged because of a change in
the nature of the solvent, or if the ionic concentration of the solvent is
suddenly changed or due to other changes in its environment. We obtained
transient helices even when we replaced Coulomb interactions by long range
repulsive power-law interactions.

Here we introduce a more generic model and establish that a semiflexible
polymer chain with charged beads, each carrying a different value of charge
and polarity, can also attain a transient helical conformation even due to the
short range screened (spherically symmetric) Coulomb interactions between the
monomers. This scenario is likely to occur in many natural systems.
Additionally, we show that asymmetry in the distribution of charges along the
polymer chain plays a pivotal role in helix formation. A chain with an
approximately equal number of positive and negative charges does not form
helices. The helix formation takes place only for a certain range of
parameters of the polymer, such as the persistence length, the spring
constant, the Debye length etc. We further show that the helices formed by
these screened electrostatic interactions can be made long-lived under
geometric confinement, in a cuboid of appropriate dimensions.

In the case of bacterial nucleoids, this geometric confinement can be imposed
by the bacterial cell leading to stable helical structures of nucleoids. These
beads can be molecular units of a heteropolymer, or groups of amino acid
sequences with length scale of the size of the beads, or even larger units of
molecules such as nucleosome complex in the DNA dressed by suitable (charged)
proteins, or even polymeric chains where the monomeric units are micron sized
colloids. In this case, the colloidal surface can be dressed by microgel
particles which have been cross-linked together as shown by Biswas2017 ;
Biswas2021 . The colloidal particles then form a polymeric chain held together
by cross-links with tunable semiflexibility (persistence length). If one uses
colloids that are chemically different to each other, to form the chain, a
change in the ionic concentration in the solution can result in a different
value of charge on each colloid.

##  II Model

We use Brownian dynamics simulations to study helix formation in a single
semiflexible, bead-spring model of a polymer chain, where the monomers have
different values of charges along the chain contour as determined by a
probability distribution. The center of each monomer is connected by harmonic
springs of spring constant $\kappa$ and of equilibrium length $a$ to the
center of two neighbouring monomers. We choose $a$ to be the unit of length
for our simulations. Furthermore, there is a bending energy cost per every
triplet of monomers along the chain contour to incorporate effects of
semiflexibility. The energy cost of having bends along the chain, is realized
by the bending potential $V(\theta)=\epsilon_{B}\cos(\theta)$, where $\theta$
is the angle between the bond vectors ($-\mathbf{r}_{i},\mathbf{r}_{i+1}$).
The vector $\mathbf{r}_{i}$ is the vector joining monomer $i-1$ to its
neighbouring monomer $i$ along the chain contour. The quantity $\epsilon_{B}$
can also be directly related to the persistence length $\ell_{p}$,

$\ell_{p}\approx a\epsilon_{B}/k_{B}T$ (1)

for small deviations of $\theta$ from $\pi$, refer appendix of Mitra2020 . In
our simulations, energy is measured in units of the thermal energy $k_{B}T$.
The excluded volume of the monomers are modelled by the WCA (Weeks, Chandler,
Andersen) potential and all the monomers have diameters $\sigma=0.7a$.

Additionally, any two monomers $i$ and $j$ situated at a distance $r_{ij}$
from each other, with charges $q_{i}$ and $q_{j}$ interact with each other
through the screened Coulomb interaction.

$V(r_{ij})=\frac{q_{i}q_{j}}{4\pi\epsilon
r_{ij}}\times\exp^{(-r_{ij}/\lambda_{D})}...\forall r_{ij}<r_{c}$ (2)

where the Debye length $\lambda_{D}$ is a parameter and sets the length scale
over which Coulomb interactions get screened out. The permittivity $\epsilon$
of the background medium within which the chain is suspended, can be expressed
in terms of the permittivity of vacuum $\epsilon_{0}$ and the relative
permittivity $\epsilon_{r}$. The strength of interactions is determined by the
value of $q_{i}$ and $q_{j}$, but for our calculation we will express charge
in units of $Z_{0}$, such that the interaction between any two charged
monomers can be expressed as:

$V(r_{ij})=Z_{i}Z_{j}/r_{ij}\times\exp^{(-r_{ij}/\lambda_{D})}...\forall
r_{ij}<r_{c}$ (3)

where, $Z_{i}$ and $Z_{j}$ denote the value of charges and is expressed in
units of $Z_{0}$, on the interacting monomers in simulation units. In the
subsequent section, we elaborate on how the values of $Z$ can be related to
charge in real units.

We assign charges to the monomers in the chain such that there is
stochasticity in the distribution of charges along the polymer chain. The
charge $Z_{i}$ of each monomer $i$ in simulations will be allocated in terms
of the asymmetry parameter $\psi$ and $Z_{0}$ such that
$Z_{i}=Z_{0}\times(r-\psi)$, where $r$ is a random number whose values are
chosen from a uniform distribution of random numbers between $0$ and $1$. We
vary $\psi$ between $0$ and $1$ to estimate the net charge of the chain. A
value of $\psi=1$ implies that all monomers have random values of charges
between $Z_{1}=0$ and $Z_{2}=-Z_{0}$ s.u. A value of $\psi=0$ implies that all
monomers have random values of charges between $Z_{1}=0$ and $Z_{2}=Z_{0}$
s.u. Correspondingly, for $\psi=0.5$, the charge on the monomers can take
random values between $Z_{1}=-Z_{0}/2$ to $Z_{2}=Z_{0}/2$ s.u. Thus, in
general, the charge on a monomer $i$ can vary between $Z_{1}$ and $Z_{2}$.

We will establish later in the manuscript that if $\psi=0.5$ such that there
are statistically equal number of positive and negative charges, then we do
not obtain helices. We need to introduce asymmetry in the distribution of
charges using $\psi$, such that $Z_{1}\neq Z_{2}$. The degree of asymmetry in
the distribution of charges along the polymer chain is denoted by the quantity
$\psi$. Thus, $\psi=0.1$ corresponds to a chain that statistically has $90\%$
of the monomers having a positive charge.

For our Brownian dynamics simulations where the friction constant is
$\mathbf{\zeta}$, and the unit of time $\tau$ is set by
$\tau=a^{2}\zeta/k_{B}T$, the time taken for an isolated monomer to diffuse a
distance of $a$.. If we set $\zeta=1$ such that $\tau=1$ with $k_{B}T=1$ and
$a=1$ (units of energy and length, respectively, the over-damped stochastic
Brownian dynamics equation is integrated with time step $dt=0.0001\tau$.

###  II.1 Relevance to experimental systems

We now establish a correspondence between the magnitudes of charges used in
the simulation (expressed in s.u.), and the magnitudes of charges in real
units. If two monomers having unit charges in the simulations, are at a unit
distance away from each other then they interact with Coulomb potential energy
of strength $k_{B}T$. Let $q_{o}$ be the charge in real units, corresponding
to $Z=1s.u$, where $Z$ denotes the charge on a monomer in the simulations.

In real units two charges ($q_{0}$) of identical charges have potential energy
given by

$\frac{(q_{0})^{2}}{4\pi\epsilon a}=\frac{Z^{2}}{a}=k_{B}T$ (4)

Thus at room temperature of $T=300K$, the value of $q_{o}$ in water
($\epsilon_{r}=80$) placed at unit distance ($a$) is given by,

$q_{o}=\sqrt{4\pi\epsilon ak_{B}T}=\sqrt{37\times 10^{-30}a}\,C.$ (5)

where the charge $q_{0}$ is measured in Coulomb (C). If we choose $a=1nm$,
then $q_{o}=1.9\times 10^{-19}$C, or $1.2e$, where $e$ denotes the electronic
charge. If $a=10nm$ then we obtain $q_{o}\approx 4e$. If $a=100nm$ then,
$q_{o}\approx 12e$ and if $a=1\mu m$ then $q_{o}\approx 40e$. In this study we
often use $Z_{0}\sim 10s.u$ which corresponds to a charge of $40e$ for a
monomer with size $10$ nm. But that is the maximum charge on a monomer for
$\psi=1$. For lower values of $\psi$, the maximum charge that a monomer can
take will be lower. Thus, the charges used in this study may feasibly occur in
many macro-molecular systems.

Figure 1:  Helix formation: The snapshots shows various stages of the helix
formation in a bead-spring model of a semi-flexible polymeric chain with $100$
monomers, with contour length $R_{c}=99a$. Initially, the uncharged polymer is
in equilibrium with a thermal bath : representative snapshot shown in [a] at
time $t=3\tau$. Snapshot [b] shows a representative conformation of the chain
soon after the monomers acquires charge at time $t=10\tau$, where we see
formation of helices along the chain contour. At (c) $t=20\tau$, we see that
the helix gradually “unwinds” and relaxes to a helical conformation having a
larger pitch. The charge on each monomer has a random value, between $-4.0$ to
$36$ s.u. (simulation unit). This corresponds to a asymmetry parameter of
$\psi=0.1$ (refer to the Methods section for the definition of $\psi$) and
$Z_{0}=40$ s.u. The bending energy cost $\epsilon_{B}=200k_{B}T$
(corresponding to a persistence length of $200a$ and spring constant
$\kappa=100k_{B}T/a^{2}$, and the Debye length $\lambda_{d}=5a$. In (d) we
show the helical order parameter $H2$ (running averaged over $0.33\tau$) as a
function of simulation time $t/\tau$, for $3$ independent runs. We see a sharp
rise in $H2$ at time $t=10\tau$, which decays thereafter, indicating transient
helix formation. Subfigure (e) shows the end to end distance $R_{ee}/R_{c}$
versus $t/\tau$.

Figure 2:  Mechanism of Helix formation:  In (a), (b) and (c) we show the
harmonic spring energy ($U_{H}$) , the screened coulomb energy ($U_{sc}$) and
the bending energy ($U_{b}$) per monomer. We see that just after the screened
Coulomb interaction is switched on, the screened coulomb energy and the
bending energies peak, and subsequently relax leading to the “unwinding” of
the helix. The springs in the polymer chains however continue to remain
stretched due to the screened Coulomb interactions, as shown in (a). The
sudden switching on of screened Coulomb interactions at $t=10\tau$, leads to
the formation of sharp angular conformations. This can vbe inferred by noting
that $U_{b}$ shows a peak at this point,in the plot given in (b).
Subsequently, the transient helix formation takes place by the gradual
relaxation of these sharp bends. Furthermore, if the springs are made too
stiff, these sharp bends are not formed since a high value of $\kappa$
prevents the monomer-bonds from stretching. Thus, in (d) we show that $\langle
H2\rangle$ shows a monotonic decrease with increase in $\kappa$. We calculate
the time averaged $\langle H2\rangle$ over $10\tau$ to $20\tau$. The values of
other parameters are same as that of the data presented in Fig.1.

##  III Results

###  III.1 Helix Formation

We observe spontaneous emergence of helices if a uncharged semiflexible
polymeric chain acquired charges at time $t=T_{0}$ due to change in external
conditions, e.g. change in pH of the solvent in which it is immersed. The
helices formed are transient but may be stabilized by other factors such as
hydrogen bond and geometric confinement, as we show later. We establish that
the stochasticity in the value or sign of the charges on the monomers along
the chain contour do not affect helix formation as long as there is sufficient
asymmetry (as determined by $\psi$) in the charge distribution. We have tried
out different initial conditions, such that monomers $i-1,i,i+1,i+2$ have
different value of charges in different runs for a particular value of $\psi$
(refer to the Methods section for the definition of $\psi$) and monitor the
helicity developed as we switch on charges on the monomers.

We measure helicity using the order parameters $H2$ which is defined as

$H2=\frac{1}{N-3}(\sum_{i=2}^{i=N-2}\mathbf{u}_{i}.\mathbf{u}_{i+1})$ (6)

where $\mathbf{u}_{i}$ is the unit vector of
$\mathbf{U}_{i}=\mathbf{r}_{i}\times\mathbf{r}_{i+1}$. A simple uncharged,
semi-flexible polymer chain shows $H2$ values $\approx 0$ (or negative values
of $H2$) as expected for a chain locally bent due to thermal fluctuations,
while calculating $H2$ over helical segments of a polymer chain would lead to
high positive values of $H2$ Biswas2021 ; Mitra2020 .

Past studies Sabeur2008 ; Mitra2020  have also used another order parameter
which is given by,

$H4=\frac{1}{N-2}(\sum_{i=2}^{i=N-1}\mathbf{u}_{i})^{2};$ (7)

A perfect helix with uniform handedness, will have vectors $u_{i}$ pointing
along the helix axis, and hence $H4$ will have a value of $\approx 1$.
However, if one obtains a helical polymer where half of the chain is right-
handed, and the rest of it is left-handed, $H4$ will be zero. In this study,
we do not have any control over the handedness of the helical structures and
thus for long chains, the chain is likely to have an equal number of right-
handed and left-handed segments. Thus, we use the parameter $H2$ to quantify
helicity, in the remainder of the manuscript. In the future, we will report
how we can design left handed and right handed helices, made to order.

We start our simulations with a uncharged semiflexible polymer chain in a heat
bath, allow the chain to reach equilibrium over $t=T_{0}=10\tau$ ($10^{5}$
iterations), and switch on charge on the monomers thereafter. We show the
various stages of helix formation through representative snapshots give in
Fig.1(a-c). We also show that the helical polymer chain can easily be
distinguished from a non-helical, uncharged, semiflexible polymer chain using
$H2$, which fluctuates near zero till $t=10\tau$ and then show a finite non-
zero value. In Fig.1d we show $H2$ as a function of time $t$ for three
independent runs. In Fig.1e, we plot the end to end distance $R_{ee}$, scaled
by the contour length $R_{C}$ ($R_{c}=99a$) as a function of time. We see that
the end to distances fluctuate at values $\approx 1$ for three independent
runs till $t=10\tau$, and thereafter the $R_{ee}/R_{c}$ shows a sudden
increase. Thus we show that the helix formation is independent of the initial
configuration of the monomers as well on the stochastic distribution of
charges along the chain contour.

Mechanism of helix formation:  The uncharged semiflexible polymer chain
undergoes stochastic conformational fluctuations due to the thermal noise from
$t=0$ to $t=10\tau$. When the electrostatic interactions are switched on,
electrostatic forces leads to a reinforcement of the angular conformations
with larger deviations in the angle $\theta$ from the chain axis. This is
because neighbouring charges repel each other, and move neighbouring monomers
along the line joining their centers, and away from the chain axis. These
angular conformations (with large anglular deviations) are penalised due to
the semiflexibility. As the different chain segments along the chain contour
locally straighten and stretch out from the axis to reduce bending energy
costs, the polymer consequently attains a helical conformation.

This can be surmised from Fig.2 where we plot the different energy
contributions in the system, for parameters identical to that of Fig.1. In
Fig.2a we plot the energy per spring, $U_{H}$ versus time $t$. We see that at
$t<T_{0}$, $U_{H}$ fluctuates around $0.5k_{B}T$, as expected from the
equipartition theorem. However at $t=10\tau$, we see that $U_{H}$ shows a
drastic increase which indicates that the bonds get stretched severely as a
consequence of the electrostatic (e.s) interactions. This is further
corroborated by the data presented in Fig.2b where we plot the screened
Coulomb energy per monomer $U_{sc}$ as a function of time. We see that at
$t=10\tau$, $U_{sc}$ shows a sudden spike which subsequently relaxes to lower
values.

We also look at the bending energy $U_{b}$ per triplet of monomers, in Fig.2c
and we find that at $t=T_{0}=10\tau$, the bending energy shows a sudden spike,
indicating a sharply kinked (angular) conformation. However, such a
conformation is penalised by the bending energy which prevents such sharp
angles. Thus the helix formation occurs as a relaxation of transiently formed
sharp angles. We emphasize again that the helix formed is transient and
eventually the helical conformation dissolves. Thereafter, the polymer chain
adopts an extended linear configuration due to the electrostatic interactions.
Furthermore, thermal fluctuations play a pivotal role in the helix formation
in creating the initial small bends which get accentuated after switching on
the charge on the monomers. The transient helices do not form, if we switch on
charge at temperature $T=0$, as there are no thermal fluctuations. This has
been explored in detail in Mitra2020 , previously.

The importance of the formation of kinks in the subsequent emergence of
helicity can also be gauged by the increasing the simulation $\kappa$
parameter to high values, keeping $k_{B}T=1$. We monitor the time averaged
value of $H2$ with increasing $\kappa$ in Fig.2d, and observe that $\langle
H2\rangle$ decreases with increasing $\kappa$. An increase in the spring
constant prevents the bonds from stretching as much, and thereby formation of
kinks on switching on of charges, and that in turn in turn suppresses
subsequent helix formation.

Figure 3:  Unwinding of the helix: Change in helical pitch To quantitatively
establish the unwinding of the helical polymer we look at the periodicity
parameter W as a function of time. W is a scalar quantity resulting from the
dot product of bond vectors along the chain contour with a vector
perpendicular to the axis of the helical polymer chain. The periodicity in W
indicates the pitch of the helix. In subfigure (a) we show $W$ for different
times for a chain of $100$ monomers, where we have switched on electrostatic
interactions at $t=10\tau$. The other parameters are also identical to that of
the data presented in Fig.1 We note that due to the gradual unwinding of the
helical polymer the periodicity in W decreases.

Unwinding of the transient helices
The emergent structure of helices “unwind” with time. There are two factors
that contribute to the unwinding: the bending energy and the screened coulomb
interaction. The high energy cost of bending leads to the gradual relaxation
of ”kinks” (sharp angles) along the chain contour and thus leads to a
straightening of the chain, such that $\theta\approx\pi$. This results in a
gradual dissolution of the helical structures. The repulsive electrostatic
interaction also leads to an “effective” additional bending energy cost
es_stiff  and causes the polymer chain to adopt an extended configuration.
Thus the helix formation occurs only as a kinetic pathway to an extended
linear equilibrium conformation.

Figure 4:  Range of parameters conducive to helix formation: For this set of
subfigures (a-d), we keep the parameters $Z_{0}$, $\epsilon_{B}$,
$\lambda_{D}$, $\kappa$, and the asymmetry parameter $\psi$ identical to that
in Fig.1, but then vary one parameter at a time keeping the others fixed. We
plot the time averaged helicity order parameter $\langle H2\rangle$ to analyze
the helix formation as we modify different system parameters. Subfigure (d)
shows the dependence of $\langle H2\rangle$ on the charge asymmetry parameter
$\psi*=1-2\psi$. The quantity $\psi*$ indicates the asymmetry in the charge
distribution in the polymer chain. The data in (c) and (d) has been produced
by averaging over $10$ independent runs.

Figure 5:  State diagram of helix formation:  Figure shows the parameter space
conducive to helix formation. In (a) we show the phase diagram with respect to
the Debye length $\lambda_{d}$ and the bending energy $\epsilon_{B}$. The
maximum value of the charge in the polymer chain is $Z_{0}=40$ s.u and the
asymmetry parameter $\psi=0.1$ or $\psi*=0.8$. In (b) we show the phase
diagram with respect to the charge $Z_{0}$ and the asymmetry parameter $\psi*$
which is related to $\psi$ as $\psi*=-2(\psi-0.5)$. We keep the Debye length
$\lambda_{d}=5a$ and the bending energy $\epsilon_{B}=200k_{B}T$. In both (a)
and (b) we switch on charges at $t=10\tau$ and the average is computed over an
interval of $10\tau$.

To quantify this unwinding process, we look at the periodicity parameter W as
a function of time. W is essentially the dot product of bond vectors along the
contour with a vector perpendicular to the axis of the helical polymer chain.
We initialize the chain along the $\hat{y}$ direction. The helical axis of the
emergent helix is also along $\hat{y}$. Thus, we calculate W as

$W(i)=\hat{x}.r_{i}$ (8)

where $r_{i}$ denotes the bond vectors along the chain contour, and $i$
denotes the monomer indices along the chain contour. We now quantitatively
establish that the helical conformations of the polymer relax by gradual
unwinding. To that end we plot in Fig.3 the periodicity parameter W at various
times, during the simulation run. We note that the periodicity in W decreases
as simulation time elapses. This indicates that the helical polymer gradually
unwinds and the pitch keeps on increasing. At any given time there may be
helical segments of varying pitch, which gradually increase. Note that the
most frequently occurring pitch of the helix along the chain contour, can be
found from the data presented in Fig.3 by conducting a Fourier transform
Mitra2020 .

The most frequently occurring pitch of the helix $P$ at a particular time also
depends on the value of the bending energy $\epsilon_{B}$. A high value of
$\epsilon_{B}$ leads to larger values of $P$ at a particular instant of
simulation time. The other parameters do not affect the pitch of the helix.

###  III.2 Parameters controlling helix formation

There are $4$ system parameters which can be tuned to obtain the helical
conformations in polymers. They are:

  1. 1.

The magnitude of the charge on a monomer which is proportional to $Z_{0}$.

  2. 2.

The Debye length of the screened Coulomb interaction: $\lambda_{D}$.

  3. 3.

The bending energy $\epsilon_{B}$, which is directly proportional to the
persistence length Mitra2020 .

  4. 4.

The degree of asymmetry in the charge distribution along the polymer chain.

In Fig.4, we investigate how helix formation depends on a balance of system
parameters, and thereby plot the time averaged $\langle H2\rangle$ as we vary
the parameters one at a time. Thereby, we keep the simulation parameters
identical to those used to obtain data of Fig.1, and vary one parameter at a
time. We compute the time averaged $\langle H2\rangle$ over a duration of
$10\tau$ from $t=10\tau$ to $t=20\tau$ .

In Fig.4a, we show that an increase in $Z_{0}$, i.e. maximum value and range
of charges of the monomers along the chain, leads to an increase in helical
order obtained. We observe an increase in $\langle H2\rangle$ with increase in
$Z_{0}$ before it plateaus around $0.6$. Moreover, the quantity $\langle
H2\rangle$ shows a decrease with increase in $\epsilon_{B}$ as shown in
Fig.4b. This is because an increase in the bending energy (or equivalently the
persistence length $\ell_{p}$) prevents the formation of sharp angular
conformations as a response to the turning on of electrostatic forces. As
mentioned before, the formation of sharp angular conformations (kinks) at time
$t=T_{0}$ are crucial to the subsequent formation of helices. Furthermore a
higher chain rigidity also leads to the faster “unwinding” and dissolution of
the transient helices.

An increase in the Debye length $\lambda_{D}$ leads to a larger number of
electrostatic interactions between monomers and thus leads to a greater
propensity of helix formation. This leads to high values of $\langle
H2\rangle$ for $\lambda_{D}$ in the range of $5-10a$. Interestingly we see in
Fig.4c that $\langle H2\rangle$ shows a non-monotonic dependence on
$\lambda_{D}$. Note that here we have $\psi=0.1$ (or $\psi*=0.8$) which
implies that the majority of the charges along the chain have a positive
value. When $\lambda_{D}$ is increased then the strength of repulsive
interactions increase leading to an increase in the helicity. However when
$\lambda_{D}$ is increased further then there are also attractive interactions
along the chain (due to some monomers having charges of opposite polarity)
that each monomer experiences. This leads to a decrease in the propensity for
helix formation and consequently lower values for $\langle H2\rangle$ as there
are strong attractions between different chain segments. This in turn
suppresses the effects of bending energy costs which led to helix formation.
We discuss the consequence of increasing $\lambda_{D}$ for a lower value of
$\psi$ subsequently. We emphasize that this suppression of helix formation is
only due to the introduction of attractive interactions and not due to the
increase in repulsive interactions. One may infer this from the data of
Fig.4a. There we show that increase in $Z_{0}$, leads to larger repulsive
forces which increases the propensity of helix formation.

The asymmetry in the distribution of charges along the polymer chain plays a
crucial role in helix formation. In Fig.4e we show how helicity as measured by
$\langle H2\rangle$ depends on the asymmetry parameter $\psi$ for a constant
value of $Z_{0}$ which is set at $Z_{0}=40$s.u. For convenience, we now define
a new asymmetry parameter $\psi*$ which depends on $\psi$ as $\psi*=1-2\psi$.
Note that $\psi*=+1$ implies that all the charges along the polymer chain are
negative while $\psi=-1$ denotes that all the charges are positive. If
$\psi*=0$ then there are statistically equal number of positive and negative
charges in the polymer chain. Thus greater the value of $|\psi*|$ greater is
the asymmetry in the distribution of charges.

We see that for $\psi*\sim 0$ helicity goes down, which indicates that when
the strength of attractive interactions become comparable to the strength of
repulsive interactions then helicity goes down. Thus helicity crucially relies
on the presence of strong repulsive interactions which may arise from both
positive and negative charges. This symmetry gives rise to the ‘U’ shape seen
in Fig.4d.

In Fig.4a-c we examine the dependence of $\langle H2\rangle$ for a high value
of asymmetry in the charge distribution ($\psi=0.1$ or $\psi*=0.8$). We also
establish that the trends observed in Fig.4 are robust even for lower
asymmetry in charge distribution, i.e for $\psi*=0.4$ (or $\psi=0.3$) albeit
with a smaller value of $\langle H2\rangle$ as expected. This is shown in
Fig.S1 of the Supplementary Information (SI).

Finally, we plot a state diagram using these parameters, using
$\kappa=200k_{B}T/a^{2}$ (same as in Fig.1) to obtain the parameter space
which is conducive to helix formation. In Fig.5a we show the state diagram
with Debye length $\lambda_{D}$ on the y-axis and the bending energy
$\epsilon_{B}$ on the x-axis. We keep $Z_{0}$ and the charge distribution
asymmetry parameter $\psi$ the same as in Fig.1. We note that for
$\lambda_{D}<6a$, $\langle H2\rangle$ shows a decrease with increase in
$\epsilon_{B}$ beyond $1200k_{B}T$. This indicates that for relatively more
rigid chains the helix formation is suppressed as has already been shown in
Fig.4b. However, for intermediate values of $\lambda_{D}$ (i.e.
$\lambda_{D}\approx 10a$), this suppression in propensity to form helices is
not observed. This is because each monomer interacts electrostatically with a
larger number of monomers along the polymer chain. However, for higher values
of $\lambda_{D}$ ($\lambda_{D}>15a$), the propensity to form helices again
decreases. This is consistent with the non-monotonic behaviour of $\langle
H2\rangle$ with increase in $\lambda_{D}$ as established in Fig.4c and is for
the same reason as explained. In Fig.5b we plot another state diagram keeping
the $\lambda_{D}$ and $\epsilon_{B}$ fixed with the values being the same as
that of Fig.1. Here we vary $Z_{0}$ and $\psi*$ to plot the state diagram. We
note that for small values of $Z_{0}$, $\langle H2\rangle\approx 0$ while for
higher values of $Z_{0}$, we obtain high values of $\langle H2\rangle$ only
when $\psi*\approx\pm 1$. These trends are consistent with those established
in Fig.4a and Fig.4e. This once again emphasises that asymmetry in charge
distribution of monomers along the chain is crucial to helix formation.

Figure 6:  Stabilization of transient helices: To examine whether geometric
confinement is able to stabilize the transient helices we induce helix
formation in a cuboid. The length of the cuboidal box $L_{c}=1.1R_{c}$, where
$R_{c}$ is the contour length of the polymer chain having $50$ monomers. The
width of the cuboid was $W_{c}=0.1R_{c}$. We implement walls of the cuboid by
WCA interactions and we keep the parameters identical to that in Fig.1. We
switch on electrostatic interactions due to charges at $10\tau$. In (a) we
show $H2$ (running averaged over $0.33\tau$) as a function of simulation time
for an unconfined chain of $50$ monomers for $3$ independent runs. We see that
the helix formed is transient and gradually dissolves with time. In (b) we
show again a chain of $50$ monomers similar to that of (a) but under
confinement. We see that $H2$ (running averaged over $0.33\tau$) shows a high
value even at long times for $3$ independent runs. Thus geometric confinement
lead to the stabilization of helices.

###  III.3 Confinement stabilizes helices

We now study whether imposing geometric confinement leads to the emergence of
long-lived helices in our model. To investigate this, we simulate the same
system confined within a cuboid, where the rigid walls which are modelled by
inducing repulsive WCA interactions to prevent the monomers crossing the
walls, refer Fig.6. Thus we show that the transient helices formed due to
electrostatic interactions between charged monomers may be stabilized or made
long lived under confinement. In Fig.6a we show that $H2$ or helicity for an
unconfined chain dissolves away by $50\tau$ whereas the helicity ($H2$) for a
(b) chain under confinement shows that $H2$ fluctuates about a high value even
at long times. Thus geometric confinement leads to stable helices due to
electrostatic interactions.

How does geometric confinement lead to the formation of stable helices? As
mentioned before, the transient helices dissolves as the pitch of the helix
increases and the polymer chain stretches out axially. This is due to the
predominantly repulsive, electrostatic interactions between monomers and the
bending energy cost due to semiflexibility. However, if the polymer is placed
in a cuboidal box with length $L_{c}=1.1R_{c}$ ($R_{c}$ is the initial contour
length of the uncharged chain), the confinement prevents the stretching of the
polymer chain axially and thus stops the unwinding process. This leads to the
formation of long lived stable helices. Thus a geometric confinement with the
axial dimension $L_{c}\sim R_{c}$ successfully leads to long lived helical
structures.

Figure 7:  Aspect ratios of cuboidal confinement conducive to helix formation:
To examine the ranges of $W_{c}$ (width of the confining cuboid) and $L_{c}$
(length of the confining cuboid) for which we obtain stable helix formation.
We switch on electrostatic interactions due to charges at $10\tau$ and measure
$\langle H2\rangle$ over the next $190\tau$. The other parameters are
identical to that of Fig1. The value of $R_{c}$ is $49a$ where $a$ is the unit
of length in our simulations.

This again leads to questions about the range of aspect ratios that are
conducive to helix formation. To investigate this, we systematically vary the
width $W_{c}$ and length of the cuboid $L_{c}$ of the cuboid with a square
cross-section. We switch on charges at $t=10\tau$ and compute $\langle
H2\rangle$ over the next $190\tau$. We plot a colour map showing the range of
$W_{c}$ and $L_{c}$ conducive to stable helix formation, refer Fig.7. We see
that we obtain helix formation only when $L_{c}\sim R_{c}$ and also requires
intermediate values of $W_{c}$. When the values of $W_{c}$ are small the
monomers are unable to move significantly from the axis, thus leading to a
suppression of helix formation. For large values of $W_{c}$ the polymer may
stretch out diagonally as a response to repulsive forces. Thus one may attain
long-lived, helical structures due to electrostatic interactions between
charged monomers in a polymer under confinement with the aspect ratio chosen
suitably.

##  IV Discussions

Through this study, we have established a generic mechanism by which the
helical motif may spontaneously emerge in several bio-molecules. We posit that
such emergent structures may arise due to the induction of charges along the
bio-polymer chain contour due to local changes in solvent conditions. However,
these structures have to rely on other mechanisms to remain long-lived. Free-
standing helices formed due to the switching on of screened Coulomb
interactions are transient. In the presence of confinement, these helices
become long-lived. In our study, we often use large Debye lengths ($\approx
10a$). Such large values of the Debye length are unlikely to arise in natural
polyelectrolytes. However, in this study we show that the mechanism of helix
formation is robust even for such high values of the Debye lengths, which may
be realized in experiments with highly deionized solvents.

The helices we observe in our simulations do not have a preferential
handedness and may have an equal number of left-handed and right-handed
segments. For most, bio-molecules however, there is a preferred handedness
when the molecules attain helical conformations. Future investigations may
shed light on this aspect and elucidate how a preferred handedness emerges.
This will require incorporating specific constraints beyond the mechanism
outlined here.

##  V Acknowledgements

A.C., with DST-SERB identification SQUID-1973-AC-4067, acknowledges funding by
DST-India, project MTR/2019/000078 and CRG/2021/007824. A.C also acknowledges
discussions in meetings organized by ICTS, Bangalore.

## References

  * [1] Nancy Kleckner, Jay K Fisher, Mathieu Stouf, Martin A White, David Bates, and Guillaume Witz.  The bacterial nucleoid: nature, dynamics and sister segregation.  Current Opinion in Microbiology, 22:127–137, 2014.  Growth and development: eukaryotes/ prokaryotes.
  * [2] Roberto Dominguez and Kenneth C. Holmes.  Actin structure and function.  Annual Review of Biophysics, 40(1):169–186, June 2011.
  * [3] Matthew D. Shoulders and Ronald T. Raines.  Collagen structure and stability.  Annual Review of Biochemistry, 78(1):929–958, June 2009.
  * [4] Jayanth R. Banavar, Trinh X. Hoang, Amos Maritan, Flavio Seno, and Antonio Trovato.  Unified perspective on proteins: A physics approach.  Physical Review E, 70(4), October 2004.
  * [5] Jay K. Fisher, Aude Bourniquel, Guillaume Witz, Beth Weiner, Mara Prentiss, and Nancy Kleckner.  Four-dimensional imaging of e. coli nucleoid organization and dynamics in living cells.  Cell, 153(4):882–895, May 2013.
  * [6] Amos Maritan, Cristian Micheletti, Antonio Trovato, and Jayanth R. Banavar.  Optimal shapes of compact strings.  Nature, 406(6793):287–290, July 2000.
  * [7] B. Pokroy, S. H. Kang, L. Mahadevan, and J. Aizenberg.  Self-organization of a mesoscale bristle into ordered, hierarchical helical assemblies.  Science, 323(5911):237–240, January 2009.
  * [8] Sid Ahmed Sabeur, Fatima Hamdache, and Friederike Schmid.  Kinetically driven helix formation during the homopolymer collapse process.  Physical Review E, 77(2), February 2008.
  * [9] Huaping Li and Alkan Kabakçıoğlu.  Role of helicity in DNA hairpin folding dynamics.  Physical Review Letters, 121(13), September 2018.
  * [10] Daniel A. Vega, Andrey Milchev, Friederike Schmid, and Mariano Febbo.  Anomalous slowdown of polymer detachment dynamics on carbon nanotubes.  Physical Review Letters, 122(21), May 2019.
  * [11] Y. Snir.  Entropically driven helix formation.  Science, 307(5712):1067–1067, February 2005.
  * [12] S. J. Gerbode, J. R. Puzey, A. G. McCormick, and L. Mahadevan.  How the cucumber tendril coils and overwinds.  Science, 337(6098):1087–1091, August 2012.
  * [13] Y. Forterre and J. Dumais.  Generating helices in nature.  Science, 333(6050):1715–1716, September 2011.
  * [14] Bipul Biswas, Debarshi Mitra, Fayis KP, Suresh Bhat, Apratim Chatterji, and Guruswamy Kumaraswamy.  Rigidity dictates spontaneous helix formation of thermoresponsive colloidal chains in poor solvent.  ACS Nano, 15(12):19702–19711, December 2021.
  * [15] Debasish Chaudhuri and Bela M. Mulder.  Spontaneous helicity of a polymer with side loops confined to a cylinder.  Physical Review Letters, 108(26), June 2012.
  * [16] S. Srivastava, A. Santos, K. Critchley, K.-S. Kim, P. Podsiadlo, K. Sun, J. Lee, C. Xu, G. D. Lilly, S. C. Glotzer, and N. A. Kotov.  Light-controlled self-assembly of semiconductor nanoparticles into twisted ribbons.  Science, 327(5971):1355–1359, February 2010.
  * [17] Tamaki Nakano and Yoshio Okamoto.  Synthetic helical polymers: conformation and function.  Chemical Reviews, 101(12):4013–4038, November 2001.
  * [18] Sheng Zhong, Honggang Cui, Zhiyun Chen, Karen L. Wooley, and Darrin J. Pochan.  Helix self-assembly through the coiling of cylindrical micelles.  Soft Matter, 4(1):90–93, 2008.
  * [19] Lingxiang Jiang, Julius W. J. de Folter, Jianbin Huang, Albert P. Philipse, Willem K. Kegel, and Andrei V. Petukhov.  Helical colloidal sphere structures through thermo-reversible co-assembly with molecular microtubes.  Angewandte Chemie, 125(12):3448–3452, February 2013.
  * [20] G. Singh, H. Chan, A. Baskin, E. Gelman, N. Repnin, P. Kral, and R. Klajn.  Self-assembly of magnetite nanocubes into helical superstructures.  Science, 345(6201):1149–1153, July 2014.
  * [21] Q. Guo, A. K. Mehta, M. A. Grover, W. Chen, D. G. Lynn, and Z. Chen.  Shape selection and multi-stability in helical ribbons.  Applied Physics Letters, 104(21):211901, May 2014.
  * [22] Yunlong Zhou, Ryan L. Marson, Greg van Anders, Jian Zhu, Guanxiang Ma, Peter Ercius, Kai Sun, Bongjun Yeom, Sharon C. Glotzer, and Nicholas A. Kotov.  Biomimetic hierarchical assembly of helical supraparticles from chiral nanoparticles.  ACS Nano, 10(3):3248–3256, March 2016.
  * [23] Xiang Yang Kong and Zhong Lin Wang.  Spontaneous polarization-induced nanohelixes, nanosprings, and nanorings of piezoelectric nanobelts.  Nano Letters, 3(12):1625–1631, December 2003.
  * [24] Kevin Stratford, Oliver Henrich, Juho S Lintuvuori, ME Cates, and Davide Marenduzzo.  Self-assembly of colloid-cholesteric composites provides a possible route to switchable optical materials.  Nature communications, 5(1):1–8, 2014.
  * [25] Qun li Lei, Ran Ni, and Yu qiang Ma.  Self-assembled chiral photonic crystals from a colloidal helix racemate.  ACS Nano, 12(7):6860–6870, June 2018.
  * [26] Wenchun Feng, Ji-Young Kim, Xinzhi Wang, Heather A Calcaterra, Zhibei Qu, Louisa Meshi, and Nicholas A Kotov.  Assembly of mesoscale helices with near-unity enantiomeric excess and light-matter interactions for chiral semiconductors.  Science Advances, 3(3):e1601159, 2017.
  * [27] Yong Wang, Jun Xu, Yawen Wang, and Hongyu Chen.  Emerging chirality in nanoscience.  Chem. Soc. Rev., 42(7):2930–2962, 2013.
  * [28] W. Wang, K. Yang, J. Gaillard, P. R. Bandaru, and A. M. Rao.  Rational synthesis of helically coiled carbon nanowires and nanotubes through the use of tin and indium catalysts.  Advanced Materials, 20(1):179–182, January 2008.
  * [29] Hai-Feng Zhang, Chong-Min Wang, and Lai-Sheng Wang.  Helical crystalline SiC/SiO2core-shell nanowires.  Nano Letters, 2(9):941–944, September 2002.
  * [30] Hai-Feng Zhang, Chong-Min Wang, Edgar C. Buck, and Lai-Sheng Wang.  Synthesis, characterization, and manipulation of helical SiO2nanosprings.  Nano Letters, 3(5):577–580, May 2003.
  * [31] Lichun Liu, Sang-Hoon Yoo, Sang A. Lee, and Sungho Park.  Wet-chemical synthesis of palladium nanosprings.  Nano Letters, 11(9):3979–3982, September 2011.
  * [32] D. C. Rapaport.  Molecular dynamics simulation of polymer helix formation using rigid-link methods.  Physical Review E, 66(1), July 2002.
  * [33] Tatjana Škrbić, Jayanth R. Banavar, and Achille Giacometti.  Chain stiffness bridges conventional polymer and bio-molecular phases.  The Journal of Chemical Physics, 151(17):174901, November 2019.
  * [34] Debarshi Mitra and Apratim Chatterji.  Transient helix formation in charged semiflexible polymers without confinement effects.  Journal of Physics: Condensed Matter, 33(4):044001, October 2020\.
  * [35] Bipul Biswas, Raj Kumar Manna, Abhrajit Laskar, P. B. Sunil Kumar, Ronojoy Adhikari, and Guruswamy Kumaraswamy.  Linking catalyst-coated isotropic colloids into “active” flexible chains enhances their diffusivity.  ACS Nano, 11(10):10025–10031, September 2017.
  * [36] Andrey V. Dobrynin.  Electrostatic persistence length of semiflexible and flexible polyelectrolytes.  Macromolecules, 38(22):9304–9314, October 2005.


# COVID-19 Pneumonia Severity Prediction using
Hybrid Convolution-Attention Neural Architectures

Nam Nguyen, and J. Morris Chang M. The authors are with the Department of
Electrical Engineering, University of South Florida, Tampa, FL 33620.
E-mail<EMAIL_ADDRESS>
###### Abstract

This study proposed a novel framework for COVID-19 severity prediction, which
is a combination of data-centric and model-centric approaches. First, we
propose a data-centric pre-training for extremely scare data scenarios of the
investigating dataset. Second, we propose two hybrid convolution-attention
neural architectures that leverage the self-attention from the Transformer and
the Dense Associative Memory (Modern Hopfield Network). Our proposed approach
achieves significant improvement from the conventional baseline approach. The
best model from our proposed approach achieves $R^{2}=0.85\pm 0.05$ and
Pearson correlation coefficient $\rho=0.92\pm 0.02$ in geographic extend and
$R^{2}=0.72\pm 0.09,\rho=0.85\pm 0.06$ in opacity prediction.

###### Index Terms:

COVID-19 severity prediction, Attention Learning, Hybrid Convolution-Attention
Neural Architecture.

##  1 Introduction

The coronavirus disease 2019 (COVID-19) was declared a global pandemic by the
World Health Organization in early 2020. There are $184$ million cases with
approximately $4$ million deaths recorded up to July 2021 [1]. Early detection
not only ameliorates the survival rate of COVID-19 patients but also prevents
the spread of diseases. Moreover, severity prediction significantly impacts
the resource allocation in hospitals [2, 3, 4], which is crucial during the
pandemic. Many studies [5, 6, 7, 8] shows the high correlation between
severity progression of COVID-19 and the length of hospital stay, ICU
admission, which is fruitful for optimal planning of follow-up medical care.

Computer-aided diagnosis based on machine learning and deep learning has
become potential solutions for COVID-19 detection [9] and severity prediction
[10, 11, 12]. The dominant solution for COVID-19 prediction is delivered
through transfer learning, in which databases are abundant and adequate for
the good-performed model. In contrast, severity prediction copes with
extremely small cohorts, where the number of samples is inadequate to deliver
well-calibrated deep learning solutions. Similar works that tackle COVID-19
severity prediction in literature are introduced [10, 11] with existing deep
neural architectures.

In this work, we proposed a novel approach for COVID-19 severity prediction,
which combines data-centric and model-centric improvement. We summarize our
contribution as follows:

  1. 1.

We proposed a data-centric pre-training framework to tackle the extremely
scare data scenario of COVID-19 prediction. Our proposed data-centric pre-
training significantly ameliorate the performance of deep neural architectures
in term of predictive power.

  2. 2.

We proposed two hybrid convolution-attention neural architectures that
leverage the self-attention from state-of-the-art Transformer and Dense
associative memory.

  3. 3.

The experimental results yield a noticeable improvement compared to
conventional counterparts, which includes transfer learning from ImageNet and
existing neural architectures.

The organization of our work is as follows: Section 2 briefly introduces
related works, Section 3 gives a detailed description of our proposed
approach, Section 4 reports our experimental design and results, Section 5
gives the discussion and conclusion of our study.

##  2 Related works

###  2.1 Neural Architecture Design

The design of deep neural architectures can be categorized into two
approaches: (1) manual and (2) automated. In the manual design, we aim to
develop the architecture of neural blocks, which requires considerable expert
knowledge. For example, the residual block is introduced in [13] enables more
convenient optimization with residual connection from the inputs; Inception
blocks enable the approximation of optimal neural spare structure by ”split-
transform-merge” strategy [14]. On the other hand, automated neural
architecture search (NAS) attempts to search the optimal neural architecture
on a given datasets [15, 16, 17]. The dominant approach for neural encoding
for the NAS algorithm is through directed acyclic graphs, which represent
blocks in neural architecture (also known as a cell). These discovered cells
are then stacked to form final neural architectures.

The recent development of manual neural architecture design leverages self-
attention to capture the global contextual information within the input space.
The common choice for the self-attention module is Transformer [18], which was
originally designed for the natural language process. Vision Transformer [19]
(ViT) split the input images into sequences of patches, which are taken as the
input of the Transformer encoder. It is noted that in the ViT architecture,
only the Transformer encoder is used. Detection Transformer leverages full
architecture of Transformer for object detection tasks [20]. The HybridCA [21]
introduced learnable image representation queries for the full Transformer
model, which enhances the capacity of neural architectures. CoAtNets [22]
introduced a family of hybrid models that can improve the generalization,
capacity, and efficiency.

##  3 Methodology

Figure 1: Illustration of our proposed data-centric pre-training framework. We
highlight the difference between transfer learning and data-centric pre-
training in different boxes. Figure 2: Illustration of Hybrid Convolution-
Attention neural architecture.

###  3.1 Data-centric framework for Pre-training Deep Neural Network

The improvements of an AI system can be achieved through two main approaches:
(1) model-centric and (2) data-centric development. In the model-centric
approach, we aim to develop AI algorithms on given datasets, commonly fixed
throughout the process. The main concentration on such an approach is
delivering the optimized models for desired learning tasks, enabling state-of-
the-art neural solutions. The advantage of such an approach is the convenient
comparison between algorithms due to fixed collections of pre-defined
datasets. However, there are several issues associated with the model-centric
approach. The performance of the model-centric approach highly depends on data
scenarios, which in some cases are intrinsically challenging. For example,
inadequate training samples in scare data scenarios potentially lead to poor-
performed models and non-robust inferences.

In the data-centric approach, we leverage additional data for improving the
performance of the AI system. The fundamental assumption of such an approach
is simple but practical, which is supposed that the performance of ML/DL
algorithms can be ameliorated with more relevant data samples. Transfer
learning can be considered an example of the data-centric approach, which
transfers knowledge from a huge domain dataset to a smaller target dataset.
This approach assists in learning low-level representations from domain sets,
enabling more efficient learning on target sets. However, the inherited
limitation of transfer learning is negative transfer, in which domain and
target datasets are irrelevant.

Directly addressing these issues, we proposed a data-centric framework for
pre-training deep neural architecture, which is illustrated in Figure 1. In
the pre-training phase, proxy datasets curation is required, in which proxy
datasets need to be highly similar but strictly separated from the datasets of
interest. The size of proxy data can be smaller or larger than the target
datasets depends on desired purposed. Take automated neural architecture
search (NAS) as an example, where we aim to discover the best neural solutions
for a given dataset. Early works of NAS [15, 16] search and evaluate on
ImageNet [23] with $~{}14$ million samples, which leads to extensively
searching time of $2250-3000$ GPU-days. Following works show that CIFAR-10
[24] ($60k$ samples) is a good proxy data for ImageNet [16], reducing the
search time to only $1-4$ GPU days [17]. In our used case, we aim to develop a
high-performance model for an extremely minimal dataset. Thus, the desired
proxy dataset needs to include more samples while maintaining the similarity
to the dataset-of-interest. Moreover, pre-training tasks enable learning good
representations, which ensures similarity to dataset-of-interest. The pre-
training tasks depend on the availability of the proxy data. In a labeled
proxy, we can adopt supervised learning tasks such as classification or object
detection for models to learn data representations. Regarding unlabeled proxy
datasets, unsupervised and self-supervised learning [25, 26] can be used to
pre-train deep neural networks.

The main objective of the data-centric pre-training phase is to help neural
architecture learning good representations, which can lead to significant
improvements in the downstream tasks. Good representations are expensive,
which reasonable-sized representations can capture the abstraction from a vast
number of inputs and mitigate the variance [27]. The design of data-centric
pre-training for COVID-19 severity prediction will be given in Section 4.

###  3.2 Hybrid Convolution-Attention Neural Architecture

####  3.2.1 Architecture

We generalized the hybrid convolution-attention (HybridCA) neural architecture
in [21], which include two main components: (1) convolution backbone module
and (2) self-attention module (Figure 2). First, the backbone convolution
transforms input features into intermediate feature maps. These image
representations is then projected and vectorized inter-channel to from a
collection of entities $\\{\bm{x_{1},x_{2},...,x_{n}}\\}$, in which each
$\bm{x_{i}}$ is $p-$dimensional vector in the latent space. These embedded
vectors are taken as inputs of self-attention modules to extract the global
contextual information and relationship amongst entities. Since the attention
module possesses the permutation-invariant property, we apply the fixed
positional encoding before the self-attention encoder. These encoded entities
are then fed-forward into self-attention decoder and same size learnable image
representation queries (IRQ), which can be considered learnable parameters of
the architecture. The final prediction of HybridCA architecture is delivered
by a multi-layer perceptron, customized to the desired learning tasks.

The original architecture of HybridCA only considers the entire Transformer
architecture as the self-attention module. In this work, we extend the study
by investigating the effectiveness of an additional self-attention model, a
dense associative memory.

####  3.2.2 Transformer Model

The Transformer is an encoder-decoder neural architecture, which contains a
stack of encoder layers followed by decoder layers. In the Transformer’s
encoder, the core component is a multi-head self-attention block followed by a
sub-sequence element-wise feed-forward network. Moreover, residual connections
can be established within the encoder together with layer-wise normalization.
The Transformer’s decoder is similar to the encoder, except required multi-
head attention for encoded representation entities. The building block for the
Transformer model is multi-head attention, which is formed by the self-
attention mechanism.

Self-attention: Given a set of image representation entities
$\\{\bm{x_{1},x_{2},...,x_{n}}\\}$, we denote

$\bm{X}_{n\times p}=\begin{bmatrix}|&|&\dots&|\\\
\bm{x_{1}}&\bm{x_{2}}&\dots&\bm{x_{n}}\\\ |&|&\dots&|\\\ \end{bmatrix}.$ (1)

Th self-attention attempt to learn the relationship amongst input entities,
producing encoded representations which captures the global contextual
information from the entities. Such task requires learning three weight
matrices: (1) Queries matrix $\bm{W}_{Q}^{n\times d_{q}}$, (2) Keys matrix
$\bm{W}_{K}^{n\times d_{k}}$ and (3) Values matrix $\bm{W}_{V}^{n\times
d_{v}}$. The collection of input entities $\bm{X}$ is projected onto the three
learnable matrix as following

$\begin{split}\bm{Q}&=\bm{XW}_{Q}\\\ \bm{K}&=\bm{XW}_{K}\\\
\bm{V}&=\bm{XW}_{V}\end{split}$ (2)

The encoded representations $\bm{Z}^{n\times d_{v}}$ is computed as

$\bm{Z}=\text{softmax}\bigg{(}\frac{\bm{Q}\bm{K}^{T}}{\sqrt{d_{q}}}\bm{V}\bigg{)},$
(3)

where $1/\sqrt{d_{p}}$ is temperature of the dot product in the softmax
function, preventing extremely small gradients [18]. As a result, each element
of encoded representation matrix $\bm{Z}$ is the weighted sum of all original
entities in the latent space, in which weight matrix is computed by the dot-
product of queries and all keys.

Multi-head Attention Given $B$ blocks of self-attention, multi-head attention
can be formed by simultaneously computing multiple individual self-attention.
We denote $\\{\bm{W}_{Q}^{(i)},\bm{W}_{K}^{(i)},\bm{W}_{V}^{(i)}\\}$ for
$i=1,2,\dots B$ for each self-attention head and $\bm{Z}^{(i)}$ for each
corresponding computed encoded entities. Output of multi-head attention is
formed by the projection of concatenation of all elements $\bm{Z}^{(i)}$ onto
$\bm{W}^{Bd_{v}\times d}$. Hence, multi-head attention’s outputs capture
multiple complex interactions form projected convolution feature maps in
parallel, which provide a larger receptive field.

####  3.2.3 Dense associative memory

The Dense Associated Memory (or Modern Hopfield Networks) is introduced in
[28], which extends to continuous-valued patterns and states. The new energy
function introduced in [29] enables an exponential number of stored patterns
with exponentially small retrieval errors, which is given as

$E=-\text{lse}(\beta,\bm{X}^{T}\bm{p})+\frac{1}{2}\bm{p}^{T}\bm{p}+\beta^{-1}\log
n+\frac{1}{2}M^{2},$ (4)

where $\text{lse}(.)$ is the log-sum function, $\beta$ is the temperature, $p$
is state pattern, $n$ is the number of stored representations and $M$ is the
largest norm among all stored representations. The update rule of such network
by using the Concave-Convex-Procedure yields

$\bm{p}_{\text{new}}=\bm{X}\text{softmax}(\beta\bm{X}^{T}\bm{p})$ (5)

This proposed update rule is equivalent to the self-attention used in the
Transformer model, which enables attention learning from the input data.
Moreover, the new energy function with associated update rule ensure the
convergence to local minimum of the energy function, leading to fast
convergence.

##  4 Experiments

Model | DenseNet121 | ResNet50 | EfficientNet-B1 | EfficientNet-B2 | EfficientNet-B3 | EfficientNet-B4
---|---|---|---|---|---|---
Atelectasis | $0.7586$ | $0.6933$ | $0.7558$ | $0.7614$ | $0.7551$ | $0.8015$
Cardiomegaly | $0.8482$ | $0.8001$ | $0.8526$ | $0.8406$ | $0.8365$ | $0.7578$
Effusion | $0.8109$ | $0.7606$ | $0.8157$ | $0.8176$ | $0.8179$ | $0.8889$
Infiltration | $0.6943$ | $0.6316$ | $0.6988$ | $0.6965$ | $0.6917$ | $0.7028$
Mass | $0.7637$ | $0.7273$ | $0.7573$ | $0.7536$ | $0.7589$ | $0.7796$
Nodule | $0.7081$ | $0.6554$ | $0.7149$ | $0.7106$ | $0.7196$ | $0.7324$
Pneumonia | $0.7030$ | $0.6488$ | $0.6902$ | $0.6948$ | $0.6987$ | $0.7313$
Pneumothorax | $0.8412$ | $0.8037$ | $0.8383$ | $0.8455$ | $0.8435$ | $0.8206$
Consolidation | $0.7282$ | $0.6912$ | $0.7171$ | $0.7272$ | $0.7235$ | $0.7998$
Edema | $0.8363$ | $0.8087$ | $0.8382$ | $0.8316$ | $0.8314$ | $0.8868$
Emphysema | $0.8680$ | $0.8456$ | $0.8319$ | $0.8737$ | $0.8736$ | $0.8330$
Fibrosis | $0.7689$ | $0.7334$ | $0.7839$ | $0.7714$ | $0.7729$ | $0.7667$
Pleural Thickening | $0.7460$ | $0.7098$ | $0.7349$ | $0.7381$ | $0.7446$ | $0.7781$
Hernia | $0.8097$ | $0.7331$ | $0.8225$ | $0.7972$ | $0.7829$ | $0.7444$
COVID-19 | $0.9992$ | $0.9993$ | $0.9998$ | $0.9998$ | $0.9998$ | $0.9994$
Mean AUC | $0.7923$ | $0.7495$ | $0.7901$ | $0.7907$ | $0.7900$ | $0.8018$
TABLE I: Experimental results of pre-training phase. Details of training
setting is given in Section 4.1.1

###  4.1 Experimental Designs

Figure 3: Experimental results of pre-training phase. The same experiment
setting is used across all training sessions, which is reported in Section.
Figure 4: Effects of data-centric pre-training on training progress of
different models. We report the cross-valiation loss computed by Equation 6.
The results from non-pretrain (transfer learning) is depicted in yellow.

####  4.1.1 Design of Data-centric pre-training task

In the pre-training phase, we collect two databases for the proxy dataset of
COVID-19 severity prediction: (1) NIH Chest X-ray Dataset [30], which include
$112,120$ Non-COVID-19 X-ray images with $14$ disease labels from $30,805$
patients and (2) COVID-19 database includes $3,671$ images collected from
various resources [31, 32]. It is noted that these databases are completely
separated from the dataset used for severity prediction.

The pre-training task for the proxy data is multi-label classification. The
target vector (label) for each input instance is a $15$-dimensional vector
($14$ types of disease plus COVID-19 class) with binary entries, representing
the presence of related diseases. In other words, the zero vector
$y=[0,...,0]$ represents normal case, while an unit entry at location $C$
represents the appearance of $C^{th}$ disease. This label encoding guarantees
that the model cannot infer normal and disease classes concurrently and that
learning tasks are considered a regression-like problem.

Within the scope of this study, we investigate five backbone convolution
neural networks with different model complexity, which are DenseNet121 [33],
ResNet50 [13], EfficientNet-B1 to B5 [34]. The loss function for the pre-
training task is binary cross-entropy loss. To optimizing model parameters, we
use AdamW optimizer with an initial learning rate of $10^{-6}$ and weight
decay $0.01$. We pre-train backbone CNN with initial ImageNet weights for
$100$ epochs. We discuss the experimental results of the pre-training phase in
Section 4.

Model | Pre-trained | Attention Module | MAE | MSE | Geographic Extend | Opacity
---|---|---|---|---|---|---
$R^{2}$ | $\rho$ | $R^{2}$ | $\rho$
DenseNet121 | - | - | $1.28\pm 0.23$ | $2.68\pm 0.75$ | $0.39\pm 0.13$ | $0.62\pm 0.1$ | $0.32\pm 0.18$ | $0.51\pm 0.23$
$\checkmark$ | - | $0.94\pm 0.15$ | $1.39\pm 0.42$ | $0.74\pm 0.04$ | $0.86\pm 0.02$ | $0.55\pm 0.05$ | $0.74\pm 0.03$
$\checkmark$ | Transformer | $0.74\pm 0.07$ | $0.93\pm 0.19$ | $0.82\pm 0.06$ | $0.91\pm 0.04$ | $0.73\pm 0.1$ | $0.85\pm 0.06$
$\checkmark$ | Hopfield | $0.73\pm 0.13$ | $0.92\pm 0.29$ | $0.81\pm 0.05$ | $0.9\pm 0.03$ | $0.71\pm 0.04$ | $0.84\pm 0.03$
ResNet50 | - | - | $4.51\pm 1.11$ | $30.75\pm 12.73$ | $0.08\pm 0.05$ | $0.1\pm 0.26$ | $0.09\pm 0.04$ | $0.11\pm 0.27$
$\checkmark$ | - | $0.96\pm 0.28$ | $3.01\pm 3.65$ | $0.63\pm 0.25$ | $0.77\pm 0.19$ | $0.54\pm 0.23$ | $0.72\pm 0.17$
$\checkmark$ | Transformer | $0.79\pm 0.05$ | $0.98\pm 0.12$ | $0.83\pm 0.06$ | $0.91\pm 0.03$ | $0.64\pm 0.13$ | $0.8\pm 0.08$
$\checkmark$ | Hopfield | $0.77\pm 0.09$ | $0.96\pm 0.16$ | $0.82\pm 0.06$ | $0.91\pm 0.03$ | $0.62\pm 0.12$ | $0.78\pm 0.08$
EfficientNet-B1 | - | - | $1.41\pm 0.27$ | $3.52\pm 1.1$ | $0.29\pm 0.11$ | $0.52\pm 0.11$ | $0.26\pm 0.08$ | $0.51\pm 0.08$
$\checkmark$ | - | $0.93\pm 0.09$ | $1.39\pm 0.28$ | $0.69\pm 0.11$ | $0.83\pm 0.07$ | $0.63\pm 0.14$ | $0.79\pm 0.08$
$\checkmark$ | Transformer | $0.75\pm 0.05$ | $0.93\pm 0.07$ | $0.79\pm 0.06$ | $0.89\pm 0.04$ | $0.71\pm 0.09$ | $0.84\pm 0.06$
$\checkmark$ | Hopfield | $0.78\pm 0.11$ | $0.98\pm 0.24$ | $0.8\pm 0.05$ | $0.89\pm 0.03$ | $0.69\pm 0.09$ | $0.82\pm 0.05$
EfficientNet-B2 | - | - | $1.46\pm 0.13$ | $3.37\pm 0.65$ | $0.27\pm 0.11$ | $0.5\pm 0.11$ | $0.23\pm 0.1$ | $0.46\pm 0.1$
$\checkmark$ | - | $1.05\pm 0.13$ | $1.72\pm 0.38$ | $0.65\pm 0.07$ | $0.81\pm 0.04$ | $0.51\pm 0.14$ | $0.71\pm 0.1$
$\checkmark$ | Transformer | $0.8\pm 0.12$ | $1.03\pm 0.17$ | $0.8\pm 0.05$ | $0.89\pm 0.03$ | $0.64\pm 0.12$ | $0.8\pm 0.07$
$\checkmark$ | Hopfield | $0.78\pm 0.09$ | $0.99\pm 0.18$ | $0.81\pm 0.06$ | $0.9\pm 0.03$ | $0.64\pm 0.1$ | $0.8\pm 0.06$
EfficientNet-B3 | - | - | $1.59\pm 0.27$ | $4.28\pm 1.26$ | $0.11\pm 0.09$ | $0.31\pm 0.12$ | $0.18\pm 0.14$ | $0.37\pm 0.2$
$\checkmark$ | - | $1.15\pm 0.06$ | $2.21\pm 0.45$ | $0.58\pm 0.12$ | $0.76\pm 0.08$ | $0.53\pm 0.11$ | $0.73\pm 0.09$
$\checkmark$ | Transformer | $0.68\pm 0.02$ | $0.76\pm 0.07$ | $0.85\pm 0.05$ | $0.92\pm 0.02$ | $0.72\pm 0.09$ | $0.85\pm 0.06$
$\checkmark$ | Hopfield | $0.74\pm 0.1$ | $0.93\pm 0.14$ | $0.82\pm 0.05$ | $0.9\pm 0.03$ | $0.7\pm 0.11$ | $0.84\pm 0.06$
EfficientNet-B4 | - | - | $1.96\pm 0.36$ | $6.29\pm 1.87$ | $0.13\pm 0.07$ | $0.34\pm 0.1$ | $0.15\pm 0.06$ | $0.38\pm 0.09$
$\checkmark$ | - | $1.08\pm 0.21$ | $1.77\pm 0.62$ | $0.62\pm 0.11$ | $0.79\pm 0.07$ | $0.5\pm 0.1$ | $0.7\pm 0.07$
$\checkmark$ | Transformer | $0.81\pm 0.14$ | $1.04\pm 0.33$ | $0.81\pm 0.08$ | $0.9\pm 0.04$ | $0.63\pm 0.09$ | $0.79\pm 0.05$
$\checkmark$ | Hopfield | $0.81\pm 0.09$ | $1.18\pm 0.2$ | $0.75\pm 0.06$ | $0.86\pm 0.04$ | $0.63\pm 0.07$ | $0.8\pm 0.04$
TABLE II: Experimental results of COVID-19 severity prediction. Each block
report the performance of individual backbone CNNs under four experimental
setting. The global evaluation metrics are MAE and MSE, while evaluation
metrics for individual attributes are $R^{2}$ and Pearson correlation
coefficient $\rho$. The best results withing each box are shaded in gray. The
same training setting is used across all experiments, which is reported in
Section 4.1.2

####  4.1.2 COVID-19 severity prediction

The COVID-19 dataset for severity prediction is from [10], which is completely
separated from the proxy dataset. The database includes $94$ posteroanterior
(PA) chest X-ray images. All patients in the cohort have been reported
positive to COVID-19 from December 2019 to March 2020. The labels of the
database are based on radiological scoring, which involved three blinded
experts. Two chest radiologist (with 20 years of experience) and a radiology
resident score the COVID-19 severity based on [35], which includes extent of
lung involvement (geographic extend) and degree of opacity (opacity).

The COVID-19 severity prediction dataset can be considered an extremely small
dataset, so we decided to perform 5-fold cross-validation to evaluate
competitors’ performance. First, we split the dataset into five independent
folds, which guarantee no overlapped patient across folds. The evaluation
metrics for severity prediction are: (1) mean squared error (MSE), (2) mean
absolute error (MAE), (3) R-squared, and Pearson correlation between actual
and predicted scores.

The loss function used in this phase is smoothed L1 loss with $\beta=1$, which
is given by

$\mathcal{L}(\bm{y,\hat{y}})=\begin{cases}0.5(\bm{y}-\bm{\hat{y}})^{2}/\beta\\\
|\bm{y}-\bm{\hat{y}}|-0.5*\beta\end{cases}$ (6)

We train the model on each fold for $400$ epochs with the SGD optimizer with
an initial learning rate of $10^{-3}$, momentum $0.9$, and weight decay
$3\times 10^{-5}$. In order to prevent the over-fitting problem, we reduce the
learning rate with a decay rate of $0.98$ for every $2$ epoch, and the dropout
rate for the self-attention module is set at $0.1$.

###  4.2 Pre-training Results

Figure 3 depicts the performances of five investigating backbone CNNs, in
terms of multi-label classification. The mean area under curve (AUC) scores of
Densenet121, EfficientNet-B1, B2, and B3 are nearly the same, while
EfficientNet-B4 achieve a slightly higher AUC of $0.8018$ on $15$ classes. On
the other hand, ResNet50 achieves the least AUC score of $0.7495$, although it
has the largest model complexity.

Table I reports the AUC score for each class from five CNNs models. As we can
see, the AUC score of the COVID-19 class is very close to perfect prediction.
However, the phenomenon is potentially attributed to the cross-domain design
of proxy dataset, where COVID-19 images are from entirely different
institutions [36]. Thus, we do not attempt to compare the results of the pre-
training phase to other works or emphasize the COVID-19 detection ability.
Instead, the main objective of the pre-training phase is assisting backbone
models to learn valuable representations for the downstream task, which is
severity prediction. We will investigate the effectiveness of such knowledge
transfer and expansion here in Section 4.

###  4.3 Severity Prediction

####  4.3.1 Effects of Data-centric pre-training

We report the main results of the COVID-19 severity prediction task in Table
II, which contains six blocks corresponding to the choices of backbone CNN.

In the first line of each block, we report the performance of stand-alone CNNs
under transfer learning setup, in which model’s weights are inherited from
ImageNet. The outcomes are consistent from all backbone CNNs, showing that
transfer learning is ineffective in the case of extremely small and irrelevant
target datasets. The performance on the test set of higher complexity such as
ResNet50 and EfficientNet-B4 is lower than small neural networks even though
over-fitted on the training set. Moreover, from the diagnosis of learning
curves (not shown here), these models stop gaining test accuracy after half of
the training process, even being applied over-fitting prevention such as
adaptive learning rate or dropout.

In the second line, we train stand-alone backbone CNNs with weights from our
data-centric pre-training task. The consistent pattern appears across all
models, yielding improvement noticeably in comparison to transfer learning
from ImageNet. First, the test accuracy is improved significantly, which can
be observed through test MAE and MSE. Moreover, the $R^{2}$ and Pearson
correlation between actual and predicted values increases with a considerably
large gap, indicating a more precise prediction. The most accuracy gain can be
observed from DenseNet121, while deeper CNNs such as ResNet50 gain a minor
improvement. However, the agreement between actual and predicted is not
remarkable, which achieves only $R^{2}$ of $0.74$ and $0.55$ to predict
geographic extend and opacity from DenseNet121.

Figure 4 illustrates the cross-validation loss computed by Equation 6.
Transfer learning fails to achieve good performance in comparison to data-
centric pre-training.

####  4.3.2 Effects of Hybrid Convolution-Attention Architecture

The third and fourth line of Table II shows the performance of proposed hybrid
convolution-attention architectures with different self-attention modules. We
denote HCT for Transformer and HCH for Hopfield network. The initial weights
for these experiments are adopted from the data-centric pre-training phase.
Overall, the MSE of hybrid neural architectures drops approximately $0.2$
points across all backbone models, while MAS drops $1.5$ points on average. We
can see the significant improvement when observing the $R^{2}$ from each
hybrid model. For example, the $R^{2}$ in geographic extend from DenseNet121
increase from $0.74$ to $0.82$, while that in opacity prediction enhances from
$0.55$ to $0.73$. Moreover, Figure 3 depicts that the CV-loss of hybrid
architectures is lower than stand-alone CNNs in general, while the difference
between two self-attention modules is not noticeable. We report the alignment
of DenseNet121, HCT-DenseNet121 and HCH-DenseNet121 in Figure 5. In general,
hybrid architectures achieve better performance across all folds. Moreover,
the alignment from geographic extent is slightly better than the predictions
of opacity.

Figure 5: An exemplar alignment between actual and predicted values from the
experiment on DenseNet121. Each column shows different investigating models,
while each row shows the results on each fold. The solid straight line
indicates perfect prediction, and $\rho$ is the Pearson correlation
coefficient.

##  5 Conclusion

We have presented a novel framework for COVID-19 severity prediction. Our
data-centric pre-training design enables high performance models when
transferring knowledge to the downstream task. Moreover, we introduce new
class of deep neural architecture, which capture the global contextual
information from the input space through self-attention modules. Further
improvement of our work considers different self-attention modules for the
hybrid architecture. Additionally, extending the framework to more
applications is also a potential research direction.

## Acknowledgments

Effort sponsored in part by United States Special Operations Command
(USSOCOM), under Partnership Intermediary Agreement No. H92222-15-3-0001-01.
The U.S. Government is authorized to reproduce and distribute reprints for
Government purposes, notwithstanding any copyright notation thereon. 111The
views and conclusions contained herein are those of the authors and should not
be interpreted as necessarily representing the official policies or
endorsements, either expressed or implied, of the United States Special
Operations Command.

## References

  * [1] “Wikipedia,” 2021.
  * [2] Z. Feng, Q. Yu, S. Yao, L. Luo, W. Zhou, X. Mao, J. Li, J. Duan, Z. Yan, M. Yang _et al._ , “Early prediction of disease progression in covid-19 pneumonia patients with chest ct and clinical characteristics,” _Nature communications_ , vol. 11, no. 1, pp. 1–9, 2020.
  * [3] A. L. Booth, E. Abels, and P. McCaffrey, “Development of a prognostic model for mortality in covid-19 infection using machine learning,” _Modern Pathology_ , vol. 34, no. 3, pp. 522–531, 2021.
  * [4] J. Zhang, Y. Xie, G. Pang, Z. Liao, J. Verjans, W. Li, Z. Sun, J. He, Y. Li, C. Shen _et al._ , “Viral pneumonia screening on chest x-ray images using confidence-aware anomaly detection,” _arXiv preprint arXiv:2003.12338_ , 2020.
  * [5] Y. Fang, H. Zhang, J. Xie, M. Lin, L. Ying, P. Pang, and W. Ji, “Sensitivity of chest ct for covid-19: comparison to rt-pcr,” _Radiology_ , vol. 296, no. 2, pp. E115–E117, 2020.
  * [6] F. Pan, T. Ye, P. Sun, S. Gui, B. Liang, L. Li, D. Zheng, J. Wang, R. L. Hesketh, L. Yang _et al._ , “Time course of lung changes on chest ct during recovery from 2019 novel coronavirus (covid-19) pneumonia,” _Radiology_ , 2020.
  * [7] A. Bernheim, X. Mei, M. Huang, Y. Yang, Z. A. Fayad, N. Zhang, K. Diao, B. Lin, X. Zhu, K. Li _et al._ , “Chest ct findings in coronavirus disease-19 (covid-19): relationship to duration of infection,” _Radiology_ , p. 200463, 2020.
  * [8] F. Liu, Q. Zhang, C. Huang, C. Shi, L. Wang, N. Shi, C. Fang, F. Shan, X. Mei, J. Shi _et al._ , “Ct quantification of pneumonia lesions in early days predicts progression to severe illness in a cohort of covid-19 patients,” _Theranostics_ , vol. 10, no. 12, p. 5613, 2020.
  * [9] H. Panwar, P. Gupta, M. K. Siddiqui, R. Morales-Menendez, and V. Singh, “Application of deep learning for fast detection of covid-19 in x-rays using ncovnet,” _Chaos, Solitons & Fractals_, vol. 138, p. 109944, 2020.
  * [10] J. P. Cohen, L. Dao, K. Roth, P. Morrison, Y. Bengio, A. F. Abbasi, B. Shen, H. K. Mahsa, M. Ghassemi, H. Li _et al._ , “Predicting covid-19 pneumonia severity on chest x-ray with deep learning,” _Cureus_ , vol. 12, no. 7, 2020.
  * [11] N. Lassau, S. Ammari, E. Chouzenoux, H. Gortais, P. Herent, M. Devilder, S. Soliman, O. Meyrignac, M.-P. Talabard, J.-P. Lamarque _et al._ , “Integrating deep learning ct-scan model, biological and clinical variables to predict severity of covid-19 patients,” _Nature communications_ , vol. 12, no. 1, pp. 1–11, 2021.
  * [12] M. Fridadar, R. Amer, O. Gozes, J. Nassar, and H. Greenspan, “Covid-19 in cxr: From detection and severity scoring to patient disease monitoring,” _IEEE journal of biomedical and health informatics_ , 2021.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [14] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [15] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in _Proceedings of the aaai conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 4780–4789.
  * [16] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8697–8710.
  * [17] N. Nguyen and J. M. Chang, “Contrastive self-supervised neural architecture search,” _arXiv preprint arXiv:2102.10557_ , 2021.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [19] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [20] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European Conference on Computer Vision_. Springer, 2020, pp. 213–229.
  * [21] J. M. C. Nam Nguyen, “Attention learning for classification of dermoscopy image,” _arXiv preprint_ , 2021.
  * [22] Z. Dai, H. Liu, Q. V. Le, and M. Tan, “Coatnet: Marrying convolution and attention for all data sizes,” _arXiv preprint arXiv:2106.04803_ , 2021.
  * [23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [24] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [25] I. Misra and L. v. d. Maaten, “Self-supervised learning of pretext-invariant representations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6707–6717.
  * [26] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [27] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 8, pp. 1798–1828, 2013.
  * [28] D. Krotov and J. J. Hopfield, “Dense associative memory for pattern recognition,” _Advances in neural information processing systems_ , vol. 29, pp. 1172–1180, 2016.
  * [29] H. Ramsauer, B. Schäfl, J. Lehner, P. Seidl, M. Widrich, T. Adler, L. Gruber, M. Holzleitner, M. Pavlović, G. K. Sandve _et al._ , “Hopfield networks is all you need,” _arXiv preprint arXiv:2008.02217_ , 2020.
  * [30] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2097–2106.
  * [31] M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al Emadi _et al._ , “Can ai help in screening viral and covid-19 pneumonia?” _IEEE Access_ , vol. 8, pp. 132 665–132 676, 2020.
  * [32] T. Rahman, A. Khandakar, Y. Qiblawey, A. Tahir, S. Kiranyaz, S. B. A. Kashem, M. T. Islam, S. Al Maadeed, S. M. Zughaier, M. S. Khan _et al._ , “Exploring the effect of image enhancement techniques on covid-19 detection using chest x-ray images,” _Computers in biology and medicine_ , vol. 132, p. 104319, 2021.
  * [33] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [34] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6105–6114.
  * [35] H. Y. F. Wong, H. Y. S. Lam, A. H.-T. Fong, S. T. Leung, T. W.-Y. Chin, C. S. Y. Lo, M. M.-S. Lui, J. C. Y. Lee, K. W.-H. Chiu, T. W.-H. Chung _et al._ , “Frequency and distribution of chest radiographic findings in patients positive for covid-19,” _Radiology_ , vol. 296, no. 2, pp. E72–E78, 2020.
  * [36] J. P. Cohen, M. Hashir, R. Brooks, and H. Bertrand, “On the limits of cross-domain generalization in automated x-ray prediction,” in _Medical Imaging with Deep Learning_. PMLR, 2020, pp. 136–155.