††thanks: These two authors contributed equally††thanks: These two authors
contributed equally

# Low-coercive-field ferroelectric hafnia with mobile domain walls

Jing Wu  Key Laboratory for Quantum Materials of Zhejiang Province, Department
of Physics, School of Science, Westlake University, Hangzhou, Zhejiang 310024,
China  Jiyuan Yang  Key Laboratory for Quantum Materials of Zhejiang Province,
Department of Physics, School of Science, Westlake University, Hangzhou,
Zhejiang 310024, China  Yang Sun  Department of Physics, Xiamen University,
Xiamen 361005, China  Shi Liu<EMAIL_ADDRESS>Key Laboratory for
Quantum Materials of Zhejiang Province, Department of Physics, School of
Science, Westlake University, Hangzhou, Zhejiang 310024, China  Institute of
Natural Sciences, Westlake Institute for Advanced Study, Hangzhou, Zhejiang
310024, China

###### Abstract

The high coercive field ($\mathcal{E}_{c}$) of hafnia-based ferroelectrics
presents a major obstacle to their applications. The ferroelectric switching
mechanisms in hafnia that dictate $\mathcal{E}_{c}$, especially those related
to nucleation-and-growth at the domain wall (DW), have remained elusive.
Through deep-learning-assisted multiscale simulations, we determine the
finite-temperature thermodynamics and switching mechanisms for diverse types
of 180∘ DWs, revealing a complex, stress-sensitive mobility landscape. The
propagation velocities for mobile DW types under various thermal conditions
can be characterized with a single creep equation, featuring a creep exponent
of 2. This unconventional critical exponent results from the nucleation of a
half-unit-cell-thin, elliptically-shaped critical nucleus. Our multiscale
approach not only reproduces the experimental thickness ($d$) scaling,
$\mathcal{E}_{c}\propto d^{-\frac{2}{3}}$, but also predicts that
$\mathcal{E}_{c}$ of HfO2 can be engineered to $\approx$0.1 MV/cm, even lower
than perovskite ferroelectrics. The theoretical lower bound of
$\mathcal{E}_{c}$ afforded by ferroelectric hafnia offers opportunities to
realize power-efficient, high-fidelity ferroelectric nanoelectronics.

Ferroelectric hafnia is emerging as a viable candidate for incorporating
ferroelectric capabilities into semiconductor technologies for its nanoscale
ferroelectricity and proven compatibility with silicon [1, 2]. The $Pca2_{1}$
phase of HfO2 is widely acknowledged as the ferroelectric phase in thin films
[3, 4, 5, 6]. The discovery of switchable spontaneous polarization in this
fluorite-structured binary oxide has significantly advanced our understanding
of ferroelectric physics, an area historically focused on perovskite oxides.
For example, recent investigations have uncovered several unique aspects of
hafnia-based ferroelectrics, including its metastable nature compared to the
bulk monoclinic phase [5], the presence of flat phonon bands [7], and
intertwined polar-nonpolar structural polymorphism with oxygen vacancy
diffusion [8, 9].

However, the widespread applications of ferroelectric hafnia-based devices are
mainly hindered by reliability issues, especially their limited endurance
[10]. Unlike devices based on perovskite ferroelectrics like Pb(Zr,Ti)O3 and
BiFeO3, which can be made close to fatigue-free [11], hafnia-based devices,
depending on their architectures, typically exhibit poor endurance, with
switching cycles ranging between $10^{5}$ and $10^{12}$ [12, 13, 14, 15]. This
falls short of the required $>10^{15}$ cycles for embedded random access
memory. Such a detrimental issue is attributed to the high coercive field
($\mathcal{E}_{c}$, the electric field needed to switch the polarization) of
hafnia-based ferroelectrics (typically $>$1 MV/cm in polycrystalline thin
films and $>$2 MV/cm in epitaxial thin films) [16, 17, 18]. Repeated
applications of high electric fields near the breakdown strength to reverse
the polarization are expected to accelerate breakdown, causing rapid
performance deterioration. Reducing $\mathcal{E}_{c}$ without sacrificing
nanoscale ferroelectricity of hafnia has become a pressing challenge for the
development of fast, durable, and energy-efficient nanoelectronics.

The atomic-level switching mechanisms affecting $\mathcal{E}_{c}$ in
ferroelectric hafnia remain elusive, due to the multiple competing pathways
and the complex variety of domain walls (DWs), which are interfaces separating
differently polarized domains. Previous studies using zero-Kelvin density
functional theory (DFT) have focused on polarization switching at the unit-
cell level [3, 19] and the behavior of specific types of DWs [7, 20, 21, 22].
These studies overlooked the thermally activated nucleation-and-growth
process, a crucial aspect for accurate $\mathcal{E}_{c}$ predictions [23].
Here, utilizing a deep neural network-based force field trained exclusively on
first-principles data of hafnia [24, 25], we employ large-scale molecular
dynamics (MD) simulations to investigate the finite-temperature thermodynamics
and kinetics of all types of 180∘ DWs, identified through comprehensive
symmetry analysis. Computational details are provided in Supplementary Sect. I
and an online notebook [26] is developed to share details regarding the
machine-learning force field.

Our MD simulations unveil, without a priori assumptions, a stress-tunable,
spontaneous transformation of DW type at room temperatures, which plays a
critical role in the “divide-and-conquer”-like mechanism underpins the low
switching fields for two types of DWs. We quantify the temperature- and field-
dependence of propagation velocities for these walls in HfO2, revealing a
universal creep exponent of $\mu=2$ and the underlying nucleation-and-growth
mechanisms characterized by a half-unit-cell-thin, elliptically-shaped
critical nucleus. A Landau–Ginzburg–Devonshire (LGD) analytical model
incorporating this insight is developed, reproducing the experimentally
observed thickness ($d$) scaling of $\mathcal{E}_{c}$ in thin films,
$\mathcal{E}_{c}\propto d^{-2/3}$. Importantly, we demonstrate that the
intrinsic $\mathcal{E}_{c}$ of HfO2 determined by mobile DWs is, unexpectedly,
on par with Pb(Ti,Zr)O3, and can be as low as 0.1 MV/cm in thin films.

Classification of 180∘ DWs based on symmetry

We start with a symmetry analysis to classify all possible 180∘ DWs,
interfaces separating domains with opposite polarization. The unit cell of
$Pca2_{1}$ HfO2 has a nonpolar layer of fourfold-coordinated oxygen ions (Onp)
and a polar layer of threefold-coordinated oxygen ions (Op), and these layers
are ordered alternately (Fig. 1a). Application of Euclidean transformations
within the unit cell results in eight different representations (Fig. S7).
This feature, refereed to as conditional chirality [27], leads to a rich
spectrum of 180∘ DWs. We introduce a mode vector
$(T_{x}^{\alpha},A_{z}^{\beta},P_{z}^{\gamma})$ with ($\alpha$, $\beta$,
$\gamma$=$\pm 1$) denoting the sign of the mode amplitude, to label the unit-
cell configuration, where the tetragonal mode ($T_{x}^{\alpha}$), the
antipolar mode ($A_{z}^{\beta}$), and the polar mode ($P_{z}^{\gamma}$) are
the three main modes bridging the cubic phase and polar orthorhombic phase
with polarization aligned along the $z$ axis ($P_{z}$) [28]. The $T_{x}$ and
$A_{z}$ mode are characterized by antiparallel $x$-displacements and
$z$-displacements of neighboring oxygen ions (Fig. 1a), respectively.

A general 180∘ wall can be expressed as
$[T_{x}^{\alpha}A_{z}^{\beta}P_{z}^{-}|T_{x}^{\alpha^{\prime}}A_{z}^{\beta^{\prime}}P_{z}^{+}$],
allowing for 16 configurations (Fig. S8). As illustrated in Fig. 1b, these
configurations can be further classified based on the sign of
$\alpha\cdot\alpha^{\prime}$ and $\beta\cdot\beta^{\prime}$. In cases where
${\rm sgn}(\alpha\cdot\alpha^{\prime})>0$, the $x$-displacement pattern of
oxygen atoms across the DW remains unchanged, whereas ${\rm
sgn}(\alpha\cdot\alpha^{\prime})<0$ indicates a sign reversal of the $T_{x}$
mode. For ${\rm sgn}(\beta\cdot\beta^{\prime})>0$, the oxygen atoms at the
wall are of the same type (either Op or Onp), while a negative sign indicates
Op and Onp atoms are adjacent. Symmetry analysis reveals six unique 180∘ DWs,
which are labeled as $[01|\bar{1}0]$, $[10|01]$, and $[10|\bar{1}0]$, along
with their counterparts having the sign of the $T_{x}$ mode reversed across
the wall, denoted as $[01|\bar{1}0]^{*}$, $[10|01]^{*}$, and
$[10|\bar{1}0]^{*}$. Here, “1”(“${\bar{1}}$”) labels the Op atom displaced
along $+z$ ($-z$), and “0” represents the Onp atom (Fig. 1c). For clarity in
the following discussion, $[01|\bar{1}0]$, $[10|01]$, and $[10|\bar{1}0]$
walls are also named as type-P, type-NP, and type-PNP, based on the type of
oxygen atoms at the wall; their counterparts with reversed $T_{x}$ mode are
type-P∗, type-NP∗, and type-PNP∗.

As shown in Fig. 1d, our DFT calculations of DW energy ($\sigma_{\rm DW}$)
reveal that the type-PNP∗ wall, resembling the antiferroelectric-like $Pbca$
phase, possesses the lowest energy ($\sigma_{\rm PNP^{*}}$), followed by
type-P, type-NP, and type-PNP walls at 0 GPa. In contrast, the other two
$T_{x}$-sign-reversed DWs, type-P∗ and type-NP∗, are highly unstable.
Interestingly, applying a compressive hydrostatic pressure of 2.5 GPa
substantially destabilizes type-P compared to type-NP and type-PNP walls. It
is noted that $\sigma_{\rm PNP^{*}}<0$ is an expected consequence of $Pbca$
HfO2 being lower in energy than $Pca2_{1}$ HfO2. Previously, this intriguing
negative DW energy led to the proposal that each polar layer in $Pca2_{1}$
HfO2 is strongly localized, allowing for individual switching and scale-free
ferroelectricity [7]. Many subsequent studies focused on the motion of a type-
PNP∗ wall [20, 22], despite its substantial energy barrier of $1.3$ eV [21].
We will demonstrate with MD simulations that the polarization switching in
HfO2 does not involve the type-PNP∗ wall as it is immobile.

Spontaneous transformation of DW types at room temperatures

An important discovery from our MD simulations, contrasting with earlier
findings from DFT modeling, is the spontaneous change in DW type at room
temperatures. At 300 K and 0 GPa, the type-NP wall is unstable, automatically
transforming into the type-P wall through Onp$\rightarrow$Op
($\mathbf{0}\rightarrow\mathbf{1}$). This change can be intuitively expressed
as $[10|\mathbf{0}\bar{1}0\bar{1}]\rightarrow[10\mathbf{1}|\bar{1}0\bar{1}]$,
where the wall shifts laterally by half a lattice constant along the $y$-axis,
$b/2$. At a higher pressure of 2.5 GPa, a reverse transition of
type-P$\rightarrow$type-NP occurs spontaneously via
$\mathbf{\bar{1}}\rightarrow\mathbf{0}$, and can be represented as
$[01|\mathbf{\bar{1}}0\bar{1}0]\rightarrow[01\mathbf{0}|0\bar{1}0]$. The high-
energy type-PNP wall remains stable at 300 K under both stress conditions. As
we will elaborate below, this thermally-induced spontaneous transition between
Onp and Op atoms at the interface is pivotal for the dynamics of type-P and
type-NP walls.

We then gauge DW mobility at 300 K by determining the lowest electric field
strength ($\mathcal{E}_{s}$) that triggers DW propagation within 1 nanosecond
in MD simulations, with results summarized in Fig. 1e. Our simulations reveal
that the type-PNP∗ wall is essentially immobile, showing no movement under a
substantial $\mathcal{E}$ of 7 MV/cm. Rather, the adjacent $-P_{z}$ domain
undergoes polarization reversal, since this field strength is sufficient to
drive single-domain (SD) switching within 1 nanosecond. This aligns with the
high DFT-calculated energy barrier of 1.3 eV for moving a type-PNP∗ wall,
compared to a considerably lower barrier of 0.22 eV for homogeneous SD
switching [19]. The resistance of a type-PNP∗ wall to $\mathcal{E}$ stems from
the inability of a $z$-axis field to flip $T_{x}$ mode that features antipolar
$x$-displacements of oxygen atoms. For $T_{x}$-sign-conserved DWs, a wall with
a lower $\sigma_{\rm DW}$ value mobilize more readily under $\mathcal{E}$,
similar to the trend observed in perovskite ferroelectrics [23]. Notably, the
type-P wall at 0 GPa exhibits a minimal $\mathcal{E}_{s}$ of 0.36 MV/cm. At
2.5 GPa, the type-NP wall has the lowest $\mathcal{E}_{s}$ of 0.53 MV/cm. Our
following discussions will focus on the most mobile type-P and type-NP walls.

Mechanisms and kinetics of domain wall motions from MD simulations

Figure 2a and c compare the SD switching to DW motion mechanisms extracted
from finite-field MD simulations at 300 K. At 0 GPa, a type-P wall initially
transforms into a type-NP wall that serves as a bridging state. Conversely, at
2.5 GPa, a type-NP wall progresses via a transient type-P wall. A common
feature is the lateral one-unit-cell translation being accomplished through
two half-unit-cell hops ($b/2$), with the initial hop kinetically determinant.
This is markedly different from perovskite ferroelectrics, where DWs advance
by one unit cell at a time [29, 23]. Such a “first-hop-limiting” mechanism
naturally results from the thermally-induced spontaneous change in DW types.
Specifically, as shown in Fig. 2c, under a downward $\mathcal{E}$, a type-P
wall transitions through $\bar{1}\rightarrow 0$ to a type-NP wall. This type
is thermodynamically unstable at 0.0 GPa, causing a spontaneous hop of
$0\rightarrow 1$. At 2.5 GPa, the type-NP wall moves $b/2$ via $0\rightarrow
1$, forming an unstable type-P wall that then shifts another $b/2$ via
$\bar{1}\rightarrow 0$. These findings also imply that the rate-limiting step
can be manipulated by mechanical boundary conditions, being either
$\bar{1}\rightarrow 0$ at 0 GPa or $0\rightarrow 1$ under higher pressure.

The unusual mechanisms of DW motions in HfO2 revealed by MD are corroborated
by DFT-based nudged elastic band (NEB) calculations. As plotted in Fig. 2d,
the minimum energy pathways (MEPs) at both pressures display clear two-hop
characteristics: a larger barrier followed by a shallower one. The MEP for SD
switching (Fig. 2b) also involves successive $0\rightarrow 1$ and
$\bar{1}\rightarrow 0$ hops, but the first hop is associated with a larger
barrier. We propose that the presence of DWs decouples the movements of Op and
Onp atoms, rendering only one hop kinetically significant. This “divide-and-
conquer”-like mechanism explains the lower switching fields for type-P and
type-NP walls.

The velocities ($v$) for type-P and type-NP walls at 0 and 2.5 GPa,
respectively, are quantified using MD simulations across various temperatures
($T$) and field strengths. For both stress conditions, as shown in Fig. 2e-f,
the $\mathcal{E}$-dependence of $v$ follows a creep process [30, 31]:

$v=v_{0}\exp\left[-\frac{\mathcal{U}}{k_{B}T}\left(\frac{\mathcal{E}_{\rm
C0}}{\mathcal{E}}\right)^{\mu}\right],$ (1)

where $v_{0}$ is the DW velocity under an infinite field, $k_{B}$ is
Boltzmann’s constant, and $\mathcal{U}$ and $\mathcal{E}_{\rm C0}$ represent
the characteristic energy barrier and electric field at zero Kelvin,
respectively; $\mu$ is the creep exponent, which depends on the dimensionality
of the interface and the universality class of the disorder landscape pinning
the interface [32]. Equation (1) can be reformulated as:

$v=v_{0}\exp\left[-\left(\frac{\mathcal{E}_{a}}{\mathcal{E}}\right)^{\mu}\right],$
(2)

where $\mu=1$ corresponds to the famous Merz’s law [33, 34], and
$\mathcal{E}_{a}$ is the $T$-dependent activation field. By comparing
Equations (1) and (2), $\mathcal{E}_{a}$ is given as $\mathcal{E}_{a}=(T_{\rm
C0}/T)^{1/\mu}$ with $T_{\rm C0}=(\mathcal{U}/k_{B})\mathcal{E}_{\rm
C0}^{\mu}$. Remarkably, all velocity data fits well with $\mu=2$. This is
evidenced by the linear relationship between $\ln v$ and $1/\mathcal{E}^{2}$
and an accurate representation of $\mathcal{E}_{a}$’s $T$-dependence as
$\mathcal{E}_{a}=\sqrt{T_{\rm C0}/T}$ (Fig. 2g and insets). A creep exponent
of 2 is higher than the well-known value of $\mu=0.25$ for 1D magnetic domain
walls in ultrathin magnetic films [32] and $\mu=1$ for 2D ferroelectric domain
walls in typical perovskite ferroelectrics [29, 23].

Nucleation-and-growth at domain walls

We now develop an analytical model based on classical nucleation theory and
Landau–Ginzburg–Devonshire (LGD) theory to explain the origin of $\mu=2$ for
both types of DWs. The DW motion can be understood as the formation of a
quasi-two-dimensional nucleus with polarization aligned with $\mathcal{E}$ at
the wall, followed by the growth of the nucleus. The energy of the critical
nucleus gives the nucleation barrier ($\Delta U_{\rm nuc}^{*}$) which can be
further related to $\mathcal{E}_{a}$ in equation (2) via the Avrami theory of
transformation kinetics. We use a revised persistent-embryo method ($r$PEM)
[35] based on MD simulations to determine the size ($N^{*}$) and shape of the
critical nucleus at the DW (see Supplementary Sect. I). Specifically, a two-
dimensional (2D) embryo with $N_{0}$ switched oxygen atoms is initially
embedded in the unswitched DW, and a tunable harmonic potential is applied to
each oxygen atoms in the embryo to prevent back-switching. The potential is
gradually weakened and completely removed when the nucleus size ($N$)
surpasses a subcritical threshold ($N_{\rm sc}$ and $N_{0}<N_{\rm sc}<N^{*}$).
The plateau of the $N(t)$ curve then corresponds to the period over which the
nucleus size fluctuates around $N^{*}$.

Figure 3a displays the $N(t)$ curve for the type-P wall at 0.0 GPa, 300 K, and
$\mathcal{E}=0.45$ MV/cm, revealing multiple plateaus, from which the average
of $N^{*}$ is determined to be 38. The rate-limiting step of
$\bar{1}\rightarrow 0$ involves a half-unit-cell-thin critical nucleus in the
$xz$ plane consisting of Onp atoms. This nucleus is elliptical, measuring
$l_{x}^{*}=1.7$ nm in width and $l_{z}^{*}=2.7$ nm in height (Fig. 3c). For
the type-NP wall at 2.5 GPa and 300 K under $\mathcal{E}=$ 0.55 MV/cm, the
$N(t)$ curve in Fig. 3b yields a half-unit-cell-thin, elliptically-shaped
critical nucleus of Op atoms, and the dimensions are $l_{x}^{*}=3.2$ nm and
$l_{z}^{*}=5.9$ nm.

Having determined the shape of the critical nucleus, we can now proceed to
develop a LGD nucleation model. For an elliptical nucleus of size $l_{x}\times
l_{y}\times l_{z}$, its boundary within the $xz$ plane is conveniently
described with polar coordinates $l_{\theta}$ and $\theta$. Here, $l_{\theta}$
is given by
$l_{\theta}=l_{x}l_{z}/\sqrt{(l_{x}\sin\theta)^{2}+(l_{z}\cos\theta)^{2}}$. In
terms of the local displacement of oxygen atom ($p_{z}$) along the polar axis,
the atomistic profile for a type-P or type-NP wall containing a nucleus can be
represented by a single generalized equation:

$p_{z}^{\mathrm{nuc}}(y,\rho,\theta)=p_{s}\left[f(y,l_{y},\delta_{y})f(\rho,l_{\theta},\delta_{\theta})+g(y,l_{y},\delta_{y})\right],$
(3)

where
$f(q,l,\delta)=\frac{1}{2}\left[\tanh\left(\frac{q+l/2}{\delta/2}\right)-\tanh\left(\frac{q-l/2}{\delta/2}\right)\right]$
and $\delta$ is the diffuseness parameter;
$g(y,l_{y},\delta_{y})=-\frac{1}{2}\left[\tanh\left(\frac{y+l_{y}/2}{\delta_{y}/2}\right)+\lambda(-1)^{n}\right]$
with $n$ being an integer derived from $\lfloor\frac{y+l_{y}/2}{b/2}\rfloor$
and $\lambda$ taking $1$ for a nucleus of Onp atoms and $-1$ for a nucleus of
Op atoms; $p_{s}$ is the ground-state value for the local displacement of Op.
The $p_{z}$ profiles in the $yz$ and $xz$ planes generated by equation (3) are
shown in Fig. 3e and 3f for type-P and type-NP walls, respectively. In the
presence of $\mathcal{E}$, the energy of a wall with a nucleus relative to the
wall without any nucleus can be expressed as:

$\Delta U_{\rm nuc}=\Delta U_{\mathrm{V}}+\Delta U_{\mathrm{I}},$ (4)

with

$\Delta
U_{\mathrm{V}}=-\mathcal{E}\eta\int_{-\infty}^{\infty}\mathrm{d}y\int_{0}^{2\pi}\mathrm{d}\theta\int_{0}^{\infty}\rho\mathrm{d}\rho\left[p_{z}^{\mathrm{nuc}}(y,\rho,\theta)-p_{z}^{\mathrm{DW}}(y,\rho,\theta)\right],$
(5)

and

$\displaystyle\Delta U_{\mathrm{I}}=$
$\displaystyle\int_{-\infty}^{\infty}\mathrm{d}y\int_{0}^{2\pi}\mathrm{d}\theta\int_{0}^{\infty}\rho\mathrm{d}\rho\left\\{\left[U_{\mathrm{loc}}(p_{z}^{\mathrm{nuc}})+U_{x}(p_{z}^{\mathrm{nuc}})+U_{y}(p_{z}^{\mathrm{nuc}})+U_{z}(p_{z}^{\mathrm{nuc}})\right]-\right.$
(6)
$\displaystyle\left.\left[U_{\mathrm{loc}}(p_{z}^{\mathrm{DW}})+U_{y}(p_{z}^{\mathrm{DW}})\right]\right\\}.$

Here, $\Delta U_{\mathrm{V}}$ is the electric field-polarization coupling term
and $\Delta U_{\mathrm{I}}$ is the energy penalty for creating new interfaces
due to nucleation; $\eta$ is a scaling constant that relates oxygen
displacement $p_{z}$ to local polarization;
$p_{z}^{\mathrm{DW}}=p_{s}g(y,l_{y},\delta_{y})$ is the atomistic profile for
a nucleus-free wall. Taking advantage of the 2D nature of the nucleus
($l_{y}=b/2$, $\delta_{y}=0$) revealed from MD simulations, we can derive (see
details in Supplementary Sect. IV) that $\Delta U_{\rm
V}=-\frac{b}{16}\mathcal{E}\eta p_{s}\int_{0}^{2\pi}l_{\theta}^{2}d\theta$,
and $\Delta U_{\rm I}$ is given as:

$\displaystyle\Delta U_{\mathrm{I}}$ $\displaystyle=\Delta
U_{\mathrm{i}}^{\prime}+\Delta U_{\mathrm{i}}$ (7)
$\displaystyle=\frac{b}{16}\int_{0}^{2\pi}\zeta^{\prime}(l_{\theta}-\delta_{\theta})^{2}\mathrm{d}\theta+\left(\frac{b}{4}\int_{0}^{2\pi}\zeta
l_{\theta}\delta_{\theta}\mathrm{d}\theta+\frac{b}{6}\int_{0}^{2\pi}\frac{l_{\theta}g_{\theta}p_{s}^{2}}{\delta_{\theta}}\mathrm{d}\theta\right)$

We note that $\Delta U_{\mathrm{i}}^{\prime}$ results from the change in DW
type during the formation of a half-unit-cell-thin nucleus, where
$\zeta^{\prime}$ is proportional to the DW energy difference between type-P
and type-NP walls, $|\sigma_{\rm P}-\sigma_{\rm NP}|$. This energy penalty is
distinctive to DWs in HfO2. In contrast, nucleation at DWs in perovskite
ferroelectrics such as PbTiO3 does not involve a change in the DW type that
separates differently polarized domains (see Fig. S10). The term $\Delta
U_{\mathrm{i}}$ quantifies the energy penalty arising from polarization
changes and their gradient at the nucleus boundary, where $g_{\theta}$ is the
$\theta$-dependent gradient coefficient along $\rho$, and $\zeta$ scales with
the energy barrier of DW motion.

For a given $\mathcal{E}$, the energy of a 2D nucleus with the shape and size
specified by $l_{\theta}$ depends on just a few parameters: $p_{s}$, $\eta$,
$\zeta^{\prime}$, $\zeta$, and $g_{\theta}$. All these parameters can be
estimated straightforwardly with low costs. This enables the facile
determination of $N^{*}$ and $\Delta U_{\rm nuc}^{*}$ across a range of field
strengths. As demonstrated in Fig. 3g and 3i, for both wall types, the values
of $N^{*}$ predicted by the LGD model show remarkable agreement with the
results obtained from MD-based $r$PEM, confirming the validity and accuracy of
the nucleation model as universally applicable to both type-P and type-NP
walls. Interestingly, we find that $\Delta U_{\rm nuc}^{*}$ scales with the
$1/\mathcal{E}^{2}$ (Fig. 3h and j), indicating the nucleation rate
$J\propto\exp(-\frac{\Delta U_{\rm nuc}^{*}}{k_{B}T})$ can be reformulated as:

$J=J_{0}\exp\left[-\left(\frac{\mathcal{E}_{a,n}}{\mathcal{E}}\right)^{2}\right],$
(8)

where $J_{0}$ is a pre-exponential factor and $\mathcal{E}_{a,n}$ is the
effective activation field for nucleation. This further corroborates the
finding that the creep exponent is $\mu=2$ in equation (2). A series of model
calculations are performed using a nucleation model without $\Delta
U_{\mathrm{i}}^{\prime}$ to assess the impact of this unique energy term on
the field-dependence of $\Delta U_{\rm nuc}^{*}$. We observe that omitting
$\Delta U_{\mathrm{i}}^{\prime}$ recovers a linear relationship between
$\Delta U_{\rm nuc}^{*}$ and $1/\mathcal{E}$. These results demonstrate
$\Delta U_{\mathrm{i}}^{\prime}$ is the atomistic origin for $\mu=2$.

Intrinsic coercive fields from multiscale simulations

Based on Avrami’s theory of transformation kinetics, we derive the
relationship between $\Delta U_{\rm nuc}^{*}$ and $\mathcal{E}_{a}$ for DW
motions with $\mu=2$ as
$\mathcal{E}_{a}=\sqrt{\frac{1}{D+1}\frac{\lambda\Delta U_{\rm
nuc}^{*}}{k_{B}T}}\mathcal{E}$, where $D$ represents the dimensionality and
$\lambda$ is a scaling parameter mapping $\Delta U_{\rm nuc}^{*}$, associated
with a half-unit-cell hop, to the effective barrier encountered by a DW moving
one unit cell at a time (see Supplementary Sect. V). With $D=2$ and
$\lambda=0.5$, we compute $\mathcal{E}_{a}$ values using the LGD model at
various temperatures, and the analytical values for two types of DWs agree
with MD results (Fig. 4a). Utilizing the LGD model’s efficient
$\mathcal{E}_{a}$ prediction capability, we can conveniently predict the
values of $\mathcal{E}_{c}$ by simulating the polarization-electric
($P$-$\mathcal{E}$) hysteresis loops resulting from DW motions (see
Supplementary Sect. VI). As depicted in Fig. 4b, the $P$-$\mathcal{E}$ loop
for type-P walls in HfO2 at 0 GPa yields $\mathcal{E}_{c}\approx$0.1 MV/cm,
even lower than that of $\approx$0.2 MV/cm due to 180∘ DWs in Pb(Zr,Ti)O3 thin
films [23].

The analysis above suggests that the intrinsic coercive fields of hafnia-based
ferroelectrics could be quite low. Yet, why do they often exhibit much larger
$\mathcal{E}_{c}$ values experimentally [16, 17, 18]? We propose that this is
due to a “thermodynamic-kinetic” dilemma unique to 180∘ DWs in HfO2. The most
thermodynamically stable 180∘ DW is the $Pbca$-like type-PNP∗ wall, which is
immobile; whereas the type-P wall is the most mobile at room temperatures but
its higher thermodynamic energy makes its formation in thin films rare.
Therefore, the magnitude of $\mathcal{E}_{c}$ in hafnia-based thin films is
likely determined by the SD switching process, which can also be predicted by
our LGD nucleation model.

We first consider a single-crystal, single-domain HfO2 thin film with a
thickness $d$ and a (111) orientation that is commonly observed in
experiments. Assuming that $\mathcal{E}_{c}$ is the minimum electric field
required to extend the critical nucleus length to $d$, we determine the
intrinsic $E_{c}$ values as a function of $d$, plotted as a blue line in Fig.
4c. Notably, the theoretical $\mathcal{E}_{c}$ is $\approx$4 MV/cm for $d=5$
nm, an excellent agreement with experimental values for La-doped HfO2 [18] and
Hf0.5Zr0.5O2 films [36, 37] in the thin thickness limit where they are
predominantly composed of ferroelectric phases. Furthermore, we consider the
decreasing volume fraction of the polar phase with increasing thickness in
polycrystalline thin films, by introducing a thickness-dependent $p_{s}$ into
the nucleation model (see Supplementary Sect. VII). Despite this simple
treatment, our analytical values (red line in Fig. 4c) not only reproduce
experimental results across various film thickness, but also the scaling
relationship, $E_{c}\propto d^{-\frac{2}{3}}$, in epitaxial polycrystalline
thin films [18, 36, 37].

Conclusions

In conclusion, we have developed a multiscale approach to elucidate the
pivotal role of DW dynamics in dictating the coercive fields of ferroelectric
HfO2. Our finite-field MD simulations indicate that the $Pbca$-like DW, though
extensively studied, actually plays a rather limited role in ferroelectric
switching, and we have identified two mobile DW types. A thermally-induced
spontaneous transformation of DW types is responsible for the unusual “first-
hop-limiting” mechanism in the movements of mobile walls at room temperatures.
The formation of a half-unit-cell-thin elliptical nucleus is identified, which
supports the unconventional creep exponent of $\mu=2$. The LGD analytical
model, incorporating atomistic insights of nucleation, not only reproduces the
experimentally observed thickness scaling of coercive fields in hafnia-based
thin films but also uncovers an unexpectedly low intrinsic coercive field for
HfO2 due to mobile walls, rivaling that of perovskite ferroelectrics. Our
results offer a deeper understanding of the fundamental mechanisms governing
ferroelectric switching in this promising material, highlighting the potential
for achieving ultralow coercive fields in hafnia-based ferroelectrics through
domain wall engineering.

Acknowledgments J.W., J.Y., and S.L. acknowledge the supports from National
Key R&D Program of China (2021YFA1202100), National Natural Science Foundation
of China (12074319, 12361141821), and Westlake Education Foundation. The
computational resource is provided by Westlake HPC Center.

Author Contributions S.L. conceived and led the project. J.W. developed the
force field, performed MD simulations, and analyzed the MD data. J.Y.
identified the critical nucleus using $r$PEM with assistance from Y.S.,
derived the LGD analytical model, and predicted the coercive fields. All
authors contributed to the discussion and the manuscript preparation.

Competing Interests The authors declare no competing financial or non-
financial interests.

## References

  * Böscke  _et al._ [2011] T. S. Böscke, J. Müller, D. Bräuhaus, U. Schröder, and U. Böttger, Ferroelectricity in hafnium oxide thin films, Appl. Phys. Lett. 99, 102903 (2011).
  * Schroeder  _et al._ [2022] U. Schroeder, M. H. Park, T. Mikolajick, and C. S. Hwang, The fundamentals and applications of ferroelectric HfO2, Nat. Rev. Mater. 7, 653–669 (2022).
  * Huan  _et al._ [2014] T. D. Huan, V. Sharma, G. A. Rossetti, and R. Ramprasad, Pathways towards ferroelectricity in hafnia, Phys. Rev. B 90, 064111 (2014).
  * Sang  _et al._ [2015] X. Sang, E. D. Grimley, T. Schenk, U. Schroeder, and J. M. LeBeau, On the structural origins of ferroelectricity in HfO2 thin films, Appl. Phys. Lett. 106, 162905 (2015).
  * Materlik  _et al._ [2015] R. Materlik, C. Künneth, and A. Kersch, The origin of ferroelectricity in Hf1-xZrxO2: A computational investigation and a surface energy model, J. Appl. Phys. 117, 134109 (2015).
  * Park  _et al._ [2015] M. H. Park, Y. H. Lee, H. J. Kim, Y. J. Kim, T. Moon, K. D. Kim, J. Müller, A. Kersch, U. Schroeder, T. Mikolajick, and C. S. Hwang, Ferroelectricity and antiferroelectricity of doped thin HfO2-based films, Adv. Mater. 27, 1811 (2015).
  * Lee  _et al._ [2020] H.-J. Lee, M. Lee, K. Lee, J. Jo, H. Yang, Y. Kim, S. C. Chae, U. Waghmare, and J. H. Lee, Scale-free ferroelectricity induced by flat phonon bands in HfO2, Science 369, 1343 (2020).
  * Nukala  _et al._ [2021] P. Nukala, M. Ahmadi, Y. Wei, S. de Graaf, E. Stylianidis, T. Chakrabortty, S. Matzen, H. W. Zandbergen, A. Björling, D. Mannix, D. Carbone, B. Kooi, and B. Noheda, Reversible oxygen migration and phase transitions in hafnia-based ferroelectric devices, Science 372, 630 (2021).
  * Ma and Liu [2023] L.-Y. Ma and S. Liu, Structural polymorphism kinetics promoted by charged oxygen vacancies in HfO2, Phys. Rev. Lett. 130, 096801 (2023).
  * Silva  _et al._ [2023] J. P. B. Silva, R. Alcala, U. E. Avci, N. Barrett, L. Bégon-Lours, M. Borg, S. Byun, S.-C. Chang, S.-W. Cheong, D.-H. Choe, J. Coignus, V. Deshpande, A. Dimoulas, C. Dubourdieu, I. Fina, H. Funakubo, L. Grenouillet, A. Gruverman, J. Heo, M. Hoffmann, H. A. Hsain, F.-T. Huang, C. S. Hwang, J. Íñiguez, J. L. Jones, I. V. Karpov, A. Kersch, T. Kwon, S. Lancaster, M. Lederer, Y. Lee, P. D. Lomenzo, L. W. Martin, S. Martin, S. Migita, T. Mikolajick, B. Noheda, M. H. Park, K. M. Rabe, S. Salahuddin, F. Sánchez, K. Seidel, T. Shimizu, T. Shiraishi, S. Slesazeck, A. Toriumi, H. Uchida, B. Vilquin, X. Xu, K. H. Ye, and U. Schroeder, Roadmap on ferroelectric hafnia- and zirconia-based materials and devices, APL Mater. 11, 089201 (2023).
  * de Araujo  _et al._ [1995] C. A.-P. de Araujo, J. Cuchiaro, L. McMillan, M. Scott, and J. Scott, Fatigue-free ferroelectric capacitors with platinum electrodes, Nature 374, 627 (1995).
  * Park  _et al._ [2020] J. Y. Park, K. Yang, D. H. Lee, S. H. Kim, Y. Lee, P. R. S. Reddy, J. L. Jones, and M. H. Park, A perspective on semiconductor devices based on fluorite-structured ferroelectrics from the materials–device integration perspective, J. Appl. Phys. 128, 240904 (2020).
  * Pešić  _et al._ [2016] M. Pešić, F. P. G. Fengler, L. Larcher, A. Padovani, T. Schenk, E. D. Grimley, X. Sang, J. M. LeBeau, S. Slesazeck, U. Schroeder, and T. Mikolajick, Physical mechanisms behind the field-cycling behavior of HfO2-based ferroelectric capacitors, Adv. Funct. Mater. 26, 4601 (2016).
  * Yurchuk  _et al._ [2014] E. Yurchuk, J. Müller, J. Paul, T. Schlösser, D. Martin, R. Hoffmann, S. Müeller, S. Slesazeck, U. Schröeder, R. Boschke, R. van Bentum, and T. Mikolajick, Impact of scaling on the performance of HfO2-based ferroelectric field effect transistors, IEEE Trans. Electron Devices 61, 3699 (2014).
  * Ambriz-Vargas  _et al._ [2017] F. Ambriz-Vargas, G. Kolhatkar, M. Broyer, A. Hadj-Youssef, R. Nouar, A. Sarkissian, R. Thomas, C. Gomez-Yáñez, M. A. Gauthier, and A. Ruediger, A complementary metal oxide semiconductor process-compatible ferroelectric tunnel junction, ACS Appl. Mater. Interfaces 9, 13262 (2017).
  * Mimura  _et al._ [2018] T. Mimura, T. Shimizu, H. Uchida, O. Sakata, and H. Funakubo, Thickness-dependent crystal structure and electric properties of epitaxial ferroelectric Y2O3-HfO2 films, Appl. Phys. Lett. 113, 102901 (2018).
  * Wei  _et al._ [2018] Y. Wei, P. Nukala, M. Salverda, S. Matzen, H. J. Zhao, J. Momand, A. S. Everhardt, G. Agnus, G. R. Blake, P. Lecoeur, B. J. Kooi, J. Íñiguez, B. Dkhil, and B. Noheda, A rhombohedral ferroelectric phase in epitaxially strained Hf0.5Zr0.5O2 thin films, Nat. Mater. 17, 1095–1100 (2018).
  * Song  _et al._ [2021] T. Song, R. Bachelet, G. Saint-Girons, N. Dix, I. Fina, and F. Sánchez, Thickness effect on the ferroelectric properties of La-doped HfO2 epitaxial films down to 4.5 nm, J. Mater. Chem. C 9, 12224 (2021).
  * Ma  _et al._ [2023] L. Ma, J. Wu, T. Zhu, Y. Huang, Q. Lu, and S. Liu, Ultrahigh oxygen ion mobility in ferroelectric hafnia, Phys. Rev. Lett. 131, 256801 (2023).
  * Ding  _et al._ [2020] W. Ding, Y. Zhang, L. Tao, Q. Yang, and Y. Zhou, The atomic-scale domain wall structure and motion in HfO2-based ferroelectrics: A first-principle study, Acta Mater. 196, 556 (2020).
  * Choe  _et al._ [2021] D.-H. Choe, S. Kim, T. Moon, S. Jo, H. Bae, S.-G. Nam, Y. S. Lee, and J. Heo, Unexpectedly low barrier of ferroelectric switching in HfO2 via topological domain walls, Mater. Today 50, 8 (2021).
  * Wu  _et al._ [2023] Y. Wu, Y. Zhang, J. Jiang, L. Jiang, M. Tang, Y. Zhou, M. Liao, Q. Yang, and E. Y. Tsymbal, Unconventional polarization-switching mechanism in (Hf, Zr)O2 ferroelectrics and its implications, Phys. Rev. Lett. 131, 226802 (2023).
  * Liu  _et al._ [2016] S. Liu, I. Grinberg, and A. M. Rappe, Intrinsic ferroelectric switching from first principles, Nature 534, 360 (2016).
  * Zhang  _et al._ [2018] L. Zhang, J. Han, H. Wang, R. Car, and W. E, Deep potential molecular dynamics: A scalable model with the accuracy of quantum mechanics, Phys. Rev. Lett. 120, 143001 (2018).
  * Wu  _et al._ [2021] J. Wu, Y. Zhang, L. Zhang, and S. Liu, Deep learning of accurate force field of ferroelectric HfO2, Phys. Rev. B 103, 024108 (2021).
  * [26] Low-coercive-field ferroelectric hafnia with ultrafast domain walls, https://nb.bohrium.dp.tech/user/update/52795761357.
  * Zhao  _et al._ [2022] G.-D. Zhao, X. Liu, W. Ren, X. Zhu, and S. Yu, Symmetry of ferroelectric switching and domain walls in hafnium dioxide, Phys. Rev. B 106, 064104 (2022).
  * Zhou  _et al._ [2022] S. Zhou, J. Zhang, and A. M. Rappe, Strain-induced antipolar phase in hafnia stabilizes robust thin-film ferroelectricity, Sci. Adv. 8, eadd5953 (2022).
  * Shin  _et al._ [2007] Y. H. Shin, I. Grinberg, I. W. Chen, and A. M. Rappe, Nucleation and growth mechanism of ferroelectric domain-wall motion, Nature 449, 881 (2007).
  * Tybell  _et al._ [2002] T. Tybell, P. Paruch, T. Giamarchi, and J. Triscone, Domain wall creep in epitaxial ferroelectric Pb(Zr0.2Ti0.8)O3 thin films, Phys. Rev. Lett. 89, 097601 (2002).
  * Jo  _et al._ [2009] J. Jo, S. Yang, T. Kim, H. Lee, J. Yoon, S. Park, Y. Jo, M. Jung, and T. W. Noh, Nonlinear dynamics of domain-wall propagation in epitaxial ferroelectric thin film, Phys. Rev. Lett. 102, 045701 (2009).
  * Lemerle  _et al._ [1998] S. Lemerle, J. Ferré, C. Chappert, V. Mathet, T. Giamarchi, and P. Le Doussal, Domain wall creep in an ising ultrathin magnetic film, Phys. Rev. Lett. 80, 849 (1998).
  * Merz [1954] W. J. Merz, Domain formation and domain wall motions in ferroelectric BaTiO3 single crystals, Phys. Rev. 95, 690 (1954).
  * Miller and Weinreich [1960] R. C. Miller and G. Weinreich, Mechanism for the sidewise motion of 180° domain walls in barium titanate, Phys. Rev. 117, 1460 (1960).
  * Sun  _et al._ [2018] Y. Sun, H. Song, F. Zhang, L. Yang, Z. Ye, M. I. Mendelev, C.-Z. Wang, and K.-M. Ho, Overcoming the time limitation in molecular dynamics simulation of crystal nucleation: A persistent-embryo approach, Phys. Rev. Lett. 120, 085703 (2018).
  * Song  _et al._ [2020] T. Song, R. Bachelet, G. Saint-Girons, R. Solanas, I. Fina, and F. Sánchez, Epitaxial ferroelectric La-doped Hf0.5Zr0.5O2 thin films, ACS Appl. Electron. Mater. 2, 3221 (2020).
  * Lyu  _et al._ [2019] J. Lyu, I. Fina, J. Fontcuberta, and F. Sanchez, Epitaxial integration on Si (001) of ferroelectric Hf0.5Zr0.5O2 capacitors with high retention and endurance, ACS Appl. Mater. Interfaces 11, 6224 (2019).

Figure 1: Classification of 180∘ domain walls in ferroelectric HfO2. a,
Schematic representation of the unit cell of $Pca2_{1}$ HfO2 and three modes,
$T_{x}^{-}$, $A_{z}^{+}$, and $P_{z}^{+}$, in the cubic phase. According to
the $T_{x}^{-}$ mode, oxygen atoms with $+x$ and $-x$ displacements are
colored in purple and salmon, respectively. b, Classification of a 180∘ wall
represented as
$[T_{x}^{\alpha}A_{z}^{\beta}P_{z}^{-}|T_{x}^{\alpha^{\prime}}A_{z}^{\beta^{\prime}}P_{z}^{+}$].
c, Illustrations of four different types of DWs, named based on the type of
oxygen atoms (Op and Onp) at the wall. The Op atom displaced along $+z$ ($-z$)
is labeled as “1” (“${\bar{1}}$”), while the Onp atom is denoted as “0”. d,
Domain wall energies ($\sigma_{\rm DW}$) under two different stress conditions
determined by zero-Kelvin DFT calculations. e, Comparison of switching fields
($\mathcal{E}_{s}$) for a single domain (SD) and DWs at 0 and 2.5 GPa, as
obtained with finite-field MD simulations at 300 K. The value of
$\mathcal{E}_{s}$ is defined the minimum field strength required to initiate
DW propagation within 1 nanosecond.

Figure 2: Kinematics of 180∘ domain walls in HfO2. a, Mechanism of
polarization reversal in a unit cell. Oxygen atoms are color-coded based on
the local displacements along the $z$ axis, represented as arrows for Op and
green dots for Onp in the schematics. b, Energy profile for unit-cell-
switching obtained with DFT-based NEB calculations. c, Mechanisms deduced from
MD simulations for the motions of type-P walls at 0 GPa (left) and type-NP
walls at 2.5 GPa (right). d, DFT energy profiles for DW motions. The rate
limiting step is $\bar{1}\rightarrow 0$ for type-P walls at 0 GPa and
$0\rightarrow 1$ for type-NP walls at 2.5 GPa. Temperature ($T$)- and field
($\mathcal{E}$)-dependent DW velocity ($v_{y}$) for e, type-P walls and f,
type-NP walls. Solid lines are fits with
$v_{y}=v_{0}\exp\left(-(\frac{\mathcal{E}_{a}}{\mathcal{E}})^{\mu}\right)$ and
$\mu=2$. g, Plots of $\ln(v_{y})$ versus $1/\mathcal{E}^{2}$ for two wall
types under varying temperatures. Insets show that the temperature dependence
of the activation field can be well described by $\mathcal{E}_{a}=\sqrt{T_{\rm
C0}/T}$, supporting $\mu=2$.

Figure 3: Nucleation and growth at domain walls. Nucleus size ($N$) versus
time during the nucleation process at a a, type-P wall at 0 GPa and b, type-NP
wall at 2.5 GPa, both obtained using the MD-based $r$PEM method at 300 K. Blue
and green dashed lines indicate the size of the persistent 2D embryo ($N_{0}$)
and sub-critical nucleus ($N_{\rm sc}$, at which the harmonic bias is
removed), respectively. The critical size ($N^{*}$) is shown with a red solid
line. Insets illustrate plateaus where $N$ fluctuates around $N^{*}$.
Simulated atomistic profiles of a quasi-2D half-unit-cell-thin nucleus at a c,
type-P and d, type-NP wall. The corresponding analytical $p_{z}$ profiles
generated by equation (3) are depicted in e and f, respectively. Comparison of
$N^{*}$ values from MD simulations with those from a LGD-based analytical
model across various electric fields are presented in g and i for two wall
types. In both cases shown in h and j, $\Delta U_{\rm i}^{\prime}$ is
responsible for the quadratic dependence of critical nucleation barrier
($\Delta U_{\rm nuc}^{*}$) on $1/\mathcal{E}$, explaining the emergence of a
creep exponent $\mu=2$.

Figure 4: Coercive fields from multiscale simulations. a, Comparison of the
activation fields $\mathcal{E}_{a}$ in equation (2) for type-P and type-NP
walls obtained from MD simulations with results estimated using the LGD
analytical model. b, Intrinsic polarization-electric field hysteresis loops of
HfO2 thin films resulting from 180∘ DW motions. The simulated hysteresis loop
for Pb(Zr,Ti)O3 thin films is also plotted for comparison. c, Theoretical
thickness-dependent coercive fields at the nucleation limit for both single-
crystal and polycrystalline HfO2 thin films, compared to experimental results
for epitaxial thin films of La-doped HfO2 [18] and Hf0.5Zr0.5O2 (HZO) grown on
common substrates such as SrTiO3 (STO) [36] and yttria-stabilized zirconia
(YSZ) [37].


# Multimodal E-Commerce Product Classification Using Hierarchical Fusion

Tsegaye Misikir Tashu1 3, Sara Fattouh1, Péter Kiss1 and Tomáš Horváth12
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1 Department of Data
Science and Engineering, Faculty of Informatics, ELTE - Eötvös Loránd
University
Pázmány Péter sétány 1/C, 1117 Budapest, Hungary  2Faculty of Science,
Institute of Computer Science, Pavol Jozef Šafárik University,
Jesenná 5, 040 01 Košice, Slovakia  3College of Informatics, Kombolcha
Institute of Technology, Wollo University,
208 Kombolcha, Ethiopia

###### Abstract

In this work, we present a multi-modal model for commercial product
classification, that combines features extracted by multiple neural network
models from textual (CamemBERT and FlauBERT) and visual data (SE-ResNeXt-50),
using simple fusion techniques. The proposed method significantly outperformed
the performance of the unimodal models, as well as the reported performance of
similar models on our specific task. We made experiments with multiple fusing
techniques, and found, that the best preforming technique to combine the
individual embedding of the unimodal network is based on the combination of
concatenation and averaging the feature vectors. Each modality complemented
the shortcomings of the other modalities, demonstrating that increasing the
number of modalities can be an effective method for improving the performance
of multi-label and multimodal classification problems.

###### Index Terms:

Transformers, pretrained models, Ensemble, E-commerce, Multi-modal, Fusion.

††publicationid: pubid: 978-1-6654- 9653-7/22/$31.00 © 2022 IEEE

##  I Introduction

Product profiles on e-commerce platforms typically contain unstructured text,
product titles, and images, and provide details about the product that is
important to e-commerce. The profiles help customers make purchasing decisions
and, on the other hand, provide input for automated services of e-commerce
sites, such as question-answering and product recommendation systems. Since
the number of products and the number of product categories in an e-commerce
platform can be so high that it is unmanageable for humans, it is important to
have methods that are able to accurately assign these products to their
categories [3].

Most research on product classification has focused on the use of text-based
cues, even though products on e-commerce platforms contain more than textual
information and ignore the valuable information that associated images
contain. Therefore, it might be useful to experiment with multimodal models
that can combine features extracted from different modalities.

In this work, we proposed a multimodal product classification model based on
several pre-trained transformer-based architectures for language modeling and
image classification. The transformer-based language models were used to learn
a representation of text features from the textual information, and pre-
trained visual approaches are used to obtain a representation from images. We
used hierarchical fusion to combine the representations obtained from the
language models and the visual model, and a fully connected layer to perform
the final classification.

We empirically evaluated the proposed method in the Multimodal Product
Classification Task of the SIGIR 111https://sigir-ecom.github.io/data-
task.html 2020 E-Commerce Workshop Data Challenge. The results of the
experiments show superior performance compared to unimodal and other
multimodal methods. Our experiments have also shown that text-based
classification models generally perform better than visual models. On the
other hand, the inclusion of both modalities allows us to benefit from
complimentary feature information and outperformed baseline approaches.

The remainder of the paper is organized into six sections. Section II
describes related work relevant to our research. Section III presents our
proposed multimodal hierarchical fusion model architecture, section IV
presents the general experimental settings and implementation, while in
section V we summarize the experimental results and discussion based on the
experiments. Finally, section VI presents our conclusions.

##  II Related Work

In 2020, a large-scale competition to classify multimodal product data was
organized as part of the SIGIR 2020 e-commerce workshop. The dataset was
provided by Rakuten France, and each product included a title and a detailed
description in French, as well as a product image. The task was to predict 27
category labels from four major genres: books, children, households, and
entertainment. Several teams and researchers participated [9, 2, 4, 11]. The
most common solution proposed by most authors was to fine-tune pre-trained
text and image models as feature extractors and then use a bimodal fusion
mechanism to combine predictions. Most teams used the Transformer-based model
BERT [5] for text feature extraction and ResNet [6] for image feature
extraction. For bimodal fusion, the methods used were even more diverse.
Roughly in order of increasing complexity, the methods included simple late
decision-level fusion [9] and co-attention [2].

Bidirectional Encoder Representations from Transformers (BERT) is a state-of-
the-art open-source language model[1]. BERT Models are fine-tuned for specific
goals. Sun et al. [12] demonstrated improved text classification performance
by using various strategies, such as layer selection, layer-wise learning
rate, etc. BERT has been used in a variety of ways to improve language
processing tasks. Variations of BERT, trained on large French corpora, include
FlauBERT [8] and CamemBERT [10].

In this study, we will fine-tune the pre-trained French language models for
each textual modality separately and the SE -ResNeXt-50 [7] for image products
to extract features for each modality. The extracted features are
hierarchically fused using different fusion strategies to make the prediction.

##  III Hierarchical Fusion Model

In this section, we describe our proposed model architecture for classifying
product types.

The core of our method is to project the unimodal representation sequences
generated by the transformers $P$, $T_{f},T_{c}$ and $D_{f},D_{c}$ into a
common representation space and then learn a classifier in this common space:

$y=f(Z(P,Z(Z(T_{f},D_{f}),Z(T_{c},D_{c})))),$ (1)

where $Z$ is a fusion function that combines the extracted features, and $f$
is a neural network classifier that maps the features of the resulting joint
space to a class prediction $y$. $P$ is the reduced image representation in SE
-ResNeXt-50 [7], $T_{f},T_{c},D_{f},D_{c}$ are the representation sequences
extracted from the titles and the descriptions by FlauBERT [8] and CamemBERT
[10], respectively.

The proposed hierarchical model uses state-of-the-art pre-trained
architectures in the base models. In developing the fusion model, we have
tried to keep it as simple as possible, since a simple architecture can
provide good performance with less complexity and cost. The fusion
architecture is a simple trainable module.

###  III-A Text Encoding Models

We fine-tuned the two French-language models, FlauBERT and CamemBERT, for
encoding textual information of the product title and description into the
textual sequence representation. FlauBERT was trained with unsupervised masked
language modeling (MLM, similar to BERT), while CamemBERT was pre-trained on
French text based on Facebook’s RoBERTa model.

Given a textual product title sentence $T=(t_{1},t_{2},\dots t_{n})$, FlauBERT
and CamemBERT give embedding sequences $T_{f}=(t_{f1},t_{f2},\dots,t_{fn})$,
and $T_{c}=(t_{c1},t_{c2},\dots,t_{cn})$, while for a given textual product
description $D=(d_{1},d2,\dots,dm)$, FlauBERT and CamemBERT produce
$D_{f}=(d_{f1},d_{f2},\dots,dfm)$ $D_{c}=(D_{c1},D_{c2},\dots,D_{cm})$
embedding sequences, respectively.

###  III-B Image Encoding Model

For the purpose of obtaining representations of product images, we use the SE-
ResNeXt-50 [7] model. The last fully-connected (FC) layer of the pre-trained
model has been replaced, and the model has been fine-tuned for our
categorization task (that will be removed for the feature extraction). Based
on the work of [13, 15, 14], an input image $I$ is re-sized to $224\times 224$
and divided into $16\times 16$ regions. A raw image vector, a.k.a. a regional
feature representation, is then produced by applying the SE-ResNeXt-50 model
to each region $I_{i}(i=1,2,\dots,256)$. The final image feature vector ($P$)
is then obtained by averaging all regional image vectors.

$P=\frac{\sum\limits_{i=1}^{N_{r}}\text{ResNeXt}(I_{i})}{N_{r}}$ (2)

where ResNeXt$(I_{i})$ is the raw image vector extracted via SE-ResNeXt-50
from region $i$, $N_{r}$ ($256$ in this work) is the number of regions. The
final visual representation $P_{\text{full}}$ is the average of all regional
image vectors.

###  III-C Multi-modal Joint Representation Learning

$P$ represents the visual features extracted by running the input images
through the SE-ResNeXt-50 model. We used a 1D convolution layer with max-
pooling ($P\in\mathbb{R}^{d}$) to fit the dimension of the image
representation to the textual representations. Similarly, textual features
extracted by passing titles to CamemBERT and FlauBERT models are represented
as $T\\_c,T\\_f\in(\mathbb{R}^{d})$ and descriptions extracted by passing
descriptions through CamemBERT and FlauBERT models are represented as
$D_{f},D_{c}\in(\mathbb{R}^{d})$. To combine the text and image features, we
implemented the following fusing functions ($Z$):

  * •

Addition Fusion: The sum of input representations is defined as the addition
operation. Given two inputs ($x_{1},x_{2}$) of any modalities with the same
dimension $d$, the addition operation yields the output $X_{add}=x_{1}+x_{2}$,
where $X_{add}\in\mathbb{R}^{d}$.

  * •

Concatenation Fusion: The concatenation operation is defined as the
concatenation of representations across a dimension. In terms of the base
models: $X_{con}=<x_{1},x_{2}>$ , where $X_{con}\in\mathbb{R}^{d\times 2}$ and
$<>$ is the concatenation operation.

  * •

Average Fusion: The mean of the input representations is used to define the
average operation.
$X_{avg}(x_{1},x_{2})=mean(x_{{1}},x_{{2}})$, where
$X_{avg}\in\mathbb{R}^{d}$.

Figure 1: Multi-Modal Representation: Hierarchical Fusion

###  III-D Architecture

Our suggested strategy involves training a fully connected neural network
model that uses the hierarchically fused embedding vectors generated by the
three unimodal networks to construct a new probability vector over the target
categories. Our research aims to discover the most effective strategies to
combine the representations discovered by several distinct models. The
proposed architecture shown in Figure 1 consists of the following layers:

  * •

Input layer: The input layer receives and preprocesses visual and textual
modalities . We feed the textual base models with the product titles and
descriptions individually, resulting in four different textual embeddings, and
split the input images into regions before passing them to SE-ResReXt-50.

  * •

Representation layer: Using fine-tuned unimodal networks, the Representation
layer extracts feature vectors from preprocessed input data. It also combines
the SE-ResReXt-50 outputs by region.

  * •

Intermediate fusion layer: The embeddings of the text modalities (product
title and description) learned from each language model (CamemBERT, FlauBERT)
were first fused using one of our fusion methods in the Intermediate fusion
layer, then the fused results from CamemBERT and FlauBERT were further fused
to obtain the final vector representation for text modalities.

  * •

Feed forward layer. Finally, the fused vector is passed through a three-layer
fully connected neural network to produce a classification result. The
classifications are generated using the softmax activation function.

When training the fusion model, the uni-modal network layers are frozen, and
the refined product description and title representations are combined to
produce a single refined representation, which, along with the reduced image
representation, serves as input to our fully connected neural network. The
fusion layers’ operations produced various variants of the proposed
hierarchical architecture.

##  IV Experimental Settings

###  IV-A Dataset

We used a dataset provided by the Rakuten Institute of Technology for the
Rakuten Multi-modal Product Data Classification and Retrieval Challenge to
test the effectiveness of our proposed model. This task’s data set contains
99K products, nearly 84K of which were included in the training dataset. There
are 27 different product type codes in the text and image dataset, totaling
55K unique products. The four parent domains that group the entire product
catalog are Children, Household, Entertainment, and Books. The text data is in
French and consists of the title and description of the product.

The description field has a mean length of 35 words and a maximum length of
2068 words, while the title field has a mean length of 11 words and a maximum
length of 56 words. The product images are all square, with white or black
borders and a $500\times 500$ pixel resolution. Each product is associated
with an image. We preprocessed the data and performed the following steps:

  * •

Text Pre-processing: We tokenize the product title and description using the
NLTK 222http://www.nltk.org tokenizer, lowercase the text and remove HTML
tags.

  * •

Product images: To fine tune the pretrained mage models were augmented by
randomly rotating, flipping, and extracting random crops from the original
images.

  * •

Product images were resized to a uniform size of 224 × 224 and normalized per
channel using ImageNet mean and standard deviation.

Figure 2: Example of the multimodal information in the training set

###  IV-B Implementation Details

PyTorch was used to create all of our models. We employed the 138M parameter
$FlauBERT_{base}$ architecture with encoded vocabulary, as well as the 110M
parameter $CamemBERT_{base}$ architecture. With a batch size of 32 and a
sequence length of 256 for 10 epochs, the FlauBERT and CamemBERT models were
fine-tuned With a batch size of 32 for 20 epochs, the SE -ResNeXt-50 model was
fine-tuned as well.

The pre-trained weights of the trained uni-modal base models were used to
fine-tune the learning module for the multimodal representation. The uni-modal
network layers were frozen, and we used feature fusion methods to train our
suggested multimodal design, followed by a fully connected. network with a
Softmax output layer to perform product classification. As a loss function, we
employed the categorical cross-entropy minimization objective, the AdamW
optimizer with a learning rate value of $2\times 10^{-5}$ for text models, and
the Adam optimizer with $10^{-3}$ for image models.

##  V Results and Discussion

Multi-modal models were created with the goal of combining the representations
given by uni-modal models to produce more accurate, reliable, and robust
predictions. Several versions have been developed, with varying levels of
overall performance. In the following sections, we present our findings. We
split the original training set and utilized it for training and testing
because the labels for the test set were not available.

We employed two alternative splits to train and test our proposed methods in
order to assess the generalization power of our model. We used 90% for
training and 10% for testing in the first phase. 10% from the training was
also used as a validation set. We used 80% for training and 20% for testing in
the second phase. A validation set made up of 10% of the 80 percent training
was also utilized. The 90/10 split was utilized to create a test set that was
around the same size as the challenge.

###  V-A 90/10, Train/test Split

In the first experiment, we used a 90/10 split between training and testing,
using 90% for training and 10% for testing the model. Table I shows the
performance of each uni-modal model, namely CamemBERT-title, CamemBERT-
description, FlauBERT-title, FlauBERT-description, and SE -ResNeXt
individually. TableII shows the multimodal hierarchical fusion models on the
test set using the concatenation/average fusion method in the first fusion
layer (fusion of text embeddings) and different combination methods in the
last fusion layer.

TABLE I: Macro F1 scores and Accuracy of uni-modal models on test Set - 90/10 Split Model | Accuracy | F1 Score
---|---|---
Validation Set
CamemBERT_description | 86.98% | 84.83%
CamemBERT_title | 90.19% | 87.38%
FlauBERT_description | 84.43% | 84.43%
FlauBERT_title | 92.31% | 90.32%
SE-ResNeXt-50 | 59.25% | 54.30 %
TABLE II:  Hierarchical Fusion Results for various groupings of models on test Set - 90/10 Split Model | Accuracy | F1 Score
---|---|---
Concatenation Based
Concatenation Fusion | 92.80% | 92.47%
Addition Fusion | 92.75% | 92.45%
Average Fusion | 92.86% | 92.47%
Average Based
Average Fusion | 93.20% | 92.67%

In the proposed hierarchical fusion architecture, it is noticeable that the
model using the average combination in both fusion layers performs well.

###  V-B Introducing More Layers

After running different experiments, the model using the average fusion method
in both fusion layers showed the best performance compared to other models
trained with a 90/10 split of the dataset. We started incrementally
experimenting with Decision Layer by adding additional layers to further
improve the performance of this model as follows:

  * •

Adding a dropout layer after the last fusion layer.

  * •

Stacking a dropout layer followed by a fully connected (FC) layer and a non-
linearity before the final projection layer.

Table III presents the performance of the basic proposed architecture and the
modified architecture having stacked dropout out, FC and non-linearity after
the final fusion layer.

TABLE III: Average Fusion performance with Different Architectures - 90/10 Split Model | Accuracy | F1 Score
---|---|---
Validation Set
Basic Model | 93.20% | 92.67%
With Dropout | 92.49% | 90.00%
With More Layers | 92.42% | 91.11%

As we can see in Table III, the performance of the models decreased when we
added more additional layers. Therefore, the modified model with more layers
did not seem to improve the results, while the simplest architecture (Figure
1) showed better overall performance and reduced the training time, giving
room for further experiments.

It is noticeable in the simplest hierarchical architecture that the model that
used average combination in both fusion layers has a good performance on the
test set. However, to obtain a robust model, further investigation is needed
to verify that the model is not overfitted. To address the issue, we split the
dataset in a way that allows us to obtain more test set, which is the second
split strategy mentioned in section V.

###  V-C 80/20, Train/Test Split

This spilt was performed to see how the models performs on less training set
and a bit more test set. Table IV and table V shows the performance of uni-
modal models and the proposed hierarchical model variations in terms of
accuracy and F1 Score on the test set of the given dataset split, using
Concatenation/Average fusion method in the first fusion layer (textual
embeddings’ fusion) and different combination methods in the final fusion
layer.

TABLE IV:  Macro F1 scores and Accuracy of uni-modal models on test Set - 80/20 Split Model | Accuracy | F1 Score
---|---|---
test Set
CamemBERT_description | 86.40% | 83.13%
CamemBERT_title | 89.64% | 84.01%
FlauBERT_description | 85.81% | 81.08%
FlauBERT_title | 88.98% | 83.10%
SE-ResNeX-50 | 57.20% | 52.32%
TABLE V:  Hierarchical Fusion Models Performance on Test Sets- 80/20 Split Model | Accuracy | F1 Score
---|---|---
Concatenation Based
Addition Fusion | 91.97% | 89.39%
Concatenation Fusion | 92.36% | 90.03%
Average Fusion | 92.86% | 92.46%
Average Based
Average Fusion | 92.25% | 89.94%

We had the best performance when we used concatenation in the first and then
averaging in the second fusion layer in the 80/20 split as shown in table V,
but full average fusion outperformed the concatenation based models in the
validation and test sets as evidenced in table II. The best model has achieved
an overall F1 Score of $92.67$%, which indicates that merging the two
modalities boosts the performance compared to uni-modal networks.

##  VI Conclusion

In this work, multiple Deep Learning based pre-trained and fine-tuned models
were used with different fusion strategies with the goal of producing a
simple, easily trainable and good quality model for multimodal e-commerce
product classification. We used a dataset provided by Rakuten Institute of
Technology for the Rakuten Multi-modal Product Data Classification and
Retrieval Challenge that contains 27 different product type codes and a total
of 55K unique products.

The best performance was obtained using the average fusion operation in all
fusion layers, where the model reached a F1-Score of $92.67$% on the test set
given 90/10 dataset split. When it came to 80/20 dataset split, the best
achieved model was concatenation-based in fusing textual features, and
average-based fusion in the final fusion layer with F1-Score of $92.46$%. Our
experimental results in general showed that fine-tuning pre-trained models for
feature vector representation with simple hierarchical fusion approaches can
yield good results in multimodal e-commerce product classification.

## Acknowledgements

This research is supported by the ÚNKP-21-4 New National Excellence Program of
the Ministry for Innovation and Technology from the source of the National
Research, Development and Innovation Fund.

Supported by the Telekom Innovation Laboratories (T-Labs), the Research and
Development unit of Deutsche Telekom.

Project no. ED_18-1-2019-0030 (Application domain specific, highly reliable IT
solutions subprogramme) has been implemented with the support provided from
the National Research, Development and Innovation Fund of Hungary, financed
under the Thematic Excellence Programme funding scheme.

## References

  * [1] Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin.  Docbert: BERT for document classification.  CoRR, abs/1904.08398, 2019.
  * [2] Ye Bi, Shuo Wang, and Zhongrui Fan.  A multimodal late fusion model for e-commerce product classification.  arXiv preprint arXiv:2008.06179, 2020.
  * [3] Lei Chen, Houwei Chou, Yandi Xia, and Hirokazu Miyake.  Multimodal item categorization fully based on transformer.  In Proceedings of The 4th Workshop on e-Commerce and NLP, pages 111–115, Online, August 2021. Association for Computational Linguistics.
  * [4] Varnith Chordia and Vijay Kumar BG.  Large scale multimodal classification using an ensemble of transformer models and co-attention.  arXiv preprint arXiv:2011.11735, 2020.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [7] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [8] Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab.  FlauBERT: Unsupervised language model pre-training for French.  In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2479–2490, Marseille, France, May 2020. European Language Resources Association.
  * [9] Younghun Lee, Lei Chen, Yandi Xia, and Wei-Te Chen.  Cbb-fe, camembert and bit feature extraction for multimodal product classification and retrieval.  2020\.
  * [10] Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot.  CamemBERT: a tasty French language model.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online, July 2020. Association for Computational Linguistics.
  * [11] Barbara Rychalska and Jacek Dąbrowski.  Synerise at sigir rakuten data challenge 2020: Efficient manifold density estimator for cross-modal retrieval.  SIGIR eCom Challenge 2020, 2020.
  * [12] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.  How to fine-tune bert for text classification?  In Maosong Sun, Xuanjing Huang, Heng Ji, Zhiyuan Liu, and Yang Liu, editors, Chinese Computational Linguistics, pages 194–206, Cham, 2019. Springer International Publishing.
  * [13] Tsegaye Misikir Tashu, Sakina Hajiyeva, and Tomas Horvath.  Multimodal emotion recognition from art using sequential co-attention.  Journal of Imaging, 7(8), 2021.
  * [14] Tsegaye Misikir Tashu and Tomáš Horváth.  Attention-based multi-modal emotion recognition from art.  In Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani, editors, Pattern Recognition. ICPR International Workshops and Challenges, pages 604–612, Cham, 2021. Springer International Publishing.
  * [15] P. Wang, Q. Wu, C. Shen, and A. van den Hengel.  The vqa-machine: Learning how to use existing vision algorithms to answer new questions.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3909–3918, 2017.