# On a Quaternary Non-Local Isoperimetric Problem

Stanley Alama Lia Bronsard Xinyang Lu Chong Wang Department of Mathematics and
Statistics, McMaster University. E-mail<EMAIL_ADDRESS>of
Mathematics and Statistics, McMaster University. E-mail<EMAIL_ADDRESS>Department of Mathematical Sciences, Lakehead University. Email:
<EMAIL_ADDRESS>Department of Mathematics, Washington and Lee University.
Email<EMAIL_ADDRESS>
###### Abstract

We study a two-dimensional quaternary inhibitory system. This free energy
functional combines an interface energy favoring micro-domain growth with a
Coulomb-type long range interaction energy which prevents micro-domains from
unlimited spreading. Here we consider a limit in which three species are
vanishingly small, but interactions are correspondingly large to maintain a
nontrivial limit. In this limit two energy levels are distinguished: the
highest order limit encodes information on the geometry of local structures as
a three-component isoperimetric problem, while the second level describes the
spatial distribution of components in global minimizers. Geometrical
descriptions of limit configurations are derived.

This paper is a heartfelt tribute to Bob Pego, acknowledging his profound and
enduring contributions to the field of PDEs and applied mathematics.

##  1 Introduction

Block copolymers, a class of soft materials, have gained much interest in
recent years due to their ability of self-organization into nanoscale ordered
structures [9]. An $ABCD$ tetrablock copolymer is a linear-chain molecule
consisting of four subchains, joined covalently to each other. Because of the
repulsive forces between different types of monomers, different types of
subchain tend to segregate. However, since subchains are chemically bonded in
molecules, segregation can lead to a phase separation only at microscopic
level, where $A,B,C$ and $D$-rich micro-domains emerge, forming morphological
phases, many of which have been observed experimentally.

To model tetrablock copolymers, one uses a free energy

$\displaystyle\mathcal{E}(u):=\frac{1}{2}\sum_{i=0}^{3}\int_{\mathbb{T}^{2}}|\nabla
u_{i}|+\sum_{i,j=1}^{3}\frac{\gamma_{ij}}{2}\int_{\mathbb{T}^{2}}\int_{\mathbb{T}^{2}}G_{\mathbb{T}^{2}}(x-y)\;u_{i}(x)\;u_{j}(y)dxdy$
(1.1)

defined on $BV(\mathbb{T}^{2};\\{0,1\\})$. In (1.1), $u=(u_{1},u_{2},u_{3})$,
and $u_{0}=1-\sum_{i=1}^{3}u_{i}$. The order parameters $u_{i},i=0,1,2,3$ are
defined on
$\mathbb{T}^{2}=\mathbb{R}^{2}/\mathbb{Z}^{2}=[-\frac{1}{2},\frac{1}{2}]^{2}$,
i.e., the two dimensional flat torus of unit volume, with periodic boundary
conditions. Each $u_{i}$, which represents the relative monomer density, has
two preferred states: $u_{i}=0$ and $u_{i}=1$. Case $u_{i}=1,i=1,2,3$ or $0$
corresponds to a pure-$A$, pure-$B$, pure-$C$, or pure-$D$ region,
respectively. Thus, each $u_{i}=\chi_{\Omega_{i}}$ with supports $\Omega_{i},\
i=0,1,2,3$ which partition $\mathbb{T}^{2}$: $\Omega_{i}$ are assumed to be
mutually disjoint and $\sum_{i=0}^{3}u_{i}=1$ almost everywhere on
$\mathbb{T}^{2}$.

The energy is minimized under three mass or area constraints

$\displaystyle\frac{1}{|\mathbb{T}^{2}|}\int_{\mathbb{T}^{2}}u_{i}=M_{i},i=1,2,3.$
(1.2)

Here $M_{i},i=1,2,3$ are the area fractions of type-$A$, type-$B$ and type-$C$
regions, respectively. Constraints (1.2) model the fact that, during an
experiment, the compositions of the molecules do not change.

The first term in (1.1) counts the perimeter of the interfaces: indeed, for
$u_{i}\in BV(\mathbb{T}^{2};\\{0,1\\})$,

$\int_{\mathbb{T}^{2}}|\nabla u_{i}|:=\sup\left\\{\int_{\mathbb{T}^{2}}u_{i}\
\text{div}\varphi\ dx:\varphi=(\varphi_{1},\varphi_{2})\in
C^{1}(\mathbb{T}^{2};\mathbb{R}^{2}),|\varphi(x)|\leq 1\right\\},$

defines the total variation of the characteristic function $u_{i}$. The factor
$\frac{1}{2}$ acknowledges that each interface between the phases is counted
twice in the sum.

The second part of (1.1) is the long range interaction energy, associated with
the connectivity of sub-chains in the tretrablock copolymer molecule. The long
range interaction coefficients $\gamma_{ij}$ form a symmetric matrix

$\gamma=[\gamma_{ij}]\in\mathbb{R}^{3\times 3}.$

Here $G_{\mathbb{T}^{2}}$ is the zero-mean Green’s function for $-\triangle$
on $\mathbb{T}^{2}$ with periodic boundary conditions, satisfying

$-\Delta G_{\mathbb{T}^{2}}(\cdot-y)=\delta(\cdot-y)-1\text{ in
}\mathbb{T}^{2};\int_{\mathbb{T}^{2}}{G_{\mathbb{T}^{2}}(x-y)}dx=0$ (1.3)

for each $y\in\mathbb{T}^{2}$. In two dimensions, the Green’s function
$G_{\mathbb{T}^{2}}$ has the local representation

$G_{\mathbb{T}^{2}}(x-y)=-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y),$
(1.4)

for $|x-y|<\frac{1}{2}$. Here $R_{\mathbb{T}^{2}}\in
C^{\infty}(\mathbb{T}^{2})$ is the regular part of the Green’s function.

As were the cases for the Ohta-Kawasaki model of diblock copolymers and the
Nakazawa-Ohta model of triblock copolymers, nonlocal quaternary systems are of
high mathematical interest because of the diverse patterns which are expected
to be observed by its minimizers. Just as the diblock copolymer problem or the
triblock copolymer problem may be formulated as a nonlocal isoperimetric
problem (NLIP) which partitions space into two or three components,
respectively, the tetrablock model is an NLIP based on partitions into four
disjoint components.

Here we consider an asymptotic regime where three minority phases have
vanishingly small areas. Like [13, 3], introduce a new parameter $\eta$,
controlling the vanishing areas. That is,

$\int_{\mathbb{T}^{2}}u_{i}=\eta^{2}M_{i}$

for some fixed $M_{i}$, $i=1,2,3$, and rescale $u_{i}$ as

$\displaystyle v_{i,\eta}=\frac{u_{i}}{\eta^{2}},\;i=0,1,2,3,\;\;\text{ with
}{\int}_{\mathbb{T}^{2}}v_{i,\eta}=M_{i},\;i=1,2,3.$ (1.5)

The matrix $\gamma=[\gamma_{ij}]$ is also scaled, in such a way that both
terms in (1.1) contribute at the same order in $\eta$. This can be achieved by
choosing

$\displaystyle\gamma_{ij}=\frac{1}{|\log\eta|\eta^{3}}\Gamma_{ij},$

with fixed constants $\Gamma_{ij}$. Let
$v_{\eta}=(v_{1,\eta},v_{2,\eta},v_{3,\eta})$. Assume $v_{\eta}$ lies in the
space

$X_{\eta}:=\left\\{(v_{1,\eta},v_{2,\eta},v_{3,\eta})\ |\
\eta^{2}v_{i,\eta}\in BV({\mathbb{T}^{2}};\\{0,1\\}),\
v_{i,\eta}\,v_{j,\eta}=0\ \ a.e.\ \ 1\leq i<j\leq 3\right\\}.$ (1.6)

For $v_{\eta}\in X_{\eta}$, we define

$\displaystyle
E_{\eta}(v_{\eta}):=\frac{1}{\eta}\mathcal{E}(u)=\frac{\eta}{2}\sum_{i=0}^{3}\int_{\mathbb{T}^{2}}|\nabla
v_{i,\eta}|+\sum_{i,j=1}^{3}\
\frac{\Gamma_{ij}}{2|\log\eta|}\int_{\mathbb{T}^{2}}\int_{\mathbb{T}^{2}}G_{\mathbb{T}^{2}}(x-y)v_{i,\eta}(x)v_{j,\eta}(y)dxdy,$
(1.7)

and $E_{\eta}(v_{\eta})=+\infty$ otherwise. Consider $v_{i,\eta}$ of
components of the form

$\displaystyle v_{i,\eta}=\sum_{k=1}^{\infty}v_{i,\eta}^{k},\text{ and
}v_{i,\eta}^{k}=\frac{1}{\eta^{2}}\chi_{\Omega_{i,\eta}^{k}},$ (1.8)

where $|\Omega_{i,\eta}^{k}\cap\Omega_{j,\eta}^{l}|=0$ when $i\neq j$ or
$k\neq l$, each $\Omega_{i,\eta}^{k}$ is a connected subset of
$\Omega_{i,\eta}$, and
$\Omega_{i,\eta}=\cup_{k=1}^{\infty}\Omega_{i,\eta}^{k}$, $i=1,2,3$. Here
$|\Omega_{i,\eta}^{k}\cap\Omega_{j,\eta}^{l}|$ denotes the area (i.e., the
Lebesgue measure) of $\Omega_{i,\eta}^{k}\cap\Omega_{j,\eta}^{l}$. Assume that
each $\Omega_{i,\eta}^{k}$ does not intersect $\partial\mathbb{T}^{2}$, and
then we extend $v_{i,\eta}^{k}$ to zero outside of $\mathbb{T}^{2}$. Let

$\displaystyle z_{i}^{k}(x):=\eta^{2}v_{i,\eta}^{k}(\eta x).$ (1.9)

Note that $z_{i}^{k}(x):\mathbb{R}^{2}\rightarrow\\{0,1\\}$ and
$\int_{\mathbb{R}^{2}}z_{i}^{k}=\int_{\mathbb{T}^{2}}v_{i,\eta}^{k}$ and
$\int_{\mathbb{R}^{2}}|\nabla z_{i}^{k}|=\eta\int_{\mathbb{T}^{2}}|\nabla
v_{i,\eta}^{k}|.$ Calculation yields

$\displaystyle
E_{\eta}(v_{\eta})=\sum_{k=1}^{\infty}\left(\frac{1}{2}\sum_{i=0}^{3}\int_{\mathbb{R}^{2}}|\nabla
z_{i}^{k}|+\sum_{i,j=1}^{3}\frac{\Gamma_{ij}}{4\pi}\left(\int_{\mathbb{R}^{2}}z_{i}^{k}\right)\left(\int_{\mathbb{R}^{2}}z_{j}^{k}\right)\right)+O\left(\frac{1}{|\log\eta|}\right).$
(1.10)

Let $m:=(m_{1},m_{2},m_{3})$. Define

$\displaystyle e_{0}(m)$ $\displaystyle:=$
$\displaystyle\inf\Bigg{\\{}\frac{1}{2}\sum_{i=0}^{3}\int_{\mathbb{R}^{2}}|\nabla
z_{i}|+\sum_{i,j=1}^{3}\frac{\Gamma_{ij}m_{i}m_{j}}{4\pi}:z_{i}\in
BV(\mathbb{R}^{2};\\{0,1\\}),i=0,1,2,3,$ (1.11)
$\displaystyle\qquad\qquad\int_{\mathbb{R}^{2}}z_{i}=m_{i},i=1,2,3,\;\
z_{0}=1-\sum_{i=1}^{3}z_{i}\Bigg{\\}}$

Since a triple bubble is the unique solution to the three-component
isoperimetric problem [56]

$\displaystyle\inf\Bigg{\\{}\frac{1}{2}\sum_{i=0}^{3}\int_{\mathbb{R}^{2}}|\nabla
z_{i}|:z_{i}\in
BV(\mathbb{R}^{2};\\{0,1\\}),i=0,1,2,3,\int_{\mathbb{R}^{2}}z_{i}=m_{i},i=1,2,3,\;\
z_{0}=1-\sum_{i=1}^{3}z_{i}\Bigg{\\}},$

(1.11) can be simplified as

$\displaystyle
e_{0}(m)=p(m_{1},m_{2},m_{3})+\sum_{i,j=1}^{3}\frac{\Gamma_{ij}m_{i}m_{j}}{4\pi},$
(1.12)

where $p(m_{1},m_{2},m_{3})$ is the perimeter of a triple bubble in
$\mathbb{R}^{2}$ when $m_{i}>0,i=1,2,3$. In the case of one $m_{i}$ equals
$0$, that is, $m=(m_{1},m_{2},0)$ or $(m_{1},0,m_{3})$ or $(0,m_{2},m_{3})$, a
triple bubble degenerates to a double bubble and (1.12) becomes

$\displaystyle\text{
}e_{0}(m)=e_{0}^{d}(m_{i},m_{j}):=p^{d}(m_{i},m_{j})+\frac{\Gamma_{ii}(m_{i})^{2}}{4\pi}+\frac{\Gamma_{jj}(m_{j})^{2}}{4\pi}+\frac{\Gamma_{ij}m_{i}m_{j}}{2\pi},$
(1.13)

where $1\leq i,j\leq 3,i\neq j$, and $p^{d}(m_{i},m_{j})$ is the perimeter of
a double bubble in $\mathbb{R}^{2}$. In the case of two $m_{i}$ equal $0$,
that is, $m=(m_{1},0,0)$ or $(0,m_{2},0)$ or $(0,0,m_{3})$, a triple bubble
degenerates to a single bubble and (1.12) becomes

$\displaystyle\text{ }e_{0}(m)=e_{0}^{s}(m_{i}):=2\sqrt{\pi
m_{i}}+\frac{\Gamma_{ii}(m_{i})^{2}}{4\pi}.$ (1.14)

Thus, we expect that minimizers of $E_{\eta}$ in two dimensions will always
consist of single, double or triple bubbles (or mixture); no other shapes are
expected for the components of $\Omega_{i,\eta}$. The spatial distribution of
the single, double or triple bubbles on ${\mathbb{T}^{2}}$ should be
determined by the higher order terms of $E_{\eta}$ in a more detailed energy
expansion.

To further discuss how the total masses $M=(M_{1},M_{2},M_{3})$ are to be
divided, define

$\displaystyle\overline{e_{0}}(M):=\inf\left\\{\sum_{k=1}^{\infty}e_{0}(m^{k}):m^{k}=(m_{1}^{k},m_{2}^{k},m_{3}^{k}),\
m_{i}^{k}\geq 0,\ \sum_{k=1}^{\infty}m_{i}^{k}=M_{i},i=1,2,3\right\\}.$ (1.15)

The main result of this paper concerns the minimizers of
$\overline{e_{0}}(M)$.

######  Theorem 1.1.

(coexistence) There exist global minimizers with arbitrarily large number of
single, double or triple bubbles. That is, given $N_{i}$, $i=1,2,3$, there
exist parameters $M_{i}$ and $\Gamma_{ij}$ such that all global minimizers
have at least $N_{1}$ (resp. $N_{2}$, $N_{3}$) triple bubbles (resp. double,
single bubbles).

These are proven via delicate comparison arguments based on the geometry of
triple bubbles, in Section 2. In Section 3 we discuss minimizers at $\eta$
level.

While mathematical study of nonlocal quaternary systems is relatively limited,
recently there has been progress in mathematical analysis of nonlocal ternary
systems:

$\displaystyle\mathcal{E}(u):=\frac{1}{2}\sum_{i=0}^{2}\int_{\mathbb{T}^{2}}|\nabla
u_{i}|+\sum_{i,j=1}^{2}\frac{\gamma_{ij}}{2}\int_{\mathbb{T}^{2}}\int_{\mathbb{T}^{2}}G_{\mathbb{T}^{2}}(x-y)\;u_{i}(x)\;u_{j}(y)dxdy.$
(1.16)

For ternary systems, one-dimensional stationary points to the Euler-Lagrange
equations of (1.16) were found in [49, 15]. Two and three dimensional
stationary configurations were studied recently in [52, 51, 46, 47, 23]. There
are also plenty of mathematical work on nonlocal binary systems. Much early
work concentrated on the diffuse interface Ohta-Kawasaki density functional
theory for diblock copolymers [45, 39, 14],

$\displaystyle\mathcal{E}(u):=\int_{\mathbb{T}^{n}}|\nabla
u|+\gamma\int_{\mathbb{T}^{n}}\int_{\mathbb{T}^{n}}G_{\mathbb{T}^{n}}(x-y)\;u(x)\;u(y)dxdy,$
(1.17)

with a single mass or volume constraint. The dynamics for a gradient flow for
(1.17) with small volume fraction were developed in [28, 24]. All stationary
solutions to the Euler-Lagrange equation of (1.17) in one dimension were known
to be local minimizers [48], and many stationary points in two and three
dimensions have been found that match the morphological phases in diblock
copolymers [44, 50, 55, 32, 18, 1]. The sharp interface nonlocal isoperimetric
problems have been the object of great interest, both for applications and for
their connection to problems of minimal or constant curvature surfaces. Global
minimizers of (1.17), and the related Gamow’s Liquid Drop model describing
atomic nuclei, were studied in [7, 43, 13, 54, 25, 34, 35, 30, 37, 22] for
various parameter ranges. Variants of the Gamow’s liquid drop model with
background potential or with an anisotropic surface energy replacing the
perimeter, are studied in [5, 33, 12]. Higher dimensions are considered in
[10, 17]. Applications of the second variation of (1.17) and its connections
to minimality and $\Gamma$-convergence are to be found in [16, 2, 31]. The
bifurcation from spherical, cylindrical and lamellar shapes with Yukawa
instead of Coulomb interaction has been done in [19]. Blends of diblock
copolymers and nanoparticles [4, 6] and blends of diblock copolymers and
homopolymers are also studied by [11, 15]. Extension of the local perimeter
term to nonlocal $s$-perimeters is studied in [20].

### Acknowledgements

Stanley Alama, Lia Bronsard, and Xinyang Lu gratefully acknowledge the support
of the Natural Science and Engineering Research Council (Canada) through the
Discovery Grants program.

##  2 Geometric properties of global minimizers

In this section we analyze the geometric properties of minimizers of
$\overline{e_{0}}(M)$. One of the key difficulties is that we lack an explicit
formula for the perimeter of triple bubbles. Recall that a triple bubble is a
group of three adjacent sets bounded by six circular arcs; see Figure 1.

Figure 1:  Symmetric (left) and asymmetric (right) triple bubbles.

Figure 2 shows a triple bubble $\Omega=(\Omega_{1},\Omega_{2},\Omega_{3})$,
where $\Omega_{1}$ is bounded by three circular arcs $C_{1}$, $C_{4}$ and
$C_{5}$ of radii $r_{1}$, $r_{4}$ and $r_{5}$; $\Omega_{2}$ is bounded by
three circular arcs $C_{2}$, $C_{4}$ and $C_{6}$ of radii $r_{2}$, $r_{4}$ and
$r_{6}$; $\Omega_{3}$ is bounded by three circular arcs $C_{3}$, $C_{5}$ and
$C_{6}$ of radii $r_{3}$, $r_{5}$ and $r_{6}$.

Figure 2: An asymmetric triple bubble with radii $r_{i},1\leq i\leq 6$.

In the case of a double bubble, the radii of curvature of the three circular
arcs are related by a reciprocal relation [3]. For a triple bubble, the
reciprocal relation holds for each pair of bubbles [29], that is,

$\displaystyle\frac{1}{r_{4}}+\frac{1}{r_{1}}-\frac{1}{r_{2}}=0,\qquad\frac{1}{r_{5}}+\frac{1}{r_{1}}-\frac{1}{r_{3}}=0,\qquad\frac{1}{r_{6}}+\frac{1}{r_{2}}-\frac{1}{r_{3}}=0.\qquad$
(2.1)

The area constraints $|\Omega_{i}|=m_{i},i=1,2,3$ can be expressed as

$\displaystyle\frac{1}{2}r_{1}^{2}(\theta_{1}-\sin\theta_{1})-\frac{1}{2}r_{4}^{2}(\theta_{4}-\sin\theta_{4})-\frac{1}{2}r_{5}^{2}(\theta_{5}-\sin\theta_{5})+\frac{1}{2}h_{4}h_{5}\sin
s_{4}$ $\displaystyle=$ $\displaystyle m_{1},$ (2.2)
$\displaystyle\frac{1}{2}r_{2}^{2}(\theta_{2}-\sin\theta_{2})-\frac{1}{2}r_{6}^{2}(\theta_{6}-\sin\theta_{6})+\frac{1}{2}r_{4}^{2}(\theta_{4}-\sin\theta_{4})+\frac{1}{2}h_{4}h_{6}\sin
s_{5}$ $\displaystyle=$ $\displaystyle m_{2},$ (2.3)
$\displaystyle\frac{1}{2}r_{3}^{2}(\theta_{3}-\sin\theta_{3})+\frac{1}{2}r_{5}^{2}(\theta_{5}-\sin\theta_{5})+\frac{1}{2}r_{6}^{2}(\theta_{6}-\sin\theta_{6})+\frac{1}{2}h_{5}h_{6}\sin
s_{6}$ $\displaystyle=$ $\displaystyle m_{3},$ (2.4)

where $\theta_{i}$ and $h_{i}$ are the angle and chord associated with the
circular arc $C_{i}$, respectively, $s_{4}$ is the angle between chords
$h_{4}$ and $h_{5}$, $s_{5}$ is the angle between chords $h_{4}$ and $h_{6}$,
and $s_{6}$ is the angle between chords $h_{5}$ and $h_{6}$.

The four points where the three arcs meet are termed triple junction points.
At triple junction points, the three arcs meet at $120$ degree angle. Let
$O_{i}$ denotes the center corresponding to the circular arc $C_{i}$. The
radii conditions (2.1) and the 120 degree angle conditions imply that centers
$O_{1},O_{2},O_{4}$ lie on a straight line, centers $O_{1},O_{3},O_{5}$ lie on
a straight line, centers $O_{2},O_{3},O_{6}$ lie on a straight line, and
centers $O_{4},O_{5},O_{6}$ lie on a straight line.

######  Lemma 2.1.

It holds

$\frac{\partial}{\partial m_{i}}p(m_{1},m_{2},m_{3})=\frac{1}{r_{i}},\qquad
i=1,2,3,$

where $r_{i}=r_{i}(m_{1},m_{2},m_{3})$ denotes the radius of the lobe of type
$i$ constituent; see Figure 2.

###### Proof.

The proof is similar to the double bubble case, and uses similar constructions
[3]. ∎

######  Proposition 2.2.

Let the total masses $M_{i}$ and interaction coefficients $\Gamma_{ij}$ be
given. Consider a local minimizer of $\overline{e_{0}}(M)$, and denote by
$B^{k}=(m_{1}^{k},m_{2}^{k},m_{3}^{k})$ its bubbles. Then

$\min_{k}m_{i}^{k}\geq m_{i}^{-},\qquad i=1,2,3$

where

$m_{i}^{-}\geq c_{1}\min_{k}(r_{i}^{k})^{2}\geq
c_{1}\min\bigg{\\{}\bigg{[}\frac{1}{\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j}\bigg{]}^{-2},\bigg{[}\frac{C_{2}M_{i}}{4c_{2}\inf
E_{0}}\bigg{]}^{2}\bigg{\\}},\qquad i=1,2,3$

is some lower bound depending only on $M_{i}$ and $\Gamma_{ij}$. Here
$r_{i}^{k}$ stands for the radius of the lobe with mass $m_{i}^{k}$, and
$c_{1},c_{2},C_{2}>0$ are universal geometric constants (see (2.6) and (2.11)
below).

Consequently, the number of bubbles of type $i$ constituent does not exceed
$M_{i}/m_{i}^{-}$.

###### Proof.

Based on (1.12) and the stability condition

$\frac{\partial}{\partial
m_{i}^{k}}\left(\sum_{k=1}^{\infty}e_{0}(m^{k})\right)=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{j}^{k}+\frac{\partial
p}{\partial
m_{i}^{k}}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{j}^{k}+\frac{1}{r_{i}^{k}}=\lambda_{i},\qquad
i=1,2,3,$ (2.5)

where $\lambda_{i}$ is the Lagrange multiplier, clearly independent of $k$.
Recall that there exist universal constants $c_{2}>c_{1}>0$ such that

$c_{1}(r_{j}^{k})^{2}\leq m_{j}^{k}\leq c_{2}(r_{j}^{k})^{2},\qquad\forall
j,k.$ (2.6)

Case A. We first assume that $\min_{k}r_{i}^{k}$ and $\max_{k}r_{i}^{k}$
exist. By (2.5), both of them are positive. Let

$r_{i}^{1}=r_{i}^{\min}:=\min_{k}r_{i}^{k},\qquad
r_{i}^{2}=r_{i}^{\max}:=\max_{k}r_{i}^{k}.$

The aim is to bound $r_{i}^{1}$ from below. By (2.5) we get

$\lambda_{i}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{j}^{k}+\frac{1}{r_{i}^{k}}\qquad\forall
k\geq 1.$

In particular, taking $k=1,2$,

$\lambda_{i}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{j}^{1}+\frac{1}{r_{i}^{1}}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{j}^{2}+\frac{1}{r_{i}^{2}},$

hence

$\frac{1}{r_{i}^{1}}-\frac{1}{r_{i}^{2}}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}[m_{j}^{2}-m_{j}^{1}].$
(2.7)

Let

$\displaystyle\alpha:=r_{i}^{2}/r_{i}^{1}\geq 1.$ (2.8)

Two possibilities arise now.

Subcase A.1: $\alpha\geq 2$. In this case, (2.7) read

$\frac{(\alpha-1)}{\alpha
r_{i}^{1}}=\frac{1}{r_{i}^{1}}-\frac{1}{r_{i}^{2}}=\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}[m_{j}^{2}-m_{j}^{1}]\leq\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j},$

hence

$\frac{1}{r_{i}^{1}}\leq\frac{\alpha}{\alpha-1}\frac{1}{2\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j}\leq\frac{1}{\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j},$

where the last inequality is due to $\alpha\geq 2$, which gives
$\alpha/(\alpha-1)\leq 2$. Thus in this case we get

$r_{i}^{1}\geq\bigg{[}\frac{1}{\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j}\bigg{]}^{-1}.$
(2.9)

Subcase A.2: $\alpha\leq 2$. In this case, (2.7) alone is not sufficient to
give a lower bound on $r_{i}^{1}$. We must use the energy term. Note that, in
this case, all radii $r_{i}^{k}$ satisfy $r_{i}^{1}\leq r_{i}^{k}\leq
2r_{i}^{1}$. By (2.6), we have at least

$\frac{M_{i}}{\sup_{k}m_{i}^{k}}\geq\frac{M_{i}}{c_{2}\sup_{k}(r_{i}^{k})^{2}}=\frac{M_{i}}{c_{2}(r_{i}^{2})^{2}}\geq\frac{M_{i}}{4c_{2}(r_{i}^{1})^{2}}$
(2.10)

lobes of type $i$ constituent.

The total energy is not smaller than the sum of perimeters, which is, in turn,
bounded via the isoperimetric inequality by

$\sum_{k\geq 1}p(m_{1}^{k},m_{2}^{k},m_{3}^{k})\geq C_{1}\sum_{k\geq
1}\sqrt{m_{i}^{k}}\overset{\eqref{mass estimate}}{\geq}C_{2}\sum_{k\geq
1}r_{i}^{k}\geq\frac{C_{2}M_{i}}{4c_{2}r_{i}^{1}}$ (2.11)

for some universal geometric constants $C_{1},C_{2}>0$. The last inequality
follows from the fact that there are at least
$\frac{M_{i}}{4c_{2}(r_{i}^{1})^{2}}$ lobes of type $i$ constituent. Thus
(2.11) gives $r_{i}^{1}\geq\frac{C_{2}M_{i}}{4c_{2}\inf E_{0}}$. Combining
with (2.9) and (2.6) concludes the proof for this case.

Case B. We now remove the assumption that both $\min_{k}r_{i}^{k}$ and
$\max_{k}r_{i}^{k}$ exist. Consider sequences

$r_{i}^{1,\varepsilon(k)}\searrow r_{i}^{\min}:=\inf_{k}r_{i}^{k}\geq 0,\qquad
r_{i}^{2,\varepsilon(k)}\nearrow r_{i}^{\max}:=\sup_{k}r_{i}^{k},$

and denote by $m_{i}^{j,\varepsilon(k)}$, $j=1,2$, the associated masses.
Without loss of generality, we can ensure $r_{i}^{1,\varepsilon(k)}\leq
r_{i}^{2,\varepsilon(k)}$. Repeat the same arguments as Case A with replacing
$r_{i}^{j},m_{i}^{j}$ by $r_{i}^{j,\epsilon(k)},m_{i}^{j,\epsilon(k)}$
respectively . Similarly let

$\alpha(k):=r_{i}^{2,\varepsilon(k)}/r_{i}^{1,\varepsilon(k)}\geq 1,$

and again we have two cases.

Subcase B.1: $\inf_{k}\alpha(k)\geq 2$. Similar arguments yield

$r_{i}^{1,\varepsilon(k)}\geq\bigg{[}\frac{1}{\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j}\bigg{]}^{-1}.$
(2.12)

Subcase B.2: $\inf_{k}\alpha(k)\leq 2$. Note that, in this case, all radii
$r_{i}^{k}$ satisfy $r_{i}^{1,\varepsilon(k)}-\delta\leq r_{i}^{k}\leq
2r_{i}^{1,\varepsilon(k)}+\delta$, for some small $\delta$ which vanishes as
$\varepsilon(k)\to 0$. Similarly we get

$\displaystyle r_{i}^{1,\varepsilon(k)}-\delta\geq\frac{C_{2}M_{i}}{4c_{2}\inf
E_{0}}.$ (2.13)

Taking the limit $\varepsilon(k)\to 0$, which in turn forces $\delta\to 0$,
concludes the proof.

∎

To prove Theorem 1.1, we need first to get an upper bound for the mass of
lobes.

######  Lemma 2.3.

Let the total masses $M_{i}$ and interaction coefficients $\Gamma_{ij}$ be
given. Consider a local minimizer of $\overline{e_{0}}(M)$, and denote by
$B^{k}=(m_{1}^{k},m_{2}^{k},m_{3}^{k})$ its bubbles. Then

$\sup_{k}m_{i}^{k}\leq m_{i}^{+}:=\frac{8\pi}{\Gamma_{ii}^{2/3}},\qquad
i=1,2,3.$

###### Proof.

We write the proof for $i=1$ only, since $i=2,3$ are dealt with analogously.
Consider a bubble $B$ with lobes of masses $m_{1},m_{2},m_{3}$, and assume
$m_{1}$ is bigger than $m_{2}$ and $m_{3}$. The proof follows by comparing its
energy with that of the two bubbles $B^{\prime}$ and $B^{\prime\prime}$, where
$B^{\prime}$ has lobes of masses $m_{1}/2,m_{2},m_{3}$, and $B^{\prime\prime}$
is a single bubble of mass $m_{1}/2$. That is, we compare the energy of the
original minimizer with that of the competitor obtained by replacing $B$ with
$B^{\prime}\cup B^{\prime\prime}$. The former is

$E_{B}:=\frac{1}{4\pi}\sum_{i,j=1}^{3}\Gamma_{ij}m_{i}m_{j}+p(m_{1},m_{2},m_{3}),$

while the latter is $E_{B^{\prime}}+E_{B^{\prime\prime}}$, where

$\displaystyle E_{B^{\prime}}$
$\displaystyle:=\frac{1}{8\pi}\sum_{j=1}^{3}\Gamma_{1j}m_{1}m_{j}+\frac{1}{4\pi}\sum_{i,j=2}^{3}\Gamma_{ij}m_{i}m_{j}+p(m_{1}/2,m_{2},m_{3}),$
(2.14) $\displaystyle E_{B^{\prime\prime}}$
$\displaystyle:=\frac{\Gamma_{11}m_{1}^{2}}{16\pi}+\sqrt{2\pi m_{1}}.$ (2.15)

By the minimality of $B$, we need

$\displaystyle 0$ $\displaystyle\geq
E_{B}-E_{B^{\prime}}-E_{B^{\prime\prime}}=\frac{1}{8\pi}\sum_{j=1}^{3}\Gamma_{1j}m_{1}m_{j}-\frac{\Gamma_{11}m_{1}^{2}}{16\pi}+\Delta_{Per}$
$\displaystyle=\frac{\Gamma_{11}m_{1}^{2}}{16\pi}+\frac{1}{8\pi}\sum_{j=2}^{3}\Gamma_{1j}m_{1}m_{j}+\Delta_{Per},$
(2.16)

where

$\displaystyle\Delta_{Per}:=p(m_{1},m_{2},m_{3})-p(m_{1}/2,m_{2},m_{3})-\sqrt{2\pi
m_{1}}.$

Lemma 2.1 gives

$\displaystyle p(m_{1},m_{2},m_{3})-p(m_{1}/2,m_{2},m_{3})$
$\displaystyle=\int_{m_{1}/2}^{m_{1}}\frac{\partial p(m,m_{2},m_{3})}{\partial
m}\,\text{d}m=\int_{m_{1}/2}^{m_{1}}\frac{1}{r_{1}}\,\text{d}m\geq 0.$

Thus (2.16) reads

$\displaystyle 0$
$\displaystyle\geq\frac{\Gamma_{11}m_{1}^{2}}{16\pi}+\frac{1}{8\pi}\sum_{j=2}^{3}\Gamma_{1j}m_{1}m_{j}+\Delta_{Per}\geq\frac{\Gamma_{11}m_{1}^{2}}{16\pi}-\sqrt{2\pi
m_{1}},$

which concludes the proof. ∎

In the proof of Theorem 1.1, we will take $\Gamma_{ij}=0$ if $i\neq j$. Since
we will choose the total masses $M_{i}$ one after another, it is important
that the the lower bound $m_{i}^{-}$ can be made independent of $M_{j}$,
$j\neq i$. The next result shows that this is really the case.

######  Lemma 2.4.

Under the hypotheses $\Gamma_{ij}=0$ if $i\neq j$, then $m_{i}^{-}$ can be
made independent of $M_{j}$, $j\neq i$.

###### Proof.

Here we only discuss the scenario that $\min_{k}r_{i}^{k}$ and
$\max_{k}r_{i}^{k}$ exist. For the case of nonexistence, the proof is similar
to Case B in Proposition 2.2. Let $\alpha$ be the ratio defined in (2.8).
There are two cases.

Case $\alpha\geq 2$. Then (2.9) combined with (2.6) give directly

$\min_{k}m_{i}^{k}\geq c_{1}\min_{k}(r_{i}^{k})^{2}\geq
c_{1}\bigg{[}\frac{1}{\pi}\sum_{j=1}^{3}\Gamma_{ij}M_{j}\bigg{]}^{-2}=\frac{c_{1}\pi^{2}}{\Gamma_{ii}^{2}M_{i}^{2}}.$

Case $\alpha\leq 2$. By (2.6) we get

$\frac{m_{i}^{+}}{m_{i}^{-}}\leq\frac{c_{2}(r_{i}^{+})^{2}}{c_{1}(r_{i}^{-})^{2}}\leq\frac{4c_{2}}{c_{1}}.$

We analyze what happens if we remove a certain number, say $n$, of
bubbles/lobes of type $i$ constituent, and combine them into a single bubble.
Removing one such bubble/lobe decreases the energy at least by an amount

$c\sqrt{m_{i}}+\frac{1}{4\pi}\sum_{j=1}^{3}\Gamma_{ij}m_{i}m_{j}=c\sqrt{m_{i}}+\frac{\Gamma_{ii}m_{i}^{2}}{4\pi},$

for some geometric constant $c>0$, hence removing $n$ such bubbles/lobes
decreases the energy by

$E^{-}:=\sum_{k=1}^{n}c\sqrt{m_{i}^{k}}+\sum_{k=1}^{n}\frac{\Gamma_{ii}(m_{i}^{k})^{2}}{4\pi}.$

Adding a single bubble of mass $\sum_{k=1}^{n}m_{i}^{k}$ increases the energy
by

$E^{+}:=\frac{\Gamma_{ii}}{4\pi}\bigg{[}\sum_{k=1}^{n}m_{i}^{k}\bigg{]}^{2}+2\sqrt{\pi\sum_{k=1}^{n}m_{i}^{k}}.$

By the minimality of our configuration, we need

$\displaystyle 0\geq E^{-}-E^{+}$
$\displaystyle=\Delta_{Per}+\frac{\Gamma_{ii}}{4\pi}\bigg{\\{}\sum_{k=1}^{n}(m_{i}^{k})^{2}-\bigg{[}\sum_{k=1}^{n}m_{i}^{k}\bigg{]}^{2}\bigg{\\}},\qquad\Delta_{Per}:=c\sum_{k=1}^{n}\sqrt{m_{i}^{k}}-2\sqrt{\pi\sum_{k=1}^{n}m_{i}^{k}},$
$\displaystyle=\Delta_{Per}-\frac{\Gamma_{ii}}{4\pi}\sum_{k\neq
h}m_{i}^{k}m_{i}^{h}\geq\Delta_{Per}-\frac{n(n-1)\Gamma_{ii}}{4\pi}(m_{i}^{+})^{2}.$
(2.17)

On the other hand,

$\displaystyle\Delta_{Per}$
$\displaystyle=\frac{c^{2}(\sum_{k=1}^{n}\sqrt{m_{i}^{k}})^{2}-4\pi\sum_{k=1}^{n}m_{i}^{k}}{c\sum_{k=1}^{n}\sqrt{m_{i}^{k}}+2\sqrt{\pi\sum_{k=1}^{n}m_{i}^{k}}}=\frac{(c^{2}-4\pi)\sum_{k=1}^{n}m_{i}^{k}+c^{2}\sum_{k\neq
h}\sqrt{m_{i}^{k}m_{i}^{h}}}{c\sum_{k=1}^{n}\sqrt{m_{i}^{k}}+2\sqrt{\pi\sum_{k=1}^{n}m_{i}^{k}}}.$

Choose $c$ such that $c^{2}-4\pi$ be negative. The numerator is bounded from
below by

$\displaystyle(c^{2}-4\pi)\sum_{k=1}^{n}m_{i}^{k}+c^{2}\sum_{k\neq
h}\sqrt{m_{i}^{k}m_{i}^{h}}$
$\displaystyle\geq(c^{2}-4\pi)nm_{i}^{+}+c^{2}n(n-1)m_{i}^{-}$
$\displaystyle\geq\bigg{[}(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}\bigg{]}nm_{i}^{+},$

while the denominator is bounded from above by

$c\sum_{k=1}^{n}\sqrt{m_{i}^{k}}+2\sqrt{\pi\sum_{k=1}^{n}m_{i}^{k}}\leq[cn+2\sqrt{n\pi}]\sqrt{m_{i}^{+}}.$

Therefore,

$\Delta_{Per}\geq\bigg{[}(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}\bigg{]}\cdot[c+2\sqrt{\pi/n}]^{-1}\sqrt{m_{i}^{+}},$

and (2.17) now reads

$0\geq\bigg{[}(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}\bigg{]}\cdot[c+2\sqrt{\pi/n}]^{-1}\sqrt{m_{i}^{+}}-\frac{n(n-1)\Gamma_{ii}}{4\pi}(m_{i}^{+})^{2}.$
(2.18)

This inequality must hold for all $n\leq N_{i}$, defined as the number of
bubbles/lobes of type $i$ constituent. If for all such admissible $n$, we have

$(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}<0,$

then

$N_{i}\leq 1+\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}},$

where the right hand side term is clearly independent of $M_{j}$, $j\neq i$.
Now consider the case $N_{i}>1+\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}}$ Until
now, we have not chosen $n$ yet. So choose

$n:=\begin{cases}\bigg{\lceil}1+\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}}\bigg{\rceil},&\text{if
}\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}}\notin\mathbb{N},\\\
2+\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}},&\text{if
}\frac{4c_{2}(4\pi-c^{2})}{c^{2}c_{1}}\in\mathbb{N}.\end{cases}$

Such $n$ ensures $(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}>0$, and (2.18)
now gives

$(m_{i}^{+})^{3/2}\geq\frac{4\pi}{n(n-1)\Gamma_{ii}}\bigg{[}(c^{2}-4\pi)+\frac{c^{2}(n-1)c_{1}}{4c_{2}}\bigg{]}\cdot[c+2\sqrt{\pi/n}]^{-1},$

with the last term being purely geometric constant. Recalling that
$\frac{m_{i}^{+}}{m_{i}^{-}}\leq\frac{4c_{2}}{c_{1}}$ concludes the proof. ∎

###### Proof of Theorem 1.1.

Let $\Gamma$ be a diagonal matrix, i.e. $\Gamma_{ij}=0$ if $i\neq j$. Based on
(1.12) - (1.14):

  1. 1.

there cannot be two single bubbles of different type constituent. This because
merging them into a double bubble is energetically favorable.

  2. 2.

there cannot be a single bubble of type $i$ constituent, and a double bubble
without lobes of type $i$ constituent. This again because merging them into a
triple bubble is energetically favorable.

Note the number of bubbles/lobes of type $i$ constituent is between
$M_{i}/m_{i}^{+}$ and $M_{i}/m_{i}^{-}$. It is of crucial importance that the
bounds $m_{i}^{\pm}$ are independent of $M_{j}$ (see Lemmas 2.3 and 2.4),
since this allows $M_{i}/m_{i}^{\pm}$ to be completely independent of
$M_{j}/m_{j}^{\pm}$, $i\neq j$. Thus by choosing suitably large $M_{i}$, we
can ensure to have arbitrarily large $M_{i}/m_{i}^{+}$, which ensures an
arbitrarily large numbers of single/double/triple bubbles. Indeed, Choosing
$M_{2},M_{3}\geq M_{1}$, with $M_{1}/m_{1}^{+}>N_{1}$ ensures the presence of
at least $N_{1}$ triple bubbles. On the other hand, the number of lobes of
type $1$ constituent cannot exceed $M_{1}/m_{1}^{-}$.

Thus, the masses of type $i=2,3$ constituents in triple bubbles is at most
$m_{i}^{+}M_{1}/m_{1}^{-}$. Then choosing sufficiently large $M_{3}\geq
M_{2}$, such that $M_{i}/m_{i}^{+}>m_{i}^{+}N_{d}+m_{i}^{+}M_{1}/m_{1}^{-}$,
$i=2,3$, ensure the presence of at least $N_{2}$ double bubbles. Finally,
noting again that the number of double bubbles cannot exceed
$(M_{2}-m_{i}^{+}M_{1}/m_{1}^{-})/m_{2}^{-}$, so the mass of type $3$
constituent in double bubbles cannot exceed
$m_{3}^{+}(M_{2}-m_{i}^{+}M_{1}/m_{1}^{-})/m_{2}^{-}$, we can finally choose
large $M_{3}$ to ensure the presence of at least $N_{1}$ single bubbles.
Clearly, such constructions leads to all single bubble being of type $3$
constituent, and all double bubbles of types $2$ and $3$ constituents. The
proof is thus complete.

∎

##  3 Minimizers at $\eta$ level

Now we investigate the behavior of minimizers of $E_{\eta}$, $\eta>0$. The
next result shows that the number of connected components is again bounded
from above.

######  Lemma 3.1.

Let $M_{i}$ and $\Gamma_{ij}$ be given. Consider a sequence of minimizers
$v_{\eta}=(v_{1,\eta},v_{2,\eta},v_{3,\eta})$ of $E_{\eta}$, subject to the
mass constraint $\int_{\mathbb{T}^{2}}v_{i,\eta}=M_{i}$. Denote by

$v_{0}=(v_{1,0},v_{2,0},v_{3,0}),\qquad
v_{i,0}=\sum_{k=1}^{K}m_{i}^{k}\delta_{x^{k}},\quad\sum_{i=1}^{3}m_{i}^{k}>0,$

the limit measure such that $v_{\eta}\rightharpoonup v_{0}$, and
$E_{0}(v_{0})=\min E_{0}$. Then, for all sufficiently small $\eta>0$, the
number of connected components of each $v_{i,\eta}$ does not exceed
$\sharp\\{k:m_{i}^{k}>0\\}$, i.e. the number of nonzero masses $m_{i}^{k}$.

###### Proof.

The proof is completely analogous to the ternary case [3]. ∎

The next result shows that, both at $\eta>0$ and $\eta=0$ levels, two
connected components cannot be too close to each other.

######  Proposition 3.2.

Let $M_{i}$ and $\Gamma_{ij}>0$ be given. Consider a minimizer
$v_{\eta}=(v_{1,\eta},v_{2,\eta},v_{3,\eta})$ of $E_{\eta}$, subject to the
mass constraint $\int_{\mathbb{T}^{2}}v_{i,\eta}=M_{i}$. Then, denoting by
$\Omega_{i,\eta}^{k}$ the connected components of $v_{i,\eta}$, it holds

$\inf_{x\in\Omega_{i,\eta}^{k},\ y\in\Omega_{i,\eta}^{h},\ k\neq
h}|x-y|\geq\delta>0,$

where $\delta$ is some computable constant depending only on $M_{i}$ and
$\Gamma_{ij}$.

###### Proof.

Consider a sequence of minimizers $v_{\eta}$, such that
$E_{\eta}(v_{\eta})=\inf E_{\eta}$, and denote by $v_{0}$ the limit function,
corresponding to the $\eta=0$ level.

Denote by $\Omega_{i,\eta}^{k}$ the lobes of the connected components of
$v_{i,\eta}$, and by $K_{i}$ the number of connected components of
$v_{i,\eta}$, $i=1,2,3$. The main idea of the proof is that, if two connected
components are too close to each other, then the logarithmic part in the
nonlocal interaction term would be too large.

Step 1. Finding an uniform lower bound on the masses. Lemma 3.1 gives that,
for all sufficiently small $\eta$, the number of connected components of
$v_{i,\eta}$ does not exceed $\\#\\{k:m_{i}^{k}>0\\}$. Therefore,

$m_{i,\eta}^{+}:=\sup_{k}|\Omega_{i,\eta}^{k}|\geq\frac{M_{i}}{\\#\\{k:m_{i}^{k}>0\\}}>0.$

Note the right hand side is crucially independent of $\eta$. In particular,
there exists a lobe with radius $r_{i,\eta}^{+}\geq c_{1}m_{i,\eta}^{+}$, for
some geometric constant $c_{1}$. Since each $v_{\eta}$ is a minimizer, the
Lagrange multiplier satisfies

$\Lambda\leq\frac{1}{r_{i,\eta}^{+}}+\sum_{k=1}^{3}\frac{\Gamma_{ik}m_{k}}{2\pi}\leq\frac{1}{c_{1}m_{i,\eta}^{+}}+\sum_{k=1}^{3}\frac{\Gamma_{ik}M_{k}}{2\pi}\leq\frac{\\#\\{k:m_{i}^{k}>0\\}}{M_{i}}+\sum_{k=1}^{3}\frac{\Gamma_{ik}M_{k}}{2\pi}.$

Note that, crucially, the right hand side term is independent of $\eta$. Thus
any $m_{i,\eta}^{k}$ satisfies

$\Lambda\geq\frac{1}{r_{i,k}^{\eta}}\Longrightarrow
r_{i,k}^{\eta}\geq\Lambda^{-1}\Longrightarrow m_{i,\eta}^{k}\geq
c_{2}\Lambda^{-2}$ (3.1)

again for some geometric constant $c_{2}$.

Step 2. Constructing a grid. With the lower bound on the mass, we can now
focus our attention on bounding the lowest distance between two bubbles.
Denote by $C^{k}_{\eta}$ the bubbles of $v_{\eta}$. Note the main the
difference between $C^{k}_{\eta}$ and $\Omega^{k}_{i,\eta}$ is that the former
does not distinguish between the different lobes in the same bubbles, while
the latter does. Denote by

$d_{k,h,\eta}:=d_{\mathcal{H}}(C^{k}_{\eta},C^{h}_{\eta})=\max\left\\{\sup_{z\in
C^{h}_{\eta}}\inf_{z^{\prime}\in C^{k}_{\eta}}|z-z^{\prime}|,\sup_{z\in
C^{k}_{\eta}}\inf_{z^{\prime}\in C^{h}_{\eta}}|z-z^{\prime}|\right\\}$

the Hausdorff distance between two bubbles $C^{k}_{\eta}$, $C^{h}_{\eta}$.
Note the total number of bubbles $C^{k}_{\eta}$ does not exceed the total
number of lobes $\Omega^{k}_{i,\eta}$ across all the three constituent types.
Denote by $K$ such quantity. Choose some $n\geq\lceil\sqrt{K}\rceil+1$.
Clearly, as $K\leq 3\\#\\{k:m_{i}^{k}>0\\}$, such $n$ can also be taken
independent of $\eta$. Partition $\mathbb{T}^{2}$ into $n^{2}$ squares
$S_{k}$, each of which with side length $1/n$.

Step 3. Constructing a competitor. Denote by

$d_{\eta}^{-}:=\inf_{k\neq h}d_{k,h,\eta}$

the smallest distance between two bubbles, and let $(C^{k}_{\eta})^{-}$,
$(C^{h}_{\eta})^{-}$ the corresponding bubbles. We can always construct the
following competitor $u_{\eta}$:

  * •

let $O_{k}$ (resp. $B_{k}$) be the barycenter of $S_{k}$ (resp. $C^{k}_{\eta}$
). Replace $C^{k}_{\eta}$ by $C^{k}_{\eta}+O_{k}-B_{k}$. This ensures that in
each $S_{k}$, there is only one bubble $C^{k}_{\eta}$.

We now find the energy $E_{\eta}(u_{\eta})$: since the above construction is
done through a translation, all the perimeter and the self interaction terms

$\sum_{k}\int_{C_{\eta}^{k}\times
C_{\eta}^{k}}\left[-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y)\right]\,\text{d}x\,\text{d}y$

are unchanged. Since $v_{\eta}$ is a minimizer, we have
$E_{\eta}(u_{\eta})\geq E_{\eta}(v_{\eta})$, thus we need

$\displaystyle\sum_{k\neq
h}\int_{(C_{\eta}^{k}+O_{k}-B_{k})\times(C_{\eta}^{h}+O_{k}-B_{k})}$
$\displaystyle\left[-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y)\right]\,\text{d}x\,\text{d}y$
$\displaystyle\geq\sum_{k\neq h}\int_{C_{\eta}^{k}\times
C_{\eta}^{h}}\left[-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y)\right]\,\text{d}x\,\text{d}y.$
(3.2)

Since, by construction, $C_{\eta}^{k}+O_{k}-B_{k}$ is contained in
$B(O_{k},R\eta)$, and the side length of $S_{k}$ is $1/n$, and $S_{k}$
contains no other bubble, we get

$\inf_{k\neq
h}d_{\mathcal{H}}(C_{\eta}^{h}+O_{k}-B_{k},C_{\eta}^{k}+O_{k}-B_{k})\geq\frac{1}{n}-2R\eta\geq\frac{1}{2n}$

for sufficiently small $\eta$. The left hand side term of (3.2) can be
estimated through

$\displaystyle\sum_{k\neq h}$
$\displaystyle\int_{C_{\eta}^{k}\times(C_{\eta}^{h}+O_{k}-B_{k})}\left[-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y)\right]\,\text{d}x\,\text{d}y$
$\displaystyle\leq\sum_{k\neq
h}\int_{C_{\eta}^{k}\times(C_{\eta}^{h}+O_{k}-B_{k})}\left[-\frac{1}{2\pi}\log\frac{1}{2n}+\sup
R_{\mathbb{T}^{2}}\right]\,\text{d}x\,\text{d}y=\left[-\frac{1}{2\pi}\log\frac{1}{2n}+\sup
R_{\mathbb{T}^{2}}\right]\sum_{k\neq h}|C_{\eta}^{k}||C_{\eta}^{h}|.$ (3.3)

Similarly, the right hand side of (3.2) can be estimated through

$\displaystyle\sum_{k\neq h}$ $\displaystyle\int_{C_{\eta}^{k}\times
C_{\eta}^{h}}\left[-\frac{1}{2\pi}\log|x-y|+R_{\mathbb{T}^{2}}(x-y)\right]\,\text{d}x\,\text{d}y$
$\displaystyle\geq|(C^{k}_{\eta})^{-}||(C^{k}_{\eta})^{-}|\left[-\frac{1}{2\pi}\log(d_{\eta}^{-}+2R\eta)+\inf
R_{\mathbb{T}^{2}}\right]+\sum_{k\neq
h}|C_{\eta}^{k}||C_{\eta}^{h}|\left[-\frac{1}{2\pi}\log\sqrt{2}+\inf
R_{\mathbb{T}^{2}}\right].$ (3.4)

Combining with (3.3) gives the necessary condition

$\displaystyle|(C^{k}_{\eta})^{-}||(C^{k}_{\eta})^{-}|\bigg{[}-\frac{1}{2\pi}\log(d_{\eta}^{-}+2R\eta)+\inf
R_{\mathbb{T}^{2}}\bigg{]}$ $\displaystyle+\sum_{k\neq
h}|C_{\eta}^{k}||C_{\eta}^{h}|\left[-\frac{1}{2\pi}\log\sqrt{2}+\inf
R_{\mathbb{T}^{2}}\right]$
$\displaystyle\leq\left[-\frac{1}{2\pi}\log\frac{1}{2n}+\sup
R_{\mathbb{T}^{2}}\right]\sum_{k\neq h}|C_{\eta}^{k}||C_{\eta}^{h}|.$

Finally, noting that the masses $|C_{\eta}^{k}|$ are bounded from above by
$M_{1}+M_{2}+M_{3}$, and from below by (3.1), both of which are clearly
independent of $\eta$, we conclude that $d_{\eta}^{-}\geq\delta$ for some
$\delta$ independent of $\eta$ too. ∎

## References

  * [1] E. Acerbi, N. Fusco, V. Julin and M. Morini. Nonlinear stability results for the modified Mullins-Sekerka and the surface diffusion flow. J. Differential Geom., 113(1):1-53, 2019.
  * [2] E. Acerbi, N. Fusco, and M. Morini. Minimality via second variation for a nonlocal isoperimetric problem. Comm. Math. Phys., 322(2): 515-557, 2013.
  * [3] S. Alama, L. Bronsard, X. Lu and C. Wang, Periodic minimizers of a ternary non-local isoperimetric problem. Indiana Univ. Math. J., 70(6):2557-2601,2021.
  * [4] S. Alama, L. Bronsard, I. Topaloglu. Sharp Interface Limit of an Energy Modeling Nanoparticle-Polymer Blends. Interfaces and Free Boundaries, 18(2): 263-290, 2016.
  * [5] S. Alama, L. Bronsard, Rustum Choksi, I. Topaloglu. Droplet breakup in the liquid drop model with background potential. Commun. Contemp. Math., 21(3), 1850022, 2019.
  * [6] S. Alama, L. Bronsard, Rustum Choksi, I. Topaloglu. Droplet phase in a nonlocal isoperimetric problem under confinement. Comm. Pure Appl. Anal., 19, 175-202, 2020.
  * [7] G. Alberti, R. Choksi and F. Otto. Uniform energy distribution for an isoperimetric problem with long-range interactions. J. Amer. Math. Soc., 22(2):569-605, 2009.
  * [8] L. Ambrosio, N. Fusco and D. Pallara. Functions of Bounded Variation and Free Discontinuity Problems. The Clarendon Press, Oxford University Press, New York, 2000.
  * [9] F.S. Bates and G.H. Fredrickson, Block copolymers - designer soft materials.  Phys. Today, 52(2): 32-38, 1999.
  * [10] M. Bonacini and R. Cristoferi. Local and global minimality results for a nonlocal isoperimetric problem on $\mathbb{R}^{N}$. SIAM J. Math. Anal., 46(4): 2310-2349, 2014.
  * [11] M. Bonacini and H. Knüepfer. Ground states of a ternary system including attractive and repulsive Coulomb-type interactions. Calc. Var. Partial Dif., 55:114, 2016.
  * [12] R. Choksi, R. Neumayer and I. Topaloglu. Anisotropic liquid drop models, submitted.
  * [13] R. Choksi and M.A. Peletier, Small Volume Fraction Limit of the Diblock Copolymer Problem: I. Sharp Interface Functional. SIAM J. Math. Anal., 42(3):1334-1370, 2010.
  * [14] R. Choksi and X. Ren. On the derivation of a density functional theory for microphase separation of diblock copolymers. J. Statist. Phys., 113:151-176, 2003.
  * [15] R. Choksi and X. Ren. Diblock copolymer-homopolymer blends: derivation of a density functional theory. Physica D. 203(1-2):100-119, 2005.
  * [16] R. Choksi and P. Sternberg. On the first and second variations of a nonlocal isoperimetric problem. J. Reine Angew. Math., 611, 75-108, 2007.
  * [17] M. Cicalese and E. Spadaro. Droplet minimizers of an isoperimetric problem with long-range interactions. Comm. Pure Appl. Math., 66(8):1298-1333, 2013.
  * [18] R. Cristoferi. On periodic critical points and local minimizers of the Ohta-Kawasaki functional. Nonlinear Anal., 168, 81-109, 2018.
  * [19] M. Fall. Periodic patterns for a model invollving short-range and long-range interactions. Nonlinear Anal., 175:73-107, 2018.
  * [20] A. Figalli, N. Fusco, F. Maggi, V. Millot and M. Morini. Isoperimetry and stability properties of balls with respect to nonlocal energies. Comm. Math. Phys., 336(1):441-507, 2015.
  * [21] J. Foisy, M. Alfaro, J. Brock, N. Hodges, and J. Zimba. The standard double soap bubble in $\mathbb{R}^{2}$ uniquely minimizes perimeter. Pacific J. Math., 159(1):47-59, 1993.
  * [22] R. Frank and E. Lieb. A compactness lemma and its application to the existence of minimizers for the liquid drop model. SIAM J. Math. Anal.,47(6):4436-4450, 2015.
  * [23] K. Glasner. Evolution and competition of block copolymer nanoparticles.  SIAM J. Appl. Math., 79(1):28-54, 2019.
  * [24] K. Glasner and R. Choksi. Coarsening and self-organization in dilute diblock copolymer melts and mixtures. Phys. D, 238:1241-1255, 2009.
  * [25] D. Goldman, C.B. Muratov and S. Serfaty. The $\Gamma$-limit of the two-dimensional Ohta-Kawasaki energy. I. droplet density. Arch. Rat. Mech. Anal., 210(2):581-613, 2013.
  * [26] J. Hass and R. Schlafly. Double bubbles minimize. Ann. Math., 151(2):459-515, 2000.
  * [27] M. Hutchings, R. Morgan, M. Ritoré, and A. Ros. Proof of the double bubble conjecture. Ann. Math., 155(2):459-489, 2002.
  * [28] M. Helmers, B. Niethammer, and X. Ren. Evolution in off-critical diblock copolymer melts. Netw. Heterog. Media, 3:615-632, 2008.
  * [29] C. Isenberg. The science of soap films and soap bubbles. Dover Publications, 1992.
  * [30] V. Julin, Isoperimetric problem with a Coulomb repulsive term. Indiana Univ. Math. J., 63(1): 77-89, 2014.
  * [31] V. Julin and G. Pisante. Minimality via second variation for microphase separation of diblock copolymer melts. J. Reine Angew. Math., 729, 81, 2017.
  * [32] V. Julin. Remark on a nonlocal isoperimetric problem. Nonlinear Anal., 154:174-188, 2017.
  * [33] J. Lu and F. Otto. An isoperimetric problem with Coulomb repulsion and attraction to a background nucleus, preprint, arXiv:1508.07172, 2015.
  * [34] H. Knüpfer and C.B. Muratov. On an isoperimetric problem with a competing non-local term. I. The planar case.  Comm. Pure Appl. Math., 66, 1129?1162, 2013.
  * [35] H. Knüpfer and C.B. Muratov. On an isoperimetric problem with a competing non-local term. II. The general case.  Comm. Pure Appl. Math., 67, 1974-1994, 2014.
  * [36] P.L. Lions, The Concentration-Compactness Principle in the Calculus of Variations. The Limit Case, Part I.  Rev. Mat. Iberoamercana, 1(1):145-201, 1984.
  * [37] M. Morini and P. Sternberg. Cascade of minimizers for a nonlocal isoperimetric problem in thin domains. SIAM J. Math. Anal., 46(3):2033-2051, 2014.
  * [38] H. Nakazawa and T. Ohta. Microphase separation of ABC-type triblock copolymers. Macromolecules, 26(20):5503-5511, 1993.
  * [39] Y. Nishiura and I. Ohnishi. Some mathematical aspects of the micro-phase separation in diblock copolymers. Phys. D, 84, 31-39, 1995.
  * [40] F. Maggi. Sets of finite perimeter and geometric variational problems: An introduction to geometric measure theory. New York: Cambridge University Press. 2012.
  * [41] N. Min, T. Choi, S. Kim. Bicolored Janus microparticles created by phase separation in emulsion drops. Macromol. Chem. Phys., 218:1600265, 2017.
  * [42] Y. Mogi, et al., Superlattice structures in morphologies of the ABC Triblock copolymers. Macromolecules, 27(23): 6755-6760, 1994.
  * [43] C.B. Muratov. Droplet phases in non-local Ginzburg-Landau models with Coulomb repulsion in two dimensions. Comm. Math. Phys., 299(1):45-87,2010.
  * [44] Y. Oshita. Singular limit problem for some elliptic systems. SIAM J. Math. Anal., 38(6):1886-1911, 2007.
  * [45] T. Ohta and K. Kawasaki. Equilibrium morphology of block copolymer melts. Macromolecules, 19(10):2621-2632,1986.
  * [46] X. Ren and C. Wang. A stationary core-shell assembly in a ternary inhibitory system. Discrete Contin. Dyn. Syst., 37(2):983-1012, 2017.
  * [47] X. Ren and C. Wang. Stationary disk assemblies in a ternary system with long range interaction. Commun. Contemp. Math., 1850046, 2018.
  * [48] X. Ren and J. Wei. On the multiplicity of solutions of two nonlocal variational problems. SIAM J. Math. Anal., 31(4):909-924, 2000.
  * [49] X. Ren and J.Wei. Triblock copolymer theory: Ordered ABC lamellar phase. J. Nonlinear Sci., 13(2):175-208, 2003.
  * [50] X. Ren and J.Wei. Many dropet pattern in the cylindrical phase of diblock copolymer morphology. Rev. Math. Phys., 19(8):879-921, 2007.
  * [51] X.Ren and J.Wei. A double bubble assembly as a new phase of a ternary inhibitory system. Arch. Rat. Mech. Anal., 215(3):967-1034, 2015.
  * [52] X. Ren and J. Wei. A double bubble in a ternary system with inhibitory long range interaction. Arch. Rat. Mech. Anal., 46(4):2798-2852, 2014.
  * [53] H. A. Schwarz. Beweis des Satze, dass die Kugel kleinere Oberfläche besitzt, als jeder andere Körper gleichen Volumens. Nach. Königlichen Ges. Wiss. Göttingen, pages 1-13, 1884.
  * [54] P. Sternberg and I. Topaloglu. A note on the global minimizers of the nonlocal isoperimetric problem in two dimensions. Interfaces Free Bound., 13(1):155-169, 2011.
  * [55] I. Topaloglu. On a nonlocal isoperimetric problem on the two-sphere. Comm. Pure Appl. Anal., 12(1):597-620, 2013.
  * [56] W. Wichiramala. Proof of the planar triple bubble conjecture. J. Reine Angew. Math., 567:1-49, 2004.


# Generalization of Fitness Exercise Recognition from Doppler Measurements by
Domain-adaption and Few-Shot Learning

Biying Fu1,2, Naser Damer1,2, Florian Kirchbuchner1,2, Arjan Kuijper1,2
1Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt, Germany
2Mathematical and Applied Visual Computing, TU Darmstadt, Darmstadt, Germany
Email<EMAIL_ADDRESS>
###### Abstract

In previous works, a mobile application was developed using an unmodified
commercial off-the-shelf smartphone to recognize whole-body exercises. The
working principle was based on the ultrasound Doppler sensing with the device
built-in hardware. Applying such a lab-environment trained model on realistic
application variations causes a significant drop in performance, and thus
decimate its applicability. The reason of the reduced performance can be
manifold. It could be induced by the user, environment, and device variations
in realistic scenarios. Such scenarios are often more complex and diverse,
which can be challenging to anticipate in the initial training data. To study
and overcome this issue, this paper presents a database with controlled and
uncontrolled subsets of fitness exercises. We propose two concepts to utilize
small adaption data to successfully improve model generalization in an
uncontrolled environment, increasing the recognition accuracy by two to six
folds compared to the baseline for different users.

##  I Introduction

Human activity recognition (HAR) covers a wide range of application areas.
Understanding human actions in daily living enables application designers to
build assisting smart home applications for elderly care [1], security
applications with video surveillance [2], applications for Quantified-self [3,
4], or associate physiological signals with emotions [5] to build interactive
applications. HAR is the key to enable human-centered application designs and
natural interaction in a smart environment [6].

However, human motion is highly complex and possesses a high degree of
freedom. This is expressed with the term user-diversity. Learning a
generalized model for all possible variations of a human motion is very
challenging. Training a model on limited amount of individuals under
constrained environment often leads to a large performance drop when applying
the model on individuals/environments disjoint from the training data. This
reduction in performance originates from the large variations between the
controlled training data and the real-world application scenarios. It is
caused by the diversity and complexity of the users actions, the device
hardware or other environmental variations. However, all possible reasons lead
to a degradation in the usability of the proposed application. One possible
solution is to reduce the inherent difference between the development dataset
and the real-world dataset by making the development data resemble the real-
world data. However, due to the diversity in the real-world applications,
there is no generalized model that is applicable in all possible situations.

This work addresses sport exercise recognition from a stationary smartphone
using the Doppler measurement. We propose a set of methods to improve the
generalization of a pre-trained model (trained on controlled data) to
scenarios containing a combination of unseen environments, individuals, and
devices. To achieve this, we propose and investigate two concepts, along with
a clear baseline that demonstrate the generalization problem. We have
developed a mobile application that aims at collecting data that is used to
deal with this challenge. Our application is based on the built-in hardware of
a commercial smartphone to measure whole-body exercise activities. The main
contribution of this work is grouped as follows:

  * •

A novel database for investigating micro-Doppler motion in relation to whole-
body exercise data with built-in smartphone hardware. The database contains
sessions in controlled environment, as well as a disjoint subset containing
variations of environments, individuals, and devices.

  * •

Propose and adapt two concepts (with variations) to improve the recognition
generalization. These concepts are based on domain adaptions, as well as few-
shot learning. Both concepts proved to enhance the generalizability on data
variations in comparison to a clear baseline.

The structure of the paper is organized as follows: in section II we provide
current researches on the topic of finetuning approaches with the focus on
model generalization to fit new data and categorized these approaches under
two main categories (retrain required and not). In Section III we introduce
one of the main contributions of this paper by presenting our collected
database. We first motivate the need for such a database. We then introduce
the sensing principle and describe the details about the database and what it
enables us to study. In section IV, we propose the baseline model and the new
approaches targeting our problem, under two main concepts. Section V
introduces evaluation results of our proposed individual approaches, along the
baseline. We further discuss the advantages and disadvantages of certain
methods and provide some guidelines in design choice for such an application.
Finally, we conclude our work in Section VI and provide relevant future
research directions.

##  II Related Works

Andrew Ng once stated [7], that the most frequent fail of inference model in
reality is the inherent difference between your development set and test set.
To overcome this problem, effort can be put into developing highly-
representative development/training databases. Though, it is impossible to
make the development set identical to the test set (application scenario), due
to the large variety and complexity in human activities. Common methods to
adapt the pre-trained model on individual new data samples without enforcing
much restrictions on the development set is thus desirable. We distinguish
between two main categories: with and without retraining the base model to
adapt to new data.

#### Retraining of the Base Model

Domain adaptation builds on transferring knowledge from similar domains to
cope with unknown target domains. This method is especially useful, when you
do not have enough annotated datasets for the particular problem at hand. The
goal is to extract knowledge from related, known datasets, and use this
knowledge to learn the new task at hand.

Wang [8] benefited from using similar labeled source domain data to annotate
the target domain that has only a few or even no labels. They evaluated their
approach on acceleration dataset from different body positions as different
domains. To alleviate the problem of negative knowledge transfer, they
proposed an unsupervised similarity measure to choose the right source domain
with respect to the target domain. Khan and Roy et al. [9] proposed a CNN
based transductive transfer learning model to adapt action recognition
classifiers trained in one context to be applicable to a different contextual
domain. The limitation is that the set of activities being monitored is the
same in both context domains, as they are transferring knowledge from
individual convolutional layers. Evaluated on their acquired smartphone and
smartwatch acceleration dataset on 15 users and 8 activities, they
demonstrated the ability of their proposed methods on transferring knowledge
from smartphone to smartwatch domain and vice versa. By incorporating a small
amount of target labels, they were able to further increase the performance.

These methods pose less constraints on the target domain, however are
difficult to train, as the knowledge transfer is solely based on the source
domains. Thus the choice of the appropriate source domain is critical for the
performance of the inference model on the unknown target domain. A retraining
is required to relate the source domain to the new target domain due to
knowledge transfer.

#### No Retraining of the Base Model

Few-shot learning is currently an active research field in machine learning.
The ability of deep neural networks to learn complex correlations and patterns
from a vast dataset is proven. However, current deep learning approaches
suffer from the problem of poor sample efficiency. To make a model learn on a
new class, sufficient amount of labeled samples from this class is required to
avoid overfitting.

Few-shot learning methods increase the model generalizability with limited
data. They have been mostly used in image classification tasks [10, 11]. Feng
[12] recently applied few-shot learning-based classification on human activity
recognition tasks. They applied a deep learning framework to automatically
extract features and perform classification. However, instead of transferring
knowledge in a common feature space, their proposed method tends to perform
knowledge transfer in terms of model parameter transfer. Based on two
benchmark datasets, they evaluated their proposed technique. A metric to
measure the cross-domain class-wise relevance is introduced to mitigate the
challenging issue of negative transfer. These datasets consist of only sparse
sensory input, mostly acceleration sensor data attached to different body
parts.

We consider this category to be the most realistic in designing applications
for human activity recognition tasks with sensory data. Since we can not adopt
to the complexity and diversity of all persons actions during the training
phase, we need the network to have the ability to adapt to individual users by
introducing only a small amount of this users data to optimize the trained
network. Few-shot classification methods are methods that can be leveraged to
unseen classes, even when less labels from these classes are available without
retraining the base model.

Exercise detection on personal devices is often applied to track daily
ambulation activities [13, 14]. For tracking of more stationary activities
involving whole-body interaction, a remote system is better than wearing a
smartphone on the body, as the detection is more unobtrusive. However, the
complexity and diversity in human action makes it difficult to develop one
single system to fits all situations. To overcome this issue, it requires more
advanced machine learning methods to improve the model generalization.

##  III Ultrasonic smartphone exercises: sensing and database

In this paper, we contribute a database collected with built-in hardware of
commercial smartphones. This database deals with exercise data using Doppler
sensing by utilising the smartphone as a sonar device. Using smartphone to
collect activity data is not novel per se. Most existing databases are,
however, focused on acceleration data using the smartphone as a wearable
device. Popular databases are for example provided in the works [15, 16, 13].
Though tracking and recognizing for various aerobic training exercises are
popular, there exists limited research on recognizing more stationary
exercises, such as strength-based training without the use of wearable. These
exercises are essential for rehabilitation purposes [17]. Typically they are
even harder to track, as they rely on coordinated movement of specific body
parts. Instead of building customized hardware designs such as in [3, 4] to
target these exercises, we leveraged the existent infrastructure of a
commercial mobile device to build an ubiquitous application. The set of
stationary and strength-based exercises are depicted in Figure 1.

|  |  |
---|---|---|---
push-up | sit-up | squat | segmental rot.
|  |  |
trunk rot. | swim | bridge | quadruped
Figure 1: Figure shows the eight workout exercises we present in our novel
database. The smartphone is placed roughly 50 cm away from the body of the
performing user.

###  III-A Sensing Principle

By emitting a 20 kHz continuous audio signal from the built-in speaker, we
turned the commercial smartphone into an active sonar device. The echo encoded
with the Doppler modulation is received from the device internal microphone.
Doppler measurement allows us to catch relative movement in close range to the
sensing device. A positive Doppler is received with a relative speed towards
the device and a negative Doppler vice versa. The device speaker can typically
emit tones in the range of 18-22 kHz on a commodity audio system without
performance degradation. Thus, we can detect a one-sided Doppler speed up to
$17.4\frac{m}{s}$ (2.05 kHz). To extract the relative movement from Doppler
signal, a Short Time Fourier Transformation (STFT) [18] is used to convert the
continuous time signal to the spectrum domain of interest. The parameter of
STFT determines the resolution in time and frequency domain. The selected
frequency resolution corresponds to a relative speed of $3\frac{cm}{s}$ (3.6
Hz) with a time resolution of 46.5 ms. A sliding window approach of 6 s
windows and an overlap of 50 % are chosen to prepare data samples used to
train the classification task. To further reduce the computational effort, we
restrict the spectrum amplitudes within frequency band between 19.5 kHz to
20.5 kHz, as other signals beyond the motion information are irrelevant. A
typical push-up exercise takes around 2-3 s each [19], depending on the
individual fitness. We conclude that with the current system setting, we are
able to resolve both the slowest and fastest movement of the targeted exercise
set.

###  III-B Database

This database allows us to study the body motion in relation to Doppler
profiles from built-in hardware of various commercial smartphones. The effect
of fine-grained movements from both limbs and arms cause micro-Doppler
patterns in addition to the main Doppler reflection. Studying these micro-
Doppler events enhances the ability of recognizing more complex and
naturalistic human activities including whole-body interactions. To the best
of our knowledge, there does not exist such a database so far. Due to the
similarity of ultrasound sensing and electromagnetic waves in physical
characteristics, this database can also be leveraged for machine learning
practitioners to design radar-based applications and gain useful insight
without the additional cost of customized hardware.

The presented database consist of two different setups, in order to
investigate the effects of various methods on improving model generalization
for different data distributions. Data of the first setup is called Lab-Data.
The Lab-Data consists of data collected in a laboratory setup as depicted in
Figure 1 from 14 individuals. The group consists of 4 females (157 cm-172 cm)
and 10 males (172 cm-193 cm). The affinity towards sport exercises varies
across the test participants. The built-in microphone of the sensing device is
placed 50 cm apart, facing the exercising individuals on the floor, aligned
with the hip as depicted in Figure 1. For each individual, two separate
sessions were collected with 10 repetitions of each exercise class. Left and
right variations for exercises such as segmented rotation, trunk rotation, and
quadruped are counted as one repetition. Swim is performed in average for 30 s
in each session to reach similar time duration comparable to the other
exercise types. In order to collect the micro-Doppler motions from the arms,
the device is aligned with the shoulder for swim and trunk rotation. The
duration of each session is approximately 7-9 minutes in average. The
smartphone used for data acquisition has the brand Samsung Galaxy A6 (2018)
and the placement of the sensing device to the exercising body is constrained
to the same position for all participants.

The goal of the second setup is to be leveraged on testing various finetuning
approaches, as these data are collected under individual, different hardware,
and uncontrolled environments independent of the Lab setup. This part of data
is called the Uncontrolled-Data. It consists of data collected from five
different individuals. Due to logistic and privacy constrains related to the
experimental setup, the second setup contains a smaller number of participants
compared to the Lab-Data. The hardware device is not limited to the smartphone
used in the Lab-Data. Each individual was asked to collect eight individual
sessions distributed over several days in their familiar surroundings and
without any supervision. Each session has a comparable length to the collected
data from the Lab-Data. Some general statistics about the participants and the
hardware devices used, are listed in Table I.

Participant | Sex | Height | Exercise Frequency | Device | Location
---|---|---|---|---|---
P1 | male | 180 cm | Frequent | SONY Xperia XZ2 Compact | Environment 1
P2 | male | 181 cm | Frequent | Samsung Galaxy A5 (2017) | Environment 2
P3 | male | 181 cm | Frequent | SONY Xperia Z5 Dual | Environment 3
P4 | male | 182 cm | Frequent | Samsung Galaxy A6 (2018) | Environment 4
P5 | female | 168 cm | Less Frequent | SONY Xperia XZ2 Compact | Environment 1
TABLE I: Description of Software and Hardware Setups for the Uncontrolled-
Data.

The data acquisition app is installed on the individual mobile device. The
participants from the uncontrolled setup were asked to collected data from
their home environment to simulate the real-world scenarios. Figure 2
illustrates the different data acquisition environments that affect the signal
strength of the underlying hardware device. In contrast to the Lab setup, the
other apartments all have wooden floors which makes the back reflected signal
strength stronger compared to the Lab setup.

|  |  |
---|---|---|---
Environment 1 | Environment 2 | Environment 3 | Environment 4
Figure 2: Illustration depicts the four different data acquisition setups from
the individual test participants.

This paper aims at investigating methods to improve the generalizability of
pre-trained models on new individuals under more realistic conditions. Based
on the underlying method, we split the data as follows:

  * •

Basic training data contain all individuals and sessions of the Lab-Data.

  * •

Subject development data contains 4 sessions (out of 8) of each of the users
in the uncontrolled setup, the Uncontrolled-Data.

  * •

Testing data contains the 4 sessions (out of 8) of each of the users in the
uncontrolled setup, the Uncontrolled-Data, that were not used in the Subject
development data.

##  IV Methods

This paper is motivated by our observations from our previous experiences in
realistic use of mobile device to detect workout exercises without using
additional external hardware [20]. However, previous work faced a major
usability challenge as it did not adapt well on individuals and environments
unseen in the training phase. This issue is the main target of the methods
presented in this paper. To state the problem, we observed the input signal of
the same participant performing the same set of physical activities under two
different environments with the same hardware and sensor position. Despite the
overall speed and appearance remain similar, the strength and noise embedded
reveal a strong difference in both settings. We noticed a strong decay in
signal power due to the material-dependent attenuation of the transmit power.

Realistically, we can not train a classifier adapting to every possible
sensing environment, unless our training data unrealistically contain
unlimited variations. The quality of hardware devices integrated in the
smartphone may also introduce strong variations in the signal power. But the
basic physical characteristics remain. To adapt to new, real-world
circumstances, we need to individually finetune the trained model. In this
paper, we investigate several approaches to improve the model generalizability
given limited individual data.

###  IV-A The baseline method

The base inference model is built using a stacked bidirectional LSTM network.
To baseline our proposed solution, we need to demonstrate the exercises
detection performance when the uncontrolled environment is not considered. The
choice of using this sequence model is due to its ability to consider the
global and sequence structure within a sample time window. The architecture of
this sequence modeling network consists of 2 stacked bidirectional LSTM layers
with 128 hidden nodes in each LSTM cell. For each input node, a slice of
spectrogram with the frequency bands (ranging from 19.5 kHz to 20.5 kHz) from
a time step resolution (46.5 ms) is provided to the network. The bidirectional
structure permits the network to look forward and backward in time to extract
fine-grained sequence information from the spectrum domain.

An 2-D instance normalization layer is applied on the sample spectrogram prior
to the network input in order to reduce the hardware specific power
dependencies. The number of output classes are nine that includes the eight
true activity classes with the additional none class describing all the
transitions and noisy samples between two successive action classes. The
learning rate is set to 0.001 and the Adam Optimizer is used to optimize the
network parameters. Cross entropy loss is used as the cost function to
minimize the loss of the misclassification error from the training samples.
Batch-wise training is used, while in each batch, 15 samples of each class are
randomly selected from the training data to construct similar training
procedures compared to other network structures.

The training contains the Lab-Data only. Subject development data with 4
sessions from the Uncontrolled-Data is used in the validation stage, while the
4 sessions of the remaining Uncontrolled-Data is used in the test phase.
Batch-wise approach is applied, while each batch contains 15 samples randomly
selected from each class.

###  IV-B Our proposed method 1: Domain adaptation

To improve the model generalization ability, the first method we propose is
from the domain adaptation (DA) network family. DA is commonly used to
transfer knowledge from labeled source domain to target domain where data is
unlabeled or only partially labeled. In our case, the source domain refers to
the Lab-Data, while the target domain refers to the Uncontrolled-Data
collected by the different individuals under changed conditions. The aim of
such an adaption network is to make the distribution of the source and target
embeddings in the common embedding space more similar, such that the
classification network works well on both domains. By applying this
architecture, we aim at adopting the base feature extractors to be sensitive
to deterministic features from the Uncontrolled-Data domain with the knowledge
draw from the source domain. In such a way, the classification performance
does improve for the uncontrolled setup.

The metric to measure the similarity of both distributions is based on the
maximum mean discrepancy (MMD) on the final feature level. The model
architecture of the adaptation network is depicted in Figure 3. Common
approaches using domain adaption do not require to include target labels.
However, without any label information from the target domain, the knowledge
transfer does not work well on the targeted use-case, as both domains are
quite dissimilar. In order to improve the knowledge transfer characteristic,
we included partial labels (50 %) from the subject development set of the
Uncontrolled-Data to increase the performance on feature level adaptation. An
instance normalization layer is used prior to the ConvNet to mitigate the
hardware dependent effects of the transmit power from different smartphone
models.

Figure 3: depicts the domain adaptation model. Source and target data are
projected into the same embedding space using the common base ConvNet.
Adaptation is realized by minimizing the classification losses for source and
target and reducing differences in both feature distributions.

A base ConvNet is used to extract common features from source and target
domain. The ConvNet structure consists of 4 successive convolutional layers,
each followed by a batch normalization layer, leaky rectified linear unit
(ReLU) activation layer with the negative slope coefficient of 0.2 and a max
pooling layer to reduce the input dimensions. The successive layers increased
the number of filters from 32-32-64-64. The same ConvNet structure is used for
all networks as the feature extraction component in this work. The embeddings
in the embedding space are used to minimize the classification loss of the
source and target domain. The objective is to minimize the combination of
three different losses as given in Equation (1) : both cross entropy losses
from the source and target domain classification, and the MMD loss of the
feature embeddings originated from the source and target domain in the same
embedding space.

$\mathcal{L}=\mathcal{L}(g_{\phi}(f_{\theta}(\textbf{x}_{s})),y_{s})+\mathcal{L}((g_{\phi}(f_{\theta}(\textbf{x}_{t})),y_{t})+\mathcal{L}_{MMD}$
(1)

The network is trained with 100 epochs. Each batch composites of 15 samples
from each of the 9 classes. Adam optimizer with a learning rate of 0.001 is
set to learn the network hyperparameters. Since the base feature extraction
network needs to be optimized on both data domains, a retraining of the base
model is required. We applied the Lab-Data as the source domain and the
uncontrolled subject development set as the target domain. The adapted model
is evaluated on the testing data of the uncontrolled setup.

This method is good to adopt the feature extraction layers to work for
features from different domains. A negative aspect is that if both domains
differ too much, it could lead to a negative adaptation and causing the
performance on the source data to decrease. To address this issue, we present
the following methods which do not need to retrain the pre-trained inference
model to finetune to new individuals. We benefit from a few labels of the new
datasets to label this unknown dataset.

###  IV-C Our proposed method 2: Few-shot classification

This section deals with three methods from the research domain of the few-shot
classification learning. The network is trying to learn common features within
a subset of tasks without retraining.

#### F1: Siamese Network with Few-Shot Classification

The Siamese network consists of two identical feature extraction base networks
with shared weight parameters. The learned feature embeddings from both inputs
are then compared with each other to form a similarity score. Commonly it is
used for verification tasks and the score indicates how similar two input
samples are. Instead of the verification task as performed in common Siamese
networks, we extended it for the few-shot classification task. In contrast to
the domain adaptation method, this method can train on disjoint data. The
Uncontrolled-Data is not used in the training stage. The working principle is
depicted in Figure 4. Our network structure aims at learning the optimum
separation between all multiple classes at once.

Figure 4: The Siamese network for few-shot classification task is to learn the
optimum separability for the multiclass classification problem. The ConvNet
structure is used to generate feature embeddings. A distance measure is
calculated for the query sample with all possible support classes. The output
label is the highest similarity score within the multiple classes.

During training time, each training batch consists of 15 query samples from
each class. The support set consists of 5 support samples from each class.
They are fed through the feature extraction network ConvNet $f_{\theta}$ to
get the embeddings. In case the support of each class is larger than one
sample, a mean embedding is calculated to reduce the computational complexity.
A similarity score is determined for the query sample embedding
$f_{\theta}(\textbf{x}_{q})$ and the individual support class embeddings
$f_{\theta}(\textbf{x}_{i})$, where $i=1..C$ and $C$ represents the number of
classes. To determine the similarity score, an euclidean distance vector
between the feature embeddings is calculated by
$\left|f_{\theta}(\textbf{x}_{q})-f_{\theta}(\textbf{x}_{i})\right|$. This
distance measure is further fed through a dense layer to learn the final
similarity score
$\delta_{i}=\sigma(\phi(\left|f_{\theta}(\textbf{x}_{q})-f_{\theta}(\textbf{x}_{i})\right|))$.
This parametric dense layer is optimized by a pair of input samples every time
step. Based on the similarity measure between the query sample and all other
support classes $(\delta_{1},\delta_{2},\delta_{3},...,\delta_{C})$, the
objective of the Siamese network is to maximize the maximum likelihood
estimation of data pairs or equally to minimize the negative log likelihood
cost function.

Adam optimizer is used to train the network parameter with a learning rate of
0.0005. 500 Epochs were performed during the training phase with
9-ways-5-shots. Fifteen query samples each class are used to build the query
set in each epoch. Nine classes are learned in each individual training task
with 5 support samples from each class. In the evaluation phase, the number of
supports each class can be tuned between 5 and 10 samples each. We perform 100
iterations to obtain the average accuracy on the test set.

The similarity measure of the Siamese network here is learnt by a parametric
dense layer, in the next two paragraphs we introduce two other alternatives
from the few-shot classification task where a non-parametric distance metric-
based learning is used in the classification stage. These methods are more
robust against small difference in source and target domains, and thus more
generalized.

#### F2: Prototypical Few-Shot Classification Network

Inspired by [10], we adopted the prototypical network as the second method
(F2) in the few-shot classification problem. In comparison to the first few-
shot method (F1) with Siamese network, this network is non-parametric in the
classification stage. This approach is a feature metric-based learning
approach. Instead of transferring network parameters, this learning approach
is based on learning the similarity distance between the feature embeddings
from new target data to the prototypes of the samples from the same setup.
This method is similar to a clustering-based method to find the k-nearest
neighbours used for the classification.

The negative log-likelihood (NLL) function is applied to the negative
euclidean distance of the embedding vectors to each class center embeddings
(prototype). The objective is to reduce the NLL loss of the query output to
the true classification label. Adam optimizer is used to optimize the weights
of the ConvNet hyperparameters. A learning rate of 0.001 is used to learn the
hyperparameters. 500 Epochs were performed to train 5-ways-10-shots
classification tasks. 15 query images each class are used to build the query
set. In the evaluation phase, the number of class recognition task has
increased to the total 9 classes. The number of supports from each class is
varied between 5 and 10 samples each. We perform 100 iterations to obtain the
average accuracy on the test set.

The disadvantage of using euclidean measure to express the similarity is that
this measure is not bounded. In contrast to Siamese network, there is non
parametric learning after the feature extraction ConvNet structure. The
classification is based on the k-nearest neighbour approach from the class
embedding centers. The last method uses the cosine similarity measure which is
a bounded metric.

#### F3: Local Descriptor Correlated Few-Shot Classification Network

The third method of the few-shot classification (F3) is inspired by Li [21].
The author introduced a new distance metric to label the query sample. Instead
of using image-level feature based measure, they introduced a local descriptor
based image-to-class measure. We think this architecture will work on the 2D
time-frequency spectrum, as this network is sensitive to local features within
a global context. In contrast to object images, the Doppler spectrum contains
less fine-grained contextual information, but local features caused by
repetitive movements and micro-movements from limbs and arms are clearly
observable. In this paper, we examine the local descriptor correlation from
the structural information extracted from the micro-Doppler range caused by
different whole-body activities.

The local features from the ConvNet output is correlated with all other local
descriptors of the support embeddings each class using a cosine similarity.
This provides a locally feature-based image-to-class mapping. The advantage of
the cosine metric is because the cosine distance measures the pattern
similarity without being largely effected by the magnitude. The objective is
to reduce the cross entropy loss for this multiclass classification problem.
Adam optimizer with a learning rate of 0.001 is chosen. The parameters for the
few-shot learning and the batch-wise composition are the same as in the method
F2 for the prototypical network.

In few-shot classification learning task, the training data can be disjoint of
the test data. We used the Lab-Data to build the training tasks and used the
subject development data as support set and the testing data from the
uncontrolled setup to evaluate the model.

In this section, we introduced three methods from few-shot learning: (F1)
Siamese, (F2) ProtoNet, and (F3) LocalNet. These methods are theoretically
suitable to enhance the generalization for our use-case, since we want to
mitigate the retraining phase for similar tasks.

##  V Evaluation and Discussion

The database details are introduced in Section III. In this section, we
present and discuss the results of the proposed approaches in comparison to
the baseline. We aim to optimize the trained model with Lab-Data under
controlled condition to be generalizable to individuals under the uncontrolled
setups.

The results of the baseline model and models which require adaptation with the
Uncontrolled-Data during the training phase are displayed in Table II.

Method |  Ratio of labels used from Uncontrolled-Data | P1 | P2 | P3 | P4 | P5
---|---|---|---|---|---|---
Baseline | - | 16.25 % | 46.35 % | 11.17 % | 25.73 % | 15.5 %
DA | 0 % | 35,20 % | 37,80 % | 20.14 % | 57.67 % | 38.04 %
DA | 50 % | 77.61 % | 85.39 % | 58.61 % | 78.86 % | 75.87 %
DA | 100 % | 87.13 % | 98.14 % | 84.10 % | 76.92 % | 96.85 %
TABLE II: The accuracy results of baseline method and methods with model
retraining is depicted. The term $P_{i}$ indicates the ID of the participants.
Enabling the knowledge from the Uncontrolled-Data to modify the base feature
extraction increases the performance on individual finetuning.

###  V-A Baseline results

Given the Lab-Data in the base training, the trained inference model does not
generalize well on test data collected under uncontrolled conditions, if not
provided in the base training. The stacked bidirectional LSTM network failed
to cope with data from real-world environments, as both data distributions
differ too much. According to the baseline result provided in Table II, in
most of the cases the accuracy equals to a uniform distribution. The model
performs no better than random guessing on the Uncontrolled-Data. In our
application, the total number of classes is nine, random guessing corresponds
to an average accuracy of $\frac{1}{9}=11.1\,\%$ .

###  V-B Our proposed approaches

#### The results of using the domain adaptation (DA) method

are depicted in Table II. Here we successively increased the amount of target
labels from the subject development set to be included into the training to
improve the performance of the domain adaptation. Without including any target
labels, the network only minimizes the distribution of the source and target
embeddings in an unsupervised way. The performance is only 10-20 percentage
points better compared to the baseline. By incorporating more label
information from the target domain, the knowledge transfer improves on the
target domain classification. With only 50 % of the target labels, the results
increased about 40 percentage points and with 100 % of the labels, the results
increased about two to six folds in average.

#### Results of Few-shot Classification Networks

Here, we evaluated three alternatives of the few-shot classification approach.
These models typically do not need to retrain the pre-trained inference model,
as the tasks are disjoint. An overview of the evaluation results is given in
Table III. A general tendency is that the classification accuracy increases at
least 5 percentage points with increasing number of support samples per class
used in the evaluation. This is intuitive, due to an increased reliability and
an improved decision boundary related to more support samples. The Siamese
network (F1) provides similar results compared to the prototypical network
(F2), as both network architectures work with euclidean similarity scores on
the sample-base. Their performances are increased around 50 percentage points
compared to the baseline model. The image-to-class measure in the LocalNet
(F3) performs the best as depicted in Table III with an increase of two to six
folds in average compared to the baseline.

No. | Method | P1 | P2 | P3 | P4 | P5 | P1 | P2 | P3 | P4 | P5
---|---|---|---|---|---|---|---|---|---|---|---
- | Baseline | 16.25 % | 46.35 % | 11.17 % | 25.73 % | 15.5 % | - | - | - | - | -
|  | 5 support samples each class | 10 support samples each class
F1 | Siamese | 65.53 % | 83.77 % | 72.81 % | 65.61 % | 68.33 % | 69.69 % | 90.07 % | 75.65 % | 69.08 % | 72.56 %
F2 | ProtoNet | 67.06 % | 79.9 % | 67.85 % | 77.54 % | 65.92 % | 74.18 % | 88.59 % | 69.31 % | 87.88 % | 70.80 %
F3 | LocalNet | 85.28 % | 79.9 % | 64.85 % | 94.33 % | 86.98 % | 89.55 % | 97.84 % | 67.51 % | 98.0 % | 91.63 %
TABLE III: The accuracy for the three alternatives of few-shot classification
task is shown. The number (5 or 10) indicates the number of support samples
each class used in the evaluation. The term $P_{i}$ indicates the ID of the
participants. The performance increases in general with the increasing number
of supports. The LocalNet with the cosine similarity measure outperforms the
other methods, as it includes the image-to-class feature correlation.

###  V-C Discussion

The performance of activity recognition based on ultrasound sensing using a
mobile device is subject to many variables. The same hardware applied under
changed conditions for the same person show variability in the signal
strength. To deploy a fixed application to real-world scenarios is therefore
not easy and often has to overcome some difficulties. In many cases, the
performance drops due to the dissimilarity in both domains. To overcome this
issue, we investigated several methods in this paper. We provide a database
collected under various conditions to allow researchers perform experiments on
it to solve the problem of lack of generalization on new individuals. The base
data consists of Lab-Data under controlled environment and same sensing
device. Uncontrolled setups from five different individuals are used to
evaluate the methods for finetuning on individual dataset.

Finetuning a base model on new domains requires sufficient amount of labeled
samples from the Uncontrolled-Data. Otherwise, the model would overfit
adopting on this small data amount. However, labels are most difficult to
acquire and the individual labeling process might be error prone. In such
cases, domain adaption method can be leveraged, where no label information of
the target domain is required. Though, such network could benefit from
including a small amount of the target labels in case both domains differ as
investigated in our use-cases.

Few-shot classification is suitable for adopting finetuning on limited data
without retraining. This method can cope with individual hardware
characteristics without modifying the base training. By comparing knowledge
extracted from support samples of different categories, an unknown sample is
able to assign to the correct category under the assumption that samples of
similar categories are close in the embedding space. As no feature adaptation
from the target domain is applied, this model requires both domains behave
similarly.

To leverage few-shot classification, the user has to pre-label a small amount
of individual sessions before the model is adopted to this user. These labels
are used to classify the new samples based on certain distance metrics. The
developer does not need to modify the feature extraction network to
individually adapt to each new user. In case of the domain adaptation, the
developer needs to modify the base feature extraction according to the user
data. It further assumes the similarity of both domains in order to avoid
negative knowledge transfer.

##  VI Conclusion

In this paper, we investigated different approaches to improve the
generalizability of pre-trained classification models under controlled
condition in uncontrolled real-world scenarios based on a mobile application
for workout exercise recognition. We first presented a database to enable us
analysing this problem and building novel solutions. This database allows us
to study the body motion in relation to Doppler profiles from built-in
hardware of different commercial smartphones. The gap between the development
setup and the real-world scenarios, as we prove, often lead to performance
drop and bad usability. The reason can be manifold, as it could rely on
individual difference, hardware specifics or environmental changes. Our
database is built to overcome this gap.

We proposed two methods: domain adaptation and few-shot classification, to
resolve the issue of lack of model generalizability. Our evaluations showed
that the baseline method fails when faced with realistic data. Our proposed
concept of using domain adaption without including the target labels improved
the baseline only by 10-20 percentage points in most cases. However, this
method benefits from including target labels, as with increasing amount of
target labels in the training phase, the recognition performance increases by
two to six folds compared to the baseline. Our proposed solution that is based
on few-shot classification improved the accuracy to the same range, however,
without the effort of retraining.

## Acknowledgment

This research work has been partially funded by the German Federal Ministry of
Education and Research and the Hessen State Ministry for Higher Education,
Research and the Arts within their joint support of the National Research
Center for Applied Cybersecurity ATHENE

## References

  * [1] L. Chen, C. D. Nugent, and H. Wang, “A knowledge-driven approach to activity recognition in smart homes,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 24, no. 6, pp. 961–974, 2011.
  * [2] S. Saxena, F. Brémond, M. Thonnat, and R. Ma, “Crowd behavior recognition for video surveillance,” in _Advanced Concepts for Intelligent Vision Systems_ , J. Blanc-Talon, S. Bourennane, W. Philips, D. Popescu, and P. Scheunders, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 970–981.
  * [3] M. Sundholm, J. Cheng, B. Zhou, A. Sethi, and P. Lukowicz, “Smart-Mat: Recognizing and Counting Gym Exercises with Low-cost Resistive Pressure Sensing Matrix,” in _Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp ’14 Adjunct_ , A. J. Brush, A. Friday, J. Kientz, J. Scott, and J. Song, Eds. New York, New York, USA: ACM Press, 2014, pp. 373–382.
  * [4] B. Fu, L. Jarms, F. Kirchbuchner, and A. Kuijper, “Exertrack—towards smart surfaces to track exercises,” _Technologies_ , vol. 8, no. 1, p. 17, 2020.
  * [5] K. H. Kim, S. W. Bang, and S. R. Kim, “Emotion recognition system using short-term monitoring of physiological signals,” _Medical and biological engineering and computing_ , vol. 42, no. 3, pp. 419–427, 2004.
  * [6] B. Fu, N. Damer, F. Kirchbuchner, and A. Kuijper, “Sensing technology for human activity recognition: A comprehensive survey,” _IEEE Access_ , vol. 8, pp. 83 791–83 820, 2020.
  * [7] A. Ng, “Machine learning yearning,” _URL: http://www. mlyearning. org/(96)_ , 2017.
  * [8] J. Wang, V. W. Zheng, Y. Chen, and M. Huang, “Deep transfer learning for cross-domain activity recognition,” in _proceedings of the 3rd International Conference on Crowd Science and Engineering_ , 2018, pp. 1–8.
  * [9] M. A. A. H. Khan, N. Roy, and A. Misra, “Scaling human activity recognition via deep learning-based domain adaptation,” in _2018 IEEE International Conference on Pervasive Computing and Communications (PerCom)_. IEEE, 2018, pp. 1–9.
  * [10] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Advances in neural information processing systems_ , 2017, pp. 4077–4087.
  * [11] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networks for one-shot image recognition,” in _ICML deep learning workshop_ , vol. 2. Lille, 2015.
  * [12] S. Feng and M. F. Duarte, “Few-shot learning-based human activity recognition,” _Expert Systems with Applications_ , vol. 138, p. 112782, 2019.
  * [13] J. R. Kwapisz, G. M. Weiss, and S. A. Moore, “Activity recognition using cell phone accelerometers,” _ACM SigKDD Explorations Newsletter_ , vol. 12, no. 2, pp. 74–82, 2011.
  * [14] N. Ravi, N. Dandekar, P. Mysore, and M. L. Littman, “Activity recognition from accelerometer data,” in _Aaai_ , vol. 5, no. 2005, 2005, pp. 1541–1546.
  * [15] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, “A public domain dataset for human activity recognition using smartphones.” in _Esann_ , 2013.
  * [16] T. L. van Kasteren, G. Englebienne, and B. J. Kröse, “Human activity recognition from wireless sensor network data: Benchmark and software,” in _Activity recognition in pervasive intelligent environments_. Springer, 2011, pp. 165–186.
  * [17] U.S. Department of Health and Human Services, “Physical Activity Guidelines for Americans,” 2008. [Online]. Available: https://health.gov/paguidelines/pdf/paguide.pdf
  * [18] L. Durak and O. Arikan, “Short-time fourier transform: two fundamental properties and an optimal implementation,” _IEEE Transactions on Signal Processing_ , vol. 51, no. 5, pp. 1231–1242, 2003.
  * [19] W.-g. Yoo, “Effect of exercise speed and isokinetic feedback on the middle and lower serratus anterior muscles during push-up exercises,” _Journal of physical therapy science_ , vol. 26, no. 5, pp. 645–646, 2014.
  * [20] B. Fu, F. Kirchbuchner, and A. Kuijper, “Unconstrained workout activity recognition on unmodified commercial off-the-shelf smartphones,” in _The 13th PErvasive Technologies Related to Assistive Environments Conference (PETRA ’20), June 30-July 3, 2020, Corfu, Greece_ , 2020, p. 10.
  * [21] W. Li, L. Wang, J. Xu, J. Huo, Y. Gao, and J. Luo, “Revisiting local descriptor based image-to-class measure for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7260–7268.