# On the Cauchy Problem for the Dispersion Generalized Camassa-Holm Equation

Nesibe Ayhan  Nilay Duruk Mutlubaş

Abstract

In this paper, we establish local well-posedness of the Cauchy problem for a
recently proposed dispersion generalized Camassa-Holm equation by using Kato’s
semigroup approach for quasi-linear evolution equations. We show that for
initial data in the Sobolev space $H^{s}(\mathbb{R})$ with $s>\frac{7}{2}+p$,
the Cauchy problem is locally well-posed, where $p$ is an even real number
determined by the order of the positive differential operator $L$
corresponding to the dispersive effect added to the Camassa-Holm equation.

Keywords: Camassa-Holm equation, Dispersion, Local well-posedness, Semigroup
Theory

MSC classification 2010: Primary: 35G25, 35L30, 47D60; Secondary: 35B65,
47D03.

##  1 Introduction

The nonlinear dispersive wave equation

$u_{t}-u_{xxt}+3uu_{x}=2u_{x}u_{xx}+uu_{xxx},$ (1.1)

was introduced by Camassa and Holm [1] to model the unidirectional propagation
of shallow water waves over a flat bottom. Here, $u(x,t)$ represents the fluid
velocity at time $t$ and in the spatial direction $x$. The equation is known
as the Camassa–Holm (CH) equation, whose various generalizations have appeared
in the literature in recent years. These generalizations are usually done
qualitatively on the structure of the equation, without paying much attention
on the physical meaning and derivation.

The Korteweg deVries (KdV) and Benjamin-Bona-Mahony (BBM) equations are
canonical wave equations which characterizes unidirectional dispersive wave
propagation in continuum having different physical properties. Although the CH
equation was proposed primarily to model propagation of waves of moderate
amplitude in shallow water, it is expected to have the equation arising as a
model equation in different continuum where dispersive and nonlinear affects
are balanced properly. As a matter of fact there are studies in this direction
(for instance, see [5] about the nonlinear waves propagating along a circular
cylindrical rod composed of a compressible elastic material and [2] about the
nonlinear waves propagating in a pre-stressed, thin elastic plate composed of
a compressible hyperelastic material). Since linear dispersive affect can
occur in different forms for different continuum due to various reasons such
as nonlocal effects, inhomogeneity, anisotropy, multidimensionality, etc and
this changes the inertia term $u_{t}-u_{xxt}$, having generalized form of
inertia term, writing the corresponding equation and analyzing the new model
are interesting research problems. Therefore, in this paper, we focus on the
dispersive nature of CH equation and analyze how the behavior of the solutions
change when the dispersive effect is generalized.

When the terms in the CH equation are examined in detail, the $uu_{x}$ term
denotes nonlinear steeping, the $u_{xxt}$ term denotes the linear dispersion
effect, and the $2u_{x}u_{xx}+uu_{xxx}$ terms denote the nonlinear dispersion
effect. When the momentum density $m=(1-\partial_{x}^{2})u$ is defined, the CH
equation becomes

$m_{t}+m_{x}u+2mu_{x}=0.$ (1.2)

There are generalizations of the CH equation for different momentum density
forms in the literature. Important examples of these can be given as follows:

$(i)$ Hunter and Saxton in [10] considered $m=-\partial_{x}^{2}u$.

$(ii)$ Holm et al. in [9] considered $m=(1-\alpha^{2}\partial_{x}^{2})u$,
where $\alpha$ is a constant.

$(iii)$ Khesin et al. in [12] introduced a $\mu$-version of Camassa-Holm
equation as follows

$m_{t}+2mu_{x}+m_{x}u=0,\ \ \ \ m=(\mu-\partial_{x}^{2})u$,

where $u(x,t)$ is a time-dependent function on the unit circle
$\mathbb{S}=\mathbb{R}/\mathbb{Z}$ and $\mu(u)=\int_{\mathbb{S}}udx$ denotes
its mean. This equation describes evolution of rotators in liquid crystals
with external magnetic field and self-interaction. It is also studied in [6],
[8], [22], [23].
For the periodic case, Wang [21] considered $m=\mu(u)-u_{xx}+u_{xxxx}$.
Moreover, Wang studied the modified $\mu$-version of Camassa-Holm equation in
[19] as follows

$m_{t}+(2\mu(u)u-u_{x}^{2})m)_{x}=0$.

$(iv)$ Wang considered $m=\mu(u)+u_{xxxx}$ in [20].

$(v)$ Ding et al. considered $m=u-u_{xx}+u_{xxxx}$ in [7].

$(vi)$ There are also higher order forms of the CH equation, where
$m=(1-\partial_{x}^{2})^{k}u$ for positive integer $k$ [4], which describe
exponential curves of the manifold of smooth orientation-preserving
diffeomorphisms of the unit circle in the plane. Studies for $k=2$ can be
found in [14], [15], [17].

$(vii)$ For $r\geq 1$, Camassa-Holm system with two components, where
$m=(1-\partial_{x}^{2})^{r}u$ is studied in [3].

The common feature of the examples in the literature is that they put
different effects on linear and non-linear dispersive terms to observe the
results. In this paper, our main aim is to study

$m_{t}+bm_{x}u+amu_{x}=0,\quad a,b>0,$ (1.3)

with the following form of momentum density:

$m=(1-L\partial_{x}^{2})u.$ (1.4)

Here, $L$ is a general differential operator in spatial variable $x$ whose
order is an even real number $p$. With this momentum density, the dispersive
effect in (1.1) is generalized in (1.3). Note that choosing $L$ as the
identity operator with $a=2$ and $b=1$, (1.3) corresponds to the CH equation
given by $(\ref{eqn1})$. Furthermore, choosing $L$ as $\alpha^{2}$ multiple of
identity operator corresponds to the example $(ii)$, as $Id-\partial_{x}^{2}$
corresponds to the example $(v)$, as $2-\partial_{x}^{2}$ corresponds to the
example $(vi)$.

The CH equation with the generalized dispersion feature defined in (1.4) has
been proposed for the first time in the literature. In the equation (1.3),
both the linear and the non-linear dispersion effects have changed. In terms
of qualitative analysis, this makes a big difference since $L$ operator is
given in closed form and applies also on nonlinear terms in the equation.

This paper presents the dispersion generalized Camassa-Holm equation given by
(1.3)-(1.4) and proves local well-posedness of the solutions for the
corresponding Cauchy problem. It can be considered as a non-local and
nonlinear dispersive partial differential equation and a mathematical
generalization of the classical Camassa-Holm equation rather than a physical
generalization. We prove that the Cauchy problem is locally well-posed on the
real line for the initial data in Sobolev space $H^{s}(\mathbb{R})=H^{s}$,
$s>\frac{7}{2}+p$. The even real number $p$ is the order of the general
differential operator $L$ which appears in closed form.

Our proof is based on Kato’s semigroup approach for quasilinear equations. For
this reason, in Section 2 we present a short review of the theorem we rely on
and establish local well-posedness for the dispersion generalized Camassa-Holm
equation. We make use of commutator operators to obtain a suitable form of the
equation to use Kato’s semigroup approach. Main reason for rewriting is that
$L$ is in closed form among the nonlinear terms in the equation as well and
usual differentiation rules do not apply.

In Section 3, we compare the results for Camassa-Holm type equations with
those of the dispersion generalized Camassa-Holm equation. The changes in the
dispersive effect, the differences in their non-local forms, the initial data
classes chosen for the corresponding Cauchy problems are discussed.

We end the paper with Section 4 in which we provide open problems to be
discussed. According to mathematical analysis questions for the Camassa-Holm
equation appearing in the literature, we can say that further qualitative
analysis is also possible for the dispersion generalized Camassa-Holm
equation, such as global well-posedness and finite time blow-up.

##  2 Local Well-posedness

###  2.1 Semigroup Approach

Consider the abstract quasi-linear evolution equation in the Hilbert space
$X$:

$u_{t}+A(u)u=f(u),\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ t\geq 0,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ u(0)=u_{0}.$ (2.1)

Let $Y$ be a second Hilbert space such that $Y$ is continuously and densely
injected into $X$ and let $S:Y\rightarrow X$ be a topological isomorphism.
Assume that

  * (A1)

For any given $r>0$ it holds that for all $u\in\mathrm{B}_{r}(0)\subseteq Y$
(the ball around the origin in $Y$ with radius $r$), the linear operator
$A(u)\colon X\to X$ generates a strongly continuous semigroup $T_{u}(t)$ in
$X$ which satisfies

$\|T_{u}(t)\|_{\mathcal{L}(X)}\leq\mathrm{e}^{\omega_{r}t}\quad\text{for
all}\quad t\in[0,\infty)$

for a uniform constant $\omega_{r}>0$.

  * (A2)

$A$ maps $Y$ into $\mathcal{L}(Y,X)$, more precisely the domain $D(A(u))$
contains $Y$ and the restriction $A(u)|_{Y}$ belongs to $\mathcal{L}(Y,X)$ for
any $u\in Y$. Furthermore $A$ is Lipschitz continuous in the sense that for
all $r>0$ there exists a constant $C_{1}$ which only depends on $r$ such that

$\|A(u)-A(v)\|_{\mathcal{L}(Y,X)}\leq C_{1}\,\|u-v\|_{X}$

for all $u,\leavevmode\nobreak\ v$ inside $\mathrm{B}_{r}(0)\subseteq Y$.

  * (A3)

For any $u\in Y$ there exists a bounded linear operator
$B(u)\in\mathcal{L}(X)$ satisfying $B(u)=SA(u)S^{-1}-A(u)$ and $B\colon
Y\to\mathcal{L}(X)$ is uniformly bounded on bounded sets in $Y$. Furthermore
for all $r>0$ there exists a constant $C_{2}$ which depends only on $r$ such
that

$\|B(u)-B(v)\|_{\mathcal{L}(X)}\leq C_{2}\,\|u-v\|_{Y}$

for all $u,\leavevmode\nobreak\ v\in\mathrm{B}_{r}(0)\subseteq Y$.

  * (A4)

For all $t\in[0,\infty)$, $f$ is uniformly bounded on bounded sets in $Y$.
Moreover, the map $f\colon Y\to Y$ is locally $X$-Lipschitz continuous in the
sense that for every $r>0$ there exists a constant $C_{3}>0$, depending only
on $r$, such that

$\|f(u)-f(v)\|_{X}\leq C_{3}\,\|u-v\|_{X}\quad\text{for
all}\;u,\leavevmode\nobreak\ v\in\mathrm{B}_{r}(0)\subseteq Y$

and locally $Y$-Lipschitz continuous in the sense that for every $r>0$ there
exists a constant $C_{4}>0$, depending only on $r$, such that

$\|f(u)-f(v)\|_{Y}\leq C_{4}\,\|u-v\|_{Y}\quad\text{for
all}\;u,\leavevmode\nobreak\ v\in\mathrm{B}_{r}(0)\subseteq Y.$

######  Theorem 2.1.

[11] Assume that (A1)-(A4) hold. Then for given $u_{0}\in Y$, there is a
maximal time of existence $T>0$, depending on $u_{0}$, and a unique solution
$u$ to (2.1) in $X$ such that

$u=u(u_{0},.)\in C([0,T),Y)\cap C^{1}([0,T),X).$

Moreover, the solution depends continuously on the initial data,
i.e. the map $u_{0}\rightarrow u(u_{0},.)$ is continuous from $Y$ to
$C([0,T),Y)\cap C^{1}([0,T),X)$.

###  2.2 Local Well-posedness Theorem

In this subsection, we apply Kato’s semigroup approach to establish local
well-posedness for the Cauchy problem associated to the generalized Camassa-
Holm equation:

$\begin{cases}u_{t}-L\partial^{2}_{x}u_{t}+(a+b)uu_{x}-au_{x}L\partial^{2}_{x}u-buL\partial^{2}_{x}u_{x}=0,&t>0,\
x\in\mathbb{R},\\\ u(x,0)=u_{0}(x),&x\in\mathbb{R},\\\ \end{cases}$ (2.2)

where $a$ and $b$ are positive constants, $L$ is a positive operator with even
order $p$. Note that we rewrite the equation (1.3) for which (1.4) is valid.
To construct a non-local form of this equation, we use the usual commutator of
two operators $[.,.]$:

$[L\partial^{2}_{x},u]u_{x}=L\partial^{2}_{x}(uu_{x})-uL\partial^{2}_{x}u_{x}$.

Then,

$\displaystyle(a+b)[L\partial^{2}_{x},u]u_{x}$ $\displaystyle=\
(a+b)L\partial^{2}_{x}(uu_{x})-(a+b)uL\partial^{2}_{x}u_{x}$ $\displaystyle=\
(a+b)L\partial^{2}_{x}(uu_{x})-auL\partial^{2}_{x}u_{x}-buL\partial^{2}_{x}u_{x}$
.

So, we can write

$(a+b)uu_{x}-buL\partial^{2}_{x}u_{x}=(a+b)(1-L\partial^{2}_{x})(uu_{x})+(a+b)[L\partial^{2}_{x},u]u_{x}+auL\partial^{2}_{x}u_{x}$.

Then, we have

$(1-L\partial^{2}_{x})u_{t}+(a+b)(1-L\partial^{2}_{x})(uu_{x})+(a+b)[L\partial^{2}_{x},u]u_{x}+auL\partial^{2}_{x}u_{x}-au_{x}L\partial^{2}_{x}u=0$.

Also, it can be seen that

$\displaystyle auL\partial^{2}_{x}u_{x}-au_{x}L\partial^{2}_{x}u$
$\displaystyle=a(uL\partial^{2}_{x}u_{x}-u_{x}L\partial^{2}_{x}u)$
$\displaystyle=a(uL\partial^{2}_{x}u_{x}-(L\partial^{2}_{x}u)u_{x})$
$\displaystyle=a(uL\partial^{2}_{x}-(L\partial^{2}_{x}u))u_{x}$
$\displaystyle=-a[L\partial^{2}_{x},u]u_{x}.$

Then, the equation (2.2) becomes

$(1-L\partial^{2}_{x})u_{t}+(a+b)(1-L\partial^{2}_{x})(uu_{x})+b[L\partial^{2}_{x},u]u_{x}=0$.

With $\Gamma^{s}=(1-L\partial^{2}_{x})^{s/p+2}$, this equation takes the
following quasi-linear form:

$u_{t}+A(u)u=f(u)$,

where

$A(u)=(a+b)u\partial_{x}+b\Gamma^{-(p+2)}[L\partial^{2}_{x},u]\partial_{x}=r(u)\partial_{x},$
(2.3)

and

$f(u)=0.$ (2.4)

Here, notice that the operator $L$ is in a closed form, and even though there
are more than one possible way of writing non-local form of an equation, the
form where we collect nonlinear effects in the operator $A(u)$, as above,
holds and serves for our purpose. Furthermore, recalling the approach, we
choose the spaces $X:=(H^{s-1},||.||_{s-1})$, $Y:=(H^{s},||.||_{s})$ and
consider the topological isomorphism
$S\coloneqq\Gamma=(1-L\partial^{2}_{x})^{1/p+2}\colon Y\to X$ between these
spaces, which defines an isometry, i.e. $\|\Gamma u\|_{s-1}=\|u\|_{s}$ for all
$u\in H^{s}$.

Hence, the main result of this paper is the following:

######  Theorem 2.2.

Assume that assumptions $(A1)-(A4)$ hold for (2.2)-(2.4). Let $u_{0}\in
H^{s}$, $s>\frac{7}{2}+p$ be given. Then, there exists a maximal time of
existence $T>0$, depending on $u_{0}$, such that there is a unique solution
$u$ satisfying

$u\in C([0,T),H^{s})\cap C^{1}([0,T),H^{s-1})$.

Moreover, the solution depends continuously on the initial data, i.e, the map
$u_{0}\rightarrow u(u_{0},.)$ is continuous from $H^{s}$ to
$C([0,T),H^{s})\cap C^{1}([0,T),H^{s-1})$.

In order to prove this result, we will apply Kato’s semigroup approach. Since
$f(u)=0$ in our Cauchy problem, we only need to verify assumptions (A1)-(A3).

Note that if $L$ is the identity operator, $a=2$ and $b=1$, we get the
Camassa-Holm equation. Considering this, the steps in the proof can be
followed clearly.

### Proof of Assumption (A1):

Below, you will find the lemmas to be used in the proof of assumption $(A1)$.

######  Lemma 2.1.

The operator $A(u)=r(u)\partial_{x}$ in $L^{2}$, with $u\in
H^{s},s>\frac{7}{2}+p$ is quasi-m-accretive.

###### Proof.

Since $L^{2}$ is a Hilbert space with standard inner product $(.,.)_{0}$,
$A(u)$ is quasi-m-accretive if and only if there is a real number $\beta$ such
that

  * (a)

$(A(u)w,w)_{0}\geq-\beta||w||_{0}^{2}$,

  * (b)

The range of $A(u)+\lambda$ is all of $X$ for some (or all) $\lambda>\beta$.

First, we will prove part $(a)$. By using integration by parts, the left-hand
side of the equality can be written as follows:

$(A(u)w,w)_{0}=(r(u)\partial_{x}w,w)_{0}=\frac{-1}{2}((r(u))_{x}w,w)_{0}$

since if we let

$\displaystyle K=(r(u)w_{x},\ w)_{0}\ $ $\displaystyle=-(r(u)w_{x},\ w)_{0}\
-\ ((r(u))_{x}w,\ w)_{0},$

where $-(r(u)w_{x},\ w)_{0}=-K$. Then,

$\displaystyle 2K=-((r(u))_{x}w,\ w)_{0}$ $\displaystyle
K=-\frac{1}{2}((r(u))_{x}w,\ w)_{0}.$

Then, it follows that

$\displaystyle|(r(u)\partial_{x}w,w)_{0}|$
$\displaystyle=|\frac{-1}{2}((r(u))_{x}w,w)_{0}|$ $\displaystyle\leq
c||(r(u))_{x}w||_{0}||w||_{0}$ $\displaystyle\leq
c||(r(u))_{x}||_{L^{\infty}}||w||_{0}^{2}$ $\displaystyle\leq
c||(r(u)||_{s}||w||_{0}^{2}$ $\displaystyle\leq\beta||w||_{0}^{2},$

where $\beta$ is a constant depending on $||u||_{s}$. Since $u\in H^{s}$ with
$s>\frac{7}{2}+p$, it follows that $||u_{x}||_{L^{\infty}}\leq||u||_{s}$. We
show that the operator $A$ is dissipative for all $\lambda>\beta$. That means
the operator $-A$ is accretive, where we have
$(A(u)w,w)_{0}\geq-\beta||w||_{0}^{2}$ as required.

Now, we will prove part $(b)$. Note that if $A$ is a closed operator, then
$A(u)+\lambda$ has closed range in $X$ for all $\lambda>\beta$. So, it is
enough to show that $A(u)+\lambda$ has dense range in $x$ for all
$\lambda>\beta$.

First, we will show that $A$ is a closed operator in $L^{2}$. Let $(v_{n})$ be
a sequence in $\mathcal{D}(A)$ which converges to $v\in L^{2}$ and $Av_{n}$
converges to $w\in L^{2}$. Then, since $v_{n}\in\mathcal{D}(A)$ and
$\mathcal{D}(A)=\\{w\in L^{2}\ |\ r(u)w\in H^{1}\\}\subset L^{2}$, we can
conclude that $rv_{n}\in H^{1}$. Also, by the continuity of the multiplication
$H^{r}\times L^{2}\rightarrow L^{2}$ for $r>\frac{1}{2}$, both
$rv_{n}\rightarrow rv$ and $r_{x}v_{n}\rightarrow r_{x}v$ in $L^{2}$, which
implies that $(rv_{n})_{x}\rightarrow w+r_{x}v$ in $L^{2}$. Then, we have the
sequences $(rv_{n})$ and $(rv_{n})_{x}$ converges in $L^{2}$. Then we can
conclude that $(rv_{n})$ converges to $rv$ in $H^{1}$, which implies that
$v\in\mathcal{D}(A)$. Moreover, by continuity of
$\partial_{x}:H^{1}\rightarrow L^{2}$ implies that
$\lim_{n\to\infty}(rv_{n})_{x}=(rv)_{x}$, thus we get $w=(rv)_{x}-r_{x}v=Av$.
Hence, by definition, we showed that A is a closed operator.

Now, we need to show that $(A(u)+\lambda)$ has dense range in $L^{2}$ for all
$\lambda>\beta$. Note that the adjoint operator of the $A(u)=r(u)\partial_{x}$
can be written as

$A^{*}(u)=-r_{x}(u)-r(u)\partial_{x}$.

Then,

$A^{*}(u)w=-r_{x}(u)w-r(u)w_{x}=-(r(u)w)_{x}$,

where $r_{x}(u)w\in L^{2}$ since $u_{x}\in L^{\infty}$ and $w\in L^{2}$, and
$r(u)w_{x}=A(u)\in L^{2}$ for $w\in\mathcal{D}(A)$. Hence, we can obtain that

$\mathcal{D}(A^{*})=\\{w\in L^{2}\ |\ A^{*}(u)w\in L^{2}\\}$.

On the contrary, assume that the range of $(A(u)+\lambda)$ is not all of
$L^{2}$. Then, there exists $z\neq 0\in L^{2}$ such that

$(A(u)w,z)_{0}=0$, $\forall w\in\mathcal{D}(A(u))$.

Since $H^{1}\subset\mathcal{D}(A)$, $\mathcal{D}(A)$ is dense in $L^{2}$.
Then, due to $\mathcal{D}(A^{*})$ is closed, $z\in\mathcal{D}(A^{*})$. Then,
by using the fact that $\mathcal{D}(A)=\mathcal{D}(A^{*})$, we can write

$((A(u)+\lambda)w,z)_{0}=(w,(A(u)+\lambda)^{*}z)_{0}=0$,

which implies that $(A(u)^{*}+\lambda)z=0$ in $L^{2}$. After multiplying this
equality by $z$, we can rewrite it as

$\displaystyle 0=((A^{*}(u)+\lambda)z,z)_{0}$
$\displaystyle=(A^{*}(u)z,z)_{0}+(\lambda z,z)_{0}$
$\displaystyle=(z,A(u)z)_{0}+(\lambda z,z)_{0}$
$\displaystyle\geq(\lambda-\beta)||z||_{0}^{2},\ \ \ \forall\lambda>\beta.$

Since for all $\lambda>\beta$, the term $(\lambda-\beta)>0$. Therefore, $z=0$.
However, it contradicts with the assumption $z\neq 0$, which completes the
proof of Lemma 2.1.
∎

Now, we give the commutator estimate needed for the upcoming lemma:

######  Proposition 2.1 ([13])).

Let $n>0$, $s\geq 0$, and $3/2<s+n\leq\sigma$. Then, for all $f\in H^{\sigma}$
and $g\in H^{s+n-1}$, one has

$||[\Lambda^{n},\ f]g||_{s}\leq c||f||_{\sigma}||g||_{s+n-1}$,

where $c$ is a constant which is independent of $f$ and $g$.

######  Lemma 2.2.

The operator $A(u)=r(u)\partial_{x}$ in $H^{s-1}$, with $u\in
H^{s},s>\frac{7}{2}+p$ is quasi-m-accretive.

###### Proof.

Since $H^{s-1}$ is a Hilbert space, $A(u)=r(u)\partial_{x}$ is quasi-m-
accretive if and only if there is a real number $\beta$ such that

  * $(a)$

$(A(u)w,w)_{s-1}\geq-\beta||w||_{s-1}^{2},$

  * $(b)$

-$A(u)$ is the infinitesimal generator of a $C_{0}$-semigroup on $H^{s-1}$, for some (or all) $\lambda>\beta$.

First, we will prove part $(a)$. Since $u\in H^{s}$, with $s>\frac{7}{2}$, we
can say that $u$ and $u_{x}$ belong to $L^{\infty}$. Then, it follows that
$||u_{x}||_{L^{\infty}}\leq||u||_{s}$. Note that

$\displaystyle\Gamma^{s-1}(r(u)\partial_{x}w)$
$\displaystyle=[\Gamma^{s-1},r(u)]\partial_{x}w+r(u)\Gamma^{s-1}(\partial_{x}w)$
$\displaystyle=\
[\Gamma^{s-1},r(u)]\partial_{x}w+r(u)\partial_{x}\Gamma^{s-1}w.$

Then, we have

$\displaystyle(A(u)w,w)_{s-1}$ $\displaystyle=\ (r(u)\partial_{x}w,\ w)_{s-1}$
$\displaystyle=\ (\Gamma^{s-1}r(u)\partial_{x}w,\ \Gamma^{s-1}w)_{0}$
$\displaystyle=\ ([\Gamma^{s-1},r(u)]\partial_{x}w,\
\Gamma^{s-1}w)_{0}+(r(u)\partial_{x}\Gamma^{s-1}w,\ \Lambda^{s-1}w)_{0}.$

For the first term $([\Gamma^{s-1},r(u)]\partial_{x}w,\ \Gamma^{s-1}w)_{0}$,
use the commutator estimate (Proposition 2.1) with $n=s-1$, and $\sigma=s$.
Then, we get

$\displaystyle|([\Gamma^{s-1},r(u)]\partial_{x}w,\ \Gamma^{s-1}w)_{0}|$
$\displaystyle\leq\ c||(r(u)||_{s}||\partial_{x}w||_{s-2}||w||_{s-1}$
$\displaystyle\leq\ \tilde{c}||w||^{2}_{s-1},$

where $\tilde{c}$ is a constant depending on $||u||_{s}$.

For the second term $(r(u)\partial_{x}\Gamma^{s-1}w,\ \Gamma^{s-1}w)_{0}$, use
the integration by parts to get

$\displaystyle|(r(u)\partial_{x}\Gamma^{s-1}w,\ \Gamma^{s-1}w)_{0}|$
$\displaystyle=\ \big{|}-\frac{1}{2}(r(u)_{x},\
\Gamma^{s-1}w)^{2})_{0}\big{|}$ $\displaystyle\leq\
c||r(u)_{x}||_{L^{\infty}}||\Gamma^{s-1}w||^{2}_{0}$ $\displaystyle\leq\
c||r(u)_{x}||_{L^{\infty}}||w||^{2}_{s-1}$ $\displaystyle\leq\
\tilde{c}||w||^{2}_{s-1},$

where $\tilde{c}$ is a constant depending on $||u||_{s}$.

Set $\beta=\tilde{c}||u||_{s}$. Then, we get
$(A(u)w,w)_{s-1}\geq-\beta||w||_{s-1}^{2}$, as required.

Now, we will prove part $(b)$. Let $Q=\Gamma^{s-1}$, note that $Q$ is an
isomorphism of $H^{s-1}$ to $L^{2}$, and $H^{s-1}$ is continuously and densely
embedded into $L^{2}$ as $s>\frac{3}{2}$. Define

$\displaystyle A_{1}(u):=QA(u)Q^{-1}$ $\displaystyle=\
\Gamma^{s-1}A(u)\Gamma^{1-s}$ $\displaystyle=\
\Gamma^{s-1}r(u)\partial_{x}\Gamma^{1-s}$ $\displaystyle=\
\Gamma^{s-1}r(u)\Gamma^{1-s}\partial_{x}$

and $B_{1}=\ A_{1}(u)+\ A(u)$.
Let $w\in L^{2}$ and $u\in H^{s}$, where $s>\frac{7}{2}+p$. Then, we have

$\displaystyle||B_{1}(u)||_{0}$ $\displaystyle=\ ||[\Gamma^{s-1},\
A(u)]\Gamma^{1-s}w||_{0}$ $\displaystyle=\ ||[\Gamma^{s-1},\
r(u)]\Gamma^{1-s}\partial_{x}w||_{0}$ $\displaystyle\leq\
c||r(u)||_{s}||\Gamma^{1-s}\partial_{x}w||_{s-2}$ $\displaystyle\leq\
c||r(u)||_{s}||w||_{0}$ ,

where we again use the commutator estimate (Proposition 2.1) with $n=s-1$, and
$\sigma=s$. Therefore, we obtain $B_{1}(u)\in\mathcal{L}(L^{2})$. ∎

######  Lemma 2.3.

[16] Let $X$ and $Y$ be two Banach spaces such that $Y$ is continuously and
densely embedded in $X$. Let -$A$ be the infinitesimal generator of the
$C_{0}$-semigroup $T(t)$ on $X$ and let $Q$ be an isomorphism from $Y$ onto
$X$. Then $Y$ is -$A$-admissible (i.e. $T(t)Y\subset Y$ for all $t\geq 0$, and
the restriction of $T(t)$ to $Y$ is a $C_{0}$-semigroup on $Y$ ) if and only
if -$A_{1}=$ \- $QAQ^{-1}$ is the infinitesimal generator of the
$C_{0}$-semigroup $T_{1}(t)=QT(t)Q^{-1}$ on $X$. Moreover, if $Y$ is
-$A$-admissible, then the part of -$A$ in $Y$ is the infinitesimal generator
of the restriction $T(t)$ to $Y$.

We show that $A(u)$ is quasi-m-accretive in $L^{2}$, i.e, -$A(u)$ is the
infinitesimal generator of $C_{0}$-semigroup on $H^{s-1}$ by Lemma 2.2. Thus,
by using the perturbation theorem for semigroup [16], we can say that
$A_{1}(u)=A(u)+B_{1}(u)$ is also the infinitesimal generator of
$C_{0}$-semigroup on $L^{2}$. Then, we can conclude that $H^{s-1}$ is
-$A$-admissible. Hence, -$A(u)$ is the infinitesimal generator of
$C_{0}$-semigroup on $H^{s-1}$ by Lemma 2.3 with $X=L^{2}$, $Y=H^{s-1}$, and
$Q=\Gamma^{s-1}$. This completes the proof of Lemma 2.2, and thus assumption
$(A1)$.

### Proof of Assumption (A2):

Below, you will find the needed lemmas to be used in the proof of assumption
$(A2)$.

######  Lemma 2.4.

Let the operator $A(u)=r(u)\partial_{x}$, with $u\in H^{s}$,
$s>\frac{7}{2}+p$. Then, $A(u)\in\mathcal{L}(H^{s},H^{s-1})$, for any $u\in
H^{s}$. Moreover,

$||(A(u)-A(v))w||_{s-1}\leq\lambda_{1}||u-v||_{s-1}||w||_{s}$, $u,v,w\in
H^{s}$.

###### Proof.

Let $u,v,w\in H^{s}$ with $s>\frac{7}{2}+p$, and note that $H^{s-1}$ is a
Banach algebra.
Then, we have

$\displaystyle||(A(u)-A(v))w||_{s-1}$
$\displaystyle=||((a+b)(u-v)\partial_{x}+b\Gamma^{-(p+2)}[L\partial^{2}_{x},(u-v)]\partial_{x})w||_{s-1}$
$\displaystyle\leq||(a+b)(u-v)\partial_{x}w||_{s-1}+||b\Gamma^{-(p+2)}[L\partial^{2}_{x},(u-v)]\partial_{x}w||_{s-1}$
$\displaystyle\leq||(u-v)||_{s-1}||\partial_{x}w||_{s-1}+||[L\partial^{2}_{x},(u-v)]\partial_{x}w||_{s-p-3}$
.

We will use the commutator estimate (Proposition 2.1) with $n=p+2$, $s=s-p-3$,
and $\sigma=s-1$, which implies $s+n-1=s-2$. Then, for $f=u-v$ and
$g=\partial_{x}w$, we get

$\displaystyle||[L\partial^{2}_{x},(u-v)]\partial_{x}w||_{s-p-3}$
$\displaystyle\leq c||(u-v)||_{s-1}||\partial_{x}w||_{s-2}$ $\displaystyle\leq
c||u-v||_{s-1}||w||_{s-1}$
$\displaystyle\leq\lambda_{1}||u-v||_{s-1}||w||_{s},$

where $\lambda_{1}$ is a constant. Then, we get

$||(A(u)-A(v))w||_{s-1}\leq\lambda_{1}||u-v||_{s-1}||w||_{s}$.

Take $v=0$ in the above inequality to obtain $A(u)\in\mathcal{L}(H^{s},\
H^{s-1})$. This completes the proof of Lemma 2.4, and thus assumption $(A2)$.

∎

### Proof of Assumption (A3):

Below, you will find the needed lemmas to be used in the proof of assumption
$(A3)$.

######  Lemma 2.5.

For any $u\in H^{s}$, there exists a bounded linear operator
$B(u)\in\mathcal{L}(H^{s-1})$ satisfying $B(u)=\Gamma A(u)\Gamma^{-1}-A(u)$,
where $B:H^{s}\rightarrow\mathcal{L}(H^{s-1})$ is uniformly bounded sets in
$H^{s-1}$. Moreover,

$||(B(u)-B(v))w||_{s-1}\leq\lambda_{2}||u-v||_{s}||w||_{s-1}$, $u,v\in H^{s}$,
$w\in H^{s-1}$.

###### Proof.

Note that since $\partial_{x}$ and $\Gamma$ commute, we can rewrite $B(u)$ as

$B(u)=\Gamma A(u)\Gamma^{-1}-A(u)=\Gamma
r(u)\partial_{x}\Gamma^{-1}-r(u)\partial_{x}=[\Gamma,r(u)]\Gamma^{-1}\partial_{x}$.

First, we will show that $B(u)$ is bounded. To do that again we will use the
commutator estimate (Proposition 2.1) with $n=1$, $s=s-1$, and $\sigma=s-1$,
which implies $s+n-1=s-1$. Then, for $f=r(u)$ and
$g=\Gamma^{-1}\partial_{x}w$, where $w\in H^{s-1}$, we can write

$\displaystyle||B(u)w||_{s-1}$
$\displaystyle=||[\Gamma,r(u)]\Gamma^{-1}\partial_{x}w||_{s-1}$
$\displaystyle\leq||r(u)||_{s}||\Gamma^{-1}\partial_{x}w||_{s-1}$
$\displaystyle\leq||r(u)||_{s}||w||_{s-1}$ $\displaystyle\leq c||w||_{s-1},$

where $c$ depends on $||u||_{s}$.

Moreover,

$\displaystyle||(B(u)-B(v))w||_{s-1}$
$\displaystyle=||[\Gamma,r(u)-r(v)]\Gamma^{-1}\partial_{x}w||_{s-1}$
$\displaystyle\leq||r(u)-r(v)||_{s}||\Gamma^{-1}\partial_{x}w||_{s-1}$
$\displaystyle\leq||r(u)-r(v)||_{s}||w||_{s-1}$
$\displaystyle\leq||(a+b)(u-v)+a\Gamma^{-(p+2)}[L\partial_{x}^{2},u-v]||_{s}||w||_{s-1}$
$\displaystyle\leq\big{(}||u-v||_{s}+||[L\partial_{x}^{2},u-v]_{s-p-2}\big{)}||w||_{s-1}$
$\displaystyle\leq\big{(}||u-v||_{s}+||u-v||_{s}\big{)}||w||_{s-1}$
$\displaystyle\leq\lambda_{2}||w||_{s-1},$

where $\lambda_{2}$ is a constant depending on $||u||_{s}$ and $||v||_{s}$.
Take $v=0$ in the above inequality to obtain $B(u)\in\mathcal{L}(H^{s-1})$.
This completes the proof of Lemma 2.5, and thus assumption $(A3)$. ∎

### Proof of Assumption (A4):

Since $f(u)=0$, it is trivial.

As we verify all the assumptions $(A1)-(A4)$ in Theorem 2.2, local well-
posedness for the dispersion generalized Camassa-Holm equation is established.

##  3 Comparison of Regularity

As we mentioned before, there are numerous studies in which different forms of
momentum density appear. Starting from the Camassa-Holm equation, we will
refer to some of the regularity results for the solutions of Cauchy problems
associated with these different types of evolution equations. We will verify
that the result we provide in this paper is consistent with the ones appeared
in the literature. Generalizing the dispersive effect also generalizes the
degree of the regularity obtained for the solutions.

Below, we give sample cases:

  * •

Case 1: [18] $m=1-\partial_{x}^{2}$ which leads to the classical Camassa-Holm
equation for which $L$ is the identity operator, local well-posedness is
proved for $s>3/2.$

  * •

Case 2: [15] $m=(1-\partial_{x}^{2})^{2}u$, which is a special case of L with
$p=2$, local well-posedness is proved for $s>7/2.$

  * •

Case 3: [20] $m=(\mu-\partial_{x}^{2}+\partial_{x}^{4})u$, which is a special
case of L with $p=2$, local well-posedness is proved for $s>7/2.$

Checking these results, we can confirm that the result we found in this paper
provide the sufficient regularity of the solution corresponding to the Cauchy
problem with initial data $u_{0}\in H^{s}$ and it generalizes the results
previously reported in the literature.

##  4 Discussion

Working on one of the main qualitative analysis questions related to the
dispersive equation proposed in the present work, we can declare follow up
questions as future work:

  * 1.

Can we find globally well-posed solution?

  * 2.

Are there any conserved quantities? Is the energy conserved as it is for the
Camassa-Holm equation?

  * 3.

Is there a finite time at which blow-up occurs? If there is finite-time blow
up, which norm of $u$ becomes unbounded? Is it in the form of wave breaking as
it is for the Camassa-Holm equation? How does the change in the dispersive
effect change the blow-up time?

##  5 Conclusion

In this study, we qualitatively generalize the dispersive effect through the
generalized Camassa-Holm equation and establish the local well-posedness of
the corresponding Cauchy problem by using Kato’s semigroup approach. There are
various studies in the literature for Camassa-Holm equation since it is a
dispersive equation which can model wave breaking in shallow water wave
theory. The 2:1 ratio of the coefficients corresponding to nonlinear terms
enables to write non-local form of CH equation in a simple manner. In our
case, having the operator $L$ in a closed form and being applied on nonlinear
terms as well makes things difficult. The operator $L$ with even order $p$
represents the generalization of the dispersive effect. Choosing $L$ as the
identity operator and constants $a=2$, $b=1$, the equation reduces to Camassa-
Holm equation. We obtain the results by making assumptions only on the order
of $L$. So, getting those results without writing the operator $L$ explicitly
will enable to evaluate the results of the equation in a very wide class.
After a large number of trials, we observe that it is needed to write the
quasi-linear form of the equation by collecting all nonlinear terms in the
operator $A(u)$. Therefore, in this case, $f$ becomes zero. Thus, it is enough
to verify the assumptions $(A1)-(A3)$ of the theorem. At the end, we see that
choosing the initial data $u_{0}$ from $H^{s}$, where $s>\frac{7}{2}+p$, the
local well-posedness is established. One can observe that the initial data
class needs to be more regular compared to Camassa-Holm equation.

##  6 Acknowledgement

N. D. Mutlubaş is supported by the Turkish Academy of Sciences within the
framework of the Outstanding Young Scientists Awards Program (TÜBA-
GEBIP-2022).

## References

  * [1] R. Camassa, D.D. Holm, An integrable shallow water equation with peaked solitons, Phys. Rev. Lett., 71 (1993), 1661–1664.
  * [2] R.M. Chen, Some nonlinear dispersive waves arising in compressible hyperelastic plates, Int.Journal of Eng. Sci., 44 (2006) 1188–1204.
  * [3] R. Chen, S. Zhou, Well-posedness and persistence properties for two-component higher order Camassa–Holm systems with fractional inertia operator, Nonlinear Analysis: Real World Applications, 33 (2017), 121–138. Constantin, J. Escher, On the Cauchy Problem for a family of quasilinear Hyperbolic Equations, Comm. in Part. Diff. Eq., 23 (1998), 1449-1458.
  * [4] A. Constantin, B.Kolev, Geodesic flow on the diffeomorphism group of the circle, Commentarii Mathematici Helvetici, 78(4) (2003), 787–804.
  * [5] H.H. Dai, Model equations for nonlinear dispersive waves in a compressible Mooney-Rivlin rod, Acta Mechanica, 127 (1998) 193-207.
  * [6] X. Deng, A. Chen, Global weak conservative solutions of the $\mu$-Camassa-Holm equation, Boundary Value Problems, 1 (2020), 1-10.
  * [7] D. Ding, S. Zhang, Lipschitz metric for the periodic second-order Camassa–Holm equation, Journal of Mathematical Analysis and Applications, 451(2) (2017), 990–1025.
  * [8] Y. Fu, Y. Liu, C. Qu, On the blow-up structure for the generalized periodic Camassa–Holm and Degasperis–Procesi equations, Journal of Functional Analysis, 262 (7) (2012), 3125–3158.
  * [9] D.D. Holm, L. Ó. Náraigh, C. Tronci, Singular solutions of a modified two-component Camassa-Holm equation, Physical Review E, 79 (1) (2009), 016601
  * [10] J.K. Hunter, R. Saxton, Dynamics of director fields, SIAM Journal on Applied Mathematics, 51 (6) (1991), 1498–1521.
  * [11] T. Kato, Quasi-linear equations of evolution, with applications to partial differential equations, in: Spectral Theory and Differential Equations, in: Lecture Notes in Math., Springer-Verlag, Berlin, 448 (1975), 25-70.
  * [12] B. Khesin, J. Lenells, G. Misiołek, Generalized Hunter–Saxton equation and the geometry of the group of circle diffeomorphisms, Mathematische Annalen, 342 (3) (2008), 617–656.
  * [13] D. Lannes,”The Water Waves Problem: Mathematical Analysis and Asymptotics”, AMS Mathematical Surveys and Monographs, 188, American Mathematical Society, Providence, RI, 2013.
  * [14] R. McLachlan, X. Zhang, Well-posedness of modified Camassa–Holm equations, Journal of Differential Equations, 246(8) (2009), 3241–3259.
  * [15] C. Mu, S. Zhou, R. Zeng, Well-posedness and blow-up phenomena for a higher order shallow water equation, Journal of Differential Equations, 251 (12) (2011), 3488–3499.
  * [16] A. Pazy, Semigroups of linear operators and applications to partial differential equations, 44 (2012), Springer Science & Business Media.
  * [17] E.G.Reyes, M. Zhu, Z. Qiao, Pseudo-peakons and Cauchy analysis for an integrable fifth-order equation of Camassa-Holm type, arXiv:2111.07733, 2021.
  * [18] G. Rodríguez-Blanco, On the Cauchy problem for the Camassa-Holm equation, Nonlinear Analysis: Theory, Methods & Applications, 46 (3) (2001), 309–327.
  * [19] H. Wang, Y. Fu, Non-uniform dependence on initial data for the modified $\mu$-Camassa–Holm equation, Journal of Differential Equations, 261 (11) (2016), 6099–6124.
  * [20] F. Wang, F. Li, Z. Qiao, Well-posedness and peakons for a higher-order Camassa-Holm equation, Nonlinear Analysis, 175 (2018), 210-236.
  * [21] F. Wang, F. Li, Z. Qiao, Well-posedness and peakons for a higher-order $\mu$-Camassa–Holm equation, Nonlinear Analysis, 175 (2018) 210–236.
  * [22] H. Wang, T. Luo, Y. Fu, C. Qu, Blow-up and peakons for a higher-order $\mu$-Camassa–Holm equation, Journal of Evolution Equations, 22(1) (2022) 1–33.
  * [23] H. Yamane, Local and global analyticity for $\mu$-Camassa–Holm equation, Discrete Cont. Dyn. Syst, 40 (2020), 4307-4340.


# PolyLoss: A Polynomial Expansion Perspective of Classification Loss
Functions

Zhaoqi Leng1, Mingxing Tan1, Chenxi Liu1, Ekin Dogus Cubuk2, Xiaojie Shi2,
Shuyang Cheng1,Dragomir Anguelov1
1Waymo LLC 2Google LLC
{lengzhaoqi, tanmingxing, cxliu, shuyangcheng<EMAIL_ADDRESS>
{cubuk<EMAIL_ADDRESS>

###### Abstract

Cross-entropy loss and focal loss are the most common choices when training
deep neural networks for classification problems. Generally speaking, however,
a good loss function _can_ take on much more flexible forms, and _should_ be
tailored for different tasks and datasets. Motivated by how functions can be
approximated via Taylor expansion, we propose a simple framework, named
_PolyLoss_ , to view and design loss functions as a linear combination of
polynomial functions. Our PolyLoss allows the importance of different
polynomial bases to be easily adjusted depending on the targeting tasks and
datasets, while naturally subsuming the aforementioned cross-entropy loss and
focal loss as special cases. Extensive experimental results show that the
optimal choice within the PolyLoss is indeed dependent on the task and
dataset. Simply by introducing one extra hyperparameter and adding one line of
code, our Poly-1 formulation outperforms the cross-entropy loss and focal loss
on 2D image classification, instance segmentation, object detection, and 3D
object detection tasks, sometimes by a large margin.

Task | ImageNet classification | COCO det. and seg. | Waymo Open Dataset 3D detection
---|---|---|---
Default loss | Cross-entropy | Cross-entropy | Focal loss
Model | ENetV2-L(21K) | ENetV2-L(1K) | Mask R-CNN | PointPillars Car | PointPillars Ped | RSN Car | RSN Ped
Baseline | 45.8 | 86.8 | 47.2 | 42.3 | 63.3 | 68.9 | 78.4 | 79.4
PolyLoss |  46.4 (+0.6) |  87.2 (+0.4) |  49.7 (+2.5) |  44.4 (+2.1) |  63.7 (+0.4) |  69.6 (+0.7) |  78.9 (+0.5) |  80.2 (+0.8)

Table 1: PolyLoss outperforms cross-entropy and focal loss on various models
and tasks. Results are for the simplest Poly-1, which has only a single
hyperparameter. On ImageNet (Deng et al., 2009), our PolyLoss improves both
pretraining and finetuning for the recent EfficientNetV2 (Tan & Le, 2021); on
COCO (Lin et al., 2014), PolyLoss improves both 2D detection and segmentation
AR for Mask-RCNN (He et al., 2017); on Waymo Open Dataset (WOD) (Sun et al.,
2020), PolyLoss improves 3D detection AP for the widely used PointPillars
(Lang et al., 2019) and the very recent Range Sparse Net (RSN) (Sun et al.,
2021). Details are in Table 4, 5, 7.

##  1 Introduction

Loss functions are important in training neural networks. In principle, a loss
function could be any (differentiable) function that maps predictions and
labels to a scalar. Therefore, designing a good loss function is generally
challenging due to its large design space, and designing a universal loss
function that works across different tasks and datasets is even more
challenging: for example, L1 / L2 losses are commonly used for regression
tasks, but they are rarely used for classification tasks; focal loss is often
used to alleviate the overfitting issue of cross-entropy loss for imbalanced
object detection datasets (Lin et al., 2017), but it is not shown to
consistently help other tasks. Many recent works have also explored new loss
functions via meta-learning, ensembling or compositing different losses
(Hajiabadi et al., 2017; Xu et al., 2018; Gonzalez & Miikkulainen, 2020b; a;
Li et al., 2019).

In this paper, we propose _PolyLoss_ : a novel framework for understanding and
designing loss functions. Our key insight is to decompose commonly used
classification loss functions, such as cross-entropy loss and focal loss, into
a series of weighted polynomial bases. They are decomposed in the form of
$\sum_{j=1}^{\infty}\alpha_{j}(1-P_{t})^{j}$, where
$\alpha_{j}\in\mathbb{R}^{+}$ is the polynomial coefficient and $P_{t}$ is the
prediction probability of the target class label. Each polynomial base
$(1-P_{t})^{j}$ is weighted by a corresponding polynomial coefficient
$\alpha_{j}$, which enables us to easily adjust the importance of different
bases for different applications. When $\alpha_{j}=1/j$ for all $j$, our
PolyLoss becomes equivalent to the commonly used cross-entropy loss, but this
coefficient assignment may not be optimal.

Our study shows that, in order to achieve better results, it is necessary to
adjust polynomial coefficients $\alpha_{j}$ for different tasks and datasets.
Since it is impossible to adjust an infinite number of $\alpha_{j}$, we
explore various strategies with a small degree of freedom. Perhaps
surprisingly, we observe that simply adjusting the single polynomial
coefficient for the leading polynomial, which we denote $L_{\text{Poly-1}}$,
is sufficient to achieve significant improvements over the commonly used
cross-entropy loss and focal loss. Overall, our contribution can be summarized
as:

  * •

Insights on common losses: We propose a unified framework, named PolyLoss, to
rethink and redesign loss functions. This framework helps to explain cross-
entropy loss and focal loss as two special cases of the PolyLoss family (by
horizontally shifting polynomial coefficients), which was not recognized
before. This new finding motivates us to investigate new loss functions that
vertically adjust polynomial coefficients, shown in Figure 1.

  * •

New loss formulation: We evaluate different ways of vertically manipulating
polynomial coefficients to simplify the hyperparameters search space. We
propose a simple and effective Poly-1 loss formulation which only introduces
one hyperparameter and one line of code.

  * •

New findings: We identify that focal loss, though effective for many detection
tasks, is suboptimal for the imbalanced ImageNet-21K. We find the leading
polynomial contributes to a large portion of the gradient during training, and
its coefficient correlates to the prediction confidence $P_{t}$. In addition,
we provide an intuitive explanation on how to leverage this correlation to
design good PolyLoss tailored to imbalanced datasets.

  * •

Extensive experiments: We evaluate our PolyLoss on different tasks, models,
and datasets. Results show PolyLoss consistently improves the performance on
all fronts, summarized in Table 1, which includes the state-of-the-art
classifiers EfficientNetV2 and detectors RSN.

##  2 Related Work

Cross-entropy loss is used in popular and current state-of-the-art models for
perception tasks such as classification, detection and semantic segmentation
(Tan & Le, 2021; He et al., 2017; Zoph et al., 2020; Tao et al., 2020).
Various losses are proposed to improve cross-entropy loss (Lin et al., 2017;
Law & Deng, 2018; Cui et al., 2019; Zhao et al., 2021). Unlike prior works,
the goal of this paper is to provide a unified framework for systematically
designing a better classification loss function.

#### Loss for class imbalance

Training detection models, especially single-stage detectors, is difficult due
to class imbalance. Common approaches such as hard example mining and
reweighing are developed to address the class imbalance issue (Sung, 1996;
Viola & Jones, 2001; Felzenszwalb et al., 2010; Shrivastava et al., 2016; Liu
et al., 2016; Bulo et al., 2017). As one of these approaches, focal loss is
designed to mitigate the class imbalance issue by focusing on the hard
examples and is used to train state-of-the-art 2D and 3D detectors (Lin et
al., 2017; Tan et al., 2020; Du et al., 2020; Shi et al., 2020; Sun et al.,
2021). In our work, we found that focal loss is suboptimal for the imbalanced
ImageNet-21K. Using the PolyLoss framework, we discover a better loss
function, which performs the opposite role of focal loss. We further provide
intuitive understanding of why it is important to design different loss
functions tailored to different imbalanced datasets using the PolyLoss
framework.

#### Robust loss to label noise

Another direction of research is to design loss functions that are robust to
label noise (Ghosh et al., 2015; 2017; Zhang & Sabuncu, 2018; Wang et al.,
2019; Oksuz et al., 2020; Menon et al., 2019). A commonly used approach is to
incorporate noise robust loss function such as Mean Absolute Error (MAE) into
cross-entropy loss. In particular, Taylor cross entropy loss is proposed to
unify MAE and cross-entropy loss by expanding the cross-entropy loss in
$(1-P_{t})^{j}$ polynomial bases (Feng et al., 2020). By truncating the
higher-order polynomials, they show truncated cross-entropy loss function is
closer to MAE, which is more robust to label noise on datasets with synthetic
label noise. In contrast, our PolyLoss provides a more general framework to
design loss functions for different datasets by manipulating polynomial
coefficients, which includes dropping higher-order polynomials proposed in
Feng et al. (2020). Our experiments in subsection 4.1 show the loss proposed
in Feng et al. (2020) performs worse than cross-entropy loss on the clean
ImageNet dataset.

#### Learned loss functions

Several recent works demonstrate learning the loss function during training
via gradient descent or meta learning (Hajiabadi et al., 2017; Xu et al.,
2018; Gonzalez & Miikkulainen, 2020a; Li et al., 2019; 2020). Notably,
TaylorGLO utilizes CMA-ES to optimize multivariate Taylor parameterization of
a loss function and learning rate schedule during training (Hansen &
Ostermeier, 1996; Gonzalez & Miikkulainen, 2020b). Due to the search space
scale with the order of polynomials, the paper demonstrates that using the
third-order parameterization (8 parameters), the learned loss function
schedule outperforms cross-entropy loss on 10-class classification problems.
Our paper (2(a)), on the other hand, shows for 1000-class classification
tasks, hundreds of polynomials are needed. This results in a prohibitively
large search space. Our proposed Poly-1 formulation mitigates the challenge of
the large search space and do not rely on advanced black-box optimization
algorithms. Instead, we show a simple grid search over one hyperparameter can
lead to significant improvement on all tasks that we investigate.

##  3 PolyLoss

Figure 1: Unified view of cross-entropy loss, focal loss, and PolyLoss.
PolyLoss $\sum_{j=1}^{\infty}\alpha_{j}(1-P_{t})^{j}$ is a more general
framework, where $P_{t}$ stands for prediction probability of the target
class. Left: Polyloss is more flexible: it can be steeper (deep red) than
cross-entropy loss (black) or flatter (light red) than focal loss (green).
Right: Polynomial coefficients of different loss functions in the bases of
$(1-P_{t})^{j}$, where $j\in\mathbb{Z}^{+}$. Black dash lines are drawn to
show the trend of polynomial coefficients. In the PolyLoss framework, focal
loss can only shift the polynomial coefficients horizontally (green arrow),
see Equation 2, whereas the proposed PolyLoss framework is more general, which
also allows vertical adjustment (red arrows) of the polynomial coefficient for
each polynomial term.

PolyLoss provides a framework for understanding and improving the commonly
used cross-entropy loss and focal loss, visualized in Figure 1. It is inspired
from the Taylor expansion of cross-entropy loss (Equation 1) and focal loss
(Equation 2) in the bases of $(1-P_{t})^{j}$:

$\displaystyle L_{\text{CE}}=-\log(P_{t})$
$\displaystyle=\sum_{j=1}^{\infty}1/j(1-P_{t})^{j}=(1-P_{t})+1/2(1-P_{t})^{2}...$
(1) $\displaystyle L_{\text{FL}}=-(1-P_{t})^{\gamma}\log(P_{t})$
$\displaystyle=\sum_{j=1}^{\infty}1/j(1-P_{t})^{j+\gamma}=(1-P_{t})^{1+\gamma}+1/2(1-P_{t})^{2+\gamma}...$
(2)

where $P_{t}$ is the model’s prediction probability of the target ground-truth
class.

#### Cross-entropy loss as PolyLoss

Using the gradient descent method to optimize the cross-entropy loss requires
taking the gradient with respect to $P_{t}$. In the PolyLoss framework, an
interesting observation is that the coefficients $1/j$ exactly cancel the
$j$th power of the polynomial bases, see Equation 1. Thus, the gradient of
cross-entropy loss is simply the sum of polynomials $(1-P_{t})^{j}$, shown in
Equation 3.

$\displaystyle-\frac{\differential L_{\text{CE}}}{\differential
P_{t}}=\sum_{j=1}^{\infty}(1-P_{t})^{j-1}=1+(1-P_{t})+(1-P_{t})^{2}...$ (3)

The polynomial terms in the gradient expansion capture different sensitivity
with respect to $P_{t}$. The leading gradient term is $1$, which provides a
constant gradient regardless of the value of $P_{t}$. On the contrary, when
$j\gg 1$, the $j$th gradient term is strongly suppressed when $P_{t}$ gets
closer to $1$.

#### Focal loss as PolyLoss

In the PolyLoss framework, Equation 2, it is apparent that the focal loss
simply shifts the power $j$ by the power of a modulating factor $\gamma$. This
is equivalent to horizontally shifting all the polynomial coefficients by
$\gamma$ as shown in Figure 1. To understand the focal loss from a gradient
prospective, we take the gradient of the focal loss (Equation 2) with respect
to $P_{t}$:

$\displaystyle-\frac{\differential L_{\text{FL}}}{\differential
P_{t}}=\sum_{j=1}^{\infty}(1+\gamma/j)(1-P_{t})^{j+\gamma-1}=(1+\gamma)(1-P_{t})^{\gamma}+(1+\gamma/2)(1-P_{t})^{1+\gamma}...$
(4)

For a positive $\gamma$, the gradient of focal loss drops the constant leading
gradient term, $1$, in the cross-entropy loss, see Equation 3. As discussed in
the previous paragraph, this constant gradient term causes the model to
emphasize the majority class, since its gradient is simply the total number of
examples for each class. By shifting the power of all the polynomial terms by
$\gamma$, the first term then becomes $(1-P_{t})^{\gamma}$, which is
suppressed by the power of $\gamma$ to avoid overfitting to the already
confident (meaning $P_{t}$ close to $1$) majority class. More details are
shown in section 12.

#### Connection to regression and general form

Representing the loss function in the PolyLoss framework provides an intuitive
connection to regression. For classification tasks where $y=1$ is the
effective probability of the ground-truth label, the polynomial bases
$(1-P_{t})^{j}$ can be expressed as $(y-P_{t})^{j}$. Thus both cross-entropy
loss and focal loss can be interpreted as a weighted ensemble of distances
between the prediction and label to the $j$th power. However, a fundamental
question in those losses: Are the coefficients in front of the regression
terms optimal?

In general, PolyLoss is a monotone decreasing function111We only consider the
case all $\alpha_{j}\geq 0$ in this paper for simplicity. There exist monotone
decreasing functions on $[0,1]$ with some $\alpha_{j}$ negative, for example
$\sin(1-P_{t})=\sum_{j=0}^{\infty}(-1)^{j}/(2j+1)!(1-P_{t})^{2j+1}$. on
$[0,1]$ which can be expressed as $\sum_{j=1}^{\infty}\alpha_{j}(1-P_{t})^{j}$
and provides a flexible framework to adjust each coefficient222To ensure
series converges, we require
$1/\limsup_{j\rightarrow\infty}{\sqrt[j]{|\alpha_{j}|}}\geq 1$ for $P_{t}$ in
$(0,1]$. For $P_{t}=0$ we don’t require point-wise convergence; in fact cross-
entropy and focal loss both go to $+\infty$.. PolyLoss can be generalized to
non-integer $j$, but for simplicity we only focus on integer power
($j\in\mathbb{Z}^{+}$) in this paper. In the next section, we investigate
several strategies on designing better loss functions in the PolyLoss
framework via manipulating $\alpha_{j}$.

##  4 Understanding the Effect of Polynomial Coefficients

In the previous section, we established the PolyLoss framework and showed that
cross-entropy loss and focal loss simply correspond to different polynomial
coefficients, where focal loss horizontally shifts the polynomial coefficients
of cross-entropy loss.

In this section, we propose the final loss formulation Poly-1. We study in
depth how vertically adjusting polynomial coefficients, shown in Figure 1, may
affect training. Specifically, we explore three different strategies in
assigning polynomial coefficients: dropping higher-order terms; adjusting
multiple leading polynomial coefficients; and adjusting the first polynomial
coefficient, summarized in Table 2. We find adjusting the first polynomial
coefficient (Poly-1 formulation) leads to _maximal_ gain while requiring
_minimal_ code change and hyperparameter tuning.

In these explorations, we experiment with 1000-class ImageNet (Deng et al.,
2009) classification. We abbreviate it as ImageNet-1K to differentiate it from
the full version, which contains 21K classes. We use ResNet-50 (He et al.,
2016) and its training hyperparameters without modification.333Code at
https://github.com/tensorflow/tpu/tree/master/models/official/

| Polynomial expansion in the basis of $(1-P_{t})$ | Loss
---|---|---
Cross-entropy loss | $(1-P_{t})+1/2(1-P_{t})^{2}+...+1/N(1-P_{t})^{N}+1/(N+1)(1-P_{t})^{N+1}+...$ | $L_{\text{CE}}=-\log(P_{t})$
Drop poly. (Sec 4.1) |  $(1-P_{t})+1/2(1-P_{t})^{2}+...+1/N(1-P_{t})^{N}$ (drop the remaining terms) | $L_{\text{Drop}}=L_{\text{CE}}-\sum_{j=N}^{\infty}1/j(1-P_{t})^{j}$
Poly-N (Sec 4.2) | $(\hbox{\pagecolor{red!10}$\displaystyle\epsilon_{1}$}+1)(1-P_{t})+...+(\hbox{\pagecolor{red!10}$\displaystyle\epsilon_{N}$}+1/N)D_{t}^{N}+1/(N+1)(1-P_{t})^{N+1}+...$ | $L_{\text{Poly-N}}=L_{\text{CE}}+\sum_{j=1}^{N}\epsilon_{j}(1-P_{t})^{i}$
Poly-1 (Sec 4.3) | $(\hbox{\pagecolor{red!10}$\displaystyle\epsilon_{1}$}+1)(1-P_{t})+1/2(1-P_{t})^{2}+...+1/N(1-P_{t})^{N}+1/(N+1)(1-P_{t})^{N+1}+...$ | $L_{\text{Poly-1}}=L_{\text{CE}}+\epsilon_{1}(1-P_{t})$

Table 2: Comparing different losses in the PolyLoss framework. Dropping higher
order polynomial, proposed in prior works, truncates all higher order
($N+1\rightarrow\infty$) polynomial terms. We propose Poly-N loss, which
perturbs the leading N polynomial coefficients. Poly-1 is the final loss
formulation, which further simplifies Poly-N and only requires a simple grid
search over one hyperparameter. The differences compared to cross-entropy loss
are highlighted in red.

###  4.1 $L_{Drop}$: Revisiting dropping higher-order polynomial terms

Prior works (Feng et al., 2020; Gonzalez & Miikkulainen, 2020b) have shown
dropping the higher-order polynomials and tuning the leading polynomials can
improve model robustness and performance. We adopt the same loss formulation
$L_{\text{Drop}}=\sum_{j=1}^{N}1/j(1-P_{t})^{j}$, as in Feng et al. (2020),
and compare their performance with the baseline cross-entropy loss on
ImageNet-1K. As shown in 2(a), we need to sum up more than 600 polynomial
terms to match the accuracy of cross-entropy loss. Notably, removing higher-
order polynomials cannot simply be interpreted as adjusting the learning rate.
To verify this, 2(b) compares the performance for different learning rates
with various cutoffs: no matter we increase or decrease the learning rate from
the original value of $0.1$, the accuracy worsens. Additional hyperparameter
tuning is shown in section 9.

(a) Truncating the infinite sum of polynomials in cross-entropy loss to $N$
reduces accuracy.

(b) Adjusting the learning rate (default 0.1) of $L_{\text{Drop}}$ does not
improve the classification accuracy.

Figure 2: Training ResNet-50 on ImageNet-1K requires hundreds of polynomial
terms to reproduce the same accuracy as cross-entropy loss.

To understand why higher-order terms are important, we consider the residual
sum after removing the first $N$ polynomial terms from cross-entropy loss:
$R_{\text{N}}=L_{\text{CE}}-L_{\text{Drop}}=\sum_{j=N+1}^{\infty}1/j(1-P_{t})^{j}$.

######  Theorem 1.

For any small $\zeta>0$, $\delta>0$ if
$N>\log_{1-\delta}{(\zeta\cdot\delta)}$, then for any $p\in[\delta,1]$, we
have $|R_{N}(p)|<\zeta$ and $|R_{N}^{\prime}(p)|<\zeta$. (Proof in section 7)

Hence, taking a large $N$ is necessary to ensure $L_{\text{Drop}}$ is
uniformly close to $L_{\text{CE}}$ in the perspectives of loss and loss
derivative on $[\delta,1]$. For a fixed $\zeta$, as $\delta$ approaches $0$,
$N$ grows rapidly. Our experimental results align with the theorem. The
higher-order ($j>N+1$) polynomials play an important role during the early
stages of training, where $P_{t}$ is typically close to zero. For example,
when $P_{t}\sim 0.001$, according to Equation 3, the coefficient of the 500th
term’s gradient is $0.999^{499}\sim 0.6$, which is fairly large. Different
from aforementioned prior works, our results show that we cannot easily reduce
the number of polynomial coefficients $\alpha_{j}$ by excluding the higher-
order polynomials.

Dropping higher order polynomials is equivalent to pushing all the higher
order ($j>N+1$) polynomial coefficients $\alpha_{j}$ vertically to zero in the
PolyLoss framework. Since simply setting coefficients to zero is suboptimal
for training ImageNet-1K, in the following sections, we investigate how to
manipulate polynomial coefficient beyond setting them to zero in the PolyLoss
framework. In particular, we aim to propose a simple and effective loss
function that requires minimal tuning.

###  4.2 $L_{\text{Poly-N}}$: Perturbing leading polynomial coefficients

In this paper, we propose an alternative way of designing a new loss function
in the PolyLoss framework, where we adjust the coefficients of each
polynomial. In general, there are infinitely many polynomial coefficients
$\alpha_{j}$ need to be tuned. Thus, it is infeasible to optimize the most
general loss:

$\displaystyle L_{\text{Poly}}$
$\displaystyle=\alpha_{1}(1-P_{t})+\alpha_{2}(1-P_{t})^{2}+...+\alpha_{N}(1-P_{t})^{N}+...=\sum_{j=1}^{\infty}\alpha_{j}(1-P_{t})^{j}$
(5)

The previous section (subsection 4.1) has shown that hundreds of polynomials
are required in training to do well on tasks such as ImageNet-1K
classification. If we naively truncate the infinite sum in Equation 5 to the
first few hundreds terms, tuning coefficients for so many polynomials still
results in a prohibitively large search space. In addition, collectively
tuning many coefficients also does not outperform cross-entropy loss, details
in section 10.

To tackle this challenge, we propose to perturb the leading polynomial
coefficients in cross-entropy loss, while keeping the rest the same. We denote
the proposed loss formulation as Poly-N, where N stands for the number of
leading coefficients that will be tuned.

$\displaystyle L_{\text{Poly-N}}$
$\displaystyle=\underbrace{(\epsilon_{1}+1)(1-P_{t})+...+(\epsilon_{N}+1/N)(1-P_{t})^{N}}_{\text{perturbed
by }\epsilon_{j}}+\underbrace{1/(N+1)(1-P_{t})^{N+1}+...}_{\text{same as
}L_{\text{CE}}}$
$\displaystyle=-\log(P_{t})+\sum_{j=1}^{N}\epsilon_{j}(1-P_{t})^{j}$ (6)

| CE loss | N=1 | N=2 | N=3
---|---|---|---|---
N-dim. grid search | 76.3 | 76.7 | 76.8 | –
Greedy grid search | 76.3 | 76.7 | 76.7 | 76.7

Table 3: $\bm{L_{\text{Poly-N}}}$ outperforms cross-entropy loss on
ImageNet-1K.

Here, we replace the $j$th polynomial coefficient in cross-entropy loss $1/j$
with $1/j+\epsilon_{j}$, where $\epsilon_{j}\in[-1/j,\infty)$ is the
perturbation term. This allows us to pinpoint the first $N$ polynomials
without the need to worry about the infinitely many higher-order ($j>N+1$)
coefficients, as in Equation 5.

Table 3 shows $L_{\text{Poly-N}}$ outperforms the baseline cross-entropy loss
accuracy. We explore N-dimensional grid search and greedy grid search of
$\epsilon_{j}$ in $L_{\text{Poly-N}}$ up to $N=3$ and find that simply
adjusting the coefficient of the first polynomial ($N=1$) leads to better
classification accuracy. Performing 2D grid search ($N=2$) can further boost
the accuracy. However, the additional gain is small (+0.1) compared to
adjusting only the first polynomial (+0.4).

###  4.3 $L_{\text{Poly-1}}$: simple and effective

(a) PolyLoss family $L_{\text{Poly-1}}=-\log(P_{t})+\epsilon_{1}(1-P_{t})$,
where $\epsilon_{1}\in\\{-1,0,1,\ldots,8\\}$.

(b) Percentage of gradient from the first polynomial versus the rest
(infinitely many) polynomials.

Figure 3: The first polynomial plays an important role for training ResNet-50
on ImageNet-1K. (a) Increasing the coefficient of the first polynomial term
($\epsilon_{1}>0$) consistently improves the ResNet50 prediction accuracy. Red
dash line shows the accuracy when using cross-entropy loss. Mean and stdev of
three runs are plotted. (b) The first polynomial $(1-P_{t})$ contributes more
than half of the cross-entropy gradient at the last 65% of the training steps,
which highlights the importance of tuning the first polynomial. The red dash
line shows the crossover.

As shown in the previous section, we find tuning the first polynomial term
leads to the most significant gain. In this section, we further simplify the
Poly-N formulation and focus on evaluating Poly-1, where only the first
polynomial coefficient in cross-entropy loss is modified.

$L_{\text{Poly-1}}=(1+\epsilon_{1})(1-P_{t})+1/2(1-P_{t})^{2}+...=-\log(P_{t})+\epsilon_{1}(1-P_{t})$
(7)

We study the effect of different first term scaling on the accuracy and
observe that increasing the first polynomial coefficient can systematically
increase the ResNet-50 accuracy, as shown in 3(a). This result suggests that
the cross-entropy loss is suboptimal in terms of polynomial coefficient
values, and increasing the first polynomial coefficient leads to consistent
improvement, which is comparable to other training techniques (section 11).

3(b) shows the leading polynomial contributes to more than half of the cross-
entropy gradient during training for the majority of the time, which
highlights the significance of the first polynomial term ($1-P_{t}$) compared
to the rest of the infinite many terms. Therefore, in the remaining of the
paper, we adopt the form of $L_{\text{Poly-1}}$ and primarily focus on
adjusting the leading polynomial coefficient. As is evident from Equation 7,
it only modifies the original loss implementation by a single line of code
(adding a $\epsilon_{1}(1-P_{t})$ term on top of cross-entropy loss).

Note that, all the training hyperparameters are optimized for cross-entropy
loss. Even so, a simple grid search on the first polynomial coefficients in
the Poly-1 formulation significantly increases the classification accuracy. We
find optimizing other hyperparameters for $L_{\text{Poly-1}}$ leads to higher
accuracy, and show more details in section 8.

##  5 Experimental Results

In this section, we compare our PolyLoss against the commonly used cross-
entropy loss and focal loss on various tasks, models, and datasets. For the
following experiments, we adopt the default training hyperparameters in the
public repositories without any tuning. Nevertheless, Poly-1 formulation leads
to consistent advantage over default loss functions at the cost of a simple
grid search.

###  5.1 $L_{\text{Poly-1}}$ improves 2D image classification on ImageNet

Image classification is a fundamental problem in computer vision, and progress
on image classification has led to progress on many related computer vision
tasks. In terms of the network architecture, in addition to the ResNet-50
already used in section 4, we also experiment with the state-of-the-art
EfficientNetV2 (Tan & Le, 2021). We use the ImageNet settings in (Tan & Le,
2021) except for replacing the original cross-entropy loss with our PolyLoss
$L_{Poly-1}$ with different values of $\epsilon_{1}$. In terms of the dataset,
in addition to the ImageNet-1K dataset already used in section 4, we also
consider ImageNet-21K, which has about 13M training images with 21,841
classes. We will study both the ImageNet-21K pretraining results and the
ImageNet-1K finetuning results.

Pretraining EfficientNetV2-L on ImageNet-21K, then finetuning it on
ImageNet-1K can improve classification accuracy (Tan & Le, 2021). Here, we
follow the same pretraining and finetuning schedule as reported in Tan & Le
(2021) without modification444 Code at
https://github.com/google/automl/tree/master/efficientnetv2 but replace the
cross-entropy loss with
$L_{\text{Poly-1}}=-\log(P_{t})+\epsilon_{1}(1-P_{t})$. We reserve 25,000
images from the training set as minival to search the optimal $\epsilon_{1}$.

Figure 4: PolyLoss improves EfficientNetV2 family on the speed-accuracy Pareto
curve. Validation accuracy of EfficientNetV2 models pretrained on ImageNet-21K
are plotted. PolyLoss outperforms cross-entropy loss with about $\times$2
speed-up.

#### Pretraining on ImageNet-21K

Figure 4 highlights the importance of using tailored loss function when
pretraining model on ImageNet-21K dataset. A simple grid search over
$\epsilon_{1}\in\\{0,1,2,\ldots,7\\}$ in $L_{\text{Poly-1}}$ without changing
other default hyperparameters leads to around 1% accuracy gain for all SOTA
EfficientNetV2 models with different sizes. The accuracy improvement of using
a better loss function nearly matches the improvement of scaling up the model
architecture (S to M and M to L).

Surprisingly, see 5(a), increasing the weight of the leading polynomial
coefficient improves the accuracy of pretraining on ImageNet-21K (+0.6),
whereas reducing it lowers the accuracy (-0.9). Setting $\epsilon_{1}=-1$
truncates the leading polynomial term in the cross-entropy loss (Equation 1),
which is similar to having a focal loss with $\gamma=1$ (Equation 2). However,
the opposite change, where $\epsilon_{1}>0$, improves the accuracy on the
imbalanced ImageNet-21K.

We hypothesize the prediction of the imbalanced ImageNet-21K is not confident
enough ($P_{t}$ is small), and using positive $\epsilon_{1}$ PolyLoss leads to
more confident predictions. To validate our hypothesis, we plot $P_{t}$ as a
function of training steps in 5(b). We observe that $\epsilon_{1}$ directly
controls the mean $P_{t}$ over all classes. Using positive $\epsilon_{1}$
PolyLoss leads to more confident prediction (higher $P_{t}$). On the other
hand, negative $\epsilon_{1}$ PolyLoss lowers the confidence.

(a) Validation accuracy of EfficientNetV2-L on ImageNet-21K. PolyLoss with
positive $\epsilon_{1}$ outperforms baseline cross-entropy loss (red dash
line).

(b) Positive $\epsilon_{1}=1$ (dark) increases the prediction confidence,
while negative $\epsilon_{1}=-1$ (light) decreases the prediction confidence.

Figure 5: PolyLoss improves EfficientNetV2-L by increasing prediction
confidence $P_{t}$.

EfficientNetV2-L | $L_{\text{CE}}$ | $L_{\text{Poly-1}}$ | Improv.
---|---|---|---
ImageNet-21K | 45.8 | 46.4 | +0.6
ImageNet-1K | 86.8 | 87.2 | +0.4

Table 4: PolyLoss improves classification accuracy on ImageNet validation set.
We set $\epsilon_{1}=2$ for both.

#### Fine tuning on ImageNet-1K

After pretraining on ImageNet-21K, we take the EfficientNetV2-L checkpoint and
finetune it on ImageNet-1K, using the same procedure as Tan & Le (2021) except
for replacing the original cross-entropy loss with the Poly-1 formulation.
PolyLoss improves the finetuning accuracy by 0.4%, advancing the ImageNet-1K
top-1 accuracy from 86.8% to 87.2%.

###  5.2 $L_{\text{Poly-1}}$ improves 2D instance segmentation and object
detection on COCO

Instance segmentation and object detection require localizing objects in an
image in addition to recognizing them: the former in the form of arbitrary
shapes and the latter in the form of bounding boxes. For both instance
segmentation and object detection, we use the popular COCO (Lin et al., 2014)
dataset, which contains 80 object classes. We choose Mask R-CNN (He et al.,
2017) as the representative model for instance segmentation and object
detection. These models optimize multiple losses, e.g.
$L_{\text{MaskRCNN}}=L_{\text{cls}}+L_{\text{box}}+L_{\text{mask}}$. For the
following experiments, we only replace the $L_{\text{cls}}$ with PolyLoss and
leave other losses intact. Results are summarized in Table 5.

| Loss | Box | Mask
---|---|---|---
|  | AP | AR | AP | AR
Mask R-CNN $L_{\text{CE}}$ | $-\log(P_{t})$ | 35.0$\pm$ 0.09 | 47.2$\pm$ 0.16 | 31.3 $\pm$ 0.09 | 42.3 $\pm$ 0.02
Mask R-CNN $L_{\text{Poly-1}}$ | $-\log(P_{t})-(1-P_{t})$ | 35.3 $\pm$ 0.12 | 49.7$\pm$ 0.07 | 31.6 $\pm$ 0.11 | 44.4 $\pm$ 0.07
Improvement | - | +0.3 | +2.5 | +0.3 | +2.1

Table 5: PolyLoss improves detection results on COCO validation set. Bounding
box and instance segmentation mask average-precision (AP) and average-recall
(AR) are reported for Mask R-CNN model with a ResNet-50 backbone. Mean and
stdev of three runs are reported.

Reducing the leading polynomial coefficient improves Mask R-CNN AP and AR. In
training Mask R-CNN, we use the training schedule optimized for cross-entropy
loss,555Code at https://github.com/tensorflow/tpu/tree/master/models/official
and replace the cross-entropy loss with
$L_{Poly-1}=-\log(P_{t})+\epsilon_{1}(1-P_{t})$ for the classification loss
$L_{cls}$, where $\epsilon_{1}\in\\{-1.0,-0.8,-0.6,-0.4,-0.2,0,0.5,1.0\\}$. We
ensure the leading coefficient is positive, i.e. $\epsilon_{1}\geq-1$. Our
results in 6(a) show systematic improvements of box AP, box AR, mask AP, and
mask AR as we reduce the weight of the first polynomial by using negative
$\epsilon_{1}$ values. Note that Poly-1 ($\epsilon=-1$) not only improves AP
but also significantly increases AR, shown in Table 5.

(a) Bound box AP, AR and Mask AP, AR increase as $\epsilon_{1}$ decreases.
Negative $\epsilon_{1}$ outperforms cross-entropy loss (red dash line).

(b) Negative $\epsilon_{1}=-1$ (light) reduces the overconfident prediction
$P_{t}$.

Figure 6: PolyLoss improves Mask R-CNN by lowering overconfident predictions.
Mean and stdev of three runs are plotted.

Tailoring loss function to datasets and tasks is important. ImageNet-21K and
COCO are both imbalanced but the optimal $\epsilon$ for PolyLoss are opposite
in sign, i.e. $\epsilon=2$ for ImageNet-21K classification and $\epsilon=-1$
for Mask R-CNN detection. We plot the $P_{t}$ of the Mask R-CNN classification
head and found the original prediction is overly confident ($P_{t}$ is close
to $1$) on the imbalanced COCO dataset, thus using a negative $\epsilon$
lowers the prediction confidence, as shown in 6(b). This effect is similar to
label smoothing (Szegedy et al., 2016) and confidence penalty (Pereyra et al.,
2017), but unlike those methods, as long as $0>\epsilon>-1$, PolyLoss lowers
the gradients of overconfident predictions but will not encourage incorrect
predictions or directly penalize prediction confidence.

###  5.3 $L_{\text{Poly-1}}$ improves 3D object detection on Waymo Open
Dataset

| Polynomial expansion in the basis of $(1-P_{t})$ | Loss
---|---|---
Focal loss | $(1-P_{t})^{\gamma+1}+1/2(1-P_{t})^{\gamma+2}+1/3(1-P_{t})^{\gamma+3}+...$ | $L_{\text{FL}}=-(1-P_{t})^{\gamma}\log(P_{t})$
Poly-1 (PointPillars) | $(\hbox{\pagecolor{red!10}$\displaystyle\epsilon_{1}$}+1)(1-P_{t})^{\gamma+1}+1/2(1-P_{t})^{\gamma+2}+1/3(1-P_{t})^{\gamma+3}+...$ | $L^{\text{FL}}_{\text{Poly-1}}=L_{\text{FL}}+\epsilon_{1}(1-P_{t})^{\gamma+1}$
Poly-1∗ (RSN) | $\hbox{\pagecolor{red!10}$\displaystyle\text{(drop first) }$}(1/2+\hbox{\pagecolor{red!10}$\displaystyle\epsilon_{2}$})(1-P_{t})^{\gamma+2}+1/3(1-P_{t})^{\gamma+3}+...$ | $L^{\text{FL}}_{\text{Poly-1}^{*}}=L_{\text{FL}}-(1-P_{t})^{\gamma+1}+\epsilon_{2}(1-P_{t})^{\gamma+2}$

Table 6: PolyLoss vs. focal loss for 3D detection models. Differences are
highlighted in red. We found the best Poly-1 for PointPillars is
$\epsilon_{1}=-1$, which is equivalent to dropping the first term. Therefore,
for RSN, we drop the first term and tune the new leading polynomial
$(1-P_{t})^{\gamma+2}$.

Detecting 3D objects from LiDAR point clouds is an important topic and can
directly benefit autonomous driving applications. We conduct these experiments
on the Waymo Open Dataset (Sun et al., 2020). Similar to 2D detectors, 3D
detection models are commonly based on single-stage and two-stage
architectures. Here, we evaluate our PolyLoss on two models: a popular single-
stage PointPillars model (Lang et al., 2019); and a state-of-the-art two-stage
Range Sparse Net (RSN) model (Sun et al., 2021). Both models rely on multi-
task loss functions during training. Here, we focus on improving the
classification focal loss by replacing it with PolyLoss. Similar to the 2D
perception cases, we adopt the Poly-1 formulation to improve upon focal loss,
shown in Table 6.

| Loss | BEV | 3D
---|---|---|---
|  | AP/APH L1 | AP/APH L2 | AP/APH L1 | AP/APH L2
Vehicle (IoU=0.7)
PointPillars $L_{\text{FL}}$ | $-(1-P_{t})^{2}\log(P_{t})$ | 82.5/81.5 | 73.9/72.9 | 63.3/62.7 | 55.2/54.7
PointPillars $L^{FL}_{\text{Poly-1}}$ | $-(1-P_{t})^{2}\log(P_{t})-(1-P_{t})^{3}$ | 83.6/82.5 | 74.8/73.7 | 63.7/63.1 | 55.5/55.0
Improvement | - | +1.1/+1.0 | +0.9/+0.8 | +0.4/+0.7 | +0.3/+0.3
RSN $L_{\text{FL}}$ | $-(1-P_{t})^{2}\log(P_{t})$ | 91.3/90.8 | 82.6/82.2 | 78.4/78.1 | 69.5/69.1
RSN $L^{FL}_{\text{Poly-1}^{*}}$ | $-(1-P_{t})^{2}\log(P_{t})-(1-P_{t})^{3}-0.4(1-P_{t})^{4}$ | 91.5/90.9 | 82.7/82.1 | 78.9/78.4 | 69.9/69.5
Improvement | - | +0.2/+0.1 |  +0.1/-0.1 | +0.5/+0.3 | +0.4/+0.4
Pedestrian (IoU=0.5)
PointPillars $L_{\text{FL}}$ | $-(1-P_{t})^{2}\log(P_{t})$ | 76.0/62.0 | 67.2/54.6 | 68.9/56.6 | 60.0/49.1
PointPillars $L^{FL}_{\text{Poly-1}}$ | $-(1-P_{t})^{2}\log(P_{t})-(1-P_{t})^{3}$ | 77.1/62.9 | 67.7/55.1 | 69.6/57.1 | 60.2/49.3
Improvement | - | +1.1/+0.9 | +0.5/+0.5 | +0.7/+0.5 | +0.2+0.2
RSN $L_{\text{FL}}$ | $-(1-P_{t})^{2}\log(P_{t})$ | 85.0/81.4 | 75.5/72.2 | 79.4/76.2 | 69.9/67.0
RSN $L^{FL}_{\text{Poly-1}^{*}}$ | $-(1-P_{t})^{2}\log(P_{t})-(1-P_{t})^{3}+0.2(1-P_{t})^{4}$ | 85.4/81.8 | 75.8/72.5 | 80.2/77.0 | 70.6/67.7
Improvement | - | +0.4/+0.4 | +0.3/+0.3 | +0.8/+0.8 | +0.7/+0.7

Table 7: PolyLoss improves detection results on Waymo Open Dataset validation
set. Two detection models: single-stage PointPillars (Lang et al., 2019) and
two-stage SOTA RSN (Sun et al., 2021) are evaluated. Bird’s eye view (BEV) and
3D detection average precision (AP) and average precision with heading (APH)
at Level 1 (L1) and Level 2 (L2) difficulties are reported. The IoU threshold
is set to 0.7 for vehicle detection and 0.5 for pedestrian detection.

PolyLoss improves single-stage PointPillars model. The PointPillars model
converts the raw 3D point cloud to a 2D top-down pseudo image, and then detect
3D bounding boxes from the 2D image in a similar way to RetinaNet (Lin et al.,
2017). Here, we replace the classification focal loss ($\gamma=2$) with
$L^{\text{FL}}_{\text{Poly-1}}=-(1-P_{t})^{2}\log
P_{t}+\epsilon_{1}(1-P_{t})^{3}$ and adopt the same training schedule
optimized for focal loss without any modification666Code at
https://github.com/tensorflow/lingvo/tree/master/lingvo/tasks/car. Table 7
shows that $L^{\text{FL}}_{\text{Poly-1}}$ with $\epsilon=-1$ leads to
significant improvement on all the metrics for both vehicle and pedestrian
models.

Figure 7: Visualizing $L^{FL}_{\text{Poly-1}}$ and
$L^{FL}_{\text{Poly-1}^{*}}$ in the PolyLoss framework.

Advancing the state-of-the-art with RSN. RSN segments foreground points from
the 3D point cloud in the first stage, and then applies sparse convolution to
predict 3D bounding boxes from the selected foreground points. RSN uses the
same focal loss as the PointPillars model, i.e.,
$L_{\text{FL}}=-(1-P_{t})^{2}\log P_{t}$. Since the optimal
$L^{\text{FL}}_{\text{Poly-1}}$ for PointPillars ($\epsilon_{1}=-1$) is
equivalent to dropping the first polynomial, we adapt the same loss
formulation for RSN and tune the new leading polynomial $(1-P_{t})^{4}$ by
defining
$L^{\text{FL}}_{\text{Poly-1}^{*}}=-(1-P_{t})^{2}\log(P_{t})-(1-P_{t})^{3}+\epsilon_{2}(1-P_{t})^{4}$,
shown in Figure 7. We follow the same training schedule optimized for focal
loss described in Sun et al. (2021) without adjustment. Our results, in Table
7, show that tuning the new leading polynomial improves all metrics (except
vehicle detection BEV APH L2) for the SOTA 3D detector.

##  6 Conclusion

In this paper, we propose the PolyLoss framework, which provides a unified
view on common loss functions for classification problems. We recognize that,
under polynomial expansion, focal loss is a horizontal shift of the polynomial
coefficients compared to the cross-entropy loss. This new insight motivates us
to explore an alternative dimension. i.e. vertically modify the polynomial
coefficients.

Our PolyLoss framework provides flexible ways of changing the loss function
shape by adjusting the polynomial coefficients. In this framework, we propose
a simple and effective Poly-1 formulation. By simply adjusting the coefficient
of the leading polynomial coefficient with just one extra hyperparameter
$\epsilon_{1}$, we show our simple Poly-1 improves a variety of models across
multiple tasks and datasets. We hope Poly-1 formulation’s simplicity (one
extra line of code) and effectiveness will lead to adoption in more
applications of classification than the ones we have managed to explore.

More importantly, our work highlights the limitation of common loss functions,
and simple modification could lead to improvements even on well established
state-of-the-art models. We hope these findings will encourage exploring and
rethinking the loss function design beyond the commonly used cross-entropy and
focal loss, as well as the simplest Poly-1 loss proposed in this work.

## Acknowledgements

We thank James Philbin, Doug Eck, Tsung-Yi Lin and the rest of Waymo Research
and Google Brain teams for valuable feedback.

## Reproducibility statement

Our experiments are based on public datasets and open source code
repositories, shown in footnote 3-6. We do not tune any default training
hyperparameters and only modify the loss functions, which are shown in Table
2-7. The proposed final formulation $L_{\text{Poly-1}}$ requires one line of
code change.
Example code for $L_{\text{Poly-1}}^{\text{CE}}$ with softmax activation is
shown below.

⬇

def poly1_cross_entropy(logits, labels, epsilon):

# epsilon >=-1.

# pt, CE, and Poly1 have shape [batch].

pt = tf.reduce_sum(labels * tf.nn.softmax(logits), axis=-1)

CE = tf.nn.softmax_cross_entropy_with_logits(labels, logits)

Poly1 = CE + epsilon * (1 - pt)

return Poly1

Example code for $L_{\text{Poly-1}}^{\text{CE}}$ with $\alpha$ label smoothing
is shown below.

⬇

def poly1_cross_entropy(logits, labels, epsilon, alpha = 0.1):

# epsilon >=-1.

# one_minus_pt, CE, and Poly1 have shape [batch].

num_classes = labels.get_shape().as_list()[-1]

smooth_labels = labels * (1-alpha) + alpha/num_classes

one_minus_pt = tf.reduce_sum(

smooth_labels * (1 - tf.nn.softmax(logits)), axis=-1)

CE_loss = tf.keras.losses.CategoricalCrossentropy(

from_logits=True, label_smoothing=alpha, reduction=’none’)

CE = CE_loss(labels, logits)

Poly1 = CE + epsilon * one_minus_pt

return Poly1

Example code for $L_{\text{Poly-1}}^{\text{FL}}$ with sigmoid activation is
shown below.

⬇

def poly1_focal_loss(logits, labels, epsilon, gamma=2.0):

# epsilon >=-1.

# p, pt, FL, and Poly1 have shape [batch, num of classes].

p = tf.math.sigmoid(logits)

pt = labels * p + (1 - labels) * (1 - p)

FL = focal_loss(pt, gamma)

Poly1 = FL + epsilon * tf.math.pow(1 - pt, gamma + 1)

return Poly1

Example code for $L_{\text{Poly-1}}^{\text{FL}}$ with $\alpha$ balance is
shown below.

⬇

def poly1_focal_loss(logits, labels, epsilon, gamma=2.0, alpha=0.25):

# epsilon >=-1.

# p, pt, FL, weight, and Poly1 have shape [batch, num of classes].

p = tf.math.sigmoid(logits)

pt = labels * p + (1 - labels) * (1 - p)

FL = focal_loss(pt, gamma, alpha)

weight = labels * alpha + (1 - labels) * (1 - alpha)

Poly1 = FL + epsilon * tf.math.pow(1 - pt, gamma + 1) * weight

return Poly1

## Supplementary material

##  7 Proof of theorem 1

######  Theorem 1.

For any small $\zeta>0$, $\delta>0$ if
$N>\log_{1-\delta}{(\zeta\cdot\delta)}$, then for any $p\in[\delta,1]$, we
have $|R_{N}(p)|<\zeta$ and $|R_{N}^{\prime}(p)|<\zeta$.

###### Proof.

$\displaystyle|R_{N}(p)|$
$\displaystyle=\sum_{j=N+1}^{\infty}1/j(1-p)^{j}\leq\sum_{j=N+1}^{\infty}(1-p)^{j}=\frac{(1-p)^{N+1}}{p}\leq\frac{(1-\delta)^{N+1}}{\delta}\leq\frac{(1-\delta)^{N}}{\delta}$
$\displaystyle|R_{N}^{\prime}(p)|$
$\displaystyle=\sum_{j=N}^{\infty}(1-p)^{j}=\frac{(1-p)^{N}}{p}\leq\frac{(1-\delta)^{N}}{\delta}$

##  8 Adjusting other training hyperparameters leads to higher gain.

All the experiments shown in the main text are based on hyperparameters
optimized for the baseline loss function, which actually puts PolyLoss at a
disadvantage. Here we use weight decay rate for ResNet50 as an example. The
default weight decay (1e-4) is optimized for cross-entropy loss. Adjusting the
decay rate may reduce the model performance of cross-entropy loss but leads to
much higher gain for PolyLoss (+0.8%), which is better than the best accuracy
(76.3%) trained using cross-entropy loss (+0.8%).

Weight decay | 1e-4† | 2e-4 | 9e-5
---|---|---|---
Cross-entropy | 76.3 | 76.3 | 76.1
PolyLoss | 76.7 | 77.1 | 76.7
Improv. @ the same weight decay | +0.4 | +0.8 | +0.6
Improv. compared to the best $L_{\text{CE}}$ (76.3%) | +0.4 | +0.8 | +0.4

Table 8: ResNet50 performances on ImageNet-1K using different weight decays.
†The default weight decay value is 1e-4.

Here, we add additional ablation studies on COCO detection using RetinaNet.
The optimal $\gamma$ and $\alpha$ balance values for Focal loss are (2.0,
0.25) (Lin et al., 2017). Since all the hyperparameters are optimized with
respect to the optimal ($\gamma$, $\alpha$) values, we observe no improvement
when tuning the leading polynomial term. We suspect the detection AP is at a
’local maximum’ of hyperparameters. By adjusting ($\gamma$, $\alpha$) values,
we show PolyLoss consistently outperforms the best Focal Loss AP (33.4), i.e.,
adjusting only $\gamma$ value (column 3, 4) or both $\gamma$ and $\alpha$
values (column 5, 6).

Focal loss $(\gamma,\alpha)$ |  (2.0, 0.25)† | (1.5, 0.25) | (2.5, 0.25) | (1.5, 0.3) | (2.5, 0.15)
---|---|---|---|---|---
Focal loss | 33.4 | 33.4 | 33.2 | 33.2 | 32.9
PolyLoss | 33.4 | 33.6 | 33.7 | 33.8 | 33.8
Improv. @ same $(\gamma,\alpha)$ | 0 | +0.2 | +0.5 | +0.6 | +0.9
Improv. compared to the best $L_{\text{FL}}$ (33.4) | 0 | +0.2 | +0.3 | +0.4 | +0.4

Table 9: RetinaNet (ResNet50 backbone) performances on COCO using different
Focal loss $(\gamma,\alpha)$. †The default $(\gamma,\alpha)$ used in Focal
loss is (2.0, 0.25).

##  9 $L_{\text{Drop}}$ with more hyperparameter tuning

For $L_{\text{Drop}}$ (N = 2), besides adjusting the learning rate, we further
tune the coefficient ($\alpha$) of the second polynomial, similar to a prior
work (Gonzalez & Miikkulainen, 2020b), and weight decay.

$\displaystyle L_{\text{Drop*}}=(1-P_{t})+\alpha(1-P_{t})^{2}$ (8)

Unlike Feng et al. (2020), where $\alpha=0.5$ after dropping all higher-order
polynomial, we find the optimal $\alpha=8$, while the optimal learning rate is
the same as the default setting (0.1). This alone increases the accuracy to
70.9, which shows simply dropping polynomial terms is not enough and adjusting
the polynomial coefficients is critical. Further tuning weight decay leads to
less than 0.1% model quality improvement.

Comparing to prior works (Gonzalez & Miikkulainen, 2020b; Feng et al., 2020),
Poly-1 is more effective and only contains one hyperparameter. Tuning weight
decay of Poly-1 further increases the accuracy while having less
hyperparameters compared to $L_{\text{Drop}^{*}}$, shown in Table 10.

| Cross-entropy | Poly-1 | Poly-1 (weight decay) | $L_{\text{Drop*}}$
---|---|---|---|---
Accuracy | 76.3 | 76.7 | 77.1 | 70.9
Num. of parameters | – | 1 | 2 | 3

Table 10: Poly-1 outperforms $L_{\text{Drop}^{*}}$ with hyperparameter tuning.
Accuracy of ResNet50 on ImageNet-1K is reported.

##  10 Collectively tuning multiple polynomial coefficients

Besides adjusting individual polynomial coefficients, in this section, we
explore collectively tuning multiple polynomial coefficients in the PolyLoss
framwork. In particular, we change the coefficients in the original cross-
entropy loss from $1/j$ (Equation 1) to exponential decay. Here, we define

$\displaystyle L_{\text{exp}}=\sum_{j=1}^{2N}e^{-(j-1)/N}(1-P_{t})^{j}$ (9)

where we cut off the infinite sum at twice the decay factor $N$. We performed
2D grid search on $N\in\\{5,20,80,320\\}$ and learning rate
$\in\\{0.1,0.4,1.6,6.4\\}$. The best accuracy is 72.3, where $N=80$ and
learning rate $=1.6$, shown in Table 11.

| Cross-entropy | Poly-1 | $L_{\text{exp}}$
---|---|---|---
Accuracy | 76.3 | 76.7 | 72.3
Num. of parameters | – | 1 | 2

Table 11: Comparing Poly-1 with exponential decay coefficients. Accuracy of
ResNet50 on ImageNet-1K is reported.

Though Poly-1 is better than using $L_{\text{exp}}$, there are a lot more
possibilities besides using exponential decay. We believe understanding how
collectively tuning multiple coefficients affects the training is an important
topic.

##  11 Comparing to other training techniques

As shown in recent works (He et al., 2019; Bello et al., 2021; Wightman et
al., 2021), though independent novel training techniques often lead to sub 1%
improvement, combining them could lead to significant overall improvements. To
put things into perspective, Poly-1 achieves similar improvements as other
commonly used training techniques, such as label smoothing and dropout on FC,
shown in Table 12.

| Cross-entropy | Poly-1 | Label smoothing | Dropout on FC
---|---|---|---|---
Accuracy | 76.3 | 76.7 | 76.7 | 76.4
Num. of parameters | – | 1 | 1 | 1

Table 12: Comparing Poly-1 with common training techniques. Accuracy of
ResNet50 on ImageNet-1K is reported.

##  12 Rediscovering focal loss from PolyLoss

Figure 8: Dropping leading polynomial terms can improve RetinaNet.

Focal loss was first developed for single-stage detector RetinaNet to address
strong class imbalance presented in object detection (Lin et al., 2017). Here,
we provide an additional ablation study on how to systemically discover focal
loss in the PolyLoss framework and investigate how the leading terms affect
training in the presence of class imbalance.

#### Rediscovering the concept of focal loss from cross-entropy loss.

Here, we take a step back and attempt to systematically rediscover the concept
of focal loss via our PolyLoss framework. Focal loss is commonly used for
training detection models. Coming up with such an insight to address the class
imbalance issue in detection requires strong domain expertise. We start with
the PolyLoss representation of cross-entropy loss and improve it from the
PolyLoss gradient perspective.

Figure 9: Dropping leading polynomials reduces overfitting to the majority
class. $P_{t}$ during RetinaNet training are plotted. Top: overall. Bottom
left: background. Bottom right: foreground object. Dark blue curves represents
$P_{t}$ for cross-entropy loss. Blue curves represents dropping the first
polynomial in the cross-entropy loss. Light blue curves represents dropping
both the first and second polynomials in the cross-entropy loss.

We start with the cross-entropy loss and define PolyLoss by dropping the first
$N$ polynomials in cross-entropy loss, i.e. $L_{\text{Drop-
front}}=\sum_{j=N+1}^{\infty}1/j(1-P_{t})^{j}=L_{\text{CE}}-\sum_{j=1}^{N}1/j(1-P_{t})^{j}$.
Dropping the first two polynomial terms $(1-P_{t})$ significantly improves
both the detection AP and AR, see Figure 8. Dropping the first two polynomials
($N=2$) leads to the best RetinaNet performance, which is similar to setting
$\gamma=2$ in focal loss, i.e. focal loss $\gamma=2$ pushes all the polynomial
coefficients to the right by 2, shown in Figure 1 right, which is similar to
truncating the first two polynomial terms.

#### Leading polynomials cause overfitting to the majority class.

In the PolyLoss framework, the leading polynomial of cross-entropy loss is a
constant, shown in Equation 3. For binary classification, the leading gradient
for each class is simply $N_{background}-N_{object}$, where $N_{background}$
and $N_{object}$ are the counts of background and object instances in the
training mini-batch. When the class counts are extremely imbalanced, the
majority class will dominate the gradient which will lead to significant bias
towards optimizing the majority class.

Dropping polynomials reduces the extremely confident prediction $P_{t}$, see
Figure 9. To examine the composition of the overall prediction confidence, we
also plot the $P_{t}$ for background only and $P_{t}$ for object only. Due to
the extreme imbalance between the background and the object class, the overall
$P_{t}$ is dominated by the background only $P_{t}$. So reducing the overall
$P_{t}$ decreases the background $P_{t}$. On the other hand, reducing
overfitting to the majority background class leads to more confident
prediction $P_{t}$ on the object class.

## References

  * Bello et al. (2021) Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, and Barret Zoph.  Revisiting resnets: Improved training and scaling strategies.  _arXiv preprint arXiv:2103.07579_ , 2021.
  * Bulo et al. (2017) Samuel Rota Bulo, Gerhard Neuhold, and Peter Kontschieder.  Loss max-pooling for semantic image segmentation.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 7082–7091. IEEE, 2017.
  * Cui et al. (2019) Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9268–9277, 2019.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Du et al. (2020) Xianzhi Du, Tsung-Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoc V Le, and Xiaodan Song.  Spinenet: Learning scale-permuted backbone for recognition and localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11592–11601, 2020.
  * Felzenszwalb et al. (2010) Pedro F Felzenszwalb, Ross B Girshick, and David McAllester.  Cascade object detection with deformable part models.  In _2010 IEEE Computer society conference on computer vision and pattern recognition_ , pp. 2241–2248. IEEE, 2010.
  * Feng et al. (2020) Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An.  Can cross entropy loss be robust to label noise.  In _Proceedings of the 29th International Joint Conferences on Artificial Intelligence_ , pp. 2206–2212, 2020.
  * Ghosh et al. (2015) Aritra Ghosh, Naresh Manwani, and PS Sastry.  Making risk minimization tolerant to label noise.  _Neurocomputing_ , 160:93–107, 2015.
  * Ghosh et al. (2017) Aritra Ghosh, Himanshu Kumar, and PS Sastry.  Robust loss functions under label noise for deep neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Gonzalez & Miikkulainen (2020a) Santiago Gonzalez and Risto Miikkulainen.  Improved training speed, accuracy, and data utilization through loss function optimization.  In _2020 IEEE Congress on Evolutionary Computation (CEC)_ , pp. 1–8. IEEE, 2020a.
  * Gonzalez & Miikkulainen (2020b) Santiago Gonzalez and Risto Miikkulainen.  Optimizing loss functions through multivariate taylor polynomial parameterization.  _arXiv preprint arXiv:2002.00059_ , 2020b.
  * Hajiabadi et al. (2017) Hamideh Hajiabadi, Diego Molla-Aliod, and Reza Monsefi.  On extending neural networks with loss ensembles for text classification.  _arXiv preprint arXiv:1711.05170_ , 2017.
  * Hansen & Ostermeier (1996) Nikolaus Hansen and Andreas Ostermeier.  Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation.  In _Proceedings of IEEE international conference on evolutionary computation_ , pp. 312–317. IEEE, 1996.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2961–2969, 2017.
  * He et al. (2019) Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li.  Bag of tricks for image classification with convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 558–567, 2019.
  * Lang et al. (2019) Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12697–12705, 2019.
  * Law & Deng (2018) Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 734–750, 2018.
  * Li et al. (2019) Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang.  Am-lfs: Automl for loss function search.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8410–8419, 2019.
  * Li et al. (2020) Hao Li, Chenxin Tao, Xizhou Zhu, Xiaogang Wang, Gao Huang, and Jifeng Dai.  Auto seg-loss: Searching metric surrogates for semantic segmentation.  _arXiv preprint arXiv:2010.07930_ , 2020.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pp. 740–755. Springer, 2014.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2980–2988, 2017.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In _European conference on computer vision_ , pp. 21–37. Springer, 2016.
  * Menon et al. (2019) Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar.  Can gradient clipping mitigate label noise?  In _International Conference on Learning Representations_ , 2019.
  * Oksuz et al. (2020) Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas.  Imbalance problems in object detection: A review.  _IEEE transactions on pattern analysis and machine intelligence_ , 2020.
  * Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton.  Regularizing neural networks by penalizing confident output distributions.  _arXiv preprint arXiv:1701.06548_ , 2017.
  * Shi et al. (2020) Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10529–10538, 2020.
  * Shrivastava et al. (2016) Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 761–769, 2016.
  * Sun et al. (2020) Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2446–2454, 2020.
  * Sun et al. (2021) Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Christian Sminchisescu, and Dragomir Anguelov.  Rsn: Range sparse net for efficient, accurate lidar 3d object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Sung (1996) Kah-Kay Sung.  Learning and example selection for object and pattern detection.  1996\.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2818–2826, 2016.
  * Tan & Le (2021) Mingxing Tan and Quoc V Le.  Efficientnetv2: Smaller models and faster training.  In _International Conference on Machine Learning_ , 2021.
  * Tan et al. (2020) Mingxing Tan, Ruoming Pang, and Quoc V Le.  Efficientdet: Scalable and efficient object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10781–10790, 2020.
  * Tao et al. (2020) Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  _arXiv preprint arXiv:2005.10821_ , 2020.
  * Viola & Jones (2001) Paul Viola and Michael Jones.  Rapid object detection using a boosted cascade of simple features.  In _Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001_ , volume 1, pp. I–I. IEEE, 2001.
  * Wang et al. (2019) Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 322–330, 2019.
  * Wightman et al. (2021) Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  _arXiv preprint arXiv:2110.00476_ , 2021.
  * Xu et al. (2018) Haowen Xu, Hao Zhang, Zhiting Hu, Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing.  Autoloss: Learning discrete schedules for alternate optimization.  _arXiv preprint arXiv:1810.02442_ , 2018.
  * Zhang & Sabuncu (2018) Zhilu Zhang and Mert R Sabuncu.  Generalized cross entropy loss for training deep neural networks with noisy labels.  _arXiv preprint arXiv:1805.07836_ , 2018.
  * Zhao et al. (2021) Guangxiang Zhao, Wenkai Yang, Xuancheng Ren, Lei Li, and Xu Sun.  Well-classified examples are underestimated in classification with deep neural networks.  _arXiv preprint arXiv:2110.06537_ , 2021.
  * Zoph et al. (2020) Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le.  Rethinking pre-training and self-training.  _arXiv preprint arXiv:2006.06882_ , 2020.