# Binary differential equations associated to congruences of lines in
Euclidean 3-space

J. W. Bruce and F. Tari

###### Abstract

We study quotients of quadratic forms and associated polar lines in the
projective plane. Our results, applied pointwise to quadratic differential
forms, shed some light on classical binary differential equations (BDEs)
associated to congruences of lines in Euclidean 3-space and allows us to
introduce a new one. The new BDE yields a new singular surface in the
Euclidean 3-space associated to a congruence of lines. We determine the
generic local configurations of the above BDEs on congruences.

0002010 Mathematics Subject classification: 53A05, 34A09, 000Key Words and
Phrases. Congruences of lines, binary differential equations, quadratic forms,
polarity in the projective plane, singularities.

##  1 Introduction

Let $\mathbb{L}$ denote the space of lines in real Euclidean 3-space; the
geometry of submanifolds of $\mathbb{L}$ is well studied, with contributions
from such luminaries as Hamilton, Dupin, Study and Blaschke. The set
$\mathbb{L}$ can be modelled by the tangent bundle to the unit sphere
$T\mathbb{S}^{2}\subset\mathbb{S}^{2}\times\mathbb{R}^{3}$, with a point
$(n,x)$ corresponding to the line $x+tn$, although this does depend on an
arbitrary choice of origin. In this paper we consider generic surfaces in
$\mathbb{L}$, known classically as congruences, and some associated BDEs. We
classify the singularities of these BDEs, relating them to other aspects of
the geometry.

We shall suppose that the congruence $Z\subset\mathbb{L}$ is a smooth regular
surface. Classically, congruences were specified by a smooth immersion

$(n,x):U\to\mathbb{S}^{2}\times\mathbb{R}^{3}$

with a point $(n(u,v),x(u,v))$ determining the line $x(u,v)+tn(u,v)$; $x(U)$
is called the director surface or directrix. The directrix is clearly not
unique: if $f:U\to\mathbb{R}$ is any smooth function the map $(n,x+fn)$
represents the same family of lines. In what follows we will only be
interested in local properties of congruences so we are considering germs of
mappings $(n,x):\mathbb{R}^{2},0\to\mathbb{S}^{2}\times\mathbb{R}^{3}$, though
sometimes we specify a domain, that is an open set $U\subset\mathbb{R}^{2}$.

There are a range of ways to study the geometry of congruences, but because we
need to carry out explicit calculations we prefer the Gaussian approach,
employed for example in [12], using quadratic forms, that is families of
quadratic differential forms $Q:T_{z}Z\to\mathbb{R}$ which depends smoothly on
$z\in Z$. Clearly a curve on $Z$ through a point $z$ yields the germ of a
ruled surface in $\mathbb{R}^{3}$. Some properties of these ruled surfaces
depend only on the direction of the tangent to the curve and classically these
were used, as below, to pick up distinct directions in $T_{z}Z$ and to
associate to $Z$ some generally singular surfaces in $\mathbb{R}^{3}$. For
example consider a curve $\gamma=(n,x):I\to Z\subset\mathbb{L}$ with $0\in I$
an open interval and $\gamma(0)=(n(0),x(0))=L$. We have an associated ruled
surface parametrised by $(t,s)\mapsto x(t)+sn(t)\subset\mathbb{R}^{3}$.

On the line $L$ there is a central point, in antiquated terminology the foot
of the common perpendicular of consecutive lines. These central points sweep
out the striction curve of the ruled surface. We shall denote by
$r=-(x^{\prime}\cdot n^{\prime}/n^{\prime}\cdot n^{\prime})(0)$ the signed
distance from $x(0)$ to the central point in the $n(0)$ direction. Clearly the
central point only depends on the tangent to the curve at $z$.

There is also a classical invariant
$\lambda=[x^{\prime},n,n^{\prime}]/n^{\prime}\cdot n^{\prime}$, called the
distribution parameter or pitch of the ruled surface, where $[-,-,-]$ denote
the usual scalar triple product in $\mathbb{R}^{3}$. Clearly this only depends
on the point $\gamma(0)$ and $\gamma^{\prime}(0)$. When this is identically
zero we have a developable surface, generally the set of tangent lines to a
smooth space curve $C$, and the line of striction here is the edge of
regression, that is $C$ which does not depend on the choice of directrix. We
have the following concepts (see for example, [12]).

  1. 1.

The directions in $T_{z}Z$ along which the values of $r$ are extreme are
called principal directions; as we shall see generally there are two such
directions. The corresponding central points on $L\in\mathbb{R}^{3}$ are
called boundary points.

  2. 2.

A point $z$ is called an umbilic point if every direction in $T_{z}Z$ is a
principal direction. The integral curves on $Z$ of the principal directions
are the principal curves and the surfaces swept out by the boundary points in
$\mathbb{R}^{3}$ are called the boundary surfaces.

  3. 3.

The surface in $\mathbb{R}^{3}$ swept out by the midpoints of the boundary
points is called the middle surface.

  4. 4.

There are $0,1$ or $2$ directions in $T_{z}Z$, called torsal directions, for
which the pitch $\lambda$ of any associated ruled surface vanishes
(infinitesimally we have a developable surface in $\mathbb{R}^{3}$). The
associated central points on the ray $L\subset\mathbb{R}^{3}$ are called foci.
The point $z\in Z$ is called a parabolic point when the two torsal directions
coincide. The surface in $\mathbb{R}^{3}$ swept out by the foci is called a
focal surface. At any point the midpoint of the two foci (which is always real
even if the two focal points are imaginary, see [6]) is also the midpoint of
the boundary points.

  5. 5.

For each $z\in Z$ there are at most two directions in $T_{z}Z$, called mean
directions, along which the values of the pitch is extreme. Their associated
points on $L$ are precisely the midpoints of the focii (or boundary) points.

The principal, torsal and mean directions are solutions of BDEs. These are
equations of the form

${\bf a}(u,v)dv^{2}+2{\bf b}(u,v)dvdu+{\bf c}(u,v)du^{2}=0,$ (1)

where ${\bf a},{\bf b},{\bf c}$ are smooth functions on some open set
$U\subset\mathbb{R}^{2}$. Clearly any quadratic differential form $\omega$ on
$Z$ yields a BDE, given by $\omega=0$.

We establish the generic local configurations of the solutions of the above
BDEs, and a new BDE associated to $Z$ called the characteristic BDE. The new
BDE yields a new singular surface in the Euclidean 3-space associated to $Z$
(§4.3). In §4 we see how the above BDEs are related using the approach in [5],
where the elementary geometry of binary quadratic forms casts light on the
geometry of BDEs (see Figure 3). The relations between the above BDEs can be
deduced from a more general result on quotient of quadratic forms given in §3.

##  2 Preliminaries

We denote the BDE (1) by $\omega=({\bf a},2{\bf b},{\bf c})$ and refer to
${\bf a},{\bf b},{\bf c}$ as the coefficients of the BDE. Such an equation
determines a pair of distinct directions at each point in the region where
$\delta={\bf b}^{2}-{\bf ac}>0$ and no direction at points where $\delta<0$.
The set $\delta=0$ is the discriminant of the BDE. If the coefficients of the
BDE do not all vanish at a point $(u,v)$ on the discriminant, then it
determines a unique (double) direction there; if they do, then all directions
at $(u,v)$ are considered solutions.

The solution curves of a BDE form a pair of transverse foliations in the
region $\delta>0$. These foliations together with the discriminant curve
constitute the so-called configuration of the BDE. Two BDEs are said to be
topologically equivalent if there is a local homeomorphism in the plane which
maps the configuration of one equation to that of the other.

Let $q$ be a regular point on the discriminant curve. If the unique solution
of $\omega$ at $q$ is transverse to the discriminant curve, then the local
configuration of the BDE is a family of cusps (see [8] for references and
Figure 1). When the direction is tangent to the discriminant curve, there are
three generic (i.e., stable) local topological models: folded saddle, folded
node and folded focus ([8], Figure 1). If we write $p=dv/du$ and the 2-jet of
the BDE in the form

$a_{0}p^{2}+(b_{0}+b_{1}u+b_{2}v)p+c_{1}u+c_{2}v+c_{3}u^{2}+c_{4}uv+c_{5}v^{2},$

then according to Lemma 2.1 in [4], the origin is a fold singularity if
$a_{0}\neq 0$ and $b_{0}=c_{1}=0$ (when $b_{0}=0$ and $a_{0}c_{1}\neq 0$ we
get a family of cusps). At a folded singularity, setting
$\lambda=(4a_{0}c_{3}-b_{1}^{2}-b_{1}c_{2})/(4c_{2}^{2})$, the singularity is
of type folded saddle if $\lambda<0$, folded node if $0<\lambda<1/16$ and
folded focus if $\lambda>1/16$.

Figure 1: Stable configurations of a BDE at points on its discriminant.

At a point where all the coefficients of the BDE vanish, the discriminant is
singular. Our interest here is when the singularity is of type $A_{1}^{+}$
(i.e., $\delta\circ h^{-1}(u,v)=\pm(u^{2}+v^{2})$ for some germ of a
diffeomorphism $h$). Then the BDE has three generic possible configurations as
shown in Figure 2, classified in ([1, 2, 9]), called the star, monstar or
lemon (here, by generic we mean within the set of BDEs with coefficients
vanishing at the origin and whose discriminants have an
$A_{1}^{+}$-singularity). If we write the 1-jet of the coefficients of the BDE
at the origin in the form
$j^{1}\omega=(a_{1}u+a_{2}v,2b_{1}u+2b_{2}v,c_{1}u+c_{2}v)$, then according to
[2], at an $A_{1}^{+}$-singularity of the discriminant we get a star or
monstar (resp. lemon) if the cubic
$\phi(p)=a_{2}p^{3}+(2b_{2}+a_{1})p^{2}+(2b_{1}+c_{2})p+c_{1}$ has three
distinct roots (resp. one root). The cases star and monstar are distinguished
by the signs of $\alpha(p_{i})\phi^{\prime}(p_{i})$ at the roots of $\phi$,
where $\alpha(p)=a_{2}p^{2}+(b_{2}+a_{1})p+b_{1}$. If all of them are positive
we get a star, otherwise we get a monstar. The star, monstar and lemon
singularities of BDEs are not stable within the set of all BDEs ([3]).
However, those that appear here associated to congruences are stable in that
context.

Figure 2: Generic configurations of BDEs of Morse Type $A_{1}^{+}$: lemon
(left), star (centre), monstar (right).

We parametrise locally a congruence by
$(n,x):U\to\mathbb{S}^{2}\times\mathbb{R}^{3}$, where $U$ is an open set in
$\mathbb{R}^{2}$ and identify germs of congruences with germs of mappings
$C^{\infty}(U,\mathbb{R}^{5})$. We endow the set
$C^{\infty}(U,\mathbb{R}^{5})$ with the Whitney $C^{\infty}$-topology and say
that a property of congruences is generic if it is satisfied in an open and
dense subset of $C^{\infty}(U,\mathbb{R}^{5})$. To prove that a property is
generic we consider the map $\Phi:U\mapsto J^{k}(2,5)$ given
$\Phi(u,v)=j^{k}_{(u,v)}(n,x)$. A given property is usually represented by a
real algebraic variety $V$ in $J^{k}(2,5)$ with a smooth stratification.
Thom’s transversality theorem asserts that for generic congruences, $\Phi$ is
transverse to the stratification of $V$. This means, in particular, that a
property defined by more than three independent conditions, that is the
codimension of $V>3$, is not generic.

##  3 Quotients of quadratic forms

Some key properties of the special directions on congruences of lines and
their associated BDEs given in the introduction can be derived from a more
general result below on quotients of quadratic forms.

###### Theorem 3.1

(1) Let $q_{i}=a_{i}\alpha^{2}+2b_{i}\alpha\beta+c_{i}\beta^{2},i=1,2,$ be two
quadratic forms and consider the quotient

$f(\alpha,\beta)=\frac{q_{2}(\alpha,\beta)}{q_{1}(\alpha,\beta)},$

defined off the set $q_{1}(\alpha,\beta)=0$. There are two local (global if
$q_{1}$ is positive definite) extremal sets of points of the function $f$
which lie along the rays $(t\alpha,t\beta)$ where $(\alpha,\beta)$ are
solutions of the quadratic equation

$\left|\begin{array}[]{ccc}\beta^{2}&-\alpha\beta&\alpha^{2}\\\
a_{1}&b_{1}&c_{1}\\\ a_{2}&b_{2}&c_{2}\end{array}\right|=0.$

This expression is classically known as the Jacobian of $q_{1},q_{2}$, written
$Jac(q_{1},q_{2})$ that is the determinant of the $2\times 2$ matrix of
partial derivatives of $q_{1},q_{2}$. These directions coincide if and only if
$q_{1}$ and $q_{2}$ have a common root.

(2) Representing $q_{i}$ by symmetric matrices
$E_{i}=\left(\begin{array}[]{cc}a_{i}&b_{i}\\\ b_{i}&c_{i}\end{array}\right)$
and writing $w=(\alpha,\beta)^{T}$, the directions in $(1)$ are the solutions
of $E_{2}w=\mu E_{1}w$. The corresponding values of $\mu$, denoted
$\mu_{1},\mu_{2}$, are real if $E_{1}$ is positive or negative definite.

(3) The directions $(\alpha_{1},\beta_{1}),(\alpha_{2},\beta_{2})$
corresponding to $\mu_{1}$, $\mu_{2}$ are orthogonal with respect to $q_{2}$.
With respect to the same form they bisect the directions given by $q_{1}=0$
when these are real; this only occurs when $q_{2}$ is positive definite.

  * Proof

(1) Taking an affine chart the extremal points of $f(\alpha,1)$ are the
solutions of

$(a_{1}\alpha+b_{1})(a_{2}\alpha^{2}+2b_{2}\alpha+c_{2})-(a_{2}\alpha+b_{2})(a_{1}\alpha^{2}+2b_{1}\alpha+c_{1})=0,$

equivalently,

$(a_{1}b_{2}-b_{1}a_{2})\alpha^{2}+(a_{1}c_{2}-c_{1}a_{2})\alpha+(b_{1}c_{2}-c_{1}b_{2})=0,$

which can be written in the determinantal form as stated. It is easy to see
that the solutions coincide if and only if $q_{1},q_{2}$ have a common root.

(2) It is easy to check that if $E_{2}w=\mu E_{1}w$ then $(\alpha,\beta)$ is a
‘root’of the quadratic form $Jac(q_{1},q_{2})=0$ and conversely. The second
part follows from the usual argument: if $E_{2}w=\mu E_{1}w$, then taking
complex conjugates $E_{2}{\bar{w}}={\bar{\mu}}E_{1}{\bar{w}}$ so
$wE_{2}{\bar{w}}={\bar{\mu}}wE_{1}{\bar{w}}$ and
${\bar{w}}E_{2}w=\mu{\bar{w}}E_{1}w$. So
$(\mu-{\bar{\mu}})({\bar{w}}E_{1}w)=0$ and the second factor is zero if and
only if $w=0$.

(3) If $q_{2}$ has rank $2$ we can assume it is $\pm(\alpha^{2}+\beta^{2}$) or
$\alpha\beta$. In the first case we know we can reduce
$q_{1}=a\alpha^{2}+c\beta^{2}$, and $Jac(q_{1},q_{2})=4(a-c)\alpha\beta$ the
results easily follow. In the second case we can clearly suppose that
$q_{1}=a\alpha^{2}+c\beta^{2}$. Now
$Jac(q_{1},q_{2})=2(a\alpha^{2}-c\beta^{2})$ and if $ac\geq 0$ clearly the two
directions given by $Jac(q_{1},q_{2})=0$ are orthogonal with respect to
$q_{2}$. Note that $q_{1}=0$ then has no roots. $\Box$

There is a classical (elementary) geometry of quadratic forms used in the
study of BDEs in [5]. A non-zero quadratic form
$a\alpha^{2}+2b\alpha\beta+c\beta^{2}$ can be represented by the point
$q=(a:2b:c)$ in the projective plane $\mathbb{R}\mathbb{P}^{2}$. In
$\mathbb{R}\mathbb{P}^{2}$ the set of singular quadratic forms is the conic
$\Gamma=\\{q:b^{2}-ac=0\\}$. The polar line $\widehat{q}$ of a point $q$ with
respect to $\Gamma$ is the line that contains all points $p$ such that $q$ and
$p$ are harmonic conjugate points with respect to the intersection points
$R_{1}$ and $R_{2}$ of the conic $\Gamma$ and a variable line through $q$. If
the polar line $\widehat{q}$ meets $\Gamma$, then the tangents to $\Gamma$ at
the points of intersection meet at $q$. A point $(a_{1}:2b_{1}:c_{1})$ is on
the polar line of a point $p=(a:2b:c)$ if and only if
$2bb_{1}-ac_{1}-a_{1}c=0$. Three points in the projective plane are said to
form a self-polar triangle if the polar line of any vertex of the triangle is
the line through the remaining two points. In our case the points represent
quadratic forms, so any vertex of the self-polar triangle is the Jacobian of
the remaining two vertices.

##  4 BDEs on congruences

A quadratic differential form $\omega(du,dv)={\bf a}(u,v)dv^{2}+2{\bf
b}(u,v)dvdu+{\bf c}(u,v)du^{2}$ on a congruence $Z$ determines a BDE
$\omega=0$ on $Z$. So the associated BDEs of two quadratic differential forms
$\omega_{1},\omega_{2}$ yields a Jacobian BDE $Jac(\omega_{1},\omega_{2})=0$;
the formula gives a quadratic form at every point $(u,v)$. This yields, at
each point, the directions in which the quotient $\omega_{2}/\omega_{1}$ has
extrema. Note that Theorem 3.1 shows that $Jac(\omega_{1},\omega_{2})=0$ has a
single solution at points where the resultant of $\omega_{1},\omega_{2}$
vanishes. Moreover the directions determined by $Jac(\omega_{1},\omega_{2})=0$
are orthogonal with respect to $\omega_{1}$ or $\omega_{2}$ and bisect the
directions $\omega_{1}=0$ (resp. $\omega_{2}=0)$ with respect to the possibly
indefinite metric $\omega_{2}$ (resp. $\omega_{1}$). So for example if we
consider a smooth surface in $\mathbb{R}^{3}$ with first fundamental form
$\omega_{1}$ and second fundamental form $\omega_{2}$ then the Jacobian BDE
$Jac(\omega_{1},\omega_{2})=0$ determines the principle directions, which from
above are orthogonal, with the asymptotic directions at hyperbolic points
bisecting those directions.

For a congruence $Z\subset\mathbb{L}$ there are three key quadratic forms
$Q_{1},Q_{3},Q$ on the tangent spaces $T_{z}Z$, with $Q$ only defined up to a
multiple of $Q_{1}$, from which two other (well-defined) quadratic forms can
be constructed (see [12]).

###### Definition 4.1

(1) Define $Q_{1},Q,Q_{3}:T_{z}Z\to\mathbb{R}$ by
$Q_{1}(du,dv)=||n_{u}du+n_{v}dv||^{2}$,
$Q(du,dv)=(x_{u}du+x_{v}dv)\cdot(n_{u}du+n_{v}dv)$,
$Q_{3}(du,dv)=[x_{u}du+x_{v}dv,n_{u}du+n_{v}dv,n]$, so that

$\begin{array}[]{ccl}Q_{1}(du,dv)&=&n_{v}\cdot n_{v}dv^{2}+2n_{u}\cdot
n_{v}dudv+n_{u}\cdot n_{u}du^{2},\\\ Q(du,dv)&=&n_{v}\cdot
x_{v}dv^{2}+(n_{u}\cdot x_{v}+n_{v}\cdot x_{u})dudv+n_{u}\cdot x_{u}du^{2},\\\
Q_{3}(du,dv)&=&[x_{v},n_{v},n]dv^{2}+([x_{u},n_{v},n]+[x_{v},n_{u},n])dudv+[x_{u},n_{u},n]du^{2}.\end{array}$

We write

$\begin{array}[]{c}A=n_{u}\cdot n_{u},B=n_{u}\cdot n_{v},C=n_{v}\cdot
n_{v},\\\ a=-n_{u}\cdot x_{u},b_{1}=-n_{u}\cdot x_{v},b_{2}=-n_{v}\cdot
x_{u},c=-n_{v}\cdot x_{v},\\\
b=-\frac{1}{2}(b_{1}+b_{2}),{\bar{b}}=-\frac{1}{2}(b_{1}-b_{2}).\end{array}$

(2) Define $Q_{2},Q_{4}:T_{z}Z\to\mathbb{R}$ by $Q_{2}=Jac(Q,Q_{1})$ and
$Q_{4}=Jac(Q_{3},Q_{1})$.

At each point $z\in Z$, we shall write $Q_{i}$ for $Q_{i}(du,dv)$.

###### Proposition 4.2

(1) $Q_{1},Q_{2},Q_{3},Q_{4}$ are all well defined.

(2) The quadratic form $Q_{2}$ is given by

$\begin{array}[]{rcl}Q_{2}&=&\left|\begin{array}[]{ccc}dv^{2}&-dudv&du^{2}\\\
A&B&C\\\ a&b&c\end{array}\right|\\\ &&\\\ &=&(Bc-Cb)dv^{2}+(Ac-Ca)dudv+(Ab-
Ba)du^{2},\end{array}$

and the quadratic form $Q_{3}$ is a non-zero multiple of

$\begin{array}[]{rcl}Q_{3}&=&(Bc-Cb-C{\bar{b}})dv^{2}+(Ac-
Ca-2B{\bar{b}})dudv+(Ab-Ba-A{\bar{b}})du^{2}\\\
&=&Q_{2}-{\bar{b}}Q_{1}.\end{array}$

(3) The quadratic form $Q_{4}=Jac(Q_{3},Q_{1})=Jac(Q_{2},Q_{1})$ is given by

$\begin{array}[]{rcl}Q_{4}&=&\left|\begin{array}[]{ccc}dv^{2}&-dudv&du^{2}\\\
A&B&C\\\ Ab-Ba&\frac{1}{2}(Ac-Ca)&Bc-Cb\end{array}\right|\end{array},$

alternatively,

$\begin{array}[]{rcl}Q_{4}&=&(2B^{2}c-2BCb-
ACc+C^{2}a)dv^{2}+2(ABc-2ACb+BCa)dudv+\\\
&&(2B^{2}a-2ABb+A^{2}c-ACa)du^{2}.\end{array}$

  * Proof

(1) Recall that we may replace $x:U\to\mathbb{R}^{3}$ by $x+fn$ for any
function $f:U\to\mathbb{R}$. This replaces $Q$ by $Q+fQ_{1}$, and clearly
$Jac(q_{2}+cq_{1},q_{1})=Jac(q_{2},q_{1})$ for any quadratic forms
$q_{1},q_{2}$ and constant $c$. Similarly $Q_{3}$ is replaced by

$[\alpha(x_{u}+f_{u}n+fn_{u})+\beta(x_{v}+f_{v}n+fn_{v}),\alpha n_{u}+\beta
n_{v}.n]$

which clearly yields the same value.

(2) The expression of $Q_{2}$ follows from the fact that it is $Jac(Q,Q_{1})$.
For $Q_{3}$, we first observe that all coefficients of $Q_{3}$ vanish at
points where $n$ is singular. Indeed, for a generic congruence, the singular
set of $n$ is a regular curve and along it $n_{v}=\alpha n_{u}$ (or
$n_{v}=\alpha n_{u}$) for some scalar function $\alpha$. Differentiating
$[x_{u},n,n]\equiv 0$ and $[x_{v},n,n]\equiv 0$ along the curve proves the
claim.

Suppose now that $n$ is not singular. Then $n=n_{u}\wedge n_{v}/||n_{u}\wedge
n_{v}||$. We are interested in the solutions of $Q_{3}=0$, so do not
distinguish differential forms that are non-zero multiples of each other and
multiply $Q_{3}$ by $||n_{u}\wedge n_{v}||$. Then

$\begin{array}[]{rcl}[x_{u},n_{u},n_{u}\wedge
n_{v}]&=&x_{u}.(n_{u}\wedge(n_{u}\wedge
n_{v}))=x_{u}.((n_{u}.n_{v})n_{u}-(n_{u}.n_{u}).n_{v}))\\\ &=&Ba-
Ax_{u}.n_{v}=Ba-Ab+A\overline{b}\\\ \end{array}$

Similarly,

$\begin{array}[]{rcl}[x_{u},n_{v},n_{u}\wedge
n_{v}]&=&x_{u}.(n_{v}\wedge(n_{u}\wedge
n_{v}))=x_{u}.((n_{v}.n_{v})n_{u}-(n_{v}.n_{u}).n_{v}))\\\ &=&Ca-
Bb+B\overline{b},\\\ \end{array}$
$\begin{array}[]{rcl}[x_{v},n_{u},n_{u}\wedge
n_{v}]&=&x_{v}.(n_{u}\wedge(n_{u}\wedge
n_{v}))=x_{v}.((n_{u}.n_{v})n_{u}-(n_{u}.n_{u}).n_{v}))\\\
&=&Bb+B\overline{b}-Ac,\\\ \end{array}$

and

$\begin{array}[]{rcl}[x_{v},n_{v},n_{u}\wedge
n_{v}]&=&x_{v}.(n_{v}\wedge(n_{u}\wedge
n_{v}))=x_{v}.((n_{v}.n_{v})n_{u}-(n_{v}.n_{u}).n_{v}))\\\
&=&Cb+C\overline{b}-Bc.\\\ \end{array}$

It follows that $Q_{3}$ is a non-zero multiple of

$(Bc-bC-C{\bar{b}})dv^{2}+(Ac-Ca-2B{\bar{b}})dudv+(Ab-Ba-A{\bar{b}})du^{2},$

and we shall take this as our $Q_{3}$ from now on, so that
$Q_{3}=Q_{2}-\overline{b}Q_{1}.$

Part (3) is a straightforward calculation. $\Box$

###### Remark 4.3

Interpreting, as previously, the forms (at each point of $Z$) as elements of
the real projective plane $\mathbb{R}\mathbb{P}^{2}$, note that although $Q$
is not well defined the line joining $Q$ to $Q_{1}$ is, and is the polar line
$\widehat{Q_{2}}$ of $Q_{2}=Jac(Q,Q_{1})$. Then
$Q_{3}=Q_{2}-\overline{b}Q_{1}$ is on the line joining $Q_{1}$ and $Q_{2}$
which is the polar line $\widehat{Q_{4}}$ of
$Q_{4}=Jac(Q_{3},Q_{1})=Jac(Q_{2},Q_{1})$. It is not hard to see that this
lies on the line joining $Q$ to $Q_{1}$, so we have a configuration like that
in Figure 3. We also get a new quadratic form $Q_{5}=Jac(Q_{3},Q_{4})$ which
we deal with in more details in §4.3. We prove in Theorem 4.8 (resp. Theorem
4.9) that $Q_{1},Q_{2},Q_{4}$ (resp. $Q_{3},Q_{4},Q_{4}$) are vertices of a
self-polar triangle.

Figure 3: The configuration of the quadratic forms $Q_{i}$ and their polar
lines.

As in the introduction a curve $\gamma(s)=(n,x)(u(s),v(s))$ through a point
$z=\gamma(0)$ on $Z$ represents a ruled surface in $\mathbb{R}^{3}$. Along any
generator of a ruled surface the central point on $L$ is the directed distance

$r=-\frac{Q(\gamma(0))(u^{\prime}(0),v^{\prime}(0))}{Q_{1}(\gamma(0))(u^{\prime}(0),v^{\prime}(0))}$

from $x(0)$, which evidently only depends on the derivative
$\gamma^{\prime}(0)$ and $\gamma(0)$. As the tangent direction rotates in
$T_{z}Z$ the central point moves up and down $L$. These ruled surfaces also
have a parameter of distribution or in the terminology of Blaschke a pitch
which is constant along a generator and again only depends on
$\gamma^{\prime}(0)$. The pitch of the ruled surface $(n,x)(u(s),v(s))$ at
$s=0$ is

$\lambda=\frac{Q_{3}(\gamma(0))(u^{\prime}(0),v^{\prime}(0))}{Q_{1}(\gamma(0))(u^{\prime}(0),v^{\prime}(0))}.$

We can now describe the geometry behind the quadratic forms above using
Theorem 3.1.

###### Proposition 4.4

(1) The principal directions: the directions in which the distance from the
director surface to the central point has extreme value are given by the BDE
$Jac(Q,Q_{1})=Q_{2}=0$.

(2) The torsal directions: the directions in which the ruled surfaces are
(infinitesimally) developable are given by $Q_{3}=0$.

(3) The mean directions: the pitch is $Q_{3}/Q_{1}=Q_{2}/Q_{1}-{\bar{b}}$ so
the directions in which the pitch has its maximal values are given by
$Q_{4}=Jac(Q_{2},Q_{1})$.

We determine the generic local topological configurations of the BDEs in
Proposition 4.4 and introduce a new BDE determining the so-called
characteristic directions. We deal in §4.1 with the pairs determined by the
geometry of the central points and in §4.2 with those determined by the pitch.
These BDEs are discussed in [12]; our approach reveals the way they are
related at each point via polarity in the projective plane. We also obtain the
generic configurations of the solution curves of the BDEs at their singular
points and on the singular set $\Sigma(n)$ of the map $n$.

As our study is local in nature, in the rest of the paper we take the line of
interest $L$ to be the oriented $z$-axis and parametrise $Z$ locally at $z$ by
$(u,v)\mapsto(n(u,v),x(u,v))$, with $(u,v)$ in a neighbourhood $U$ of the
origin in $\mathbb{R}^{2}$. We have $n(0,0)=(0,0,1)$ and take the directrix
$x(u,v)=(x_{1}(u,v),x_{2}(u,v),0)$ with

$\begin{split}j^{3}x_{1}&=\alpha_{10}u+\alpha_{11}v+\alpha_{20}u^{2}+\alpha_{21}uv+\alpha_{22}v^{2}+\displaystyle{\sum_{i=0}^{3}\alpha_{3i}u^{3-i}v^{i}},\\\
j^{3}x_{2}&=\beta_{10}u+\beta_{11}v+\beta_{20}u^{2}+\beta_{21}uv+\beta_{22}v^{2}+\displaystyle{\sum_{i=0}^{3}\beta_{3i}u^{3-i}v^{i}}.\end{split}$
(2)

The map $n:U\to S^{2}$ is a local diffeomorphism on $U\setminus\Sigma(n)$
where, for generic congruences, $\Sigma(n)$ is empty or a regular curve. The
map $n$ has locally a fold singularity at most points on $\Sigma(n)$ and a
cusp singularity at isolated points on that curve. With the notation in
Definition 4.1, $(u,v)\in\Sigma(n)$ if and only if $(B^{2}-AC)(u,v)=0$, that
is, $Q_{1}$ is degenerate.

To simplify notation, we shall omit mention of selecting $(u,v)\in U$, but
implicitly we are referring to the quadratic forms and projective plane at
each point $(n(u,v),x(u,v))\in Z$.

###  4.1 BDEs associated to central points

As above the principle directions are given by the BDE $Q_{2}=0$; its
integrals curves are the lines of principal curvature; point $z$ on $Z$ is
called an umbilic point if the all the coefficients of $Q_{2}$ vanish at $z$.
We observe that umbilic points are stable on a generic congruence, that is,
they persists when deforming the congruence.

Suppose that $Q_{1}$ is not degenerate, that is, $n$ is not singular. If
$\mu_{1},\mu_{2}$ are the values described in Theorem 3.1(2), the discriminant
function $\delta(Q_{2})$ of $Q_{2}$ is given by

$\delta(Q_{2})=4(B^{2}-AC)^{2}(H^{2}-K),$ (3)

where

$\begin{array}[]{c}H=\frac{1}{2}(\mu_{1}+\mu_{2}),\,K=\mu_{1}\mu_{2},\end{array}$

so $\delta(Q_{2})$ vanishes if and only if $H^{2}-K=0$, that is precisely at
umbilic points. These are points where all the coefficients of $Q_{2}$ vanish,
that is, where $Q_{1}$ and $Q$ are linearly dependent. Note that because of
the ambiguity in the choice of $Q$, $\mu_{1}$ and $\mu_{2}$ are only defined
up to the addition of the (same) constant, that is only $|\mu_{1}-\mu_{2}|$ is
well defined, and so is $H^{2}-K=(\mu_{1}-\mu_{2})^{2}/4$.

###### Theorem 4.5

(1) Away from $\Sigma(n)$ and umbilic points, the lines of principal curvature
form a $Q_{1}$-orthogonal net. The three generic configurations of BDEs of
Morse Type $A_{1}^{+}$ (Figure 2) can occur at umbilic points on generic
congruences and only these (see also [7]).

(2) For each $(u,v)\in U\setminus\Sigma(n)$ the quadratic form
$Q_{2}\in\mathbb{R}\mathbb{P}^{2}$ is the unique point on the polar line
$\widehat{Q}$ of $Q$ that has $Q_{1}$-orthogonal roots. It is the intersection
of the polar lines $\widehat{Q}_{1}$ and $\widehat{Q}$ (all forms evaluated at
$(u,v)$).

(3) On $\Sigma(n)$, $Q_{2}$ factors as a product of two $1$-forms
$\sigma_{i}$, $i=1,2$. At fold singularities of $n$ and at most points on
$\Sigma(n)$, the two $1$-forms are regular and their foliations have $2$-point
contact along $\Sigma(n)$. Their common tangent direction is along the kernel
of $dn$, so they are transverse to $\Sigma(n)$ (Figure 4 left). The pair
$(\sigma_{1},\sigma_{2})$ is topologically equivalent to $(du,d(u-v^{2}))$.

At isolated fold points on $\Sigma(n)$, one of the $1$-forms is regular and
the other has generically a saddle, node or focus singularity with eigenspaces
when real transverse to $\Sigma(n)$ (Figure 4 middle three figures). The pair
$(\sigma_{1},\sigma_{2})$ is topologically equivalent to

$\begin{array}[]{ll}(du,(v-u)du+vdv),&\mbox{if $\sigma_{2}$ is a saddle},\\\
(du,(v+\frac{1}{8}u)du+vdv),&\mbox{if $\sigma_{2}$ is a node},\\\
(du,(u+v)du+vdv),&\mbox{if $\sigma_{2}$ is a focus}.\end{array}$

At a cusp singularity of $n$, the two $1$-forms are regular and their leaves
have $3$-point contact at the singular point of $n$ and are tangent to
$\Sigma(n)$ (Figure 4, right). The pair $(\sigma_{1},\sigma_{2})$ is
topologically equivalent to $(du,d(u+vu^{2}+v^{3}))$.

Figure 4: Principal curvature on $\Sigma(n)$: left at generic fold singular
points of $n$, middle three at special fold singularities of $n$, left at a
cusp singularity of $n$. The curve in black is $\Sigma(n)$.

  * Proof

(1) For generic congruences, the umbilic points are not on $\Sigma(n)$ (three
independent conditions need to be satisfied for that to happen), so at umbilic
points, we can presume that $n$ is non-singular, that is $Q_{1}$ is positive
definite, and we can take

$n(u,v)=(u,v,\sqrt{1-u^{2}-v^{2}}).$ (4)

Taking $j^{3}x_{i},i=1,2,$ as in (2) we find that the 1-jets of the
coefficients of $Q_{2}$ are

$\begin{array}[]{rcl}j^{1}(Bc-
Cb)&=&\frac{1}{2}\left(\alpha_{11}+\beta_{10}+(\alpha_{21}+2\beta_{20})u+(2\alpha_{22}+\beta_{21})v\right),\\\
j^{1}(Ac-
Ca)&=&\alpha_{10}-\beta_{11}+(2\alpha_{20}-\beta_{21})u+(\alpha_{21}-2\beta_{22})v,\\\
j^{1}(Ab-
Ba)&=&\frac{1}{2}\left(\alpha_{11}+\beta_{10}+(\alpha_{21}+2\beta_{20})u)+(2\alpha_{22}+\beta_{21})v\right).\\\
\end{array}$

Then the origin is an umbilic point if and only if $\beta_{10}=-\alpha_{11}$
and $\beta_{11}=\alpha_{10}$. In that case,
$j^{1}Q_{2}=(a_{1}u+a_{2}v,b_{1}u+b_{2}v,-a_{1}u-a_{2}v),$ with

$\begin{array}[]{ll}\begin{array}[]{rcl}a_{1}&=&\frac{1}{2}\alpha_{21}+\beta_{20},\\\
b_{1}&=&2\alpha_{20}-\beta_{21},\\\
\end{array}&\begin{array}[]{rcl}a_{2}&=&\alpha_{22}+\frac{1}{2}\beta_{21},\\\
b_{2}&=&\alpha_{21}-2\beta_{22}.\end{array}\end{array}$

Clearly, we can choose the 2-jets of $x_{1}$ and $x_{2}$ so that
$a_{i},b_{i},i=1,2$, can take any values in $\mathbb{R}$. It follows by the
results in [1, 2] that the lines of principal curvature can have any of the
three generic topological configurations lemon, star or monstar in Figure 2,
and these are the only configurations that generically occur at umbilic
points. This result on the configurations at umbilics was obtained previously
by Craizer and Garcia in [7].

(2) The statement follows from Theorem 3.1.

(3) Suppose the origin is a fold singularity of $n$. We can take
$\Sigma(n)=\\{v=0\\}$ and $n_{v}=0$ on $\Sigma(n)$. Then $n_{vv}\neq 0$, so
$n_{v}(u,v)=vw(u,v)$ for some smooth $w$ with $w(0,0)\neq 0$ and $n_{u},w$
linearly independent in a neighbourhood of the origin. We have

$\begin{array}[]{c}A=n_{u}\cdot n_{u}\neq 0,B=vn_{u}\cdot w,C=v^{2}w_{v}\cdot
w_{v},\\\ a=-n_{u}\cdot x_{u},\,b=-\frac{1}{2}(n_{u}\cdot x_{v}+vw\cdot
x_{v}),\,c=-vw\cdot x_{v},\end{array}$

so the coefficients of $\omega(Q_{2})$ are

$\begin{array}[]{rcl}a_{P}=Bc-Cb&=&-v^{2}\left((n_{u}\cdot w)(w\cdot
x_{v})-\frac{1}{2}(n_{u}\cdot x_{v}+vw\cdot x_{u})(w\cdot w)\right),\\\
b_{P}=Ac-Ca&=&-v\left((n_{u}\cdot n_{u})(w\cdot x_{v})-v(w\cdot w)(n_{u}\cdot
x_{u}))\right),\\\ c_{P}=Ab-Ba&=&-\frac{1}{2}(n_{u}\cdot n_{u})(n_{u}\cdot
x_{v}+vw\cdot x_{u})+v(n_{u}\cdot w)(n_{u}\cdot x_{u}).\\\ \end{array}$

It follows that

$\delta(Q_{2})=v^{2}(n_{u}\cdot n_{u})\left[(n_{u}\cdot n_{u})(w\cdot
x_{v})^{2}-2(n_{u}\cdot w)(n_{u}\cdot x_{u})(w\cdot x_{v})+(w\cdot
w)(n_{u}\cdot x_{v})^{2}+vO(u,v)\right]$

for some smooth function $O(u,v)$. As $n_{u}$ and $w$ are linearly
independent, it follows that $\overline{Q_{1}}(du,dv)=w\cdot
wdv^{2}+2n_{u}\cdot wdudv+n_{u}\cdot n_{u}du^{2}$ defines (locally) a metric
on $Z$. The coefficient of $v^{2}$ in $\delta(Q_{2})$ is then
$\Lambda=(n_{u}\cdot n_{u})\overline{Q_{1}}(w\cdot x_{v},-n_{u}\cdot x_{v})$.
It vanishes if and only if $w\cdot x_{v}=n_{u}\cdot x_{v}=0$, which does not
occur on generic congruences at points on $\Sigma(n)$. Therefore, $\Lambda>0$.
We now have two cases, $c_{P}(0,0)=(n_{u}\cdot n_{u})(n_{u}\cdot x_{v})$
vanishing or not.

If $n_{u}\cdot x_{v}(0,0)\neq 0$, $c_{P}(0,0)\neq 0$ so $Q_{2}$ is a non-zero
multiple of the product of the following two regular 1-forms

$\sigma_{i}=2c_{P}du+(-b_{P}-(-1)^{i}\sqrt{\delta(Q_{2})}\,)dv,\,i=1,2.$

On $v=0$, the above one forms reduce to $du=0$ which is the kernel direction
of $dn$. Their leaves have $2$-point contact at points on $v=0$. The result on
their topological configurations follows from Theorem 2.2 in [8].

If $n_{u}\cdot x_{v}(0,0)=0$, then all the coefficients of $Q_{2}=0$ vanish at
the origin. Away from the curve $c_{P}(u,v)=0$, $Q_{2}$ is still a product of
the above 1-forms $\omega_{i}$. Writing $a_{P}=v^{2}\bar{a}_{P}$,
$b_{P}=v\bar{b}_{P}$, $c_{P}=\bar{c}_{P}$, and assuming $\bar{b}_{P}(0,0)\neq
0$ for generic congruences (we have already two independent condition at the
origin), and without loss of generality that $\bar{b}_{P}(0,0)>0$, we have

$\sigma_{i}=2\bar{c}_{P}du+v(-\bar{b}_{P}-(-1)^{i}\bar{b}_{P}(1-\frac{4\bar{a}_{P}\bar{c}_{P}}{\bar{b}_{P}^{2}})^{\frac{1}{2}}\,)dv,\,i=1,2.$

The 1-form $\sigma_{1}=\bar{c}_{P}(2du+h(u,v)dv)$ for some regular function
$h$. So it defines a regular foliation away from $\bar{c}_{P}=0$ which extends
to a regular foliation on $\bar{c}_{P}=0$ (we get a line of singularities on
$\bar{c}_{P}=0$ which does not alter the configuration of the foliation
determined by $\sigma_{1}$). For the analysis of the singularity of the 1-form
$\sigma_{2}$ we can take the first two component of $n$ in the form
$(u,(1+k_{10}u+k_{11}v+O(2))v^{2}+l_{2}u^{2}+O(u^{3})))$. Then $n_{u}\cdot
x_{v}(0,0)=\alpha_{11}=0$ and
$j^{1}\sigma_{2}=\left((2l_{2}\beta_{11}+\alpha_{21})u)+2(\alpha_{22}+\beta_{10})v\right)du+2\beta_{11}vdv.$
Clearly, $\sigma_{1}$ can have generically a saddle, node or focus
singularity. The result on the topological configurations of the pair
$(\sigma_{1},\sigma_{2})$ follows from Theorem 2.2 in [8].

At a cusp singularity of $n$, we can still take $n_{v}=0$ on
$\Sigma(n)=\\{\lambda(u,v)=0\\}$ for some smooth function $\lambda$ with
$\lambda_{u}(0,0)\lambda_{vv}(0,0)\neq 0$. We can then proceed as above and
write $n_{v}(u,v)=\lambda(u,v)w(u,v)$ with $w(0,0)\neq 0$ and $x_{u}$ and $w$
linearly independent. The coefficients $a_{P},b_{P},c_{P}$ are as above but
replacing the factor $v$ by $\lambda(u,v)$. For generic congruences we cannot
have any extra conditions, so $c_{P}(0,0)\neq 0$ and $Q_{2}$ factors as a
product of two $1$-forms $\sigma_{i}$, $i=1,2$. We can show that the leaves at
the origin have $3$-point contact between them and $2$-point contact with
$\Sigma(n)$. The result on the topological configurations of the pair
$(\sigma_{1},\sigma_{2})$ also follows from Theorem 2.2 in [8]. $\Box$

###  4.2 BDEs associated to the parameter distribution

We consider as in §4.1 a curve $\gamma$ on the surface $Z$ with
$\gamma(0)=(n(0),x(0))=z$ and seek the directions $\gamma^{\prime}(0)\in
T_{z}Z$ for which the parameter distribution $\lambda$ of the associated ruled
surface in $\mathbb{R}^{3}$ vanishes. These are the torsal directions and
their integral curves are the torsal curves. We have the following
observation.

###### Proposition 4.6

For a generic congruence, the coefficients of the BDE $Q_{3}=0$ of the torsal
curves vanish simultaneously at $z$ if and only if $n$ is singular at $z$.

  * Proof

We showed in the proof of Proposition 4.2 (2) that the coefficients of $Q_{3}$
all vanish on $\Sigma(n)$. For the converse, setting to zero the coefficients
of $Q_{3}$ leads to $(AC-B^{2})(Ac-aC)=0$. The first factor vanishes if and
only if $n$ is singular. If the second factor vanishes then it is easy to see
that $\bar{b}=0$ and $z$ an umbilic point, so we have three independent
conditions which cannot be satisfied simultaneously for a generic congruence.
$\Box$

###### Theorem 4.7

(1) At each point $z$ on $Z\setminus\Sigma(n)$ there are $2,1$ or $0$ torsal
directions in $T_{z}Z$ for which the pitch $\lambda$ vanishes. These are given
by the BDE $Q_{3}=Q_{2}-{\bar{b}}Q_{1}=0$.

We have $\delta(Q_{3})=\delta(Q_{2})+\overline{b}^{2}\delta(Q_{1})$. The
discriminant of the BDE $Q_{3}=0$ is generically a smooth curve on $Z$, called
the parabolic curve, and all the configurations in Figure 1 can occur, and
generically only these. The set on $Z$ where $\delta(Q_{3})>0$ (resp.
$\delta(Q_{3})<0$) is called the hyperbolic (resp. elliptic) region of $Z$.

(2) The torsal curves extend to $\Sigma(n)$. For generic congruences, the
curve $\Sigma(n)$ lies in the closure of the hyperbolic region of $Z$. The
parabolic curve intersect $\Sigma(n)$ tangentially at isolated points. At such
points, the torsal curves form locally a family of cusps as in Figure 5.

Figure 5: Generic configuration of the torsal curves on $\Sigma(n)$ and at a
parabolic point on $\Sigma(n)$. The thick curve in black represents
$\Sigma(n)$ and the curve in grey the parabolic curve.

  * Proof

(1) We have already found the form of the BDE which is also
$Q_{2}-\overline{b}Q_{1}=0.$ This means that $Q_{3}$ belongs to the pencil
determined by $Q_{1}$ and $Q_{2}$, that is, to the polar line of
$Jac(Q_{1},Q_{2}).$

A straightforward calculation shows that
$\delta(Q_{3})=\delta(Q_{2})+4\overline{b}^{2}\delta(Q_{1})$. Using (3), we
get
$\delta(Q_{3})=4\delta(Q_{1})^{2}(H^{2}-K)+4\overline{b}^{2}\delta(Q_{1})$,
with $\delta(Q_{1})=B^{2}-AC$, so

$\delta(F)=0\iff H^{2}-K+\frac{\overline{b}^{2}}{B^{2}-AC}=0.$ (5)

Equation (5) generically determines a regular curve on the surface. For the
configuration of the BDE $Q_{3}=0$, we use the setting of the proof of Theorem
4.5(1). Then the 2-jets of the coefficients $(a_{T},b_{T},c_{T})$ of $Q_{3}$
are given by

$\begin{array}[]{rl}j^{2}a_{T}=&\alpha_{11}+\alpha_{21}u+2\alpha_{22}v+\alpha_{31}u^{2}-(\beta_{11}-2\alpha_{32})uv+(3\alpha_{33}+\alpha_{11})v^{2},\\\
j^{2}b_{T}=&\alpha_{10}-\beta_{11}+(2\alpha_{20}-\beta_{21})u+(\alpha_{21}-2\beta_{22})v+(3\alpha_{30}-\beta_{11}-\beta_{31})u^{2}+\\\
&(\alpha_{11}+2\alpha_{31}-2\beta_{32}-\beta_{10})uv+(\alpha_{10}+\alpha_{32}-3\beta_{33})v^{2},\\\
j^{2}c_{T}=&-\beta_{10}-2\beta_{20}u-\beta_{21}v-(3\beta_{30}+\beta_{10})u^{2}+(\alpha_{10}-2\beta_{31})uv-\beta_{32}v^{2}.\end{array}$

The origin is on the discriminant if and only if
$(\alpha_{10}-\beta_{11})^{2}+4\alpha_{11}\beta_{10}=0$. It is a singular
point of $Q_{3}=0$ if and only if

$(\alpha_{10}-\beta_{11})^{3}\beta_{20}+2(\alpha_{10}-\beta_{11})^{2}(\beta_{21}-\alpha_{20})\beta_{10}+4\beta_{10}^{2}(\alpha_{10}-\beta_{11})(\beta_{22}-\alpha_{21})-8\alpha_{22}\beta_{10}^{3}=0.$
(6)

Following the arguments in §2, we can show that all the stable singularities
in Figure 1 can occur. For generic congruences, those are the only
singularities that can occur.

(2) We follow the setting of the proof of Theorem 4.5 (3) at a fold or a cusp
singularity of $n$. We set $n_{v}(u,v)=\lambda(u,v)w(u,v)$ with
$\Sigma(n)=\\{\lambda(u,v)=0\\}$ and $w,n_{u}$ linearly independent. Then
$(a_{T},b_{T},c_{T})=(\lambda^{2}\tilde{a}_{T},\lambda\tilde{b}_{T},\lambda\tilde{c}_{T})$
for some smooth functions $\tilde{a}_{T},\tilde{b}_{T},\tilde{c}_{T}$. This
means that the torsal curves have a removable singularity on $\Sigma(n)$. We
define their extension to $\Sigma(n)$ as the solutions of the BDE with
coefficients $(\lambda\tilde{a}_{T},\tilde{b}_{T},\tilde{c}_{T})$. Then the
parabolic curve extended to points on $\Sigma(n)$ and is given by
$\tilde{b}_{T}^{2}-4\lambda\tilde{a}_{T}\tilde{c}_{T}=0$. In particular, it
intersects $\Sigma(n)$ at the origin if and only if $\tilde{b}_{T}(0,0)=0$ and
for generic congruences, the two curves have ordinary tangency at that point.

The remaining part of the proof follows by computing the initial jets of the
coefficients $(\lambda\tilde{a}_{T},\tilde{b}_{T},\tilde{c}_{T})$. Following
the setting in the proof of Theorem 4.5 (3) at a fold singularity of $n$, we
get

$\begin{array}[]{rcl}j^{1}(v\tilde{a}_{T})&=&4\alpha_{11}v,\\\
j^{1}\tilde{b}_{T}&=&-2\beta_{11}+2(2l_{2}\alpha_{11}-\beta_{11}k_{10}-\beta_{21})u+(4\alpha_{10}-3\beta_{11}k_{11}-4\beta_{22})v,\\\
j^{1}\tilde{c}_{T}&=&-2\beta_{10}+2(2l_{2}\alpha_{10}-\beta_{10}k_{10}-2\beta_{20})u-(3\beta_{10}l_{11}+2\beta_{21})v.\end{array}$

The origin is on the parabolic curve if and only if $\beta_{11}=0$. The
parabolic set is generically a smooth curve ($\alpha_{11}\beta_{10}\neq 0$)
and has ordinary tangency with $\Sigma(n)$
($2l_{2}\alpha_{11}-\beta_{11}k_{10}-\beta_{21}\neq 0$) at the origin. The
torsal curves form a family of cusps along the parabolic curve as
$\alpha_{11}\beta_{10}\neq 0$. Observe that along $\Sigma(n)$,
$\delta(Q_{3})=\tilde{b}_{T}^{2}(u,0)>0$, so there are always two real torsal
directions away from parabolic points.

The claim at a cusp singularity of $n$ follows similarly observing that for
generic congruences the cusp singularity of $n$ is not on the parabolic curve.
$\Box$

We turn now to the extremes of the pitch, these occur along the mean
directions.

###### Theorem 4.8

(1) At every non-umbilic point on $Z\setminus\Sigma(n)$ there are two
$Q_{1}$-orthogonal directions along which the parameter distribution has an
extremum. These are given by the BDE $Q_{4}=Jac(Q_{2},Q_{1})=0$, that is,

$Q_{4}=\left|\begin{array}[]{ccc}dv^{2}&-dudv&du^{2}\\\ A&B&C\\\ Ab-Ba&(Ac-
Ca)/2&Bc-Cb\end{array}\right|=0.$ (7)

The discriminant of the BDE (7) consists of the umbilic points, and all the
configurations in Figure 2 can occur at such points and only these.

(2) The triple $Q_{1},Q_{2},Q_{4}$ are vertices of a self-polar triangle.

(3) For generic congruences, the solution curves of $Q_{4}=0$ extend to
$\Sigma(n)$. Their configurations are as those in Figure 4 with the special
fold points generically distinct from those of the principal curves.

  * Proof

(1) We have already established the form of the BDE, and the orthogonality
condition follows from Theorem 3.1. We have
$\delta(Q_{4})=-\delta(Q_{1})\delta(Q_{2})$ which vanishes only at umbilic
points or on $\Sigma(n)$.

At an umbilic point which is generically not on $\Sigma(n)$, and with the same
setting as for the previous cases, we have
$j^{1}Q_{4}=(\bar{a}_{1}u+\bar{a}_{2}v,\bar{b}_{1}u+\bar{b}_{2}v,-\bar{a}_{1}u-\bar{a}_{2}v)$
with

$\begin{array}[]{lcl}\bar{a}_{1}=-2\alpha_{20}+\beta_{21}&&\bar{a}_{2}=-\alpha_{21}+2\beta_{22},\\\
\bar{b}_{1}=2(\alpha_{21}+2\beta_{20})&&b_{2}=2(2\alpha_{22}+\beta_{21}).\end{array}$

Clearly, as in the proof of Theorem 4.5, we can obtain the three generic
configurations in Figure 2 and show that only these occur. Observe that
$\bar{a}_{1}u+\bar{a}_{2}v=-b_{1}u-b_{2}v$ and
$\bar{b}_{1}u+\bar{b}_{2}v=4(a_{1}u+a_{2}v)$ with
$j^{1}Q_{2}=(a_{1}u+a_{2}v,b_{1}u+b_{2}v,-a_{1}u-a_{2}v)$ as in the proof of
Theorem 4.5.

(2) We have $Jac(Q_{2},Q_{4})=\delta(Q_{2})Q_{1}$,
$Jac(Q_{1},Q_{4})=\delta(Q_{1})Q_{2}$ and $Q_{4}=Jac(Q_{1},Q_{2})$, so
$Q_{1},Q_{2},Q_{4}$ are vertices of a self-polar triangle.

(3) The analysis at points on $\Sigma(n)$ is identical to that for principal
curves and is omitted. $\Box$

###  4.3 Characteristic directions and characteristic points

The BDEs of the torsal and mean directions determine another BDE on the
surface, namely, $Q_{5}=Jac(Q_{3},Q_{4})=0$. We call its solutions at each
point $z$ the characteristic directions and their integral curves the
characteristic curves.

###### Theorem 4.9

(1) At each point $z$ on the surface $Z\setminus\Sigma(n)$ there are $2,1$ or
$0$ characteristic directions in $T_{z}Z$. These are given by the BDE

$Q_{5}=-\delta(Q_{2})Q_{1}-\overline{b}\delta(Q_{1})Q_{2}=0,$

so $Q_{5}$ is on the polar line $\widehat{Q_{4}}$ of $Q_{4}$. The triple
$Q_{3},Q_{4},Q_{5}$ are vertices of a self-polar triangle.

(2) The discriminant of the BDE $Q_{5}=0$ is given by
$\delta(Q_{5})=\delta(Q_{1})\delta(Q_{2})\delta(Q_{3})$. Consequently, when
the torsal directions are real the characteristic ones are imaginary and vice-
versa, that is, the characteristic curves lie on the closure of the elliptic
region of $Z$.

(3) The discriminant of $Q_{5}$ is the union of the parabolic curve together
with the umbilic points. The folded singularities of the characteristic
directions occur at the same points as those of the torsal directions. The two
configurations have opposite indices (when one has a folded saddle, the other
is a folded node or focus). At umbilic points, the characteristic curves have
the same configurations as those of the principal curves.

(4) The characteristic curves BDE extend to $\Sigma(n)$. For generic
congruences and away from parabolic points, there are no characteristic
directions on $\Sigma(n)$. At a parabolic point on $\Sigma(n)$, we get a
family of cusps along the parabolic curve.

  * Proof

(1) and (2). The expression for $Q_{5}$ follows by calculating
$Jac(Q_{2},Q_{3})$. Clearly it is on the polar line $\widehat{Q_{4}}$ of
$Q_{4}=Jac(Q_{1},Q_{2})$. We have
$Jac(Q_{4},Q_{5})=\delta(Q_{1})\delta(Q_{2})Q_{3}$ so the polar line
$\widehat{Q_{3}}$ of $Q_{3}$ contains $Q_{4}$ and $Q_{5}$. The polar line
$\widehat{Q_{5}}$ of $Q_{5}$ contains $Q_{3}$ and $Q_{4}$ by definition of
$Q_{5}.$ This proves that $Q_{3},Q_{4},Q_{5}$ are vertices of a self-polar
triangle.

Observe that away from $\Sigma(n)$ and umbilic points $\delta(Q_{1})<0$ and
$\delta(Q_{2})>0$, so $\delta(Q_{5})$ and $\delta(Q_{3})$ have opposite signs.
It follows that the torsal curves and the characteristic curves live on
opposite sides of the parabolic curve.

(3) We follow the setting of the proof of the previous cases and compute the
2-jet of the coefficients of BDE $Q_{5}$. At a parabolic point, we find that
the BDE is singular if and only if the condition (6) is satisfied. We compute
the scalar $\lambda$ that determines the type of the folded singularity (see
§2) and find that it is the opposite of that associated with of the folded
singularity of the torsal direction.

At umbilic points, the 1-jet of $Q_{5}$ is a scalar multiple of that of the
principal curves, so at generic umbilic points, the two BDEs have the same
configurations. $\Box$

When the characteristic directions are real at a point $z$ on $Z$, they
determine points on the line $L\subset\mathbb{R}^{3}$ associated to $z$ which
we call characteristic points. We call the surface in $\mathbb{R}^{3}$ that
they trace as $z$ varies in $Z$ the characteristic surface.

###### Theorem 4.10

There are $2,1$ or $0$ characteristic points on the line
$L\subset\mathbb{R}^{3}$. When there are two of them, there are no focal
points and vice-versa. The midpoint of the characterise points is the midpoint
of the boundary points. The characteristic points lie between the boundary
points; see Figure 6.

  * Proof

We consider the case when $n$ is not singular; the singular case is similar.
Then each characteristic direction (when it exists) at a point $z\in Z$
determines a characteristic point on the line $L\subset\mathbb{R}^{3}$ with
distance $r=-x^{\prime}.n^{\prime}/n^{\prime}.n^{\prime}$ from $x$, with
$(u^{\prime},v^{\prime})$ a solution of $Q_{5}=0$. Eliminating
$u^{\prime},v^{\prime}$ gives

$r^{2}-2Hr+K+\frac{AC-B^{2}}{\bar{b}^{2}}(H^{2}-K)^{2}=0.$

It follows that the midpoint $(x+Hn)(0,0)$ of the characteristic points is the
midpoint of the boundary points (these are given by $r^{2}-2Hr+K=0,$ [12]).
The square of the distance between the characteristic points is
$H^{2}-K-(AC-B^{2})(H^{2}-K)^{2}/{\bar{b}^{2}}$ and that between the boundary
points is $H^{2}-K$. Therefore the characteristic points are between the
boundary points. $\Box$

Figure 6: Relative position of the boundary, focal/characteristic and
midpoints.

##  5 Normal congruences

A congruence $Z\subset\mathbb{L}$ is said to be a normal congruence if for
some smooth surface $X\subset\mathbb{R}^{3}$ the set $Z$ consists of the
normals to $X$ denoted by $N(X)$.

Normal congruences are important in geometrical optics, where $X$ can be
thought of as a wavefront and the associated lines as rays. In this case the
quadratic forms in the previous section are determined by the first and second
fundamental forms of $X$.

We parametrise the directrix $X$ away from its umbilic points so that the
coordinate curves represent the lines of principal curvature. Then the
coefficients of the first fundamental form are $x_{u}\cdot x_{u}=E$,
$x_{u}\cdot x_{v}=0$, $x_{v}\cdot x_{v}=G$.

The quadratic form $Q_{1}$ is the third fundamental form of $X$ and has
coefficients $A=\kappa_{1}^{2}E$, $B=0$ and $C=\kappa_{2}^{2}G$.

For normal congruences, $\bar{b}=0$ (in fact $\bar{b}=0$ if and only if $Z$ is
a normal congruence, [12], p. 196) and $a=\kappa_{1}E,b=0,c=\kappa_{2}G$ are
the coefficients of the second fundamental form of $X$.

The quadratic form $Q_{2}=\kappa_{1}\kappa_{2}EG(\kappa_{1}-\kappa_{2})dudv$,
so the solution curves of $Q_{2}=0$, i.e., the principal curves of the
congruence $Z$ coincide with the lines of principal curvature of $X$ in the
parameter space. In fact, the natural map $X\to\mathbb{L},x\mapsto(x,N(x))$
with image $N(X)$, takes the classical principle directions of $X$ to the
principal directions of the congruence $N(X)$. (Here $\Sigma(n)$ is the
parabolic set of $X$ given by $\kappa_{1}\kappa_{2}=0$, so the principal
directions on $Z$ have removable singularities on $\Sigma(n)$.)

We have $Q_{3}=Q_{2}$, so the torsal curves coincide with the principal
curves, so every non-umbilic point on $N(X)$ is a hyperbolic point.

The BDE of the mean directions is given by

$Q_{4}=Jac(Q_{1},Q_{2})=\kappa_{2}^{2}Gdv^{2}-\kappa_{1}^{2}Edu^{2}=0.$

The directions determined by $Q_{4}$ are called minimal orthogonal spherical
image directions in [10]. They are the unique pair of tangent directions to
$X$ that have orthogonal images under $dn$ and that are inclined at a minimal
angle at each point of $X$.

The characteristic BDE is
$Q_{5}=Jac(Q_{4},Q_{2})=\kappa_{2}^{2}Gdv^{2}+\kappa_{1}^{2}Edu^{2}=0,$ so
does not have real solutions (there are no elliptic points on $N(X)$).

  * Acknowledgments.

The work in this paper was partially supported by the FAPESP Thematic project
grant 2019/07316-0.

## References

  * [1] J. W. Bruce and D. Fidal, On binary differential equations and umbilics. Proc. Royal Soc. Edinburgh 111A (1989), 147–168.
  * [2] J. W. Bruce and F. Tari, On binary differential equations. Nonlinearity 8 (1995), 255–271.
  * [3] J. W. Bruce and F. Tari, , Generic 1-parameter families of binary differential equations. Discrete Contin. Dyn. Syst. 3 (1997), 79–90.
  * [4] J. W. Bruce and F. Tari, Duality and implicit differential equations. Nonlinearity 13 (2000), 791–812.
  * [5] J. W. Bruce and F. Tari, Dupin indicatrices and families of curve congruences. Trans. Amer. Math. Soc. 357 (2005), 267–285.
  * [6] J. W. Bruce and F. Tari, On the affine geometry of congruences of lines. Preprint, 2023.
  * [7] M. Craizer and R. A. Garcia, Singularities of generic line congruences. To appear in J. Math. Soc. Japan.
  * [8] A.A. Davydov, Qualitative control theory. Translations of Mathematical Monographs 142, AMS, Providence, R.I., 1994.
  * [9] V. Guíñez, Positive quadratic differential forms and foliations with singularities on surfaces. Trans. Amer. Math. Soc. 309 (1988), 477–502.
  * [10] J. M. Oliver, Pairs of geometric foliations on regular and singular surfaces. Doctoral thesis, Durham University, 2010.
  * [11] H. Pottmann and J. Wallner, Computational Line Geometry. Springer, 2010.
  * [12] C. E. Weatherburn, Differential Geometry of Three Dimensions. Cambridge University Press, 1955.

JWB: Department of Mathematical Sciences, University of Liverpool, Liverpool,
L69 3BXl
E-mail<EMAIL_ADDRESS>

FT: Instituto de Ciências Matemáticas e de Computacão - USP, Avenida
Trabalhador são-carlense, 400 - Centro, CEP: 13566-590 - São Carlos - SP,
Brazil.
E-mail<EMAIL_ADDRESS>


# Estimating the Robustness of Classification Models by the
Structure of the Learned Feature-Space

Kalun Ho, Franz-Josef-Pfreundt
CC-HPC, Fraunhofer ITWM
Fraunhofer-Platz 1
67663 Kaiserslautern
Germany
<EMAIL_ADDRESS>Janis Keuper
Institute for
Machine Learning
and Analytics
Offenburg University
Germany  Margret Keuper
Data and Web Science Group
University of Mannheim
Germany

###### Abstract

Over the last decade, the development of deep image classification networks
has mostly been driven by the search for the best performance in terms of
classification accuracy on standardized benchmarks like ImageNet. More
recently, this focus has been expanded by the notion of model robustness,
_i.e_. the generalization abilities of models towards previously unseen
changes in the data distribution. While new benchmarks, like ImageNet-C, have
been introduced to measure robustness properties, we argue that fixed testsets
are only able to capture a small portion of possible data variations and are
thus limited and prone to generate new overfitted solutions. To overcome these
drawbacks, we suggest to estimate the robustness of a model directly from the
structure of its learned feature-space. We introduce robustness indicators
which are obtained via unsupervised clustering of latent representations from
a trained classifier and show very high correlations to the model performance
on corrupted test data.

##  1 Introduction

Deep learning approaches have shown rapid progress on computer vision tasks.
Much work has been dedicated to train ever deeper models with improved
validation and test accuracies and efficient training schemes [53, 19, 33,
20]. Recently, this progress has been accompanied by discussions on the
robustness of the resulting model [9]. Specifically, the focus shifted towards
the following two questions: 1. How can we train models that are robust with
respect to specific kinds of perturbations? 2. How can we assess the
robustness of a given model? These two questions represent fundamentally
different perspectives on the same problem. While the first question assumes
that the expected set of perturbations is known during model training, the
second question rather aims at estimating a models behavior in unforeseen
cases and predict its robustness without explicitly testing on specific kinds
of corrupted data.

Figure 1: Predicting the robustness of models using our proposed cluster
purity indicator $(p_{purity})$: The correlation between $p_{purity}$ of
models trained on the original ImageNet with the measured test accuracy on
ImageNet-C is $R^{2}=0.87$.

In this paper, we address the second research question. We argue that the
clustering performance in a model’s latent space can be an indicator for a
model’s robustness. For this purpose, we introduce cluster purity as a
robustness measure in order to predict the behavior of models against data
corruption and adversarial attacks. Specifically, we evaluate various
classification models [29, 53, 21, 15, 44, 52, 22, 47] on the ImageNet-C [16]
dataset of corrupted ImageNet images where we measure the robustness of a
model as the ratio between the accuracy on corrupted data and clean data. The
key result of this paper is illustrated in figure 1: it shows that the model
robustness is strongly correlated to the relative clustering performance on
the models’ latent spaces, i.e. the ratio between the cluster purity and the
classification accuracy, both evaluated on clean data. The clusterability of a
model’s feature space can therefore be considered as an easily accessible
indicator for model robustness.

In summary, our work contributes the following:

  * •

We study the feature spaces of several ImageNet pre-trained models including
the state-of-the-art CNN models [53, 21, 15, 44, 52] and the recently proposed
transformer models [47] and evaluate their model robustness on the ImageNet-C
dataset and against adversarial attacks.

  * •

We show that intra- and inter-class distances extracted from classification
models are not suitable as a direct indicator for a model’s robustness.

  * •

We provide a study of two clustering methods, K-means and the Minimum Cost
Multicut Problem (MP) and analyze the correlation between classification
accuracy, robustness and clusterability.

  * •

We show that the relative clustering accuracy, _i.e_. the ratio between
classification and clustering performance, is a strong indicator for the
robustness of the classification model under ImageNet-C corruptions.

This paper is structured as follows: We first review the related work on image
classification, model robustness and deep clustering approaches in Section 2,
then we propose the methodology for the feature space analysis in Section 3.
Our experiments and results are discussed in Section 4.

##  2 Related Work

Image Classification. Convolutional neural networks (CNN) have shown great
success in computer vision. In particular, from the classification of
handwritten characters [32] to images [28], CNN-based methods consistently
achieve state-of-the-art in various benchmarks. With the introduction of
ImageNet [39], a dataset with higher resolution images and one thousand
diverse classes is available to benchmark the classification accuracy of ever
better performing networks [29, 53, 21, 15, 44, 52], ranging from small and
compact netork [19] to large models [43] with over 100 millions of parameters.

Transformers. Recently, transformer network architectures, which were
originally introduced in the area of natural language processing [49], have
been successfully applied to the image classification task [6, 10]. The
performance of transformer networks is competitive despite having no
convolutional layers. However, transformer models require long training times
and large amounts of data [10] in order to generalize well. A more efficient
approach for training has been proposed in [47], which is based on a teacher-
student strategy (distillation). Similarly, [5] uses the same strategy on
self-supervised tasks.

Model Robustness. Convolutional neural networks are susceptible to
distribution shifts [38] between train and test data [36, 11, 16, 40]. This
concerns both visible input domain shifts by for example considering
corrupted, noisy or blurred data, as well as imperceptible changes in the
input, induced by [35, 14, 31]. These explicitly maximize the error rate of
classification models [45, 3] and thereby reveal model weaknesses. Many
methods have been proposed to improve the adversarial robustness by specific
training procedures, _e.g_. [35, 24]. In contrast, input distribution shifts
induced by various kinds of noise as modeled in the ImageNet-C [16] dataset
mimic the robustness of a model in unconstrained environments, for example
under diverse weather conditions. This aspect is crucial if we consider
scenarios like autonomous driving, where we want to ensure robust behaviour
for example under strong rain. Therefore, we focus on the latter aspect and
investigate the behaviour of various pre-trained models under ImageNet-C
corruptions but also evaluate the proposed robustness measure on adversarial
perturbations [35, 24].

Clustering. Clustering approaches, deep clustering approaches in particular,
have shown to benefit from well structured feature spaces. Such approaches
therefore aim at optimizing the latent representations for example using
variational autoencoders or Gaussian mixture model or K-means priors [37, 51,
13, 12, 4]. [4] iteratively groups points using K-means during the latent
space optimization. Conversely, we are investigating the actual feature space
learned from image classification tasks using clusterability as a measure for
its robustness. Therefore, we apply clustering approaches on pre-trained
feature spaces. Further, while the above mentioned methods rely on a K-means-
like clustering, _i.e_. data is clustered into a given number of clusters, we
also evaluate clusters from a similarity driven clustering approach, the
Minimum Cost Multicut Problem [1].
The Multicut Problem, aka. Correlation Clustering, groups similar data points
together by pairwise terms: data (_e.g_. images) are represented as nodes in a
graph. The real valued weight of an edge between two nodes measures their
similarity. Clusters are obtained by cutting edges in order to decompose the
graph and minimize the cut cost. This problem is known to be NP-hard [8]. In
practice, heuristic solvers often perform reasonably [25, 2]. Correlational
Clustering has various applications in computer vision, such as motion
tracking and segmentation [26, 50], image clustering [17] or multiple object
tracking [46, 17].

##  3 Feature Space Analysis

Our aim is to establish indicators for a model’s robustness from the structure
of its induced latent space. Therefore, we first extract latent space samples,
_i.e_. feature representations of input test images. The latent space
structure is subsequently analyzed using two different clustering approaches.
K-means is clustering data based on distances to a fixed number of cluster
means and can therefore be interpreted as a proxy of how well the latent space
distribution can be represented by a univariate Gaussian mixture model. The
Minimum Cost Multicut problem formulation clusters data points based on their
pairwise distances and therefore imposes less constraints on the data manifold
to be clustered. Figure 2 gives an overview of the methodology. First, we
briefly recap classification models as feature extractors in Section 3.1. The
K-means and Minimum Cost Multicut Problem on the image clustering task are
explained in Section 3.2. In Section 3.3, we review evaluation metrics for
measuring the clustering performance and in Section 3.4, we present our
proposed metrics for robustness estimation.

Figure 2: The robustness of a model is measured by its relative classification
performance, which is the ratio between clean and corrupted (in red arrow)
data.. The latent space or features (in blue) of various classification models
is sampled using ImageNet images. The feature representations are then
clustered with the K-means and Multicut clustering approaches. The correlation
is visualized in 1.

###  3.1 Extracting Features from Classification Models

Classification models with multiple classes are often trained with softmax
cross-entropy and it has been shown that features, learned from vanilla
softmax cross-entropy achieve a high performance in transfer accuracy [27]. In
order to obtain the learned features from images, the last layer of the
trained model (classifier) is removed, which is often done for instance in
transfer learning [41, 42] or clustering tasks [51]. The model encodes an
image $x_{i}$ with a function $f_{\theta}(.)$, with pre-trained parameters
$\theta$ . Table 1 shows the different classification models with their
according feature dimensions as well as the number of parameters and their top
1 classification accuracy in %. We investigate models which vary significantly
in their architectures, including CNNs and transformer models, their number of
parameters, ranging from 3.5M to 138M, as well as their test accuracy, ranging
from 56.4% to 81.2% top-1 scores. We use features extracted from the full
ImageNet test set as latent space samples for our analysis as shown in 2.

Table 1: Classification models: all models are trained and evaluated on the ImageNet [39] dataset, sorted by performance. We report the Top1 classification accuracy in %. The first ten models are based on convolutional layers while the last two are transformer networks. Model | Features | Param | Top1 %
---|---|---|---
AlexNet [29] | 4096 | 61.1M | 56.4
vgg11 [53] | 4096 | 132.9M | 69.0
vgg16 [53] | 4096 | 138.4M | 71.6
bninception [22] | 1024 | 11.3M | 73.5
nasnetamobile [53] | 1056 | 5.3M | 74.1
Densenet121 [21] | 1024 | 7.9M | 74.6
Resnet50 [15] | 2048 | 25.6M | 76.0
Resnet101 [15] | 2048 | 44.5M | 77.4
incResNv2 [44] | 1536 | 55.8M | 80.2
Polynet [52] | 2048 | 95.3M | 81.0
Deit-tiny [47] | 192 | 5.9M | 74.5
Deit-small [47] | 384 | 22.4M | 81.2

###  3.2 Latent Space Clustering

K-means. K-means is a simple and effective method to cluster $N$ data points
into $K$ clusters $S_{k}$, $k=1,\dots,K$. As $K$ is set a priori, this method
produces exactly the number of defined clusters by minimizing the intra-
cluster distance:

$\sum_{k=1}^{K}\sum_{x_{i}\in S_{k}}||f(x_{i})-{\mu_{k}}||^{2}\,$ (1)

where the centroid ${\mu_{k}}$ is computed as the mean of features
$\frac{1}{|S_{k}|}\sum_{x_{i}\in S_{k}}f(x_{i})$ in cluster $k$.

Multicut Clustering.  The Minimum Cost Multicut Problem is a graph based
clustering approach. Considering an undirected graph $G=(V,E)$, with $v\in V$
being the images $x_{i}$ of the dataset $X$ with $|V|=N$ samples, a complete
graph with $N$ nodes has in total $|E|=\frac{N(N-1)}{2}$ edges. A real valued
cost $w:E\rightarrow\mathbb{R}$ is assigned to every edge $e\in E$. While the
decision, whether an edge is joined or cut, is made based on the edge label
$y:E\rightarrow\\{0,1\\}$, the decision boundary can be derived from training
parameters of the model [18], directly learned from the dataset [17, 46] or
simply estimated empirically (via parameter search). The inference of such
edge labels is defined as follows:

$\displaystyle\min\limits_{y\in\\{0,1\\}^{E}}\sum\limits_{e\in E}w_{e}y_{e}$
(2)

$\displaystyle\mathrm{s.t.}\quad\forall C\in cycles(G)\quad\forall e\in
C:y_{e}\leq\sum\limits_{e^{\prime}\in C\backslash\\{e\\}}y_{e^{\prime}}$ (3)

Here, edges with negative costs $w_{e}$ have a high probability to be cut.
Equation (3) enforces that for each cycle in $G$, a cut is only allowed if
there is at least another edge being cut as well, which was shown in [7] to be
sufficient to enforce on all _chordless_ cycles.Practically, the edge costs
are computed from pairwise distances in the feature space. The distance
$d_{i,j}$ between two features $f(x_{i})$ and $f(x_{j})$ is calculated from
the pre-trained model or encoder $f$, where $x_{i}$ and $x_{j}$ are two
distinct images from the test dataset, respectively, as

$d_{i,j}=||f(x_{i})-f(x_{j})||^{2}\,.$ (4)

A logistic regression model estimates the probability $d_{i,j}$ of the edge
between $f(x_{i})$ and $f(x_{j})$ to be cut. This cut probability is then
converted into real valued edge costs $w$ using the _logit_ function
$\text{logit}(p)=\log\frac{p}{1-p}$ such that similar features are connected
by an edge with positive,  _i.e_. attractive weight and dissimilar features
are connected by edges with negative,  _i.e_. repulsive weight. The decision
boundary (_i.e_. the threshold on $d$, which indicates when to cut or to join)
is estimated empirically and we will provide the details in the supplementary
material.

Figure 3: Evaluation metrics with 4 clusters with 3 unique classes. Cluster
Accuracy: The best match for class dark circle is cluster 3, since it contains
the most frequent items from the same class. Cluster 4 is considered as false
positive. Purity score on the other hand does not penalize cluster 4. Thus,
the purity score is higher than the cluster accuracy (80% vs. 73%).

###  3.3 Cluster Quality Measures

We use two popular external evaluation metrics (_i.e_. label information are
used) to measure the clustering performance: Cluster Accuracy (ACC) and Purity
Score. The former metric is calculated based on the Hungarian algorithm [30],
where the best match between the predicted and the true labels are found. The
purity score assigns data in a cluster to the class with the most frequent
label [23]. Formally, given a set of $K$ clusters $S_{k}$ and a set of classes
$L$ with a total number of $N$ data samples, the purity is computed as
follows:

$\frac{1}{N}\sum_{k\in K}\max_{\ell\in L}|S_{k}\cap\ell|$ (5)

The advantage of using this metric is two-fold: on one hand, it is suitable if
the dataset is balanced and on the other hand, purity score does not penalize
having a large number of clusters. Figure 3 depicts an example of both
metrics.

###  3.4 Performance Measure

Next, we derive a measure based on the latent space clustering performance,
that allows to draw conclusions on a model’s robustness without evaluating the
model on corrupted data. Thereby, we measure a model’s robustness as its
relative classification accuracy, _i.e_. the ratio between its classification
accuracy on corrupted data and on clean data:

$Robustness=\frac{\mathrm{Model}_{\mathrm{ACC}_{s,c}^{*}}}{\mathrm{Model}_{\mathrm{ACC}}}$
(6)

Parameters $c$ and $s$ are corruption type and severity level (or intensity),
respectively for non-adversarial attacks such as ImageNet-C. The aggregated
value over all severity levels $s\in\tilde{S}$ on all corruption types $c\in
CORR$ is calculated as follows:

$\mathrm{ACC}_{all}^{*}=\frac{1}{|\mathrm{CORR}|}\sum_{c\in\mathrm{CORR}}\frac{1}{|\mathrm{\tilde{S}}|}\sum_{s=1}^{|\mathrm{\tilde{S}}|}\mathrm{ACC}_{s,c}^{*}$
(7)

According to equation 6, perfectly robust models therefore have a robustness
of 1, smaller values indicate lower robustness. Based on the above
considerations on model robustness and clustering performance, we propose to
consider the relative clustering performance as an indicator for the model
robustness and show empirically that there exists a strong correlation between
both. The relative clustering performance, _i.e_. the ratio between clustering
performance and classification accuracy $\mathrm{Model}_{\mathrm{ACC}}$ is
defined as follows:

$p=\frac{\mathrm{clustering~{}performance}}{\mathrm{Model}_{\mathrm{ACC}}}$
(8)

Here, we consider the clustering accuracy $\mathrm{C}_{\mathrm{ACC}}$ and
purity score $\mathrm{C}_{\mathrm{purity}}$ as a performance measures for our
experiments,  _i.e_.

$\displaystyle
p_{\mathrm{ACC}}=\frac{\mathrm{C}_{\mathrm{ACC}}}{\mathrm{Model}_{\mathrm{ACC}}}\quad\mathrm{and}\quad\displaystyle
p_{\mathrm{purity}}=\frac{\mathrm{C}_{\mathrm{purity}}}{\mathrm{Model}_{\mathrm{ACC}}}$

respectively.

Correlation Metrics. The degree of correlation is computed based on the
coefficient of determination $R^{2}$ and Kendall rank correlation coefficient
$\tau$, respectively with a value of $1.0$ being perfectly correlated while
$0$ means no correlation at all. An example for $R^{2}$ is illustrated in
Figure 1 and $\tau$ in Figure 7.

Baseline Indicator: Class Overlap $\Delta$. Our hypothesis is that an initial
well-separated feature space of a classification model provides a good
estimate regarding the model robustness. A simple method to determine such a
separation would be to observe the intra- and inter-class distances between
data samples in the feature space. If an overlap between classes exists, they
are not well separated, which may indicate weak models. We define this setting
as a baseline in order to show that latent space clustering provides
significantly more information.

To investigate this, we define the overlap $\Delta$ between the intra- and
inter-class distances as follows:

$\Delta=(\mu_{intra}+\sigma_{intra})-(\mu_{inter}+\sigma_{inter})$ (9)

$\mu$ and $\sigma$ represent the mean and standard deviation of the intra- and
inter-class distances.

##  4 Experiments

Figure 4: ImageNet-C dataset: first row shows the original image and the
corruption brightness for different severity levels. Second row: examples of
other corruption types at severity level 5.

This section is structured as follows: we first explain the setup of our
experiments in 4.1. Then, we present the clustering results in Section 4.2
where we analyse the clustering accuracy and purity for the two considered
clustering approaches on the feature spaces of the different models. Section
4.3 shows that the intra- and inter-class distances cannot directly be used as
robustness indicators. In Section 4.4, we consider the relationship between
the model classification robustness under corruptions and the relative
clustering performance of the considered clustering methods and metrics. We
show that both clustering accuracy and cluster purity, computed on the feature
spaces of clean data, allow to derive indicators for a model’s expected
robustness under corruptions. Thereby, the purity score is more stable than
the clustering accuracy and the information provided by k-means clustering and
multicuts complement one another. In Section 4.5, we evaluate the proposed
robustness indicator in the context of adversarial attacks.

###  4.1 Setup

Our experiments are based on the ImageNet [39] dataset. All models were pre-
trained on the original training dataset. We evaluate 10 CNN-based models and
2 transformer architectures deit-t and deit-s (t stands for tiny and s for
small). An overview is provided in Table 1. We evaluate the robustness of the
considered models against corruptions using the ImageNet-C [16] dataset and
report the model accuracy for classification and clustering tasks. Figure 4
illustrates an example of considered image corruptions: the first row shows
the different severity levels $s=1,\dots,5$ of the corruption brightness, with
1 being the lowest and 5 the strongest corruption. The second row shows other
kinds of image perturbations $c$ at severity level 5 such as fog, frost,
gaussian blur, jpeg_compression or pixelate. Each corruption $c$ has 5
severity levels $s=1,\dots,5$. All models are trained on the clean dataset and
the numbers are evaluated on the full test dataset, as done in [16].

###  4.2 Classification vs. Clustering

Table 2 summarizes the evaluation in three categories: classification, K-means
and Multicuts. There are in total $|\mathrm{CORR}|=19$ corruption types with
each $|\tilde{S}|=5$ severity levels on ImageNet-C. For the classification
task, the numbers are reported in top 1% accuracy for all five levels of
corruption (denoted as $1-5$). On K-means and Multicuts, we report the
clustering metrics as presented in 3.3.

The transformer deit-s shows the highest top 1% accuracy on the classification
task both on clean and on corrupted data for all severity levels.
Inceptionresnetv2 and polynet perform only slightly worse on clean data but
are more strongly affected by the ImageNet-C data corruptions than deit-s.
Alexnet shows the worse performance across all corruption levels. Although
resnet50 outperforms bninception, nasnetamobile and densenet121 it is less
robust against corruption. This is also illustrated in Figure 6 (right).

Considering the clustering accuracy and purity, the K-means and the Multicut
behave significantly different from one another. K-means clustering achieves
about 70% accuracy for models with the highest clean classification accuracy.
Yet, its accuracy is much better for the deit-t latent space than for example
for the densenet121 induced latent space, although the clean classification
accuracy of both networks is comparable. Overall, the K-means clustering works
surprisingly well on the transformer models. The Multicut clustering showed
the highest clustering accuracy on the inceptionresnetv2 model. The cluster
purity was comparably high for the best transformer model deit-s. Note that
our goal is to derive from the clustering performance an indicator for model
robustness, _i.e_. we expect clustering to be less accurate when models are
less robust to noise.

Table 2: Evaluation of robustness on classification and clustering tasks with the ImageNet C dataset, evaluated on corruption severity levels 1 to 5. Column CLEAN represents the classification performance of the models on the clean dataset. Columns 1-5 show the classification accuracy (top 1%) under different severity levels of over all 19 corruptions and column $\mathrm{ACC}^{*}_{all}$ shows the mean over corruptions on all 5 severity levels. On K-means, ACC and purity are clustering performance on clean test data. The numbers on multicuts are evaluated on a subset. The best score on each column is marked in bold. | Classifcation Accuray (top 1%) | K-means | Multicuts
---|---|---|---
Model | CLEAN | 1 | 2 | 3 | 4 | 5 | $\mathrm{ACC}^{*}_{all}$ | ACC | Purity | $\mathrm{ACC}^{*}_{all}$ | ACC | Purity | $\mathrm{ACC}^{*}_{all}$
alexnet | 56.4 | 35.9 | 25.4 | 18.9 | 12.7 | 8.0 | 20.2 | 14.6 | 18.4 | 8.0 | 8.0 | 28.1 | 2.6
vgg11 | 69.0 | 47.3 | 35.3 | 25.7 | 16.7 | 10.1 | 27.0 | 28.0 | 32.8 | 12.4 | 15.8 | 27.2 | 2.5
vgg16 | 71.6 | 50.9 | 38.6 | 28.5 | 18.7 | 11.4 | 29.6 | 32.3 | 37.5 | 14.4 | 19.3 | 27.6 | 2.8
bninception | 73.5 | 59.4 | 48.4 | 38.8 | 27.2 | 17.7 | 38.3 | 40.5 | 44.0 | 18.0 | 11.5 | 46.6 | 7.9
nasnetamobile | 74.1 | 60.7 | 51.3 | 43.7 | 33.6 | 22.5 | 42.4 | 41.0 | 45.3 | 23.6 | 41.9 | 70.2 | 19.3
densenet121 | 74.6 | 60.2 | 50.9 | 42.2 | 31.4 | 21.0 | 41.1 | 48.9 | 52.1 | 23.4 | 16.8 | 80.5 | 8.3
resnet50 | 76.0 | 60.1 | 49.8 | 40.1 | 28.9 | 18.8 | 39.6 | 55.8 | 58.6 | 24.9 | 29.3 | 64.3 | 11.1
resnet101 | 77.4 | 63.6 | 54.4 | 45.5 | 34.0 | 22.9 | 44.1 | 59.1 | 61.9 | 29.3 | 28.6 | 53.7 | 16.6
inceptionresnetv2 | 80.2 | 68.8 | 60.8 | 53.5 | 43.5 | 31.6 | 51.7 | 70.0 | 71.2 | 37.4 | 71.3 | 81.3 | 39.8
polynet | 81.0 | 68.0 | 58.9 | 49.9 | 38.2 | 26.3 | 48.3 | 67.8 | 69.7 | 34.8 | 54.4 | 76.6 | 24.1
deit-t | 74.5 | 63.3 | 55.9 | 48.7 | 38.9 | 28.1 | 47.0 | 57.4 | 60.0 | 31.7 | 33.0 | 91.9 | 19.5
deit-s | 81.2 | 72.1 | 66.1 | 60.2 | 51.3 | 39.7 | 57.9 | 68.8 | 70.8 | 43.4 | 49.4 | 81.1 | 29.6

###  4.3 Baseline Indicators: Intra- and Inter class-distances

Table 3 shows the correlation (as $R^{2}$) and the ranking correlation $\tau$
between the class overlap baseline indicator $\Delta$, which we detailed in
section 3.3, and the model robustness, grouped by severity level. We use
equation 6 to calculate the robustness for severity level $s$ over all
corruptions and compare them with $\Delta$. The last column shows the
correlation on all corruption levels. All 12 models are considered. The rank
correlation $\tau$ is calculated by comparing the model’s robustness rank and
the overlap $\Delta$ ranking. Initial well-separated feature spaces (thus a
low $\Delta$) should have a high correlation with their model’s robustness.
Despite its simplicity, this metric $\Delta$ correlates poorly with a highest
score of $R^{2}=0.29$ and $\tau=0.52$. This observation rejects the simple
hypothesis about the overlap of intra- and inter-class distances and it
suggests that using $\Delta$ is not sufficiently informative as an indicator
for model robustness.

Table 3: Baseline indicators for model robustness: The table shows the correlation between overlap $\Delta$ and model robustness for different corruption severity levels. Second row shows the rank correlation $\tau$ between the actual model robustness rank and the predicted rank using $\Delta$. | Severity
---|---
Metric | 1 | 2 | 3 | 4 | 5 | Total
$R^{2}$ | 0.27 | 0.29 | 0.27 | 0.25 | 0.26 | 0.27
$\tau$ | 0.48 | 0.52 | 0.52 | 0.52 | 0.52 | 0.48

###  4.4 Robustness Indicators: Clustering Measures

Figure 5: Visualization of feature space on alexnet and deit-s using umap. The
colors correspond to the class labels, where only 10 classes were selected at
random. First column shows the initial, clean latent space from the
classification model. Each new column depicts the corresponding severity level
of the corruption brightness. While alexnet collapses as the severity
increases, the most robust model deit-s preserves the clusters very well even
after significant corruptions and thus our proposed clusterability of latent
space provides a good indicator about the model robustness.

In the following we evaluate our proposed clustering driven robustness
indicator. Specifically, we want to investigate the effects of different
clustering measures on the correlation coefficient $R^{2}$. Table 4 gives an
overview of the strength of correlation on different severity levels and
clustering metrics on K-means and Multicuts. Column $\Delta$ shows the
correlation on robustness using the overlap of intra- and inter-class
distances as previously discussed. Furthermore, the columns ACC and P are
showing the correlation between the model robustness and the clustering
accuracy and purity, respectively. The last column shows the combination of
both clustering methods in one metric. K-means and Multicuts have an $R^{2}$
value of $R^{2}=0.83$ and $R^{2}=0.55$ for clustering accuracy on all
corruption levels. On the purity score, both methods show a slightly higher
correlation of $R^{2}=0.83$ and $R^{2}=0.71$, respectively for the sum over
all corruptions (last row of Table 4). This indicated that latent space
clusterablity of clean test images K-means is a valid indicator for model
robustness under corruptions. However, we show that both clustering methods
are complementary when combining their purity scores with

$p_{\mathrm{purity}^{\mathit{k-means}\cdot\mathit{multicuts}}}=\frac{\mathrm{C}_{\mathrm{purity}}^{\mathit{k-means}}\cdot\mathrm{C}_{\mathrm{purity}}^{\mathit{multicut}}}{\mathrm{Model}_{\mathrm{ACC}}}.$
(10)

This measure shows the highest correlation with the model robustness with
$R^{2}=0.87$ (see fig. 1 for the full correlation plot). Additionally, the
combination of purity scores of both methods also yields more consistent
results across different severity levels.

Table 4: Correlation with different metrics and severity levels: the reported numbers are the coefficient of determination ($R^{2}$) on different clustering metrics. Column $\Delta$ is the overlap (from Table 3). Column ACC and Purity (denoted as P.) are used to compute the correlation coefficient $R^{2}$. The last column is the combination of both clustering methods, _i.e_. last column $Purity$ is equation 10. The highest score is marked in bold. Metric: $R^{2}$ |  | K-means | Multicuts | Combined
---|---|---|---|---
Severity | $\Delta$ | ACC | P. | ACC | P. | ACC | P.
1 | 0.27 | 0.85 | 0.85 | 0.48 | 0.67 | 0.54 | 0.82
2 | 0.29 | 0.87 | 0.87 | 0.51 | 0.70 | 0.58 | 0.86
3 | 0.27 | 0.84 | 0.83 | 0.55 | 0.73 | 0.61 | 0.87
4 | 0.25 | 0.79 | 0.79 | 0.58 | 0.72 | 0.64 | 0.87
5 | 0.26 | 0.75 | 0.84 | 0.57 | 0.68 | 0.64 | 0.84
All | 0.27 | 0.83 | 0.83 | 0.55 | 0.71 | 0.62 | 0.87

Figure 6: Change in robustness ranking based on predicted (left) vs. actual
(right) model robustness on ImageNet-C using clustering metric
$p_{\mathrm{ACC^{K-means}}}$ on total corruptions $\tau=0.79$. Top is the
least robust model ($Rank=12$) while the bottom shows the most robust model
($Rank=1$). The highest score is marked in bold.

#### Model Ranking.

Next, we evaluate whether our proposed robustness indicator is able to
retrieve the correct ranking in terms of model robustness for our set of
classification models. The rank correlation is measured as the Kendall rank
coefficient $\tau$. Table 5 shows the results for different setups. Here,
K-means shows a more consistent and better correlation with highest rank
correlation of $\tau=0.82$ on ACC and Purity. Again, all clustering metrics
outperform the $\Delta$ baseline. Figure 6 illustrates one example of the
change of rank between the predicted (left) and actual (right) model
robustness. The prediction is done using
$p_{\mathrm{ACC}^{\mathit{K-means}}}$, which has a rank correlation of
$\tau=0.79$. Our proposed measure is able to rank different models according
to their robustness. The three worse performing models (alexnet, vgg11 and
vgg16) are correctly retrieved. The largest ranking gap of 3 positions is
observed for nasnetamobile and resnet50. In this particular example, the value
for alexnet is calculated as follows: $\frac{14.6}{56.4}*100=25.9$ and
$\frac{20.2}{56.4}*100=35.8$ for predicted and actual values, respectively.

Table 5: Rank correlation with different metrics and severity levels: the reported numbers are the coefficient of determination ($\tau$) on different clustering metrics. Column $\Delta$ is the overlap (from Table 3). Column ACC and Purity (denoted as P.) are the used compute the rank correlation coefficient $\tau$. The last column is the combination of both clustering methods, _i.e_. last column $Purity$ is equation 10. The highest score is marked in bold. Metric: $\tau$ |  | K-means | Multicuts | Combined
---|---|---|---|---
Severity | $\Delta$ | ACC | P. | ACC | P. | ACC | P.
1 | 0.48 | 0.79 | 0.79 | 0.61 | 0.52 | 0.73 | 0.73
2 | 0.52 | 0.82 | 0.82 | 0.64 | 0.55 | 0.76 | 0.76
3 | 0.52 | 0.82 | 0.82 | 0.70 | 0.61 | 0.82 | 0.76
4 | 0.52 | 0.82 | 0.82 | 0.70 | 0.61 | 0.82 | 0.76
5 | 0.52 | 0.82 | 0.82 | 0.70 | 0.61 | 0.82 | 0.76
All | 0.48 | 0.79 | 0.79 | 0.67 | 0.58 | 0.79 | 0.73

#### Latent Space Visualization.

Umap [34], a scalable dimensionality reduction method similar to the popular
technique TSNE[48], has been applied to features on 10 randomly selected
classes of the ImageNet dataset for visualization. Figure 5 shows one example
of the corruption brightness for 2 different models: the first column shows
features without any corruptions (clean). As the severity level increases, a
collapse is observed for instance on alexnet: well-separated clusters (_i.e_.
different colors) are being pulled into a direction in the latent space as the
severity increases. The model with the highest robustness, _i.e_. deit-s,
preserves the clusters well, which explains the high relative clustering
performance. This verifies our assumption on the correlation between
clusterability and robustness of classification models, that were evaluated in
ImageNet-C dataset.

Figure 7: Correlation of our proposed clustering metric on different
adversarial attacks with different strengths (Epsilon). Dashed line represents
the classification accuracy and solid line the coefficient of determination
$R^{2}$, respectively.

###  4.5 Adversarial Robustness

So far, we have shown that our proposed approach can effectively indicate the
robustness of classification models towards visible image corruptions and
shifts in the data distributions provided by the ImageNet-C benchmark. Here,
we extend this evaluation to intentional, non-visible corruptions induced by
adversarial attacks. Using the proposed clustering metric
$p_{\mathrm{purity}^{\mathit{k-means}\cdot\mathit{multicuts}}}$ as an
estimator, we evaluate all 12 models with ImageNet test dataset under two
adversarial attacks: DeepFool [35] and FGSM [14] with different perturbation
sizes $epsilon$. Figure 7 shows the results of both attacks. The solid lines
represent the correlation of determination $R^{2}$ while the dashed lines show
the average adversarial accuracy across all 12 models. Epsilon (x-axis) is the
perturbation size of the attacks. For small epsilon, we expect lower
correlations since the model accuracy should hardly be affected. As epsilon
increases, some models are more robust than others, _i.e_. better preserve
their classification accuracy. In this range, we see a relatively strong
correlation of the proposed indicator and the relative robust accuracy, albeit
weaker than the correlation with robustness to corruptions, with $R^{2}=0.66$
and $R^{2}=0.44$ for DeepFool and FGSM, respectively. When epsilon becomes too
large, the model accuracies eventually drop to very low values for all models,
_i.e_. the correlation becomes weaker.

##  5 Conclusion

In this work, we presented a study of the feature space of several pre-trained
models on ImageNet including state-of-the-art CNN models and the recently
proposed transformer models and we evaluated the robustness on ImageNet-C
dataset and extended our evaluation on adversarial robustness as well. We
propose a novel way to estimate the robustness behavior of trained models by
analyzing the learned feature-space structure. Specifically, we presented a
comprehensive study of two clustering methods, K-means and the Minimum Cost
Multicut Problem on ImageNet, where the classification accuracy,
clustarability and robustness are analyzed. We show that the relative
clustering performance gives a strong indication regarding the model’s
robustness. Both considered clustering methods show complementary behaviour in
our analysis: the coefficient of determination is $R^{2}=0.87$ when combining
the purity scores of both methods. Our experiments also show that this
indicator is lower, albeit still significant for adversarial robustness
($R^{2}=0.66$ and $R^{2}=0.44$). Additionally, our proposed method is able
estimate the order of robust models ($\tau=0.79$) on ImageNet-C. This novel
method is simple yet effective and allows the estimation of robustness of any
given classification model without explicitly testing on any specific test
data. To the best of our knowledge, we are the first to propose such technique
for estimating model robustness.

## References

  * [1] Nikhil Bansal, Avrim Blum, and Shuchi Chawla.  Correlation clustering.  Machine Learning, 56(1–3):89–113, 2004.
  * [2] Thorsten Beier, Thorben Kroeger, Jorg H Kappes, Ullrich Kothe, and Fred A Hamprecht.  Cut, glue & cut: A fast, approximate solver for multicut partitioning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 73–80, 2014.
  * [3] Battista Biggio and Fabio Roli.  Wild patterns: Ten years after the rise of adversarial machine learning.  Pattern Recognition, 84:317–331, 2018.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  arXiv preprint arXiv:2104.14294, 2021.
  * [6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In International Conference on Machine Learning, pages 1691–1703. PMLR, 2020.
  * [7] Sunil Chopra and M.R. Rao.  The partition problem.  Mathematical Programming, 59(1–3):87–115, 1993.
  * [8] Erik D. Demaine, Dotan Emanuel, Amos Fiat, and Nicole Immorlica.  Correlation clustering in general weighted graphs.  Theoretical Computer Science, 361(2–3):172–187, 2006.
  * [9] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al.  On robustness and transferability of convolutional neural networks.  arXiv preprint arXiv:2007.08558, 2020.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [11] Robert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann.  Generalisation in humans and deep neural networks.  arXiv preprint arXiv:1808.08750, 2018.
  * [12] Kamran Ghasedi, Xiaoqian Wang, Cheng Deng, and Heng Huang.  Balanced self-paced learning for generative adversarial clustering network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4391–4400, 2019.
  * [13] Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang.  Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization.  In Proceedings of the IEEE international conference on computer vision, pages 5736–5745, 2017.
  * [14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [16] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  arXiv preprint arXiv:1903.12261, 2019.
  * [17] Kalun Ho, Amirhossein Kardoost, Franz-Josef Pfreundt, Janis Keuper, and Margret Keuper.  A two-stage minimum cost multicut approach to self-supervised multiple person tracking.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [18] Kalun Ho, Janis Keuper, Franz-Josef Pfreundt, and Margret Keuper.  Learning embeddings for image clustering: An empirical study of triplet loss approaches.  arXiv preprint arXiv:2007.03123, 2020.
  * [19] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [20] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [22] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [23] Himanshi Jain, Reshu Grover, and Alwar LIET.  Clustering analysis with purity calculation of text and sql data using k-means clustering algorithm.  IJAPRR, 4(44557):47–58, 2017.
  * [24] Daniel Jakubovitz and Raja Giryes.  Improving dnn robustness to adversarial attacks using jacobian regularization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 514–529, 2018.
  * [25] Brian W Kernighan and Shen Lin.  An efficient heuristic procedure for partitioning graphs.  The Bell system technical journal, 49(2):291–307, 1970.
  * [26] Margret Keuper, Siyu Tang, Bjoern Andres, Thomas Brox, and Bernt Schiele.  Motion segmentation & multiple object tracking by correlation co-clustering.  IEEE transactions on pattern analysis and machine intelligence, 42(1):140–153, 2018.
  * [27] Simon Kornblith, Honglak Lee, Ting Chen, and Mohammad Norouzi.  What’s in a loss function for image classification?  arXiv preprint arXiv:2010.16402, 2020.
  * [28] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [29] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [30] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval Research Logistics (NRL), 52(1):7–21, 2005.
  * [31] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.  Adversarial machine learning at scale.  arXiv preprint arXiv:1611.01236, 2016.
  * [32] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [33] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In Proceedings of the European conference on computer vision (ECCV), pages 19–34, 2018.
  * [34] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [35] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  * [36] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  arXiv preprint arXiv:1906.02530, 2019.
  * [37] Vignesh Prasad, Dipanjan Das, and Brojeshwar Bhowmick.  Variational clustering: Leveraging variational autoencoders for image clustering.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE, 2020.
  * [38] Joaquin Quiñonero-Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer.  Dataset shift in machine learning.  Mit Press, 2009.
  * [39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [40] Tonmoy Saikia, Cordelia Schmid, and Thomas Brox.  Improving robustness against common corruptions with frequency biased models.  arXiv preprint arXiv:2103.16241, 2021.
  * [41] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  Cnn features off-the-shelf: an astounding baseline for recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–813, 2014.
  * [42] Hoo-Chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Ziyue Xu, Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers.  Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning.  IEEE transactions on medical imaging, 35(5):1285–1298, 2016.
  * [43] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [44] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [45] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [46] Siyu Tang, Mykhaylo Andriluka, Bjoern Andres, and Bernt Schiele.  Multiple people tracking by lifted multicut and person re-identification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3539–3548, 2017.
  * [47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  arXiv preprint arXiv:2012.12877, 2020.
  * [48] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [50] Steffen Wolf, Alberto Bailoni, Constantin Pape, Nasim Rahaman, Anna Kreshuk, Ullrich Köthe, and Fred A Hamprecht.  The mutex watershed and its objective: Efficient, parameter-free graph partitioning.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [51] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In International conference on machine learning, pages 478–487. PMLR, 2016.
  * [52] Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin.  Polynet: A pursuit of structural diversity in very deep networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 718–726, 2017.
  * [53] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018.