# Invariance of Polarization Induced by Symplectomorphisms

Ethan Ross

(2021)

##  1 Introduction

Geometric quantization roughly amounts to associating complex vector spaces
(preferably Hilbert spaces) to symplectic manifolds. The terminology arises
from the realization of a symplectic manifold as a classical phase space and a
Hilbert space as the space of quantum wave functions. One fruitful direction
in quantization has been the Kirillov-Kostant-Souriau picture in which one
associates to a given symplectic manifold $(M,\omega)$ a complex Hermitian
line bundle with compatible covariant derivative $(L,\nabla)\to M$ so that the
curvature $curv(\nabla)$ is given by the symplectic form $\omega$. The Hilbert
space of $L^{2}$ functions then has some great geometric properties, but it
can be “too large” in some sense. The standard first step in reducing this
Hilbert space is to introduce a possibly singular Lagrangian foliation
$P\subset TM$ on the symplectic manifold. From here, one has two choices for
what a quantization could be. First, via the covariant derivative $\nabla$ we
can define a sheaf of polarized sections $\mathscr{S}_{P}$ of $L$. We then
define the sheaf quantization

$Q_{shf}(M,P)=\bigoplus_{n}\check{H}^{n}(M,\mathscr{S}_{P}),$ (1)

where $\check{H}^{n}(M,\mathscr{S}_{P})$ are the Čech cohomology groups
associated to the sheaf. Second, we can use the fact that $P$ is a foliation
to define special leaves $\iota:B\to M$ called Bohr-Sommerfeld leaves which
admit non-trivial covariantly constant sections. That is sections $s$ of the
pullback bundle $(\iota^{*}B,\iota^{*}L)$ satisfying $\iota^{*}\nabla s=0$
along $TB$. Writing $BS(P)$ for the Bohr-Sommerfeld leaves of $P$, we then get
our second quantization, Bohr-Sommerfeld quantization

$Q_{BS}(M,P)=\bigoplus_{B\in BS(P)}\check{H}^{0}(B,\mathscr{S}_{P}|_{B}),$ (2)

where $\mathscr{S}_{P}|_{B}$ denotes the sheaf of covariantly constant
sections on $B$.

Sometimes these two quantizations agree as shown by Sniatycki[8] and
Hamilton[8], and other times they do not. Since both are of interest, I will
use both in this paper and be clear which one I am considering.

Now, a famous body of results in geometric quantization are the “invariance”
of polarization results, where it can be shown that two naturally arising
polarizations induce the same quantization. Fantastic examples include the
Gelfand-Zeitlin system [4] and the moduli space of flat $SU(2)$ connections
[6]. In this paper, I show a new kind of toy invariance theorem arising from
the action of a symplectomorphism on the polarizations. In particular, given a
polarization $P$ and a symplectomorphism $\Phi$ on $(M,\omega)$, I define a
new polarization $\Phi(P)$. It then appears to be a tautology that the
quantizations of $M$ with respect to $P$ and $\Phi(P)$ should agree, however
this only appears to be the case under special topological conditions on $M$.
In particular, I show the following two theorems.

######  Main Theorem 1.

Suppose that $\check{H}^{1}(M,\underline{\mathbb{C}}_{M}^{\times})=0$. Then,
$\Phi$ induces an isomorphism between the sheaf quantizations

$Q_{shf}(M,P)\to Q_{shf}(M,\Phi(P)),$

where $Q_{shf}(M,P)$ is defined in Equation 1.

######  Main Theorem 2.

Suppose that $\check{H}^{1}(M,\underline{\mathbb{C}}_{M}^{\times})=0$. Then,
$\Phi$ induces an isomorphism between the Bohr-Sommerfeld quantizations

$Q_{BS}(M,P)\to Q_{BS}(M,\Phi(P)),$

where $Q_{BS}(M,P)$ is defined in Equation 2

The upshot of the my proofs of these theorems is that the isomorphism can
explicitly be defined via local data arising from the line bundle. In
particular, the isomorphism is in terms of a choice of local trivialization of
the line bundle. Historically, most invariance results arise from counting the
dimensions of the quantizations with respect to two choices of polarization,
then showing these agree. So, even though the isomorphisms of Main Theorems 1
and 2 depend on choices, they could be considered more “canonical” in some
sense.

##  2 Quantization

Let $(M,\omega)$ be a symplectic manifold.

######  Definition 2.1.

A prequantum line bundle over $M$ is a complex Hermitian line bundle with
compatible connection $(L,\nabla)\to M$ such that

$curv(\nabla)=\omega.$

A quantization should naively attach to a symplectic manifold (thought of as a
classical phase space) a Hilbert space and a map taking the classical
observables $C^{\infty}(M)$ to self-adjoint operators on the Hilbert space.
Prequantum line bundles achieve both of these objectives. The Hermitian
structure of the complex line bundle together with the canonical volume form
$\omega^{n}/n!$ allows us to equip the space $\Gamma_{0}(L)$ of compactly
supported sections with the structure of a pre-Hilbert space. Then, using the
compatible covariant derivative $\nabla$, we can construct the desired map
between classical and quantum observables. The only problem with the
construction and the reason why these are called “prequantum” and not
“quantum” is due to the fact that even in the simplest cases, the resulting
Hilbert space is too large in some sense. The idea of a polarization is then
introduced to cut this Hilbert space down in size.

######  Definition 2.2.

A (singular, real) polarization of $(M,\omega)$ is a smooth singular subbundle
$P\subset TM$ such that its sheaf of sections $\mathfrak{F}_{P}$ satisfies the
following axioms.

  * (i)

(Involutivity) If $X,Y\in\mathfrak{F}_{P}$, then so is $[X,Y]$.

  * (ii)

(Locally Finitely Generated) For any $x\in M$, there exists an open
neighbourhood $U\subset M$ of $x$ and sections
$X_{1},\dots,X_{k}\in\mathfrak{F}_{P}(U)$ such that

$\mathfrak{F}_{P}(U)=\text{span}_{C^{\infty}(U)}\\{X_{1},\dots,X_{k}\\}.$

  * (iii)

(Lagrangian) $P_{x}:=P\cap T_{x}M$ is generically Lagrangian. That is, due to
(i) and (ii) there exists a dense open subset $U\subset M$ so that $P|_{U}\to
U$ is a smooth distribution[1]. We demand $P_{x}$ is Lagrangian for any $x\in
U$.

Since Lagrangian subspaces have half the dimension of the symplectic manifold,
demanding sections be covariantly constant along the directions associated to
$P$ is our mechanism for cutting down the variables which the Hilbert space
depends. That is, we define the sheaf of covariantly constant sections
$\mathscr{S}_{P}$ by

$\mathscr{S}_{P}(U):=\\{s\in\Gamma(U,L)\ |\ \nabla_{X}s=0\ \forall\
X\in\mathfrak{F}_{P}(U)\\},$ (3)

for open $U\subset M$.

This enables us to formally define the sheaf quantization for the purposes of
this paper.

######  Definition 2.3.

Define the sheaf quantization of $(M,\omega)$ with respect to the prequantum
line bundle $(L,\nabla)\to M$ and the polarization $P\subset TM$ by

$Q_{shf}(M,P):=\bigoplus_{n}\check{H}^{n}(M,\mathscr{S}_{P}),$ (4)

where $\check{H}^{n}(M,\mathscr{S}_{P})$ denotes the $n$-th Ĉech cohomology
group of $M$ with respect to the sheaf $\mathscr{S}_{P}$.

Another common way of quantizing a symplectic manifold is by a Bohr-Sommerfeld
quantization. To do this, observe that if $P\subset TM$ is a polarization,
then by a Theorem of Sussman[9], $P$ induces a decomposition of $M$ into
disjoint immersed submanifolds called leaves. The leaves of maximal dimension
are immersed Lagrangian submanifolds.

Now, if $\iota:B\to M$ is a leaf of $P$, we can pullback the covariant
derivative $\nabla$ on $L$ to the pull-back bundle $\iota^{*}L\to B$ and write
$\iota^{*}\nabla$ for this pull-back covariant derivative. Thus, we can define
a sheaf $\mathscr{S}_{P}|_{B}$ on $B$ by

$\mathscr{S}_{P}|_{B}(U)=\\{s\in\Gamma(U,\iota^{*}L)\ |\ \nabla_{X}s=0\
\forall X\in\mathfrak{X}(U)\\},$ (5)

for open $U\subset B$. With this set up, we can define the Bohr-Sommerfeld
quantization.

######  Definition 2.4.

A leaf $B$ of $P$ is called Bohr-Sommerfeld if

$H^{0}(B,\mathscr{S}_{P}|_{B})\neq 0.$

Writing $BS(P)$ for the set of Bohr-Sommerfeld leaves of $P$, we define the
Bohr-Sommerfeld quantization of $(M,\omega)$ with respect to the prequantum
line bundle $(L,\nabla)\to M$ and the polarization $P$ by

$Q_{BS}(M,P):=\bigoplus_{B\in BS(P)}H^{0}(B,\mathscr{S}_{P}|_{B}).$ (6)

######  Remark 1

  * (i)

An equivalent definition of a Bohr-Sommerfeld leaf used in the literature is
as follows. A Bohr-Sommerfeld leaf is a leaf $\iota:B\to M$ that admits a non-
trivial covariantly constant section $s:B\to\iota^{*}B$, i.e. a covariantly
constant section which is not identically the zero section.

  * (ii)

If $B$ is a Bohr-Sommerfeld leaf and is connected, then it follows that

$H^{0}(B,\mathscr{S}_{P}|_{B})\cong\mathbb{C}.$

##  3 Trivialization Covers and Cohomology

######  Definition 3.1.

A trivialization cover for the line bundle $L\to M$ is a pair
$(\mathscr{U},s)$, where

  * (i)

$\mathscr{U}=\\{U_{\alpha}\\}_{\alpha\in J}$ is an open cover of $M$.

  * (ii)

$s=\\{s_{\alpha}\in\Gamma(U_{\alpha},L)\\}_{\alpha\in J}$ is a collection of
non-vanishing local sections of $L$.

######  Remark 2

A trivialization cover $(\mathscr{U},s)$ gives us useful local data.

  * •

(Transition functions) If $U_{\alpha\beta}=U_{\alpha}\cap U_{\beta}$ is non-
empty, then there exists a unique non-vanishing function
$\lambda_{\alpha\beta}:U_{\alpha\beta}\to\mathbb{C}^{\times}$ defined by

$s_{\alpha}|_{U_{\alpha\beta}}=\lambda_{\alpha\beta}s_{\beta}|_{U_{\alpha\beta}}.$

By construction, these transition functions satisfy the cocycle condition

$\lambda_{\alpha\beta}\lambda_{\beta\gamma}|_{U_{\alpha\beta\gamma}}=\lambda_{\alpha\gamma}|_{U_{\alpha\beta\gamma}}$

on triple overlaps. Thus, they define a Čech cocycle

$\lambda\in\check{Z}^{1}(\mathscr{U},\mathscr{O}^{\times}).$

  * •

(Local primitives of $\omega$) On $U_{\alpha}$ there exists a unique complex
one-form $\theta_{\alpha}^{s}\in\Omega^{1}_{\mathbb{C}}(U_{\alpha})$ defined
by

$\theta_{\alpha}(X)s_{\alpha}=-i\nabla_{X}s_{\alpha},$

for all $X\in\mathfrak{X}(U_{\alpha})$. These $\theta_{\alpha}$ satisfy

$d\theta_{\alpha}=\omega|_{U_{\alpha}}.$

Thus, to each trivialization cover $(\mathscr{U},s)$ we get a pair

$(\lambda,\theta)\in\check{Z}^{1}(\mathscr{U},\mathscr{O}^{\times})\times\check{C}^{0}(\mathscr{U},\Omega^{1}_{\mathbb{C}}).$

We will call this pair the local data associated to $(\mathscr{U},s)$.

We can use trivialization covers to locally represent sections as functions.
This is needed since a symplectomorphism naturally acts on functions via
precomposition. To get the idea, fix a trivialization cover
$(\mathscr{U},s)=(\\{U_{\alpha}\\}_{\alpha\in J},\\{s_{\alpha}\\}_{\alpha\in
J})$ and choose $\alpha\in J$. Then, for each open $U\subset U_{\alpha}$ and
each $s\in\mathscr{S}_{P}(U)$ there exists a unique $f\in
C^{\infty}(U,\mathbb{C})$ such that

$fs_{\alpha}|_{U}=s.$

Write $s/s_{\alpha}:=f$. Furthermore, observe that for any
$X\in\mathfrak{F}_{P}(U_{\alpha})$, we have

$X\frac{s}{s_{\alpha}}+i\theta_{\alpha}(X)\frac{s}{s_{\alpha}}=0.$ (7)

Thus, for each open $U\subset U_{\alpha}$, define

$C^{\infty}_{P,\alpha}(U):=\\{f\in C^{\infty}(U)\ |\ f\text{ satisfies
Equation (\ref{polarizedfunction})}\\}.$ (8)

This defines a sheaf on $U_{\alpha}$ for each $\alpha$. Further, we obtain a
recipe for giving a functions representation of
$s\in\check{C}^{n}(\mathscr{U},\mathscr{S}_{P})$ as follows: For each
$\alpha_{0},\dots,\alpha_{n}\in J$ such that
$U_{\alpha_{0},\dots,\alpha_{n}}\neq\emptyset$, write

$\displaystyle\psi^{P}_{\alpha_{0},\dots,\alpha_{n}}:\mathscr{S}_{P}(U_{\alpha_{0},\dots,\alpha_{n}})$
$\displaystyle\to\prod_{j=0}^{n}C^{\infty}_{P,\alpha_{j}}(U_{\alpha_{0},\dots,\alpha_{n}})$
(9) $\displaystyle s$
$\displaystyle\mapsto\big{(}\frac{s}{s_{\alpha_{0}}},\dots,\frac{s}{s_{\alpha_{n}}}\big{)}.$
(10)

This map is injective with left inverse

$\displaystyle\prod_{j=0}^{n}C^{\infty}_{P,\alpha_{j}}(U_{\alpha_{0},\dots,\alpha_{n}})$
$\displaystyle\to\mathscr{S}_{P}(U_{\alpha_{0},\dots,\alpha_{n}})$
$\displaystyle(f^{0},\dots,f^{n})$
$\displaystyle\mapsto\frac{1}{n+1}\sum_{j=0}^{n}f^{j}s_{\alpha_{j}}.$

######  Remark 3

Using the above, it is easy to see that
$(f^{0},\dots,f^{n})\in\text{image}(\psi^{P}_{\alpha_{0},\dots,\alpha_{n}})$
if and only if for each $j=0,\dots,n$

$f^{j}=\frac{1}{n+1}\sum_{k=0}^{n}\lambda_{\alpha_{k}\alpha_{j}}f^{k}.$ (11)

######  Definition 3.2.

Define the trivialization complex
$(C^{\bullet}_{triv}(\mathscr{U},s,P),\delta)$ by

$C^{n}_{triv}(\mathscr{U},s,P):=\prod_{\begin{subarray}{c}\alpha_{0},\dots,\alpha_{n}\in
J\\\
U_{\alpha_{0},\dots,\alpha_{n}}\neq\emptyset\end{subarray}}\text{image}(\psi^{P}_{\alpha_{0},\dots,\alpha_{n}})$
(12)

with differential $\delta$ defined as follows. For
$\alpha_{0},\dots,\alpha_{n+1}\in J$ and

$f=\\{f_{\beta_{0},\dots,\beta_{n}}\\}\in C^{n}_{triv}(\mathscr{U},s,P)$

we define $\delta f\in C^{n+1}_{triv}(\mathscr{U},s,P)$ by

$(\delta f)_{\alpha_{0},\dots,\alpha_{n+1}}=\big{(}(\delta
f)_{\alpha_{0},\dots,\alpha_{n+1}}^{0},\dots,(\delta
f)_{\alpha_{0},\dots,\alpha_{n+1}}^{n+1}\big{)},$ (13)

where for $j=0,\dots,n$ we have

$(\delta
f)_{\alpha_{0},\dots,\alpha_{n+1}}^{j}=\frac{1}{n+1}\sum_{k=0}^{n+1}\sum_{\ell\neq
j}(-1)^{k}\lambda_{\alpha_{\ell}\alpha_{j}}f_{\alpha_{0},\dots,\hat{\alpha_{j}},\dots,\alpha_{n+1}}^{\ell}.$
(14)

######  Remark 4

We can make the above definition of $\delta$ a little mysterious as follows.
If $\mathscr{U}=\\{U_{\alpha}\\}_{\alpha\in J}$, let

$\displaystyle A$ $\displaystyle=(\alpha_{0},\dots,\alpha_{n})\in
J^{n+1}\text{ and}$ $\displaystyle B$
$\displaystyle=(\beta_{0},\dots,\beta_{n})\in J^{m+1}$

be an $n+1$ and an $m+1$ tuple of indices of indices, respectively. Write
$A\leq B$ if

  * (i)

$n\leq m$.

  * (ii)

There exists a permutation $\sigma$ on $\\{0,\dots,m\\}$ so that

$\beta_{\sigma(j)}=\alpha_{j}$

for all $0\leq j\leq n$.

With this, we can define a map called the restriction map

$\displaystyle\text{res}_{B,A}:\prod_{j=0}^{n}C_{P,\alpha_{j}}(U_{\alpha_{0},\cdots,\alpha_{n}})$
$\displaystyle\to\prod_{j=0}^{m}C_{P,\beta_{j}}(U_{\beta_{0},\cdots,\beta_{n}});$
$\displaystyle f=(f_{0},\dots,f_{n})$
$\displaystyle\mapsto(\text{res}_{B,A}(f)^{0},\dots,\text{res}_{B,A}(f)^{m}),$

where for any $0\leq j\leq m$

$\text{res}_{B,A}(f)^{j}:=\frac{1}{n+1}\sum_{k=0}^{n}\lambda_{\alpha_{k},\beta_{j}}f^{k}.$
(15)

These restriction maps behave like the restriction homomorphisms of a
presheaf. Namely, we have the following.

  * (i)

If $A,B,C$ are tuples of indices satisfying $A\leq B\leq C$, then

$\text{res}_{C,B}\circ\text{res}_{B,A}=\text{res}_{C,A}.$

  * (ii)

For any $(n+1)$-tuple $A=(\alpha_{0},\dots,\alpha_{n})\in I^{n+1}$ of indices,
we have on $\text{image}(\psi^{P}_{\alpha_{0},\dots,\alpha_{n}})$

$\text{res}_{A,A}=\text{Id}.$

(ii) is an easy application of Equation (11).

Now, if $f\in C^{n}_{triv}(\mathscr{U},s,P)$ and
$B=(\beta_{0},\dots,\beta_{n})$, write

$f_{B}=f_{\beta_{0},\dots,\beta_{n}}.$

Thus, for any $(n+2)$-tuple $A=(\alpha_{0},\dots,\alpha_{n+1})$ and the same
$f$, we can define $(\delta f)_{A}$ to be the $(n+2)$-tuple defined by

$(\delta f)_{A}=\sum_{j=0}^{n+1}(-1)^{j}\text{res}_{A,A_{j}}(f_{A_{j}}),$ (16)

where $A_{j}=(\alpha_{0},\dots,\hat{\alpha_{j}},\dots,\alpha_{n+1})$. This
mirrors the usual definition of the differential in Čech cohomology, except
the restriction homomorphism of a presheaf has been replaced by this family of
maps $\text{res}_{A,A_{j}}$.

######  Proposition 3.1.

We have the following

  * (i)

$\delta^{2}=0$.

  * (ii)

The obvious maps from Equation (9)

$\psi^{P}_{\alpha_{0},\dots,\alpha_{n}}:\mathscr{S}_{P}(U_{\alpha_{0},\dots,\alpha_{n}})\to\text{image}(\psi^{P}_{\alpha_{0},\dots,\alpha_{n}})$

defines an isomorphism of cochain complexes

$\check{C}^{\bullet}(\mathscr{U},\mathscr{S}_{P})\to
C_{triv}^{\bullet}(\mathscr{U},s,P).$ (17)

###### Proof.

Using the res notation introduced in Equation (15) makes this quite
straightforward. Indeed, let $A=(\alpha_{0},\dots,\alpha_{n+2})$ be a
$(n+3)$-tuple of indices. We have

$(\delta^{2}f)_{A}=\sum_{j=0}^{n+2}(-1)^{j}\text{res}_{A,A_{j}}((\delta
f)_{A_{j}}).$

Write $A_{j}=(\beta_{0}^{j},\dots,\beta_{n+1}^{j})$ and for each $0\leq k\leq
n+1$, write

$A_{jk}:=(\beta_{0}^{j},\dots,\hat{\beta_{k}^{j}},\dots,\beta^{j}_{n+1}).$

Then, we have

$(\delta
f)_{A_{j}}=\sum_{k=0}^{n+1}(-1)^{k}\text{res}_{A_{j},A_{jk}}(f_{A_{jk}}).$

Therefore, using the earlier remark, we obtain

$(\delta^{2}f)_{A}=\sum_{j=0}^{n+2}\sum_{k=0}^{n+1}(-1)^{j+k}\text{res}_{A,A_{jk}}(f_{A_{jk}}).$

Now, for $0\leq j\neq k\leq n+2$, set

$C_{j,k}:=(\alpha_{0},\dots,\hat{\alpha_{j}},\dots,\hat{\alpha_{k}},\dots,\alpha_{n+2}).$

Now, for any $0\leq j\leq n+2$ and any $0\leq k\leq n+1$, we observe that

$A_{jk}=\begin{cases}C_{j,k},\quad&k<j\\\ C_{j,k+1},\quad&k\geq j\end{cases}$

Thus, we can write

$\displaystyle(\delta^{2}f)_{A}$
$\displaystyle=\sum_{j=0}^{n+2}(-1)^{j}\bigg{[}\sum_{k<j}(-1)^{k}\text{res}_{A,C_{j,k}}(f_{C_{j,k}})+\sum_{k\geq
j}(-1)^{k}\text{res}_{A,C_{j,k+1}}(f_{C_{j,k+1}})\bigg{]}$
$\displaystyle=\sum_{j,k=1}^{n+1}(-1)^{j+k}\Omega_{j,k},$

where

$\Omega_{j,k}:=\begin{cases}\text{res}_{A,C_{j,k}}(f_{C_{j,k}}),\quad&k<j\\\
-\text{res}_{A,C_{j,k}}(f_{C_{j,k}}),\quad&k>j\\\ 0,\quad&k=j\end{cases}$

In particular, $\Omega_{j,k}=-\Omega_{k,j}$ for any choice of $j,k$. Thus,

$\sum_{j,k=1}^{n+2}(-1)^{j+k}\Omega_{j,k}=0.$

Therefore, $(\delta^{2}f)_{A}=0$.

Fix a non-negative integer $n\geq 0$. We define two maps. First

$\psi^{n}:\check{C}^{n}(\mathscr{U},\mathscr{S}_{P})\to
C^{n}_{triv}(\mathscr{U},s,P),$

where for any $s\in\check{C}^{n}(\mathscr{U},\mathscr{S}_{P})$, any indices
$\alpha_{0},\dots,\alpha_{n}$, and any $0\leq j\leq n$, define

$\psi^{n}(s)_{\alpha_{0},\dots,\alpha_{n}}^{j}:=\psi^{P}_{\alpha_{0},\dots,\alpha_{n}}(s_{\alpha_{0},\dots,\alpha_{n}})^{j}=\frac{s}{s_{\alpha_{j}}}.$

This well-defined by definition.

Going the other way, define

$\phi^{n}:C^{n}_{triv}(\mathscr{U},s,P)\to\check{C}^{n}(\mathscr{U},\mathscr{S}_{P})$

by

$\psi^{n}(f)_{\alpha_{0},\dots,\alpha_{n}}:=\frac{1}{n+1}\sum_{j=0}^{n}f^{j}_{\alpha_{0},\dots,\alpha_{n}}s_{\alpha_{j}}.$

This also is clearly well-defined. Showing $\psi^{n}$ and $\phi^{n}$ are
inverses amounts to Equation (11). The collection of maps $\psi^{\bullet}$
becomes a chain map almost by definition and $\phi^{\bullet}$ being a chain
map amounts to the identity

$\lambda_{\alpha,\beta}s_{\beta}=s_{\alpha}.$

∎

As is well known, Čech cohomology is highly dependent on the choice of an open
cover. The usual way to make it an invariant is to introduce refinement then
take a direct limit. See Bott and Tu [2] for the details. As such, if we want
to show two cohomologies with respect to different sheaves are isomorphic, one
technique is to find an isomorphism for a suitable class of open covers, then
show that these isomorphisms behave well under refinement. We will need a
similar device for our cohomology.

######  Definition 3.3.

Let $(\mathscr{U},s)$ and $(\mathscr{V},t)$ be two trivializing covers of
$L\to M$. We say $(\mathscr{V},t)$ is a refinement of $(\mathscr{U},s)$ and
write

$(\mathscr{V},t)\leq(\mathscr{U},s)$ (18)

if $\mathscr{V}$ is a refinement of $\mathscr{U}$.

This imports the idea of refinement directly from open covers. Thus,
trivializing covers with refinement naturally form a directed category and via
the natural isomorphisms in Proposition 3.1 it follows that if
$(\mathscr{V},t)$ is a refinement of $(\mathscr{U},s)$, then there is a unique
morphism

$H^{\bullet}_{triv}(\mathscr{U},s,P)\to H^{\bullet}_{triv}(\mathscr{V},t,P)$

allowing us to define the trivialization cohomology of the polarized line
bundle $L\to M$ with $P$.

######  Definition 3.4.

Give a prequantum line bundle $(L,\nabla)\to M$ and a polarization $P\subset
TM$, define the trivialization cohomology of the triple $(M,L,P)$ to be the
direct limit

$H^{\bullet}_{triv}(M,L,P):=\lim_{(\mathscr{U},s)}H^{\bullet}_{triv}(\mathscr{U},s,P),$
(19)

where the direct limit is taken over the directed set of trivialization cover
$(\mathscr{U},s)$ of the line bundle $L\to M$.

######  Lemma 3.1.

Via the natural isomorphisms in Proposition 3.1, we have a canonical
isomorphism

$\check{H}^{\bullet}(M,\mathscr{S}_{P})\cong H^{\bullet}_{triv}(M,L,P).$ (20)

##  4 Action by Symplectomorphisms on Polarizations

###  4.1 Effect of Čech Cohomology

Now, given a symplectomorphism $\Phi:M\to M$ and a polarization $P\subset TM$,
we can define a new polarization $\Phi(P)\subset TM$ by the pushforward by
$\Phi$:

$\Phi(P)_{x}:=\Phi^{-1}_{*}(P_{\Phi(x)}).$ (21)

Since $\Phi$ is a diffeomorphism it easily follows that $\Phi(P)$ is an
involutive, locally finitely generated, smooth singular subbundle of $TM$.
Further, $\Phi$ being a symplectomorphism gives us that $\Phi(P)$ is
generically Lagrangian.

######  Definition 4.1.

Given a trivialization cover $(\mathscr{U},s)$ of $L\to M$ and a
symplectomorphism $\Phi:M\to M$, say a trivialization cover $(\mathscr{V},t)$
is complementary to $(\mathscr{U},s)$ with respect to $\Phi$ if

  * (i)

$\mathscr{V}=\Phi^{*}\mathscr{U}$, where

$\Phi^{*}\mathscr{U}=\\{\Phi^{-1}(U)\ |\ U\in\mathscr{U}\\}$ (22)

  * (ii)

If $(\lambda,\theta)$ and $(\lambda^{\prime},\theta^{\prime})$ denote the
local data associated to $s$ and $t$, respectively, then we have

$\lambda^{\prime}=\Phi^{*}\lambda\quad\theta^{\prime}=\Phi^{*}\theta.$ (23)

######  Lemma 4.1.

Fix a symplectomorphism $\Phi$. If
$\check{H}^{1}(M,\underline{\mathbb{C}}_{M}^{\times})=0$, then any
trivialization cover $(\mathscr{U},s)$ with $\mathscr{U}$ good admits a
complementary pair.

###### Proof.

Observe that since each element of $\mathscr{U}$ is contractible, it follows
that the elements of $\Phi^{*}\mathscr{U}$ admit non-vanishing sections.
Hence, there exists $t^{\prime}\in\check{C}^{0}(\Phi^{*}\mathscr{U},L)$ such
that $(\Phi^{*}\mathscr{U},t^{\prime})$ is a trivialization cover of $L\to M$.
Now we just have to massage the local data induced by
$(\Phi^{*}\mathscr{U},t^{\prime})$ .

Write $\mathscr{U}=\\{U_{\alpha}\\}_{\alpha\in J}$ and fix $\alpha\in J$.
Since $U_{\alpha}$ is contractible, there exists
$f_{\alpha}\in\mathscr{O}(\Phi^{-1}(U_{\alpha}))$ such that if
$\theta^{\prime\prime}$ is the associated local trivialization, then

$\theta^{\prime\prime}_{\alpha}-\Phi^{*}\theta_{\alpha}=df_{\alpha}.$

Defining $t_{\alpha}=e^{-if_{\alpha}}t_{\alpha}^{\prime}$, it follows that
$t_{\alpha}$ is non-vanishing and for any
$X\in\mathfrak{X}(\Phi^{-1}(U_{\alpha}))$

$\Phi^{*}\theta_{\alpha}(X)t_{\alpha}=-i\nabla_{X}t_{\alpha}.$

Now, for the transition functions. It is a simple matter of moving definitions
around to show for any $\alpha,\beta\in J$ with $U_{\alpha\beta}\neq\emptyset$
that the equation

$\theta_{\alpha}-\theta_{\beta}=-id\log\lambda_{\alpha\beta}$

holds (see Kostant[7] for the computation) for any choice of branch of
logarithm. Pulling back by $\Phi$, we see then

$-id\log\Phi^{*}\lambda_{\alpha\beta}=-id\log\lambda^{\prime\prime}_{\alpha\beta},$

where $\lambda_{\alpha\beta}^{\prime\prime}$ are the associated transitions
functions to the $t_{\alpha}$.

Hence, there exists $e_{\alpha\beta}\in\mathbb{C}^{\times}$ so that

$\lambda^{\prime\prime}_{\alpha\beta}=e_{\alpha\beta}\Phi^{*}\lambda_{\alpha\beta}.$

Since $\lambda^{\prime\prime}$ and $\lambda$ satisfy the cocycle conditons, it
follows that so does

$e=\\{e_{\alpha\beta}\\}.$

Since $\check{H}^{1}(\mathscr{U},\underline{\mathbb{C}}_{M}^{\times})=0$,
there exists
$w=\\{w_{\alpha}\\}\in\check{C}^{0}(\mathscr{U},\underline{\mathbb{C}}_{M}^{\times})$
such that

$w_{\beta}w_{\alpha}^{-1}=e_{\alpha\beta}.$

Redefining $t_{\alpha}=w_{\alpha}t_{\alpha}$ obtain a collection
$t\in\check{C}^{0}(\Phi^{*}\mathscr{U},L)$ making $(\Phi^{*}\mathscr{U},t)$
into a trivialization cover and so that the local data
$(\lambda^{\prime},\theta^{\prime})$ satisfy the desired relations. ∎

######  Lemma 4.2.

Given a trivialization cover $(\mathscr{U},s)$, a symplectomorphism $\Phi:M\to
M$, and a complementary pair $(\Phi^{*}\mathscr{U},t)$, then $\Phi$ induces an
isomorphism of cochain complexes

$\Phi^{\bullet}:C_{triv}^{\bullet}(\mathscr{U},s,P)\to
C_{triv}^{\bullet}(\Phi^{*}\mathscr{U},t,\Phi(P))$

###### Proof.

Let $U\subset U_{\alpha}$ be open and let $f\in C^{\infty}(U,\mathbb{C})$
satisfy

$Xf+i\theta_{\alpha}(X)f=0$

for all $X\in\mathfrak{F}_{P}(U)$. Let
$Y\in\mathfrak{F}_{\Phi(P)}(\Phi^{-1}(U))$. Then it follows by definition of
$\Phi(P)$ that

$Y(f\circ\Phi)+i\Phi^{*}\theta_{\alpha}(Y)(f\circ\Phi)=0.$

Hence, for each $\alpha_{0},\dots,\alpha_{n}\in J$ such that
$U_{\alpha_{0},\dots,\alpha_{n}}\neq\emptyset$ we have a well-defined map

$\displaystyle\Phi_{\alpha_{0},\dots,\alpha_{n}}:\prod_{j=0}^{n}C^{\infty}_{P,\alpha_{j}}(U_{\alpha_{0},\dots,\alpha_{n}})$
$\displaystyle\to\prod_{j=0}^{n}C^{\infty}_{\Phi(P),\alpha_{j}}(\Phi^{-1}(U_{\alpha_{0},\dots,\alpha_{n}}))$
$\displaystyle(f^{0},\dots,f^{n})$
$\displaystyle\mapsto(f^{0}\circ\Phi,\dots,f^{n}\circ\Phi).$

We now just need to check that for each $j=0,\dots,n$ that if

$(f^{0},\dots,f^{n})\in\text{image}(\psi^{P}_{\alpha_{0},\dots,\alpha_{n}}),$

then

$(f^{0}\circ\Phi,\dots,f^{n}\circ\Phi)\in\text{image}(\psi^{\Phi(P)}_{\alpha_{0},\dots,\alpha_{n}}).$

This is straightforward using the characterization that

$f^{j}=\frac{1}{n+1}\sum_{k=0}^{n}\lambda_{\alpha_{k}\alpha_{j}}f^{k}$

for all $j=0,\dots,n$. Applying $\Phi$ to both sides and using the fact that
the transition functions for $(\Phi^{*}\mathscr{U},t)$ are given by
$\lambda_{\alpha_{k}\alpha_{j}}\circ\Phi$, we obtain the desired result.

To finish off, one then checks that the maps

$\displaystyle\Phi^{n}:C_{triv}^{n}(\mathscr{U},s,P)$ $\displaystyle\to
C_{triv}^{n}(\Phi^{*}\mathscr{U},t,\Phi(P));$ $\displaystyle
f=\\{f_{\alpha_{0},\dots,\alpha_{n}}\\}$
$\displaystyle\mapsto\Phi^{n}(f)=\\{\Phi_{\alpha_{0},\dots,\alpha_{n}}(f_{\alpha_{0},\dots,\alpha_{n}})\\}$

define a cochain map for any non-negative integer. Applying the exact same
argument to $\Phi^{-1}$ allows us to then construct an inverse. ∎

Fix a trivializing cover $(\mathscr{U},s)$ and write
$\mathscr{U}=\\{U_{\alpha}\\}_{\alpha\in I}$ and let $(\Phi^{*}\mathscr{U},t)$
be a complementary cover along $\Phi$. The map

$\Phi^{\bullet}:C^{\bullet}_{triv}(\mathscr{U},s,P)\to
C^{\bullet}_{triv}(\Phi^{*}\mathscr{U},t,\Phi(P))$

constructed in Lemma 4.2 does not obviously behaves well with respect to
refinement as we have defined it. And so, we will now introduce a restricted
notion of refinement.

######  Definition 4.2.

Say a trivializing cover $(\mathscr{V},q)=(\\{V_{\beta}\\}_{\beta\in
J},\\{q_{\beta}\\}_{\beta\in J})$ is a restricted refinement of
$(\mathscr{U},s)$ and write

$(\mathscr{V},q)\leq^{r}(\mathscr{U},s)$ (24)

if there exists a refinement map $\phi:J\to I$ such that for all $\beta\in J$
we have

$q_{\beta}=s_{\phi(\beta)}|_{V_{\beta}}.$

######  Proposition 4.1.

Let $(\mathscr{U},s)$ be a trivializing cover and fix a complementary pair
$(\Phi^{*}\mathscr{U},t)$.

  * (i)

The collection of trivializing covers of $L\to M$ which are restricted
refinements of $(\mathscr{U},s)$ is cofinal in the amongst trivializing
covers. That is if $(\mathscr{W},u)$ is an trivializing cover, then there
exists $(\mathscr{V},q)\leq^{r}(\mathscr{U},s)$, such that
$(\mathscr{V},q)\leq(\mathscr{W},u)$. Furthermore, if
$(\mathscr{V},q),(\mathscr{W},u)\leq^{r}(\mathscr{U},s)$ and
$(\mathscr{V},q)\leq(\mathscr{W},u)$, then
$(\mathscr{V},q)\leq^{r}(\mathscr{W},u)$.

  * (ii)

If $(\mathscr{V},q)=(\\{V_{\beta}\\}_{\beta\in J},\\{q_{\beta}\\}_{\beta\in
J})\leq^{r}(\mathscr{U},s)$ then
$(\Phi^{*}\mathscr{V},q^{\Phi})\leq^{r}(\Phi^{*}\mathscr{U},t)$, where for
each $\beta\in J$ we define

$q^{\Phi}_{\beta}:=t_{\phi(\beta)}|_{\Phi^{-1}(V_{\beta})}\in\Gamma(\Phi^{-1}(V_{\beta}),L)$

where $\phi:J\to I$ is the refinement map as in Definition 4.2 and $t$ is the
collection of sections in the complementary cover $(\Phi^{*}\mathscr{U},t)$.

  * (iii)

The map $(\mathscr{V},q)\mapsto(\Phi^{*}\mathscr{V},q^{\Phi})$ defines an
equivalence of categories between the collection of trivializing covers which
are restricted refinements of $(\mathscr{U},s)$ and the trivializing covers
which are restricted refinements of $(\Phi^{*}\mathscr{U},t)$.

###### Proof.

  * (i)

Suppose $(\\{V_{\beta}\\},\\{q_{\beta}\\})$ is a refinement of
$(\mathscr{U},s)$ via the map $\phi:J\to I$. For each $\beta\in J$, define

$u_{\beta}:=s_{\phi(\beta)}|_{V_{\beta}}\in\Gamma(V_{\beta},L).$

It follows that this defines a trivializing cover
$(\\{V_{\beta}\\},\\{u_{\beta}\\})$ which is a restricted refinement of
$(\mathscr{U},s)$ and is a refinement of $(\\{V_{\beta}\\},\\{q_{\beta}\\})$.

  * (ii)

Follows immediately.

  * (iii)

The map $(\mathscr{V},q)\mapsto(\Phi^{*}\mathscr{V},q^{\Phi})$ clearly
respects refinement and so does the inverse.

∎

######  Corollary 4.1.

A choice of a trivializing cover $(\mathscr{U},s)$ and a complementary
trivializing cover $(\Phi^{*}\mathscr{U},t)$ along $\Phi$ induces an
isomorphism

$H^{\bullet}_{triv}(M,L,P)\cong H^{\bullet}_{triv}(M,L,\Phi(P)).$

###### Proof.

Fixing trivializing cover $(\mathscr{U},s)$ and a complementary cover
$(\Phi^{*}\mathscr{U},t)$ along $\Phi$, we obtain for each restricted
refinement $(\mathscr{V},q)$ of $(\mathscr{U},s)$ an isomorphism via Lemma 4.2

$\Phi^{\bullet}_{(\mathscr{V},q)}:H^{\bullet}_{triv}(\mathscr{V},q,P)\to
H^{\bullet}_{triv}(\Phi^{*}\mathscr{V},q^{\Phi},\Phi(P)).$

Furthermore, if $(\mathscr{W},u)\leq^{r}(\mathscr{V},q)$, then it follows that
the diagram commutes

${H^{\bullet}_{triv}(\mathscr{V},q,P)}$${H^{\bullet}_{triv}(\Phi^{*}\mathscr{V},q^{\Phi},\Phi(P))}$${H^{\bullet}_{triv}(\mathscr{W},u,P)}$${H^{\bullet}_{triv}(\Phi^{*}\mathscr{W},u^{\Phi},\Phi(P))}$$\scriptstyle{\Phi^{\bullet}_{(\mathscr{V},q)}}$$\scriptstyle{\Phi^{\bullet}_{(\mathscr{W},u)}}$

It then follows by abstract nonsense and the fact that the restricted
refinements of $(\mathscr{U},s)$ and $(\Phi^{*}\mathscr{U},t)$ are cofinal,
that after taking the direct limit, we have the desired isomorphism

$\Phi^{\bullet}:H^{\bullet}_{triv}(M,L,P)\to H^{\bullet}_{triv}(M,L,\Phi(P)).$

∎

###### Proof.

(Of Main Theorem 1) Apply Lemma 4.1 to obtain an isomorphism

$\check{H}^{\bullet}(M,\mathscr{S}_{P})\to H^{\bullet}_{triv}(M,L,P)\to
H^{\bullet}_{triv}(M,L,\Phi(P))\to\check{H}^{\bullet}(M,\mathscr{S}_{\Phi(P)}).$

It then follows immediately from the definition of sheaf quantization that

$Q_{shf}(M,P)=\bigoplus_{n}\check{H}^{n}(M,\mathscr{S}_{P})\cong\bigoplus_{n}\check{H}^{n}(M,\mathscr{S}_{\Phi(P)})=Q_{shf}(M,\Phi(P))$

as desired. ∎

###  4.2 Effect on Bohr-Sommerfeld Leaves

Now, for the Bohr-Sommerfeld quantization.

######  Lemma 4.3.

If $\iota:B\to M$ is a leaf of $P$, then $\Phi^{-1}\circ\iota:B\to M$ is a
leaf of $\Phi(P)$.

###### Proof.

By construction of $\Phi(P)$ it is clear that if $\iota:B\to M$ is an immersed
integral submanifold of $P$, then $\Phi^{-1}\circ\iota:B\to M$ is an immersed
integral submanifold of $\Phi(P)$. Indeed, for any $x\in B$, we have

$\iota_{*}(T_{x}B)=P_{\iota(x)}.$

Hence,

$(\Phi^{-1}\circ\iota)_{*}(T_{x}B)=\Phi^{-1}_{*}P_{\iota(x)}=\Phi(P)_{\Phi^{-1}\circ\iota(x)}.$

The only issue now is maximality of $\Phi^{-1}\circ\iota:B\to M$. Suppose
$\iota^{\prime}:B^{\prime}\to M$ is another integral submanifold of $\Phi(P)$
through $\Phi^{-1}\circ\iota(x)$ for some $x\in B$. Then, by the same
argument, $\Phi\circ\iota^{\prime}:B^{\prime}\to M$ is an integral submanifold
of $P$ through $\iota(x)$. Thus, by the maximality of $\iota:B\to M$ there
exists an open embedding

$F:B^{\prime}\to B$

making the diagram commute

${M}$${B}$${B^{\prime}}$$\scriptstyle{\iota}$$\scriptstyle{\Phi\circ\iota^{\prime}}$$\scriptstyle{F}$

Since $\Phi$ is a diffeomorphism, it then follows that the below diagram
commutes

${M}$${B}$${B^{\prime}}$$\scriptstyle{\Phi^{-1}\circ\iota}$$\scriptstyle{\iota^{\prime}}$$\scriptstyle{F}$

Hence, $\Phi^{-1}\circ\iota:B\to M$ is a maximal integral submanifold of
$\Phi(P)$. ∎

For the sake of convenience, we simply write $B$ for the data of a leaf
$\iota:B\to M$ of $P$ and $\Phi(B)$ for the data of $\Phi^{-1}\circ\iota:B\to
M$. It then becomes clear that we have defined a bijection induced by $\Phi$.

######  Corollary 4.2.

If $Lf(P)$ denotes the set of leaves of $P$, then the map

$\displaystyle Lf(P)\to Lf(\Phi(P));\quad B\mapsto\Phi(B).$

is a bijection.

Let $(\mathscr{U},s)$ be a trivialization cover of $L\to M$. Then, for any
leaf $\iota:B\to M$, we can define a pullback trivialization cover
$(\iota^{*}\mathscr{U},\iota^{*}s)$ of $\iota^{*}L\to B$. If
$\mathscr{U}=\\{U_{\alpha}\\}_{\alpha\in J}$ and
$\iota^{-1}(U_{\alpha})\neq\emptyset$, we simply define

$\iota^{*}s_{\alpha}:\iota^{-1}(U_{\alpha})\to\iota^{*}L;\quad
x\mapsto(x,s_{\alpha}(\iota(x))).$

######  Lemma 4.4.

Let $(\mathscr{U},s)$ be a trivialization cover, $\Phi$ a symplectomorphism,
and $(\Phi^{*}\mathscr{U},t)$ a complementary pair with respect to $\Phi$.
Then, for any leaf $\iota:B\to M$ of $P$, $\Phi$ induces an isomorphism

$\Phi_{B}:H^{0}(B,\mathscr{S}_{P}|_{B})\to
H^{0}(B,\mathscr{S}_{\Phi(P)}|_{\Phi(B)}).$ (25)

###### Proof.

Fix a leaf $\iota:B\to M$ of $P$. The trivialization cover $(\mathscr{U},s)$
pulls back to a trivialization cover $(\iota^{*}\mathscr{U},\iota^{*}s)$ of
$\iota^{*}L\to B$. These naturally induce local data

$(\lambda^{*},\theta^{*})\in\check{Z}^{1}(\iota^{*}\mathscr{U},\mathscr{O}^{\times}_{B})\times\check{C}^{0}(\iota^{*}\mathscr{U},\Omega^{1}_{\mathbb{C}}).$

Now, doing the exact same construction on the leaf $(B,\Phi^{-1}\circ\iota)$
of $\Phi(P)$ with respect to the complementary pair $(\Phi^{*}\mathscr{U},t)$,
we get a trivialization cover

$((\Phi^{-1}\circ\iota)^{*}\Phi^{*}\mathscr{U},(\Phi^{-1}\circ\iota)^{*}t)=(\iota^{*}\mathscr{U},(\Phi^{-1}\circ\iota)^{*}t)$

with its own induced local data

$((\lambda^{\prime})^{*},(\theta^{\prime})^{*})\in\check{Z}^{1}((\iota^{*}\mathscr{U},\mathscr{O}^{\times}_{B})\times\check{C}^{0}(\iota^{*}\mathscr{U},\Omega^{1}_{\mathbb{C}}).$

It is straightforward to show that if $(\lambda,\theta)$ are the local data of
$(\mathscr{U},s)$, then

$\iota^{*}\lambda=\lambda^{*},\quad\iota^{*}\theta=\theta^{*}$

and further

$\lambda^{*}=(\lambda^{\prime})^{*},\quad\theta^{*}=(\theta^{\prime})^{*}.$

Now, for the construction. Fix $\tau\in
H^{0}(\iota^{*}\mathscr{U},\mathscr{S}_{P}|_{B})$. Then for each $\alpha\in J$
there exists smooth $f^{\alpha}:\iota^{-1}(U_{\alpha})\to\mathbb{C}$ such that

$\tau|_{\iota^{-1}(U_{\alpha})}=f^{\alpha}\iota^{*}s_{\alpha}.$

Define $\Phi(\tau):B\to(\Phi^{-1}\circ\iota)^{*}L$ locally by

$\Phi_{B}(\tau)|_{\iota^{-1}(U_{\alpha})}=f^{\alpha}(\Phi^{-1}\circ\iota)^{*}t_{\alpha}.$

Since the transition functions of
$(\iota^{*}\mathscr{U},(\Phi^{-1}\circ\iota)^{*}t)$ are identical to that of
$(\iota^{*}\mathscr{U},\iota^{*}s)$, it follows that these local descriptions
agree on overlaps, hence define a well-defined section
$\Phi(\tau):B\to(\Phi^{-1}\circ\iota)^{*}L$. This section is covariantly
constant since the local 1-forms $\theta^{*}$ of
$(\iota^{*}\mathscr{U},\iota^{*}s)$ agree with the local 1-forms of
$(\iota^{*}\mathscr{U},(\Phi^{-1}\circ\iota)^{*}t)$. Thus, we have a well-
defined map

$\Phi_{B}:H^{0}(B,\mathscr{S}_{P}|_{B})\to
H^{0}(B,\mathscr{S}_{\Phi(P)}|_{\Phi(B)}).$

Applying the exact same reasoning to $\Phi^{-1}$, we obtain an inverse. ∎

###### Proof.

(Of Main Theorem 2) Suppose
$\check{H}^{1}(M,\underline{\mathbb{C}}_{M}^{\times})=0$, $P\subset TM$ is a
real polarization, and $(\mathscr{U},s)$ is a trivializing cover of $L\to M$.
Due to Corollary 4.4, the bijection in Lemma 4.2 restricts to a bijection

$BS(P)\to BS(\Phi(P));\quad B\mapsto\Phi(B).$

Furthermore, taking the direct sum of the isomorphisms in Equation (25), we
obtain the desired isomorphism

$\bigoplus_{B\in BS(P)}\Phi_{B}:Q_{BS}(M,P)\to Q_{BS}(M,\Phi(P)).$

∎

##  5 Application to Toric Geometry

To finish off, we discuss an application to toric varieties. In particular, we
can mildly generalize the Bohr-Sommerfeld rigidity discussed in Hamilton [5].
There are easier proofs that don’t require the machinery developed above, but
it at least illustrates how the vanishing of the first cohomology with
coefficients $\mathbb{C}^{\times}$ can appear.

######  Theorem 5.1.

Let $(M,\omega)$ be a smooth compact symplectic toric variety together with a
prequantum line bundle $(L,\nabla)\to M$. Then, for any choice of
symplectomorphism $\Phi:M\to M$ and any choice of real polarization $P\subset
TM$, we have isomorphisms

$\displaystyle Q_{shf}(M,P)$ $\displaystyle\cong Q_{shf}(M,\Phi(P))$
$\displaystyle Q_{BS}(M,P)$ $\displaystyle\cong Q_{BS}(M,\Phi(P))$

###### Proof.

By the first Proposition in section 3.2 of Fulton [3], we have $\pi_{1}(M)=0$.
Since

$H^{1}(M,\mathbb{C}^{\times})\cong\text{Hom}(\pi_{1}(M),\mathbb{C}^{\times}),$

it then follows that $H^{1}(M,\mathbb{C}^{\times})=0$. Thus, the hypotheses of
Main Theorems 1 and 2 hold. ∎

## References

  * [1] Iakovos Androulidakis and Georges Skandalis. The Holonomy Groupoid of a Singular Foliation. Journal für die Reine und Angewandte Mathematik, vol 2009, pp. 1-37, 2009.
  * [2] Raoul Bott and Loring W. Tu. Differential Forms in Algebraic Topology. Springer-Verlag, New York, N.Y., 1982.
  * [3] William Fulton. Introduction to Toric Varieties. Princeton University Press, Princeton, N.J., 1993.
  * [4] V. Guillemin and S. Sternberg. The Gelfand-Cetlin System and Quantization of the Complex Flag Manifolds. Journal of Functional Analysis, vol. 52, 106-128, 1983.
  * [5] Mark Hamilton. Locally Toric Manifolds and Singular Bohr-Sommerfeld Leaves. Memoirs of the American Mathematical Society, vol. 207, 2010.
  * [6] Lisa C. Jeffrey and Jonathan Weitsman. Bohr-Sommerfeld Orbits in the Moduli Space of Flat Connections and the Verlinde Dimension Formula. Communications in Mathematical Physics, vol. 150, pp. 593-630, 1992.
  * [7] Bertram Kostant. Quantization and Unitary Representations. Lectures in Modern Analysis and Applications III. Lecture Notes in Mathematics, vol 170. Springer, Berlin, Heidelberg, 1970.
  * [8] Jedrzej Sniatycki. On Cohomology Groups Appearing in Geometric Quantization Differential Geometrical Methods in Mathematical Physics, Lectures Notes in Mathematics, vol. 570, pp. 46-66, Springer, Berlin, 1977.
  * [9] Héctor J. Sussmann. Orbits of Families of Vector Fields and Integrability of Distributions. Transactions of the American Mathematical Society, vol 180, pp. 171-188, 1973.


11institutetext: University of Ljubljana, Faculty of Computer and Information
Science
11email: {vitjan.zavrtanik, matej.kristan<EMAIL_ADDRESS>
# DSR – A dual subspace re-projection network for surface anomaly detection

Vitjan Zavrtanik  Matej Kristan  Danijel Skočaj

###### Abstract

The state-of-the-art in discriminative unsupervised surface anomaly detection
relies on external datasets for synthesizing anomaly-augmented training
images. Such approaches are prone to failure on near-in-distribution anomalies
since these are difficult to be synthesized realistically due to their
similarity to anomaly-free regions. We propose an architecture based on
quantized feature space representation with dual decoders, DSR, that avoids
the image-level anomaly synthesis requirement. Without making any assumptions
about the visual properties of anomalies, DSR generates the anomalies at the
feature level by sampling the learned quantized feature space, which allows a
controlled generation of near-in-distribution anomalies. DSR achieves state-
of-the-art results on the KSDD2 and MVTec anomaly detection datasets. The
experiments on the challenging real-world KSDD2 dataset show that DSR
significantly outperforms other unsupervised surface anomaly detection
methods, improving the previous top-performing methods by $10\%$ AP in anomaly
detection and $35\%$ AP in anomaly localization. Code is available at:
https://github.com/VitjanZ/DSR_anomaly_detection.

###### Keywords:

Surface anomaly detection, discrete feature space, simulated anomaly
generation

##  1 Introduction

Surface anomaly detection addresses localization of image regions that deviate
from normal object appearance. This is a fundamental problem in industrial
inspection, in which the anomalies are defects on production line objects. In
the most challenging situations, the distribution of the normal appearance of
the inspected objects is very close to the distribution of anomaly
appearances, while anomalies often occupy only a small portion of the object.
Furthermore, the anomalies are rare in practical production lines, making the
acquisition of a suitable data set for training supervised methods infeasible.
The methods thus focus on leveraging only anomaly-free images, since these can
be abundantly obtained.

Figure 1:  The dual decoder architecture with discrete feature space allows
DSR robust object-specific reconstruction ($R_{DSR}$) and accurate detection
of near-in-distribution anomalies, which present a considerable challenge for
the recent state-of-the-art ($M_{[22]}$).

Most anomaly detection approaches are based on computing the difference
between the inspected image and its image-level or feature-level
reconstruction [1, 2, 18, 5, 22, 3], with their reconstruction method trained
only on anomaly-free images. These approaches assume that anomalies will be
poorly reconstructed since they have never been observed during training and
that the reconstruction failure on anomalies can be well detected by $L_{2}$
or SSIM [20] difference with the input image.

However, $L_{2}$ and SSIM measures can only detect anomalies that differ
substantially from normal appearance. Subsequent works have addressed this
problem by either learning the distance measure with a discriminative network
[21] or by classifying the anomalies directly on the input image [11]. These
methods require annotated anomalies at training time, and resort to simulation
of anomalies from auxiliary datasets and copy-pasting and blending them with
the anomaly-free training images. While these methods by far outperform the
reconstruction-only methods, they rely substantially on the quality of the
auxiliary dataset and the simulation process quality; their performance still
degrades on near-in-distribution anomalies (Figure 1) since it is difficult to
simulate these realistically.

In this paper we address two drawbacks of the surface anomaly detection state-
of-the-art: the reliance on the auxiliary anomaly simulation datasets and poor
near-in-distribution anomaly detection. We propose a dual subspace re-
projection surface anomaly detection network (DSR). The network leverages the
framework of discretized latent feature space image representation [16], and
jointly learns a general and a normal-appearance-specific subspace re-
projection to emphasize the anomaly detection capability.

The proposed architecture avoids reliance on auxiliary anomaly datasets in
training anomaly discrimination. We propose a new anomaly simulation technique
that generates the anomalies directly from the network’s discretized latent
space of natural images, leading to significant performance improvements on
near-in-distribution anomaly detection (Figure 1).

Our contribution is thus twofold: (i) the dual image reconstruction branch
architecture with discretized latent representation and (ii) the latent space
anomaly generation method that leverages the learned representation of natural
images. DSR substantially outperforms the state-of-the-art in near-in-
distribution anomaly detection on the recent challenging KSDD2 [6] and
delivers state-of-the-art performance on the standard MVTec anomaly detection
dataset [3].

##  2 Related Work

Many recent surface anomaly detection methods are based on image
reconstruction [1, 2, 5], where an encoder-decoder network is trained for
image reconstruction on anomaly free images. The anomaly detection capability
of these methods is based on the assumption that the trained networks will be
unable to accurately reconstruct anomalous regions due to never seeing them
during training, making anomalies detectable by comparing the input image to
its reconstruction. The core assumption often does not hold, especially in
more diverse datasets, as the reconstruction networks learn to generalize well
which enables accurate anomaly reconstruction, hampering the downstream
anomaly detection performance. Similarly, in [22] iterative inpainting is used
for image reconstruction, however, the method is sensitive to random pattern
regions that are difficult to inpaint and which cause false positive
detections.

Several recent approaches utilize the ability of pretrained networks to
extract informative features from an image. In [4] a network is trained on
anomaly free images to reconstruct features extracted by the pretrained
network. In [19] a feature map is generated as a concatenation of several
layers of a pretrained network. An auto-encoder is then trained to reconstruct
the resulting feature map. As with image reconstruction based anomaly
detection methods, the networks in [4, 19] are assumed to be unable to
reconstruct the features extracted by the pretrained networks. In [17], a
pretrained network is also used to extract informative features from anomaly
free data, however instead of relying on reconstruction for anomaly detection,
a multivariate gaussian is fit to the anomaly-free data. A Mahalanobis
distance is then used as an anomaly score. A similar approach is used in [7],
however, a Gaussian is fit at each location in the feature map.

Discriminative unsupervised anomaly detection methods [21, 12, 11] utilize
synthetically generated anomalies to train a discriminative anomaly detection
network. In order to alleviate overfitting on the synthetic anomaly
appearance, in [21] a reconstruction network is used to restore the normal
appearance of the synthetic anomalies. The discriminative network then learns
a distance function between the original image and its reconstruction to
perform anomaly detection. Due to a limited distribution of the generated
synthetic anomalies, the reconstruction network may overfit to the synthetic
appearance and fail to restore normality in near-distribution anomalies,
leading to a reduction in performance in downstream anomaly detection.

Figure 2:  The DSR architecture. During training, the non-anomalous image
quantized feature maps $(Q_{hi},Q_{lo})$ are replaced by the anomaly augmented
feature maps $(Q_{a,hi},Q_{a,lo})$ generated by the latent space sampling
procedure (shaded block). The pathway marked with green arrows are used when
training the Upsampling module with simulated smudges and at inference.

##  3 DSR

The DSR architecture is outlined in Figure 2. The input image is projected
into a quantized latent space (Section 3.1) and then decoded in parallel by
two decoders specialized on different appearance subspaces. One decoder, the
general object appearance decoder (Section 3.2), is trained for high-fidelity
reconstruction of arbitrary natural images, while the second, object-specific
decoder (Section 3.3), is restricted to reconstructing only normal local
appearances of the selected object. The two reconstructed images are then
analyzed by the anomaly detection module (Section 3.4). The output
localization map is at feature resolution and is upsampled to the input image
resolution by the Upsampling module (Section 3.5).

DSR is trained by a novel technique of sampling the quantized latent feature
space for generating near-in-distribution anomalies (Section 3.6). The DSR
training regime is explained in Section 3.7.

###  3.1 Quantized latent space encoder

DSR leverages quantized feature space representation, which has recently
demonstrated strong modelling capabilities of complex natural image
distributions ,e.g., [9, 15]. The approach is based on quantizing the
extracted features with features from a codebook $\mathbf{K}$ which has been
trained for optimal decoding of spatial configurations of quantized features
into high-fidelity images.

In particular, the quantized latent space encoder module in Figure 2 accepts
the input image $\mathbf{I}$ and projects it into a feature space $\mathbf{F}$
using a Resnet-based encoder. A quantized feature representation $\mathbf{Q}$
of the input image is obtained by replacing each feature vector
$\mathbf{F}_{ij}$ with its nearest neighbor $\mathbf{e}_{l}$ in $\mathbf{K}$,
i.e,

$Q_{i,j}=q(F_{i,j})=\operatorname*{argmin}_{\mathbf{e}_{l}\in\mathbf{K}}\big{(}||\mathbf{F}_{i,j}-\mathbf{e}_{l}||\big{)}.$
(1)

In the following, we refer to this operation as vector quantization (VQ). Note
that the input image is encoded at two levels of detail using low- and high-
resolution codebooks ($\mathbf{K}_{\mathrm{lo}}$, $\mathbf{K}_{\mathrm{hi}}$),
producing $\mathbf{Q}_{\mathrm{lo}}$ and $\mathbf{Q}_{\mathrm{hi}}$. The two-
level VQ has recently been reported to produce superior reconstructions [16].
The architecture of the quantized latent space encoder is shown in Figure 3.
The quantized feature maps $Q_{hi}$ and $Q_{low}$ produced by the quantized
latent space encoder are $4\times$ and $8\times$ smaller than the original
input image, respectively.

Figure 3: The quantized latent space encoder architecture. Two residual blocks
extract image features at different spatial resolutions. The low-resolution
features $F_{lo}$ are quantized by the codebook $K_{lo}$ into a feature
representation $Q_{lo}$. The high-resolution features are concatenated by
upsampled $Q_{lo}$, followed by a convolutional block and quantized by the
codebook $K_{hi}$ into a high-resolution representation $Q_{hi}$.

###  3.2 General object appearance decoder

The subspace of VQ encoded natural images is captured by specific spatial
configurations of quantized feature vectors. We apply a general object
appearance decoder to learn decoding of these configurations into image
reconstructions. The decoder first upsamples the low resolution
$\mathbf{Q}_{\mathrm{lo}}$ and concatenates it with the
$\mathbf{Q}_{\mathrm{hi}}$, which is followed by two ResNet blocks and two
transposed convolution upsampling blocks that map into the reconstructed image
$\mathbf{I}_{\mathrm{gen}}$.

###  3.3 Object-specific appearance decoder

The tasks of the second decoder, the object-specific appearance decoder (see
Figure 4) is to restore local visual anomalies into feasible normal
appearances of the object instances observed during training. In particular,
we would like to restrict the appearance subspace, i.e., the allowed spatial
VQ feature configurations, into configurations that agree with normal
appearances. This is achieved by a subspace restriction module (Figure 4),
which transforms both high- and low-resolution general input VQ
representations ($\mathbf{Q}$=$\\{\mathbf{Q}_{\mathrm{hi}}$,
$\mathbf{Q}_{\mathrm{lo}}\\}$) into non-quantized object-specific subspace
configurations ($\tilde{\mathbf{F}}$=$\\{\tilde{\mathbf{F}}_{\mathrm{lo}}$,
$\tilde{\mathbf{F}}_{\mathrm{hi}}\\}$). This is followed by a VQ projection
(with codebooks $\mathbf{K}$={$K_{hi},K_{lo}\\}$) into object subspace-
restricted quantized feature configurations
($\tilde{\mathbf{Q}}$=$\\{\tilde{\mathbf{Q}}_{\mathrm{hi}}$,
$\tilde{\mathbf{Q}}_{\mathrm{lo}}\\}$). The quantized representations are
concatenated (the low-resolution representation is upsampled first) and
decoded into reconstructed anomaly-free image $\mathbf{I}_{\mathrm{spc}}$
using a convolutional decoder. The subspace restriction modules (Figure 4) are
encoder-decoder convolutional networks with three downsampling and
corresponding upsampling blocks. The image reconstruction network consists of
two downsampling convolutional blocks, followed by four transposed convolution
upsampling blocks. Examples of images with anomalies present and their
anomaly-free reconstructions are shown in Figure 5.

Figure 4:  The object-specific appearance decoder architecture. The features
in the quantized features maps $\mathbf{Q}$ are reduced by a subspace
restriction module into non-quantized features $\tilde{\mathbf{F}}$ that are
then vector-quantized (VQ) by the codebooks $\mathbf{K}$ into
$\tilde{\mathbf{Q}}$, which are then decoded into an anomaly-free image
$\mathbf{I}_{spc}$.  Figure 5:  The object-specific decoder reconstructs the
anomalous images (left) into anomaly-free images (right) with remarkable
fidelity. In the second example, it even correctly reconstructs the anomalous
green wire into blue.

###  3.4 Anomaly detection module

The purpose of the anomaly detection module is to localize the anomaly by
inspecting the input image reconstruction generated by the general object
appearance decoder ($\mathbf{I}_{\mathrm{gen}}$) and the object-specific
appearance decoder ($\mathbf{I}_{\mathrm{spc}}$). The reconstructed images are
concatenated depth-wise and decoded into a segmentation mask $\mathbf{M}$ by a
Unet-based architecture. $\mathbf{M}$ is the output anomaly map indicating the
pixel-level location of the anomalies in the image. To compute also the image-
level anomaly score, we apply a simple segmentation mask interpretation
procedure as in [22] – the segmentation mask is smoothed by a $21\times 21$
averaging filter and globally max-pooled into a single score.

###  3.5 Upsampling module

The generated segmentation mask $\mathbf{M}$ is of the same resolution as the
feature maps. A simple Unet-like upsampling module is thus used to resample
the mask to full resolution. The input to the network is a depth-wise
concatenation of the input image $\mathbf{I}$ and bilinearly upsampled mask
$\mathbf{M}$. The output is the final full-resolution mask $\mathbf{M_{r}}$
(Figure 2).

###  3.6 Feature-space surface anomaly generation

The purpose of anomaly generator is to simulate near-in-distribution anomalies
of various shapes and sizes with diverse visual appearances to (i) learn
normal appearance subspace restriction in the object-specific appearance
decoder (Section 3.3) and (ii) to specialize the anomaly detection module
(Section 3.4) for detection of potentially gentle appearance deviations of
anomalies from a diverse within-class normal appearance.

We propose a method that leverages the learned quantized subspace in DSR to
generate such training anomalies as follows. An anomaly-free input image
$\mathbf{I}$ is encoded into a VQ subspace representation $\mathbf{Q}$ and an
anomaly mask $\mathbf{M}_{\mathrm{gt}}$ is generated by sampling a Perlin
noise [14], with values 1 indicating anomalous pixels. The features in
$\mathbf{Q}$ corresponding to the anomaly indicators in
$\mathbf{M}_{\mathrm{gt}}$ are replaced by sampling from the set of codebook
features $\mathbf{K}$.

Sampling without constraints will likely lead to significant appearance
changes on anomalous pixels, resulting in trivial out-of-distribution
reconstructed images. On the other hand, if the closest vectors are sampled
that are too similar to the normal appearance, they will likely lead to false-
positive detections.

We thus first define a similarity bound on all features for a given image
$n\sim\mathcal{U}[\lambda_{s}N_{K},N_{K}]$, where $N_{K}$ is the number of
codebook vectors and $\lambda_{s}$ is the similarity bound parameter. Then we
replace each feature in $\mathbf{Q}$, indicated by $\mathbf{M}_{\mathrm{gt}}$,
with one of its near neighbors from the codebook feature vectors, sampled
uniformly, i.e., $k\sim\mathcal{U}[\lambda_{s}N_{K},n]$. In all experiments
$\lambda_{s}$ is $0.05$, therefore, excluding the $5\%$ of most similar
vectors to prevent false-positive generation, while prioritizing the selection
of the features close to the vector to be replaced to encourage the generation
of near-in-distribution anomalies.

###  3.7 DSR training procedure

The DSR is trained in three stages. In the first stage, the quantized latent
space encoder (Section 3.1), along with the VQ codebook and the general object
appearance decoder are trained on ImageNet [8] to learn the subspace of
natural images, which allows a high-fidelity reconstruction of general images.
We use the procedure from [13] that minimizes the image reconstruction loss as
well as the difference between the feature space projection $\mathbf{F}$
computed in the quantized latent space encoder (Figures 2 and 3) and its
quantized version $\mathbf{Q}$, i.e,

$\mathcal{L}_{\mathrm{st1}}=L_{2}(\mathbf{I},\mathbf{I}_{\mathrm{gen}})+L_{2}(sg[\mathbf{F}],\mathbf{Q})+\lambda_{1}L_{2}(\mathbf{F},sg[\mathbf{Q}]),$
(2)

where $L_{2}(\cdot)$ is the Euclidean distance and $sg[\cdot]$ is the stop
gradient operator constraining the operand to be a non-updated constant [13].
After training, the discrete latent space encoder, the codebook and the
general object appearance decoder (coloured in magenta in Figure 2) are fixed.

In the second, anomaly detection training stage, the detection parts of DSR
are trained on images of the selected object type. Anomaly-free training
images are projected through the quantized latent space encoder into their
quantized feature representation $\mathbf{Q}$. The surface anomaly generation
method presented in Section 3.6 then generates the anomalies at the feature
level, $\mathbf{Q}_{\mathrm{a}}$, along with their ground truth masks
$\mathbf{M}_{\mathrm{gt}}$. The representation $\mathbf{Q}_{\mathrm{a}}$ (that
replaces $\mathbf{Q}$ in Figure 2) is then forward passed and decoded into the
anomaly mask $\mathbf{M}$.

The anomaly detection module and the object-specific appearance decoder are
trained by minimizing the focal loss between the ground truth
$\mathbf{M}_{\mathrm{gt}}$ and predicted anomaly mask $\mathbf{M}$, the
$L_{2}$ distance between the subspace-restricted configurations
$\tilde{\mathbf{F}}$ computed in the object-specific appearance decoder
(Figure 4) and the non-anomalous input image quantized representation
$\mathbf{Q}$, and the $L_{2}$ distance between the non-anomalous input image
$\mathbf{I}$ and its object-specific reconstruction
$\mathbf{I}_{\mathrm{spc}}$, i.e.,

$\mathcal{L}_{\mathrm{st2}}=\mathcal{L}_{\mathrm{foc}}(\mathbf{M}_{\mathrm{gt}},\mathbf{M})+\lambda_{2}L_{2}(\tilde{\mathbf{F}},\mathbf{Q})+\lambda_{3}L_{2}(\mathbf{I},\mathbf{I}_{\mathrm{spc}}).$
(3)

Finally, the Upsampling module is trained. After the anomaly detection network
has been trained, images with copy-pasted smudges are generated and the low-
resolution anomaly masks are computed as detection network outputs. Since the
full-resolution masks are known from the smudge pasting on the original image,
the upsampling network can be trained with a focal loss.

##  4 Experiments

###  4.1 Implementation details

In the first training stage (Section 3.7) quantized latent space encoder,
general object appearance decoder and the latent vector codebook are trained
for image reconstruction on ImageNet [8] for $200,000$ iterations with a batch
size of 32 and a learning rate of $0.0002$. The codebooks $K_{hi}$ and
$K_{lo}$ each contain $4096$ latent vectors of dimension $128$. In the second
training stage (Section 3.7) the object-specific appearance decoder and the
anomaly detection model are trained for $100,000$ iterations with a batch size
of $8$ with a learning rate of $0.0002$. The learning rate is decreased by a
factor of $10$ after $80,000$ iterations. The $\lambda_{2}$ and $\lambda_{3}$
values in Eq.3 are set to $1$ and $10$, respectively. In the third stage the
Upsampling module is then trained for $20,000$ iterations with a learning rate
of $0.0002$ and a batch size of $8$. The training hyperparameters and network
architectures remain constant throughout our experiments.

###  4.2 Experimental results on KSDD2

Figure 6:  Normal (left) and anomalous (right) images from KSDD2. The
anomalies (marked with white arrows) are very similar to the normal appearance
in this dataset.

The recently proposed KSDD2 [6] surface anomaly detection dataset is currently
the most challenging dataset with near-in-distribution surface anomalies (see
Figure 6). KSDD2 was acquired on real industrial production lines and contains
a wide variety of anomalies, many of which are particularly challenging due to
their similarity to the normal appearances in the training set.

The dataset contains $2085$ anomaly-free and $246$ anomalous training images
to test anomaly detection under unsupervised and supervised setups. For
example, in the unsupervised setup, only anomaly-free training images may be
used. The test set contains $894$ anomaly-free and $110$ anomalous images. We
follow the evaluation procedure defined in [6], with the AP metric for image-
level anomaly detection ($AP_{det}$), and present also anomaly localization
results in terms of pixel-level AP ($AP_{loc}$).

Additional anomalous training samples available in KSDD2 [6] enable comparison
of DSR with the supervised defect detection methods. Note that, although the
acquisition of anomalous examples is difficult in practice, in many cases
these are available, albeit in small quantities. Most of unsupervised anomaly
detection methods, however, do not, or even can not, make use of this
additional information, if available. The proposed DSR method can be easily
adapted to take such annotations of anomalous images into account. In addition
to the generated synthetic anomalies, the real-world anomalies with known
ground truth can be utilised to train the anomaly detection module.

Figure 7: Qualitative results of the unsupervised DSR on the KSDD2 dataset:
the input image, the overlaid predicted mask and the ground truth.

####  4.2.1 Comparison with the state-of-the-art

The results of unsupervised surface anomaly detection methods are shown in
Table 1. DSR significantly outperforms other state-of-the-art unsupervised
methods such as MAD [17], PaDim [7] and DRAEM [21], achieving higher AP scores
for anomaly detection and localization. It outperforms the previous best
image-level AP score by $7.9$ p.p. Qualitative examples of the unsupervised
DSR are presented in Figure 7 and show that despite a quite heterogeneous
normal appearance, that can also be observed in Figure 6, the visual defects
are successfully detected and localized.

####  4.2.2 Extension to supervised learning

In contrast to most of the unsupervised visual anomaly detection methods, DSR
can also utilise pixel-level anotations, if they are available, making it
applicable in low-shot anomaly detection scenarios. We evaluated the proposed
approach in the supervised setting. Table 2 shows the comparison with the
method from [6] that is specifically designed for supervised defect detection
and can operate with various levels of supervision. In KSDD2, there are 246
anomalous samples available, and if image-level labels are available but none
of them is segmented, the method [6] operates in the weakly supervised mode
($N=0$).

Method | US [4] | MAD [17] | DR$\AE$M [21] | PaDim [7] | DSR
---|---|---|---|---|---
$AP_{det}$ | 65.3 | 79.3 | 77.8 | 55.6 | 87.2
$AP_{loc}$ | - | - | 42.4 | 45.3 | 61.4

Table 1: Anomaly detection ($AP_{det}$) and localization ($AP_{loc}$) on the
KSDD2 dataset.

The completely unsupervised DSR, without taking these additional positive
training samples into account, outperforms the weakly supervised method [6]
for 13.9 p.p. in $AP_{det}$ and achieves a good localization result, whereas
[6] is unable to produce meaningful segmentation maps. When a number of
anomalous training images are also segmented ($N>0$), the results improve even
further, and significantly outperform the results of [6]. When using the full
annotated training set of 246 segmented examples, DSR achieves anomaly
detection performance near that of the fully supervised method [6] and
outperforms the recent fully supervised method presented in [10]. Furthermore,
DSR achieves the highest localization performance $AP_{loc}$ under all
training settings, significantly outperforming [6]. These results demonstrate
that DSR can efficiently work in supervised settings as well, utilising all
information available.

| Method |  $N$: | 0 | 16 | 53 | 246
---|---|---|---|---|---|---
$AP_{det}$ | [6] |  | 73.3 | 83.2 | 89.1 | 95.4
DSR |  | 87.2 | 91.4 | 94.6 | 95.2
[10] |  | - | - | - | 93.3
$AP_{loc}$ | [6] |  | 1.0 | 45.1 | 52.2 | 67.6
DSR |  | 61.4 | 71.2 | 81.6 | 85.5

Table 2: Anomaly detection and localization on the KSDD2 dataset in a
supervised settings w.r.t. number of used anomalous training images $N$ with
ground truth masks.

###  4.3 Experimental results on MVTec

Figure 8: Qualitative results on the MVTec dataset: the input images, outputs
of the reconstruction network, input images with overlaid output masks, the
output masks and the ground truth masks are shown in individual rows.

We also perform experiments on the MVTec anomaly detection dataset [3], which
contains $15$ different texture and object classes, and has been established
as the standard surface anomaly evaluation benchmark [11, 7, 21, 17]. The
training set consists only of anomaly-free images, while the test set is
comprised of anomalous as well as anomaly-free images. The widely used AUROC
metric is applied for image-level anomaly detection. Because only a fraction
of the pixels in the test set are anomalous, the pixel-wise average-precision
metric AP [21] is used to evaluate the anomaly localization performance. It is
more robust to class imbalance and better suited for anomaly localization
evaluation than the commonly used pixel-wise AUROC.

####  4.3.1 Comparison with the state-of-the-art

Method | bottle | capsule | grid | leather | pill | tile | trans. | zipper | cable | carpet | hazelnut | m. nut | screw | toothbrush | wood | average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
[4] | 99.0 | 86.1 | 81.0 | 88.2 | 87.9 | 99.1 | 81.8 | 91.9 | 86.2 | 91.6 | 93.1 | 82.0 | 54.9 | 95.3 | 97.7 | 87.7
[22] | 99.9 | 88.4 | 99.6 | 100 | 83.8 | 98.7 | 90.9 | 98.1 | 81.9 | 84.2 | 83.3 | 88.5 | 84.5 | 100 | 93.0 | 91.7
[17] | 100 | 92.3 | 92.9 | 100 | 83.3 | 97.4 | 95.9 | 97.9 | 94.0 | 95.5 | 98.7 | 93.1 | 81.2 | 95.8 | 97.6 | 94.4
[7] | 99.8 | 91.5 | 95.7 | 100 | 94.4 | 97.4 | 97.8 | 90.9 | 92.2 | 99.9 | 93.3 | 99.2 | 84.4 | 97.2 | 98.8 | 95.5
[11] | 98.2 | 98.2 | 100 | 100 | 94.9 | 94.6 | 96.1 | 99.9 | 81.2 | 93.9 | 98.3 | 99.9 | 88.7 | 99.4 | 99.1 | 96.1
[21] | 99.2 | 98.5 | 99.9 | 100 | 98.9 | 99.6 | 93.1 | 100 | 91.8 | 97.0 | 100 | 98.7 | 93.9 | 100 | 99.1 | 98.0
DSR | 100 | 98.1 | 100 | 100 | 97.5 | 100 | 97.8 | 100 | 93.8 | 100 | 95.6 | 98.5 | 96.2 | 99.7 | 96.3 | 98.2

Table 3: Results of anomaly detection on MVTec dataset (AUROC) with the
average score over all classes ($avg$) in the last column.

We evaluate DSR against recent state-of-the-art surface anomaly detection
methods. The experimental results are presented in Tables 3 and 4 for image-
level anomaly detection and for pixel-level anomaly localization,
respectively, and show that DSR outperforms the recent state-of-the-art
approaches. It achieves an average AUROC of $98.2\%$, while maintaining a
strong anomaly localization performance with an AP score of $70.2\%$. DSR
outperforms the previous top performer in anomaly detection DR$AE$M [21] on
the mean AUROC score by $0.2$ percentage points (p.p.) and achieves superior
performance on classes such as transistor, cable, carpet and screw, where
near-distribution anomalies such as deformations are more prevalent.
Qualitative results on the MVTec dataset are shown in Figure 8 and demonstrate
that the detected anomalous regions resemble the ground truth anomaly maps to
a high degree.

Method | bottle | capsule | grid | leather | pill | tile | trans. | zipper | cable | carpet | hazelnut | m. nut | screw | toothbrush | wood | average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
[4] | 74.2 | 25.9 | 10.1 | 40.9 | 62.0 | 65.3 | 27.1 | 36.1 | 48.2 | 52.2 | 57.8 | 83.5 | 7.8 | 37.7 | 53.3 | 45.5
[22] | 76.4 | 38.2 | 36.4 | 49.1 | 51.6 | 52.6 | 39.2 | 63.4 | 24.4 | 61.4 | 33.8 | 64.3 | 43.9 | 50.6 | 38.2 | 48.2
[7] | 77.3 | 46.7 | 35.7 | 53.5 | 61.2 | 52.4 | 72.0 | 58.2 | 45.4 | 60.7 | 61.1 | 77.4 | 21.7 | 54.7 | 46.3 | 55.0
[21] | 86.5 | 49.4 | 65.7 | 75.3 | 48.5 | 92.3 | 50.7 | 81.5 | 52.4 | 53.5 | 92.9 | 96.3 | 58.2 | 44.7 | 77.7 | 68.4
DSR | 91.5 | 53.3 | 68.0 | 62.5 | 65.7 | 93.9 | 41.1 | 78.5 | 70.4 | 78.2 | 87.3 | 67.5 | 52.5 | 74.2 | 68.4 | 70.2

Table 4: Results (AP) of anomaly localization on MVTec dataset. Figure 9:
Qualitative reconstruction results. DSR far better reconstructs the anomalies
by their corresponding in-distribution appearance than the state-of-the-art
DR$\AE$M [21].

Further qualitative comparison is shown in Figure 9. Both DSR and DR$\AE$M
[21] are trained to restore the normality of images corrupted by simulated
anomalies. However, DR$\AE$M [21] generates the anomalies in the image space
from an out-of-distribution dataset, while DSR generates the anomalies within
the quantized feature space, making it more difficult for the reconstruction
network to overfit to the simulated anomaly appearance. This results in DSR
having a more robust image normality restoration capability that is
insensitive to near-in-distribution anomalies such as deformations. Figure 9
compares the reconstruction results of both methods. Note that DSR produces a
more realistic reconstruction. Due to the reconstruction network not
overfitting to the synthetic anomaly appearance, DSR can recognize
deformations as deviations from normality and reconstruct them accordingly.

###  4.4 Ablation Study

Additional experiments on the MVTec dataset provide further insights into DSR.

Anomaly source evaluation. We evaluate the impact of training using our
feature-based anomalies and compare it to the training with image-based
anomalies generated from out-of-distribution datasets as proposed in [21]. The
results are shown in Table 6(a), where DSRimg denotes the results when using
image-based anomalies during training. We can see that feature-based anomaly
generation (DSR) outperforms the image-based approach in anomaly localization
and detection.

Anomaly feature sampling. To generate simulated anomalies, DSR samples vectors
from the codebook $K$ according to a designed process presented in Section
3.6. We compare this approach to one that samples the codebook with a uniform
probability for each vector. The results (DSRrandom) presented in Table 6(b)
show a significant drop in performance. The sampling method, therefore, plays
an important role. The proposed approach generates anomalous regions from
vectors that are close to the extracted vectors generating simulated near-
distribution anomalies that leads to superior results in terms of anomaly
detection as well as localization. DSR is robust to the choice of the
similarity bound parameter, retaining a good anomaly detection performance for
a wide range of $\lambda_{s}$ values (Table 5).

$\lambda_{s}$ | 0.01 | 0.02 | 0.05 | 0.1 | 0.2 | 0.5
---|---|---|---|---|---|---
AUROCdet | 97.5 | 98.0 | 98.2 | 97.7 | 97.1 | 95.5
Table 5: Results on MVTec using various similarity bound $\lambda_{s}$ values.

Reconstruction loss components. Table 6(c) shows the impact of individual
reconstruction loss components of Eq. (3), where the reconstruction is
conditioned both on the feature $L_{feat}$ and image $L_{img}$ reconstruction
losses, which are the second and the third components in Eq. (3),
respectively. During training DSR${}_{L_{img}}$ uses only the image
reconstruction loss and DSR${}_{L_{feat}}$ uses only the feature
reconstruction loss. There is a significant loss in performance when using
only individual loss components. Relying on both image and feature
reconstruction losses during training results in a more robust normality
reconstruction model leading to a $2$ p.p. higher average image-level AUROC as
well as significantly higher localization AP scores.

Upsampling module Table 6(d) shows the effect of removing the Upsampling
module of the network, leading to a drop in localization performance. The
anomaly detection performance remains the same as the image-level score is
extracted from the lower resolution output anomaly map $M$.

| (a) | (b) | (c) | (d)
---|---|---|---|---
Experiment | DSR | DSRimg | DSRrandom | DSR${}_{L_{img}}$ | DSR${}_{L_{feat}}$ | DSRU-
AUROCdet | 98.2 | 97.8 | 95.6 | 96.3 | 95.2 | 98.2
APloc | 70.2 | 67.5 | 62.5 | 67.0 | 61.8 | 65.2

Table 6: Ablation study results on MVTec: (a) using out-of-distribution
texture-based anomalies (DSRimg) in training; (b) unconstrained uniform
anomaly sampling (DSRrandom); (c) training with only image reconstruction loss
(DSR${}_{L_{img}}$) and with only the feature reconstruction loss
(DSR${}_{L_{feat}}$); (d) Performance without the Upsampling module (DSRU-).

##  5 Conclusion

We proposed DSR, a discriminative surface anomaly detection method based on
dual image reconstruction branch architecture with discretized latent
representation. Such representation allows controlled generation of synthetic
anomalies in feature space, which, in contrast to the state-of-the-art methods
that generate anomalies in image space, makes no assumption about the anomaly
appearance and does not rely on image-level heuristics. Our anomaly generation
approach produces near-in-distribution synthetic anomalies resulting in more
robustly trained reconstruction capabilities and detection of anomalies whose
appearance is close to the normal appearance.

On the recent challenging real-world KSDD2 dataset [6], it outperforms all
other unsupervised surface anomaly detection methods by $10\%$ AP in anomaly
detection and $35\%$ AP in anomaly localization tasks. DSR also significantly
outperforms the weakly-supervised model presented in [6]. Moreover, we showed
that the proposed approach can be extended to utilise also pixel-wise
annotated anomalous training samples. When used in such supervised settings,
it considerably improves the results of related supervised methods [6] when
only a few annotated examples are available. This demonstrates the potential
of DSR to be used in real-world settings where the number of available
anomalous images is typically too low to train fully supervised methods. DSR
also achieves state-of-the-art results on the standard MVTec anomaly detection
dataset [3].

The ablation study shows that the sampling method used for anomaly generation
impacts the final anomaly detection results significantly, which suggests that
a more complex feature sampling method may improve results further, which we
leave for our future research endeavours.

In addition to it’s good performance across several tasks and it’s ability to
utilize anomalous training samples that are available in some practical
scenarios, DSR is fast, running at $58$ FPS, making it a good choice for real-
world industrial applications with real-time requirements.

Acknowledgement This work was supported by Slovenian research agency programs
J2-2506, L2-3169, P2-0214. Vitjan Zavrtanik was supported by the Young
researcher program of the ARRS.

## References

  * [1] Akcay, S., Atapour-Abarghouei, A., Breckon, T.P.: GANomaly: Semi-supervised anomaly detection via adversarial training. In: Asian Conference on Computer Vision. pp. 622–637. Springer (2018)
  * [2] Akçay, S., Atapour-Abarghouei, A., Breckon, T.P.: Skip-GANomaly: Skip connected and adversarially trained encoder-decoder anomaly detection. In: 2019 International Joint Conference on Neural Networks (IJCNN). pp. 1–8. IEEE (jul 2019)
  * [3] Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: MVTec AD – A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9592–9600 (2019)
  * [4] Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4183–4192 (2020)
  * [5] Bergmann, P., Löwe, S., Fauser, M., Sattlegger, D., Steger, C.: Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In: 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics theory and Applications. vol. 5, pp. 372–380 (2018)
  * [6] Božič, J., Tabernik, D., Skočaj, D.: Mixed supervision for surface-defect detection: from weakly to fully supervised learning. Computers in Industry 129, 103459 (2021)
  * [7] Defard, T., Setkov, A., Loesch, A., Audigier, R.: Padim: A patch distribution modeling framework for anomaly detection and localization. In: International Conference on Pattern Recognition. pp. 475–489. Springer (2021)
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [9] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12873–12883 (June 2021)
  * [10] Lei, L., Sun, S., Zhang, Y., Liu, H., Xu, W.: PSIC-Net: Pixel-wise segmentation and image-wise classification network for surface defects. Machines 2021 9, 221 (2021). https://doi.org/10.3390/MACHINES9100221, https://www.mdpi.com/2075-1702/9/10/221/htmhttps://www.mdpi.com/2075-1702/9/10/221
  * [11] Li, C.L., Sohn, K., Yoon, J., Pfister, T.: Cutpaste: Self-supervised learning for anomaly detection and localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9664–9674 (2021)
  * [12] Lv, C., Shen, F., Zhang, Z., Xu, D., He, Y.: A novel pixel-wise defect inspection method based on stable background reconstruction. IEEE Transactions on Instrumentation and Measurement 70, 1–13 (2021). https://doi.org/10.1109/TIM.2020.3038413
  * [13] van den Oord, A., Vinyals, O., kavukcuoglu, k.: Neural discrete representation learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30, pp. 6306–6315. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf
  * [14] Perlin, K.: An image synthesizer. ACM Siggraph Computer Graphics 19(3), 287–296 (1985)
  * [15] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 (2021)
  * [16] Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with vq-vae-2. In: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 32, pp. 14866–14876. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/5f8e2fa1718d1bbcadf1cd9c7a54fb8c-Paper.pdf
  * [17] Rippel, O., Mertens, P., Merhof, D.: Modeling the distribution of normal data in pre-trained deep features for anomaly detection. ICPR (2020)
  * [18] Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis 54, 30–44 (2019)
  * [19] Shi, Y., Yang, J., Qi, Z.: Unsupervised anomaly segmentation via deep feature reconstruction. Neurocomputing 424, 9–22 (2021)
  * [20] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [21] Zavrtanik, V., Kristan, M., Skočaj, D.: Draem - a discriminatively trained reconstruction embedding for surface anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8330–8339 (October 2021)
  * [22] Zavrtanik, V., Kristan, M., Skočaj, D.: Reconstruction by inpainting for visual anomaly detection. Pattern Recognition 112, 107706 (2021). https://doi.org/https://doi.org/10.1016/j.patcog.2020.107706, https://www.sciencedirect.com/science/article/pii/S0031320320305094