# Singular hermitian metrics and the decomposition theorem of Catanese,
Fujita, and Kawamata

Luigi Lombardi  Dipartimento di Matematica
Università degli Studi di Milano Statale
Via Cesare Saldini 50, Milan 20133, Italy<EMAIL_ADDRESS>and
Christian Schnell  Department of Mathematics
Stony Brook University, Stony Brook, NY 11794-3651
<EMAIL_ADDRESS>
###### Abstract.

We prove that a torsion-free sheaf ${\mathcal{F}}$ endowed with a singular
hermitian metric with semi-positive curvature and satisfying the minimal
extension property admits a direct-sum decomposition
${\mathcal{F}}\simeq{\mathcal{U}}\oplus{\mathcal{A}}$ where ${\mathcal{U}}$ is
a hermitian flat bundle and ${\mathcal{A}}$ is a generically ample sheaf. The
result applies to the case of direct images of relative pluricanonical bundles
$f_{*}\omega_{X/Y}^{\otimes m}$ under a surjective morphism $f\colon X\to Y$
of smooth projective varieties with $m\geq 2$. This extends previous results
of Fujita, Catanese–Kawamata, and Iwai.

##  1\. Introduction

Let $f\colon X\to Y$ be a fibration of smooth projective varieties (over the
complex numbers) and let $\omega_{X/Y}=\omega_{X}\otimes f^{*}\omega_{Y}^{-1}$
be the relative canonical bundle. Motivated by earlier work of Fujita,
Catanese and Kawamata [CK19, Theorem 1.2] proved a direct-sum decomposition

(1) $f_{*}\omega_{X/Y}\;\simeq\;{\mathcal{U}}\,\oplus\,{\mathcal{A}}$

where ${\mathcal{U}}$ is a hermitian flat bundle and ${\mathcal{A}}$ is a
generically ample sheaf, if not zero. We recall that a coherent torsion-free
sheaf ${\mathcal{A}}$ on a smooth projective variety is _generically ample_ if
its restriction to a general complete intersection smooth curve is an ample
vector bundle. In dimension one we say that ${\mathcal{A}}$ is generically
ample if it is ample. When $Y$ is a smooth projective curve, the decomposition
(1) is Fujita’s second decomposition in [Fuj78] (see also [CD17, Theorem
3.3]). Motivated by (1), we introduce the following definition.

######  Definition 1.

A coherent torsion-free sheaf ${\mathcal{F}}$ admits a
_Catanese–Fujita–Kawamata decomposition_ if there exists an isomorphism
${\mathcal{F}}\simeq{\mathcal{U}}\oplus{\mathcal{A}}$ where ${\mathcal{U}}$ is
a hermitian flat bundle, and ${\mathcal{A}}$ is either a generically ample
sheaf or the zero sheaf.

In this paper we extend decomposition (1) to direct images of relative
_pluricanonical_ bundles.

######  Theorem 2.

If $f\colon X\to Y$ is a surjective morphism of smooth projective complex
varieties, then $f_{*}\omega_{X/Y}^{\otimes m}$ admits a
Catanese–Fujita–Kawamata decomposition for every $m\geq 2$.

When $Y$ is a smooth projective curve, Theorem 2 was proved by Iwai in [Iwa22,
Theorem 1.4] by showing in greater generality that the reflexive hull
$\big{(}f_{*}\omega_{X/Y}^{\otimes m}\big{)}^{**}$ admits a
Catanese–Fujita–Kawamata decomposition for any smooth projective variety $Y$
and $m\geq 1$.

While the proof of (1) is Hodge-theoretic, the proof of Theorem 2 is based on
the fact that, for any $m\geq 2$, the sheaf $f_{*}\omega_{X/Y}^{\otimes m}$
carries a canonical singular hermitian metric with semi-positive curvature.
Furthermore, this metric satisfies the so-called _minimal extension property_
(_cf_. §2, [PT18, Theorem 1.1] and [HPS18, Theorem 27.1]). This is a property
that stems from Ohsawa–Takegoshi’s extension theorem with optimal bounds and
allows one to extend local sections across subsets of measure zero with a
control on the $L^{2}$-norm. Instances of bundles satisfying the minimal
extension property are pseudo-effective line bundles and Nakano semi-positive
vector bundles. Theorem 2 is an application of the following theorem.

######  Theorem 3.

Let ${\mathcal{F}}$ be a coherent torsion-free sheaf on a smooth projective
variety $Y$ endowed with a singular hermitian metric with semi-positive
curvature and satisfying the minimal extension property. Then ${\mathcal{F}}$
admits a Catanese–Fujita–Kawamata decomposition.

###### Proof of Theorem 2.

By [PT18, Theorem 1.1] and [HPS18, Theorem 27.1] the pushforward
$f_{*}\omega_{X/Y}^{\otimes m}$ admits a singular hermitian metric with semi-
positive curvature and satisfying the minimal extension property for any
$m\geq 2$. The result follows by Theorem 3. ∎

As an application of Theorem 2, we prove a structure theorem for the sheaves
$f_{*}\omega_{X/Y}^{\otimes m}$ with $m\geq 2$.

######  Theorem 4.

Let $f\colon X\to Y$ be a fibration of smooth projective complex varieties and
denote $J=\\{\,m\in\mathbb{N}_{\geq 2}\,\big{|}\,f_{*}\omega_{X/Y}^{\otimes
m}\neq 0\,\\}$. Suppose there exists an open subset $U\subset Y$ such that:
${\rm codim}(Y\smallsetminus U)\geq 2$, the morphism $f$ is smooth over $U$,
and $\omega_{V/U}$ is $f|_{V}$-semi-ample where $V=f^{-1}(U)$. Then
$f_{*}\omega_{X/Y}^{\otimes m}$ is either generically ample for all $m\in J$,
or hermitian flat for all $m\in J$.

In §4 we collect further instances of Catanese–Fujita–Kawamata decompositions.

### Acknowledgments

We thank Victor González-Alonso for fruitful conversations. Moreover we thank
Federico Caucci, Yajnaseni Dutta, Beppe Pareschi, Mihnea Popa and Luca Tasin
for comments on a first draft of the paper. We also thank Masataka Iwai for
letting us know about the papers [HIM22], [Iwa22] and [IM22].

L.L. was partially supported by GNSAGA, PSR Linea 4, and PRIN 2020: Curves,
Ricci flat Varieties and their Interactions.

During the preparation of this paper, Ch.S. was partially supported by NSF
grant DMS-1551677 and by a grant from the Simons Foundation (817464, Schnell).
He thanks the National Science Foundation and the Simons Foundation for their
financial support; he also thanks the Max-Planck-Institute for Mathematics and
the Kavli Institute for the Physics and Mathematics of the Universe for
providing him with excellent working conditions.

##  2\. The minimal extension property

Let $Y$ be a complex manifold of positive dimension $n$ and let
$({\mathcal{F}},h)$ be a torsion-free sheaf endowed with a singular hermitian
metric. We refer to [HPS18, §19] for the definition of singular hermitian
metrics with semi-positive curvature on torsion-free sheaves. Denote by
$B\subset\mathbb{C}^{n}$ the open unit ball centered at the origin with volume
$\mu(B)=\pi^{n}/n!$. Moreover let $F_{y}$ be the fiber of ${\mathcal{F}}$ at a
point $y\in Y$ over which ${\mathcal{F}}$ is locally free. We recall the
definition of the minimal extension property introduced in [HPS18, Definition
20.1].

######  Definition 5.

The pair $({\mathcal{F}},h)$ satisfies the _minimal extension property_ if
there exists an analytic nowhere dense closed subset $Z\subset Y$ such that
${\mathcal{F}}$ is locally free on $Y\backslash Z$, and for every embedding
$\iota\colon B\hookrightarrow Y$ of the open unit ball centered at
$\iota(0)=y\in Y\backslash Z$, and vector $v\in F_{y}$ of length
$\big{|}v\big{|}_{h,y}=1$, there exists a holomorphic section $s\in
H^{0}(B,\iota^{*}{\mathcal{F}})$ such that

$s(0)\,=\,v\quad\mbox{ and
}\quad\frac{1}{\mu(B)}\int_{B}\big{|}s\big{|}_{h}^{2}\,\,d\mu\,\leq\,1.$

We recall a few properties of the minimal extension property from [HPS18].

######  Proposition 6.

Let $({\mathcal{F}},h)$ be a torsion-free sheaf endowed with a singular
hermitian metric satisfying the minimal extension property and let
$b\colon{\mathcal{F}}\hookrightarrow{\mathcal{G}}$ be an inclusion of torsion-
free sheaves. If $b$ is generically an isomorphism, then $h$ extends to a
singular hermitian metric $h_{{\mathcal{G}}}$ on ${\mathcal{G}}$ satisfying
the minimal extension property. Moreover, if $h$ has semi-positive curvature,
then $h_{{\mathcal{G}}}$ has semi-positive curvature as well.

###### Proof.

The proposition is essentially proved in [HPS18, Proposition 19.3] where it is
stated in greater generality. It only remains to note that the minimal
extension property holds for $({\mathcal{G}},h_{{\mathcal{G}}})$, but this is
true because every section of ${\mathcal{F}}$ is also a section of
${\mathcal{G}}$. ∎

######  Proposition 7.

Let $({\mathcal{F}},h)$ be a torsion-free sheaf endowed with a singular
hermitian metric satisfying the minimal extension property. If
$\varphi\colon{\mathcal{F}}\twoheadrightarrow{\mathcal{E}}$ is a quotient onto
a torsion-free sheaf and $h^{\prime}$ is the induced metric, then
$({\mathcal{E}},h^{\prime})$ satisfies the minimal extension property.

###### Proof.

Let $Z=Z({\mathcal{F}})$ be as in Definition 5 and let $S({\mathcal{E}})$ be
the locus where ${\mathcal{E}}$ is not locally free. Set $Z^{\prime}=Z\cup
S({\mathcal{E}})$ and let $y\in Y\backslash Z^{\prime}$. For $w\in E_{y}$ the
induced metric $h^{\prime}$ on ${\mathcal{E}}$ is defined by

(2) $\big{|}w\big{|}_{h^{\prime},y}={\rm
inf}\\{\,\big{|}v\big{|}_{h,y}\;\big{|}\;v\in F_{y}\mbox{ and
}\varphi_{y}(v)=w\,\\}.$

(If $\varphi_{y}=0$, then the metric is $+\infty$ for all $w\neq 0$.) Now let
$w\in E_{y}$ be such that $\big{|}w\big{|}_{h^{\prime},y}=1$ and let $B\subset
Y$ be the embedding of the unit ball centered at $y\in Y\backslash
Z^{\prime}$. Then there exists $v\in F_{y}$ such that
$\big{|}v\big{|}_{h,y}=1$ and, by the minimal extension property of
${\mathcal{F}}$, a holomorphic section $s\in H^{0}(B,{\mathcal{F}}|_{B})$ such
that $s(0)=v$ and $\frac{1}{\mu(B)}\int_{B}\big{|}s\big{|}^{2}_{h}d\mu\leq 1$.
As $\big{|}\varphi(s)\big{|}_{h^{\prime},y}\leq\big{|}s\big{|}_{h,y}$ for
almost every $y\in B$, this yields inequalities

$\frac{1}{\mu(B)}\int_{B}\big{|}\varphi(s)\big{|}^{2}_{h^{\prime}}\,d\mu\,\leq\,\frac{1}{\mu(B)}\int_{B}\big{|}s\big{|}^{2}_{h}\,d\mu\,\leq\,1.$

∎

The following result generalizes [HPS18, Theorem 26.4].

######  Proposition 8.

Suppose $Y$ is a compact complex manifold and let $({\mathcal{F}},h)$ be a
torsion-free sheaf endowed with a singular hermitian metric of semi-positive
curvature and satisfying the minimal extension property. If
$f\colon{\mathcal{F}}\twoheadrightarrow{\mathcal{U}}$ is a quotient onto a
vector bundle ${\mathcal{U}}$ endowed with a smooth hermitian flat metric,
then there exists a morphism $s\colon{\mathcal{U}}\to{\mathcal{F}}$ such that
$f\circ s={\rm id}_{{\mathcal{U}}}$.

###### Proof.

Set $r={\rm rk}\big{(}{\mathcal{U}}\big{)}>0$. The bundle ${\mathcal{U}}$ is
associated to a representation $\pi_{1}(Y)\to U(r)$ of the fundamental group
of $Y$ to the unitary group $U(r)$. Hence ${\mathcal{U}}$ decomposes as a
direct sum of vector bundles arising from irreducible unitary representations.
Without loss of generality we can suppose that ${\mathcal{U}}$ is irreducible.

Consider the quotients

${\mathcal{F}}\otimes{\mathcal{U}}^{\vee}\,\twoheadrightarrow{\mathcal{U}}\otimes{\mathcal{U}}^{\vee}\stackrel{{\scriptstyle
tr}}{{\twoheadrightarrow}}{\mathcal{O}}_{Y}$

where the first is induced by $f$ and the second is the trace map of
${\mathcal{U}}$. Since the metric on ${\mathcal{U}}$, and hence on
${\mathcal{U}}^{\vee}$, is flat, the induced singular hermitian metric on the
sheaf ${\mathcal{F}}\otimes{\mathcal{U}}^{\vee}$ has semi-positive curvature
and the minimal extension property. By [HPS18, Theorem 26.4] there exists a
splitting
$s^{\prime}\colon{\mathcal{O}}_{Y}\to{\mathcal{F}}\otimes{\mathcal{U}}^{\vee}$,
and hence a non-trivial morphism
$s^{\prime\prime}\colon{\mathcal{U}}\to{\mathcal{F}}$ such that $f\circ
s^{\prime\prime}\neq 0$. By Schur’s Lemma the composition $f\circ
s^{\prime\prime}$ is an isomorphism and $s:=s^{\prime\prime}\circ(f\circ
s^{\prime\prime})^{-1}$ splits $f$. ∎

######  Remark 9.

Following [HIM22, Theorem 1.4] the previous proposition remains valid when
${\mathcal{F}}$ is reflexive and $h$ does not necessarily satisfy the minimal
extension property.

Finally we recall the following theorem [HPS18, Theorem 26.1] which is based
on an earlier result of Cao and Păun [CP17, Lemma 5.3]. We define the
determinant of a torsion-free sheaf ${\mathcal{F}}\neq 0$ as ${\rm
det}({\mathcal{F}})=\big{(}\bigwedge^{\rm
rk({\mathcal{F}})}{\mathcal{F}}\big{)}^{**}$.

######  Theorem 10.

Let $Y$ be a compact complex manifold and let $({\mathcal{F}},h)$ be a nonzero
torsion-free sheaf endowed with a singular hermitian metric. Suppose $h$ has
semi-positive curvature and satisfies the minimal extension property. If
$c_{1}\big{(}\det({\mathcal{F}})\big{)}=0$ in $H^{2}(Y,\mathbb{R})$, then
${\mathcal{F}}$ is locally free, $h$ is smooth and $({\mathcal{F}},h)$ is
hermitian flat.

##  3\. Catanese–Fujita–Kawamata decompositions

###  3.1. Proof of Theorem 3

Set $n=\dim Y$. We may suppose $n>0$ and ${\mathcal{F}}\neq 0$. By [HPS18,
Proposition 25.1, §26 and Definition 19.1] the line bundle ${\rm
det}({\mathcal{F}})$ admits a singular hermitian metric with semi-positive
curvature. Hence ${\rm det}({\mathcal{F}})$ is pseudo-effective and for any
very ample line bundle $A$ on $Y$ the _degree_ of ${\mathcal{F}}$ satisfies

${\rm
deg}_{A}({\mathcal{F}})\;:=\;\big{(}A^{n-1}\,\cdot\,{\mathcal{F}}\big{)}\;\geq\;0$

by Nakai–Moishezon’s Theorem. We define the $A$_-slope_ of ${\mathcal{F}}$ as

$\mu^{A}({\mathcal{F}})\,:=\,\frac{\deg_{A}({\mathcal{F}})}{{\rm
rk}({\mathcal{F}})}$

and say that it is _semistable_ (with respect to $A$) if for every nonzero
coherent sub-module ${\mathcal{E}}\subset{\mathcal{F}}$ the inequality
$\mu^{A}({\mathcal{E}})\leq\mu^{A}({\mathcal{F}})$ holds. Let

(3)
$0\,=\,{\mathcal{N}}_{0}\,\subsetneq\,{\mathcal{N}}_{1}\,\subsetneq\,\ldots\,\subsetneq\,{\mathcal{N}}_{d}\,=\,{\mathcal{F}}$

be the Harder–Narasimhan filtration of ${\mathcal{F}}$. Hence for any
$i=1,\ldots,d$ the quotients ${\mathcal{N}}_{i}/{\mathcal{N}}_{i-1}$ are
torsion-free and semistable. Moreover the slopes

$\mu^{A}_{i}:=\mu^{A}({\mathcal{N}}_{i}/{\mathcal{N}}_{i-1})$

satisfy

$\mu^{A}_{1}\,>\,\mu^{A}_{2}\,>\,\ldots\,>\,\mu^{A}_{d}$

[Mar81, Proposition-Definition 1.13]. We denote by

${\mathcal{Q}}\;:=\;{\mathcal{F}}/{\mathcal{N}}_{d-1}\;=\;{\mathcal{N}}_{d}/{\mathcal{N}}_{d-1}$

the minimal destabilizing quotient of ${\mathcal{F}}$ and set

${\mathcal{L}}:={\rm det}({\mathcal{Q}})$

for the determinant of ${\mathcal{Q}}$. By [PT18, Lemma 2.4.3] and [HPS18,
Proposition 25.1] both ${\mathcal{Q}}$ and ${\mathcal{L}}$ admit a singular
hermitian metric with semi-positive curvature. Hence ${\mathcal{L}}$ is
pseudo-effective and ${\rm deg}_{A}({\mathcal{L}})\geq 0$. We distinguish two
cases: ${\rm deg}_{A}({\mathcal{L}})>0$ and ${\rm deg}_{A}({\mathcal{L}})=0$.

Let’s begin with the case ${\rm deg}_{A}({\mathcal{L}})>0$. We are going to
show that ${\mathcal{F}}$ is already generically ample. Let $H=A^{\otimes a}$
be a very ample line bundle with $a\gg 0$ so that Flenner’s Theorem [HL10,
Theorem 7.1.1] applies to a general complete intersection smooth curve $C$ cut
out by divisors in $|H|$ and contained in the locus where ${\mathcal{F}}$ is
locally free. It follows that the Harder–Narasimhan filtration (3) of
${\mathcal{F}}$ restricts to the Harder–Narasimhan filtration

$0\,=\,{\mathcal{M}}_{0}\,\subsetneq\,{\mathcal{M}}_{1}\,\subsetneq\,\ldots\,\subsetneq\,{\mathcal{M}}_{d}\,=\,{\mathcal{F}}|_{C}$

of ${\mathcal{F}}|_{C}$. Here the sheaves
${\mathcal{M}}_{i}:={\mathcal{N}}_{i}|_{C}$ are locally free and semistable,
and

$\frac{{\rm deg}\big{(}{\mathcal{M}}_{i}/{\mathcal{M}}_{i-1}\big{)}}{{\rm
rk}\big{(}{\mathcal{M}}_{i}/{\mathcal{M}}_{i-1}\big{)}}>\frac{{\rm
deg}\big{(}{\mathcal{M}}_{i+1}/{\mathcal{M}}_{i}\big{)}}{{\rm
rk}\big{(}{\mathcal{M}}_{i+1}/{\mathcal{M}}_{i}\big{)}}$

for all $i=1,\ldots,d-1$. Since

${\rm
deg}\big{(}{\mathcal{M}}_{i}/{\mathcal{M}}_{i-1}\big{)}\,=\,\Big{(}H^{n-1}\,\cdot\,{\mathcal{N}}_{i}/{\mathcal{N}}_{i-1}\Big{)}\,=\,a^{n-1}\Big{(}A^{n-1}\,\cdot\,{\mathcal{N}}_{i}/{\mathcal{N}}_{i-1}\Big{)},$

the _minimal slope_ of ${\mathcal{F}}|_{C}$ satisfies

$\mu_{{\rm min}}\big{(}{\mathcal{F}}|_{C}\big{)}\,:=\,\frac{{\rm
deg}\big{(}{\mathcal{M}}_{d}/{\mathcal{M}}_{d-1}\big{)}}{{\rm
rk}\big{(}{\mathcal{M}}_{d}/{\mathcal{M}}_{d-1}\big{)}}\,=\,a^{n-1}\mu_{d}^{A}\,=\,a^{n-1}\frac{{\deg}_{A}({\mathcal{L}})}{{\rm
rk}({\mathcal{Q}})}\,>\,0.$

By [Bre04, Theorem 2.1] ${\mathcal{F}}|_{C}$ is an ample bundle. In this case
we set ${\mathcal{U}}=0$ and ${\mathcal{A}}={\mathcal{F}}$.

Now let us suppose that ${\rm deg}_{A}({\mathcal{L}})=0$. We will first show
that $c_{1}({\mathcal{L}})=0$ in $H^{2}(Y,\mathbb{R})$. Let $H=A^{\otimes a}$
be a very ample line bundle as before with $a$ sufficiently large. Let
$D_{1},\ldots,D_{n-2}\in|H|$ be general members such that for all
$i=2,\ldots,n-2$ each partial intersection

$V_{i}\,:=\,D_{1}\,\cap\,D_{2}\,\cap\,\ldots\,\cap\,D_{i}$

is a smooth and irreducible ample divisor in $V_{i-1}$. Hence $S:=V_{n-2}$ is
a smooth surface with $[S]=H^{n-2}$ in $H_{4}(Y;\mathbb{Z})$ such that the
restriction map

(4) $H^{2}(Y;\mathbb{Z})\to H^{2}(S;\mathbb{Z})\quad\mbox{is injective}$

(see Lefschetz’s Hyperplane Theorem [Laz04, Theorem 3.1.17]). Moreover, we can
choose the general divisors $D_{i}\in|H|$ such that the restrictions
${\mathcal{L}}|_{V_{i}}$ are pseudo-effective. Hence

$0\;=\;\big{(}H^{n-1}\,\cdot\,{\mathcal{L}}\big{)}\;=\;\big{(}H|_{S}\,\cdot\,{\mathcal{L}}|_{S}\big{)}$

from which we deduce that ${\mathcal{L}}|_{S}$ is numerically trivial as
$H|_{S}$ is ample and ${\mathcal{L}}|_{S}$ is a limit of effective classes
(_cf_. [Laz04, Theorem 1.4.29]). By (4) the claim follows.

Since $c_{1}({\mathcal{L}})=0$, by Proposition 7 and Theorem 10 the bundle
${\mathcal{Q}}$ is hermitian flat. Moreover, by Proposition 8 there exists a
decomposition ${\mathcal{F}}\simeq{\mathcal{Q}}\oplus{\mathcal{N}}_{d-1}$ so
that we only need to prove that ${\mathcal{N}}_{d-1}$ is generically ample (or
zero). Let $C$ be a general complete intersection smooth curve cut out by
divisors in $|H|$ and consider the Harder–Narasimhan filtration

$0\,=\,{\mathcal{M}}_{0}\,\subsetneq\,{\mathcal{M}}_{1}\,\subsetneq\,\ldots\,\subsetneq\,{\mathcal{M}}_{d-2}\,\subsetneq\,{\mathcal{M}}_{d-1},$

of ${\mathcal{M}}_{d-1}:=\big{(}{\mathcal{N}}_{d-1}\big{)}|_{C}$ where, as
before, ${\mathcal{M}}_{i}:={\mathcal{N}}_{i}|_{C}$. As the minimal slope of
${\mathcal{M}}_{d-1}$ satisfies

$\mu_{\rm
min}\big{(}{\mathcal{M}}_{d-1}\big{)}\,:=\,\frac{\deg\big{(}{\mathcal{M}}_{d-1}/{\mathcal{M}}_{d-2}\big{)}}{{\rm
rk}\big{(}{\mathcal{M}}_{d-1}/{\mathcal{M}}_{d-2}\big{)}}\,=\,a^{n-1}\mu_{d-1}^{A}\,>\,a^{n-1}\mu_{d}^{A}\,=\,a^{n-1}\frac{{\rm
deg}_{A}({\mathcal{L}})}{{\rm rk}({\mathcal{Q}})}\,=\,0,$

the bundle ${\mathcal{M}}_{d-1}$ is ample. In this case we set
${\mathcal{U}}={\mathcal{Q}}$ and ${\mathcal{A}}={\mathcal{N}}_{d-1}$.

##  4\. Further applications

######  Corollary 11.

Let $f\colon X\to Y$ be a surjective morphism of smooth projective varieties
and let $L$ be a semiample line bundle on $X$. Then $f_{*}(\omega_{X/Y}\otimes
L)$ admits a Catanese–Fujita–Kawamata decomposition.

###### Proof.

As a consequence of the semiampleness, the line bundle $L$ admits a smooth
hermitian metric $h$ with semi-positive curvature; in particular, the
multiplier ideal ${\mathcal{I}}(h)={\mathcal{O}}_{X}$ is trivial. By
Păun–Takayama’s theorem [PT18, Theorem 1.1] and [HPS18, Theorem 21.1],
$f_{*}(\omega_{X/Y}\otimes L)$ admits a singular hermitian metric with semi-
positive curvature and the minimal extension property. The corollary follows
from Theorem 3. ∎

######  Corollary 12.

If $f\colon X\to Y$ is a smooth fibration of smooth projective varieties and
$E$ is a Nakano semi-positive vector bundle on $X$, then
$R^{j}f_{*}(\omega_{X/Y}\otimes E)$ admits a Catanese–Fujita–Kawamata
decomposition for every $j\geq 0$.

###### Proof.

The main result of [MT08, Theorem 1.1] proves that
$R^{j}f_{*}(\omega_{X/Y}\otimes E)$ is a Nakano semi-positive vector bundle
for all $j\geq 0$. On the other hand a Nakano semi-positive vector bundle
satisfies the minimal extension property (_cf_. [SY23, Example 2.16]). The
result follows by Theorem 3.

∎

######  Remark 13.

As a further example, Iwai and Matsumura prove that a nef cotangent bundle
$\Omega_{X}^{1}$ of a smooth projective variety admits a
Catanese–Fujita–Kawamata decomposition (_cf_. [IM22, Proposition 4.2]).

######  Remark 14.

Let $f\colon X\to A$ be a surjective morphism from a smooth projective complex
variety $X$ to an abelian variety $A$. By employing [LPS20, Theorem C] it is
possible to check that the hermitian flat part of the Catanese–Fujita–Kawamata
decomposition of $f_{*}\omega_{X/A}^{\otimes m}$ ($m\geq 1$) consists of a
direct-sum of finitely many torsion line bundles in
$\operatorname{Pic}^{0}(A)$.

##  5\. Proof of Theorem 4

In this section we prove Theorem 4. In order to do so, we begin by recalling
the following result of Esnault and Viehweg, which in fact holds under more
general assumptions.

Given a fibration $f\colon X\to Y$ of varieties, a line bundle $L$ on $X$ is
$f$_-semi-ample_ if for some positive integer $N>0$ the natural morphism
$f^{*}f_{*}L^{\otimes N}\to L^{\otimes N}$ is surjective.

######  Theorem 15.

Let $g\colon X\to C$ be a smooth fibration from a smooth projective variety to
a smooth projective curve such that $\omega_{X/C}$ is $g$-semi-ample. If ${\rm
det}(g_{*}\omega_{X/C}^{\otimes m})$ is an ample line bundle for some $m\geq
1$, then for all $m\geq 2$ the bundle $g_{*}\omega_{X/C}^{\otimes m}$ is
ample, if not the zero sheaf.

###### Proof.

The proof is a special case of [EV91, Theorem 0.1]. ∎

######  Corollary 16.

Let $g\colon X\to C$ be a smooth fibration from a smooth projective variety to
a smooth projective curve such that $\omega_{X/C}$ is $g$-semi-ample. Denote
$J=\\{m\in\mathbb{N}_{\geq 2}\,\big{|}\,g_{*}\omega_{X/C}^{\otimes m}\neq
0\\}$. Then either $g_{*}\omega_{X/C}^{\otimes m}$ is ample for every $m\in
J$, or hermitian flat for every $m\in J$.

###### Proof.

By Theorem 3 $g_{*}\omega_{X/C}^{\otimes m}$ decomposes as
$g_{*}\omega_{X/C}^{\otimes m}\simeq{\mathcal{U}}_{m}\oplus{\mathcal{A}}_{m}$
with ${\mathcal{U}}_{m}$ hermitian flat (or zero), and ${\mathcal{A}}_{m}$
ample (or zero). If ${\mathcal{A}}_{m}=0$ for all $m\in J$, then
$g_{*}\omega_{X/C}^{\otimes m}$ is hermitian flat for every $m\in J$ and the
proof is complete. On the other hand, if ${\mathcal{A}}_{m}\neq 0$ for some
$m\in J$, then ${\rm det}\big{(}g_{*}\omega_{X/C}^{\otimes m}\big{)}$ is
ample, and by Theorem 15 the bundle $g_{*}\omega_{X/C}^{\otimes m}$ is itself
ample for every $m\in J$. ∎

######  Lemma 17.

Let $f\colon X\to Y$ be a fibration of smooth projective varieties. Then for
any sufficiently general hyperplane section $H\subset Y$ the variety
$X_{H}:=f^{-1}(H)$ is smooth and irreducible. Moreover, if $m$ is a positive
integer, then there is an isomorphism $f_{*}\omega_{X/Y}^{\otimes
m}|_{H}\simeq g_{*}\omega_{X_{H}/H}^{\otimes m}$ where $g:=f|_{X_{H}}$.

###### Proof.

The fact that $X_{H}$ is smooth and irreducible follows by Bertini theorem, as
stated in [Jou83, Theorem 6.3]. The following argument is inspired by [Kol86].
Consider the following cartesian diagram

$\textstyle{X_{H}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{j}$$\scriptstyle{g}$$\textstyle{X\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{f}$$\textstyle{H\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{i}$$\textstyle{Y}$

where $i$ and $j$ are the natural inclusions. There is a short exact sequence
for any $m\geq 1$

$0\to\omega_{X/Y}^{\otimes
m}\otimes{\mathcal{O}}_{X}(-X_{H})\to\omega_{X/Y}^{\otimes m}\to
j_{*}\omega_{X_{H}/H}^{\otimes m}\to 0$

since $N_{X_{H}/X}\simeq g^{*}N_{H/Y}$ and $\omega_{X/Y}^{\otimes
m}|_{X_{H}}\simeq\omega_{X_{H}/H}^{\otimes m}$. By taking higher direct
images, there is a long exact sequence

$0\to f_{*}\omega_{X/Y}^{\otimes m}\otimes{\mathcal{O}}_{Y}(-H)\to
f_{*}\omega_{X/Y}^{\otimes m}\to i_{*}g_{*}\omega_{X_{H}/H}^{\otimes m}\to$
$\to R^{1}f_{*}\omega_{X/Y}^{\otimes m}\otimes{\mathcal{O}}_{Y}(-H)\to
R^{1}f_{*}\omega_{X/Y}^{\otimes m}\to i_{*}R^{1}g_{*}\omega_{X_{H}/H}^{\otimes
m}\to\ldots$ $\ldots\to R^{j}f_{*}\omega_{X/Y}^{\otimes
m}\otimes{\mathcal{O}}_{Y}(-H)\to R^{j}f_{*}\omega_{X/Y}^{\otimes m}\to
i_{*}R^{j}g_{*}\omega_{X_{H}/H}^{\otimes m}\to\ldots$

For every index $j\geq 0$ the morphism $\psi_{j}\colon
R^{j}f_{*}\omega_{X/Y}^{\otimes m}\otimes{\mathcal{O}}_{Y}(-H)\to
R^{j}f_{*}\omega_{X/Y}^{\otimes m}$ is obtained by tensoring the natural
inclusion ${\mathcal{O}}_{Y}(-H)\to{\mathcal{O}}_{Y}$ with
$R^{j}f_{*}\omega_{X/Y}^{\otimes m}$. Hence, for a general hyperplane section
$H$, which does not contain any associated subvariety of
$R^{j}f_{*}\omega_{X/Y}^{\otimes m}$, the morphism $\psi_{j}$ is injective.
For $j=0$ it follows that the following sequence

(5) $0\to f_{*}\omega_{X/Y}^{\otimes m}\otimes{\mathcal{O}}_{Y}(-H)\to
f_{*}\omega_{X/Y}^{\otimes m}\to i_{*}g_{*}\omega_{X_{H}/H}^{\otimes m}\to 0$

is short exact. The desired isomorphism is obtained by restricting (5) to $H$.
∎

Repeated applications of the previous lemma yield the following corollary.

######  Corollary 18.

Let $f\colon X\to Y$ be a fibration of smooth projective varieties. Then for a
general complete intersection smooth curve $C$ in $Y$ the variety
$X_{C}=f^{-1}(C)$ is smooth and irreducible. Moreover, if $m$ is a positive
integer, then there is an isomorphism $f_{*}\omega_{X/Y}^{\otimes
m}|_{C}\simeq g_{*}\omega_{X_{C}/C}^{\otimes m}$ where $g:=f|_{X_{C}}$.

###### Proof of Theorem 4.

Set $n=\dim Y$. Thanks to Corollary 16 we can assume that $n\geq 2$. Moreover,
without loss of generality we can suppose that $J\neq\emptyset$. Recall that
for every $m\in J$ there exists a decomposition

(6) ${\mathcal{F}}_{m}\,:=\,f_{*}\omega_{X/Y}^{\otimes
m}\,\simeq\,{\mathcal{U}}_{m}\oplus{\mathcal{A}}_{m}$

where ${\mathcal{U}}_{m}$ is hermitian flat (or zero), and ${\mathcal{A}}_{m}$
is generically ample (or zero). If ${\mathcal{A}}_{m}=0$ for all $m\in J$,
then ${\mathcal{F}}_{m}$ is hermitian flat for every $m\in J$ and the proof is
complete. We may suppose that there exists an index $m_{0}\in J$ such that
${\mathcal{A}}_{m_{0}}\neq 0$. We aim to prove that ${\mathcal{U}}_{m}=0$ for
all $m\in J$.

Let $U\subset Y$ be an open subset as in the statement of the theorem and fix
a very ample line bundle $H$ on $Y$. Moreover, let $C\subset U$ be a general
complete intersection smooth curve cut out by divisors in $|H|$ such that
${\rm deg}({\mathcal{A}}_{m_{0}}|_{C})>0$. If $g\colon X_{C}\to C$ denotes the
restriction of $f$ to $X_{C}:=f^{-1}(C)$, then by Corollary 18 there is an
isomorphism

(7)
$0\;\neq\;{\mathcal{F}}_{m_{0}}|_{C}\;\simeq\;g_{*}\omega_{X_{C}/C}^{\otimes
m_{0}}\;\simeq\;{\mathcal{A}}_{m_{0}}|_{C}\oplus{\mathcal{U}}_{m_{0}}|_{C}.$

Hence we have

${\rm deg}\big{(}\det(g_{*}\omega_{X_{C}/C}^{\otimes
m_{0}})\big{)}\;=\;\big{(}{\rm
det}({\mathcal{A}}_{m_{0}})\,\cdot\,C\big{)}\;>\;0$

and by Corollary 16 we conclude that

(8) $g_{*}\omega_{X_{C}/C}^{\otimes k}\mbox{ is ample for all }k\in J.$

If $m\in J$ is an arbitrary index, we can select a curve $C$ (depending on
$m$) as above, and general enough so that also the following isomorphism

(9) $0\,\neq\,{\mathcal{F}}_{m}|_{C}\;\simeq\;g_{*}\omega_{X_{C}/C}^{\otimes
m}$

holds. By (6), (8) and (9) the bundle $g_{*}\omega_{X_{C}/C}^{\otimes
m}\simeq{\mathcal{U}}_{m}|_{C}\oplus{\mathcal{A}}_{m}|_{C}$ is ample, and
therefore ${\mathcal{U}}_{m}|_{C}$ itself is ample, if not trivial. Since
$\big{(}{\rm det}({\mathcal{U}}_{m})\,\cdot\,C\big{)}=0$, this forces ${\rm
det}({\mathcal{U}}_{m})|_{C}=0$, and thus ${\mathcal{U}}_{m}=0$. ∎

## References

  * [Bre04] Holger Brenner, _Slopes of vector bundles on projective curves and applications to tight closure problems_ , Trans. Amer. Math. Soc. 356 (2004), no. 1, 371–392.
  * [CD17] Fabrizio Catanese and Michael Dettweiler, _Answer to a question by Fujita on variation of Hodge structures_ , Higher dimensional algebraic geometry—in honour of Professor Yujiro Kawamata’s sixtieth birthday, Adv. Stud. Pure Math., vol. 74, Math. Soc. Japan, Tokyo, 2017, pp. 73–102.
  * [CK19] Fabrizio Catanese and Yujiro Kawamata, _Fujita decomposition over higher dimensional base_ , Eur. J. Math. 5 (2019), no. 3, 720–728.
  * [CP17] Junyan Cao and Mihai Păun, _Kodaira dimension of algebraic fiber spaces over abelian varieties_ , Invent. Math. 207 (2017), no. 1, 345–387.
  * [EV91] Hélène Esnault and Eckart Viehweg, _Ample sheaves on moduli schemes_ , Algebraic geometry and analytic geometry (Tokyo, 1990), ICM-90 Satell. Conf. Proc., Springer, Tokyo, 1991, pp. 53–80.
  * [Fuj78] Takao Fujita, _The sheaf of relative canonical forms of a Kähler fiber space over a curve_ , Proc. Japan Acad. Ser. A Math. Sci. 54 (1978), no. 7, 183–184.
  * [HIM22] Genki Hosono, Masataka Iwai, and Shin-ichi Matsumura, _On projective manifolds with pseudo-effective tangent bundle_ , J. Inst. Math. Jussieu 21 (2022), no. 5, 1801–1830.
  * [HL10] Daniel Huybrechts and Manfred Lehn, _The geometry of moduli spaces of sheaves_ , second ed., Cambridge Mathematical Library, Cambridge University Press, Cambridge, 2010.
  * [HPS18] Christopher Hacon, Mihnea Popa, and Christian Schnell, _Algebraic fiber spaces over abelian varieties: around a recent theorem by Cao and Păun_ , Local and global methods in algebraic geometry, Contemp. Math., vol. 712, Amer. Math. Soc., [Providence], RI, [2018] ©2018, pp. 143–195.
  * [IM22] Masataka Iwai and Shin-ichi Matsumura, _Abundance theorem for minimal compact Kähler manifolds with vanishing second Chern class_ , arXiv:2205.10613 (2022).
  * [Iwa22] Masataka Iwai, _Almost nef regular foliations and Fujita’s decomposition of reflexive sheaves_ , Ann. Sc. Norm. Super. Pisa Cl. Sci. (5) 23 (2022), no. 2, 719–743.
  * [Jou83] Jean-Pierre Jouanolou, _Théorèmes de Bertini et applications_ , Progress in Mathematics, vol. 42, Birkhäuser Boston, Inc., Boston, MA, 1983\.
  * [Kol86] János Kollár, _Higher direct images of dualizing sheaves. I_ , Ann. of Math. (2) 123 (1986), no. 1, 11–42.
  * [Laz04] Robert Lazarsfeld, _Positivity in algebraic geometry. I_ , Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics], vol. 48, Springer-Verlag, Berlin, 2004, Classical setting: line bundles and linear series.
  * [LPS20] Luigi Lombardi, Mihnea Popa, and Christian Schnell, _Pushforwards of pluricanonical bundles under morphisms to abelian varieties_ , J. Eur. Math. Soc. (JEMS) 22 (2020), no. 8, 2511–2536.
  * [Mar81] Masaki Maruyama, _The theorem of Grauert-Mülich-Spindler_ , Math. Ann. 255 (1981), no. 3, 317–333.
  * [MT08] Christophe Mourougane and Shigeharu Takayama, _Hodge metrics and the curvature of higher direct images_ , Ann. Sci. Éc. Norm. Supér. (4) 41 (2008), no. 6, 905–924.
  * [PT18] Mihai Păun and Shigeharu Takayama, _Positivity of twisted relative pluricanonical bundles and their direct images_ , J. Algebraic Geom. 27 (2018), no. 2, 211–272.
  * [SY23] Christian Schnell and Ruijie Yang, _Hodge modules and singular hermitian metrics_ , Math. Z. 303 (2023), no. 2, Paper No. 28, 20.


# Attentive Illumination Decomposition Model
for Multi-Illuminant White Balancing

Dongyoung Kim1 Jinwoo Kim1 Junsang Yu2 Seon Joo Kim1
1Yonsei University 2Samsung Advanced Institute of Technology

###### Abstract

White balance (WB) algorithms in many commercial cameras assume single and
uniform illumination, leading to undesirable results when multiple lighting
sources with different chromaticities exist in the scene. Prior research on
multi-illuminant WB typically predicts illumination at the pixel level without
fully grasping the scene’s actual lighting conditions, including the number
and color of light sources. This often results in unnatural outcomes lacking
in overall consistency. To handle this problem, we present a deep white
balancing model that leverages the slot attention, where each slot is in
charge of representing individual illuminants. This design enables the model
to generate chromaticities and weight maps for individual illuminants, which
are then fused to compose the final illumination map. Furthermore, we propose
the centroid-matching loss, which regulates the activation of each slot based
on the color range, thereby enhancing the model to separate illumination more
effectively. Our method achieves the state-of-the-art performance on both
single- and multi-illuminant WB benchmarks, and also offers additional
information such as the number of illuminants in the scene and their
chromaticity. This capability allows for illumination editing, an application
not feasible with prior methods.

##  1 Introduction

Figure 1: Comparison of the AID framework (bottom) with existing approaches
(top). Previous methodologies did not individually consider illuminant
profiles within the scene, resulting in unnatural results. The AID framework
outperforms previous works in illumination estimation by estimating the
chromaticity and pixel-wise weight map of each individual illuminant and
combining them.

Color constancy, a unique human capability, allows us to perceive the color of
objects uniformly under any lighting conditions. Similarly, a computational
color constancy or white balancing (WB) module is integrated into the image
processing unit, designed to compensate for the effects of illumination to
recover the original color of the objects.

Many WB studies have been conducted with the goal of predicting the single
chromaticity vector of the light source for a given image, assuming uniform
illumination. Traditional statistics-based methodologies [43, 20, 16, 19],
including gray world [10] and white patch [30] algorithms, used various
statistics that could be obtained from images. Data-driven methods [4, 24]
worked by optimizing through the white balance dataset. However, these
algorithms produce distorted results when multiple illuminants affect the
scene simultaneously. For example, when a blue skylight is coming in from the
window into a room with warm-colored lighting, applying a single white balance
matrix to the entire image may fail in recovering the scene color.

Accordingly, spatially varying white balance algorithms have been proposed to
deal with multi-illuminant scenes. Early works estimate mixed illumination map
by utilizing auxiliary flash photography [25] or prior knowledge about the
chromaticity of illuminants [23, 5]. Recently, many DNN-based methods have
been introduced with the advancements in neural networks. Algorithms using
patches [8], GANs [42], U-Net [27] with transformer blocks [31] have been
proposed.

All previous multi-illuminant WB works directly generate patch- or pixel-level
predictions of illumination maps using an encoder-decoder structure without
any structural constraints. These approaches often fail to satisfy the
linearity constraint [23, 21, 27] that the chromaticity of mixed illumination
can be expressed as a linear combination of individual light source
chromaticity under the Lambertian image model. This may result in producing
unnatural illumination that does not exist in a scene (Fig. 1 top). In
addition, as the previous methods cannot offer individual light source
profiles in a multi-illuminant scene, further tuning or editing the
illumination is not possible.

To overcome the limitations of the existing multi-illuminant WB methods, we
propose the Attentive Illumination Decomposition (AID) mechanism. AID shows
strong performance and is equipped with tunability for the spatially varying
multi-illuminant WB. Our framework works in an end-to-end manner with a single
given image. In other words, it does not require any auxiliary images [25] or
post-processing procedures [25, 26] to decompose the illumination map. Our
model is based on slot attention [33], to learn the implicit representation of
illuminant chromaticity in a scene in the form of slot vectors. Specifically,
we leverage the slot vectors to represent the chromaticities of the light
sources in a scene, and use the attention map of each slot as the pixel-wise
weight map of corresponding illuminant (Fig. 1 bottom). By doing so, we can
enforce each predicted pixel-wise chromaticity to be a linear combination of
the slot chromaticities, and enable illuminant-wise tunability. The way our
model generates the final illumination maps follows the linearity constraint
so that our method can properly tackle the problem of spatially varying WB.
Furthermore, we propose a novel loss called centroid-matching loss, to
effectively train our slot attention based model by assigning specific color
ranges to slots.

We validate the robustness of AID framework through comprehensive experiments
conducted on various datasets, including the LSMI dataset [27], the Multi-
Illumination In the Wild dataset [34], and the well-established single-
illuminant dataset, NUS-8 [11]. The experimental results consistently
demonstrate superior performance compared to previous models, achieving the
state-of-the-art performance across all of the aforementioned datasets.

Our contributions can be summarized as follows:

  * •

By successfully leveraging the concept of the slot attention, we propose a
novel end-to-end framework AID, which can infer the chromaticities of
illuminants and their pixel-level weight maps separately.

  * •

We introduce the centroid-matching loss to enable more effective updates of
slots to represent specific color gamuts.

  * •

Our model not only demonstrates the state-of-the-art performance in both
single- and multi-illuminant white balance scenarios but also provides tunable
WB, thanks to its capacity to generate fully disentangled illumination maps.

##  2 Related work

###  2.1 Computational color constancy

#### Single-illuminant WB.

Classical statistics-based algorithms utilizing image statistics have been
studied [10, 30, 15, 43] for computational color constancy. Additionally,
numerous WB datasets [13, 17, 40, 11] have been proposed for data-driven
research. Methodologies have been introduced involving the learning of kernels
to detect illuminant chromaticity in the uv-histogram space [3, 4], utilizing
convolutional features [7, 41, 24, 36], and employing various learning
techniques [32, 45, 37, 46]. In particular, FC4 [24] employs a form of
attention technique by inferring spatial weighting coefficients, rather than
uniformly using all spatial features within the image. On the other hand, C4
[46] demonstrated the capability for more accurate chromaticity inference
through iterative refinement process. While they achieve impressive results
for single-illuminant WB, they cannot address the multi-illuminant WB cases.
We found that the incorporation of spatial attention maps and an iterative
refinement strategy, in conjunction with the concept of slots outlined in Sec.
2.2, is highly suitable for addressing the spatially varying multi-illuminant
decomposition task.

#### Multi-illuminant WB.

To solve the multi-illuminant WB problem, several studies have been conducted
to utilize additional prior information such as the chromaticity of the
illuminant [23, 5], flash photography [25], and human face [6]. Approaches
that apply single-illuminant WB in a spatially varying form have been
introduced in [9, 21], and a graph structure reflecting the characteristics of
spatially varying WB has been utilized in [35].

Following small-scale multi-illuminant datasets [9, 21, 5, 8] for testing
spatially varying WB algorithms, several large scale multi-illuminant datasets
have been captured [34, 27] and synthesized [22] recently. Deep learning-based
strategies such as using GANs [42], and leveraging transformer blocks with
multi-task learning strategies [31] have also been explored.

The base architecture for previous multi-illuminant WB used the encoder-
decoder structure to directly predict the chromaticity of illumination for
each individual pixel. These models fall short in estimating and incorporating
the individual chromaticities of illuminants present in the scene, leading to
inconsistencies in the generated illumination map. (Fig. 1 top). While a model
that estimates pixel-wise weights for pre-specified WB presets [2] has been
proposed recently, the resulting weight maps do not accurately reflect the the
ground-truth illuminant-wise mixing ratio due to its reliance on pre-defined
WB presets. A summary of the comparison between our framework and previous
works is presented in Table 1.

Models |  | Mixed
---
illumination
| Decomposed
---
illumination map
| Controllable
---
WB
Single AWB | $\times$ | $\times$ | $\times$
| Multi-AWB
---
[2, 42, 31]
$\checkmark$ | $\times$ | $\times$
AID (Ours) | $\checkmark$ | $\checkmark$ | $\checkmark$

Table 1: Comparison between previous WB methods and our AID framework. AID
predicts a decomposed illumination map, enabling the inference of individual
illuminant chromaticity and the number of illuminants in a scene. This new
feature enables controllable WB, allowing for individual adjustment of the
color of each illuminant in a scene.

###  2.2 Slot attention

Slot attention [33] was introduced to solve the object-centric learning (OCL),
where the model needs to cluster and compute the representation of objects
from a given scene without any human-annotated labels in an autoencoding
manner. Slot attention employs the concept of the slots, a set of vectors,
each of which captures the representation of the object in a scene. Slots are
initialized using Gaussian random sampling and are subsequently evolved to
capture the representations of objects. Dot-product based attention maps
between slots and encoded visual feature maps are used for updating the slots.
By applying slot-wise softmax mechanism on the attention map, slots are forced
to compete with each other to get more task-relevant representation, i.e.
object-centric representation.

Due to the decomposition ability of the slot attention, it has been widely
applied to various domains in computer vision such as object discovery [33,
14, 29, 28], novel view synthesis [39], panoptic segmentation [47], and visual
question answering [44]. Slot attention acts like a soft k-means clustering,
where the slots are appropriately updated to represent the target sub-element.
In this work, we adopt slot attention to the task of multi-illuminant white
balancing, enforcing slots to implicitly represent individual illuminants. In
addition, we introduce a novel loss function named centroid matching loss,
aimed at preventing all slots from indiscriminately contributing to the
inference. This improves illumination decomposition accuracy by allocating the
specific color ranges to each slot.

##  3 Method

Figure 2: (a) Overview of our framework. Image feature is extracted from the
input using an U-Net encoder. Next, the slot attention adaptively calibrates
slot representation to be bound with illuminant chromaticity in each scene.
Finally, the model fuses the chromaticity and the weight map to generate the
mixed illumination map. (b) Detailed generation flow of weight maps and
calibrated slots, where Q-Softmax denotes softmax application on the query
dimension. (c) Illustration of the slot-wise loss using the centroid based
Hungarian matching under $K=4,N=2$ assumption.

###  3.1 Image formation model

In the Lambertian image model, the RGB value of each pixel under single-
illuminant condition can be represented as follows:

$I=kR\circ\boldsymbol{\ell}.$ (1)

Here $I$, $R$ and $\boldsymbol{\ell}$ are $3\times 1$ vectors for observed RGB
pixel, surface reflectance, and normalized illuminant chromaticity,
respectively. The $\circ$ symbol represents element-wise product and the
scalar $k$ represents the integrated scaling term of illumination including
the power of illuminant and surface normal. In this paper, we normalize the
illuminant chromaticity so that the value of the green channel becomes 1.
Previous works [23, 21, 27] suggest that if multiple illuminants are present
in a scene, the chromaticity of mixed illumination can be represented by the
linear combination of the chromaticity of each illuminant. This property has
been used to calculate per-pixel illumination labels for multi-illuminant
datasets [5, 27].

Under the imaging model, the illumination chromaticity value
$\boldsymbol{\ell}$ on a given location $x$ of a single or multiple illuminant
scene can be generalized and expressed as follows:

$\displaystyle\boldsymbol{\ell}_{mixed}(x)=$
$\displaystyle\sum_{k=1}^{N}\alpha_{k}(x)\boldsymbol{\ell}_{k},$ (2) where
$\displaystyle\sum_{k=1}^{N}\alpha_{k}(x)=1,\quad\boldsymbol{\ell}_{k}=\begin{bmatrix}R_{k}\\\
B_{k}\end{bmatrix}.$

$\alpha_{k}$ and $\boldsymbol{\ell}_{k}$ represent the weight map and the
normalized chromaticity of illuminant $k$, respectively, and $N$ is the number
of illuminants in the scene. As mentioned earlier, we only consider the R and
B channels of the illuminant chromaticity $\boldsymbol{\ell}_{k}$, given that
the G channel is normalized to 1. Since $\boldsymbol{\ell}_{k}$ is the
chromaticity of each light source, it does not change with pixel location, and
only the weight map $\alpha$ is dependent on $x$.

###  3.2 Attentive illumination decomposition

To solve the multi-illuminant WB, we design the Attentive Illumination
Decomposition (AID) framework, which follows the imaging model described in
Eq. (2). The proposed method first predicts the weight map $\alpha_{k}$ and
the chromaticity $\boldsymbol{\ell}_{k}$ of each illuminant in the scene, and
then the mixed illumination map $\boldsymbol{\ell}_{mixed}$ for WB is
generated using Eq. (2). It is important to highlight that this is the first
approach to separately predict the chromaticity and the weight map of
illuminant for multi-illuminant WB, leading to enhancement in performance.

To obtain $\alpha_{k}$ and $\boldsymbol{\ell}_{k}$, our framework utilizes the
slot attention [33]. Different from the existing slot attention models, where
slots are typically designed to capture object-level features, we design the
model to make the slots to represent illuminant-level information. More
precisely, each slot in our model allows us to infer both the chromaticity and
the weight map associated with the corresponding illuminant. The overview of
our framework is illustrated in Fig. 2(a). Our model consists of three parts:
1) image feature extraction, 2) iterative slot calibration process using slot
attention, and 3) weight map & illuminant chromaticity fusion.

#### Image feature extraction.

For a given raw image $\mathbf{I}$ with the resolution $H\times W$, the
feature encoder $E$ extracts a latent feature $feat$ with the same spatial
resolution as the image and $D_{slot}$ channels:

$feat:=E(\mathbf{I})\in\mathbb{R}^{HW\times D_{slot}},$ (3)

where $D_{slot}$ represents the dimension of slots.

#### Iterative slot calibration by slot attention.

After extracting the image features, the iterative slot attention module (Fig.
2(b)) is applied to calibrate representations of the illuminant chromaticity.
The details of the calibration process are as follows.

First, $slots_{\scaleto{0}{3.1pt}}$ $\in\mathbb{R}^{K\times D_{slot}}$ are
initialized as learnable parameters where $K$ indicates the number of slots.
The slot attention module takes the image feature $feat$ and the initialized
$slots$ as inputs to produce attention map $attn$:

$\begin{gathered}attn_{i,j}:=\frac{\mathrm{exp}(M_{i,j})}{\sum_{l}{\mathrm{exp}(M_{i,l})}},\quad\textrm{where}\\\
M:=\frac{1}{\sqrt{D_{slot}}}k(feat)\cdot
q(slots_{\scaleto{n}{3pt}})^{T}\in\mathbb{R}^{HW\times K},\end{gathered}$ (4)

where $k$ and $q$ are MLPs for generating the key and query representations in
$D_{slot}$ dimension, and $slots_{n}$ represents the state of the slots in the
n-th iteration.

Subsequently, the intermediate representation vectors, $updates$, are computed
by aggregating the $values$ through spatially normalized attention map $W$:

$\begin{gathered}updates:=W^{T}\cdot v(feat)\in\mathbb{R}^{K\times
D_{slot}},\\\ \textrm{where}\quad
W_{i,j}:=\frac{attn_{i,j}}{\sum_{l=1}^{N}attn_{l,j}},\end{gathered}$ (5)

where $v$ is MLPs for generating value representation.

Finally, the calibrated $slots_{n+1}$ are refined by the GRU [12], which takes
$updates$ as input and previous $slots_{n}$ as hidden state:

$\begin{gathered}slots_{\scaleto{n+1}{5pt}}=GRU(slots_{{\scaleto{n}{3pt}}},updates_{\scaleto{n}{3pt}}).\end{gathered}$
(6)

The process from Eq. (4) to Eq. (6) is repeated $T$ times to generate final
calibrated slots, $slots_{T}$.

#### Weight map & Illuminant chromaticity fusion.

In our framework, we have carefully designed the output tensors of the slot-
attention module, $slots$ and $attn$, to represent the chromaticity of
illuminants $\boldsymbol{\ell}$ and the weight map $\alpha$, respectively.
Specifically, the $HW\times K$ shaped tensor $attn$, represents the pixel-wise
similarity score between $feat$ and each $slot$, enabling its direct use as
the set of $K$ weight maps for each illuminant
($\hat{\alpha}_{1}\dots\hat{\alpha}_{K}$). Also, chromaticities for $K$
illuminants ($\boldsymbol{\hat{\ell}}_{1}\dots\boldsymbol{\hat{\ell}}_{K}$)
can be generated through passing calibrated $slots_{\scaleto{T}{4pt}}$ to
chromaticity conversion MLPs $c$, where $c(slots_{\scaleto{T}{4pt}})$ is a
$K\times 2$ shaped tensor. Finally, we can fuse these two tensors to make
mixed illumination map $\boldsymbol{\hat{\ell}}_{mixed}$ according to Eq. (2),
by simply multiplicating them:

$\begin{gathered}\boldsymbol{\hat{\ell}}_{mixed}=\sum_{k=1}^{K}\hat{\alpha}_{k}\boldsymbol{\hat{\ell}}_{k}=attn\cdot
c(slots_{T}).\end{gathered}$ (7)

Although the above equation utilizes the final calibrated slots, $slots_{T}$,
it is noteworthy that we can visualize the chromaticity
$\boldsymbol{\hat{\ell}}_{k}$ and weight map $\hat{\alpha}_{k}$ for each
iteration by employing the respective $slots_{n}$ of iteration $n$. Fig. 3
demonstrates how the generated illuminant chromaticity
$\boldsymbol{\hat{\ell}}_{k}$ and weight map $\hat{\alpha}_{k}$ changes as
$slots$ are itaratively calibrated.

Figure 3: Slot calibration process. The chromaticity
$\boldsymbol{\hat{\ell}}_{k}$ and weight map $\hat{\alpha}_{k}$ generated from
each $slots_{n}$ are iteratively calibrated to their ground truth values.

###  3.3 Loss functions

AID framework is trained with two types of loss function: 1) mixed
illumination loss, and 2) slot-wise loss, named as centroid-matching loss. The
total training objective is to minimize the sum of these two loss functions.

#### Mixed illumination loss.

Mixed illumination loss $\mathcal{L}_{mixed}$ is simply defined by L1 distance
between the predicted mixed illumination map $\boldsymbol{\hat{\ell}}_{mixed}$
and the ground truth $\boldsymbol{\ell}_{mixed}$:

$\mathcal{L}_{mixed}=\left|\boldsymbol{\ell}_{mixed}-\boldsymbol{\hat{\ell}}_{mixed}\right|.$
(8)

#### Centroid-based matching loss.

The mixed illumination loss alone does not provide a sufficient constraint
that ensures our model activates the appropriate number of slots. Instead, it
may result in the activation of either all slots or a random number of slots.
As depicted in Fig. 2(a), the scene involves two illuminants, yet the model
employs four slots to generate a mixed illumination map. Hence, it is
necessary to strategically select and supervise the slots which are aligned
with the ground-truth chromaticity and weight map. In this context, we design
the loss term based on two assumptions that 1) each slot possesses its pre-
defined cluster (color-gamut), and 2) activation of the slot should occur when
the ground truth chromaticity lies within its cluster boundary. To this end,
we propose the centroid matching loss and the calculation process of this loss
is presented in Fig. 2(c).

First let us denote a pre-calculated set of centroids as
$\mu=\\{\mu_{i}\\}_{1}^{K}$, obtained by applying K-means algorithm on the
illuminant chromaticity distribution of the dataset. These centroids serve as
the centerpoints of each illuminant chromaticity cluster, and in AID
framework, each slot is responsible for representing one of these clusters.
Next, we obtain a set of matched indices $\sigma_{m}$ that minimizes the L1
cost between the matched centroid chromaticitiy $\mu$ and the ground truth
chromaticity $\boldsymbol{\ell}$:

$\sigma_{m}=\underset{\sigma}{argmin}\sum_{i}^{N}\left|\boldsymbol{\ell}_{i}-\mu_{\sigma(i)}\right|,$
(9)

where N is the number of ground-truth illuminants in each scene and $\sigma$
is one of the combinations of N elements from the set $\\{1,\cdots,K\\}$. Now
we can define the loss term with respect to the chromaticity and weight map of
each matched slots as follows:

$\begin{gathered}\leavevmode\resizebox{413.38667pt}{}{
$\mathcal{L}_{centroid}=\sum_{i}\left(\left|\boldsymbol{\ell}_{i}-\boldsymbol{\hat{\ell}}_{\sigma_{m}(i)}\right|+\left|\alpha_{i}-\hat{\alpha}_{\sigma_{m}(i)}\right|\right)$,}\end{gathered}$
(10)

where the centroid-matching loss $\mathcal{L}_{centroid}$ consists of both L1
loss for chromaticity and weight map of the matched slot indices. Here, the
predicted weight map $\hat{\alpha}_{k}$ and the illuminant chromaticity
$\hat{\boldsymbol{\ell}}_{k}$, are the k-th channel of $attn$ and
$c(slots_{T})$, as previously shown in Eq. (7).

##  4 Experiments

###  4.1 Experimental setup

We validate the multi-illuminant WB performance of AID using two datasets: the
LSMI dataset [27] captured with three cameras having different bit-depths and
spectral sensitivities, and the Multi-Illumination In the Wild dataset (MIIW)
[34], which is a versatile dataset covering various illumination-related
tasks, including multi-illuminant WB.

We use seven slots ($K=7$), 64 latent channels for $D_{slot}$, and calibrate
slots three times ($T=3$). For the evaluation, the green channel was inserted
(G=1) to the mixed illumination map $\boldsymbol{\hat{\ell}}_{mixed}$, and the
mean angular error (MAE) in degree was calculated with respect to the ground
truth illumination map. For more detailed information, please refer to the
supplementary materials.

###  4.2 Spatially varying white balance

|  | mean | median
---|---|---|---
(a) | AWB [1] † | 9.54 | 8.19
Patch CNN [8] † | 4.82 | 4.24
AngularGan [42] † | 4.69 | 3.88
TransCC [31] † | 2.78 | 2.15
LSMI-U [27] | 2.31 | 1.89
AID | 2.04 | 1.73
AID + MDL | 1.93 | 1.60
|  | mean | median
|  | gal. | son. | nik. | gal. | son. | nik.
(b) | LSMI-H [27] | 3.06 | 3.21 | 2.99 | 2.54 | 2.89 | 2.61
LSMI-U [27] | 2.68 | 2.15 | 1.92 | 2.17 | 1.74 | 1.54
AID | 1.66 | 1.66 | 1.71 | 1.41 | 1.35 | 1.34

Table 2: Mean angular error (MAE) for the spatially varying illumination map
on LSMI dataset: (a) all-in-one (cross-camera), (b) device-specific. †
indicates that the results of [31] are referenced.

LSMI [27] | Galaxy | Sony | Nikon
---|---|---|---
single | multi | single | multi | single | multi
LSMI-U [27] | 2.85 | 2.55 | 1.92 | 2.34 | 1.49 | 2.30
AID | 1.19 | 2.03 | 1.01 | 2.16 | 1.11 | 2.26

Table 3: Average MAE values obtained through experiments on the LSMI test set,
distinguishing between single and multi-illuminant scenarios.

MIIW [34] | Single (1) | Multi (2,3) | Mixed (1,2,3)
---|---|---|---
mean | median | mean | median | mean | median
LSMI-U [27] | 4.15 | 2.39 | 4.34 | 3.87 | 4.28 | 3.54
AID | 1.07 | 0.73 | 3.14 | 2.81 | 2.46 | 2.11

Table 4: MAE values for predicting single-, multi-, and mixed-illuminant
scenario using the MIIW test set.

#### Quantitative comparison.

For the LSMI dataset, we evaluated AID under two settings to show the
robustness of the proposed method: all-in-one cross-camera and device-specific
setting. As shown in Table 2(a), AID achieves the state-of-the-art performance
compared to all previous models in LSMI dataset. Here, we would like to inform
that LSMI-H and LSMI-U are the preceding state-of-the-art models introduced in
the LSMI dataset. LSMI-H employs HDR-Net [18], while LSMI-U utilizes U-Net
[38].

Moreover, as AID uses a concept of slots as an intermediate representation of
the illumination, the model can be easily extended to multi-domain learning
(MDL). We simply make the slot initialization different depending on the
camera model (AID + MDL) and this slight modification brings additional 5%
performance enhancement. Furthermore, in the camera-specific setting (Table 2
(b)), AID outperforms the LSMI baselines for all three cameras.

Since the LSMI dataset consists of one- to three-illuminant scenes, we also
tested device-specific models with single and multi (two to three) illuminant
subset, separately. As illustrated in Table 3, our framework consistently
outperforms the LSMI baseline [27] across all devices in both single- and
multi-illuminant settings. For the single-illuminant case, we could observe
that only one slot is activated among 7 slots, and produces near-perfect
global uniform illumination, resulting in a significant performance
improvement over LSMI baseline.

We further demonstrate the robustness of our framework using another large-
scale dataset, Multi-Illumination in the Wild [34]. Since no other algorithms
have been applied to MIIW dataset previously, we select LSMI-U as our
baseline. Table 4 demonstrates that AID outperforms LSMI-U, which had
previously shown the best performance on the LSMI dataset, further
highlighting the superior performance of AID.

Figure 4: Qualitative comparison using LSMI test set. Top three rows show
original raw image and corresponding WB results. The last two rows show the
sRGB input images and corresponding illumination maps. The two rightmost
columns demonstrate that our model, which infers illuminant-wise chromaticity
and spatially mixes them, leads to more stable illumination plots compared to
previous approaches. The x-axis and y-axis of the plot represent the ratio of
the illumination value of the R and B channels to the value of the G channel.
Figure 5: Further applications of AID framework on LSMI test set examples. The
separated weight map and the corresponding illuminant chromaticity (Decomp)
allow for individual white balance to be applied to each light (WB Illum1,2),
and for the chromaticity to be adjusted as desired (Illum manip). Full WB
shows the results of applying white balance to all illuminants for reference.
Gamma was adjusted for all images to increase visibility, and the G channel
was scaled down for the decomposed illumination map visualization.

#### Qualitative comparison.

Fig. 4 illustrates the qualitative comparison between LSMI-U [27] and our
method AID. For better visibility, we apply the following post-processing: 1)
convert the result images in the top three rows and input images in the bottom
two rows to the sRGB color space, and 2) scale down the green channel of the
illumination maps in the bottom two rows. Our model generates more natural and
ground truth-like WB results and illumination maps compared to the LSMI-U. We
contribute the improvement of AID to the model design where the final
illumination maps are generated under the condition of the physical image
model Eq. (2), and also to the proposed loss function that matches the
predictions to the proper ground truths.

We also provide the plots of chromaticity of the pixel-wise illumination
predictions with the ground truths. Through the ground-truth illumination
distributions of the top three rows, illustrated in red, it can be confirmed
that each scene has a single, dual, or triple illuminant, respectively
represented as a point, a line segment, and a triangle. It can be easily
notified that the previous model generates unrefined predictions whereas our
model performs well on reconstructing the actual distribution of the
chromaticities of the illumination. For additional visualizations, including
results related to the MIIW dataset, please refer to the supplementary
material.

###  4.3 Generalization using single-illuminant DB

We also assessed the generalizability of our framework using the established
single-illuminant white balance dataset, NUS-8 [11], which is a well-known
benchmark widely used in the literature. NUS-8 dataset comprises 8 camera
subsets, and we conducted three-fold cross-validation experiment for each
camera. We measured the following metrics in the same way as previous studies
[3, 4, 24]: mean, median, tri-mean, best 25%, worst 25%, and their geometric
mean (G.M.). For the model configuration, we use $K=5$, $T=3$, and
$D_{slot},D_{attn}=64$.

In Table 5, we report the angular error in degrees, along with the performance
of recent works. The result shows that the proposed framework works robustly
under both single- and multi-illuminant environments.

NUS-8 [11] | Mean | Med. | Tri. |  | Best
---
25%
| Worst
---
25%
G.M.
CCC [3] | 2.38 | 1.48 | 1.69 | 0.45 | 5.85 | 1.73
AlexNet-FC4 [24] | 2.12 | 1.53 | 1.67 | 0.48 | 4.78 | 1.66
FFCC [4] | 1.99 | 1.31 | 1.43 | 0.35 | 4.75 | 1.44
CLCC [32] | 1.84 | 1.31 | 1.42 | 0.41 | 4.2 | 1.42
AID | 1.57 | 1.03 | 1.16 | 0.37 | 3.67 | 1.21

Table 5: Three-fold cross-validation result on NUS-8 dataset, with mean
angular error in degrees.

| # of illum acc. | illuminant AE
---|---|---
| mean | median
Galaxy | 0.800 | 1.71 | 1.25
Sony | 0.871 | 1.50 | 0.86
Nikon | 0.813 | 1.84 | 1.20

Table 6: Additional validation metrics. We measured 1) the accuracy of
predicted number of illuminants in the scene and 2) the angular error (AE)
between predicted and the GT chromaticities.

###  4.4 Fully decomposed multi-illuminant WB

Since our model generates fully decomposed illumination map, we can calculate
the number of illuminants or the prediction accuracy of individual
illuminant’s chromaticity.

#### Count & chromaticity prediction result.

We can also evaluate the accuracy of the predicted number of illuminants in
the scene and the angular error of the chromaticities of each individual
illuminant, using decomposed illumination map. The number of illuminants was
measured by ignoring slots where the maximum value of the weight map component
was below the threshold of 0.3. Angular error of illuminant chromaticity was
measured between chromaticity vectors with matched indices $\sigma_{m}$ and
their corresponding GT vectors. Such additional information could be utilized
as an additional metric for how well the model understands and accurately
decomposes the multi-illuminant scene. Table 6 demonstrates that AID
accurately predicts the chromaticity and the number of each illuminant. One
thing to note is that it is impossible to measure this decomposition
performance in previous works as ours is the first work to enable this
illumination decomposition in multi-illuminant scenes.

#### Controllable multi-illuminant WB.

Unlike previous multi-illuminant WB methods, AID can make fully-decomposed
results with illuminant-wise chromaticities and weight maps. Therefore, we can
leverage these decomposed information to provide additional features like
manipulating the chromaticity of each light or selective WB. Fig. 5 shows
additional capabilities of AID framework.

###  4.5 Ablation study

As shown in the Table 7, we present three different ablation studies:
centroid-matching loss ($\mathcal{L}_{m}$), the number of slots ($K$), and the
number of iterations in the slot attention module ($T$). The first section of
the table shows that the centroid-based matching loss helps the model to
decompose the mixed illumination with the proper number of slots, as
demonstrated by the illuminant number prediction accuracy (# acc.). Absence of
$\mathcal{L}_{centroid}$ resulted in failure to effectively decompose mixed
illumination using slots, as all slots were indiscriminately engaged to
estimate the illumination, yielding an decomposition accuracy of 0.288. The
efficacy of the centroid matching loss is more clearly demonstrated in Section
C and Fig. 11 of the supplementary material. In addition, the second and the
third section of the study reveals that the model performance can deliver
different results depending on the number of slots ($K$) and the number of
iterations in the slot attention module ($T$).

$\mathcal{L}_{mixed}$ | $\mathcal{L}_{centroid}$ | $K$ | $T$ | Mixed illum MAE | Illuminant
---|---|---|---|---|---
mean | median | # acc. | AE
✓ |  | 7 | 3 | 1.58 | 1.26 | 0.288 | -
✓ | ✓ | 7 | 3 | 1.66 | 1.41 | 0.800 | 1.71
✓ | ✓ | 5 | 3 | 1.82 | 1.37 | 0.744 | 2.40
✓ | ✓ | 7 | 3 | 1.66 | 1.41 | 0.800 | 1.71
✓ | ✓ | 9 | 3 | 1.84 | 1.42 | 0.488 | 1.49
✓ | ✓ | 7 | 2 | 1.85 | 1.46 | 0.688 | 1.79
✓ | ✓ | 7 | 3 | 1.66 | 1.41 | 0.800 | 1.71
✓ | ✓ | 7 | 4 | 1.92 | 1.44 | 0.720 | 1.84

Table 7: Results of ablation studies on the centroid-matching loss
($\mathcal{L}_{m}$), the number of slots ($K$), and the number of iterations
of GRU ($T$) using the Galaxy camera subset of the LSMI dataset.

Among the combinations of $K$ and $T$, we choose to use $K=7$ and $T=3$
combination by considering the accuracy and the computational cost. Ablation
studies are conducted using the Galaxy camera subset of the LSMI dataset.

##  5 Conclusion and discussion

In this paper, we introduced a framework called AID, designed to extract the
chromaticity of individual illuminants along with their corresponding weights,
while satisfying the linearity constraint of the Lambertian image model. To
construct our model, we incorporated the slot attention module and applied the
centroid-based matching loss, extending upon previous multi-illuminant white
balance methods.

We demonstrated the effectiveness of AID through various experiments, and we
believe this marks as a step towards more interpretable image enhancement,
particularly in the context of white balancing. However, we acknowledge
limitations in our proposed method, such as the requirement for presets
regarding the number of clusters. Building model that can dynamically
determine the number of clusters based on input images can be a promising path
for future research.

## References

  * Afifi et al. [2019] Mahmoud Afifi, Brian Price, Scott Cohen, and Michael S Brown.  When color constancy goes wrong: Correcting improperly white-balanced images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1535–1544, 2019.
  * Afifi et al. [2022] Mahmoud Afifi, Marcus A Brubaker, and Michael S Brown.  Auto white-balance correction for mixed-illuminant scenes.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1210–1219, 2022.
  * Barron [2015] Jonathan T Barron.  Convolutional color constancy.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 379–387, 2015.
  * Barron and Tsai [2017] Jonathan T Barron and Yun-Ta Tsai.  Fast fourier color constancy.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 886–894, 2017.
  * Beigpour et al. [2013] Shida Beigpour, Christian Riess, Joost Van De Weijer, and Elli Angelopoulou.  Multi-illuminant estimation with conditional random fields.  _IEEE Transactions on Image Processing (TIP)_ , 23(1):83–96, 2013.
  * Bianco and Schettini [2014] Simone Bianco and Raimondo Schettini.  Adaptive color constancy using faces.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 36(8):1505–1518, 2014.
  * Bianco et al. [2015] Simone Bianco, Claudio Cusano, and Raimondo Schettini.  Color constancy using cnns.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPR - Workshop)_ , pages 81–89, 2015.
  * Bianco et al. [2017] Simone Bianco, Claudio Cusano, and Raimondo Schettini.  Single and multiple illuminant estimation using convolutional neural networks.  _IEEE Transactions on Image Processing (TIP)_ , 26(9):4347–4362, 2017.
  * Bleier et al. [2011] Michael Bleier, Christian Riess, Shida Beigpour, Eva Eibenberger, Elli Angelopoulou, Tobias Tröger, and André Kaup.  Color constancy and non-uniform illumination: Can existing algorithms work?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop (ICCV - Workshop)_ , pages 774–781. IEEE, 2011.
  * Buchsbaum [1980] Gershon Buchsbaum.  A spatial processor model for object colour perception.  _Journal of the Franklin institute_ , 310(1):1–26, 1980.
  * Cheng et al. [2014] Dongliang Cheng, Dilip K Prasad, and Michael S Brown.  Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution.  _Journal of the Optical Society of America A (JOSA A)_ , 31(5):1049–1058, 2014.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _Advances in Neural Information Processing Systems Workshop (NeurIPS - Workshop)_ , 2014.
  * Ciurea and Funt [2003] Florian Ciurea and Brian Funt.  A large image database for color constancy research.  In _Color and Imaging Conference_ , pages 160–164. Society for Imaging Science and Technology, 2003.
  * Engelcke et al. [2019] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner.  Genesis: Generative scene inference and sampling with object-centric latent representations.  _arXiv preprint arXiv:1907.13052_ , 2019.
  * Finlayson and Trezzi [2004] Graham D Finlayson and Elisabetta Trezzi.  Shades of gray and colour constancy.  In _Color and Imaging Conference_ , pages 37–41. Society for Imaging Science and Technology, 2004.
  * Forsyth [1990] David A Forsyth.  A novel algorithm for color constancy.  _International Journal of Computer Vision (IJCV)_ , 5(1):5–35, 1990.
  * Gehler et al. [2008] Peter Vincent Gehler, Carsten Rother, Andrew Blake, Tom Minka, and Toby Sharp.  Bayesian color constancy revisited.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–8. IEEE, 2008.
  * Gharbi et al. [2017] Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand.  Deep bilateral learning for real-time image enhancement.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–12, 2017.
  * Gijsenij et al. [2010] Arjan Gijsenij, Theo Gevers, and Joost Van De Weijer.  Generalized gamut mapping using image derivative structures for color constancy.  _International Journal of Computer Vision (IJCV)_ , 86(2-3):127–139, 2010.
  * Gijsenij et al. [2011a] Arjan Gijsenij, Theo Gevers, and Joost Van De Weijer.  Improving color constancy by photometric edge weighting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 34(5):918–929, 2011a.
  * Gijsenij et al. [2011b] Arjan Gijsenij, Rui Lu, and Theo Gevers.  Color constancy for multiple light sources.  _IEEE Transactions on Image Processing (TIP)_ , 21(2):697–707, 2011b.
  * Hao and Funt [2020] Xiangpeng Hao and Brian Funt.  A multi-illuminant synthetic image test set.  _Color Research & Application_, 45(6):1055–1066, 2020.
  * Hsu et al. [2008] Eugene Hsu, Tom Mertens, Sylvain Paris, Shai Avidan, and Frédo Durand.  Light mixture estimation for spatially varying white balance.  In _ACM SIGGRAPH_ , pages 1–7, 2008.
  * Hu et al. [2017] Yuanming Hu, Baoyuan Wang, and Stephen Lin.  Fc4: Fully convolutional color constancy with confidence-weighted pooling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4085–4094, 2017.
  * Hui et al. [2016] Zhuo Hui, Aswin C Sankaranarayanan, Kalyan Sunkavalli, and Sunil Hadap.  White balance under mixed illumination using flash photography.  In _International Conference on Computational Photography (ICCP)_ , pages 1–10. IEEE, 2016.
  * Hui et al. [2018] Zhuo Hui, Kalyan Sunkavalli, Sunil Hadap, and Aswin C Sankaranarayanan.  Illuminant spectra-based source separation using flash photography.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6209–6218, 2018.
  * Kim et al. [2021] Dongyoung Kim, Jinwoo Kim, Seonghyeon Nam, Dongwoo Lee, Yeonkyung Lee, Nahyup Kang, Hyong-Euk Lee, ByungIn Yoo, Jae-Joon Han, and Seon Joo Kim.  Large scale multi-illuminant (lsmi) dataset for developing white balance algorithm under mixed illumination.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 2410–2419, 2021.
  * Kim et al. [2023] Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim.  Shepherding slots to objects: Towards stable and robust object-centric learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 19198–19207, 2023.
  * Kipf et al. [2021] Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff.  Conditional object-centric learning from video.  _arXiv preprint arXiv:2111.12594_ , 2021.
  * Land [1977] Edwin H Land.  The retinex theory of color vision.  _Scientific american_ , 237(6):108–129, 1977\.
  * Li et al. [2022] Shuwei Li, Jikai Wang, Michael S Brown, and Robby T Tan.  Transcc: Transformer-based multiple illuminant color constancy using multitask learning.  _arXiv preprint arXiv:2211.08772_ , 2022.
  * Lo et al. [2021] Yi-Chen Lo, Chia-Che Chang, Hsuan-Chao Chiu, Yu-Hao Huang, Chia-Ping Chen, Yu-Lin Chang, and Kevin Jou.  Clcc: Contrastive learning for color constancy.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8053–8063, 2021.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.  Object-centric learning with slot attention.  _Advances in Neural Information Processing Systems_ , 33:11525–11538, 2020.
  * Murmann et al. [2019] Lukas Murmann, Michael Gharbi, Miika Aittala, and Fredo Durand.  A dataset of multi-illumination images in the wild.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4080–4089, 2019.
  * Mutimbu and Robles-Kelly [2016] Lawrence Mutimbu and Antonio Robles-Kelly.  Multiple illuminant color estimation via statistical inference on factor graphs.  _IEEE Transactions on Image Processing (TIP)_ , 25(11):5383–5396, 2016.
  * Oh and Kim [2017] Seoung Wug Oh and Seon Joo Kim.  Approaching the computational color constancy as a classification problem through deep learning.  _Pattern Recognition (PR)_ , 61:405–416, 2017.
  * Qian et al. [2017] Yanlin Qian, Ke Chen, Jarno Nikkanen, Joni-Kristian Kamarainen, and Jiri Matas.  Recurrent color constancy.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5458–5466, 2017.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , pages 234–241. Springer, 2015.
  * Sajjadi et al. [2022] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J Guibas, Klaus Greff, and Thomas Kipf.  Object scene representation transformer.  _arXiv preprint arXiv:2206.06922_ , 2022.
  * Shi [2000] Lilong Shi.  Re-processed version of the gehler color constancy dataset of 568 images.  _http://www. cs. sfu. ca/~ color/data/_ , 2000.
  * Shi et al. [2016] Wu Shi, Chen Change Loy, and Xiaoou Tang.  Deep specialized network for illuminant estimation.  In _Proceedings of Proceedings of European Conference on Computer Vision (ECCV)_ , pages 371–387. Springer, 2016.
  * Sidorov [2019] Oleksii Sidorov.  Conditional gans for multi-illuminant color constancy: Revolution or yet another approach?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPR - Workshop)_ , pages 0–0, 2019.
  * Van De Weijer et al. [2007] Joost Van De Weijer, Theo Gevers, and Arjan Gijsenij.  Edge-based color constancy.  _IEEE Transactions on Image Processing (TIP)_ , 16(9):2207–2214, 2007.
  * Wang et al. [2020] Ruocheng Wang, Jiayuan Mao, Samuel J Gershman, and Jiajun Wu.  Language-mediated, object-centric representation learning.  _arXiv preprint arXiv:2012.15814_ , 2020.
  * Xu et al. [2020] Bolei Xu, Jingxin Liu, Xianxu Hou, Bozhi Liu, and Guoping Qiu.  End-to-end illuminant estimation based on deep metric learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3616–3625, 2020.
  * Yu et al. [2020] Huanglin Yu, Ke Chen, Kaiqi Wang, Yanlin Qian, Zhaoxiang Zhang, and Kui Jia.  Cascading convolutional color constancy.  In _AAAI Conference on Artificial Intelligence (AAAI)_ , pages 12725–12732, 2020.
  * Zhou et al. [2022] Yi Zhou, Hui Zhang, Hana Lee, Shuyang Sun, Pingjun Li, Yangguang Zhu, ByungIn Yoo, Xiaojuan Qi, and Jae-Joon Han.  Slot-vps: Object-centric representation learning for video panoptic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3093–3103, 2022.