# Simple geometric mitosis

Valentina Kiritchenko<EMAIL_ADDRESS>Laboratory of Algebraic Geometry and
Faculty of Mathematics
National Research University Higher School of Economics
Usacheva str. 6, 119048 Moscow, Russia  Institute for Information Transmission
Problems, Moscow, Russia

###### Abstract.

We construct simple geometric operations on faces of the Cayley sum of two
polytopes. These operations can be thought of as convex geometric counterparts
of divided difference operators in Schubert calculus. We show that these
operations give a uniform construction of Knutson–Miller mitosis (in type $A$)
and (simplified) Fujita mitosis (in type $C$) on Kogan faces of Gelfand–Zetlin
polytopes.

###### Key words and phrases:

Schubert calculus, mitosis, push-pull operator

The study has been partially funded within the framework of the HSE University
Basic Research Program.

##  1\. Introduction

Mitosis as a mathematical notion was introduced by Knutson and Miller to
generate combinatorial objects (reduced pipe dreams or rc-graphs) from a
single object [KnM, M]. Originally, pipe dreams were used in Fomin–Kirillov
theorem to enumerate the coefficients of Schubert polynomials in type $A$.
Recall that Schubert polynomials are defined inductively by applying divided
difference operators to a single Schubert polynomial. Hence, mitosis
operations can be thought of as combinatorial counterparts of divided
difference operators from Schubert calculus. A convex geometric version of
mitosis was defined in [K16]: pipe dreams were replaced by faces of polytopes
from representation theory (such as Nakashima–Zelevinsky polytopes) and
divided difference operators were replaced by Demazure operators. In the
present paper, we introduce a different version of geometric mitosis (simple
geometric mitosis). It is more general and can be applied to classical
Gelfand–Zetlin polytopes in all types. The starting point for this paper was
the desire to understand the results of [F] from a geometric rather than
representation theoretic viewpoint.

Simple geometric mitosis is defined on a convex polytope $\Delta$ such that
$\Delta$ can be represented as the Cayley sum of two of its facets. Let $v$ be
a vertex of $\Delta$. Faces of $\Delta$ that contain $v$ play the role of pipe
dreams. Mitosis operation acts only on faces that contain $v$ and assigns to a
face a collection of faces of dimension one greater. In applications to
Schubert calculus, there are several different mitosis operations associated
with $\Delta$. This is because Gelfand–Zetlin polytopes (and other polytopes
from representation theory) admit several decompositions into Cayley sums of
facets. In this case, mitosis operations generate faces (that contain $v$)
from the single vertex $v$. The faces generated this way can be used to
represent Schubert cycles as sums of faces.

The paper is organized as follows. In Section 2, we give a definition of
simple geometric mitosis. In Section 3, we recall the combinatorial definition
of Knutson–Miller mitosis on pipe dreams in type $A$ and Fujita mitosis on
skew pipe dreams in type $C$. In Section 4, we relate the constructions of the
previous two sections using Gelfand–Zetlin polytopes in types $A$ and $C$. In
Section 4, we outline applications to Schubert calculus.

##  2\. Main construction

In this section, we give an elementary definition of simple geometric mitosis.
In Section 5, we explain the meaning of this definition in the setting of
intersection theory.

Let $P$, $Q\subset\mathbb{R}^{d}$ be two convex polytopes of full dimension.
Denote by $\Delta:=\Delta(P,Q)\subset\mathbb{R}^{d}\times\mathbb{R}$ their
Cayley sum $P*Q$, that is, the convex hull of

$(P\times 0)\cup(Q\times 1).$

In what follows, we identify $P$ and $Q$ with facets $P\times 0$ and $Q\times
1$ of $\Delta$. The mitosis operation acts only on those faces of $\Delta$
that are contained in $P$ (“horizontal faces”), and produces faces of $\Delta$
that are not contained in $P$.

######  Definition 1.

The face $F\subset P$ is called admissible, if there exists a unique face
$\exp(F)\neq F$ with the property

$F=\exp(F)\cap P.$

It is easy to check that if $F$ is admissible then $\dim\exp(F)=\dim F+1$. The
face $\exp(F)$ can be thought of as an expansion of $F$ from $P$ to $\Delta$.

######  Example 2.1.

Let $d=2$, and let $P,Q\subset\mathbb{R}^{2}$ be two triangles obtained from a
unit square by cutting along a diagonal. More precisely, take
$P=\\{(x_{1},x_{2})\in\mathbb{R}^{2}\ |\ x_{1},x_{2}\leq 1,\ x_{1}+x_{2}\geq
1\\}$ and $Q=\\{(x_{1},x_{2})\in\mathbb{R}^{2}\ |\ x_{1},x_{2}\geq 0,\
x_{1}+x_{2}\leq 1\\}$. Then $\Delta=\\{(x_{1},x_{2},x_{3})\in\mathbb{R}^{3}\
|\ x_{1},x_{2},x_{3}\leq 1,\ 1\leq x_{1}+x_{2}+x_{3}\leq 2\\}$. The face
$P\subset\Delta$ and edges of $P$ are admissible faces, while vertices of $P$
are not.

Let $v\in P$ be a vertex. In the definition below, we consider only those
faces of $P$ and $\Delta$ that contain $v$. We call them $v$-faces. In
particular, the mitosis operation ${\rm mitosis}^{v}(\cdot)$ will depend on
the choice of $v$. Let $F\subset P$ be an admissible $v$-face of dimension
$\ell$. We will define ${\rm mitosis}^{v}(F)$ geometrically as a set of
$v$-faces $\\{E_{1},\ldots,E_{k}\\}$ of dimension $\ell+1$.

######  Definition 2.

A $v$-face $E_{i}\subset\Delta$ belongs to ${\rm mitosis}^{v}(F)$ if $E_{i}$
satisfies the following two conditions:

  1. (1)

$E_{i}\not\subset P$, $Q$;

  2. (2)

$E_{i}\cap Q\subset\exp(F)\cap Q$.

Faces in ${\rm mitosis}^{v}(F)$ will be called offsprings of $F$.

Informally, the first condition means that $E_{i}$ is not a “horizontal” face,
that is, $E_{i}$ intersects both $P$ and $Q$. In particular, it is not
possible to apply the same mitosis operation twice because none of the faces
$E_{i}\in{\rm mitosis}^{v}(F)$ lies in $P$. The second condition tells us that
the face $E_{i}$ is not in general position with respect to $Q$ (unless
$E_{i}=\exp(F)$).

######  Example 2.2.

Let $\Delta\subset\mathbb{R}^{3}$ be a Feigin–Fourier–Littelmann–Vinberg
(FFLV) polytope given by inequalities $0\leq x_{1},x_{3}\leq 1$, $0\leq
x_{2}$, $x_{1}+x_{2}+x_{3}\leq 2$. Consider faces:

$P_{1}=\\{x_{3}=0\\},\quad Q_{1}=\\{x_{3}=1\\},\quad P_{2}=\\{x_{1}=1\\},\quad
Q_{2}=\\{x_{1}=0\\}.$

Let $v$ be the vertex $(1,1,0)$. Clearly, there are two ways to decompose
$\Delta$ as the Cayley sum of two polygons: $\Delta=\Delta(P_{1},Q_{1})$ and
$\Delta=\Delta(P_{2},Q_{2})$. Hence, there are two different mitosis
operations ${\rm mitosis}^{v}_{1}$ and ${\rm mitosis}^{v}_{2}$ associated with
these decompositions.

For instance, ${\rm mitosis}^{v}_{1}(F)$ for the edge $F=\\{x_{3}=0,\
x_{1}+x_{2}+x_{3}=2\\}$ consists of two offsprings, namely,
$\exp(F)=\\{x_{1}+x_{2}+x_{3}=2\\}$ and $P_{2}$. However, ${\rm
mitosis}^{v}_{2}(F)$ consists of a single offspring for all admissible
$v$-faces $F\subset P_{2}$.

######  Remark 2.3.

In what follows, we sometimes apply the mitosis operation to a set $S$ of
faces of the same dimension. By ${\rm mitosis}^{v}(S)$ we mean $\bigcup_{F\in
S}{\rm mitosis}^{v}(F).$

##  3\. Combinatorial mitosis

In this section, we recall two combinatorial rules: Knutson–Miller mitosis on
pipe dreams (type $A$) and Fujita mitosis on skew pipe dreams (type $C$). Both
rules can be defined uniformly using the same combinatorial operation called
two-row mitosis. A similar operation is implicitly used in the original
definition of Knutson–Miller mitosis [KnM, M]. We first define two-row mitosis
explicitly, and then define mitosis operations in types $A$ and $C$ by
reducing them to suitable two-row mitosis. Note that the term row here does
not necessarily mean a horizontal collection of items as in the original
definition of Knutson–Miller mitosis.

###  3.1. Two-row mitosis

Let $A$ and $B$ be two finite collections of squares such that the number of
squares in $A$ is greater by one than the number of squares in $B$. Denote the
number of squares in $B$ by $\ell$. We will label the squares in $A$ by $a1$,
$a2$,…, $a(\ell+1)$, and the squares in $B$ by $b1$, $b2$,…, $b\ell$ (like in
chess notation). Symbolically, we may represent $A$ and $B$ by $(\ell+1)$-row
and $\ell$-row of squares, respectively. In real life examples, we might need
to arrange squares of $A$ and $B$ in more intricate ways. Namely, we will
identify squares in $A$ and $B$ with specific cells in various tables.

To get a basic pipe dream we fill some squares in $A$ and $B$ with $+$. The
other squares remain empty. By size of a basic pipe dream $D$ we mean the
total number of $+$ in the squares of $D$. Two-row mitosis $M$ is an operation
on basic pipe dreams that sends a basic pipe dream $D$ of size $s$ to a
(possibly empty) set $M(D)$ of basic pipe dreams of size $(s-1)$.

To construct the set $M(D)$ we use the following rules:

  1. (1)

If square $a1$ is empty, then $M(D)=\varnothing$.

  2. (2)

If square $a1$ contains $+$, then find the maximal index $r_{D}$ such that the
squares $a1$,…, $ar_{D}$ are all filled with $+$.

  3. (3)

If square $a1$ contains $+$, define the set $\mathcal{J}(D)$ of indices:

$\mathcal{J}(D):=\\{j\leq r_{D}\ |\ \mbox{ square }aj\mbox{ contains }+,\mbox{
square }bj\mbox{ is empty or }j=\ell+1\\}.$

  4. (4)

For every $p\in\mathcal{J}(D)$, construct the offspring $D_{p}$ as follows.
First, erase $+$ in square $ap$. Then move $+$ from square $aj$ down to square
$bj$ for all $j\in\mathcal{J}(D)$ such that $j<p$.

######  Example 3.1.

(cf. [M, Example 7]) Let $\ell=5$. Figure 1 shows a pipe dream $D$ of size $6$
and three pipe dreams of size $5$ that form the set $M(D)$. In this case,
$\mathcal{J}(D)=\\{1,2,4\\}$, and $r_{D}=4$.

$+$$+$$+$$+$$+$

$\downarrow$

$+$$+$$+$$+$

$+$$+$$+$$+$

$+$$+$$+$$+$

Figure 1.

###  3.2. Mitosis in type A

Recall that a pipe dream in type $A_{n}$ can be defined as an $n\times n$
table whose cells are either filled with $+$ or empty. Furthermore, $+$ is not
allowed in cell $(i,j)$ if the cell is below the main antidiagonal of the
table (that is, $i+j>n+1$). An example of pipe dream for $n=6$ is given on
Figure 2 (left). Pipe dreams have an elegant interpretation in terms of
networks of pipes, and are related to permutations from $S_{n+1}$ (see [M] for
details).

$+$$+$$+$$+$$+$$+$$+$$+$

$+$$+$$+$$+$$+$$+$$+$$+$$+$

Figure 2. Pipe dreams in type $A_{6}$ (left) and $C_{4}$ (right).

There are $n$ different mitosis operations $M^{A}_{1}$,…, $M^{A}_{n}$ on pipe
dreams in type $A_{n}$. Informally, mitosis operation $M^{A}_{i}$ can be
defined as the two-row mitosis applied to rows $i$ and $i+1$ of a pipe dream
(the other rows are not affected by $M^{A}_{i}$). Note that mitosis operation
$M^{A}_{n}$ is also well-defined though row $n+1$ is an empty set of boxes.
This is because row $n$ might have at most one $+$, that is, no $+$ will have
to be moved down according to the mitosis rules.

We now define mitosis operation $M^{A}_{i}$ more formally. Let $D$ be a pipe
dream in type $A_{n}$. Put $\ell=n-i$, and label cells $(i,1)$,…, $(i,n-i+1)$
by $a1$,…, $a(\ell+1)$, respectively. Label cells $(i+1,1)$,…, $(i+1,n-i)$ by
$b1$,…, $b\ell$, respectively. Extract the basic pipe dream $D^{i}$ from $D$
by setting $A^{i}=(a1,\ldots,a(\ell+1))$ and $B^{i}=(b1,\ldots,b\ell)$. Apply
two-row mitosis to $D^{i}$. Complete every resulting offspring $D^{i}_{p}$ to
a pipe dream $D_{p}$ by replacing $D^{i}$ with $D^{i}_{p}$ in $D$. Define
$M^{A}_{i}(D)$ as the set $\\{D_{p}\ |\ p\in\mathcal{J}(D^{i})\\}$.

######  Example 3.2.

Let $D$ be a pipe dream depicted on Figure 2 (left). For $i=1$, the basic pipe
dream $D^{1}$ coincides with the one on Figure 1 (top) from Example 3.1.
Hence, $M^{A}_{1}(D)$ consists of the pipe dreams shown on Figure 3.

$+$$+$$+$$+$$+$$+$$+$

$+$$+$$+$$+$$+$$+$$+$

$+$$+$$+$$+$$+$$+$$+$

Figure 3.

If $i=2,3,4,6$, then $M^{A}_{i}(D)$ is empty. Finally, $M^{A}_{5}(D)$ consists
of a single offspring obtained from $D$ by erasing the $+$ in the fifth row.

######  Remark 3.3.

It is easy to check that $M^{A}_{i}$ coincides with the $i$-th mitosis
operator ${\rm mitosis}_{i}$ introduced in [M, Definition 6] (Knutson–Miller
mitosis). There is also a strong relationship between $M^{A}_{i}$ and the
operator $M_{i}$ introduced in [F, Section 5] using representation theoretic
considerations. These operations coincide whenever $D$ does not have any $+$
in cells to the right of $r_{D_{i}}$. While $M^{A}_{i}$ and $M_{i}$ might
differ on the other pipe dreams (the latter operation might produce more
offsprings than the former) it is interesting that both operations lead to the
same results in Schubert calculus (see [F, Corollary 5.13] and the preceding
discussion for more details). In particular, [M, Theorem 15] still holds if
${\rm mitosis}_{i}$ is replaced by $M_{i}$, that is, $M_{i}$ can be viewed as
an alternative version of the Knutson–Miller mitosis.

###  3.3. Mitosis in type $C$

Similarly to type $A$ case, pipe dreams in type $C_{n}$ (also called skew pipe
dreams) can be defined as $n\times(2n-1)$ tables filled with $+$. In type
$C_{n}$ case, $+$ is not allowed in cell $(i,j)$ if $i+j>2n$ or $i>j$. An
example of a skew pipe dream for $n=4$ is given on Figure 2 (right). Skew pipe
dreams were recently used in [F] to construct a convex geometric model for
Schubert calculus in type $C$ in terms of symplectic GZ polytopes (we say more
about this in Section 4), in particular, they are related to signed
permutation. It would be interesting to find an interpretation of skew pipe
dreams in terms of networks of pipes. In [ST], c-signed pipe dreams were
defined as networks of pipes with extra features. We do not know of any direct
relation between skew pipe dreams and c-signed pipe dreams.

We now define $n$ mitosis operations $M^{C}_{1}$,…, $M^{C}_{n}$ on a skew pipe
dream $D$. The case $i=1$ is special. In this case, set $\ell=n-1$. Label
cells $(1,n)$, $(2,n)$,…, $(n,n)$ by $a1$, $a2$,…, $a(\ell+1)$, respectively
(these are cells in the middle column of $D$). Label cells $(1,n+1)$,
$(2,n+1)$,…, $(n-1,n+1)$ by $b1$, $b2$,…, $b\ell$, respectively. If $i=2$,…,
$n$, set $\ell=2(n-i)+1$. Label cells $(1,n-i+1)$, $(1,n+i-1)$, $(2,n-i+1)$,
$(2,n+i+1)$,…, $(n-i+1,n-i+1)$, $(n-i+1,n+i-1)$ by $a1$, $a2$, $a3$, $a4$,…,
$a\ell$, $a(\ell+1)$, respectively (these are all cells in columns $n\pm(i-1)$
of $D$). Label cells $(1,n-i+2)$, $(1,n+i)$, $(2,n-i+2)$, $(2,n+i)$,…,
$(n-i+1,n-i+2)$ by $b1$, $b2$, $b3$, $b4$,…, $b\ell$, respectively (here we
alternate first $n-i+1$ cells in column $n-i+2$ with cells in column $n+i$ of
$D$). The rest of the definition of $M^{C}_{i}$ is completely analogous to
type $A$ case.

######  Example 3.4.

Let $D$ be a skew pipe dream from Figure 2 (right). Using Example 3.1 again we
get that $M^{C}_{1}(D)$ consists of skew pipe dreams depicted on Figure 4.

$+$$+$$+$$+$$+$$+$$+$$+$

$+$$+$$+$$+$$+$$+$$+$$+$

$+$$+$$+$$+$$+$$+$$+$$+$

Figure 4.

For $i=2,3$, the set $M^{C}_{i}(D)$ is empty. The set $M^{C}_{4}(D)$ consists
of a single offspring obtained by erasing the $+$ in the first column of $D$.

######  Remark 3.5.

Mitosis operation $M^{C}_{i}$ on skew pipe dreams is different from operator
$M_{i}$ defined in [F, Section 6]. As in type $A$ (see Remark 3.3), the
difference lies in the restriction $j\leq r_{D}$ in Definition 2. It would be
interesting to check whether both operations still produce the same results in
applications to Schubert calculus.

##  4\. Mitosis on GZ polytopes in types A and C

In this section, we apply simple geometric mitosis to GZ polytopes in types
$A$ and $C$. We compare the resulting operations with geometric realizations
of Knutson–Miller mitosis and Fujita mitosis.

###  4.1. Gelfand–Zetlin polytopes in type $A_{n}$

Put $d:=\frac{n(n+1)}{2}$. A GZ table of type $A_{n}$ is a collection of cells
organized according to the pattern on Figure 5 (left). Let
$\lambda:=(\lambda_{1}\geq\lambda_{2}\geq\ldots\geq\lambda_{n+1})$ be a
decreasing collection of real numbers. Identify a point
$(x^{1}_{1},x^{1}_{2},\ldots,x^{1}_{n};x^{2}_{1},\ldots,x^{2}_{n-1};\ldots;x^{n-1}_{1},x^{n-1}_{2};x^{n}_{1})\in\mathbb{R}^{d}$
with the GZ table whose $i$-th row is filled with coordinates
$(x^{i}_{1},x^{i}_{2},\ldots,x^{i}_{n-i+1})$ for $1\leq i\leq n$. Define the
GZ polytope $GZ^{A}_{\lambda}\subset\mathbb{R}^{d}$ by $2d$ interlacing
inequalities $x^{i-1}_{j}\geq x^{i}_{j}\geq x^{i-1}_{j+1}$ (we put
$x^{0}_{j}:=\lambda_{j}$). In terms of the GZ table, these inequalities tell
us that the coordinate in any given cell lies between coordinates in two upper
neighbors of this cell.

$\lambda_{1}$$\lambda_{2}$$\lambda_{3}$$\lambda_{4}$$\lambda_{5}$$\lambda_{6}$$\lambda_{7}$

$\lambda_{1}$$\lambda_{2}$$\lambda_{3}$$\lambda_{4}$$0$$0$$0$$0$

Figure 5. Gelfand–Zetlin patterns in type $A_{6}$ (left) and type $C_{4}$
(right).

Let $v$ be the Kogan vertex of $GZ_{\lambda}^{A}$. Recall that the Kogan
vertex is a unique vertex of the polytope $GZ^{A}_{\lambda}$ that satisfies
simultaneously all $d$ equations of type $x^{i-1}_{j}=x^{i}_{j}$. A face
$\Gamma\subset GZ^{A}_{\lambda}$ is called a Kogan face if $\Gamma$ contains
$v$. These faces were first considered by Mikhail Kogan [Ko]. In particular,
$v$-faces used in the definition of mitosis in Section 2 are exactly Kogan
faces for this choice of $v$.

Note that the polytope $GZ^{A}_{\lambda}$ for $\lambda=(n,n-1,\ldots,1,0)$ can
be represented as the Cayley sum of two polytopes in $n$ differents ways.
Namely, let $P_{i}$ and $Q_{i}$ be the facets of $GZ^{A}_{\lambda}$ given by
equations $x^{1}_{i}=\lambda_{i}$ and $x^{1}_{i}=\lambda_{i+1}$, respectively.
Then $\Delta(P_{i},Q_{i})$ coincides with $GZ^{A}_{\lambda}$ up to a parallel
translation. Clearly, $v\in P_{1},$…, $P_{n}$. Hence, we can define $n$
different mitosis operations ${\rm mitosis}^{v}_{1}$,…, ${\rm
mitosis}^{v}_{n}$ on Kogan faces of $GZ_{\lambda}^{A}$ as in Section 2. In
particular, Example 2.2 describes these operations for $n=2$ because the GZ
polytope in this case is unimodularly equivalent to the FFLV polytope.

By definition, any Kogan face is given only by equations of type
$x^{i-1}_{j}=x^{i}_{j}$. Hence, every Kogan face $F$ can be encoded by a GZ
table filled with $+$. Namely, if $F$ lies in the hyperplane
$\\{x^{i-1}_{j}=x^{i}_{j}\\}$ then put $+$ in cell $(i,j)$ (that is, in the
$j$-th cell of the $i$-th row). Otherwise, leave cell $(i,j)$ empty. The
resulting GZ table filled with $+$ will be called the diagram of the face $F$
and denoted by $D(F)$.

Define a bijective correspondence between pipe dreams in type $A_{n}$ and
diagrams of Kogan faces of $GZ^{A}_{\lambda}$ by identifying cell $(i,j)$ in a
GZ table with cell $(j,i)$ in a pipe dream. The correspondence is illustrated
by Figure 7 (left). Namely, inscribe the words “GELFAND ZETLIN POLYTOPE”
(without spaces) inside a pipe dream of type $A_{6}$ in usual way, that is,
start from the top row and write from left to right. After switching to $GZ$
table, the words will transform to the sequence of letters on Figure 7 (left).

GELFANDZETLINPOLYTOPE

SHCU

B

E

R

TCA

L

C

UL

U

S

Figure 6. Correspondence between pipe dreams and GZ patterns in type $A_{6}$
(left) and type $C_{4}$ (right).

######  Example 4.1.

The pipe dream on Figure 2 (left) corresponds to the Kogan face $F$ given by
eight equations $\lambda_{1}=x^{1}_{1}=x^{2}_{1}=x^{3}_{1}=x^{4}_{1}$,
$x^{2}_{2}=x^{3}_{2}$, $\lambda_{3}=x^{1}_{3}$, $\lambda_{4}=x^{1}_{4}$,
$\lambda_{5}=x^{1}_{5}$. In particular, $\dim F=13$. It is easy to check that
${\rm mitosis}_{1}^{v}(F)$ consists of three Kogan faces of dimension $14$
whose diagrams correspond to pipe dreams on Figure 3.

Using the bijection between pipe dreams and GZ tables we can extend mitosis
operations $M^{A}_{1}$,…, $M^{A}_{n}$ from pipe dreams to GZ tables.
Informally speaking, we replace rows with columns when applying two-row
mitosis. We will use the same notation for mitosis operations on GZ tables as
it will be clear from the context whether we talk about pipe dreams or about
GZ tables.

It turns out that geometric operation ${\rm mitosis}_{i}^{v}$ on certain Kogan
faces is combinatorially equivalent to mitosis operation $M^{A}_{i}$ on the
corresponding pipe dreams. The following theorem describes the precise
relation between the geometric mitosis on Kogan faces of a GZ polytope and the
combinatorial mitosis on GZ tables.

######  Theorem 4.2.

Let $F\subset GZ_{\lambda}^{A}$ be a Kogan face such that its diagram $D(F)$
has $+$ in cell $(1,i)$ and no $+$ in cells $(j,i+1)$ for all $j$. Then ${\rm
mitosis}_{i}^{v}(F)$ consists of Kogan faces whose diagrams are obtained from
$D(F)$ by the mitosis operation $M^{A}_{i}$ on GZ tables.

###### Proof.

In order to compute ${\rm mitosis}_{i}^{v}(F)$ we have to consider not only
Kogan faces. In general, a face of $GZ^{A}_{\lambda}$ is given by equations of
two types: either $x^{i-1}_{j}=x^{i}_{j}$ (type $A$) or
$x^{i}_{j}=x^{i-1}_{j+1}$ (type $B$).

######  Definition 3.

Following [Ko] define equations $A_{i,j}$ and $B_{i,j}$, respectively, as
$x^{i-1}_{j}=x^{i}_{j}$ and $x^{i}_{j}=x^{i-1}_{j+1}$.

We now apply Definition 2 to $P:=P_{i}$ and $Q:=Q_{i}$. Note that any Kogan
face is admissible because it contains a simple vertex, namely, the Kogan
vertex $v$. The face $E_{1}:=\exp(F)$ is obtained from $F$ by removing a
single equation $A_{1,i}$. Hence, the face $\exp(F)\cap Q$ is given by the
same equations as $F$ with one exception: equation $A_{1,i}$ is replaced by
equation $B_{1,i}$.

Let $E$ be a Kogan face of dimension $\ell+1$ (where $\ell:=\dim F$) such that
$E\cap Q\subset\exp(F)\cap Q$. Let ($X_{1}$,…, $X_{d-\ell-1}$) be the
collection of equations of type $A$ that define $E$. If $E\neq\exp(F)$ then
there exists an equation $Y$ (of type A) such that $\exp(F)$ satisfies $Y$ but
$Y$ does not follow from ($X_{1}$,…, $X_{d-\ell-1}$). However, $Y$ should
follow from equations $B_{1,i}$, $X_{1}$,…, $X_{d-\ell-1}$. Hence, the
collection of equations ($B_{1,i}$, $Y$, $X_{1}$,…, $X_{d-\ell-1}$) is
redundant, while the collection ($Y$, $X_{1}$,…, $X_{d-\ell-1}$) is not. This
is only possible if $A_{1,i+1}$ is contained among the equations $X_{1}$,…,
$X_{d-\ell-1}$ as the following lemma shows:

######  Lemma 4.3.

Let $(X_{1},\ldots,X_{s})$ be a collection of equations of type $A$ that does
not contain equation $A_{p,q}$. If $A_{p,q}$ follows from
$(B_{j,i},X_{1},\ldots,X_{s})$ then three conditions hold:

  1. (1)

$q=i$;

  2. (2)

$p>j$;

  3. (3)

$(X_{1},\ldots,X_{s})$ contains $A_{k,i+1}$ for all $k$ such that $j\leq k<p$.

###### Proof.

The statement follows directly from Definition 3. ∎

If ($X_{1}$, $X_{2}$,…, $X_{d-\ell-1}$) contains $A_{1,i+1}$, $A_{2,i+1}$,…,
$A_{k,i+1}$ but does not contain $A_{k+1,i+1}$ for some $k\geq 1$, then
$A_{2,i}$,…, $A_{k+1,i}$ are the only equations that follow from ($B_{1,i}$,
$X_{1}$,…, $X_{d-\ell-1})$. Hence, if $F$ satisfies $A_{p,q}$, and $A_{p,q}$
is not contained among ($X_{1}$, $X_{2}$,…, $X_{d-\ell-1}$)) then $A_{p,q}$
must coincide with one of the equations $A_{2,i}$,…, $A_{k+1,i}$. For
dimension reasons, all $k$ equations $A_{2,i}$,…, $A_{k+1,i}$ must be absent
in ($X_{1}$, $X_{2}$,…, $X_{d-\ell-1}$) (to compensate for the presence of $k$
equations $A_{1,i+1}$, $A_{2,i+1}$,…, $A_{k,i+1}$). In particular, if $F$ does
not satisfy $A_{2,i}$ then ${\rm mitosis}_{i}^{v}(F)$ consists of a single
offspring $\exp(F)$.

We now proceed by induction on $r$ where $r$ is the maximal number such that
$F$ satisfies the equations $A_{1,i}$,…, $A_{r,i}$. Let $\Gamma_{r,i}$ be the
facet given by equation $A_{r,i}$, and $F^{\prime}$ the face given by the same
equations as $F$ except for $A_{r,i}$, that is,
$F=F^{\prime}\cap\Gamma_{r,i}$. It is easy to check that there is a bijection
between ${\rm mitosis}^{v}_{i}(F^{\prime})$ and the subset $S\subset{\rm
mitosis}^{v}_{i}(F)$ that consists of all faces $E_{i}\in{\rm
mitosis}^{v}_{i}(F)$ such that $E_{i}\subset\Gamma_{r,i}$. Namely, a face
$E^{\prime}\in{\rm mitosis}^{v}_{i}(F^{\prime})$ corresponds to the face
$E=E^{\prime}\cap\Gamma_{r,i}\in{\rm mitosis}^{v}_{i}(F)$. Hence, the faces in
$S$ can be described by the induction hypothesis applied to $F^{\prime}$.

It remains to describe the faces $E_{i}\in{\rm mitosis}^{v}_{i}(F)\setminus
S$. By condition (2) of Definition 2 and part (3) of Lemma 4.3 applied to
$A_{r,i}$, we have that if $E_{i}\not\subset\Gamma_{r,i}$, then $E_{i}$ must
satisfy equations $A_{1,i+1}$, $A_{2,i+1}$…, $A_{r-1,i+1}$. Hence, there is a
single offspring $E_{r}\in{\rm mitosis}^{v}_{i}(F)\setminus S$ obtained from
$F$ by removing equation $A_{r,i}$ and by replacing equations $A_{j,i}$ by
equations $A_{j,i+1}$ for all $j<r$. ∎

We now adapt geometric mitosis operations so that Theorem 4.2 holds for all
reduced Kogan faces. The definition of reduced pipe dreams can be found in [M,
Definition 1] (these pipe dreams are mostly used in applications to Schubert
calculus). A Kogan face $F$ is reduced if its diagram $D(F)$ is reduced. Below
we use notation of Definition 3 from the proof of Theorem 4.2.

######  Corollary 4.4.

Let $F\subset GZ_{\lambda}^{A}$ be a reduced Kogan face. Define an adapted
geometric mitosis operation ${\rm{\underline{mitosis}}}_{i}^{v}$ as follows.

  1. (1)

Consider the face ${\rm env}(F)\subset GZ_{\lambda}^{A}$ given by equations
$A_{k,i}$ and $A_{k,i+1}$ for all $k$ such that $F$ satisfies both $A_{k,i}$
and $A_{k,i+1}$.

  2. (2)

Find the minimal index $s$ such that ${\rm env}(F)$ does not satisfy equation
$A_{s,i}$. Consider the facets $P_{i}^{F}$, $Q_{i}^{F}\subset{\rm env}(F)$
given by equations $A_{s,i}$ and $B_{s,i}$, respectively.

  3. (3)

Define ${\rm{\underline{mitosis}}}_{i}^{v}(F)$ by applying Definition 2 to
$\Delta={\rm env}(F)$, $P=P_{i}^{F}$ and $Q=Q_{i}^{F}$.

Then ${\rm{\underline{mitosis}}}_{i}^{v}(F)$ consists of exactly those Kogan
faces whose diagrams lie in the set $M^{A}_{i}(D(F))$.

The proof of Corollary 4.4 is completely analogous to the proof of Theorem
4.2.

######  Remark 4.5.

Note that the set ${\rm mitosis}_{i}^{v}(F)$ might, in general, contain more
faces than the set ${\rm{\underline{mitosis}}}_{i}^{v}(F)$. By applying
mitosis inside the face ${\rm env}(F)$ instead of the whole $GZ_{\lambda}^{A}$
we get rid of these extra faces. The definition of ${\rm env}(F)$ in Corollary
4.4 is ad hoc, it relies heavily on combinatorics of GZ polytopes in type $A$.
It would be interesting to find a more geometric definition of ${\rm env}(F)$.

The choice of $P_{i}^{F}$ and $Q_{i}^{F}$ stems from the classical inductive
construction of GZ bases and polytopes based on the chain of subgroups
$GL_{2}(\mathbb{C})\subset GL_{3}(\mathbb{C})\subset\ldots\subset
GL_{n+1}(\mathbb{C})$.

###  4.2. Gelfand–Zetlin polytopes in type $C$

We now describe analogous constructions in type $C$. We omit details that are
the same as in type $A$ case and focus on unique features of type $C$ case.
Put $d=n^{2}$, and put
$\lambda:=(\lambda_{1}\geq\lambda_{2}\geq\ldots\geq\lambda_{n}\geq\lambda_{n+1}=0)$.
A GZ table of type $C_{n}$ is defined according to the pattern on Figure 5
(right). Roughly speaking, a GZ table of type $C_{n}$ is a half of a GZ table
of type $A_{2n}$. As in type $A$ case, the GZ polytope
$GZ^{C}_{\lambda}\subset\mathbb{R}^{d}$ is defined by $2d$ interlacing
inequalities that come from a $GZ$ pattern in type $C$. Again, the polytope
$GZ_{\lambda}^{C}$ for $\lambda=(n,n-1,\ldots,0)$ can be represented as Cayley
sum of two polytopes in $n$ different ways.

Let $v$ be the symplectic Kogan vertex of $GZ_{\lambda}^{C}$ as defined in [F,
Section 6]. Using notation from Definition 3 (see the proof of Theorem 4.2)
and regarding a GZ pattern of type $C_{n}$ as part of a GZ pattern of type
$A_{2n-1}$ we may define $v$ by equations $A_{i,j}$ for all odd $i\leq 2n-1$
and equations $B_{i,j}$ for all even $i<2n-1$. Similarly to type $A$ case, a
face $\Gamma\subset GZ^{C}_{\lambda}$ is called a symplectic Kogan face if
$\Gamma$ contains $v$. Again, there are $n$ mitosis operations ${\rm
mitosis}_{1}^{v}$,…, ${\rm mitosis}_{n}^{v}$.

As in type $A$ case, every symplectic Kogan face $F$ can be encoded by a GZ
table $D(F)$ of type $C$ filled with $+$. Define bijective correspondence
between skew pipe dreams and diagrams of Kogan faces as in [F, Section 6]. For
instance, if words “SCHUBERT CALCULUS” (without spaces) are inscribed into a
skew pipe dream of type $C_{4}$ in usual way then they get transformed into
boustrophedon111I am grateful to Evgeny Smirnov from whom I learnt the proper
name of this writing style.  writing in a $GZ$ pattern of type $C_{4}$ on
Figure 7 (right). Again, we extend mitosis operations $M_{1}^{C}$,…,
$M_{n}^{C}$ from skew pipe dreams to GZ tables in type $C_{n}$.

Note that $M_{1}^{C}$ is combinatorially different from $M_{2}^{C}$,…,
$M_{n}^{C}$. In applications to Schubert calculus, mitosis operations
correspond to generators of the group of signed permutations $B_{n}$ on $n$
elements. There is a special generator
$s_{1}:(1,2,\ldots,n)\mapsto(-1,2,\ldots,n)$ (change of sign), and the
elementary transpositions $s_{2}$,…,$s_{n}$, namely, $s_{i}=(i-1~{}i)$. If we
regard $B_{n}$ as the Weyl group of the symplectic group
$Sp_{2n}(\mathbb{C})$, then $s_{1}$ is the simple reflection corresponding to
the longer root. The mitosis operation $M_{n}^{C}$ corresponds to the special
generator $s_{1}$, hence, it is natural to expect that $M_{n}^{C}$ will be
special.

The following theorem is analogous to Theorem 4.2 and can be proved using
similar arguments.

######  Theorem 4.6.

Let $F\subset GZ^{C}_{\lambda}$ be a symplectic Kogan face.

  1. (1)

Let $i=1$,…, $n-1$. If the diagram $D(F)$ has $+$ in cell $(1,i)$, and no $+$
in cells $(2k+1,i-k+1)$ and $(2k,i-k)$ for all $k$, then ${\rm
mitosis}^{v}_{i}(F)$ consists of faces whose diagrams are obtained from $D(F)$
by applying $M^{C}_{n-i+1}$.

  2. (2)

If the diagram $D(F)$ has $+$ in cell $(1,n)$, and no $+$ in cells $(2k,n-k)$
then ${\rm mitosis}^{v}_{n}(F)$ consists of faces whose diagrams are obtained
from $D(F)$ by applying $M^{C}_{1}$.

##  5\. Applications to Schubert calculus

We now explain how the notion of simple geometric mitosis fits into in the
context of intersection theory. In particular, we relate Corollary 4.4 and
Theorem 4.6 with the analogous results on Schubert calculus in types $A$ and
$C$. Below we use the notion of polytope ring (aka Khovanskii–Pukhlikov ring).
The definition can be found, for instance, in [K21, Section 2]. We refer the
reader to [F] for more details on applications of Khovanskii–Pukhlikov rings
to Schubert calculus.

We use notation of Section 2. Let $R_{\Delta}$ and $R_{P}$ be the polytope
rings of $\Delta$ and $P$. Assume that $R_{\Delta}$ and $R_{P}$, respectively,
are isomorphic to the Chow rings of smooth varieties $Y$ and $X$, where
$Y=\mathbb{P}(E)$ is the projectivization of a rank two vector bundle $E$ on
$X$ (see [K21, Section 4] for motivation behind such an assumption). Let $p$
be the natural projection from $Y$ to $X$. Then there is a push-pull operator

$p^{*}p_{*}:CH^{*}(Y)\to CH^{*-1}(Y),$

which is a homomorphism of $CH^{*}(X)$-modules. The simple geometric mitosis
is an attempt to describe explicitly the action of the push-pull operator on
$R_{\Delta}\simeq CH^{*}(Y)$ using representations of elements of $R_{\Delta}$
by linear combinations of faces of $\Delta$. It would be interesting to
formalize the connection between Definition 2 and the action of $p^{*}p_{*}$
on faces of $\Delta$ in the general setting. Below we will exhibit such a
connection in the special case of GZ polytopes and flag varieties.

Let $\Delta$ be the GZ polytope for a classical group $G$ and a strictly
dominant weight $\lambda$. Let $B\subset G$ denote a Borel subgroup of $G$,
and let $Y:=G/B$ denote the complete flag variety for $G$. The ring
$R_{\Delta}$ is isomorphic to the subring of $CH^{*}(Y)$ generated by the
first Chern classes of line bundles on $G/B$ [Ka, Corollary 5.3, Remark 2.4].
For $G=GL_{n+1}(\mathbb{C})$ and $Sp_{2n}(\mathbb{C})$, this subring coincides
with $CH^{*}(Y)$ (in general, the discrepancy between the subring and the
whole $CH^{*}(Y)$ is measured by the torsion index of $G$, see [T] for more
details). In particular, if $\lambda=\rho:=(n,n-1,\ldots,1,0)$, then
$R_{\Delta}\simeq CH^{*}(Y)$ in types $A$ and $C$.

Denote by $\alpha_{1}$,…, $\alpha_{n}$ the simple roots of $G$. Let $X_{i}$ be
the partial flag variety $G/P_{i}$ for the minimal parabolic subgroup $P_{i}$
corresponding to $\alpha_{i}$. Note that the natural projection $p_{i}:Y\to
X_{i}$ turns $Y$ into a $\mathbb{P}^{1}$-fibration (it can be realized as the
projectivization of the rank two bundle $E_{i}:={p_{i}}_{*}\mathcal{L}(\rho)$
where $\mathcal{L}(\rho)$ is the line bundle on $Y$ corresponding to the
weight $\rho$). The classical divided difference (or push-pull) operator in
Schubert calculus is defined as $\partial_{i}:=p_{i}^{*}{p_{i}}_{*}$.

Recall that the operators $\partial_{1}$,…, $\partial_{n}$ are used to
generate the Schubert classes $[X_{w}]\in CH^{*}(Y)$ starting from the class
$[X_{\rm id}]$ of a point. More precisely, let $w\in W$ be an element of the
Weyl group of $G$. Choose a reduced decomposition $w=s_{i_{1}}\cdots
s_{i_{\ell}}$. Here $s_{i}$ denotes the reflection with respect to the root
$\alpha_{i}$. Then $[X_{w}]=\partial_{i_{\ell}}\cdots\partial_{i_{1}}[X_{\rm
id}]$. If we choose the Kogan vertex $v$ as a representative of the class
$[X_{\rm id}]$ in $R_{\Delta}$, then the action of
$\partial_{i_{\ell}}\cdots\partial_{i_{1}}$ on $[X_{\rm id}]$ can be computed
using mitosis operations as follows.

######  Theorem 5.1.

Let $G=GL_{n+1}(\mathbb{C})$ and $\Delta=GZ^{A}_{\rho}$. Under the isomorphism
$CH^{*}(Y)\simeq R_{\Delta}$ the Schubert cycle $[X_{w}]$ can be represented
as the class of the sum of faces $F\subset\Delta$ where $F$ runs through the
set

$S_{w}={\rm{\underline{mitosis}}}_{i_{\ell}}^{v}\cdots{\rm{\underline{mitosis}}}^{v}_{i_{1}}(v).$

In particular, the action of divided difference operator $\partial_{i}$ on the
Schubert cycle $[X_{w}]$ gets represented by the action of
${\rm{\underline{mitosis}}}_{i}^{v}$ on faces from $S_{w}$:

$S_{ws_{i}}={\rm{\underline{mitosis}}}_{i}^{v}(S_{w}).$

This theorem follows from [F, Corollary 5.18] together with Corollary 4.4.
Note that the proof of [F, Theorem 5.17] (which implies [F, Corollary 5.18])
uses representation theoretic arguments. It would be interesting to find a
convex geometric proof.

######  Remark 5.2.

In [K16], different geometric mitosis operations are defined. They mimick
Demazure operators rather than push-pull operators $\partial_{i}$. In type
$A$, they can also be used to prove theorems analogous to Theorem 5.1 or [F,
Corollary 5.18] due to a special symmetry of GZ diagrams in type $A$. Namely,
the diagrams are symmetric with respect to the reflection $(i,j)\mapsto(j,i)$.

Note that the induction step in [K16, Corollary 3.6] goes from $s_{i}w$ to $w$
(not from $ws_{i}$ to $w$ as in Theorem 5.1 and in [F, Theorem 5.17]). In
other words, induction goes along initial subwords of
$s_{i_{1}}s_{i_{2}}\ldots s_{i_{\ell}}$ (not along terminal subwords). This
difference is matched by the difference between mitosis as defined in [K16]
and its transpose (or mirror) mitosis as defined in the present paper. In
particular, all arguments with the mitosis operations applied to
$w=s_{i_{1}}s_{i_{2}}\ldots s_{i_{\ell}}$ can be immediately translated into
analogous arguments with the transpose mitosis operations applied to
$w^{-1}=s_{i_{\ell}}s_{i_{\ell-1}}\ldots s_{i_{1}}$ (see [F, Section 5] for
more details).

In type $C$, Theorem 4.6 together with Remark 3.5 and [F, Theorem 6.8] suggest
that ${\rm mitosis}^{v}_{i}$ can be adapted so that under the isomorphism
$CH^{*}(Y)\simeq R_{\Delta}$ the Schubert class $[X_{w}]$ can be represented
as the class of the sum of faces $F\subset\Delta$ where $F$ runs through the
set

$S_{w}={\rm{\underline{mitosis}}}_{n-i_{\ell}+1}^{v}\cdots{\rm{\underline{mitosis}}}^{v}_{n-i_{1}+1}(v).$

In [F, Corollary 6.13], another presentation of Schubert cycles by faces of
$GZ^{C}_{\rho}$ is obtained using the dual Kogan faces. Similarly to the
definition of symplectic Kogan vertex in Section 4.2, the dual Kogan vertex
$v^{*}$ can be defined by equations $B_{i,j}$ for all odd $i\leq 2n-1$ and
equations $A_{i,j}$ for all even $i<2n-1$. Again there are $n$ geometric
mitosis operations ${\rm mitosis}^{v^{*}}_{1}$,…, ${\rm mitosis}^{v^{*}}_{n}$
where ${\rm mitosis}^{v^{*}}_{i}$ corresponds to the decomposition
$GZ^{C}_{\rho}=\\{x^{1}_{i}=\lambda_{i+1}\\}\star\\{x^{1}_{i}=\lambda_{i}\\}$
(that is, $P_{i}$ and $Q_{i}$ from the definition of ${\rm mitosis}^{v}_{i}$
switch places when defining ${\rm mitosis}^{v^{*}}_{i}$). In the case of
$Sp_{4}(\mathbb{C})$, these operations produce part of the presentation of
Schubert cycles from [F, Corollary 6.13] as the following example shows.

######  Example 5.3.

Let $n=2$. We encode dual symplectic Kogan faces by their diagrams. We put $A$
and $B$ instead of $+$ just for clarity (not because it carries any extra
information). This way it is easier to distinguish diagrams of Kogan faces
from diagrams of dual Kogan faces. In the second row, we use an adapted
mitosis once (this adaptation is based on the chain
$Sp_{2}(\mathbb{C})=SL_{2}(\mathbb{C})\subset Sp_{4}(\mathbb{C})$).

BB$\stackrel{{\scriptstyle{\rm mitosis}^{v^{*}}_{1}}}{{\longrightarrow}}$${\rm
id}$AB

B$\stackrel{{\scriptstyle{\rm
mitosis}^{v^{*}}_{2}}}{{\longrightarrow}}$$s_{2}$AB

B&$s_{2}s_{1}$B

$s_{2}s_{1}$$\stackrel{{\scriptstyle{\rm
mitosis}^{v^{*}}_{1}}}{{\longrightarrow}}$AB

B&$s_{2}s_{1}s_{2}$

$s_{2}s_{1}s_{2}$B

BB$\stackrel{{\scriptstyle{\rm{\underline{mitosis}}}^{v^{*}}_{2}}}{{\longrightarrow}}$$\rm
id$AB

BB$\stackrel{{\scriptstyle{\rm
mitosis}^{v^{*}}_{1}}}{{\longrightarrow}}$$s_{1}$A

B$\stackrel{{\scriptstyle{\rm
mitosis}^{v^{*}}_{2}}}{{\longrightarrow}}$$s_{1}s_{2}$A

B&$s_{1}s_{2}s_{1}$

$s_{1}s_{2}s_{1}$A

Figure 7. Mitosis for the dual Kogan vertex in type $C_{2}$.

These collections of faces are the same as the presentations of Schubert
cycles $[X_{w}]$ in [F, Corollary 6.13] for all $w\in W$ except for
$s_{2}s_{1}$. In the case $w=s_{2}s_{1}$, one face is missing.

Recall that according to [F, Corollary 6.13], the dual Kogan faces that
represent $[X_{w}]$ are in bijective correspondence with those reduced
subwords of the longest word

$\overline{w_{0}}=(s_{1})(s_{2}s_{1}s_{2})(s_{3}s_{2}s_{1}s_{2}s_{3})\ldots(s_{n}s_{n-1}\ldots
s_{2}s_{1}s_{2}\ldots s_{n-1}s_{n})$

that represent the element $w_{0}w$. The bijection is obtained by inscribing
the word $\overline{w_{0}}$ into the symplectic GZ pattern using reverse
boustrophedon (see Figure 8 left). For instance, the subword

$(s_{1})(\cancel{s_{2}}\cancel{s_{1}}s_{2})(s_{3}s_{2}s_{1}s_{2}\cancel{s_{3}})(\cancel{s_{4}}s_{3}\cancel{s_{2}}\cancel{s_{1}}s_{2}s_{3}s_{4})$

in type $C_{4}$ corresponds to the dual Kogan face on Figure 8 right.

In particular, there are three dual Kogan faces that represent $[X_{w}]$ for
$w=s_{2}s_{1}$ in type $C_{2}$. Indeed, $w_{0}w=s_{1}s_{2}$, and there are
three reduced subwords of $s_{1}(s_{2}s_{1}s_{2})$ that represent
$s_{1}s_{2}$.

$s_{4}$$s_{3}$$s_{2}$$s_{1}$$s_{2}$$s_{3}$$s_{4}$$s_{3}$$s_{2}$$s_{1}$$s_{2}$$s_{3}$$s_{2}$$s_{1}$$s_{2}$$s_{1}$

$B$$B$$B$$A$$B$$B$$A$$A$$B$$B$

Figure 8. Correspondence between subwords of $\overline{w_{0}}$ and dual Kogan
faces in type $C_{4}$

######  Remark 5.4.

In type $A$, there is no combinatorial difference between presentation of
Schubert cycles by Kogan faces and presentation by dual Kogan faces (see [F,
Theorem 5.25]). Both presentations can be related using the automorphism of
the Dynkin diagram of type $A_{n}$ (on the level of GZ diagrams this
automorphism corresponds to the reflection $(i,j)\mapsto(i,n+2-i-j)$). In type
$C$, there is a combinatorial difference already for $n=2$ (see [F, Section 1]
for more details). In type $A$, one can immediately recover mitosis on dual
Kogan faces from mitosis on Kogan faces. Both operations will be
combinatorially equivalent to the Knutson–Miller mitosis. In type $C$, the
Fujita mitosis on Kogan faces does not yield a combinatorially equivalent
mitosis on dual Kogan faces.

Example 5.3 shows that Definition 2 is too simple to capture completely the
action of $p^{*}p_{*}$ on faces of $\Delta$ in the general setting. However,
it may be regarded as the first approximation of this action. It can also be
used to make an educated guess about a possible combinatorial mitosis on dual
Kogan faces in type $C$ and mitosis in type $D$.

## References

  * [F] Naoki Fujita, Schubert calculus from polyhedral parametrizations of Demazure crystals, Adv. in Math., 397 (2022)
  * [Ka] Kiumars Kaveh, Note on cohomology rings of spherical varieties and volume polynomial, J. Lie Theory 21 (2011), no. 2, 263–283
  * [K16] Valentina Kiritchenko, Geometric mitosis, Math. Res. Lett., 23 (2016), no. 4, 1069–1096
  * [K21] Valentina Kiritchenko, Push-pull operators on convex polytopes, IMRN, https://doi.org/10.1093/imrn/rnab331
  * [KP] Valentina Kiritchenko and Maria Padalko, Schubert calculus on Newton–Okounkov polytopes, Chapter 11 in Interactions with lattice polytopes, Springer, 2022
  * [KnM] Allen Knutson and Ezra Miller, Gröbner geometry of Schubert polynomials, Ann. of Math. (2), 161 (2005), 1245–1318
  * [Ko] Mikhail Kogan, Schubert geometry of flag varieties and Gelfand–Cetlin theory, Ph.D. thesis, MIT, 2000
  * [M] Ezra Miller, Mitosis recursion for coefficients of Schubert polynomials, J. Comb. Theory A, 103 (2003), no. 2, 223–235
  * [ST] Evgeny Smirnov and Anna Tutubalina, Pipe dreams for Schubert polynomials of the classical groups, preprint arXiv:2009.14120 [math.CO]
  * [T] Burt Totaro, The torsion index of the spin groups, Duke Math. J., 129 (2005), no.2, 249–290


# A new era of intracluster light studies with JWST

Mireia Montes Instituto de Astrofísica de Canarias, c/ Vía Láctea s/n, E-38205
- La Laguna, Tenerife, Spain  Departamento de Astrofísica, Universidad de La
Laguna, E-38205 - La Laguna, Tenerife, Spain  Ignacio Trujillo Instituto de
Astrofísica de Canarias, c/ Vía Láctea s/n, E-38205 - La Laguna, Tenerife,
Spain  Departamento de Astrofísica, Universidad de La Laguna, E-38205 - La
Laguna, Tenerife, Spain

###### Abstract

Still largely unexplored, the diffuse light in cluster of galaxies traces the
past and on-going buildup of these massive structures. Here, we present the
first comprehensive study of the intracluster light (ICL) of the cluster
SMACS-J0723.3-7327 (z=0.39) using the JWST Early Release Observations. These
deep and high spatial resolution images allow the study of the ICL with high
signal-to-noise ratio up to a radial distance of $\sim 400$ kpc, twice as far
with respect to previous HST studies of intermediate redshift clusters. This
opens up the possibility of exploring the rich mixture of processes that are
building the ICL. We find that the inner parts (R$<$100 kpc) are built through
a major merger while the outer parts (R$>$ 100 kpc) are mainly produced by the
tidal stripping of Milky Way-like satellites. We also found that the slope of
the stellar mass density radial profile of the ICL of this cluster
($\alpha_{3D}=-2.47\pm 0.13$) follows closely the predicted dark matter halo
slope ($\alpha_{3D\mathrm{,DM}}=-2.6$ to $-2$), supporting the idea that both
components have a similar shape and thus the potential of using the ICL as a
tracer of the dark matter distribution in clusters of galaxies. Future JWST
studies of the ICL are set to revolutionise our understanding of cluster
formation and will be crucial to improve the gravitational lensing mass maps
of these structures and thus to accurately characterise the properties of the
first galaxies.

††facilities: JWST (NIRCam)††software: astropy (The Astropy Collaboration et
al., 2018), Source Extractor (Bertin & Arnouts, 1996) SWarp (Bertin, 2010),
photutils v0.7.2 (Bradley et al., 2019), pillow (van Kemenade et al., 2020),
numpy (Oliphant, 2006), scipy (Virtanen et al., 2020)

##  1 Introduction

During the past 20 years, observations have shown that the ICL is a ubiquitous
feature in galaxy clusters (Feldmeier et al., 2002; Krick & Bernstein, 2007;
Kluge et al., 2020). During encounters between galaxies in groups and
clusters, stars are freed from their hosts and end up populating the space
between the galaxies. After some time, these unbound stars form the
characteristic diffuse glow of the ICL. Consequently, this light is a
signature of the assembly of clusters of galaxies (see Montes, 2019; Contini,
2021; Montes, 2022, for recent reviews).

The ICL has been shown to be an unexpected tool to study groups and clusters
of galaxies in detail, allowing us to infer not only their assembly history
but their radius and even dark matter distributions (Montes & Trujillo, 2019;
Alonso Asensio et al., 2020; Deason et al., 2021; Gonzalez et al., 2021). For
example, Mahler et al. (2022) showed that the inclusion of the ICL as a prior
to model the mass distribution of the cluster SMACS-J0723.3-7327 resulted in
more accurate mass maps (1$\farcs$08 of r.m.s compared to the 1$\farcs$26 of
r.m.s without ICL). These results show the extraordinary potential of using
the ICL as an accurate dark matter tracer.

Observations of the ICL of intermediate redshift clusters have been, for the
most part, limited to the blue part of the spectrum. These observations show
clear radial gradients in colors indicating radial gradients in metallicity
(e.g., Mihos et al., 2017; Iodice et al., 2017; DeMaio et al., 2018; Montes et
al., 2021) and, in some cases, age (e.g., Montes & Trujillo, 2014, 2018;
Morishita et al., 2017). Consequently, exploring the ICL distribution and mass
fraction of intermediate redshift clusters using only blue filters is prone to
artificial biases. In this context, it is clear that deep infrared (IR)
imaging with high spatial resolution, as the one provided by JWST (Rigby et
al., 2022), would be a major step forward in the study of clusters. First, the
inclusion of IR wavelengths in the study of the stellar populations would
constrain their properties (age and metallicity) better than using optical
data alone (e.g., Worthey, 1994). Second, the IR explores the restframe
optical/NIR wavelengths of more distant systems enabling a better
characterization of the stellar mass distribution of these objects.

The advent of JWST marks a new era in the detection and study of the low
surface brightness Universe. The exceptional IR sensitivity of JWST’s
detectors holds great promise for the detection of extremely low surface
brightness features. The aim of this work is to showcase, for the first time,
the potential of JWST observations to study the diffuse light in clusters of
galaxies.

Throughout this work, we adopt a standard cosmological model with the
following parameters: $H_{0}=70$ km s-1 Mpc-1, $\Omega_{m}=0.3$ and
$\Omega_{\Lambda}=0.7$. The redshift of this cluster is $z=0.39$,
corresponding to a spatial scale of 5.3 kpc/arcsec. All magnitudes in this
paper are in the AB magnitude system.

##  2 Data

Figure 1: The cluster SMACS0723. The left panel shows an RGB composite of the
long wavelength bands with a black and white background that is an average of
the three filters. Some of the most prominent features in the ICL of the
cluster are marked with arrows. The inset on the top left is to better show
the features in the center of the cluster. The right panel shows an average of
the three long wavelength bands with two isocontours overplotted: 25 (red) and
27.5 (orange) mag/arcsec2. North is up and East is left.

The JWST Near Infrared Camera (NIRCam111https://jwst-docs.stsci.edu/jwst-near-
infrared-camera/nircam-instrumentation/nircam-detector-overview, Rieke et al.,
2005) imaging is obtained at 0.6–2.3 $\mu$m (0.031″/pix, short wavelength
channel) and 2.4-5.0 $\mu$m (0.063″/pix, long wavelength channel)
_simultaneously_ over a 9.7 arcmin2 field of view. NIRCam uses a dichroic to
observe in both channels in roughly the same field of view. The camera
consists of two modules (A and B) separated by a $\sim$44″gap. Each of the
detectors contains $2048\times 2048$ pixels. Only the inner $2040\times 2040$
pixels are for science, and the outer 4-pixel wide border are reference pixels
used for calibration. The short wavelength channel has 4 detectors, and
therefore twice the resolution, and the long wavelength channel one, in each
of the modules.

Deep observations of the cluster SMACS-J0723.3-7327 (hereafter SMACS0723, Repp
& Ebeling, 2018) were taken with the NIRCam filters F090W, F150W, F200W,
F277W, F356W, and F444W. NIRCam module B was centered on the cluster, and
module A on an adjacent field. The cluster was observed on June 7th 2022 with
several instruments aboard JWST as part of the observatory’s Early Release
Observations (ERO, Pontoppidan et al., 2022). The primary data set used for
this work is based in the images obtained with NIRCam, retrieved from the
Mikulski Archive for Space Telescopes
(MAST)222https://outerspace.stsci.edu/display/MASTDATA/JWST+AWS+Bulk+Download+Scripts.

###  2.1 Re-processing of the calibrated files

The calibrated, co-added final (_Stage 3_) released images present light
gradients across the image, making them unsuitable to study the ICL of this
cluster.

In this Section, we will explain our attempt at improving the data quality,
re-reducing these images in a more suitable way for the purposes of this
paper. In this work, we concentrate our efforts on the long wavelength
channel. The short wavelength channel needs a more elaborate re-reduction that
is beyond the scope of this work. See Appendix B for more details.

####  2.1.1 Long wavelength channel

The long wavelength channel consists of two detectors, one for each module.
The pixel scale of this channel is $0.063\arcsec$/pixel. The re-calibration of
the long wavelength channel images were done in a two-step process. The light
gradients in the final calibrated images (_Stage 3_) are clearly an artefact.
They are different from the ones observed in the short wavelength filters and
not present in the RELICS HST images (Coe et al., 2019). The origin of these
light gradients is not clear but they are observed both in the individual
frames and the final co-add of all the long wavelength images.

To correct these gradients, first, we downloaded from MAST the calibrated
(_Stage 2_) individual frames of the F277W, F356W and F444W bands. For the
first step of the process, we masked in each of the frames all galaxies,
foreground and background objects and the central parts of the cluster. Then,
we fitted a second degree 2D polynomial to the masked frame to model the
gradient seen in the individual frames. The size of this plane is larger than
the physical extent of the ICL in the images. This ensures that we are only
correcting the gradient in the image while preserving the diffuse light of the
cluster. In Appendix A, we show an example of this process.

With the background of the individual images corrected, we created a
preliminary co-add of the 9 exposure frames, per filter. This first co-add is
then used to build a more accurate mask of the cluster, adding manually masked
regions to cover the central parts of the image and minimize ICL
contamination. This improved mask is then applied to the individual frames to
obtain a more reliable model of the light gradient, for each of the frames.
Once the frames are corrected from this gradient, we use SWarp (Bertin, 2010)
to create the final co-add. Finally, we computed and subtracted a constant
background value from the final co-added images.

Note that we have used the original released images (Pontoppidan et al.,
2022). It is known that there is a systematic magnitude offset due to the
$20\%$ higher throughput of the filters in the long wavelength channel (Rigby
et al., 2022). Therefore, we correct the photometry of the images using the
offsets listed in Adams et al. (2022). The offsets are: +0.223, +0.163 and
+0.162 for the F277W, F356W and F444W bands, respectively. The zeropoints to
convert to mag/arcsec2 of each image are computed using the transformation in
Appendix Sec. C.

The re-reduced images of the long wavelength channels are available through
this link333
https://www.dropbox.com/sh/zzb7fl1j0vl1vit/AABW1uz06Vn2XGCYAmOJIbega?dl=0.
Although, we do not make use of them in this work, we have also made available
the re-reduced images of the short wavelength channel.

###  2.2 Masking

The study of the ICL in clusters of galaxies requires very careful masking of
sources to reduce contamination that can affect the intrinsic properties of
this light. In the case of deep images, this masking must be optimized for
faint and small objects and for bright and large objects, background,
foreground as well as belonging to the cluster. We use SExtractor’s
segmentation map (Bertin & Arnouts, 1996) to build the masks for these images.

As a single setup for the detection and masking of both types of sources is
unfeasible, we used a similar approach than in Montes & Trujillo (2018);
Montes et al. (2021); two SExtractor runs: one to detect small sources (“hot”
mode) and another to detect larger, more extended objects (“cold” mode). We
use this approach on a median F277W + F356W + F444W image. In the case of the
“hot” mode, we unsharp-masked the original image, to enhance the contrast of
the smallest sources, particularly in the central parts of the cluster. To
make the unsharp-masked image, we convolved the original image with a median
box filter with a side of $5$ pixels and then we subtracted it from the
original. This “hot” mask was further expanded 4 pixels. The brightest cluster
galaxy (BCG) is left unmasked in both the “hot” and “cold” masks.

The final mask was again visually inspected to manually mask any remaining
light that was missed by the process described above. The spikes of the bright
stars were also manually masked. In this final mask, all the objects are
masked except for the BCG and ICL.

###  2.3 Surface brightness limits

Our goal is to study the low surface brightness features in SMACS0723 down to
the faintest surface brightness possible. For this reason, we need to know how
deep our images are by estimating the surface brightness limits that they
reach. To obtain these limits, we calculated the r.m.s per pixel of the final
masked images and apply the formula given in Román et al. (2020) to transform
it to the typical $10\arcsec\times 10\arcsec$ size. The $3\sigma$ surface
brightness limits for the long wavelength channel are listed in Table 1.

Filter | Channel | Exp. Time | Surface Brightness limits∗
---|---|---|---
|  | (seconds) | (mag/arcsec2)
F277W | Long | $7537.2$ | 31.28
F356W | Long | $7537.2$ | 31.32
F444W | Long | $7537.2$ | 31.10
Table 1: Summary of the NIRCam observations used in this work. ∗The surface
brightness limits correspond to a sky fluctuation of 3$\sigma$ in an area of
$10\times 10$ arcsec2.

##  3 The ICL of SMACS0723

SMACS0723 is a cluster at z = 0.39 discovered as part of the southern
extension of the Massive Cluster Survey (Repp & Ebeling, 2018), with a mass of
$M_{h}=8.39^{+0.33}_{-0.34}\times 10^{14}M_{\odot}$ (Coe et al., 2019). This
cluster was selected for the JWST’s ERO for its gravitational lensing
potential, as shown by the prominent lensed arcs, and high latitude (i.e., low
Zodiacal emission, Pontoppidan et al., 2022).

Fig. 1 shows the final re-reduced images of SMACS0723. The left panel is an
RGB color composite image of the F277W, F356W and F444W filters with an
average of these three filters as a black and white background. The most
notable features in the image are highlighted: two East Streams (1 and 2), the
Giant West Loop and plumes East and West of the BCG. The East Stream 1 is
located at 150 kpc from the BCG, while the East Stream 2 is at 243 kpc. The
Giant West Loop covers a diameter of 100 kpc, from 150 kpc to 250 kpc from the
center of the BCG. The width of the loop is quite homogeneous throughout the
structure ($\sim 30$ kpc). This suggests that it could be the remains of the
tidal destruction of a galaxy of similar diameter. This will be discussed in
the next sections.

The plumes seen in the center of the cluster span between 25 to 100 kpc from
the BCG. They seem to be the result of the interaction of the BCG with a
massive galaxy (Lauer, 1988). The inset in the left panel of Fig. 1 shows a
zoom-in into the central part of the cluster to show the plumes more clearly.

The right panel of Fig. 1 shows the image average of the three filters with
two isocontours: one representative of the galaxies of the cluster (red, 25
mag/arcsec2) and one of the ICL at a distance of $\sim 350$ kpc from the
center of the BCG (orange, 27.5 mag/arcsec2).

###  3.1 Radial surface brightness profiles of the BCG + ICL

In this section we show the surface brightness radial profiles along the two
semi-major axis of the cluster. This is done to highlight the asymmetries that
are concentrated along this axis. We followed a similar approach as in Montes
et al. (2021).

We masked everything except for a 20 deg-wide section towards the East and a
20-deg wide section to the West. We derived the radial profiles in each of the
3 bands in 51 circular logarithmic-spaced bins from 0 to 72 arcsec (or 380
kpc). The profiles for the East (right panel) and West (left panel) directions
are shown in Fig. 2. The insets in Fig. 2 show the region used to derive the
profiles. The errors, smaller than the size of the markers, are computed as
the quadratic sum of the r.m.s scatter of the signal and the r.m.s scatter of
the background for each bin. The profiles are corrected for the extinction of
the Galaxy (E(B$-$V)=0.193, Schlafly & Finkbeiner, 2011) using the Cardelli et
al. (1989) extinction law. The gaps in the profiles correspond to masked
regions in the images.

The surface brightness profiles are shown down to $\mu_{F444W}$ $\sim 28$
mag/arcsec2 ($\sim 350$ kpc). This is around a factor of 20 above the limiting
surface brightness of these images (Table 1). We conservatively decided not to
explore the ICL beyond $\mu_{F444W}$ $\sim 28$ mag/arcsec2 as the current
background of the data is still affected by light gradients left during the
re-reduction process.

Figure 2: Surface brightness profiles as a function of radius for the BCG +
ICL of SMACS0723, for the F277W (yellow), F356W (orange) and F444W (red)
bands. Left panel shows the profiles towards the West while the right panel
shows the profiles towards the East. The inset in both panels indicates the
regions where the profiles were extracted.

###  3.2 Color and stellar mass density profiles

The left panel of Fig. 3 shows the $F277W-F444W$ color profiles for the East
(peach orange, squares) and West (pink, circles) profiles down to $300$ kpc
from the center of the BCG. The errors are the combination of the errors from
the radial profiles. As the outer parts of the images are more prone to
systematics on the background subtraction, we conservatively show the radial
color profiles down to a radial distance of $\sim 300$ kpc.

In the color profile plot, we also indicate the position of the features
highlighted in Fig. 1. For instance, the position of the Giant West Loop is
evidenced by a wide ($\sim 100$ kpc) dip of $\sim-0.6$ magnitudes in depth in
the West profile. Instead, the East Streams appear as bumps in the East color
profiles, indicating that the ICL is being enriched by the addition of redder
stars. The plumes at each side of the BCG coincide with the almost constant
color profile in the inner $\sim 100$ kpc in both East and West color
profiles; a signature of a major merger as the merger will lead to the
flattening of the metallicity gradient of the BCG (e.g., White, 1980).

The right panel in Fig. 3 shows the stellar mass density profiles of SMACS0723
for the East (peach yellow) and West (pink) profiles. A restframe $3.6\mu$m
filter would be the most representative tracer of the underlying stellar mass
distribution (e.g., Sheth et al., 2010). Therefore, we used as a reference the
F444W band as it is the closest to a $3.6\mu$m filter at the redshift of
SMACS0723. To estimate the surface stellar mass density profile, we apply
equation 1 in Montes & Trujillo (2014), to transform the F444W surface
brightness profile to stellar mass density. We used the the mass-to-light
ratios (M/L) predictions in the Spitzer IRAC $3.6\mu$m band from the E-MILES
models (Vazdekis et al., 2016).

The BCG+ICL profiles of clusters at intermediate redshifts have negative
gradients of both metallicity and age (e.g., Montes & Trujillo, 2014;
Morishita et al., 2017; Montes & Trujillo, 2018). Lacking precise information
about the age and metallicity of the BCG and ICL of this cluster, we took two
M/L ratios: for an age of $7$ Gyr and [M/H] = $0.22$ ($0.70$, solid lines),
and age $1.5$ Gyr and [M/H] = $-0.7$ ($0.18$, dashed lines) based on the
values of age and metallicity of the Frontier Fields clusters for the centre
and ICL from Montes & Trujillo (2018). By selecting these two extreme
predictions for the M/L, we ensure that the most likely stellar mass density
profile of the cluster is within the two lines in the right panel of Fig. 3.

Both stellar mass density profiles, East and West, follow a power-law profile.
The profiles also present the different features highlighted in Fig. 1 like
the Giant West Loop dip at $\sim 200$ kpc and the bumps of the East Streams 1
($150$ kpc) and 2 ($240$ kpc).

Figure 3: Left panel: the F277W$-$F444W radial color profile of the BCG+ICL of
SMACS0723. The features seen in the ICL are highlighted in the plot. Right
panel: Stellar mass density radial profile of the BCG+ICL of SMACS0723. The
solid line indicates the density profile derived using a M/L of 0.7 and the
dashed line with a M/L of 0.18. The grey shaded area indicates the range of
possible density profiles for this cluster. Note that the X-axis in this
right-hand plot is in logarithmic scale.

##  4 Discussion and conclusions

The results presented in this work show the extraordinary potential of JWST to
unravel the physical mechanisms behind the origin and evolution of the ICL.
However, the current JWST NIRCAM data reduction is not yet low surface
brightness-friendly. A tailored data reduction is a must in order to take the
most advantage of the observations to study the faintest surface brightness
features.

###  4.1 ICL formation in SMACS0723

The asymmetry of the diffuse light in SMACS0723, along the East-West
direction, means that the ICL of this cluster is still forming. The signature
of this formation process can be seen in the prominent features that show up
in both the surface brightness radial profiles in Fig. 2 and the radial color
profile in the left panel of Fig. 3. In the following, we will discuss these
features in detail.

####  4.1.1 The Giant West Loop and the East Streams

Early observations showed that the ICL originates from the tidal stripping of
satellite galaxies as they interact inside the cluster (e.g., Gregg & West,
1998; Mihos et al., 2005). This appears to be the origin of the features
observed at larger radii, such as the East Streams and the Giant West Loop in
Fig. 4.

The circular shape of the Giant West Loop is composed by two parts. Its left
side corresponds to the diffuse light from the center of the cluster rapidly
decreasing. The right arc-like shape is likely the debris of a single stellar
stream. The constant width of this feature ($\sim 30$ kpc) and homogeneous
color is consistent with being a single stellar stream. In Fig. 4 upper left
panel, we show the RGB color composite image of the region, labelling two of
the galaxies that could potentially be associated with the loop.

The upper right panel shows the F277W$-$F444W color map of the same region.
Although the North part of the loop seems to be associated with Galaxy A, this
galaxy is redder than the stream and the galaxies of the cluster. In fact, it
has a photometric redshift of 1.82 (Coe et al., 2019), placing it behind the
cluster and not physically associated to the stream. Galaxy B, on the other
hand, has colors that are compatible with that of the stream. The photometric
redshift of this galaxy is 0.32 (Coe et al., 2019). However, the position of
this galaxy is inconsistent with having created the entire Giant West Loop.
Therefore, it remains an open question which galaxy has created this
structure.

Both East Streams are shown in the lower panels of Fig. 4. The East Stream 1
is narrower (with a width of $\sim 15$ kpc) than the other features
highlighted in this paper. In the color map, lower right panel, this feature
does not show up as there is little contrast between the stream and the ICL
background (see also Fig. 3). The color and width of this stream potentially
associates it with Galaxy C. The East Stream 2 is wider ($\sim 30$ kpc),
likely produced by the stripping of Galaxy D. This stream has a similar color
than the ICL in the inner parts of the cluster, as seen in the color map.

The widths of both the Giant West Loop and the East Stream 2 are compatible
with the size of Milky Way-like objects (i.e. with a diameter of $\sim 30$ kpc
Trujillo et al., 2020). This is in agreement with the finding that the stars
composing the ICL are mainly stripped from the outskirts of galaxies with
stellar masses around $5\times 10^{10}$ M⊙ (Montes & Trujillo, 2014, 2018;
Morishita et al., 2017; DeMaio et al., 2018).

Figure 4: Zoom-ins into the different tidal features of SMACS0723. Left panels
show the RGB composite images, while the right panels show the F277W$-$F444W
color maps. The upper panels show the Giant West Loop and the lower panels the
East Streams. The width of the different features is indicated with arrows.

####  4.1.2 Plumes

The plumes seen in the center of the cluster ($\sim 25-100$ kpc, Fig. 1)
indicate that the BCG of this cluster is likely experiencing a merger (see
also Duc et al., 2015; Martin et al., 2022). At the same time, the measured
color is flat in the inner $\sim 100$ kpc in both East and West profile,
indicating that the stellar populations in this region are well mixed. Flat
gradients in color are indicative of stars expelled into the intracluster
medium during a major merger ($<$1:10, Krick & Bernstein, 2007; DeMaio et al.,
2015, 2018). The major merger is likely with the galaxy located 9 arcsec
($\sim 47$ kpc) away to its West, as both galaxies are connected by a stellar
bridge (zoom-in in Fig. 1). Assuming similar stellar population properties
between both galaxies, the merger ratio inferred is 1:3.

The distinct features that compose the diffuse light of this cluster (plumes,
streams, loop, etc) suggest different physical mechanisms creating the ICL
(mergers and tidal disruption). This is telling us that the mechanisms that
contribute to ICL formation do that differently depending on their distance to
the centre of the cluster.

###  4.2 The slope of the stellar mass density of the ICL

The high degree of structure in the ICL of this cluster suggests that the
diffuse extended component is being built now. Thanks to the dense environment
of the cluster, in a few Gyrs these structures will smooth and become well-
mixed (Rudick et al., 2011). Over time, the tidal stripping of more and more
satellites will build the outer ICL (DeMaio et al., 2018, 2020).

To get an estimation of the slope of the stellar mass density profile, we have
fitted a power-law to the BCG+ICL profile shown in Fig 3. As the centers of
BCGs are usually old and high in metallicity, we use this fact as the starting
point to fit the central regions. The slope and its errors encompass the
possible solutions showed in Fig 3. The slope is $\alpha_{2D,E}=-1.48\pm
0.13$, for the East profile, and $\alpha_{2D,W}=-1.46\pm 0.08$, for the West
profile. Both values are in agreement within errors. That translates into a 3D
slope of $\sim-2.47$ (eq. 5 in Stark, 1977). The value of the 3D slope is in
agreement with the values of the slopes in Montes & Trujillo (2018) for the
Frontier Fields clusters. This result implies that the stellar halo and the
dark matter of SMACS0723 could have similar slopes (the dark matter slopes
range between $-2.6$ to $-2$, Pillepich et al., 2014, 2018), something
expected for high-mass clusters ($M_{h}=8.39^{+0.33}_{-0.34}\times
10^{14}M_{\odot}$, Coe et al., 2019).

In this work, we present the first analysis of the ICL of the ERO observations
with JWST. We re-reduced the original JWST released images to minimize the
inhomogeneities and make them more suitable for ICL studies. After our
reprocessing, we measured that the ICL covers around half of the NIRCam
detector area. We derive the surface brightness radial profiles out to $380$
kpc from the center of the BCG, and the F277W$-$F444W color profiles to $300$
kpc. We found that:

  * •

The ICL of SMACS0723 presents a lot of substructure indicating that the
processes that form the ICL in this cluster are caught in action.

  * •

The color and morphological properties of the ICL in the inner $\sim 100$ kpc
are consistent with a major merger.

  * •

The outer ICL ($>150$ kpc) is being formed by the tidal stripping of Milky
Way-like galaxies.

  * •

The slope of the stellar mass density profile of the BCG+ICL
($\alpha_{3D}=-2.47\pm 0.13$) is nearly identical to the slope of the dark
matter profile of clusters of this mass. This reinforces the idea of using the
ICL as a potential dark matter tracer. Considering this, we encourage using
the ICL to improve the gravitational lensing mass maps and, consequently, the
characterization of the properties of the first galaxies.

This publication is part of the Project PCI2021-122072-2B, financed by
MICIN/AEI/10.13039/501100011033, and the European Union
“NextGenerationEU”/RTRP. I.T. acknowledges support from the ACIISI, Consejería
de Economía, Conocimiento y Empleo del Gobierno de Canarias and the European
Regional Development Fund (ERDF) under grant with reference PROID2021010044
and from the State Research Agency (AEI-MCINN) of the Spanish Ministry of
Science and Innovation under the grant PID2019-107427GB-C32 and IAC project
P/300624, financed by the Ministry of Science and Innovation, through the
State Budget and by the Canary Islands Department of Economy, Knowledge and
Employment, through the Regional Budget of the Autonomous Community. The
authors want to thank the JWST ERO team for making these extraordinary data
available.

##  Appendix A Example of the data processing of the images from the long
wavelength channel

In Sec. 2, we detailed the data re-reduction that we performed in the JWST
NIRCam images. For the long wavelength channel, we fit a second degree 2D
polynomial (a plane) to the calibrated images to eliminate the gradient seen
across the image. The left panel in Fig. 5 shows a Stage 2 calibrated frame.
The middle panel shows the plane fitted to the masked original calibrated
frame. The right panel shows our final result, without the gradient.

Fig. 6 shows the improvement of the data reduction performed here. Both panels
show RGB color composite images using the long wavelength bands and with an
average of the three filters as the black and white background. The original
images (in the left) are those using the JWST current reduction pipeline
whereas our re-reduction is shown in the right panel.

Figure 5: An example of a single frame background correction. Left panel:
Calibrated F277W frame (jw02736001001_02101_00003_nrcblong_cal.fits). A
gradient across the image can be seen, towards the upper-right corner. Middle
panel: 2nd degree polynomial plane fitted to the (masked) calibrated frame to
correct the gradient. Right panel: Final frame after subtracting the plane.
Figure 6: Left panel: RGB color composite image using the original calibrated
co-adds of the long wavelength channels. Right panel: RGB color composite
image using the re-reduced co-add images created in this work. Note the huge
improvement of correcting the strong light gradient towards the East side on
the NIRCam camera.

##  Appendix B Short wavelength channel re-reduction

The short wavelength channel consists of a mosaic of 8 detectors (4 for each
module), manufactured with a 2.5 $\mu$m wavelength cutoff. In each module, the
4 short wavelength detectors are arranged in a $2\times 2$ array. The pixel
scale of this channel is $0.031\arcsec$/pixel.

Contrary to the case of the long wavelength channel, the detectors in the
short wavelength channel are smaller than the extent of the diffuse light of
the cluster. For this reason, fitting a plane to each one of the detectors is
risky, more so when the detectors do not show any common gradient. In this
case, we opted for a more conservative approach: to correct for the readout
pattern per detector (see Pontoppidan et al., 2022; Merlin et al., 2022) and
scale the 4 detectors.

One of the characteristics of the NIRCam detectors is the presence of low-
level electronic noise manifesting as linear stripes (‘1/f noise’, see also
Pontoppidan et al., 2022). The released ERO images are already corrected from
this noise using a simple correction. However, residuals are left, clearly
visible in the short wavelength channel images (left panel Fig. 7).

The first step was to remove this remaining pattern by subtracting the
$3\sigma$-clipped median value for each row, horizontally, in the calibrated
individual frames after masking out all the objects. Once this correction was
applied, we derived an improved mask from the corrected images and compute and
subtract the median value for each row.

Once the images per detector were corrected from the remaining ‘1/f noise’, we
built the mosaics. The left panel in Fig. 7 shows that in the final calibrated
mosaics the different detectors were not properly scaled. Therefore, we have
also scaled the mosaics by multiplying each detector by a factor, taking
detector one (nrcb1) as reference. This factor is computed as the
$3\sigma$-clipped median, for each frame, in a ring of radius $\sim 47\arcsec$
and width of $6\arcsec$ from the center of the frame, after masking all
sources. After scaling the different detectors, the frames were assembled
using SWarp. The final output is created as the median of the 9 mosaic frames.

An example of the data reduction for the F090W band is shown in Fig. 7. For
comparison, we show the original image in the left panel, and our data
reduction in the right panel. Despite the clear improvement, the final images
of the short wavelength channel still show significant residuals that prevent
us to use them to study the ICL. These residuals seem to be caused by
inhomogeneities in the 4 different adjacent sectors ($512\times 2048$ pixels)
that form each of the
detectors444https://www.stsci.edu/files/live/sites/www/files/home/jwst/documentation/technical-
documents/_documents/JWST-STScI-001721.pdf.

Figure 7: Left panel: Original calibrated F090W co-add. Right panel: Final co-
add after the data re-reduction.

##  Appendix C From MJy/sr to surface brightness in JWST data

In this work, all the units of the photometry, surface brightness and color
profiles, are given in mag/arcsec2 (in the AB system). The explicit equation
to transform from MJy/sr (the units of the JWST images) to surface brightness
in the AB system is:

$\mu_{AB}=-2.5\times\log_{10}(\mathrm{Counts}\,[\mathrm{MJy/sr}]\times\frac{10^{6}}{4.25\times
10^{10}})+8.906$ (C1)

Where 4.25$\times$1010 corresponds to number of arcsec2 in 1 sr. The factor
106 stands for the transformation from MJy to Jy. Finally, the 8.906 value
corresponds to the AB system referred to 3631 Jy as the reference unit (Oke &
Gunn, 1983).

## References

  * Adams et al. (2022) Adams, N. J., Conselice, C. J., Ferreira, L., et al. 2022, arXiv e-prints, arXiv:2207.11217.  https://arxiv.org/abs/2207.11217
  * Alonso Asensio et al. (2020) Alonso Asensio, I., Dalla Vecchia, C., Bahé, Y. M., Barnes, D. J., & Kay, S. T. 2020, MNRAS, 494, 1859, doi: 10.1093/mnras/staa861
  * Bertin (2010) Bertin, E. 2010, SWarp: Resampling and Co-adding FITS Images Together.  http://ascl.net/1010.068
  * Bertin & Arnouts (1996) Bertin, E., & Arnouts, S. 1996, A&AS, 117, 393, doi: 10.1051/aas:1996164
  * Bradley et al. (2019) Bradley, L., Sipocz, B., Robitaille, T., et al. 2019, astropy/photutils: v0.7.1, v0.7.1, Zenodo, doi: 10.5281/zenodo.3478575
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245, doi: 10.1086/167900
  * Coe et al. (2019) Coe, D., Salmon, B., Bradač, M., et al. 2019, ApJ, 884, 85, doi: 10.3847/1538-4357/ab412b
  * Contini (2021) Contini, E. 2021, Galaxies, 9, 60, doi: 10.3390/galaxies9030060
  * Deason et al. (2021) Deason, A. J., Oman, K. A., Fattahi, A., et al. 2021, MNRAS, 500, 4181, doi: 10.1093/mnras/staa3590
  * DeMaio et al. (2015) DeMaio, T., Gonzalez, A. H., Zabludoff, A., Zaritsky, D., & Bradač, M. 2015, MNRAS, 448, 1162, doi: 10.1093/mnras/stv033
  * DeMaio et al. (2018) DeMaio, T., Gonzalez, A. H., Zabludoff, A., et al. 2018, MNRAS, 474, 3009, doi: 10.1093/mnras/stx2946
  * DeMaio et al. (2020) —. 2020, MNRAS, 491, 3751, doi: 10.1093/mnras/stz3236
  * Duc et al. (2015) Duc, P.-A., Cuillandre, J.-C., Karabal, E., et al. 2015, MNRAS, 446, 120, doi: 10.1093/mnras/stu2019
  * Feldmeier et al. (2002) Feldmeier, J. J., Mihos, J. C., Morrison, H. L., Rodney, S. A., & Harding, P. 2002, ApJ, 575, 779, doi: 10.1086/341472
  * Gonzalez et al. (2021) Gonzalez, A. H., George, T., Connor, T., et al. 2021, arXiv e-prints, arXiv:2104.04306.  https://arxiv.org/abs/2104.04306
  * Gregg & West (1998) Gregg, M. D., & West, M. J. 1998, Nature, 396, 549, doi: 10.1038/25078
  * Iodice et al. (2017) Iodice, E., Spavone, M., Cantiello, M., et al. 2017, ApJ, 851, 75, doi: 10.3847/1538-4357/aa9b30
  * Kluge et al. (2020) Kluge, M., Neureiter, B., Riffeser, A., et al. 2020, ApJS, 247, 43, doi: 10.3847/1538-4365/ab733b
  * Krick & Bernstein (2007) Krick, J. E., & Bernstein, R. A. 2007, AJ, 134, 466, doi: 10.1086/518787
  * Lauer (1988) Lauer, T. R. 1988, ApJ, 325, 49, doi: 10.1086/165982
  * Mahler et al. (2022) Mahler, G., Jauzac, M., Richard, J., et al. 2022, arXiv e-prints, arXiv:2207.07101.  https://arxiv.org/abs/2207.07101
  * Martin et al. (2022) Martin, G., Bazkiaei, A. E., Spavone, M., et al. 2022, MNRAS, 513, 1459, doi: 10.1093/mnras/stac1003
  * Merlin et al. (2022) Merlin, E., Bonchi, A., Paris, D., et al. 2022, arXiv e-prints, arXiv:2207.11701.  https://arxiv.org/abs/2207.11701
  * Mihos et al. (2005) Mihos, J. C., Harding, P., Feldmeier, J., & Morrison, H. 2005, ApJ, 631, L41, doi: 10.1086/497030
  * Mihos et al. (2017) Mihos, J. C., Harding, P., Feldmeier, J. J., et al. 2017, ApJ, 834, 16, doi: 10.3847/1538-4357/834/1/16
  * Montes (2019) Montes, M. 2019, arXiv e-prints, arXiv:1912.01616.  https://arxiv.org/abs/1912.01616
  * Montes (2022) —. 2022, Nature Astronomy, 6, 308, doi: 10.1038/s41550-022-01616-z
  * Montes et al. (2021) Montes, M., Brough, S., Owers, M. S., & Santucci, G. 2021, ApJ, 910, 45, doi: 10.3847/1538-4357/abddb6
  * Montes & Trujillo (2014) Montes, M., & Trujillo, I. 2014, ApJ, 794, 137, doi: 10.1088/0004-637X/794/2/137
  * Montes & Trujillo (2018) —. 2018, MNRAS, 474, 917, doi: 10.1093/mnras/stx2847
  * Montes & Trujillo (2019) —. 2019, MNRAS, 482, 2838, doi: 10.1093/mnras/sty2858
  * Morishita et al. (2017) Morishita, T., Abramson, L. E., Treu, T., et al. 2017, ApJ, 846, 139, doi: 10.3847/1538-4357/aa8403
  * Oke & Gunn (1983) Oke, J. B., & Gunn, J. E. 1983, ApJ, 266, 713, doi: 10.1086/160817
  * Oliphant (2006) Oliphant, T. E. 2006, A guide to NumPy, Vol. 1 (Trelgol Publishing USA)
  * Pillepich et al. (2014) Pillepich, A., Vogelsberger, M., Deason, A., et al. 2014, MNRAS, 444, 237, doi: 10.1093/mnras/stu1408
  * Pillepich et al. (2018) Pillepich, A., Nelson, D., Hernquist, L., et al. 2018, MNRAS, 475, 648, doi: 10.1093/mnras/stx3112
  * Pontoppidan et al. (2022) Pontoppidan, K., Blome, C., Braun, H., et al. 2022, arXiv e-prints, arXiv:2207.13067.  https://arxiv.org/abs/2207.13067
  * Repp & Ebeling (2018) Repp, A., & Ebeling, H. 2018, MNRAS, 479, 844, doi: 10.1093/mnras/sty1489
  * Rieke et al. (2005) Rieke, M. J., Kelly, D. M., & Horner, S. D. 2005, in Cryogenic Optical Systems and Instruments XI, ed. J. B. Heaney & L. G. Burriesci, Vol. 5904, International Society for Optics and Photonics (SPIE), 1 – 8, doi: 10.1117/12.615554
  * Rigby et al. (2022) Rigby, J., Perrin, M., McElwain, M., et al. 2022, arXiv e-prints, arXiv:2207.05632.  https://arxiv.org/abs/2207.05632
  * Román et al. (2020) Román, J., Trujillo, I., & Montes, M. 2020, A&A, 644, A42, doi: 10.1051/0004-6361/201936111
  * Rudick et al. (2011) Rudick, C. S., Mihos, J. C., & McBride, C. K. 2011, ApJ, 732, 48, doi: 10.1088/0004-637X/732/1/48
  * Schlafly & Finkbeiner (2011) Schlafly, E. F., & Finkbeiner, D. P. 2011, ApJ, 737, 103, doi: 10.1088/0004-637X/737/2/103
  * Sheth et al. (2010) Sheth, K., Regan, M., Hinz, J. L., et al. 2010, PASP, 122, 1397, doi: 10.1086/657638
  * Stark (1977) Stark, A. A. 1977, ApJ, 213, 368, doi: 10.1086/155164
  * The Astropy Collaboration et al. (2018) The Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, ArXiv e-prints.  https://arxiv.org/abs/1801.02634
  * Trujillo et al. (2020) Trujillo, I., Chamba, N., & Knapen, J. H. 2020, MNRAS, 493, 87, doi: 10.1093/mnras/staa236
  * van Kemenade et al. (2020) van Kemenade, H., wiredfool, Murray, A., et al. 2020, python-pillow/Pillow 7.1.1, 7.1.1, Zenodo, doi: 10.5281/zenodo.3738618
  * Vazdekis et al. (2016) Vazdekis, A., Koleva, M., Ricciardelli, E., Röck, B., & Falcón-Barroso, J. 2016, MNRAS, 463, 3409, doi: 10.1093/mnras/stw2231
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261, doi: 10.1038/s41592-019-0686-2
  * White (1980) White, S. D. M. 1980, MNRAS, 191, 1P, doi: 10.1093/mnras/191.1.1P
  * Worthey (1994) Worthey, G. 1994, ApJS, 95, 107, doi: 10.1086/192096