# Invariant splitting of a slab into infinitely many robust clusters

Péter Mester111HUN-REN Alfréd Rényi Institute of Mathematics, Budapest

###### Abstract

We give an example of an invariant bond percolation process on the slab
${\mathbb{Z}}^{2}\times\\{0,1\\}$ with the property that it has infinitely
many clusters whose critical percolation probability is strictly less than
$1$. We also show that no such process can exist in ${\mathbb{Z}}^{2}$.

##  1 Introduction

An invariant bond percolation process on a graph $G$ is a random subset of the
edges whose distribution is invariant under all the graph automorphisms of
$G$. Connected components formed by the process are called clusters, and a
cluster is called robust if its critical percolation probability is strictly
less than $1$ (i.e., if we delete edges of the cluster independently from each
other with a small enough probability $\epsilon>0$, then there is still an
infinite component). In this paper we show the following:

######  Theorem 1.1.

There is an invariant bond process $\Phi_{\beta}({\tt RF})$ on the planar slab
${\mathbb{Z}}^{2}\times\\{0,1\\}$ which has infinitely many robust clusters.

######  Theorem 1.2.

On ${\mathbb{Z}}^{2}$ there is no invariant process which has more than two
robust clusters.

With extra conditions, such as positive associations, even having more than
one robust clusters is impossible on ${\mathbb{Z}}^{2}$ — see [GKR88] and
[Sh05]. But our general negative result seems to be new. Thus the slab
${\mathbb{Z}}^{2}\times\\{0,1\\}$ is radically different from the planar
lattice ${\mathbb{Z}}^{2}$ regarding what kind of percolation processes it can
support. We think this is interesting as there are papers, such as [DST15]
(which shows the nonexistence of infinite clusters at critical Bernoulli
percolation in the case of two-dimensional slabs
${\mathbb{Z}}^{2}\times\\{0,1,\dots,k\\}$) or [NTW17] (which shows that the
minimal spanning forest of a quasi-planar slab
${\mathbb{Z}}^{2}\times\\{0,\dots,k\\}$ is a tree) where it is shown that a
particular process behaves in the same way on quasi-planar slabs as on the
planar lattice ${\mathbb{Z}}^{2}$. While the aforementioned papers deal with
processes which are well-defined and natural for a wide range of graphs, our
process is tied to ${\mathbb{Z}}^{2}\times\\{0,1\\}$, and we think it would be
interesting to see if a “more natural process” (at least well-defined on a
wide range of graphs) can also produce infinitely many robust clusters on
${\mathbb{Z}}^{2}\times\\{0,1\\}$. Regarding the corresponding question where
the base graph is non-amenable, [PT22] shows that the Free Uniform Spanning
Forest can have radically different cluster structure on different finite
extensions of a tree.

In this paper all but the last section is devoted to the description of the
process $\Phi_{\beta}({\tt RF})$ witnessing the truth of Theorem 1.1, while
the last section proves Theorem 1.2.

In [HäM09], we constructed a translation invariant site process ${\tt RF}$
(for Random Forks) with the property that it has a robust open and a robust
closed cluster (we explain in Section 5, how to get an invariant one from a
translation invariant one). We will build our process out of this ${\tt RF}$
and we will make an effort to explain ${\tt RF}$ in a self-contained way:
instead of reproducing the formalism there, we will offer illustrations and
point out the delicate points of the construction.

${\tt RF}$ is built from a collection ${\tt Rect}={\tt Rect}_{\tt V}\sqcup{\tt
Rect}_{\tt H}$ (the operation is disjoint union) of vertical and horizontal
rectangles which overlap in a very regular way, and from any $R\in{\tt RF}$ we
can start a “road” of ever widening rectangles (the intersection graph built
from the rectangles of ${\tt Rect}$ is a one-ended tree, so for $R\in{\tt
Rect}$ there will be a unique ${\tt next}(R)\in{\tt Rect}$ which is in the
direction of infinity from $R$). In a Bernoulli $p$ percolation (for some
$p>1/2$), with positive probability a so-called crossing event holds in each
rectangle of a road, which implies the existence of an infinite path within
the road (thus the road itself is robust). Figure 1.1 illustrates this.

Figure 1.1: The shaded part of the left picture is ${\tt RF}$, within which
the dark region illustrates a “road”. The right picture illustrates that if a
path exists between the opposite sides of each rectangle (in an appropriately
alternating way between vertical and horizontal ones), then we can extract an
infinite path from them. If the rectangles are getting “thicker and thicker”,
then in a Bernoulli $p$ percolation ($p\in(1/2,1)$) this happens with positive
probability due to the lower bound on the crossing probability (Lemma 6.1
below).

In the slab ${\mathbb{Z}}^{2}\times\\{0,1\\}$ we can get infinitely many
robust clusters from ${\tt RF}$ in the following way. We embed ${\tt RF}$ into
the top layer of the slab, then we cut its rectangles into slices and fold
those slices (using the extra space in the slab) in a way so that the folded
slices only intersect specific slices of the corresponding next rectangle in
the road. The result of these operation is denoted as $\Phi_{\beta}({\tt RF})$
(where $\Phi$ refers to “folding” and $\beta$ will be a collection of maps
describing the overlaps we want to maintain). By keeping a balance between not
cutting the rectangles into too thin slices but still making the number of
slices unbounded we can produce infinitely many robust clusters. Figure 1.2
illustrates schematically this folding.

Figure 1.2: Each of the three vertical slices named $AV,BV,CV$ is meant to
intersect only the horizontal slice ($AH,BH,CH$) with which its first letter
agrees. In the extra space we have in the slab, we can fold each vertical
slice in a way that the folded slices keep the intersections we wanted but all
other intersections are avoided.

##  2 A visual guide to ${\tt RF}$

We provide a quick visual introduction on how ${\tt RF}$ is built in [HäM09]
(most of what we cover here can be found in the section titled “Building
blocks” there).

A grid is a collection of congruent vertical or horizontal infinite strips of
the same width which are organized periodically so that the complement of
their union is a collection of congruent squares. The width $l$ of the strips
and the side length $d$ of the squares determine the grid up to translation
and we call such a grid an $(l,d)$-grid. There are finitely many distinct
translates of an $(l,d)$-grid and selecting one of them uniformly is a
translation invariant process. Figure 2.1 shows part of a $(2,3)$-grid. In
this picture we represented the grid using little circles for those vertices
of ${\mathbb{Z}}^{2}$ which are in the grid and little squares for those
vertices which are not. In the rest of the pictures we do not display
individual vertices of ${\mathbb{Z}}^{2}$, instead we use solid rectangles
with different shades for clearer visibility.

Figure 2.1: A piece of a $(2,3)$-grid.

We can sequentially select grids with some fine-tuned parameters (we will see
more details about those parameters in Section 6) so that consecutive grids
fit into the previously selected collection in a way that determines a nested
sequence of windows. The left part of Figure 2.2 offers a pictorial
representation of what we mean by that.

Figure 2.2: The left picture shows a nested sequence of windows. The right one
illustrates the fork (top left) and extended fork (bottom-left) associated to
a given window in a given sequence of nested windows. The vertical and
horizontal frames included in the extended forks of all the windows will
constitute the sets ${\tt Rect}_{\tt V}$ and ${\tt Rect}_{\tt H}$.

The windows are built from vertical and horizontal frames. The fork of a
window is built using the all but the leftmost vertical frames and the
horizontal frame at the very bottom of the window. In the top-left part of the
right part of Figure 2.2, we highlighted the fork of a window. ${\tt RF}$ is
defined by declaring all vertices/sites which belong to the fork of some
window to be open. This is done for each window. Instead of selecting the
fork, we could have selected the extended fork displayed at the bottom left,
where the difference is that the bottom horizontal frame is extended so that
it overlaps with a vertical frame from the next level window (so the extended
fork is not determined by the individual window alone, but we need to know the
next window as well). Even though for an individual window the fork and the
extended fork are different, if we take the increasing union of them for all
windows, we got the same ${\tt RF}$ process. Our reason for nevertheless
introducing the extended fork (not present in [HäM09]) will become clear in
the next section.

We can see a snapshot of ${\tt RF}$ (the shaded region of the left picture
including the black rectangles) in Figure 1.1 where the windows are no longer
shown. It may already suggest that the rectangles are arranged in a way which
resembles a one-ended tree.

If $G$ is a graph and $X$ is a subset of its vertices, then $G[X]$ denotes the
subgraph induced by $X$. The fact that ${\tt RF}$ is a site process means that
through the above process we select a subset $\bigcup_{R\in{\tt Rect}}R$ of
vertices, and ${\tt RF}$ is the subgraph induced by it; i.e., ${\tt
RF}:={\mathbb{Z}}^{2}[\bigcup_{R\in{\tt Rect}}R]$. In order to build the
construction in this paper, we need to change this in a way we describe next.

##  3 A modification of ${\tt RF}$ to define it through a graph union

Assume $G$ and $H$ are simple graphs. Then let $G\cup H$ be that simple graph
whose vertex set is $V(G\cup H):=V(G)\cup V(H)$ and whose edge set is $E(G\cup
H):=E(G)\cup E(H)$ (without multiplicities in the union, of course). Sometimes
we will refer to this as the “graph union” of graphs.

By a block we mean a finite consecutive subset of integers. If $Q=A\times B$
where $A$ and $B$ are blocks, then we say that $Q$ is a rectangle, and we will
use the notation ${\tt h}(Q)=A,{\tt v}(Q)=B$ to express the blocks in terms of
the rectangle, where ${\tt h}$ and ${\tt v}$ refer to horizontal and vertical.

If, for rectangles $Q$ and $R$, it holds that ${\tt h}(Q)\subset{\tt h}(R)$
and ${\tt v}(Q)\supset{\tt v}(R)$, and both containments are proper, then we
say that the pair $(Q,R)$ is well-joined in a vertical to horizontal way. If,
on the other hand, we have ${\tt v}(Q)\subset{\tt v}(R)$ and ${\tt
h}(Q)\supset{\tt h}(R)$, and both containments are proper, then we say that
the pair $(Q,R)$ is well-joined in a horizontal to vertical way.

The following is a quick illustration of why we want to avoid using a site
percolation. Consider the rectangles
$V_{1}=\\{(0,0),(0,1)\\},V_{2}=\\{(1,0),(1,1)\\},H_{1}=\\{(0,0),(1,0)\\}$.

If $G$ is the graph union of their induced subgraphs, that is
$G={\mathbb{Z}}^{2}(V_{1})\cup{\mathbb{Z}}^{2}(V_{2})\cup{\mathbb{Z}}^{2}(H_{1})$,
then any path in $G$ which connects pairs of vertices from $V_{1}$ and $V_{2}$
must go through $H_{1}$ as well. However, if we take those rectangles just as
vertex sets and declare their vertices to be open (as in a site percolation),
then we got ${\mathbb{Z}}^{2}[V_{1}\cup V_{2}\cup H_{1}]$, in which there is
an edge between $(0,1)\in V_{1}$ and $(1,1)\in V_{2}$, so a path exists
between them which does not go through $H_{1}$. In our construction later,
this kind of situation would be undesirable, as we want infinitely many
distinct clusters, and for that purpose it is not good if vertex disjoint
rectangles are automatically connected if they are next to each other.

For this reason, the random fork subgraph in the present paper will be defined
as ${\tt RF}:=\bigcup_{R\in{\tt Rect}}{\mathbb{Z}}^{2}[R]$ (instead of the
original ${\mathbb{Z}}^{2}[\bigcup_{R\in{\tt Rect}}R]$).

##  4 One-ended trees

Now we will give an abstract description of how we associate a one-ended tree
to ${\tt RF}$. A tree ${\cal T}$ is one-ended if for any vertex $v$ of ${\cal
T}$ there is a unique infinite simple path starting from $v$. In general, if
we have a forest ${\cal F}$ with the property that every component of it is a
one-ended tree, and $v\in V({\cal F})$ is a vertex of it, then there is a
unique one–way infinite path in ${\cal F}$ which starts in $v$; we will denote
by ${\tt next}_{\cal F}(v)$ the vertex right after $v$ in this path. Unless
there is some ambiguity which forest we are talking about, we will omit the
subscript, so we will just use the notation ${\tt next}(v)$ (note that this
definition allows ${\cal F}$ to be a one-ended tree itself). The rectangles in
${\tt Rect}$ will be the ones included in the extended forks of the nested
window sequence which we have not yet described formally but illustrated them
in the right part of Figure 2.2.

${\tt RF}$ can be described as a graph union of the rectangles contained in a
certain collection ${\tt Rect}={\tt Rect}_{\tt V}\sqcup{\tt Rect}_{\tt H}$
(the operation is disjoint union) of rectangles, where the rectangles in ${\tt
Rect}_{\tt V}$ (respectively in ${\tt Rect}_{\tt H}$) will be called vertical
(respectively horizontal). The way ${\tt Rect}={\tt Rect}_{\tt V}\sqcup{\tt
Rect}_{\tt H}$ is built makes sure that: Any two distinct rectangles
$S_{1},S_{2}\in{\tt Rect}_{\tt V}$ (respectively $\in{\tt Rect}_{\tt H}$) are
vertex disjoint. If $S_{V}\in{\tt Rect}_{\tt V}$ and $S_{H}\in{\tt Rect}_{\tt
H}$, they are either vertex disjoint or the pair $(S_{V},S_{H})$ is well-
joined in a vertical to horizontal way or the pair $(S_{H},S_{V})$ is well-
joined in a horizontal to vertical way.

Let ${\mathbb{T}}_{\tt RF}$ be the graph whose vertex set is
$V({\mathbb{T}}_{\tt RF})={\tt Rect}$, and there is an edge between
$S_{1},S_{2}\in{\tt Rect}$ exactly if $S_{1},S_{2}$ are not vertex-disjoint.

Due to the structure of the nested windows whose frames are used in the
definition of ${\tt Rect}={\tt Rect}_{\tt V}\sqcup{\tt Rect}_{\tt H}$, the
rectangles in ${\tt Rect}$ are indexed by the natural numbers
$0,1,\dots,n,\dots$; that is, there is an indexing map ${\tt i}:{\tt
Rect}\rightarrow{\mathbb{N}}$. Moreover, if $V\in{\tt Rect}_{\tt V}$, then
${\tt i}({\tt next}(V))={\tt i}(V)$, while if $H\in{\tt Rect}_{\tt H}$, then
${\tt i}({\tt next}(H))={\tt i}(H)+1$. This index ${\tt i}$ naturally belongs
to a full window, and we will derive from it another index ${\tt j}:{\tt
Rect}\rightarrow{\mathbb{N}}$ by defining ${\tt j}(R):=2{\tt i}(R)$, if
$R\in{\tt Rect}_{\tt V}$, and ${\tt j}(R):=2{\tt i}(R)+1$, if $R\in{\tt
Rect}_{\tt V}$.

The following is also true (but would not be if we used site percolation): if
$v_{1},v_{2}$ are vertices of ${\tt RF}$ and $v_{1}$ is a vertex of
$S_{1}\in{\tt Rect}$ and $v_{2}$ is a vertex of $S_{2}\in{\tt Rect}$, then any
path between $v_{1}$ and $v_{2}$ in ${\tt RF}$ must go through in all the
rectangles which are contained in the unique shortest path between $S_{1}$ and
$S_{2}$ in ${{\mathbb{T}}}_{\tt RF}$.

Using the tree structure, we explain at an abstract level how we make
infinitely many clusters from ${\tt RF}$.

Let ${\mathbb{T}}$ be any one-ended tree and let a function
$m:V({\mathbb{T}})\rightarrow{\mathbb{N}}^{+}$ be given with the property that
$m(v)\leq m({\tt next}(v))$. Associate to every $v\in V({\mathbb{T}})$ a set
$B_{v}$ (a “bag” at $v$) with $|B_{v}|=m(v)$, and for $v\not=u$ let $B_{v}\cap
B_{u}=\emptyset$. Fix for each $v$ an injection $\beta_{v}:B_{v}\rightarrow
B_{{\tt next}(v)}$. Since the bags are disjoint, we can write $\beta(s)$
instead of $\beta_{v}(s)$ without ambiguity. Let the forest ${\cal
F}_{\beta}({\mathbb{T}})$ be defined as follows: let the vertex set of ${\cal
F}_{\beta}({\mathbb{T}})$ be the disjoint union $\bigcup_{v\in
V({\mathbb{T}})}B_{v}$, and let us connect $s\in B_{v}$ with $\beta(s)\in
B_{{\tt next}(v)}$ with and edge (no other edges are added). It is clear that
${\cal F}_{\beta}({\mathbb{T}})$ is a forest and all of its components are
one-ended trees (and ${\tt next}_{{\cal
F}_{\beta}({\mathbb{T}})}(u)=\beta(u)$); moreover, if $s_{1},s_{2}\in
B_{v},s_{1}\not=s_{2}$, then by the injectivity of the $\beta$ maps, $s_{1}$
and $s_{2}$ are in different components of ${\cal F}_{\beta}({\mathbb{T}})$.
In particular, $m(v)$ for any $v\in V({\mathbb{T}})$ is a lower bound on the
number of clusters in ${\cal F}_{\beta}({\mathbb{T}})$, and if $m(v)$ is
unbounded, then ${\cal F}_{\beta}({\mathbb{T}})$ has infinitely many clusters.

On the other hand, if $s_{1}\in B_{v_{1}},s_{2}\in
B_{v_{2}},v_{1}\not=v_{2},{\tt next}(v_{1})={\tt next}(v_{2})$, then it may
happen that $\beta(s_{1})=\beta(s_{2})$, since in this case the two $\beta$’s
are actually different, the first is $\beta_{v_{1}}$, the second is
$\beta_{v_{2}}$ so their individual injectivity does not prevent sending
different inputs to the same output.

Note also that the collection of injections $\beta_{v}:B_{v}\rightarrow
B_{{\tt next}(v)}$ for $v\in{\mathbb{T}}$ also determines the collection of
bags $B_{v}$ and their sizes $|B_{v}|=m(v)$, so the notation ${\cal
F}_{\beta}({\mathbb{T}})$ does refer to all ingredients in the above
construction.

##  5 Cutting and folding

In [HäM09] we constructed ${\tt RF}$ as a translation invariant process (which
only requires invariance with respect to translations). We can transform it
into a invariant one on ${\mathbb{Z}}^{2}$ by randomly selecting which are the
vertical and horizontal directions, and also into which direction the
coordinates are increasing, and when we move to the slab we can also randomly
select which layer is the top and which is the bottom (in the later discussion
we will assume that these choices are made).

To realize the abstract idea of turning ${\mathbb{T}}$ into ${\cal
F}_{\beta}({\mathbb{T}})$ in the case when ${\mathbb{T}}={\mathbb{T}}_{\tt
RF}$ we associate a bag $B_{R}$ to a rectangle $R\in V({\mathbb{T}}_{\tt RF})$
in the following way. We pick a sequence $m_{0}\leq m_{1}\leq\dots$ and we cut
$R\in{\tt Rect}$ into $m_{{\tt j}(R)}$ vertical (if $R\in{\tt Rect}_{\tt V}$)
or horizontal (if $R\in{\tt Rect}_{\tt H}$) slices and the union of those
slices will be the bag $B_{R}$. Then we define a collection of injections
$\beta_{R}:B_{R}\rightarrow B_{{\tt next}(R)}$ by uniformly selecting one from
the finitely many possibilities, giving ${\cal F}_{\beta}({\mathbb{T}}_{\tt
RF})$. In this way $B_{R}$ and $B_{Q}$ will be disjoint for $Q\not=R$; but
this only means that if $S\in B_{R}$, then $S\not\in B_{Q}$, but if $Q={\tt
next}(R)$, then $S$ intersects all slices from $B_{Q}$. So while the
associated abstract forest ${\cal F}_{\beta}({\mathbb{T}}_{\tt RF})$ indeed
consists of infinitely many different clusters if $m_{i}\rightarrow\infty$, if
we form the graph union ${\cal G}(C):=\bigcup_{S\in V(C)}{\mathbb{Z}}^{2}[S]$
for a cluster $C$ of ${\cal F}_{\beta}({\mathbb{T}}_{\tt RF})$, then for any
two clusters $C_{1},C_{2}$ of ${\cal F}_{\beta}({\mathbb{T}}_{\tt RF})$ the
vertex and edge set of the graphs ${\cal G}(C_{1})$ and ${\cal G}(C_{2})$ will
intersect.

Here comes the moment when embedding everything into
${\mathbb{Z}}^{2}\times\\{0,1\\}$ helps, since we can “fold” any slice $S\in
B_{R}$ into a “folded slice” $\phi(S)$ in such a way that only those folded
slices have nontrivial intersection which are meant to by the $\beta$
injections. Figure 5.1 is meant to capture this idea of using some extra space
in order to avoid unwanted intersections. Note that this picture is only
conveying the rough idea and is not meant to be a representation of the
specific folding we will use.

Figure 5.1: Within the slab we can fold the slices (as if they were ribbons)
so that only the ones of the same shade intersect.

If $B$ is a block, $m\geq|B|$ is a positive integer, then a balanced $m$-cut
of $B$ is a (possibly random) collection of $m$ blocks ${\tt
cut}(m,B):=\\{B_{1},\dots,B_{m}\\}$ which are pairwise disjoint whose union is
$B$, and for any two $B_{i},B_{j}\in{\tt cut}(m,B)$ we have
$|B_{i}|-1\leq|B_{j}|\leq|B_{i}|+1$.

This is not uniquely defined, but for our purpose any such partition will
work; for example we can take the one which uniformly selects one from the
finitely many possibilities.

By the horizontal $m$-cut of a rectangle $R=B_{1}\times B_{2}$ we mean the
collection of rectangles (we will call them the slices of $R$):

${\tt cut}_{H}(m,R):=\\{{\tt h}(R)\times C:C\in{\tt cut}(m,{\tt v}(R))\\}.$

Similarly, by the vertical $m$-cut we mean the collection of rectangles
(slices of $R$):

${\tt cut}_{V}(m,R):=\\{C\times{\tt v}(R):C\in{\tt cut}(m,{\tt h}(R))\\}.$

We will only cut a vertical rectangle into vertical slices and a horizontal
rectangle into horizontal slices. If it is clear from the context that $R$ is
a vertical rectangle, then we may just write ${\tt cut}(m,R)$ for ${\tt
cut}_{\tt V}(m,R)$ (similar remarks apply for horizontal rectangles); we may
also omit the subscript if we want to talk about a generic rectangle which can
either be vertical or horizontal (but the cut should always be the same type
as the rectangle).

Let $m_{0}\leq m_{1}\leq\dots$ be a sequence of positive integers which has
the property that $m_{i}$ is less than the shorter side of the vertical and
horizontal rectangles for the rectangles of index $i$.

For $R\in{\tt Rect}$ let the corresponding bag be $B_{R}:={\tt cut}(m_{{\tt
j}(R)},R)$, then by fixing the appropriate $\beta$ injections from $B_{R}$ to
$B_{{\tt next}(R)}$ we have the forest ${\cal F}_{\beta}({\mathbb{T}}_{\tt
RF})$.

Let us embed ${\tt RF}$ into ${\mathbb{Z}}^{2}\times\\{0,1\\}$ by placing it
into the top layer ${\mathbb{Z}}^{2}\times\\{1\\}$. So if
$X\subset{\mathbb{Z}}^{2}$, then if we use $X$ in the description of the
folding operation, then we really mean
$\\{(x,1)\in{\mathbb{Z}}^{2}\times\\{0,1\\}:x\in X\\}$. The folding operation
will associate to any $S\in B_{R}$ a new connected subgraph $\phi(S)$ of
${\mathbb{Z}}^{2}\times\\{0,1\\}$ in such a way that for any two slices
$S_{1}\in B_{R},S_{2}\in B_{Q}$, for $R,Q\in{\tt Rect}$, the only way
$V(\phi(S_{1}))\cap V(\phi(S_{2}))\not=\emptyset$ if $Q={\tt next}(R)$ and
$S_{2}=\beta(S_{1})$, or the other way around ($QR={\tt next}(Q)$ and
$S_{1}=\beta(S_{2})$). Moreover, if this holds, then $V(\phi(S_{1}))\cap
V(\phi(S_{2}))=S_{1}\cap S_{2}$. This implies that if $C_{1},C_{2}$ are two
different clusters of the abstract forest ${\cal F}_{\beta}({\mathbb{T}}_{\tt
RF})$, then:

i, the graph union $\bigcup_{S\in V(C_{1})}\phi(S)$ will be a connected graph
(since the nonempty intersection of $S$ and $\beta(S)$ (which are neighbors in
$C_{1}$ if $S\in V(C_{1})$) is preserved by
$\phi(S)\cap\phi(\beta(S))=S\cap\beta(S)$.

ii, the graph unions $\bigcup_{S\in V(C_{1})}\phi(S)$ and $\bigcup_{S\in
V(C_{2})}\phi(S)$ are disjoint (and by the previous item they are connected
subgraphs of ${\mathbb{Z}}^{2}\times\\{0,1\\}$).

Note that this already implies that we have infinitely many clusters if
$m_{i}$ goes to infinity.

The formalization of the folding operation may hide its extreme simplicity so
we start with an intuitive description. Given $S\in B_{R}$ let us place the
rectangles $S$ and ${\tt next}(R)$ on the top layer of
${\mathbb{Z}}^{2}\times\\{0,1\\}$ and let us also mark the horizontal slice
$\beta(S)$ as distinguished from the rest of ${\tt next}(R)$. We want to keep
as much of these at the top level as we can, but we want to avoid the overlap
of $S$ and that part of ${\tt next}(R)$ which does not belong to $\beta(S)$.
Moreover, the slices in ${\tt next}(R)$ have a “higher rank” than the slices
in $R$ in the sense that they will stay in the top layer and the slices in $R$
will be folded in order to avoid the overlap (the slices of ${\tt next}(R)$
will be folded also in order to avoid the unwanted overlaps with the slices of
${\tt next}^{2}(R)$ and in this way the folding operations executed at
different slices will not interfere with each other). Imagine that ${\tt
next}(R)$ is a lake but $\beta(S)$ is an island in it, and we want to build a
road where $S$ lays (and in particular we want it to go through the island),
but we must avoid the water and we cannot build bridges; however, we can build
a subway under the lake. This subway would correspond to using the extra space
in ${\mathbb{Z}}^{2}\times\\{0,1\\}$, so that we avoid the unwanted
intersection by using the bottom layer.

Figure 5.2 is meant to depict what is happening.

Figure 5.2: A visual representation of the folding operation applied to a
slice $S\in B_{R}$. The proportion of the pieces are highly distorted: for
example, $(S\setminus{\tt next}(R),1)$ is much longer in reality than any
other piece involved.

Before moving on, we want to point out some pathological situations which we
want to avoid. If $Q\in{\tt Rect}$ and the bag $B_{{\tt next}(Q)}$ of the next
rectangle ${\tt next}(Q)$ consisted of slices whose shorter side is $1$ or
$2$, then in the first case the whole concept of “folding” seems to break down
while in the second case the relatively simple definition we give below would
break down (the below defined sets $\partial^{t}$ and $\partial_{b}$ would
force to put too many edges into the induced subgraph). So we assume that the
shortest side of any slice is at least $3$, this condition can be safely
assumed (the next section provide explicit details about how the parameters of
the process govern the size of the rectangles in ${\tt Rect}$).

Let $S\in B_{Q}$, let $R=S\cap{\tt next}(Q)$. Let $I_{\phi}$ be the following
indicator function $I_{\phi}:S\rightarrow\\{0,1\\}$: for $v\in S$,
$I_{\phi}(v)=1$ if $v\in(S\setminus R)\cup\beta(S)$ and $I_{\phi}(v)=0$ if
$v\in R\setminus\beta(S)$. Let $\partial^{t}\subset S$ be defined by: for
$v\in S$ we have $v\in\partial^{t}$ iff $I_{\phi}(v)=1$ but $v$ has a neighbor
$u\in S$, for which $I_{\phi}(v)=0$. $\partial^{t}$ is meant to stand for “top
boundary” (since that part of $S$ is meant to be kept at the top layer). Let
$\partial_{b}\subset S$ be defined by: for $v\in S$ we have $v\in\partial_{b}$
iff $I_{\phi}(v)=0$ but $v$ has a neighbor $u\in S$, for which
$I_{\phi}(v)=1$. $\partial_{b}$ is meant to stand for “bottom boundary” (since
that part of $S$ is meant to be kept at the bottom layer).

Recall that if $G=(V,E)$ is a simple graph and $X\subset V(G)$, then by $G[X]$
we mean the subgraph induced by $X$. Since using this notation for
$G={\mathbb{Z}}^{2}\times\\{0,1\\}$ is quite cumbersome, we will temporarily
denote ${\mathbb{Z}}^{2}\times\\{0,1\\}$ by ${\cal
S}_{2}:={\mathbb{Z}}^{2}\times\\{0,1\\}$ here. Let us emphasize that “union”
below means graph union. Let

$\phi(S):={\cal S}_{2}[\\{(v,I_{\phi}(v)):v\in S\\}]\cup{\cal
S}_{2}[\\{(v,i):v\in\partial^{t},i\in\\{0,1\\}\\}]\cup{\cal
S}_{2}[\\{(v,0):v\in\partial^{t}\cup\partial_{b}\\}].$

Now we are ready to define our construction. In order to see the full process,
we go through the steps we did so far as well.

We start with ${\tt RF}$ which depends on some sequence of parameters
$(l_{0},d_{0}),L_{1},L_{2}\dots$ (they are positive integers and we say more
about them in the next section). We add a sequence $m_{0}\leq m_{1}\leq\dots$,
with $m_{i}\rightarrow\infty$ of positive integers as an extra parameter
(which will need to grow slower than the shorter side of the rectangles in
${\tt RF}$). This defines splitting the rectangles of ${\tt RF}$ into slices,
and the slices of a given rectangle can be used as its bag. Choosing the
collection of injections to which we refer by $\beta$, the “abstract” forest
${\cal F}_{\beta}({\mathbb{T}}_{\tt RF})$ becomes well defined. Finally, let

$\Phi_{\beta}({\tt RF}):=\bigcup_{S\in V({\cal F}_{\beta}({\mathbb{T}}_{\tt
RF}))}\phi(S),$

where the union is again meant in the graph union sense. That is, we simply
replace every slice $S$ with its folded version $\phi(S)$, and then the graph
whose vertices are the folded slices and where an edge represents non-empty
intersection will be isomorphic to ${\cal F}_{\beta}({\mathbb{T}}_{\tt RF})$.
What we claim is that, if the parameters of ${\tt RF}$ are chosen in such a
way that it is robust and its robustness can be proven by a crossing argument
described in the next section, then the sequence $m_{0}\leq m_{1}\leq\dots$
can be chosen to grow slowly enough, so that $\Phi_{\beta}({\tt RF})$ has
infinitely many components and they are all robust.

##  6 Crossing probabilities

First we say more about the parameters in ${\tt RF}$ (in [HäM09] this is
mainly covered in the Section titled “The actual construction”).

To define ${\tt RF}$ we need an initial pair $(l_{0},d_{0})$ of positive
integers and a sequence of positive integers $L_{1},L_{2},\dots$. The first
$(l_{0},d_{0})$ parameter tells us (up to a random translation) what is the
initial grid we start with is, then the $L_{i}$ sequence tells us the number
of vertical (or, what is the same, horizontal) frames in the windows which
will be carved out from the previous grid by the new one. This determines the
parameter $(l_{i},d_{i})$ of the $i$th grid, as illustrated in the left part
of Figure 6.1. We can also express the sizes of the vertical and horizontal
rectangles of an extended fork with these parameters (see the right part of
Figure 6.1). Those sizes need to be known if we want to estimate crossing
probabilities.

Figure 6.1: From the left figure we can read off that $l_{i+1}=d_{i}$ and
$d_{i+1}=L_{i+1}l_{i}+(L_{i+1}-1)d_{i}$. For the sum $l_{i}+d_{i}$ this gives
the simple recursion $l_{i+1}+d_{i+1}=L_{i+1}(l_{i}+d_{i})$. In the right
picture we can see that the vertical rectangle in an extended fork is an
$l_{i}\times(d_{i+1}-l_{i})$ one while the horizontal rectangle is a
$(d_{i+1}+d_{i}-l_{i})\times l_{i}$ one.

With the right choice of the above parameters, the ${\tt RF}$ process will be
well-defined and robust. To be well-defined it is important that any vertex is
covered by finitely many grids only. By a Borel-Cantelli argument, the
condition $\sum_{i=0}^{\infty}{1\over{L_{i}}}<\infty$ will be sufficient for
this.

The robustness is taken care of by a crossing argument. When preforming i.i.d.
bond percolation with retention probability $p$ – indicated by writing
$\mathbf{P}_{p}$ for the probability measure – on a rectangle $R$, the event
${\tt cross}_{\tt H}(R)$ is defined as the set of those subgraphs of the
induced subgraph ${\mathbb{Z}}^{2}(R)$ that contains a path between the two
different vertical sides called a horizontal crossing in $R$. The event ${\tt
cross}_{\tt V}(R)$, which we define by interchanging the words vertical and
horizontal above is called a vertical crossing  in $R$.

If $R$ is any rectangle from ${\tt Rect}$, then let ${\tt ray}(R)$ be the
unique infinite path in ${\mathbb{T}}_{\tt RF}$ starting from $R$ (as a vertex
of ${\mathbb{T}}_{\tt RF}$), the vertices in ${\tt ray}(R)$ form a sequence of
rectangles (in an alternating way from ${\tt Rect}_{\tt V}$ and ${\tt
Rect}_{\tt H}$) in which every consecutive pair is either well joined in a
vertical to horizontal or in a horizontal to vertical way. Let ${\tt
road}(R):=\bigcup_{Q\in{\tt ray}(R)}{\mathbb{Z}}^{2}[Q]$ be the graph union of
the subgraphs induced by the rectangles in ${\tt ray}(R)$.

We show that ${\tt RF}$ is robust by showing that already ${\tt road}(R)$ is
robust for any $R\in{\tt Rect}$.

If we know a lower bound for $\mathbf{P}_{p}({\tt cross_{\tt V}}(Q))$ for
$Q\in V({\tt ray}(R))\cap{\tt Rect}_{\tt V}$ and $\mathbf{P}_{p}({\tt
cross_{\tt H}}(Q))$ for $Q\in V({\tt ray}(R))\cap{\tt Rect}_{\tt H}$, then by
the well-known Harris–FKG inequality which states that for i.i.d. percolation
any two increasing events are positively correlated (see, e.g., [G99, Thm.
2.4]) we can get a lower bound for the probability of an infinite path simply
by multiplication, since if the appropriate (horizontal ones for horizontal
rectangles, vertical ones for vertical rectangles) crossing events hold
simultaneously, then from the individual paths within the rectangles we can
extract the existence of a single infinite path (the situation is illustrated
on the right of Figure 1.1). Such a lower bound is provided by the following
lemma (stated as Corollary 3.4 in [HäM09], and derived from an estimate by
Bollobás-Riordan in [BR06]):

######  Lemma 6.1.

For any fixed $p\in(\frac{1}{2},1)$, we can find a $c>0$, a positive integer
$\nu_{0}$ and a $\gamma>0$ such that for any positive integer $L$ the
following holds. If $R$ is an $Ln\times n$ rectangle where $n>\nu_{0}$, then
$\mathbf{P}_{p}({\tt cross}_{\tt H}(R))\geq c^{L/{n^{\gamma}}}$.

Of course a similar statement holds for $n\times Ln$ rectangles regarding the
vertical crossings.

In [HäM09] (section 3.6) the robustness of ${\tt road}(R)$ is shown by the
above strategy (we assume that ${\tt j}(R)=0$, so that the indices below
matches with $m_{i}$). That is, the parameters
$(l_{0},d_{0}),L_{1},L_{2},\dots$ can be chosen in such a way that if the
$j$th rectangle in ${\tt road}(R)$ is a $\Lambda_{j}n_{j}\times n_{j}$ one (if
it is horizontal) or a $n_{j}\times\Lambda_{j}n_{j}$ one (if it is vertical),
then $\sum_{j=0}^{\infty}{\Lambda_{j}/{n_{j}^{\gamma}}}<\infty$, which makes
the product $\prod_{j=0}^{\infty}c^{\Lambda_{j}/{n_{j}^{\gamma}}}$ to be
positive. Since $\prod_{j=0}^{\infty}c^{\Lambda_{j}/{n_{j}^{\gamma}}}$ is a
lower bound for the existence of an infinite path within ${\tt road}(R)$, by
Harris-FKG in a Bernoulli $p$ percolation, ${\tt road}(R)$ is robust.

If we accept that those calculations are correct in the case of ${\tt RF}$,
then, when we move from ${\tt RF}$ to ${\Phi}_{\beta}({\tt RF})$, we will not
need to know what $\Lambda_{i}$ is because we can use a soft argument: if
$\sum_{i=0}^{\infty}a_{i}<\infty$, then there exists a sequence of positive
integers $k_{1}\leq k_{2}\leq\dots$, with $k_{i}\rightarrow\infty$ such that
$\sum_{i=0}^{\infty}a_{i}k_{i}<\infty$ still holds. One construction is to
select indices $j_{1}<j_{2}<\dots$ in such a way that
$\sum_{j_{i}}^{\infty}\leq{1\over{2^{i}}}$, then let $k_{i}:=s$ if $j_{s}\leq
i<j_{s+1}$. Then
$\sum_{i=0}^{\infty}a_{i}k_{i}<\sum_{i=0}^{j_{0}}a_{i}+\sum_{i=0}^{\infty}{i\over{2^{i}}}<\infty$.

The reason this will complete our job is that ${\Phi}_{\beta}({\tt RF})$ still
consists of folded slices which are isomorphic to planar rectangles, and the
way these folded slices overlap is also isomorphic to the way how two well-
joined planar rectangle overlap. Thus a crossing argument will also be
applicable to the folded slices in ${\Phi}_{\beta}({\tt RF})$ as well. When we
construct ${\Phi}_{\beta}({\tt RF})$ from ${\tt RF}$, then we have the
sequence $m_{0}\leq m_{1}\leq\dots$, with $m_{i}\rightarrow\infty$ as a
parameter which we have great freedom to choose. When we cut a $\Lambda
n\times n$ rectangle into $m$ horizontal slices, then a slice will
approximately be a $\Lambda n\times n/m$ rectangle (if $m$ does not divide $n$
it is only an approximation), and assuming that $n/m>\nu_{0}$ still holds, the
lower bound from Lemma 6.1 for the horizontal crossing probability is

$c^{\Lambda/{({n\over{m}})^{\gamma}}}=c^{{m^{\gamma}\Lambda}\over{n^{\gamma}}}.$

When we move from ${\tt RF}$ to $\Phi_{\beta}({\tt RF})$, then we do not only
cut a $\Lambda n\times n$ rectangle into $m$ slices (which makes slices of
$\Lambda n\times n/m$ rectangles) but we also fold those slices, and a folded
slice is then isomorphic to a $\Lambda n+h\times n/m$ rectangle, where $h$ is
some integer at most $3$ (corresponding to the need to moving up and down
between the top and bottom layer of ${\mathbb{Z}}^{2}\times\\{0,1\\}$, when we
fold the $\Lambda n\times n/m$ slice). However, if we ignore this additive
term for a moment, we basically get that it is enough to choose the sequence
$m_{0}\leq m_{1}\leq\dots$, with $m_{i}\rightarrow\infty$ in such a way that

$\sum_{j=0}^{\infty}{{m_{j}^{\gamma}{\Lambda_{j}}}\over{n_{j}^{\gamma}}}<\infty,$

since this will imply by a crossing argument that the clusters of
$\Phi_{\beta}({\tt RF})$ are robust.

Since $\sum_{j=0}^{\infty}{\Lambda_{j}/{n_{j}^{\gamma}}}<\infty$, we can find
a sequence of positive integers $k_{1}\leq k_{2}\leq\dots$, with
$k_{i}\rightarrow\infty$, such that
$\sum_{j=0}^{\infty}k_{j}{\Lambda_{j}/{n_{j}^{\gamma}}}<\infty$, so if we
choose $m_{j}\rightarrow\infty$ in such a way that $m_{j}\leq
k_{j}^{1/{\gamma}}$, then we are done, if we make sure that
$n_{j}/m_{j}>\nu_{0}$, that is if $m_{j}<n_{j}/\nu_{0}$ (which is needed to
apply Lemma 6.1). To ensure that $m_{j}<n_{j}/\nu_{0}$ holds, note that if the
sequence $n_{j}$ is nondecreasing and $n_{j}\rightarrow\infty$, then with the
above $k_{j}^{1/\gamma}$ sequence it will be true that ${\tt
min}(n_{j},k_{j}^{1/\gamma})$ is nondrecreasing and ${\tt
min}(n_{j},k_{j}^{1/\gamma})\rightarrow\infty$ still holds, so we can choose
$m_{j}\leq{\tt min}(n_{j},k_{j}^{1/\gamma})$ in way that $m_{0}\leq
m_{1}\leq\dots$ and $m_{i}\rightarrow\infty$ still holds. Finally, let us note
that is indeed possible to choose the parameters
$(l_{0},d_{0}),L_{1},L_{2},\dots,$ in such a way that $n_{j}$ (which is the
shorter side of the rectangles used) is strictly increasing.

Note also that so far we just used the approximate length of the sides of the
slices as if they were $\Lambda n\times n/m$, when in reality both sides can
be slightly different (partially by the extra length obtained from the folding
and also from the possibility that $m$ does not divide $n$). Note that the
crossing probabilities could only decrease if the longer side of the rectangle
is lengthened and the shorter side is shortened, and then we can be sure that
the longer side of the folded rectangle still will be less than $2\Lambda n$
and the shorter side cannot be shorter than $n/2m$, thus if $n/2m>\nu_{0}$
still holds, then we can use Lemma 6.1 to estimate the crossing probability of
a $2\Lambda n\times n/2m$ rectangle (and use it as a lower bound for the
crossing probability of the folded slice). In this way the sum whose
convergence will ensure the robustness of our clusters will only increase by a
constant multiplier compared to the estimates we used before (and the
condition $n/2m>\nu_{0}$ is also easy to satisfy). This completes the proof of
Theorem 1.1.

##  7 The case of ${\mathbb{Z}}^{2}$

In this section we prove Theorem 1.2. We will use some basic tools from the
theory of invariant processes in groups in general (for more background see
the research paper [BLPS99] or the the books [LP16], [P23]). The number of
ends in a graph $G$ is the supremum of the number of infinite components of
$G\setminus K$ over all finite subsets $K$ of $G$ (for an explanation what is
an end in general see section 7.3 in [LP16]). If the number of ends in a graph
is $k$, we will call it “$k$-ended” (for trees it is consistent with the
notion of one-ended trees we used before). The following is a well-known fact
(stated as an exercise in [P23], following the landmark Burton-Keane
argument):

In an invariant percolation on an amenable group (such as ${\mathbb{Z}}^{2}$)
there cannot be a cluster with more than two ends.

We will also use the Mass-Transport Principle for the case of countable groups
(see for example [LP16]). We only need the following consequence: an invariant
process on a countable group cannot produce any infinite cluster in which a
non-empty finite set of vertices are distinguished. This is true, because in
every infinite cluster $C$ where a non-empty finite set of vertices is
distinguished we could pick a single vertex $v_{C}$ uniformly and every vertex
from $C$ could send mass $1$ to $v_{C}$ in which case the expected mass sent
out is at most $1$ but the expected mass received is infinite.

Let the dual lattice of ${\mathbb{Z}}^{2}$ be the graph (isomorphic to
${\mathbb{Z}}^{2}$) whose vertices are of the form $a+(1/2,1/2)$ for
$a\in{\mathbb{Z}}^{2}$, and edges are $\\{a+(1/2,1/2),b+(1/,1/2\\}$ for
$\\{a,b\\}\in{\mathbb{Z}}^{2}$. There is a natural bijection between the edges
of the two graphs, namely the ones which are crossing each other can be put
into pairs (that is $\\{a,a+1\\}$ for $a\in{\mathbb{Z}}$ is the pair of
$\\{a+1/2,a-1/2\\}$ while $\\{a,a+1\\}$ is the pair of $\\{a-1/2,a+1/2\\}$).
We will call configurations $\omega\in\\{0,1\\}^{E({\mathbb{Z}}^{2})}$ in the
“original” ${\mathbb{Z}}^{2}$ primal and we associate to such an $\omega$ a
natural dual $\omega^{*}$: let an edge $f$ in the dual graph be in
$\omega^{*}$ iff its pair $e$ is not in $\omega$.

If $C_{1}$ and $C_{2}$ are two disjoint infinite components of
${\mathbb{Z}}^{2}$, then we can find an at least two ended dual component
between them (thus when it arises from an invariant process, it is exactly
two-ended). This is because if $P$ is a shortest path from $C_{1}$ to $C_{2}$,
and we start an exploration of the boundary of $C_{1}$ from the first edge of
$P$ into both direction, then this exploration finds an infinite path in the
dual on either side of $P$, and if we delete the dual edges corresponding to
the edges of $P$, then these two infinite paths end up in different components
in the dual, thus the dual region between $C_{1}$ and $C_{2}$ must be at least
two-ended.

Let us say that a dual cluster $C_{1}$ touches a primal cluster $C_{2}$ if
$C_{1}$ has a vetex $v_{1}$ for which
$v_{1}=v_{2}+(\sigma_{1}/2,\sigma_{2}/2)$ for a vertex $v_{2}$ of $C_{2}$,
where $\sigma_{i}\in\\{-1,+1\\}$.

Assume that ${\cal I}$ is an invariant percolation process on
${\mathbb{Z}}^{2}$ which produces at least one robust cluster. We will show
that it cannot have more than two robust clusters. The argument is cleaner if
we assume that ${\cal I}$ has robust clusters and isolated vertices only. So
let us delete the edges from any non-robust cluster. In what follows ${\cal
I}^{*}$ is the dual process (which produces a dual configuration of ${\cal
I}$).

If ${\cal I}$ has at least two infinite clusters, then there must exist a
2-ended cluster of ${\cal I}^{*}$ which separates them. Let ${\cal D}$ be any
2-ended cluster of ${\cal I}^{*}$, this separates the primal
${\mathbb{Z}}^{2}$ into two infinite pieces (and possibly into finite ones).
Let us call two dual clusters neighbors if there is an infinite primal cluster
that they both touch and consider the infinite dual clusters neighboring
${\cal D}$. By inspecting the number of ends in the primal configuration (see
Figure 7.1) there can only be at most one such neighbor on either side of
${\cal D}$ (otherwise the primal configuration must contain components with at
least three ends).

Figure 7.1: The picture shows the following steps: if there are at least two
infinite primal cluster, then there must exists at least one 2-ended dual
clusters ${\cal D}$ which separates the primal ${\mathbb{Z}}^{2}$ into two
infinite pieces. If they were more than one infinite dual clusters neighboring
${\cal D}$ on the same side of ${\cal D}$, then there would exists a primal
cluster with more than two ends contradicting the amenability of
${\mathbb{Z}}^{2}$.

Call them ${\cal N}_{1}({\cal D})$ and ${\cal N}_{2}({\cal D})$, if they
exist. We will see that, using the robustness of the clusters of ${\cal I}$,
not even a single such neighbor can exist. Assume that at least one of them
exists, let ${\cal N}_{\cal D}$ be one of them selected uniformly at random,
then the primal component ${\cal C}_{\cal D}$ between ${\cal D}$ and ${\cal
N}_{\cal D}$ must be two ended.

Let the minimal distance between ${\cal D}$ and ${\cal N}$ be $m_{\cal D}$.
Let $M({\cal D})$ be the set of those vertices of ${\cal D}$ for which there
exists a vertex $y$ of $N$ such that ${\tt dist}(x,y)=m_{\cal D}$. Because the
association of $M({\cal D})$ to two-ended dual components ${\cal D}$ is an
invariant selection of a non-empty (if at least one of ${\cal N}_{i}({\cal
D})$ exists) set of vertices, $M({\cal D})$ must be infinite.

This implies that the primal cluster ${\cal C}_{\cal D}$ between ${\cal D}$
and ${\cal N}_{\cal D}$ cannot be robust, contradicting the assumption on
${\cal I}$. This is because if $P_{\tt min}({\cal D},{\cal N}_{\cal D})$ is
the collection of the infinitely many shortest dual paths between ${\cal D}$
and ${\cal N}_{\cal D}$ and ${\tt inv}(P_{\tt min}({\cal D},{\cal N}_{\cal
D}))$ is any invariantly selected non-empty subset of $P_{\tt min}({\cal
D},{\cal N}_{\cal D})$, then deleting all the edges of ${\cal C}$
corresponding to the edges of the paths in ${\tt inv}(P_{\tt min}({\cal
D},{\cal N}_{\cal D}))$ cuts ${\cal C}$ into finite components and a Bernoulli
$p$ percolation with $p<1$ on ${\cal C}$ will indeed delete such collection of
edges from ${\cal C}$.

##  8 Acknowledgements

This work was partially supported by the ERC Consolidator Grant 772466
“NOISE”. I am indebted to Gábor Pete for useful discussions throughout, and to
Russell Lyons to ask the question answered by Theorem 1.2.

## References

  * [1]
  * [BLPS99] I. Benjamini, R. Lyons, Y. Peres and O. Schramm. Group-invariant Percolation on Graphs Geometric and Functional Analysis 9 (1999), 29–66.
  * [BR06] B. Bollobás, and O. Riordan. A short proof of the Harris–Kesten theorem, The Bulletin of the London Mathematical Society 38 (2006), 470–484.
  * [DST15] H. Duminil-Copin, V. Sidoravicius, and V. Tassion. Absence of infinite cluster for critical Bernoulli percolation on slabs. Communications in Pure and Applied Mathematics 69 (2016), 1397-1411.
  * [GKR88] A. Gandolfi, M. Keane, and L. Russo. On the uniqueness of the infinite open cluster in dependent two-dimensional site percolation, Annals of Probability 16 (1988), 1147–1157.
  * [G99] G.R. Grimmett, Percolation, 2nd ed., Springer, (1999), New York.
  * [HäM09] O. Häggström and P. Mester. Some two-dimensional finite energy processes. Electronic Communications in Probability 14 (2009), 42-54.
  * [LP16] R. Lyons and Y. Peres. Probability on trees and networks. Cambridge University Press, (2016). Available at http://mypage.iu.edu/~rdlyons
  * [NTW17] C. Newman, V. Tassion, and W. Wu. Critical Percolation and the Minimal Spanning Tree in Slabs. Communications in Pure and Applied Mathematics 70(11) (2017), 2084-2120.
  * [P23] G. Pete. Probability and Geometry on Groups. Book in preparation, (2023). Available at http://www.math.bme.hu/~gabor/PGG.pdf
  * [PT22] G. Pete, and Á. Timár. The Free Uniform Spanning Forest is disconnected in some virtually free groups, depending on the generator set, Annals of Probability 50(6), (2022), 2218-2243.
  * [Sh05] S. Sheffield. (2005) Random surfaces, Asterisque 304, vi+175 pp.


# Empirical study of the modulus as activation function in computer vision
applications

Iván Vallés-Pérez  Emilio Soria-Olivas  Marcelino Martínez-
Sober111Corresponding author. Antonio J. Serrano-López  Joan Vila-Francés
Juan Gómez-Sanchís  IDAL, Intelligent Data Analysis Laboratory, University of
Valencia, Avenida de la Universitat s/n 46100 Burjassot, Valencia, Spain.

###### Abstract

In this work we propose a new non-monotonic activation function: the modulus.
The majority of the reported research on nonlinearities is focused on
monotonic functions. We empirically demonstrate how by using the modulus
activation function on computer vision tasks the models generalize better than
with other nonlinearities - up to a 15% accuracy increase in CIFAR100 and 4%
in CIFAR10, relative to the best of the benchmark activations tested. With the
proposed activation function the vanishing gradient and dying neurons problems
disappear, because the derivative of the activation function is always 1 or
-1. The simplicity of the proposed function and its derivative make this
solution specially suitable for TinyML and hardware applications.

###### keywords:

Deep Learning , Activation Functions , Optimization

††journal: Elsevier

##  1 Introduction

The core piece of all deep learning models is the activation function. They
enable the models to produce non-linear abstract representations when applying
linear transformations in cascade (Goodfellow et al., 2016). The most common
choice in modern deep learning models is the Rectified Linear Unit (ReLU)
(Nair and Hinton, 2010). Among other advantages, ReLU nonlinearities allowed
us to train deeper neural networks (Xu et al., 2015).

Apart from the ReLU in the last years there have appeared many alternative
activation functions (Dubey et al., 2022). The following studies represent
some of the most popular examples: Leaky-ReLU and PR-ReLU (Xu et al., 2015),
ELU (Clevert et al., 2016), Swish (Ramachandran et al., 2018), SELU (Klambauer
et al., 2017), F/PFLU (Zhu et al., 2021), RSigElu (Kiliçarslan and Celik,
2021) etc. The authors of (Agostinelli et al., 2014) studied how to learn
adaptive piecewise linear activation functions. All the mentioned studies
propose functions that have the following properties in common: nonlinearity,
continuity, differentiability and low computational cost (being Swish the most
computationally expensive option). The majority of the activation functions
are monotonic, Swish, GELU (Hendrycks and Gimpel, 2016), S-ReLU (Jin et al.,
2016) and Mish (Misra, 2019) are some of the most popular exceptions. Despite
the large pool of alternatives, ReLU still appears as the default choice in
many applications due to its simplicity and low computational cost (Nair and
Hinton, 2010).

In this work, we investigate the use of the modulus function, also known as
the absolute value function, as an activation function for deep learning
models. Previous research (Karnewar, 2018) has hinted at the potential of the
modulus nonlinearity in the design of a specific neural network architecture,
but its performance has not been systematically compared to other
nonlinearities. Through empirical evaluation, we provide empirical evidence
proving that the modulus activation function allows deep learning models to
converge to better solutions in CIFAR10, CIFAR100 and MNIST datasets. Compared
with other modern non-monotonic nonlinearities such as Mish, PFLU and FPFLU or
Swish, the modulus has a very low computational cost (equivalent to ReLU).
This property makes the modulus specially useful for TinyML and hardware
applications (Sanchez-Iborra and Skarmeta, 2020) and hardware neural networks
(Misra and Saha, 2010).

The contributions of this paper are: (1) we propose the modulus activation
function, (2) the proposed function empirically shows significantly better
results in 75% of our experiments, (3) we contribute to the new trend of non-
monotonic activation functions with another example showing good results in
the same direction, (4) the simplicity of the modulus activation function
makes it very suitable for hardware applications and TinyML, and (5) we
propose two smooth approximation to the modulus function, one of them
achieving even superior results than the original modulus.

This paper is organized as follows. Section 2 describes the proposed
activation function and the ones used for benchmarking. Section 3 describes
the experiments conducted and the results achieved. Finally, we summarize our
contribution in section 5, enumerating the main conclusions.

##  2 Methods

###  2.1 Modulus activation function

In this section we introduce our proposed activation function, the modulus:
$f(x)=|x|$. This nonlinearity is a continuous, piecewise-linear function
consisting of an identity mapping for positive values of $x$, and a negative
identity mapping for negative values of $x$. Its derivative (defined below),
as well as the modulus function itself, are hardware-level bit-size
operations. This is extremely useful for hardware implementations. Besides,
the modulus function can be expressed as $f(x)=\text{sgn}(x)\cdot x$, where
$f^{\prime}(x)=\text{sgn}(x)$, and hence $f(x)=f^{\prime}(x)\cdot x$. In
practice, this means that the derivative can be calculated as part of the
forward pass and cached for the backpropagation, reducing the total
computation. This form is also specially useful in hardware implementations,
given that the whole activation function and its derivative is fully
represented as a bit-size operation (the sign function). In hardware, the
memory and computing requirements are limited, hence having mechanisms that
allow for cheap computations (1-bit in the modulus activation) and
reusability/caching (part of the gradients computed during the forward pass,
in our case) is crucial.

$f^{\prime}(x)=\mathrm{sgn}(x)\left\\{\begin{array}[]{lcc}1&\text{if}&x>0\\\
-1&\text{if}&x<0\end{array}\right.$ (1)

Notice that the modulus function is differentiable everywhere except in $x=0$.
For practical purposes, we define $f^{\prime}(0)=1$ so that the derivative is
defined for all the range of $x$ values, similar to the case of ReLU
(Goodfellow et al., 2016), and to ensure that the norm of the gradient is
constant for all values of $x$. See figure 1h for a graphical representation.

Figure 1: Nonlinearities used along this study. (a-g) are benchmark activation
functions that showed good performance in previous studies. (h) is the modulus
activation function proposed in this study. For this example, $\beta=10.0$,
$\beta=1.0$ and $\beta=1.0$ have been used for the Leaky-ReLU, ELU and Swish
activation functions, respectively.

The modulus activation function belongs, in essence, to the family of
rectifier functions (Glorot et al., 2011) . In fact, it is equivalent to a
Leaky-ReLU with $\beta=-1$ (see equation 4). However Leaky-ReLU was originally
defined to take values of beta strictly higher than 1 (Xu et al., 2015).
Furthermore, we empirically show that the modulus achieves significantly
superior results than the Leaky-ReLU.

The benefit of this activation function with respect to the other rectifiers
is that the norm of its gradient is constant ($||\nabla_{x}f||=1\quad\forall
x$) and hence, interestingly, it does not depend on the value of $x$. This
property is desirable when optimizing the parameters of a neural network with
gradient descent algorithms, as there are no input values for which the neuron
saturates or explodes (Glorot and Bengio, 2010) (i.e. values of x for which
the gradient of the activation function is close to zero, or extremely large).
This naturally removes the dying neurons (Lu, 2020) and vanishing gradient
problems (Pascanu et al., 2013; Hochreiter, 1998; Hochreiter et al., 2001),
which usually appears in activations with zero regions (such as ReLU) or with
asymptotically saturating regions (such as tanh), respectively.

###  2.2 Smooth approximations of the modulus function

The modulus function is not differentiable when $x=0$. To study if this
property harms the performance of the models in any way, two alternative
smooth approximations of the modulus function have been tested as an
additional experiment. See figure 2 for a visual representation. The two
approximations are defined below.

  * 1.

Quadratic approximation: referred subsequently as SoftModulusQ and defined in
equation 2 (full derivation in the appendix).

$f(x)=\left\\{\begin{array}[]{lcc}x^{2}\cdot(2-|x|)&\text{if}&|x|\leq 1\\\
|x|&\text{if}&|x|>1\end{array}\right.$ (2)

  * 2.

Hyperbolic tangent approximation: referred subsequently as SoftModulusT and
defined in equation 3. Starting from the modulus function $f(x)=sgn(x)\cdot
x$, we approximate the $sgn$ function as follows
$sgn(x)\approx\tanh(x/\beta)$, where $\beta\in[0,1]$ is a tunable
hyperparameter that controls how acute the “V” transition is. The smaller the
$\beta$, the closest the approximation is to the modulus. We used a value of
$\beta=0.01$ in all the models we trained.

$f(x)=x\cdot\tanh(x/\beta)$ (3)

Figure 2: Representation of the two SoftModulus activation functions compared
with the original modulus: SoftModulusQ and SoftModulusT.

###  2.3 Benchmark activation functions

The following activation functions have been used as a benchmark to compare
the performance of the proposed one.

  * 1.

Tanh: the hyperbolic tangent has been one of the most popular choices
(together with the sigmoid), before ReLU was proposed (LeCun et al., 2012).
Among many other desirable properties, its derivative is very simple: $(\tanh
x)^{\prime}=1-\tanh^{2}x$. This property was very beneficial specially before
automatic differentiation tools appeared. Besides, similar to our proposed
activation function, the derivative can be easily calculated during forward
pass and cached for the backward pass.

  * 2.

ReLU: this activation function was published in 2010 as an alternative to
train Restricted Boltzman Machines. It is defined as $f(x)=\max(0,x)$. It
allowed training deeper neural networks by solving the vanishing gradient
problems typically happening with saturating activation functions. As it can
be seen in the formula, ReLU outputs zero if the input is negative. This
brings sparsity to the non-linear representation, at the cost of potentially
finding optimization problems due to the fact that the derivative when $x<0$
is zero (dying neurons, for instance).

  * 3.

Leaky-ReLU: the Leaky-ReLU attempted to solve the problem known as dying
neurons in the ReLUs by adding a small linear term in the negative side of $x$
(see equation 4, where $\beta>1$ is a hyperparameter to be tuned).

$f(x)=\max(x/\beta,x)$ (4)

  * 4.

ELU: this is a smooth version of ReLU that approaches asymptotically to -1 as
$x\rightarrow-\infty$. Unlike ReLUs, it is differentiable everywhere and can
produce negative values. This activation function is formally defined in
equation 5, where $\beta$ is a tunable hyperparameter.

$f(x)=\left\\{\begin{array}[]{lcc}x&\text{if}&x>0\\\
\beta(e^{x}-1)&\text{if}&x\leq 0\end{array}\right.$ (5)

  * 5.

Swish: this is one of the few non-monotonic activation functions formally
published. It gained its popularity because it showed promising results in
multiple applications. It is defined as $f(x)=x\cdot\text{sigmoid}(\beta x)$,
where $\beta$ is a hyperparameter to be tuned.

  * 6.

Mish: this is a smooth, non-monotonic and self-regularized activation
function, similar to swish, showing superior performance in different
benchmarks. It is defined as $f(x)=x\cdot\tanh(\text{softplus}(x))$, where
$\text{softplus}(x)=\log(1+e^{x})$ (Dugas et al., 2001).

  * 7.

PFLU: the Power Function Linear Unit (PFLU) is a non-monotonic activation
function published on 2020 that showed good performance in convolutional
architectures. It is defined as follows:
$f(x)=x\cdot\frac{1}{2}\left(1+\frac{x}{\sqrt{1+x^{2}}}\right)$.

##  3 Experiments and results

###  3.1 Setup

Table 1: Deep learning architectures used to experiment with different activations. In the dense layers row, the output size has been represented as $C$. The Fully connected architecture (FC) is a multilayer perceptron with 2 hidden layers + the output layer. The Conv2 and Conv6 architectures are shallow variants of VGG described here: (Simonyan and Zisserman, 2015). The last pooling layer of the VGG-16 original architecture has been trimmed in order to allow this model to work with smaller image sizes. Network | Fully Connected | Conv2 | Conv6 | VGG-16
---|---|---|---|---
Conv layers |  |  64,64,pool |  64,64,pool 128,128,pool 256,256,pool |  64,64,pool 128,128,pool 256,256,256,pool 512,512,512,pool 512,512,512
Dense Layers |  256,256,$C$ |  256,256,$C$ |  256,256,$C$ |  4096,4096,$C$
Filter sizes |  | 3x3 | 3x3 | 3x3
Pooling type |  | max | max | max
# Parameters | 269k-878k | 3.3M-4.3M | 1.8M-2.3M | 33.6M-40.3M
Size | 3.1-11MB | 38-50MB | 21-27MB | 385-462MB

For the following experiments we have used CIFAR10, CIFAR100 (Krizhevsky,
2009) and MNIST (LeCun and Cortes, 2010) datasets. CIFAR10 and CIFAR100 images
are labeled into 10 and 100 classes, respectively. They are of size 32x32 and
full color. In both cases, the data sets contain 50,000 images for training
and 10,000 images for testing purposes. MNIST images belong to one of 10
classes and are in grey scale and 28x28 size. MNIST comes with 50,000 images
for training and 10,000 images for testing purposes.

The images of the datasets have been normalized so that the minimum and the
maximum values are -1 and 1. We have kept untouched the default train/test
split provided by Pytorch (Paszke et al., 2019), in order to facilitate future
potential reproducibility and benchmarking efforts.

Four different architectures have been used to test the performance of the
modulus activation function against the other nonlinearities. These
architectures consist of a multilayer perceptron (named fully connected), two
convolutional architectures with 2 and 6 convolutional layers (named conv2 and
conv6 respectively) and the VGG-16 network Simonyan and Zisserman (2015). They
are described in more detail in Table 1. The architectures choice has been
inspired on the Lottery Ticket Hypothesis paper (Frankle and Carbin, 2019),
however no pruning methods have been applied in this study.

Table 2: Test accuracy for all the datasets and activation functions tested (rows), and for all the models (columns). The results are expressed as mean $\pm$ standard deviation across the 30 random initializations. To facilitate the reading of the table, for each dataset-model combination, we have colored the results as follows: the best result (highest accuracy) has been colored in blue, the second in green, the third in yellow and the fourth in red (first $>$ second $>$ third $>$ fourth). Additionally, we marked in bold those cases where any of the proposed activation functions achieved significantly higher accuracy than the benchmarks, with a significance level of $\alpha=0.05$, for which we used a Wilcoxon one-sided Rank Sum test. Dataset | Activation | FC | Conv2 | Conv6 | VGG16
---|---|---|---|---|---
CIFAR10 | ReLU | $54.65\pm 0.22$ | $71.40\pm 0.26$ | $77.09\pm 1.21$ | $\cellcolor{Fourth}83.66\pm 0.41$
| LeakyReLU | $54.71\pm 0.24$ | $\cellcolor{Third}71.65\pm 0.32$ | $77.38\pm 1.13$ | $\cellcolor{Third}83.98\pm 0.34$
| Tanh | $49.95\pm 0.23$ | $67.46\pm 0.34$ | $77.67\pm 0.24$ | $79.69\pm 0.26$
| Swish | $\cellcolor{Third}55.39\pm 0.29$ | $69.09\pm 0.27$ | $71.66\pm 0.62$ | $80.77\pm 0.50$
| ELU | $\cellcolor{Fourth}55.16\pm 0.21$ | $69.34\pm 0.34$ | $78.98\pm 0.33$ | $81.21\pm 0.37$
| PFLU | $\cellcolor{Second}55.43\pm 0.30$ | $70.34\pm 0.33$ | $\cellcolor{Fourth}80.77\pm 0.40$ | $81.58\pm 0.38$
| Mish | $\cellcolor{First}55.44\pm 0.22$ | $69.66\pm 0.35$ | $75.66\pm 0.53$ | $80.98\pm 0.64$
| Modulus | $53.97\pm 0.24$ | $\mathbf{\cellcolor{Second}73.93\pm 0.42}$ | $\mathbf{\cellcolor{Second}84.22\pm 0.29}$ | $\mathbf{\cellcolor{Second}84.86\pm 0.32}$
| SoftModulusQ | $54.07\pm 0.29$ | $\cellcolor{Fourth}71.49\pm 0.37$ | $\mathbf{\cellcolor{Third}81.01\pm 1.27}$ | $10.00\pm 0.00$
| SoftModulusT | $54.04\pm 0.24$ | $\mathbf{\cellcolor{First}73.95\pm 0.40}$ | $\mathbf{\cellcolor{First}84.36\pm 0.28}$ | $\mathbf{\cellcolor{First}85.34\pm 0.36}$
CIFAR100 | ReLU | $27.33\pm 0.25$ | $36.72\pm 0.31$ | $36.35\pm 0.89$ | $44.61\pm 1.11$
| LeakyReLU | $27.24\pm 0.22$ | $37.03\pm 0.40$ | $37.15\pm 0.77$ | $45.19\pm 1.47$
| Tanh | $23.63\pm 0.20$ | $35.29\pm 0.47$ | $42.15\pm 0.49$ | $44.14\pm 0.37$
| Swish | $\cellcolor{Fourth}27.59\pm 0.25$ | $35.20\pm 0.34$ | $35.75\pm 0.41$ | $46.02\pm 1.10$
| ELU | $\cellcolor{First}27.92\pm 0.26$ | $35.68\pm 0.31$ | $40.74\pm 0.48$ | $\cellcolor{Fourth}47.63\pm 0.71$
| PFLU | $\cellcolor{Second}27.73\pm 0.21$ | $\cellcolor{Third}37.51\pm 0.42$ | $\cellcolor{Fourth}42.25\pm 0.45$ | $\cellcolor{Third}48.22\pm 0.63$
| Mish | $\cellcolor{Third}27.68\pm 0.25$ | $36.04\pm 0.41$ | $37.63\pm 0.75$ | $\cellcolor{First}48.69\pm 0.69$
| Modulus | $26.29\pm 0.26$ | $\mathbf{\cellcolor{Second}38.66\pm 0.56}$ | $\mathbf{\cellcolor{First}48.73\pm 0.62}$ | $45.83\pm 0.80$
| SoftModulusQ | $26.23\pm 0.25$ | $\cellcolor{Fourth}37.48\pm 0.44$ | $\mathbf{\cellcolor{Third}48.16\pm 1.97}$ | $1.00\pm 0.00$
| SoftModulusT | $26.32\pm 0.24$ | $\mathbf{\cellcolor{First}38.69\pm 0.56}$ | $\mathbf{\cellcolor{Second}48.63\pm 0.83}$ | $\cellcolor{Second}48.47\pm 0.68$
MNIST | ReLU | $98.35\pm 0.07$ | $\cellcolor{Fourth}99.27\pm 0.04$ | $99.53\pm 0.03$ | $\cellcolor{Fourth}99.58\pm 0.04$
| LeakyReLU | $98.37\pm 0.06$ | $99.27\pm 0.04$ | $99.53\pm 0.03$ | $\cellcolor{Third}99.58\pm 0.03$
| Tanh | $98.34\pm 0.07$ | $99.06\pm 0.05$ | $99.48\pm 0.04$ | $99.48\pm 0.04$
| Swish | $98.36\pm 0.05$ | $99.24\pm 0.04$ | $99.52\pm 0.03$ | $99.53\pm 0.03$
| ELU | $98.31\pm 0.04$ | $99.16\pm 0.04$ | $99.54\pm 0.03$ | $99.54\pm 0.03$
| PFLU | $\cellcolor{Fourth}98.42\pm 0.05$ | $99.21\pm 0.04$ | $\cellcolor{Fourth}99.56\pm 0.03$ | $99.57\pm 0.03$
| Mish | $98.41\pm 0.05$ | $99.23\pm 0.04$ | $99.56\pm 0.03$ | $99.57\pm 0.04$
| Modulus | $\mathbf{\cellcolor{Third}98.47\pm 0.07}$ | $\mathbf{\cellcolor{Second}99.38\pm 0.04}$ | $\mathbf{\cellcolor{Third}99.60\pm 0.03}$ | $\mathbf{\cellcolor{First}99.63\pm 0.04}$
| SoftModulusQ | $\mathbf{\cellcolor{First}98.51\pm 0.06}$ | $\mathbf{\cellcolor{Third}99.37\pm 0.03}$ | $\mathbf{\cellcolor{First}99.62\pm 0.03}$ | $11.35\pm 0.00$
| SoftModulusT | $\mathbf{\cellcolor{Second}98.47\pm 0.06}$ | $\mathbf{\cellcolor{First}99.39\pm 0.04}$ | $\mathbf{\cellcolor{Second}99.61\pm 0.03}$ | $\mathbf{\cellcolor{Second}99.62\pm 0.03}$
Figure 3: Learning curves of all the trained models. Each line represents the
average test accuracy of 30 runs of a model. The shade behind to each line
shows the 95% confidence interval around the mean. Notice that the scales in
the y-axis are not shared to allow for better zoom in each figure, comparing
the performance across models is not the objective of this study. Details like
the shades are better viewed in a screen.

We have followed the same experimental setup across models, activations and
datasets. All the networks have been trained for 100 epochs, and the best
accuracy of each run has been reported. Each run has been repeated 30 times
with different random weight initializations. No dropout (Srivastava et al.,
2014) nor batch normalization (Ioffe and Szegedy, 2015) have been used. We
used Adam (Kingma and Ba, 2014) as optimizer, with a learning rate of
$10^{-4}$, a gradual warmup (Gotmare et al., 2019) during the first 5 epochs
starting from $10^{-5}$, and a cosine annealing (Loshchilov and Hutter, 2017)
with a target learning rate of $10^{-6}$ in the last epoch. We used Python
3.7.3, Pytorch 1.7.1 and TorchVision 0.8.2, and all the models have been
trained in a single Nvidia 2080ti graphics card. The code used can be found in
the url of the footnote222https://github.com/ivallesp/abs.

###  3.2 Classification performance

Table 2 summarizes the results of the classifiers for CIFAR10, CIFAR100 and
MNIST datasets. As we can see from the tables, the modulus activation function
outperforms significantly the benchmark activations in 9 out of 12 experiments
(4 architectures x 3 datasets). In 4 of these cases, the accuracy improvement
is $3\%$ or higher, in relative terms (CIFAR10-Conv2, CIFAR10-Conv6,
CIFAR100-Conv2 and CIFAR100-Conv6 with p-value $<10^{-6}$ in all the cases).
If we consider the soft approximations of the modulus activation function, we
see that the SoftModulusT significantly outperforms the benchmark in 10 out of
12 experiments (with p-value of $6.55\cdot 10^{-6}$ for MNIST-Conv6,
$1.29\cdot 10^{-2}$ for MNIST-VGG16, $3.99\cdot 10^{-4}$ for CIFAR100-VGG16
and $<10^{-6}$ for the rest). All the p-values reported in this study have
been obtained using a Wilcoxon one-sided Rank Sum test comparing the modulus
results against the best of the benchmark activation functions in each case.

Figure 3 shows the test accuracy at every epoch for the 12 experiments (the
curves of the smooth approximations have been included in figure 4 to compare
them with the modulus). As it can be noticed in several cases (e.g.
CIFAR10-Conv2, CIFAR10-Conv6, CIFAR100-Conv6), the accuracy of the networks
with modulus activation function keeps increasing at the 100th epoch while the
benchmarks have already stabilized. That observation suggests that, if trained
for longer, probably the accuracy would increase even more.

The results of the SoftModulus approximations compared with the modulus
activation function are summarized in table 3. Additionally, figure 4 shows
the test accuracy at every epoch for the same activations. From these results,
we see that although the SoftModulusQ approximation achieves significantly
better results than the original modulus in several cases (FC model over
CIFAR10 and MNIST dataset, and Conv6 over MNIST dataset), it seems to be much
more unstable: on VGG16, the gradients of the models with SoftModulusQ
activations vanish at the beginning of the training process due to the fact
that the weights are initialized very close to zero and the gradients are too
small, causing numerical problems. This problem may be solved tweaking the
initialiation. However, the SoftModulusT approximation seems to be on-par with
the original modulus in the majority of cases, while it perform significantly
better for deep architectures like VGG16. We hypothesize that the difference
between the two approximations is due to the width of the zero gradient region
around $x=0$: wider regions lead to more training problems (see figure 2). The
$\beta$ parameter in the hyperbolic tangent approximation allows for easily
adjust this zero-gradient region. We informally tested different values of
beta concluding that for high values of $\beta$ (e.g. $\beta=1.0$) the model
struggles to train due to numerical precision problems (small values of
weights lead to tiny gradients). A value of $\beta=0.01$ seems to work well
for all the tested experiments. Finally, we see that the SoftModulusT achieves
superior results than the original modulus when used in the VGG16
architecture.

Table 3: Accuracy comparison for the soft approximations of the modulus function, compared with the original modulus definition. The results are expressed as mean $\pm$ standard deviation across the 30 random initializations. In each column we highlighted the model with higher accuracy in bold. We added a star to those cases where a SoftModulus activation function achieved significantly higher results than the modulus, with a significance level of $\alpha=0.05$. Dataset | Activation | FC | Conv2 | Conv6 | VGG16
---|---|---|---|---|---
CIFAR10 | Modulus | $53.97\pm 0.24$ | $73.93\pm 0.42$ | $84.22\pm 0.29$ | $84.86\pm 0.32$
| SoftModulusQ | $\mathbf{54.07\pm 0.29}*$ | $71.49\pm 0.37$ | $81.01\pm 1.27$ | $10.00\pm 0.00$
| SoftModulusT | $54.04\pm 0.24$ | $\mathbf{73.95\pm 0.40}$ | $\mathbf{84.36\pm 0.28}$ | $\mathbf{85.34\pm 0.36}*$
CIFAR100 | Modulus | $26.29\pm 0.26$ | $38.66\pm 0.56$ | $\mathbf{48.73\pm 0.62}$ | $45.83\pm 0.80$
| SoftModulusQ | $26.23\pm 0.25$ | $37.48\pm 0.44$ | $48.16\pm 1.97$ | $1.00\pm 0.00$
| SoftModulusT | $\mathbf{26.32\pm 0.24}$ | $\mathbf{38.69\pm 0.56}$ | $48.63\pm 0.83$ | $\mathbf{48.47\pm 0.68}*$
MNIST | Modulus | $98.47\pm 0.07$ | $99.38\pm 0.04$ | $99.60\pm 0.03$ | $\mathbf{99.63\pm 0.04}$
| SoftModulusQ | $\mathbf{98.51\pm 0.06}*$ | $99.37\pm 0.03$ | $\mathbf{99.62\pm 0.03}*$ | $11.35\pm 0.00$
| SoftModulusT | $98.47\pm 0.06$ | $\mathbf{99.39\pm 0.04}$ | $99.61\pm 0.03$ | $99.62\pm 0.03$

##  4 Discussion

The modulus activation function has shown promising results in certain
contexts, but it is worth discussing potential criticisms of this approach.

One concern may be that the modulus activation does not seem to be
biologically inspired by the way in which axon hillocks fire in biological
neurons. Despite this, the primary goal of an activation function is to
introduce non-linearity into the neural networks, and the modulus activation
is effective in achieving this. Additionally, the modulus produces non-zero
gradients on the negative side, which can be beneficial to avoid vanishing
gradients and dying neurons.

Despite the intuition that non-monotonic activations can lead to neurons which
parameters are more difficult to optimize, there are several examples in the
literature (as mentioned in the introduction) that show that when using those
nonlinearities the models tend to achieve better results. One potential
explanation for this is that non-monotonic activations can introduce more
complex nonlinearities into the network, allowing it to model complex
relationships in the data more easily, potentially improving the performance
of the resulting model. Another possible reason is that non-monotonic
activation functions may have improved gradient properties, making
optimization easier. For instance, certain non (strictly) monotonic activation
functions, such as ReLU and the modulus, have very simple gradients.

Finally, another criticism of the modulus function may be its potential to
introduce symmetry in neural networks, which may result in oscillation or
instability. While in our experiments we did not observe evidences of this
behavior, it may be a contributing factor to the slower convergence of
networks with the modulus function (see figure 3). Another potential
explanation for this observation is the lack of initialization methods
specially designed for the modulus activation. It is worth noting that
symmetry in neural networks can lead to duplication of ways to approximate the
desired function, which translates to multiple equally valid solutions in the
loss landscape. This may be advantageous, as it results in more regions of the
loss landscape containing feasible solutions. Other potential advantage is
that symmetry may improve the generalization ability of the network, as it
allows to learn patterns that are invariant under certain transformations
(e.g. input sign flip in dense layers without bias terms
$f_{\theta}(X)=f_{\theta}(-X)$). The symmetry property introduced by the
modulus can have more advantages, disadvantages or applications. Further
research could explore the potential side effects of symmetry in more depth,
as well as its impact on the performance and stability of the network in other
tasks and applications.

##  5 Conclusions

We have shown how the new modulus activation function outperforms the
benchmark activation functions in 75% of our experiments. The improvement
achieved by the networks with this activation function is often significantly
higher ($3\%$ or higher in one third of the experiments we conducted).
Additionally, we have proposed a smooth version of the modulus activation
function which performs better than the first one, at a slightly higher
computational cost.

As a next step, it would be beneficial to evaluate the proposed modulus
activation function on a wider range of problems beyond image classification.
This would provide insight into the generalizability and potential utility of
the modulus function in a variety of tasks. Further investigation in this
direction could help to identify the specific areas in which the modulus
function is especially effective, and highlight new applications for this non-
monotonic activation function.

Figure 4: Learning curves for smooth approximations compared with the modulus.
Each line represents the average test accuracy of 30 runs of a model. The
shade behind to each line shows the 95% confidence interval around the mean.
Notice that the scales in the y-scales are not shared to allow for better zoom
in each figure, comparing the performance across models is not the objective
of this study. Details like the shades are better viewed in a screen.

##  6 Acknowledgements

This work has been partially funded by the Spanish ministry of science and
innovation MCIN/AEI projects PID2019-107347RR-C33 and PID2021-127946OB-I00.

## References

  * Agostinelli et al. (2014) Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P., 2014\.  Learning activation functions to improve deep neural networks, in: 3rd International Conference on Learning Representations, 2015.
  * Clevert et al. (2016) Clevert, D., Unterthiner, T., Hochreiter, S., 2016.  Fast and accurate deep network learning by exponential linear units (ELUs), in: Bengio, Y., LeCun, Y. (Eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.  URL: http://arxiv.org/abs/1511.07289.
  * Dubey et al. (2022) Dubey, S.R., Singh, S.K., Chaudhuri, B.B., 2022.  Activation functions in deep learning: A comprehensive survey and benchmark.  Neurocomputing 503, 92–108.  URL: https://www.sciencedirect.com/science/article/pii/S0925231222008426, doi:https://doi.org/10.1016/j.neucom.2022.06.111.
  * Dugas et al. (2001) Dugas, C., Bengio, Y., Bélisle, F., Nadeau, C., Garcia, R., 2001.  Incorporating second-order functional knowledge for better option pricing, in: Leen, T., Dietterich, T., Tresp, V. (Eds.), Advances in Neural Information Processing Systems, MIT Press.  URL: https://proceedings.neurips.cc/paper/2000/file/44968aece94f667e4095002d140b5896-Paper.pdf.
  * Frankle and Carbin (2019) Frankle, J., Carbin, M., 2019\.  The lottery ticket hypothesis: Finding sparse, trainable neural networks., in: 7th International Conference on Learning Representations, ICLR, OpenReview.net.  URL: http://dblp.uni-trier.de/db/conf/iclr/iclr2019.html#FrankleC19.
  * Glorot and Bengio (2010) Glorot, X., Bengio, Y., 2010\.  Understanding the difficulty of training deep feedforward neural networks, in: Teh, Y.W., Titterington, M. (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR, Chia Laguna Resort, Sardinia, Italy. pp. 249–256.  URL: http://proceedings.mlr.press/v9/glorot10a.html.
  * Glorot et al. (2011) Glorot, X., Bordes, A., Bengio, Y., 2011.  Deep sparse rectifier neural networks, in: Gordon, G., Dunson, D., Dudík, M. (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR, Fort Lauderdale, FL, USA. pp. 315–323.  URL: http://proceedings.mlr.press/v15/glorot11a.html.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., Courville, A., 2016.  Deep Learning.  MIT Press.  doi:10.1007/s10710-017-9314-z.
  * Gotmare et al. (2019) Gotmare, A., Shirish Keskar, N., Xiong, C., Socher, R., 2019\.  A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation, in: 7th International Conference on Learning Representations, ICLR.  URL: https://openreview.net/forum?id=r14EOsCqKX.
  * Hendrycks and Gimpel (2016) Hendrycks, D., Gimpel, K., 2016\.  Bridging nonlinearities and stochastic regularizers with gaussian error linear units, in: 4th International Conference on Learning Representations, ICLR.  URL: https://openreview.net/forum?id=Bk0MRI5lg.
  * Hochreiter (1998) Hochreiter, S., 1998.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  Int. J. Uncertain. Fuzziness Knowl.-Based Syst. 6, 107–116.  URL: https://doi.org/10.1142/S0218488598000094, doi:10.1142/S0218488598000094.
  * Hochreiter et al. (2001) Hochreiter, S., Bengio, Y., Frasconi, P., Schmidhuber, J., 2001\.  Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, in: A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press.  URL: https://ml.jku.at/publications/older/ch7.pdf.
  * Ioffe and Szegedy (2015) Ioffe, S., Szegedy, C., 2015\.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: 32nd International Conference on Machine Learning - Volume 37, JMLR.org. pp. 448–456.  URL: http://proceedings.mlr.press/v37/ioffe15.html.
  * Jin et al. (2016) Jin, X., Xu, C., Feng, J., Wei, Y., Xiong, J., Yan, S., 2016.  Deep learning with s-shaped rectified linear activation units, in: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI Press. p. 1737–1743.  URL: https://arxiv.org/abs/1512.07030.
  * Karnewar (2018) Karnewar, A., 2018.  Aann: Absolute artificial neural network, in: 2018 3rd International Conference for Convergence in Technology (I2CT), pp. 1–6.  doi:10.1109/I2CT.2018.8529552.
  * Kiliçarslan and Celik (2021) Kiliçarslan, S., Celik, M., 2021\.  Rsigelu: A nonlinear activation function for deep neural networks.  Expert Systems with Applications 174, 114805.  URL: https://www.sciencedirect.com/science/article/pii/S0957417421002463, doi:https://doi.org/10.1016/j.eswa.2021.114805.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization., in: 3rd International Conference of Learning Representations (ICLR).  URL: https://arxiv.org/abs/1412.6980.
  * Klambauer et al. (2017) Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S., 2017\.  Self-normalizing neural networks., in: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (Eds.), NIPS, pp. 971–980.  URL: http://dblp.uni-trier.de/db/conf/nips/nips2017.html#KlambauerUMH17.
  * Krizhevsky (2009) Krizhevsky, A., 2009.  Learning multiple layers of features from tiny images.  Technical Report. Computer Science, University of Toronto.  URL: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
  * LeCun and Cortes (2010) LeCun, Y., Cortes, C., 2010\.  MNIST handwritten digit database.  URL: http://yann.lecun.com/exdb/mnist/.
  * LeCun et al. (2012) LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.R., 2012\.  Efficient BackProp. Springer Berlin Heidelberg, Berlin, Heidelberg.  pp. 9–48.  doi:10.1007/978-3-642-35289-8_3.
  * Loshchilov and Hutter (2017) Loshchilov, I., Hutter, F., 2017\.  Sgdr: Stochastic gradient descent with warm restarts, in: 5th International Conference of Learning Representations ICLR.  URL: https://ml.informatik.uni-freiburg.de/~staeglis/deploy/papers/17-ICLR-SGDR.pdf.
  * Lu (2020) Lu, L., 2020.  Dying ReLU and initialization: Theory and numerical examples.  Communications in Computational Physics 28, 1671–1706.  doi:10.4208/cicp.oa-2020-0165.
  * Misra (2019) Misra, D., 2019.  Mish: A self regularized non-monotonic neural activation function.  arXiv preprint arXiv:1908.08681 .
  * Misra and Saha (2010) Misra, J., Saha, I., 2010.  Artificial neural networks in hardware: A survey of two decades of progress.  Neurocomputing 74, 239–255.  URL: https://www.sciencedirect.com/science/article/pii/S092523121000216X, doi:https://doi.org/10.1016/j.neucom.2010.03.021. artificial Brains.
  * Nair and Hinton (2010) Nair, V., Hinton, G.E., 2010\.  Rectified linear units improve restricted boltzmann machines, in: Fürnkranz, J., Joachims, T. (Eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814.  URL: https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf.
  * Pascanu et al. (2013) Pascanu, R., Mikolov, T., Bengio, Y., 2013.  On the difficulty of training recurrent neural networks, in: Dasgupta, S., McAllester, D. (Eds.), Proceedings of the 30th International Conference on Machine Learning, PMLR, Atlanta, Georgia, USA. pp. 1310–1318.  URL: http://proceedings.mlr.press/v28/pascanu13.html.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  Pytorch: An imperative style, high-performance deep learning library, in: Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 32. Curran Associates, Inc., pp. 8024–8035.  URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Ramachandran et al. (2018) Ramachandran, P., Zoph, B., Le, Q., 2018.  Searching for activation functions, in: 6th International Conference on Learning Representations, ICLR.  URL: https://openreview.net/forum?id=SkBYYyZRZ.
  * Sanchez-Iborra and Skarmeta (2020) Sanchez-Iborra, R., Skarmeta, A.F., 2020\.  Tinyml-enabled frugal smart objects: Challenges and opportunities.  IEEE Circuits and Systems Magazine 20, 4–18.  doi:10.1109/MCAS.2020.3005467.
  * Simonyan and Zisserman (2015) Simonyan, K., Zisserman, A., 2015\.  Very deep convolutional networks for large-scale image recognition.  URL: https://arxiv.org/abs/1409.1556.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of machine learning research 15, 1929–1958.  URL: https://jmlr.org/papers/v15/srivastava14a.html.
  * Xu et al. (2015) Xu, B., Wang, N., Chen, T., Li, M., 2015.  Empirical evaluation of rectified activations in convolutional network.  CoRR URL: https://arxiv.org/pdf/1505.00853.pdf.
  * Zhu et al. (2021) Zhu, M., Min, W., Wang, Q., Zou, S., Chen, X., 2021\.  Pflu and fpflu: Two novel non-monotonic activation functions in convolutional neural networks.  Neurocomputing 429, 110–117.  URL: https://www.sciencedirect.com/science/article/pii/S0925231220318749, doi:https://doi.org/10.1016/j.neucom.2020.11.068.

## Appendix: SoftModulusQ derivation

We chose a fuzzy set approach to combine the modulus function with a quadratic
function. For that, three membership functions are defined in figure 5 and
equations 6, 7, 8.

Figure 5: Membership functions used to combine the quadratic and modulus
functions.)

$\mu^{x}_{\text{low}}=\left\\{\begin{array}[]{lcc}1&\text{if}&x<-1\\\
-x&\text{if}&-1\leq x<0\\\ 0&\text{if}&x\geq 0\end{array}\right.$ (6)

$\mu^{x}_{\text{med}}=\left\\{\begin{array}[]{lcc}0&\text{if}&x<-1\\\
x+1&\text{if}&-1\leq x<0\\\ 1-x&\text{if}&0\leq x<1\\\ 0&\text{if}&x\geq
1\end{array}\right.$ (7)

$\mu^{x}_{\text{high}}=\left\\{\begin{array}[]{lcc}0&\text{if}&x<0\\\
x&\text{if}&0\leq x<1\\\ 1&\text{if}&x\geq 1\end{array}\right.$ (8)

Then we define $f^{x}_{\text{low}}=-x$, $f^{x}_{\text{med}}=x^{2}$ and
$f^{x}_{\text{high}}=x$. If we combine these functions with the membership
functions we get the following.

  * 1.

For $x<-1$: we get
$\hat{f}(x)=\frac{f^{x}_{\text{low}}\cdot\mu^{x}_{\text{low}}}{\mu^{x}_{\text{low}}}=-x$

  * 2.

For $-1\leq x<0$: we get
$\hat{f}(x)=\frac{f^{x}_{\text{low}}\cdot\mu^{x}_{\text{low}}+f^{x}_{\text{med}}\cdot\mu^{x}_{\text{med}}}{\mu^{x}_{\text{low}}+\mu^{x}_{\text{med}}}=\frac{(x+1)x^{2}+(-x)(-x)}{1+x-x}=x^{3}+2x^{2}$

  * 3.

For $0\leq x<1$: we get
$\hat{f}(x)=\frac{f^{x}_{\text{med}}\cdot\mu^{x}_{\text{med}}+f^{x}_{\text{high}}\cdot\mu^{x}_{\text{high}}}{\mu^{x}_{\text{med}}+\mu^{x}_{\text{high}}}=\frac{(1-x)x^{2}+x\cdot
x}{1-x+x}=-x^{3}+2x^{2}$

  * 4.

For $x\geq 1$: we get
$\hat{f}(x)=\frac{f^{x}_{\text{high}}\cdot\mu^{x}_{\text{high}}}{\mu^{x}_{\text{high}}}=x$

Combining and simplifying all the above expressions we get the final
SoftModulusQ activation function.

$f(x)=\left\\{\begin{array}[]{lcc}x^{2}\cdot(2-|x|)&\text{if}&|x|\leq 1\\\
|x|&\text{if}&|x|>1\end{array}\right.$ (9)