# Interspecific dispersal constraints suppress pattern formation in
metacommunities

Patrick Lawton Biophysics Graduate Program, University of California,
Riverside, CA, USA  Ashkaan K. Fahimipour Department of Biological Sciences,
Florida Atlantic University, Boca Raton, FL, USA  Center for Complex Systems
and Brain Sciences, Florida Atlantic University, Boca Raton, FL, USA  Kurt E.
Anderson Department of Evolution, Ecology, & Organismal Biology, University of
California, Riverside, CA, USA

∗Correspondence to<EMAIL_ADDRESS>
Decisions to disperse from a habitat stand out among organismal behaviors as
pivotal drivers of ecosystem dynamics across scales. Encounters with other
species are an important component of adaptive decision-making in dispersal,
resulting in widespread behaviors like tracking resources or avoiding
consumers in space. Despite this, metacommunity models often treat dispersal
as a function of intraspecific density alone. We show, focusing initially on
three-species network motifs, that interspecific dispersal rules generally
drive a transition in metacommunities from homogeneous steady states to self-
organized heterogeneous spatial patterns. However, when ecologically realistic
constraints reflecting adaptive behaviors are imposed — prey tracking and
predator avoidance — a pronounced homogenizing effect emerges where spatial
pattern formation is suppressed. We demonstrate this effect for each motif by
computing master stability functions that separate the contributions of local
and spatial interactions to pattern formation. We extend this result to
species rich food webs using a random matrix approach, where we find that
eventually webs become large enough to override the homogenizing effect of
adaptive dispersal behaviors, leading once again to predominately pattern
forming dynamics. Our results emphasize the critical role of interspecific
dispersal rules in shaping spatial patterns across landscapes, highlighting
the need to incorporate adaptive behavioral constraints in efforts to link
local species interactions and metacommunity structure.

Keywords: Dispersal $|$ Metacommunity $|$ Network $|$ Pattern formation $|$
Cross-diffusion

## Introduction

Organismal behavior plays a pivotal role in shaping ecosystems across scales.
Dispersal in particular exerts a profound influence, affecting everything from
the availability of local resources [1, 2, 3] and the distribution of
predators [4] to the structure of ecological networks [5, 6], and the dynamics
of metacommunities [7, 8, 9, 10]. However, understanding the effects of
dispersal on metacommunity dynamics has been made difficult by the variation
and complexity of behaviors that ultimately influence whether an organism
leaves a given habitat [11, 8, 9]. Often, these dispersal decisions are
influenced by encounters with other species, including with resources,
competitors, or consumers [12, 13, 14, 15]. Despite the observable
dependencies of dispersal on local diversity and species interactions, many
metacommunity models still treat dispersal as function of intraspecific
density alone [7, 16, 17, 9].

The consideration of dispersal rates that respond to species interactions is
needed as previous studies suggest they may have substantial impacts on
metacommunity dynamics [14, 16, 17, 18, 10, 19, 20]. These dispersal responses
are often represented as “cross-diffusion” terms, where the movement of one
species is influenced by the spatial gradient of another species’ density or
abundance [21, 22, 23]. A recurring finding in recent spatial models
incorporating cross-diffusion is the heightened sensitivity of spatially
homogeneous steady states to spontaneous pattern formation driven by
dispersal, i.e. Turing instabilities [24, 25, 26, 27, 28]. To ensure that
these models capture ecologically relevant phenomena, it is important to
incorporate cross-diffusion terms that reflect the signs of interspecific
interactions, such that prey would avoid predators or consumers would actively
track prey [24]. However, when such constraints on cross-diffusivity have been
implemented in ecological models, they have mostly been applied to two-species
systems [29, 10], limiting the range of potential dynamical behaviors [28].
These simplified models fall short in capturing the dynamics of larger species
interaction networks, leaving critical gaps in our understanding of the link
between interaction-driven dispersal and the self-organized community
structures that emerge at realistic scales [30, 19, 31, 5, 32].

Figure 1: Food web motifs and metacommunity connectivity

Solid arrows denote feeding relationships. Light, medium and dark shading
represents primary producers, intermediate consumers, and top consumers,
respectively. Motifs are labeled as (a) food chain, (b) apparent competition,
(c) resource competition, (d) intraguild predation. N > 3 food webs are not
shown, but may include one or more of these motifs. (e) Species may disperse
from patches (dashed lines) in response to other species’ densities (shaded
areas). For webs (a)-(c), a maximum of $n_{\text{cross}}=4$ such interspecific
dispersal responses are possible. When constrained, dispersal responses are
opposite in sign from feeding relationships such that dispersal is higher for
a species with locally high densities of its predators and low densities of
its prey. (f) Dispersal connects habitats in spatial networks to create
metacommunities.

Here, we assess the susceptibility of model metacommunities to spatial pattern
formation, both with and without ecologically-relevant adaptive constraints
applied to the sign of cross-diffusion terms (i.e., predator avoidance and
prey tracking). Our results consistently indicate that increasing the
prevalence of unconstrained cross-diffusion facilitates spatial pattern
formation. However, the introduction of ecological constraints that regulate
cross-diffusion diminishes or even eliminates the tendency for pattern
formation. We first examine ecologically relevant three-species interaction
motifs (Fig. 1a-d) via a rigorous computational scan of both non-spatial
(trophic) and spatial (dispersal) model parameters. We then extend our
analysis to systems comprising more than three species using a more efficient
random matrix approach, showing that findings in our three species motifs
qualitatively hold when extended to larger metacommunities.

## modeling and analysis framework

We employ a metacommunity modeling framework in order to study the effects of
cross-diffusion on spatial food web dynamics. We assume these dynamics are
governed by a set of deterministic reaction-diffusion equations on a spatial
network describing species interactions and dispersal in separable terms.
Nodes in the spatial network represent habitat patches where species
interactions determine both local food web dynamics and decisions to emigrate
to neighboring patches. The dynamics of species $i$ on a given patch $k$ are
then captured by

$\displaystyle\dot{x}_{i}^{k}=f_{i}(\bm{x}^{k})-\sum_{l}{\text{L}_{kl}D_{i}({\bm{x}^{l}})},$
(1)

where the generally non-linear function of species $i$’s biomass density
$f_{i}(\bm{x}^{k})$ and $D_{i}({\bm{x}^{k}})$ define the rates of trophic
interactions and dispersal, respectively, and are assumed to be identical
across all habitat patches. Connections between patches are represented by the
Laplacian matrix
$\text{L}_{kl}=\delta_{kl}\sum_{l}\text{A}_{kl}-\text{A}_{kl}$, with A the
network’s adjacency matrix. As exact solutions to Eq 1. rarely exist, either
direct simulation or analysis of the linearized system can be employed to
understand the long term dynamics.

We consider the dynamics of Eq. 1 following small perturbations
$\delta\bm{x}^{k}$ from a spatially homogeneous, nontrivial steady state
$\bm{x}^{\ast}$. These perturbations can be decomposed over eigenmodes of
$\bf{L}$, each with a corresponding Laplacian eigenvalue $\kappa$ analogous to
the wavenumber in continuous space [33, 10, 30, 34]. The allowed spatial
signatures of all eigenmodes, as encoded in the respective eigenvectors, is
predetermined by the structure of $\bf{L}$, unlike in continuous space where
perturbations may be decomposed over arbitrarily high wavenumbers. To evaluate
the potential dynamical behavior across arbitrary patch networks, we therefore
refrain from specifying a particular form of $\bf{L}$ and instead consider
$\kappa$ as a real valued, positive parameter of arbitrarily high magnitude.
The exponential growth rate $\lambda$ of the $n$th mode for each species is
then given by eigenvalues of

$\displaystyle\bf{J}=\bf{P}-\kappa\bf{C},$ (2)

where the local Jacobian

$\displaystyle\text{P}_{ij}=\frac{\partial f_{i}(\bm{x}^{k})}{\partial
x_{j}^{k}}\bigg{\rvert}_{\bm{x}^{\ast}}$ (3)

encodes the linearized non-spatial component of the model (i.e., trophic
interactions) while the connectivity matrix

$\displaystyle\text{C}_{ij}=\frac{\partial D_{i}(\bm{x}^{k})}{\partial
x_{j}^{k}}\bigg{\rvert}_{\bm{x}^{\ast}}$ (4)

encodes the linearized spatial response of species $i$ to species $j$ near
$\bm{x}^{\ast}$ [34, 10].

The long term behavior of Eq 1. can be qualitatively understood via the
distribution of all possible eigenmodes’ maximum growth rates

$\displaystyle\lambda(\kappa)={\rm Ev}_{\max}\left({\bf J}\right),$ (5)

where the right hand side denotes the leading eigenvalue of the matrix ${\bf
J}$, variably referred to as the dispersion relation [22, 35] or the master
stability function [10, 34]. We delineate possible metacommunity outcomes into
three qualitatively different dynamical behaviors based on the form of
$\lambda(\kappa)$. The first, which we label ’stable’ dynamics (st), occur
when perturbations decay on an isolated patch (i.e. $\lambda(0)<0$) and the
homogeneous state $\bf{x}^{\ast}$ is similarly maintained on any spatial patch
network [10], such that $\lambda(\kappa)<0$ for all $\kappa>0$. In contrast,
’unstable’ dynamics (us) are characterized by an initially positive master
stability function (i.e. $\lambda(0)>0$) such that stable coexistence of all
species beginning from $\bm{x}^{\ast}$ is impossible in isolation or on any
spatial network. Finally, ’pattern forming’ dynamics (pf) occur when
$\lambda(0)<0$ but crosses $0$ at some critical $\kappa$ value. In these
systems, certain spatial networks will cause the system to self-organize to a
heterogeneous state where species exhibit variation in densities — either
static or oscillatory — across patches in the spatial network.

To assess the tendency of ecological systems to exhibit any of the particular
dynamical behaviors outlined above, it is important to account for the effects
of parameter variation in Eq. 1. We quantify this tendency as the robustness
of a given behavior to variation in the parameter space of Eq. 1 [36, 35].
Separating the effects of trophic and dispersal interactions, we define the
local robustness $\omega(\bf{P})$, spatial robustness $\omega(\bf{C})$, and
total robustness $\omega(\bf{P},\bf{C})$ as fractions of the appropriate
parameter spaces yielding a particular behavior (see Supplement). This
definition is conceptually akin to that of feasibility domains discussed
elsewhere in ecological literature [37]. Local robustness thus quantifies the
fraction of possible interaction models, encoded in $\bf{P}$, which result in
a particular dynamical outcome under a set of behavioral assumptions on
dispersal, encoded in $\bf{C}$. Similarly, spatial robustness quantifies the
fraction of considered dispersal behaviors with a common outcome under a set
of assumptions on trophic interactions. Finally, the total robustness
quantifies the frequency of a given outcome as both local and spatial
parameters vary.

To construct the connectivity matrix $\bf{C}$, we first assume diagonal
entries are strictly positive to accommodate mass-action diffusive effects.
Off-diagonal elements $\text{C}_{ij}$ may appear for each pairwise food web
interaction $\text{P}_{ij}\neq 0$ resulting in a total of $n_{\text{cross}}$
interspecific dispersal responses. To impose ecologically reasonable (i.e.,
adaptive) constraints we then restrict the signs of these off diagonals as

$\displaystyle\text{sgn}(\text{C}_{ij})=-\text{sgn}(\text{P}_{ij}),~{}i\neq
j,$ (6)

such that the effects of prey tracking and predator avoidance are accounted
for [24, 29]. Rather than assume a specific form for $D_{i}({\bm{x}^{k}})$, we
focus on the linearized spatial responses in Eq. 4. While this prevents us
from performing direct simulation of the system, it allows for a comprehensive
numerical sampling scheme to approximate the robustness of dynamical
behaviors, for which analytical predictions become intractable for $N>2$
species [28].

## 3 species motifs

We start by considering the effects of interspecific dispersal rules on
conventional models of 3-species ecological interaction motifs (Fig. 1). Local
population dynamics on patch $k$ for each $N=3$ interaction motif are defined
by a generalized system of equations:

$\displaystyle f_{i}(\bm{x}^{k})=$ $\displaystyle
x_{i}\Bigl{[}\smash{\underbrace{r_{i}\left(1-\frac{x_{i}}{K_{i}}\right)}_{\begin{subarray}{c}\text{primary}\\\
\text{production}\end{subarray}}}~{}-~{}\smash{\underbrace{\sum_{j\neq
i}{\frac{A_{ij}x_{j}}{B_{ij}+x_{i}}}}_{\begin{subarray}{c}\text{loss from}\\\
\text{predation}\end{subarray}}}~{}+~{}\smash{\underbrace{\sum_{j\neq
i}{\frac{A_{ji}e_{ji}x_{j}}{B_{ji}+x_{j}}}}_{\begin{subarray}{c}\text{gain
from}\\\
\text{predation}\end{subarray}}}~{}-\smash{\underbrace{{d_{i}x_{i}}}_{\text{mortality}}}\Bigr{]}$
(7)

where $r$ and $K$ are the intrinsic population growth rate and carry capacity,
respectively, of producers while $A$, $B$, $e$ and $d$ are the attack rate,
half-saturation constant, conversion efficiency, and mortality coefficient,
respectively, of consumers. The relevant gain and loss terms due to predation
interactions appear according the interaction motif. This corresponds to
ecological models with logistic producer growth, and consumers that exhibit
type-II functional responses and density-dependent mortality.

Figure 2: Robustness of pattern formation in food web motifs

Total robustness of pattern forming dynamics $\omega_{pf}(\bf{P},\bf{C})$
shown is averaged over permutations of $n_{\text{cross}}$ interspecific
dispersal responses for a given local interaction motif. The robustness of
unstable dynamics depends exclusively on local interactions and is indicated
by the hatched area, providing an upper bound on $\omega_{pf}(\bf{P},\bf{C})$.
The robustness of stable dynamics (not shown) is simply
$1-(\omega_{pf}+\omega_{us})$. Thin horizontal line indicates the threshold
above which pattern formation becomes the most robust dynamical behavior.

In accordance with previous studies on 2 species systems, for all considered 3
species systems we find that the inclusion of interspecific spatial responses
has the potential to induce pattern formation which is not possible with
intraspecific dispersal alone (i.e. $n_{\text{cross}}=0$). This potential
increases monotonically with the prevalence of cross diffusion (i.e. with
$n_{\text{cross}}$) such that pattern forming dynamics eventually become the
most robust to variation in both local and spatial interaction parameters for
3 out of 4 food web motifs (Fig. 2). That is, a random parameterization of Eq.
7 is highly likely to yield a steady state $\bf{x}^{\ast}$ susceptible to
pattern formation if spatial interactions between species are common and no
restrictions exist on the relative weight or sign of these interactions. The
increase in $\omega_{pf}(\bf{P},\bf{C})$ with such unconstrained spatial
responses is qualitatively similar across all interaction motifs, despite
consisting of markedly different trophic structures and propensities for
locally unstable dynamics.

However, under the behavioral constraints on dispersal in Eq. 6, the
robustness of pattern forming dynamics is significantly diminished. Regardless
of the local interaction motif or the value of $n_{\text{cross}}$, stable
dynamics become more robust to parameter variation than pattern forming
dynamics. For systems with only 2 directly interacting species, the
constrained value of $\omega_{pf}(\bf{P},\bf{C})$ falls to nearly zero. This
indicates that a locally stable steady state of food webs a-c in Fig. 1 will
be maintained regardless of the underlying spatial network or the particular
model parameters, so long as the behaviors of prey tracking and predator
avoidance are strictly adhered to. For the case of 3 direct interactions in
food web D, pattern forming dynamics remain likely for a randomly
parameterized metacommunity, but significantly less so relative to the case of
unconstrained cross-diffusion. Thus, Fig. 2 reflects an increased tendency for
the system in Eq. 7 to maintain spatial homogeneity rather than transition to
a heterogeneous state when spatial interactions follow reasonable expectations
for adaptive behaviors.

While the cumulative effect of adding interspecific dispersal terms on
metacommunity behavior is equivalently given by the first moment, i.e. the
mean, of either robustness metric, higher moments can shed light on
differential impacts for either spatial or local robustness. Thus, we compute
the second moment, the variance, and the third moment, the skewness, about the
shared mean for the distributions of local and spatial robustness at each
value of $n_{\text{cross}}$. To draw a direct comparison of these quantities
between each robustness metric, we consider the distributions over parameter
samples of Eq. 7 which yield food webs that are both feasible and stable in
isolation, i.e. we exclude webs which yield unstable dynamics.

Figure 3: Distributions of spatial and local robustness

Local (a, c) and spatial (b, d) robustness distributions considered exclude
locally unstable model parameterizations, such that $\omega_{pf}$ may reach
$1$ as opposed to Fig. 2. Without constraints on interspecific dispersal
(dotted line), the sensitivity of pattern forming behavior to dispersal rates
(a) is significantly higher than the sensitivity to local interaction
parameters (b). Dispersal constraints (solid line) render these sensitivities
comparable while diminishing or even prohibiting pattern formation. Example
distributions (c) and (d) for the intraguild predation motif show that, at
high $n_{\text{cross}}$, stabilization of homogeneous equilibria via dispersal
constraints is reflected most prominently in the local robustness
distribution.

Beginning with the second moment, the variance provides a heuristic measure of
model sensitivity to variation in species’ spatial responses when computed for
$\omega(\bf{P})$ or sensitivity to variation in the local parameters of Eq. 7
when computed for $\omega(\bf{C})$. As seen in Fig. 3a, the robustness of
pattern forming dynamics is highly dependent on the species’ spatial
responses, encoded in the near-equilibrium dispersal rates of Eq. 4. For all
local interaction motifs, this sensitivity is maximized at an intermediate
value of $n_{\text{cross}}$ approximately where the difference in total
robustness between stable and pattern forming dynamics is minimized (Fig. 2).
For higher values of $n_{\text{cross}}$, susceptibility to pattern formation
becomes increasingly common for initially stable steady states of Eq. 7, and
thus $\text{Var}(\omega_{pf}(\bf{P}))$ decreases as the dynamical behavior
becomes less dependent on the particular set of dispersal parameters. In
comparison, the sensitivity to local parameters of Eq. 7 shown in Fig. 3b is
relatively low, and the maxima of $\text{Var}(\omega_{pf}(\bf{C}))$ occur
prior to that of the $\omega_{pf}(\bf{P})$ distribution. This indicates that
dispersal is the more decisive factor in determining dynamical behavior
relative to local interaction conditions for systems where species respond to
one another spatially without any constraints on such responses.

The imposition of the constraints in Eq. 6 significantly alters these
sensitivities. For motifs with only two direct interactions, the variance of
either distribution is approximately zero as stable dynamics become virtually
guaranteed. In contrast, the intraguild predation motif (Fig. 1D) maintains a
high sensitivity to near-equilibrium dispersal rates while
$\text{Var}(\omega_{pf}(\bf{C}))$ increases such that the local and spatial
sensitivities become comparable. Thus, when spatial responses are possible
between all species comprising our $N=3$ metacommunity, the dynamical outcome
of Eq. 1 is highly sensitive to the choice of parameters and susceptibility to
pattern formation remains common for an initially stable food web under cross
diffusive constraints, even while stable dynamics become generally preferred.

Turning to the third central moment, the skewness indicates the tendency for
randomly selected dispersal rates to yield values of local robustness higher
or lower than the total robustness (i.e. the mean) when computed for
$\omega(\bf{P})$, and similarly indicates this tendency for randomly selected
local interaction parameters when computed for $\omega(\bf{C})$. We find that
for low values of $n_{\text{cross}}$, the robustness of pattern forming
dynamics tends to be lower than the average shown in Fig. 2 in the absence of
cross-diffusive constraints from Eq. 6. Imposition of the constraints
exacerbates this effect, in particular with respect to the choice of local
interaction parameters, and at higher $n_{\text{cross}}$ the skew towards
relatively low $\omega_{pf}$ is comparable for both robustness distributions.
In the absence of these constraints, however, the local robustness
distribution skews towards relatively high $\omega_{pf}$, while the spatial
robustness becomes symmetric about the mean.

Overall, when dispersal responses between species are few, ecologically
motivated dispersal constraints result in a more significant shift towards
preference of stable dynamics in the $\omega(\bf{C})$ distribution compared to
the $\omega(\bf{P})$ distribution. However, when interspecific dispersal
responses are prevalent, this stabilization of homogeneous equilibria is
instead reflected most prominently in the local robustness distribution. This
highlights the strong potential of interspecific dispersal responses, and
their particular dependence on food web interactions, to determine the
dynamical behavior of spatially explicit metacommunities.

## Large random metacommunities

To understand the impacts of interspecific dispersal rules on the dynamics of
large metacommunities, we employed a method to generate random Jacobian
matrices based on species interactions defined by the niche model [38]. To
construct the network Jacobians, we first generate a food web topology by
drawing niche values randomly from a uniform distribution for each of $N$
species in the metacommunity. These values depict each species’ position along
a 1-dimensional trophic niche axis. Niche ranges are then determined for each
species by randomly sampling values from a Beta distribution with parameters
that depended on the desired connectance [38]. For species pair $(i,j)$, if
the niche value of $j$ falls within the range of $i$, then $i$ is designated
the consumer and $j$ as a resource. The corresponding Jacobian $\mathbf{P}$
entries are modified accordingly: $\text{P}_{ij}$ receives a positive entry
drawn from a folded normal distribution
$\sim\left|\mathcal{N}(0,\sigma)\right|$, while $\text{P}_{ji}$ receives a
value drawn from $\sim-\left|\mathcal{N}(0,\sigma)\right|$. Finally, the
diagonal entries $\text{P}_{ii}$ are set to $-1$, reflecting density-dependent
effects and self-regulation. Resulting webs were examined to ensure that paths
exist between all species in the food web (i.e., there are no disconnected
sub-webs), otherwise the web was discarded and re-generated. To populate the
connectivity matrix $\bf{C}$, we define an auxiliary parameter $q$ which
specifies the probability that the effect of species $j$ on $i$ will lead to a
uniformly distributed dispersal kernel, such that
$\text{C}_{ij}=\text{Bern}(q)~{}\cdot~{}\sim\mathcal{U}(-1,1)$, with the sign
of $\text{C}_{ij}$ optionally constrained by Eq. 6.

The effects of cross-diffusion observed in our 3 species model with fully-
specified trophic interactions qualitatively hold when extended to larger
metacommunities with interactions strengths in Eq. 3 randomly assigned.
Firstly, the potential for pattern formation increases dramatically with the
prevalence of interspecific dispersal responses $q$. In the absence of
constraints on these responses, locally stable $\bf{P}$ matrices are
guaranteed to be susceptible to pattern formation at sufficiently high $q$ for
most food web sizes $N$. A stronger tendency towards pattern formation is
observed relative to our 3 species results, reflected in a higher maximum
robustness and in a lower number of interspecific responses needed to reach
said maximum. Secondly, imposing the ecological constraints of Eq. 6
significantly diminishes $\omega_{pf}(\bf{P},\bf{C})$, even eliminating the
possibility of pattern formation for metacommunities with relatively few
species. While the robustness of pattern forming dynamics still scales with
$q$ for large $N$, pattern forming dynamics never become guaranteed as it does
for unconstrained cross diffusion. Thus, the increased propensity for a
homogeneous state under strict behavioral constraints on interspecific
dispersal is a generic feature of our results, regardless of the number of
species.

Figure 4: Pattern formation in large metacommunities

As in Fig.3, locally unstable model parameterizations are excluded, such that
$\omega_{pf}$ may reach $1$. Vertical axis and color indicate total robustness
of pattern forming behavior in randomly generated metacommunities (methods),
with $q$ the probability of a food web interaction having a corresponding
interspecific dispersal response either without (a) or with (b) the
ecologically motivated constraints in Eq. 6.

## Discussion

In this paper we show that dispersal driven by interspecific interactions has
profound effects on spatial-pattern formation tendencies in metacommunities.
Ecological studies commonly simplify dispersal as a linear function of
intraspecific density with similar rates for all species, assumptions which
tend to stabilize spatially uniform equilibrium states [22, 23]. Generally, we
find that assuming dispersal rates which respond locally to both intra- and
inter- specific densities significantly increases susceptibility to pattern
formation. However, when interspecific dispersal responses are constrained to
follow empirically observed adaptive behaviors [12, 13, 14, 15], i.e. prey
tracking and predator avoidance, pattern formation is dramatically suppressed.
This effect is observed regardless of food web size, although for sufficiently
large number of species pattern forming dynamics take precedent over
homogeneous dynamics with or without adaptive dispersal constraints applied
(Fig. 4).

The three-species interaction motifs investigated here showed minimal pattern
formation when interspecific dispersal was constrained. The intraguild
predation module was a notable exception as it exhibited significant pattern
formation in both the unconstrained and constrained scenarios (Fig. 2). This
motif is distinctive among the three-species motifs we examined because it
involves interactions links spanning multiple trophic levels, potentially
introducing additional dispersal feedbacks that can destabilize the system.
Interestingly, we observed limited pattern formation in larger metacommunities
with fewer than approximately 20 species (Fig. 4B). While all motifs could
potentially co-occur as building blocks of larger metacommunities, it appears
that the homogeneous dynamics observed in certain motifs appear to dominate
less speciose metacommunities. The presence of pattern-forming omnivory in
such cases might be stabilized by other motifs or factors [9] or only become a
frequent module in larger metacommunities. Studies that construct large food
webs around specific interaction motifs could elucidate which ones are most
important for pattern formation and represents an important future research
direction [39, 40].

The spatially heterogeneous metacommunity dynamics observed in our models may
take different forms. Heterogeneous spatial patterns in metacommunities may
include static differences in species densities — Turing-like patterns — and
localized oscillations that may be synchronous or asynchronous across patches
[22, 30, 41, 20]. Certain forms of spatially asynchronous variability
promoting species persistence [42, 32, 31, 43, 44]. Internally generated
patterns have been observed in nature [45], although empirical links between
pattern formation and ecosystem resilience are few [46, 47]. This is
particularly the case of large scale metacommunity dynamics [44], where
variation may only manifest across longer temporal and spatial scales.

Regardless of the exact form, pattern formation mechanisms generate variation
among patches even without underlying variation in the abiotic environment.
However, environmental variation within metacommunities is commonplace and
underpins notable metacommunity paradigms [48]. Models of large random
metacommunities show that spatial environmental variation can promote
community persistence, inverting the classic relationship between community
complexity and instability [9, 49, 50]. Environmental variation may also drive
dispersal, generating another path for altering cross-diffusion terms. By
increasing the number and/or strength of community linkages, cross-diffusion
terms would likely increase complexity and therefore reduce pattern formation
when spatial environmental variation is present. Furthermore, high dispersal
among certain species can provide sufficient dispersal to increase stability
when overall dispersal is low [9, 19]; this effect could be promoted by
adaptive dispersal behavior [51, 52]. Whether constrained or unconstrained
cross-diffusion leads to greater metacommunity complexity or influences
pattern formation in spatially heterogeneous habitats is an open question.

Our models assume that dispersal decisions are based on local resource or
predator densities. Emigrants dispersing from a patch then “resettle” in
neighboring patches without an assessment of the new conditions there. In
contrast, fitness-dependent models implement dispersal as a function of
differences between both the "donor" and "recipient" habitats (e.g., [51, 17,
16]. These alternative representations of dispersal lead to different outcomes
regarding species coexistence and distributions among habitats [8, 53]. Which
model of dispersal best approximates the behavior of real organisms remains an
open question, and likely depends on the focal system in question (reviewed in
[8]). However, it is probably unlikely that individuals can rapidly assess
differences between community conditions at the scale over which most
metacommunities operate (but see [54, 55]).

Still, our assumption that dispersal is triggered directly by local resource
or predator densities may not be general to all ecological systems. While we
restrict interactions to only exist between consumers and resources, competing
species are also known to drive each other’s emigration decisions [56].
Species responding to the presence of competitors may be especially common
under contest competition (e.g. competition for space [57]). Given that direct
dispersal responses to competitors would add cross-diffusion terms similar to
“unconstrained” ones in our model, we expect the inclusion of such responses
to further increase the propensity of pattern forming dynamics.

Furthermore, species may use both intra- and interspecific social cues when
dispersing. Congregation behaviors may be reflected in directional biases in
movement among patches that have been well studied in the context of animal
grouping [58, 21, 59] and are known to facilitate spontaneous organization
into heterogeneous patterns [10, 60, 61, 62]. Organisms may also emigrate from
habitat patches in groups [63] and the cues used to synchronize group
movements may include those from other species [3, 15]. While grouping
behavior and synchronized dispersal among group members are likely to also
lead to pattern formation, the heterogeneous patterns that result may differ
strongly from those in the systems we study here [33, 34].

Teasing apart the contributions of internal and external drivers of observed
spatial patterns continues to be a major empirical challenge. Dispersal shaped
by interactions with other species is one of the many mechanisms that can
generate self-organized variation in community compositions. Given the central
role of dispersal behavior in how organisms respond to their environment,
future changes to environmental conditions may simultaneously alter external
and internal drivers of spatial heterogeneity. Modeling studies such as ours
can play an important role in understanding under what conditions internal
pattern formation is possible, guiding empirical research.

### Data accessibility

Code to reproduce all data and analyses is publicly available at [64].

### Authors’ contributions

All authors contributed to research design and the drafting, writing and
editing of the manuscript. P.L. and A.K.F. implemented data generation and
analysis.

### Acknowledgements

A.K.F. was supported by National Science Foundation grant EF-2222478 and
K.E.A. was supported by National Science Foundation grant DEB-2225098. We
acknowledge this research was primarily conducted on unceded land of the
Cahuilla, Tongva, Luiseño, and Serrano peoples native to Southern California.

## References

  * [1] Adrian C Stier, Andrew M Hein, Valeriano Parravicini and Michel Kulbicki  “Larval dispersal drives trophic structure across Pacific coral reefs”  In _Nature Communications_ 5.1 Nature Publishing Group UK London, 2014, pp. 5575
  * [2] Ashkaan K Fahimipour and Kurt E Anderson  “Colonisation rate and adaptive foraging control the emergence of trophic cascades”  In _Ecology Letters_ 18.8 Wiley Online Library, 2015, pp. 826–833
  * [3] Michael A Gil and Andrew M Hein  “Social interactions among grazing reef fish drive material flux in a coral reef ecosystem”  In _Proceedings of the National Academy of Sciences_ 114.18 National Acad Sciences, 2017, pp. 4703–4708
  * [4] Andrew M Hein and James F Gillooly  “Predators, prey, and transient states in the assembly of spatially structured communities”  In _Ecology_ 92.3 Wiley Online Library, 2011, pp. 549–555
  * [5] Pradeep Pillai, Andrew Gonzalez and Michel Loreau  “Metacommunity theory explains the emergence of food web complexity”  In _Proceedings of the National Academy of Sciences_ 108.48 National Acad Sciences, 2011, pp. 19293–19298
  * [6] Ashkaan K Fahimipour and Andrew M Hein  “The dynamics of assembling food webs”  In _Ecology Letters_ 17.5 Wiley Online Library, 2014, pp. 606–613
  * [7] Priyanga Amarasekare  “Spatial dynamics of foodwebs”  In _Annual Review of Ecology, Evolution, and Systematics_ 39 Annual Reviews, 2008, pp. 479–500
  * [8] Thilo Gross et al.  “Modern models of trophic meta-communities”  In _Philosophical Transactions of the Royal Society B_ 375.1814 The Royal Society, 2020, pp. 20190455
  * [9] Kurt E Anderson and Ashkaan K Fahimipour  “Body size dependent dispersal influences stability in heterogeneous metacommunities”  In _Scientific Reports_ 11.1 Nature Publishing Group UK London, 2021, pp. 17410
  * [10] Ashkaan K Fahimipour et al.  “Sharp thresholds limit the benefit of defector avoidance in cooperation on networks”  In _Proceedings of the National Academy of Sciences_ 119.33 National Acad Sciences, 2022, pp. e2120120119
  * [11] Andrew M Hein et al.  “An algorithmic approach to natural behavior”  In _Current Biology_ 30.11 Elsevier, 2020, pp. R663–R675
  * [12] Bianca Kreuzinger-Janik, Birgit Gansfort and Christoph Ptatscheck  “Population density, bottom-up and top-down control as an interactive triplet to trigger dispersal”  In _Scientific Reports_ 12.1 Nature Publishing Group UK London, 2022, pp. 5578
  * [13] Delphine Legrand et al.  “Ranking the ecological causes of dispersal in a butterfly”  In _Ecography_ 38.8 Wiley Online Library, 2015, pp. 822–831
  * [14] Emanuel A Fronhofer et al.  “Bottom-up and top-down control of dispersal across major organismal groups”  In _Nature Ecology & Evolution_ 2.12 Nature Publishing Group UK London, 2018, pp. 1859–1863
  * [15] Ashkaan K Fahimipour et al.  “Wild animals suppress the spread of socially transmitted misinformation”  In _Proceedings of the National Academy of Sciences_ 120.14 National Acad Sciences, 2023, pp. e2215428120
  * [16] Peter A Abrams  “Habitat choice in predator-prey systems: spatial instability due to interacting adaptive movements”  In _The American Naturalist_ 169.5 The University of Chicago Press, 2007, pp. 581–594
  * [17] Priyanga Amarasekare  “Effect of non-random dispersal strategies on spatial coexistence mechanisms”  In _Journal of Animal Ecology_ 79.1 Wiley Online Library, 2010, pp. 282–293
  * [18] Pierre Quévreux, Rémi Pigeault and Michel Loreau  “Predator avoidance and foraging for food shape synchrony and response to perturbations in trophic metacommunities”  In _Journal of Theoretical Biology_ 528 Elsevier, 2021, pp. 110836
  * [19] Andreas Brechtel, Thilo Gross and Barbara Drossel  “Far-ranging generalist top predators enhance the stability of meta-foodwebs”  In _Scientific reports_ 9.1 Nature Publishing Group UK London, 2019, pp. 12268
  * [20] Eric J Pedersen et al.  “Nonhierarchical dispersal promotes stability and resilience in a tritrophic metacommunity”  In _The American Naturalist_ 187.5 University of Chicago Press Chicago, IL, 2016, pp. E116–E128
  * [21] Peter Turchin  “Quantitative analysis of movement: measuring and modeling population redistribution in animals and plants”  Sinauer Associates, 1998
  * [22] James Dickson Murray  “Mathematical Biology: II: Spatial Models and Biomedical Applications”  Springer, 2003
  * [23] Robert Stephen Cantrell and Chris Cosner  “Spatial ecology via reaction-diffusion equations”  John Wiley & Sons, 2004
  * [24] Vladimir K Vanag and Irving R Epstein  “Cross-diffusion and pattern formation in reaction–diffusion systems”  In _Physical Chemistry Chemical Physics_ 11 Royal Society of Chemistry, 2008, pp. 897–912
  * [25] E.. Zemskov, K. Kassner, M… Hauser and W. Horsthemke  “Turing space in reaction-diffusion systems with density-dependent cross diffusion”  In _Physical Review E_ 87.032906 American Physical Society, 2013
  * [26] Duccio Fanelli, Claudia Cianci and Francesca Di Patti  “Turing instabilities in reaction-diffusion systems with cross diffusion”  In _The European Physical Journal B_ 86.142 EDP Sciences, 2013
  * [27] Anotida Madzvamuse, Raquel Barreira and Alf Gerisch  “Cross-Diffusion in Reaction-Diffusion Models: Analysis, Numerics, and Applications”  In _Progress in Industrial Mathematics at ECMI 2016_ Springer, 2017
  * [28] Andrew L. Krause, Earmonn A. Gaffney, Philip K. Maini and Václav Kilka  “Modern perspectives on near-equilibrium analysis of Turing systems”  In _Philisophical Transactions A_ 379.20200268 The Royal Society, 2021
  * [29] M.A. Tsyganov, J. Brindley, A.V. Holden and V.N. Biktashev  “Soliton-like phenomena in one-dimensional cross-diffusion systems: a predator–prey pursuit and evasion example”  In _Physica D_ 197 Elsevier, 2004, pp. 18–33
  * [30] Shigefumi Hata, Hiroya Nakao and Alexander S Mikhailov  “Dispersal-induced destabilization of metapopulations and oscillatory Turing patterns in ecological networks”  In _Scientific reports_ 4.1 Nature Publishing Group UK London, 2014, pp. 3585
  * [31] Sean M Hayes and Kurt E Anderson  “Persistence in spatial multi-species food webs: The conflicting influences of isolated food web feasibility and spatial asynchrony”  In _Communications in Nonlinear Science and Numerical Simulation_ 119 Elsevier, 2023, pp. 107089
  * [32] Tarik C Gouhier, Frédéric Guichard and Andrew Gonzalez  “Synchrony and stability of food webs in metacommunities”  In _The American Naturalist_ 175.2 The University of Chicago Press, 2010, pp. E16–E34
  * [33] Hiroya Nakao and Alexander S Mikhailov  “Turing patterns in network-organized activator–inhibitor systems”  In _Nature Physics_ 6.7 Nature Publishing Group UK London, 2010, pp. 544–550
  * [34] Andreas Brechtel et al.  “Master stability functions reveal diffusion-driven pattern formation in networks”  In _Physical Review E_ 97.3 APS, 2018, pp. 032307
  * [35] Natalie S. Scholes, David Schnoerr, Mark Isalan and Michael P.H. Stumpf  “A Comprehensive Network Atlas Reveals That Turing Patterns Are Common but Not Robust”  In _Cell Systems_ 9 Elsevier, 2019, pp. 243–257
  * [36] Z. Li, S. Liu and Q. Yang  “Incoherent Inputs Enhance the Robustness of Biological Oscillators”  In _Cell Systems_ 5 Cell Press, 2017, pp. 72–81
  * [37] Jacopo Grilli et al.  “Feasibility and coexistence of large ecological communities”  In _Nature communications_ 8.14389 Nature Publishing Group UK London, 2015
  * [38] Richard J Williams and Neo D Martinez  “Simple rules yield complex food webs”  In _Nature_ 404.6774 Nature Publishing Group UK London, 2000, pp. 180–183
  * [39] Gabriel Gellner and Kevin McCann  “Reconciling the omnivory-stability debate”  In _The American Naturalist_ 179.1 University of Chicago Press Chicago, IL, 2012, pp. 22–37
  * [40] Gabriel Gellner and Kevin S McCann  “Consistent role of weak and strong interactions in high-and low-diversity trophic food webs”  In _Nature communications_ 7.1 Nature Publishing Group UK London, 2016, pp. 11180
  * [41] Shigefumi Hata, Hiroya Nakao and Alexander S Mikhailov  “Sufficient conditions for wave instability in three-component reaction–diffusion systems”  In _Progress of Theoretical and Experimental Physics_ 2014.1 Oxford University Press, 2014, pp. 013A01
  * [42] Karen C Abbott  “A dispersal-induced paradox: Synchrony and stability in stochastic metapopulations”  In _Ecology letters_ 14.11 Wiley Online Library, 2011, pp. 1158–1169
  * [43] Kurt E Anderson and Sean M Hayes  “The effects of dispersal and river spatial structure on asynchrony in consumer–resource metacommunities”  In _Freshwater biology_ 63.1 Wiley Online Library, 2018, pp. 100–113
  * [44] Christian Guill, Janne Hülsemann and Toni Klauschies  “Self-organised pattern formation increases local diversity in metacommunities”  In _Ecology letters_ 24.12 Wiley Online Library, 2021, pp. 2624–2634
  * [45] Max Rietkerk and Johan Van de Koppel  “Regular pattern formation in real ecosystems”  In _Trends in ecology & evolution_ 23.3 Elsevier, 2008, pp. 169–175
  * [46] Kang Zhang et al.  “Self-organized mud cracking amplifies the resilience of an iconic “Red Beach” salt marsh”  In _Science Advances_ 9.18 American Association for the Advancement of Science, 2023, pp. eabq3520
  * [47] John L. Maron and Susan Harrison  “Spatial Pattern Formation in an Insect Host-Parasitoid System”  In _Science_ 278 AAAS, 1997, pp. 1619–1621
  * [48] Mathew A Leibold et al.  “The metacommunity concept: a framework for multi-scale community ecology”  In _Ecology letters_ 7.7 Wiley Online Library, 2004, pp. 601–613
  * [49] Dominique Gravel, François Massol and Mathew A Leibold  “Stability and complexity in model meta-ecosystems”  In _Nature communications_ 7.1 Nature Publishing Group UK London, 2016, pp. 12457
  * [50] “Food-web complexity, meta-community complexity and community stability”  In _Scientific reports_ 6.1 Nature Publishing Group UK London, 2016, pp. 24478
  * [51] Priyanga Amarasekare  “Spatial dynamics of communities with intraguild predation: the role of dispersal strategies”  In _The American Naturalist_ 170.6 The University of Chicago Press, 2007, pp. 819–831
  * [52] A Mougi  “Adaptive migration promotes food web persistence”  In _Scientific reports_ 9.1 Nature Publishing Group UK London, 2019, pp. 12632
  * [53] Philipp Gramlich et al.  “The influence of dispersal on a predator–prey system with two habitats”  In _Journal of Theoretical Biology_ 398 Elsevier, 2016, pp. 150–161
  * [54] Emanuel A Fronhofer, Nicolai Nitsche and Florian Altermatt  “Information use shapes the dynamics of range expansions into environmental gradients”  In _Global Ecology and Biogeography_ 26.4 Wiley Online Library, 2017, pp. 400–411
  * [55] Aurore Ponchon and Justin MJ Travis  “Informed dispersal based on prospecting impacts the rate and shape of range expansions”  In _Ecography_ 2022.5 Wiley Online Library, 2022, pp. e06190
  * [56] Emanuel A Fronhofer, Jan Klecka, Carlos J. Melián and Florian Altermatt  “Condition-dependent movement and dispersal in experimental metacommunities”  In _Nature Ecology & Evolution_ 2.12 Nature Publishing Group UK London, 2018, pp. 1859–1863
  * [57] Vincent Calcagno, Nicolas Mouquet, Philippe Jarne and Patrice David  “Coexistence in a metacommunity: the competition–colonization trade-off is not dead”  In _Ecology letters_ 9.8 Wiley Online Library, 2006, pp. 897–907
  * [58] Akira Okubo  “Diffusion and Ecological Problems: Modern Perspectives”  Springer, 2001
  * [59] Peter Turchin and William T Thoeny  “Quantifying dispersal of southern pine beetles with mark-recapture experiments and a diffusion model”  In _Ecological Applications_ 3.1 Wiley Online Library, 1993, pp. 187–198
  * [60] K.E. Anderson, F.M. Hilker and R. Nisbet  “Directional biases and resource-dependence in dispersal generate spatial patterning in a consumer-producer model”  In _Ecology Letters_ 15 CNRS, 2012, pp. 209–217
  * [61] Yimamu Maimaiti and Wenbin Yang  “Stability and Pattern Formation in a General Class of Reaction-Diffusion-Advection System”  In _Acta Applicandae Mathematicae_ 185.6 Springer, 2023
  * [62] MA Lewis  “Spatial coupling of plant and herbivore dynamics: the contribution of herbivore dispersal to transient and persistent" waves" of damage”  In _Theoretical Population Biology_ 45.3 Elsevier, 1994, pp. 277–312
  * [63] Julien Cote et al.  “Behavioural synchronization of large-scale animal movements–disperse alone, but migrate together?”  In _Biological reviews_ 92.3 Wiley Online Library, 2017, pp. 1275–1296
  * [64] Patrick Lawton, Ashkaan K. Fahimipour and Kurt E. Anderson  “Data from: Interspecific dispersal influences pattern formation in model metacommunities”  In _Philisophical Transactions B_ , 2023  URL: https://github.com/patricklawton/MetacomCrossDiffusion

## Supplement

#### Robustness definitions

First we define a binary variable $\Omega_{b}(\bm{\phi})$ which equals $1$ if
the $n$-dimensional vector of model parameters $\bm{\phi}$ yields a shape of
$\lambda(\kappa)$ corresponding to dynamical behavior $b$ and is 0 otherwise.
The fraction of parameter space resulting in behavior $b$, i.e. the
robustness, is then approximated via the mean value theorem over $N$ random
parameter set samples $\bm{\phi}_{s}$ as

$\displaystyle\omega_{b}=\frac{1}{N}\sum_{s}^{N}{\Omega_{b}(\bm{\phi}_{s})}.$
(S1)

For the local robustness $\omega_{b}(\bf{P})$, $\bm{\phi}$ contains the
parameters appearing in Eq. 5 for the analysis of 3 species motifs, which are
then sampled over $\mathcal{U}(0,10)$. For the spatial robustness
$\omega_{b}(\bf{C})$, $\bm{\phi}$ contains all potentially nonzero elements of
the connectivity matrix $\bf{C}$. As the dynamical behavior is determined by
the relative magnitude of linear coefficients appearing in $\bf{C}$ [41], we
sample elements of $\bf{C}$ on the unit $n-1$ hypersphere, where $n$ is the
number of nonzero $\bf{C}$ elements. Finally, for the total robustness
$\omega_{b}(\bf{P},\bf{C})$, $\bm{\phi}$ contains all possible model
parameters and the average is taken over all parameter combinations.

Figure S1: Skewness of robustness distributions

Skewness of pattern forming dynamics robustness distributions. Distributions
taken over locally feasible and stable webs as done for Fig. 3 in the main
text. The constrained skewness is only shown for the intraguild predation
motif, as pattern formation is prohibited under cross diffusive constraints in
all other motifs, and the skewness is thus uninformative.


# Conditional Reconstruction for Open-set Semantic Segmentation

###### Abstract

Open set segmentation is a relatively new and unexplored task, with just a
handful of methods proposed to model such tasks. We propose a novel method
called CoReSeg that tackles the issue using class conditional reconstruction
of the input images according to their pixelwise mask. Our method conditions
each input pixel to all known classes, expecting higher errors for pixels of
unknown classes. It was observed that the proposed method produces better
semantic consistency in its predictions, resulting in cleaner segmentation
maps that better fit object boundaries. CoReSeg outperforms state-of-the-art
methods on the Vaihingen and Potsdam ISPRS datasets, while also being
competitive on the Houston 2018 IEEE GRSS Data Fusion dataset. Official
implementation for CoReSeg is available at:
https://github.com/iannunes/CoReSeg.

Index Terms—  semantic segmentation, open set, convolutional neural network,
auto-encoder, open world

##  1 Introduction

Fig. 1: Overview of CoReSeg for OS3. An SSeg model and a Conditional
AutoEncoder (CAE) are trained for closed set segmentation and conditional
reconstruction of the input image, respectively. Reconstruction errors are
then thresholded according to a quantile $q$ to determine unknown pixels.

In the last decade many computer vision tasks such as object classification
[1], detection [2] and semantic segmentation [3, 4] have greatly improved
their performance. Semantic Segmentation is a fundamental task in computer
vision field, and its objective is to assign a label to each pixel of the
image [5]. Traditional Semantic Segmentation (SSeg) tasks use closed set
models, since the produced segmentation models are conceived to learn and
predict the same set of classes. However, assuming an open set of classes is
more applicable in real-world scenarios, where classes not seen during
training appear on the deploy phase. In such cases, the method must be able to
identify the pixels of unseen or unknown classes while still correctly
segmenting known classes [6, 7, 8], which is the definition of Open Set
Semantic Segmentation (OS3).

OS3 is inherently harder than open set classification due to its dense
labeling nature. This can explain the gap in the literature for OS3 methods,
with only a handful of works tackling the issue [9]. According to Silva et al.
[7] a dataset with all possible classes known in advance will hardly be found
in practice, especially when dealing with remote sensing datasets. Silva et
al. [7] adapted the well-known OpenMax algorithm [10] for patchwise OS3,
resulting in the OpenPixel method. OpenPixel, however, has a major drawback of
being extremely inefficient during both training and testing. A nonparametric
statistical adaptive OS3 method was proposed by Cui et al. [9]. Employing the
Mann-Whitney U test on a closed set segmentation output to determine the
existence of unknown classes in each image and uses an adaptive threshold that
defines which pixels are defined as unknown.

The first fully convolutional architectures for OS3 were proposed by Oliveira
et al. [8]. Open Fully Convolutional Network (OpenFCN) is a fully
convolutional extension of OpenPixel [7] for dense labeling tasks. Open
Principal Component Scoring (OpenPCS) [8] uses the intermediate feature
activations from closed set networks to fit gaussian distributions in a low-
dimensional projection, obtaining a likelihood score for OOD pixel detection.

The performance of all previously mentioned OS3 methods vary considerably
across datasets and distinct hidden class scenarios. Thus, robustness is still
lacking in OS3 algorithms, with qualitative results varying widely across
tasks and poor semantic consistency along neighboring pixels in OS3
predictions. With these limitations in mind, we propose a novel end-to-end
fully convolutional OS3 method named Conditional Reconstruction for Open Set
Segmentation (CoReSeg). Figure 1 shows an overview of the information flow in
CoReSeg, with reconstruction and closed set segmentation branches merging into
one single open set prediction. We describe this novel development in Section
2 and present experiments and our conclusions along Sections 3, 4 and 5. Our
experiments show that CoReSeg is capable of yielding OS3 predictions with
superior semantic consistency, overall robustness and better average AUROC
when compared to the state-of-the-art.

##  2 Proposed Method: CoReSeg

(a)

(b)

Fig. 2: Open Set Training (a) and Deploy (b) schematics for CoReSeg. $e_{i}$
denotes a layer block on the Closed Set Encoder, $d_{i}$ denotes a block on
the reconstruction decoder, and $f_{i}$ denotes a simplified conditioning
layer that has two encoders $\beta$ and $\gamma$ corresponding the parameters
of a pixelwise FiLM conditioning [11] operation. Introducing the match/non
match conditional reconstruction training procedure in (a), and in (b)
evaluating the model by conditioning the input image with each of the known
classes to find the pixelwise minimum loss.

CoReSeg employs a pre-trained closed set neural network to generate a latent
representation of the input image, which is then sent to a reconstruction
decoder. This decoder also receives conditioning information, whose objective
is to guide the reconstruction of the input image from its latent features
conditioned to the desired class (or classes). An overview of this process is
illustrated in Figure 2(a).

Our method is inspired in some design choices proposed by Oza et al. [12],
wherein the conditioning concerns the entire image, while for CoReSeg the
conditioning is performed in a pixel-wise manner. The main idea of the
conditioning procedure is that objects from known classes will be better
reconstructed when their pixels are conditioned to the correct class, while
unknown objects will present poor reconstructions, since there is no correct
conditioning to these pixels.

CoReSeg has 3 sequential stages to be further explained in the following
paragraphs: 1) Closed Set Training, 2) Conditional Reconstruction, and 3) Open
Set Pixel Recognition.

Closed Set Training: CoReSeg requires a pre-trained closed set segmentation
model to generate latent features that will be later used as input to the
reconstruction decoder. For this reason, the first necessary step is to train
a U-net [4] in a traditional closed set scenario, where only the known classes
are learned. We call the encoder of this U-net the Closed Set Encoder, and its
output is the desired latent representation for the conditional reconstruction
training to be performed later. The resulting model will be used in both Open
Set Training and Deploy phases. Also, the weights of this closed set U-net
will not change after its initial training, which means that the semantic
segmentation layers are frozen during the training of the rest of CoReSeg’s
framework.

Conditional Reconstruction: Conditional Reconstruction training aims at
reconstructing the image that serves as input to the closed set semantic
segmentation block of the framework from its latent representation. The
reconstruction is conducted by a conditioning input, which, in the case of a
semantic segmentation task, is comprised by a mask providing a class for each
pixel. The conditional reconstruction block of the framework can be viewed as
an AutoEncoder where the conditioning layers are the encoder and the
reconstruction layers are the decoder. Figure 2(a) shows how these different
parts of the training are connected to each other.

In order to train the model to reconstruct the image taking into account the
conditioning input, CoReSeg uses for each input image 2 different masks to
condition the reconstruction. The first mask is the one correctly labeled for
the image (match mask, $y_{m}$). The second one is the label from a different
image (non-match mask, $y_{nm}$), which means that it incorrectly conditions
the pixels from the input image. The use of a non-match mask is important to
make sure the network is actually learning to condition the input while not
simply reconstructing the image from the latent representation.

We employ the L1 loss in the reconstruction step. The final reconstruction
loss $\mathcal{L}$ is computed as follows:

$\mathcal{L}=L1(x,\hat{x}_{m})+\alpha*L1(x,\hat{x}_{nm}),$ (1)

where $x$ is the input image, $\hat{x}_{m}$ and $\hat{x}_{nm}$ are the
reconstructions conditioned on $y_{m}$ and $y_{nm}$ respectively, while
$\alpha$ weights the importance of each term.

Aiming to enforce the conditioning, the encoder from the conditional
reconstruction block applies a transformation to the intermediate features
from the frozen Closed Set Encoder layers ($e_{i}$). The result of this
transformation is then used as input on the corresponding layer of the
reconstruction decoder ($d_{i}$). In Figure 2(a), this process is represented
by the $f_{i}(e_{i})$ blocks and it is performed for both match and non-match
conditioning masks. The transformation responsible for the conditioning is the
FiLM method proposed by Perez et al. [11] extended to work in a pixelwise
problem. More specifically, in order to use the pixelwise FiLM, the
conditional reconstruction decoder is composed by two auxiliary encoders:
$\beta$ and $\gamma$. Both encoders have the same shape as $e_{i}$ and
$d_{i}$. In order to apply the transformation, we perform the following
operation $\gamma_{i}\odot e_{i}+\beta_{i}$, where $\beta_{i}$ and
$\gamma_{i}$ are the $i^{th}$ blocks of the conditional reconstruction
encoder, and $e_{i}$ is the output of the $i^{th}$ block from the Closed Set
Encoder. This procedure allows us to perform pixelwise FiLM conditioning on
$e_{i}$.

The reconstruction decoder uses as a main input the latent representation of
the input image from the Closed Set Encoder. Furthermore, each layer of the
reconstruction decoder receives two additional inputs concatenated to the
previous layer activation: (i) the corresponding FiLM transformation ($f_{i}$)
from the conditional reconstruction encoder, and (ii) the raw feature maps
from the corresponding Closed Set Encoder ($e_{i}$). These concatenations can
also be viewed in Figure 2(a).

Open Set Pixel Recognition. During deploy – shown in Figure 2(b) – we cannot
provide match and non-match masks for the conditional encoder, as the labels
for these samples are not available. So, to define which pixels are known and
unknown CoReSeg tries to condition every pixel for each known class. Then, for
each pixel the minimum loss for $k\in\\{1,2,\dots,K\\}$ is computed and
selected – where $\\{1,2,\dots,K\\}$ is the set of known classes. Pixels that
were conditioned to the right class yield a small minimum error, while unknown
pixels result in higher loss values for each one of the reconstructions, since
none of them match the right expected class. At last, a threshold operation
defines which pixels are known and unknown. We use error quantiles to set
thresholds and find the best performance for the model. If the minimum
reconstruction loss of a pixel is below the threshold, its class is deemed as
known and set to the closed set predicted output, and otherwise it is set as
unknown.

##  3 Experimental Setup

Three datasets were selected to evaluate the proposed method:
Vaihingen111https://www2.isprs.org/commissions/comm2/wg4/benchmark/ and
Potsdam1 datasets; and the Houston GRSS 2018 Data Fusion dataset [13]. As the
selected datasets for this work are intended for closed set segmentation, we
emulate open set environments using the Leave One Class Out (LOCO) protocol
[8]. As the final model for the open set pixel recognition process, we choose
the model with the best AUROC result calculated on the validation set.

Fig. 3: Visual results for Potsdam (UUC Impervious Surfaces, top row) and Vaihingen (UUC Building, bottom row) using the LOCO protocol for the closed set model, Softmax with thresholding, OpenPCS and CoReSeg. Method | Backbone | Vaihingen | Potsdam
---|---|---|---
UUCs | Avg. | UUCs | Avg.
0 | 1 | 2 | 3 | 4 | 0 | 1 | 2 | 3 | 4
Softmax | DN-121 | .78 | .63 | .73 | .67 | .58 | $.678\pm.08$ | .66 | .49 | .53 | .64 | .76 | $.616\pm.11$
Softmax | WRN-50 | .80 | .64 | .75 | .63 | .58 | $.680\pm.09$ | .68 | .53 | .60 | .62 | .66 | $.618\pm.06$
Softmax | U-net | .86 | .63 | .76 | .69 | .62 | $.708\pm.10$ | .75 | .50 | .74 | .65 | .68 | $.664\pm.10$
OpenPCS | DN-121 | .84 | .94 | .70 | .81 | .65 | $.788\pm.12$ | .83 | .87 | .28 | .54 | .77 | $.658\pm.25$
OpenPCS | WRN-50 | .87 | .94 | .77 | .63 | .87 | $.816\pm.12$ | .66 | .86 | .71 | .57 | .86 | $.732\pm.13$
OpenPCS | U-net | .90 | .80 | .51 | .84 | .54 | $.717\pm.18$ | .88 | .86 | .42 | .53 | .84 | $.705\pm.21$
CoReSeg (ours) | U-net | .88 | .93 | .71 | .87 | .87 | $\textbf{.854}\pm\textbf{.08}$ | .82 | .90 | .70 | .63 | .77 | $\textbf{.764}\pm\textbf{.07}$
Table 1: Classwise AUROC and average (Avg.) AUROC in Vaihingen and Potsdam. Numbered columns stand for: 1 – Impervious Surfaces, 2 – Building, 3 – Low Vegetation, 4 – High Vegetation and 5 – Car. Method | Backbone | UUCs | Avg.
---|---|---|---
Veg. | Build.
Softmax | DN-121 | .61 | .66 | $.635\pm.04$
Softmax | U-net | .58 | .75 | $.666\pm.12$
OpenPCS | DN-121 | .81 | .76 | $.785\pm.04$
OpenPCS | U-net | .65 | .67 | $.660\pm.01$
OpenIPCS | DN-121 | .90 | .88 | $\textbf{.890}\pm\textbf{.01}$
OpenIPCS | U-net | .69 | .68 | $.683\pm.01$
CoReSeg (ours) | U-net | .82 | .74 | $.780\pm.05$
Table 2: AUROC of Vegetation (Veg.) and Building (Build.) UUCs, as well as the
average (Avg.) AUROC in Houston.

Vaihingen and Potsdam: Both datasets have images with 4 spectral channels IR-
R-G-B paired with the normalized digital surface model (nDSM). The semantic
maps contain 6 classes: impervious surfaces, building, low vegetation, high
vegetation, car and miscellaneous. All experiments were conducted using the
IR-R-G-nDSM channels as input, leaving the Blue channel out. We separated
images from the original training/test set used by OpenPCS [8] to serve as a
validation set: area 23 for Vaihingen; and areas 3_11 and 6_14 for Potsdam.
For both datasets we tested 5 different scenarios in which one class was
selected as UUC for each case.

Houston: For this work only the RGB spectral channels and DSM image were used.
The image fusion procedure we employed was the same as described in OpenPCS
[8]. On Houston we tested 2 different scenarios: 1) Vegetation, wherein the
UUC is composed by healthy grass, stressed grass, evergreen trees, and
deciduous trees; and 2) Building, in which the UUCs are residential and non-
residential buildings.

##  4 Results and Discussion

As presented in Table 1, for Vaihingen and Potsdam, CoReSeg attained the best
average AUROC result. CoReSeg reached a higher and more uniform performance
between the different tested UUCs. As it can be seen in Figure 3 the OS3
obtained using CoReSeg resulted in a much cleaner result than other methods
with similar or even higher AUROC performance. The final segmentation is more
semantically consistent and looks closer to ground truth.

As shown in Table 1, until this work the state-of-the-art performance for both
Vaihingen and Potsdam was achieved by OpenPCS with WRN-50. CoReSeg achieves
better average results across classes, even though WRN-50 has the best AUROC
for 3 UUCs in Vaihingen and 2 UUCs in Potsdam. When the UUC is Low Vegetation
(2) or High Vegetation (3) almost all models and backbones have difficulties
in identifying the OOD pixels. One possible reason for this poor performance
is their high intraclass variability, hampering the characterization of OOD
pixels.

For the Houston dataset, OpenIPCS with DN-121 achieved the best performance as
shown in Table 2. The proposed method was not able to outperform the state-of-
the-art results given by OpenIPCS with DN-121. We hypothesize that this might
be linked to the size of the training set on Houston, which is considerably
smaller than Vaihingen and Potsdam.

##  5 Conclusion

CoReSeg outperformed, in all tested scenarios, the other models using the same
closed set backbone. For Vaihingen and Potsdam datasets, CoReSeg established
new state-of-the-art performance achieving the best average performance.
Adapting other closed set backbones to CoReSeg is a challenge. The
architecture uses spatial information from the Closed Set Encoder as input to
the reconstruction decoder. Adapting a backbone with a different architecture
will likely also imply changes in the CoReSeg architecture.

In our future work we consider: adapting other closed set backbones to work
with the CoReSeg method; modifying the conditioning mechanism to better
characterize each class; test on time series, sparsely labeled and not urban
datasets.

## References

  * [1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “ImageNet Classification with Deep Convolutional Neural Networks,”  NIPS, vol. 25, 2012.
  * [2] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,”  NIPS, vol. 28, 2015.
  * [3] Jonathan Long, Evan Shelhamer, and Trevor Darrell,  “Fully Convolutional Networks for Semantic Segmentation,”  in CVPR, 2015, pp. 3431–3440.
  * [4] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional Networks for Biomedical Image Segmentation,”  in MICCAI. Springer, 2015, pp. 234–241.
  * [5] Xiaohui Yuan, Jianfang Shi, and Lichuan Gu,  “A Review of Deep Learning Methods for Semantic Segmentation of Remote Sensing Imagery,”  Expert Systems with Applications, vol. 169, pp. 114417, 2021.
  * [6] Chuanxing Geng, Sheng-Jun Huang, and Songcan Chen,  “Recent Advances in Open Set Recognition: A Survey,”  IEEE TPAMI, vol. 43, no. 10, pp. 3614–3631, 2021.
  * [7] Caio C. V. da Silva, Keiller Nogueira, Hugo N. Oliveira, and Jefersson A. dos Santos,  “Towards Open-Set Semantic Segmentation Of Aerial Images,”  in Latin American GRSS/ISPRS Remote Sensing Conference (LAGIRS), 2020, pp. 16–21.
  * [8] Hugo Oliveira, Caio Silva, Gabriel L.S. Machado, Keiller Nogueira, and Jefersson A. dos Santos,  “Fully Convolutional Open Set Segmentation,”  Machine Learning, pp. 1–52, 7 2021.
  * [9] Zhiying Cui, Wu Longshi, and Ruixuan Wang,  “Open Set Semantic Segmentation With Statistical Test And Adaptive Threshold,”  in International Conference on Multimedia and Expo (ICME), 2020, pp. 1–6.
  * [10] Abhijit Bendale and Terrance E. Boult,  “Towards Open Set Deep Networks,”  in CVPR, 2016, pp. 1563–1572.
  * [11] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville,  “FiLM: Visual Reasoning with a General Conditioning Layer,”  AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.
  * [12] P. Oza and V. M. Patel,  “C2AE: Class Conditioned Auto-Encoder for Open-Set Recognition,”  in CVPR, Los Alamitos, CA, USA, jun 2019, pp. 2302–2311, IEEE Computer Society.
  * [13] Saurabh Prasad, Bertrand Le Saux, Naoto Yokoya, and Ronny Hansch,  “2018 IEEE GRSS Data Fusion Challenge – Fusion of Multispectral LiDAR and Hyperspectral Data,” 2020.