# On the Need and Applicability of Causality for Fair Machine Learning

Rūta Binkytė111Corresponding author<EMAIL_ADDRESS>INRIA, École
Polytechnique, IPP  Sami Zhioua  INRIA, École Polytechnique, IPP

Causal reasoning has an indispensable role in how humans make sense of the
world and come to decisions in everyday life. While $20th$ century science was
reserved from making causal claims as too strong and not achievable, the
$21st$ century is marked by the return of causality encouraged by the
mathematization of causal notions and the introduction of the non-
deterministic concept of cause [15]. Besides its common use cases in
epidemiology, political, and social sciences, causality turns out to be
crucial in evaluating the fairness of automated decisions, both in a legal and
everyday sense. We provide arguments and examples, of why causality is
particularly important for fairness evaluation. In particular, we point out
the social impact of non-causal predictions and the legal anti-discrimination
process that relies on causal claims. We conclude with a discussion about the
challenges and limitations of applying causality in practical scenarios as
well as possible solutions.

##  1 Introduction

Causal reasoning has an indispensable role in how humans make sense of the
world and come to decisions in everyday life. It is true across epochs and
continents although the particularities and points of emphasis may differ [1].
Although, in response to the critique from Karl Pearson, Ernst Mach, and
Bertrand Russel, $20th$ century science reserved from making causal claims as
too strong and not achievable, the $21th$ century is marked by the return of
causality encouraged by the mathematization of causal notions and introduction
of the non-deterministic concept of cause [15]. In social sciences, causality
is determined by explaining social mechanisms that link the cause and the
effect[21]. For formalization and quantification of the causal relationships,
social sciences mostly employ structural equation models by Boilen [5], and
probabilistic causal models by Pearl (also known as DAG222Directed Acyclic
Graph) or SCM 333Structural Causal Models framework) [27] and Potential
Outcome framework by Rubin [16]. The availability of formal methods and the
implications of links with social domains made the use of causality appealing
for Fairness in machine learning. The recent increase in the application of AI
in the domains of high social impact, such as healthcare, insurance, or
hiring, has shown the risk of amplifying historical discrimination through
automated decision systems. The best-known examples are racial bias in
recidivism prediction [6] and sex bias in Amazon job candidate selection [9].
The general Fairness in ML problem is a) evaluating the equality of labels in
the data or algorithmic predictions for groups or individuals, based on
sensitive protected attributes such as gender, race, religion, and others; b)
intervening in the biased data, algorithm or algorithmic predictions to make
them fairer. The idea of fairness is usually instrumentalized through
mathematical fairness notions. Recently, in addition to the statistical
approach, a number of causal fairness notions and causality-based methods to
detect and mitigate bias have been formulated [23]. Fair Machine Learning
domain mostly makes use of SCM and Potential Outcome causal formalization
approaches, which are viewed as equivalent and complementary as each has
better applicability for specific Fairness tasks [23, 24].

Despite the increase in specific applications of the causal approach, the
general discussion about the benefits and challenges of adaptation of causal
frameworks to Fair machine learning is very limited. Most of the articles
provide specific solutions to causal Fairness problems, give the general
arguments of avoiding spurious correlations and make strong assumptions, for
example about the availability of causal structure, without further
consideration [20, 35, 32, 8, 18]. Loftus et al. [22] summarize the advantages
of using causality in Fair machine learning. Several more recent studies warn
about the dangers of using counterfactual models either because of their
sensitivity to unmeasured confounding or incompatibility with social reality
[17, 19]. A study by Rahmattalabi and Xiang [30] touches upon extensive
assumptions made by the DAG (Pearl [27]) approach and suggests new premises
for the use of the Potential Outcome Framework. [3, 34] lay out the arguments
for causality in a legal context, however, do not clearly link the judicial
process with causal mediation analysis.

This article consolidates the arguments for causality in fair AI incorporating
both Pearl and Potential Outcome frameworks and points out the key challenges
in real-world applications.

The section 2specifies the difference between causality and correlation and
why it is particularly important for the Fairness context. Section 3
introduces causal Fairness notions most relevant for legal anti-discrimination
cases. In the last section 4 we discuss practical conditions for applying
causality frameworks. Finally, we provide conclusions in section 5.

The main contributions of the article are the following:

\- Illustrating the causal approach with general and fairness-related
examples;

\- Emphasizing the need for Causality in the Fairness domain because of the
disruptive social impact of unfair predictions and compatibility of causal
approach with anti-discrimination law;

\- Discussing the main assumptions of Pearl and Potential Outcome causal
approaches and their practical implications;

##  2 When do we need Causality? Prediction and Intervention

Obsessive-
---
Compulsive
Behavior
Happy
---
Glow
Being
---
in Love
(a)

Good
---
Looks
Being
---
Smart
Success (b)

Being
---
Happy
Being
---
Liked
Smile
---
More
Help
---
Others
(c)

Figure 1: The three basic structures of causal graphs: confounder (a),
collider (b), and mediator (c). The dotted lines account for non causal
association.

Racial
---
Minorities
Low Performing
---
Schools
Poor
---
District
Individual
---
Ability
(a)

Gender
---
Talent
---
Being
---
Famous
Banknote
---
Character
(b)

Race
---
Being
---
Hired
Last Name
---
Skill
---
(c)

Figure 2: The three basic structures of causal graphs: confounder (a),
collider (b), and mediator (c). The dotted lines account for non causal
association.

In this section, we will rely on causal graphs to illustrate the difference
between correlation and causation and how it relates to intervention and
prediction in the social domain. We introduce three main causal structures:
confounding, collider, and mediation [27]. We will start with the general
examples (Figure 1) and then go on to bring the discussion back to fairness
scenarios (Figure 2). Let’s start with examples of how the interference of a
third variable (or a group of variables) can create spurious correlations
between otherwise unrelated variables. In the first example, we will consider
a confounding structure, where two related phenomena are caused by the same
factor1(a). Let’s say we observe a correlation between Obsessive-Compulsive
behavior, like checking on the phone all the time and glowing with happiness
(Figure 1(a)). We might come up with the conclusion, that obsessive behavior
may boost your happiness, however, the real reason for both is being in love.
To confirm causal relationships we should only observe people who are not in
love and look if the link between the two effects still holds (observational
study). We could also ask people who are in love but do not demonstrate
compulsive-obsessive behavior to start acting weirdly and check if their
happiness increases (experimental study). Formally, the probability of being
happy ($\mathbb{P}(HappyGlow)$) is independent of the probability of
compulsive-obsessive behavior conditioned on being in Love
($\mathbb{P}(Compulsive-ObsessiveBehavior|InLove)$). In the Fairness related
scenario (Figure 2(a)) confounder bias is illustrated with an example, where a
poor district with lower real estate prices has a higher percentage of racial
minorities whose average income is statistically low. The schools in poor
districts also perform lower for several reasons including funding, access to
learning resources at home, etc. It is not uncommon to predict someone’s
ability by the rating of the attended school. In this case, the predictions
would unjustly harm individuals by miss-classifying them and reinforcing a
vicious cycle by pushing the group further into poverty.

Collider bias is similar to classic confounding as both are induced by the
selection of data sources. Namely, the selection of the sample is biased by
the variable that has an impact both on the treatment and the effect. However,
in the case of confounder or selection bias, the false link is induced by
failing to stratify the common cause of the treatment and the effect, whereas
collider bias is created by stratifying the common effect that precedes the
collection of the data [13]. In the next example (Figure 1(b)), let’s say that
we see a connection between good looks and being smart. We would like to
conclude, that improving your looks will also make you smarter, but then we
notice, that all the people in the sample were highly successful. Since it is
more likely to be successful if you have the brains or the looks, and even
better if you have both, sampling from a population of highly successful
individuals creates a false association between the two. Formally
$\mathbb{P}(GoodLooks)$ and $\mathbb{P}(BeingSmart)$ are correlated given the
Success. In this case, to identify the true relationship we should ensure
appropriate inclusion from the general population [13]. An example of collider
bias in a fairness context (Figure 2(b)) would be considering only famous
artists or scientists for choosing banknote characters. Historically, women
were not accepted in the community of artists and scientists so even the most
talented females had trouble being recognized. If we insist that only well-
known artists and scientists should be honored, or use gender to predict
someone’s talent we will reinforce unjust historical bias.

The third example (Figure 1(c)) illustrates the causal relationship which is
not direct but mediated by intermediary variables. For example, we discover
that happy people are much more popular and liked by their peers. Although
being happy would indeed have an effect on attracting people, it might sound
unachievable to someone who is both unhappy and disliked. However, if we break
the causal effect into indirect effects we see that happy people tend to smile
more and help others more willingly. This is a more tangible behavior change
that could be recommended in order to become more socially attractive.

To illustrate a mediation structure where the fairness of the decision is in
question, we consider a hiring scenario where Race can be connected to the
positive hiring decisions in three possible paths: directly $Race->Being$
Hired, indirectly $Race->LastName->BeingHired$ and indirectly
$Race->Skill->BeingHired$ (Figure 2(c)). Both ”Skill” and ”Last Name” are
mediators. However ”skill” is an explaining variable 444A variable that
mediates and explains the disparate outcome, but is legal and justifiable as a
business necessity., while ”Last Name” is clearly a proxy 555A variable that
is highly correlated with the sensitive attribute and is used to predict the
outcome, although it is not essential for the positive decision.
discrimination. Using the Last Name attribute for the hiring decision is both
unnecessary and harmful for the individuals and the sensitive group.

Note, that both confounder and collider-induced associations are not causal.
In practice, it means, that we cannot use a variable linked through a
confounder or collider for intervention to impose a change in another
variable. For example, we will not make a person glow with happiness by
telling her to act weirdly (compulsive-obsessive) or make someone smart by
changing how they look. However, we can use one variable to predict the
presence of another at least in some populations. For example, it is possible
to predict slippery pavement by seeing umbrellas in the street, even though
they do not cause it, because both are caused by rain (confounding). This
prediction would not have implications for the umbrellas or change the
relationship between slipper pavement and the rain. On the contrary, social
predictions always have implications for the sensitive group through the
feedback loop. In all previously discussed fairness examples (Figure 2) the
false link between the sensitive group and the positive outcome will be
amplified through the biased predictions. Therefore it is possible to argue,
that in sensitive social domains causality is crucial both for predictions and
interventions, as every prediction implies intervention in the social domain.

##  3 Causal Fairness Notions and Legal Contexts

In previous sections, we used causal graphs to illustrate the situations where
it is important to distinguish if the association between sensitive attribute
and decision or effect is causal and, if so, what causal path it traverses. It
is easy to show that one can make qualitative Fairness claims just by looking
at a graph. For example, in the figure,1(c) it is evident that there is no
direct discrimination. However, in practice, for instance, in legal scenarios,
it is essential to quantify the extent of a sensitive attribute’s effect on
the decision. Several Causal Fairness notions were proposed to quantify the
causal effect of sensitive attributes on the outcome and distinguish the
proportion of direct impact as opposed to indirect causal influence. Here we
summarize some of the basic causal metrics and discuss their applicability in
the context of Disparate Impact and Disparate Treatment anti-discrimination
court cases, where the use of causality is particularly relevant [34]. We
refer the interested readers to a more in-depth review of causal fairness
notions [23]. To facilitate the introduction of the causal fairness notions,
we propose an example of a Work visa approval scenario(Figure 3). Certain
countries have an age limit for issuing working visas in their territory.
Younger people are more mobile and willing to work abroad, so they are also
more likely to apply for working visa. In the analyzed sample, we observe more
younger candidates from Country A, which has a younger population, and more
elderly (non-eligible) applicants from Country B. Therefore, age becomes a
confounder hurting the chances of acceptance for Country B. Nationality is
also linked with a relevant skill set and family status (the nationals of some
countries are more likely to have large families). We will consider a court
case where the visa issuing country is accused of discrimination on the basis
of Nationality. We first consider the Disparate Impact framework, which
describes the disproportional negative effect on a protected group caused by a
seemingly neutral policy or requirement. The disparate impact liability
framework does not imply motivation so the evidence can be purely quantitative
[11]. Here the goal is to demonstrate the negative impact disproportionately
experienced by a sensitive group.

NationalityWork VisaAgeSkillFamily Status (a) The scope of Total Variation

NationalityWork VisaAgeSkillFamily StatusNationalityWork VisaAgeSkillFamily
Status (b) The scope of Total Effect

Figure 3:

The most basic statistical fairness notion for measuring disparity of the
decision is Total Variation 666Also known as Statistical Parity or Demographic
Parity. It describes the difference in the expected outcome conditional on the
sensitive attribute:

$TV_{a_{1},a_{0}}(Y=y^{\scriptscriptstyle+})=\mathbb{P}(Y=y^{\scriptscriptstyle+}|A=a_{1})-\mathbb{P}(Y=y^{\scriptscriptstyle+}|A=a_{0})$
(3.1)

Where $Y=y^{\scriptscriptstyle+}$ is a positive decision and $A=a_{1},A=a_{0}$
are the values of the sensitive attribute.

The causal counterpart of Total Variation is Total Effect ($TE$) [27]. Assume
that the sensitive variable $A$ (in this case Nationality) can take two
possible values $a_{0}$ and $a_{1}$ and that the positive outcome is
$y^{\scriptscriptstyle+}$ (approval of work visa), $TE$ is defined as follows:

$TE_{a_{1},a_{0}}(Y=y^{\scriptscriptstyle+})=\mathbb{P}(Y=y^{\scriptscriptstyle+}|do(A=a_{1}))-\mathbb{P}(Y=y^{\scriptscriptstyle+}|do(A=a_{0})$
(3.2)

which measures the effect of the change of $A$ from $a_{1}$ to $a_{0}$ on
$Y=y^{+}$ along all the causal paths from $A$ to $Y$: $Nationality->Visa$,
$Nationality->Skill->Visa$ and $Nationality->FamilyStatus->Visa$. The path
going through Age variable is excluded from measuring the effect, because Age
creates a confounding effect (Figure 3(b)).

The equivalent of TE in the Potential outcome framework is Average Treatment
Effect (ATE) [23]:

$ATE_{a_{1},a_{0}}=\frac{1}{n}\sum_{i=1}^{n}(Y_{i}^{a_{1}}-Y_{i}^{a_{0}})$
(3.3)

Where $Y_{i}^{a_{1}},Y_{i}^{a_{0}}$ correspond to observed and counterfactual
outcomes and $n$ is the number of the observations. In the case where all the
mediators are controlled, ATE would correspond to Direct Effect and therefore
exclude a part of the disparate impact.

All three notions can be used to evaluate the Disparate impact. However, Total
Variation would exaggerate the effect by including non-causal paths (Figure
3(a)). $\mathbb{P}(Y=y|do(A=a))$ denotes the probability of $Y=y$ after an
intervention $A=a$. Total Effect and Total Average Treatment allow for
evaluating the extent of Disparate Impact for one nationality group when the
Age factor is discarded as a non-causal co-incidence.

On the contrary, the Disparate Treatment legal framework looks for direct
discrimination. Following our example, the goal would be to estimate which
part of the disparity is due to the Nationality variable. In this case causal
inference can help to distinguish direct discrimination - the link from the
sensitive attribute to decision - from the effect created by confounding or
mediating variables ($Nationality->Skill->Visa$,
$Nationality->FamilyStatus->Visa$ and $Nationality<-Age->Visa$ ). In the US,
recent court cases have established that a proof of motivation to discriminate
is not sufficient for the court case, but the “but-for causation” standard
needs to be satisfied [10]. Namely, it requires proving that the
discrimination would not have occurred had the plaintiff not been a
disadvantaged group member (e.g., black or a woman). This situation naturally
corresponds to the counterfactual question, which can be answered with causal
counterfactual quantity [12]. The important advantage of the causal approach
is preserving the coherent ”alternative world” model in contrast to a
simplistic flip of the sensitive attribute. For example, in a setting where
height and weight variables are considered, it is not sufficient to switch the
female sensitive value to male without adjusting the weight and height
variables accordingly. This translates to the question, had this female been
denied a job, had she been a man of a height and weight of a corresponding
percentile in the male range. Traditional statistical techniques would not
detect this type of discrimination, but it can be estimated by applying the
causal Natural Direct effect ($NDE$) metric:

($NDE$) [26] defined as:

$NDE_{a_{1},a_{0}}(y^{\scriptscriptstyle+})=\mathbb{P}(y^{\scriptscriptstyle+}_{{a_{1}},\mathbf{Z}_{a_{0}}})-\mathbb{P}(y^{\scriptscriptstyle+}_{a_{0}})$
(3.4)

where $\mathbf{Z}$ is the set of mediator variables and
$\displaystyle\mathbb{P}(y^{\scriptscriptstyle+}_{{a_{1}},\mathbf{Z}_{a_{0}}})$
is the probability of $Y=y^{\scriptscriptstyle+}$ had $A$ been $a_{1}$ and had
$\mathbf{Z}$ been the value it would naturally take if $A=a_{0}$. In case
where all confounding and mediating variables are controlled, the Potential
Outcome ATE metric will correspond to direct effect.

NDE isolates the direct path from all the other indirect links between
sensitive attribute and the decision, which are measured by Natural indirect
effect ($NIE$) [26]:

$NIE_{a_{1},a_{0}}(y)=\mathbb{P}(y_{{a_{0}},\mathbf{Z}_{a_{1}}})-\mathbb{P}(y_{a_{0}})$
(3.5)

Where $X$ is set to $a_{0}$ in the single direct path $A\rightarrow Y$ and is
set to $a_{1}$ in all other indirect paths. In our example, NIE would measure
the joint impact of $Nationality->Skill->Visa$ and
$Nationality->FamilySatus->Visa$.

The second phase of Disparate Impact court case process, requires even more
fine-grained metric, that can split the NIE into to distinct paths
$Nationality->Skill->Visa$ and $Nationality->FamilySatus->Visa$. The procedure
of the inculpation, including the framework for defense, implies the awareness
of paths connecting the sensitive attribute and the decision, and measuring
their impact separately. Naming the variables on the path of disparate impact
gives the defendant a possibility to rebut the claim by proving the business
necessity of the practice [11]. For example, if the path connecting
Nationality and the decision pass through the Skill variable, it can be argued
to be a justifiable reason for the disparity (certain nationals are more
skilled). In this case, we would say that Skill is an explaining variable.
However, if the path through family status, non-essential for the job but more
characteristic to particular countries, is detected, it would be considered
proxy discrimination. In some instances, even the otherwise explaining
variable can be reclassified as a proxy if the extent of disparity proves to
be much higher than the actual business necessity [33]. For example, in the
original Griggs case [7] that set a precedent for Disparate Impact, the
plaintiff pointed out that the requirements to have a high-school diploma or
pass an intelligence test to be transferred between departments were not
informative of a person’s ability to perform the job and had a significant
negative impact on African Americans who did not have the same access to
higher education [7]. This makes the numerical analysis of the extent of path-
specific effect an important plaintiff argument. The extent of the disparity
passing through specific path can be measured by applying Path-specific Effect
(PSE) [8]:

$PSE_{a_{1},a_{0}}^{\pi}(y)=\mathbb{P}(y_{{a_{1}}|\pi,a_{0}|\bar{\pi}})-\mathbb{P}(y_{a_{0}})$
(3.6)

Where $\pi$ is set of the variables on the path of interest,$\bar{\pi}$ is the
set of variables not in $\pi$ and
$\mathbb{P}(y_{{a_{1}}|\pi,a_{0}|\bar{\pi}})$ is the counterfactual
probability of Y=y given the effect of $a_{0}$ on Y through the path in $\pi$
and the effect of $a_{1}$ on Y through the path not in $\pi$.

The Path-specific and Direct Effect formulas include a non-observable
counterfactual quantity that cannot be obtained experimentally or from
observational data. However, under specific assumptions, counterfactual
quantities can be expressed in terms of observational probabilities and hence
can be estimated from data. In such a case, the counterfactual quantities are
said to be identifiable. In the next section, we discuss in detail the
causality-related assumptions.

##  4 Practical considerations to use causality for Fairness

In the previous sections, we illustrate the situations where the causality
approach is relevant for evaluating fairness and how it can be attained using
causal fairness notions and approximation techniques. Despite the apparent
advantages, the applicability of the causal framework is limited due to its
reliance on extensive causal knowledge and strong assumptions. Here we will
list the main requirements for applying DAG and Potential Outcome causal
frameworks and discuss their practical implications.

SUTVA [31] (Potential Outcome)

The SUTVA (Stable Unit Treatment Value Assumption) requires that no influence
on the treatment effect is induced by the interaction between the individuals
or treatment mechanism [24]. Using the scenario of granting work visas (Figure
3), the violation of the first SUTVA requirement would occur if the process of
acceptance is affected by the number of applicants from a particular country,
and the number of applicants from this country increases with the number of
visas approved (individuals joining family members and friends). The second
assumption is violated if the nationals of one country would have different
levels of ”nationality,” which would induce different effects on the outcome.

Ignorability [31] (Potential Outcome) The Ignorability assumption requires
that the sensitive attribute and the outcome are independent given observable
variables. That means no unobserved variables create a significant link
between the sensitive attribute and the outcome. For example, if the Age
confounder is not present in the data, we cannot estimate the causal effect of
the Nationality on the Visa granting decision.

Positivity [31] (Potential Outcome) The positivity is satisfied if the
sensitive attribute is not deterministically related to other attributes. For
example, it would be violated if the value of required skill or family status
would always be the same for a specific Nationality group and the opposite for
another. The same violation would occur if all the nationals of one country
belonged to one age group and all the nationals of the second country belonged
to another age group.

Availability of Causal Graph [27] (DAG) Causal graph is the main requirement
in the DAG framework and provides the full specification of the relationships
in the data. Formally, a causal graph $\mathcal{G}=(\mathbf{V},\mathcal{E})$,
composed of a set of variables/vertices $\mathbf{V}$ and a set of edges
$\mathcal{E}$, is a directed acyclic graph (DAG) that describes the causal
relations between variables. It is subject to further assumptions of Causal
Markov condition, Causal Faithfulness, and Causal Sufficiency that will be
discussed next. All three assumptions together encode the same requirements as
the SUTVA and Ignorability in Potential Outcome.

Causal Markov condition [27] (DAG): Causal Markov condition is related to
graph representation and requires every node to be independent of its non-
descendants given its parents. Formally, a directed acyclic causal graph
$\mathcal{G}$ with a set of vertices $\mathbf{V}$ and a probability
distribution $\mathcal{P}$ over the vertices $\mathbf{V}$ satisfies the Causal
Markov Condition if every node $X$ in $\mathbf{V}$ is independent of
${\textstyle NonDescendants(X)}$ given ${\textstyle Parents(X)}$. It implies
the absence of cycles, a requirement equivalent to the first assumption in
SUTVA.

Causal faithfulness (DAG): Causal faithfulness is also a characteristic of a
causal graph. It requires that all the conditional independence relations that
hold in the data are also encoded in the graph. Formally, a causal graph
$\mathcal{G}$ and a probability distribution $\mathcal{P}$ over the same
variables $\mathbf{V}$ are faithful to each other if all and only the
conditional independence relations that hold in $\mathcal{P}$ are entailed by
the Markov condition and d-separation in $\mathcal{G}$. For example,
faithfulness would be violated if negatively correlated indirect paths in the
data cancel out positively correlated directed paths rending Nationality and
Visa decision variables independent, even though they are dependent (connected
with directed edges) in the graph.

Causal sufficiency [27] (DAG): causal sufficiency requires that there are no
latent (hidden) confounders between variables in the data. It corresponds to
the Ignorability assumption in the Potential Outcome Framework.

Linearity [27, 24] (DAG, Potential Outcome): the value of every variable is
determined by a linear sum of the values of its parents (causes) in addition
to the error term. This assumption makes causal relations simpler and hence
the statistical computations (regression analysis) more tractable. The
violation of linearity would occur if the one Nationality were preferred for
issuing visas for higher-paying jobs and other for lower-paying jobs.

To summarize, we can say that both DAG and Potential Outcome causality
frameworks heavily rely on background knowledge about the generating
mechanisms. Some assumptions are possible to test from the data, for example,
Linearity or Positivity.

Given, a causal graph and the data, the Causal Faithfulness assumption is also
not difficult to test. Others, like Ignorability or SUTVA (also causal Markov
condition and causal sufficiency), are very hard to evaluate. Violation of
causal sufficiency may sometimes be detected from data using causal discovery
algorithms [4]. If all the variables assumed in the causal structure cannot be
obtained for the causal effect estimation, it is often possible to identify
specific causal effects while other effects in the graph remain
unidentified.[24]. This is particularly useful for fairness, as the effect of
interest often is concentrated around the sensitive and few mediating or
confounding variables. The assumption of acyclicity or no interaction has to
be judged from the previous knowledge of the relevant social mechanisms. The
first assumption is difficult to satisfy as feedback loops and interaction are
very common in social science contexts. [24] views it as restrictive but
necessary for carefully interpreting causal analysis results. It is difficult
to satisfy as feedback loops and interaction are very common in social science
contexts. [24] views it as restrictive, but necessary for carefully
interpreting causal analysis results. However, sometimes the interaction
between data points does not affect the whole data set, so parts less affected
by data point dependency can be used for causal analysis [24]. The assumption
of the constant level of effect of sensitive attribute on the outcome, as
noted by [23] is less likely to be violated in the fairness scenarios.

One of the most significant restraints for using causality is the knowledge of
the relationship between the variables, which can range from a fully specified
causal graph with parameters to skeleton structure or partial relationships.
The fact that causal relationships are predominantly expressed as a causal
graph, and the potential outcome framework rejects the use of graphs gives an
impression that the DAG Framework implies more assumptions than the Potential
Outcome framework [30]. However, the full set of assumptions on the causal
relationships is required for both frameworks to evaluate the identifiability
of the causal relationship and point out the minimally sufficient conditioning
set to isolate the specific causal effect [24, 26]777[24] distinguishes ATE as
reliant on causal graphs, while identifiability of ATT and ATC is not defined
by the graph. Potential Outcome analysts usually avoid structural assumptions
by conditioning on as many covariates as possible, but apart from the
computational expenses, this practice can lead to increasing rather than
eliminating bias if the covariate is also a collider (Remember examples in
1(b), 2(b)) [25]. The potential outcome framework has developed certain
techniques to avoid the need for fully specified causal mechanisms for
estimating causal quantities by casting the counterfactuals as missing data
problem [24, 26]. However, the awareness of basic causal structure cannot be
avoided. This is illustrated by the fact that for the evaluation of Path-
specific effects, the most used Potential Outcome libraries require to specify
confounders and mediators before performing the analysis888For example, paths:
An Imputation Approach to Estimating Path-specific Causal Effects
https://github.com/xiangzhou09/paths. Furthermore, [23, 4] shows the
significant disagreement between estimations of causal fairness notions (using
both Potential Outcome and DAG notions) due to slight differences in the
causal structure.

Given the causal structure, Potential Outcome and DAG frameworks can be used
as complementary. While DAG provides sufficient conditions for identifiability
and correct conditioning set, Potential Outcome methods allow to simplify the
computation of counterfactuals [26, 24].

The causal structure (or causal graph) can be obtained by consulting domain
experts, transferring the results learned from experimental studies, or
learning from observational data. All of those approaches have their own
limitations. Domain experts can disagree or have biased assumptions. The
causal transplant of the experimental results can be non-applicable because of
differences between the target and the experiment populations [28]. Finally,
learning from the data requires additional assumptions on data distribution,
functional relationships, the relations of exogenous unobserved variables, and
the informed choice of the learning algorithm. [4] show how different
algorithms for recovering causal structure yield different results when
applied to the same data set. It is already a general consensus, that learning
causal relationships from observational data only may not be realistic [14].
However, the combination of causal discovery, expert knowledge, and
experimental results into a single framework could give more reliable results.
Some methods are available for this purpose. In the case where the causal
structure for the experiment and target population is known, the parametric
relationships can be transferred or the data set augmented using causal
transportability formulas [2, 29, 28]. For example, in the working visas
scenario (Figure 3 the effect of the age confounder could be added to the
analysis if it is known but not observed in the data. In the case where only
the structure from the experimental setting is available, the structure
learned from the observational data can be checked against it to inspect for
any major discrepancies. For example, in the medical scenario, we might check
with the experimental data if the link between gender and disease is indeed
causal when evaluating the possible discrimination in treatment resource
allocation given gender. Another approach frequently used in causal discovery
is the use of tiers to account for temporal priority [4]. The selection of
tiers relies on the idea that certain attributes cannot be an effect of the
others because of temporal priority. For example, sex or race is set at birth
and cannot be caused by other variables. However, this approach is very
restrictive, as it does not consider the possibility for other variables to
cause the appearance of race in the sample rather than in general and thus
does not allow for the detection of certain confounders. For example, in
Figure 2(a) individual race is not caused by living in a poor district, but a
poor (also with accessible pricing) district attracts racial minorities that
tend to be more financially vulnerable. The poor environment also impacts the
school’s performance, thus creating a confounding effect between race and
ability to perform. Given the above, it is evident that causal discovery for
fairness would benefit from a more systematic approach for integrating
background knowledge into causal discovery and specifying fairness-related
restrictions on the relationships between the variables.

##  5 Conclusions

We provide illustrative examples of the difference between causality and
correlation in general and fairness-related scenarios. We show, that while it
is possible to exploit correlation for predictions in certain scenarios, the
non-causal predictions about human beings are creating a negative feedback
loop for groups and individuals. This makes the need for causality
particularly important in Fairness-related contexts. We also provide arguments
for causality and the use of causal fairness notions for discrimination
analysis by connecting them to legal frameworks and phases of the judicial
process. Finally, we conclude that knowledge of causal structure is necessary
for causal fairness estimation. Furthermore, even answering the question if we
need to turn to causal estimations or can rely on simple statistical notions
requires the knowledge of underlying causal relations. Namely, are there any
confounding or mediating variables that need to be adjusted? This puts a
causal graph999Or other representation of the causal relationships in the data
set. in the center of the applicability of Causal Fairness notions and
causality framework in general. Apparently, there is no shortcut to obtaining
a reliable causal structure, however, the combination of expert, experimental
knowledge, and causal discovery methods can bring reasonably reliable results.
We list relevant ideas and methods that can lead to more reliable causal
structure discovery and leave the formalization and unification of those
approaches into a single framework for future research.

##  6 Acknowledgement(s)

We thank the COMETE team of Inria Saclay-Ile-de-France and particularly Carlos
Pinzón and Karima Makhlouf for rich discussions and exchange of ideas. The
project is funded by the ERC grant Hypatia (https//project.inria.fr/hypatia/)
under the Euro- pean Union’s Horizon 2020 research and innovation program.
Grant agreement No 835294.

## References

  * [1] A. Balogun.  Causality in western science and traditional african thought: a study in comparative philosophy.  Saarbrucken: Lambert Academic Publishers, 2010.
  * [2] E. Bareinboim and J. Pearl.  Causal transportability with limited experiments.  In Twenty-Seventh AAAI Conference on Artificial Intelligence, 2013\.
  * [3] S. Barocas and A. D. Selbst.  Big data’s disparate impact.  Calif. L. Rev., 104:671, 2016.
  * [4] R. Binkytė-Sadauskienė, K. Makhlouf, C. Pinzón, S. Zhioua, and C. Palamidessi.  Causal discovery for fairness.  arXiv preprint arXiv:2206.06685, 2022.
  * [5] K. A. Bollen and M. D. Noble.  Structural equation models and the quantification of behavior.  Proceedings of the National Academy of Sciences, 108(supplement_3):15639–15646, 2011.
  * [6] A. Brackey.  Analysis of Racial Bias in Northpointe’s COMPAS Algorithm.  PhD thesis, Tulane University School of Science and Engineering, 2019\.
  * [7] T. E. o. E. Britannica.  Griggs v. duke power co.  Encyclopedia Britannica, 2018.
  * [8] S. Chiappa.  Path-specific counterfactual fairness.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7801–7808, 2019.
  * [9] J. Dastin.  Amazon scraps secret ai recruiting tool that showed bias against women.  In Ethics of Data and Analytics, pages 296–299. Auerbach Publications, 2018.
  * [10] R. Dembroff and I. Kohler-Hausmann.  Supreme confusion about causality at the supreme court.  CUNY Law Review, 25(1), 2022.
  * [11] M. C. Garaud.  Legal standards and statistical proof in title vii litigation: In search of a coherent disparate impact model.  U. Pa. L. Rev., 139:455, 1990.
  * [12] M. Glymour, J. Pearl, and N. P. Jewell.  Causal inference in statistics: A primer.  John Wiley & Sons, 2016.
  * [13] S. Greenland.  Quantifying biases in causal models: classical confounding vs collider-stratification bias.  Epidemiology, 14(3):300–306, 2003.
  * [14] I. Guyon, C. Aliferis, G. Cooper, A. Elisseeff, J. P. Pellet, P. Spirtes, and A. Statnikov.  Causality workbench.  In Causality in the sciences. Oxford University Press, 2011.
  * [15] P. Illari, F. Russo, and J. Williamson.  Why look at causality in the sciences? a manifesto.  Causality in the Sciences, pages 3–22, 2011.
  * [16] G. W. Imbens and D. B. Rubin.  Causal inference in statistics, social, and biomedical sciences.  Cambridge University Press, 2015.
  * [17] A. Kasirzadeh and A. Smart.  The use and misuse of counterfactuals in ethical machine learning, 2021\.
  * [18] A. Khademi, S. Lee, D. Foley, and V. Honavar.  Fairness in algorithmic decision making: An excursion through the lens of causality.  In The World Wide Web Conference, pages 2907–2914, 2019.
  * [19] N. Kilbertus, P. J. Ball, M. J. Kusner, A. Weller, and R. Silva.  The sensitivity of counterfactual fairness to unmeasured confounding.  In Uncertainty in artificial intelligence, pages 616–626. PMLR, 2020.
  * [20] M. J. Kusner, J. Loftus, C. Russell, and R. Silva.  Counterfactual fairness.  Advances in neural information processing systems, 30, 2017.
  * [21] D. Little.  Causal mechanisms in the social realm.  Causality in the Sciences, 27395, 2011.
  * [22] J. R. Loftus, C. Russell, M. J. Kusner, and R. Silva.  Causal reasoning for algorithmic fairness.  arXiv preprint arXiv:1805.05859, 2018.
  * [23] K. Makhlouf, S. Zhioua, and C. Palamidessi.  Survey on causal-based machine learning fairness notions.  arXiv preprint arXiv:2010.09553, 2020.
  * [24] S. L. Morgan and C. Winship.  Counterfactuals and causal inference.  Cambridge University Press, 2015.
  * [25] J. Pearl.  Probabilistic reasoning in intelligent systems: networks of plausible inference.  Morgan kaufmann, 1988.
  * [26] J. Pearl.  Direct and indirect effects.  In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, pages 411–420, 2001.
  * [27] J. Pearl.  Causality.  Cambridge university press, 2009.
  * [28] J. Pearl and E. Bareinboim.  Transportability of causal and statistical relations: A formal approach.  In Twenty-fifth AAAI conference on artificial intelligence, 2011\.
  * [29] J. Pearl and E. Bareinboim.  External validity: From do-calculus to transportability across populations.  In Probabilistic and Causal Inference: The Works of Judea Pearl, pages 451–482. 2022.
  * [30] A. Rahmattalabi and A. Xiang.  Promises and challenges of causality for ethical machine learning.  arXiv preprint arXiv:2201.10683, 2022.
  * [31] D. B. Rubin.  Statistics and causal inference: Comment: Which ifs have causal answers.  Journal of the American Statistical Association, 81(396):961–962, 1986.
  * [32] B. Salimi, L. Rodriguez, B. Howe, and D. Suciu.  Interventional fairness: Causal database repair for algorithmic fairness.  In Proceedings of the 2019 International Conference on Management of Data, pages 793–810, 2019.
  * [33] K. Tobia.  Disparate statistics.  The Yale Law Journal, pages 2382–2420, 2017.
  * [34] A. Xiang.  Reconciling legal and technical approaches to algorithmic bias.  Tenn. L. Rev., 88:649, 2020.
  * [35] J. Zhang and E. Bareinboim.  Fairness in decision-making—the causal explanation formula.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.


# CloudTracks: A Dataset for Localizing
Ship Tracks in Satellite Images of Clouds

Muhammad Ahmed Chaudhry111Stanford University
<EMAIL_ADDRESS>Lyna Kim111Stanford University111Stanford University
<EMAIL_ADDRESS>Jeremy Irvin111Stanford University111Stanford University
<EMAIL_ADDRESS>Yuzu Ido111Stanford University
<EMAIL_ADDRESS>Sonia Chu111Stanford University
<EMAIL_ADDRESS>Jared Thomas Isobe111Stanford University
<EMAIL_ADDRESS>Andrew Y. Ng111Stanford University
<EMAIL_ADDRESS>Duncan Watson-Parris222UC San Diego, Scripps Institution
of Oceanography and Halıcıoğlu Data Science Institute
<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Clouds play a significant role in global temperature regulation through their
effect on planetary albedo. Anthropogenic emissions of aerosols can alter the
albedo of clouds, but the extent of this effect, and its consequent impact on
temperature change, remains uncertain. Human-induced clouds caused by ship
aerosol emissions, commonly referred to as ship tracks, provide visible
manifestations of this effect distinct from adjacent cloud regions and
therefore serve as a useful sandbox to study human-induced clouds. However,
the lack of large-scale ship track data makes it difficult to deduce their
general effects on cloud formation. Towards developing automated approaches to
localize ship tracks at scale, we present CloudTracks, a dataset containing
3,560 satellite images labeled with more than 12,000 ship track instance
annotations. We train semantic segmentation and instance segmentation model
baselines on our dataset and find that our best model substantially
outperforms previous state-of-the-art for ship track localization (61.29 vs.
48.65 IoU). We also find that the best instance segmentation model is able to
identify the number of ship tracks in each image more accurately than the
previous state-of-the-art (1.64 vs. 4.99 MAE). However, we identify cases
where the best model struggles to accurately localize and count ship tracks,
so we believe CloudTracks will stimulate novel machine learning approaches to
better detect elongated and overlapping features in satellite images. We
release our dataset openly at zenodo.org/records/10042922.

_Keywords_ ship tracks, instance segmentation, deep learning, satellite
imagery, climate change

##  1 Introduction

While anthropogenic greenhouse gasses are primarily responsible for the
warming we have experienced to date (Forster et al., 2021), some of that
warming has been masked by the cooling effect of aerosol (Bellouin et al.,
2020). These microscopic particles, such as soot and sulfate, directly reflect
some solar radiation back to space (Ångström and Angstrom, 1929) but can also
affect the albedo of clouds, making them reflect more sunlight to space
(Twomey et al., 1968). As humans mitigate these emissions, as is necessary to
improve air-quality (Cohen et al., 2017; Shindell and Smith, 2019), this
cooling effect will be removed, leading to increased warming. The magnitude of
this masking effect, and hence the additional warming, is one of the leading
uncertainties in future climate change (Watson-Parris and Smith, 2022).

This uncertainty is due, in part, to the difficulty of observing isolated
effects of anthropogenic aerosols on clouds, as they can be confounded with
natural meteorological phenomena or other aerosols that are not easily mapped
to a specific human activity. Ship emissions release sulfate aerosols that act
as cloud condensation nuclei and can enhance cloud droplet numbers, generating
ship tracks—long, thin trails of enhanced cloud brightness that were observed
in some of the very first satellite images of Earth (Conover, 1966). Given
their unambiguous source and relatively clean background environment, these
features provide a unique opportunity to study human-cloud interaction away
from confounding aerosol sources (Christensen et al., 2022). Previous works
have used ship tracks as a sandbox to advance knowledge of anthropogenic
aerosol effects (Albrecht, 1989; Ackerman et al., 2004; Gryspeerdt et al.,
2019; Christensen and Stephens, 2011; Christensen et al., 2009; Goren and
Rosenfeld, 2012).

Recent works have developed approaches for automatically identifying ship
tracks in satellite imagery, in particular using deep learning to detect such
cloud formations in satellite images of stratocumulus clouds (Watson-Parris et
al., 2022; Yuan et al., 2022). Although these preliminary models show promise,
their ability to precisely detect ship tracks is still limited. This is
primarily due to the physical characteristics of ship tracks as seen in Figure
1 that affect their discernibility in satellite imagery: they are long and
thin, they are often overlapping and densely packed, they often have sharp
turns and kinks in their paths, and they can easily be confused with natural
cloud boundaries. Furthermore, previous approaches cannot differentiate
specific instances of ship tracks, which limits the ability to accurately
assess the impact of tracks on cloud morphology. Existing training data for
developing these approaches is limited, which hinders the research community
from addressing these challenges.

(a) Single Instance

(b) Intersecting Instances

(c) Dense Instances

(d) No Instances

Figure 1: Example images in CloudTracks with ship track instance annotations
overlaid. Images in the dataset can contain a single ship track instance (a),
intersecting ship track instances (b), densely packed ship track instances
(c), and no instances (d).

To this end, we present CloudTracks, a dataset for automatically localizing
ship tracks in satellite images of clouds. CloudTracks contains $1,780$ images
collected by the Moderate Resolution Imaging Spectroradiometer (MODIS) each
hand-labeled with individual ship track instances. This dataset offers
significant value compared to previous data (Watson-Parris et al., 2022) for
the following reasons. First, CloudTracks is labeled with individual instance
annotations rather than binary segmentation masks, which enable the
development of instance segmentation models, as we show in this work. These
models can be used to identify individual ship tracks at scale, which can
allow for isolating the effect of individual ship tracks on cloud formation
and enable downstream tasks such as tracking ship movements over time. Second,
CloudTracks contains more accurate labels by employing a systematic annotation
procedure, as detailed in Section 2. We show that this leads to improved
modeling performance both qualitatively and quantitatively. Third, CloudTracks
contains additional images with no ship tracks to help reduce the false
positive detections found in prior work (Watson-Parris et al., 2022).

We develop semantic segmentation and instance segmentation models on the
dataset and show that the models achieve a new state-of-the-art in ship track
localization. However, we find that there are challenging aspects of the ship
track detection task that hinder the ability of well-established segmentation
models. First, ship tracks often manifest as long and thin objects with sharp
turns and kinks, which can be difficult to localize contiguously. Second, the
tracks often overlap, sometimes in dense clusters, which leads to challenges
with differentiating individual instances. We hope that CloudTracks helps
facilitate the development of methods to solve these challenges and helps
enable more accurate ship track detection models that can further our
understanding of anthropogenic effects on clouds and the climate.

##  2 Data

MODIS Channel | Wavelength | Spectrum | Physical Characteristic
---|---|---|---
1 | 645nm | Visible | Optical Thickness
20 | 3.75 micrometers | Near Infrared | Cloud Droplet Size
32 | 12.5 micrometers | Infrared | Cloud Temperature
Table 1: Characteristics of the channels used in the MODIS false color
composite satellite imagery.

###  2.1 MODIS Images

CloudTracks contains 1,780 NASA MODIS Terra and Aqua images collected between
2002 and 2021 inclusive over the various stratocumulus cloud regions (such as
the East Pacific and East Atlantic) where ship tracks have commonly been
observed. Each image has a dimension 1354 x 2030 and a spatial resolution of
1km. For processing images, we followed the methodology described in (Watson-
Parris et al., 2022). Of the 36 bands collected by the instruments, we
selected channels 1, 20, and 32 to capture useful physical properties of cloud
formations, as listed in Table 1. We applied histogram equalization to scale
the channels and construct false color composite images to be used in the
final dataset.

###  2.2 Labeling Methodology

We manually labeled each image in the dataset using a widely used open-source
polygonal annotation tool (Wada, 2018). We used the following criteria to
confirm whether an object was a ship track: the object had to be quasi-linear
that gradually evolved into a more diffuse track, and the track had to be
continuous above background cloud formation. Each ship track instance was
annotated as a sequence of points starting from the head of the track (if
visible) and ending at the last discernible portion of the track. To handle
cases where the end of the track was unclear, we extended the track as far as
possible until it was clear the ship track had terminated. We continued the
track through occlusions if there was a clear track before and after the
occlusion with similar directionality and shape. We began with annotations
from a previous work (Watson-Parris et al., 2022), removed or modified any
annotations that did not conform to the above labeling criteria and procedure,
and annotated any new ship tracks not already labeled in the original set of
annotations. We employed the following strategies to increase the quality of
the ship track labels. To increase inter-rater agreement, a calibration set of
125 images was used to create clear definitions of ship tracks and reduce
annotation differences on challenging cases. To further capture the possible
subjectivity of determining ship tracks, we created two classes for labels,
“ship track” and “uncertain.” We labeled objects entirely consisting of
borderline diffuseness as uncertain. Although investigating the impact of
including the “uncertain” ship tracks is an interesting research direction,
for all subsequent experiments we only retain the “ship track” labels.
Examples of images with ship track annotations overlaid are shown in Figure 1.

###  2.3 Dataset Preprocessing

We converted the ship track instance annotations to semantic segmentation and
instance segmentation masks using the following procedures. For semantic
segmentation, adjacent points within each instance were joined with a line
segment and then buffered using a pixel width of 10. The buffered line
segments from all instances were then rasterized to create a binary
segmentation mask with the same dimensions as the corresponding MODIS image.
For instance segmentation, each instance was converted to a polygon with a
pixel width of 10 for consistency with semantic segmentation and each instance
polygon was converted to pixel coordinates relative to the dimensions of the
corresponding MODIS image. Before inputting the images into the models, we
cropped the original 1354 x 2030 images to obtain two images with 1354 x 1015
resolution. The binary masks obtained from the rasterization and the
polygonized instance annotations were cropped in the same way to obtain the
corresponding ground truth labels. This cropping was selected to balance the
benefits of increased context with GPU memory constraints. Moreover, our
preliminary experiments showed that using crops smaller than 1354 x 1015
indeed harmed model performance since a smaller footprint prohibits the models
from capturing sufficient context in the images to accurately localize the
ship tracks. The resulting dataset consists of 3,560 image and mask pairs. We
randomly split the dataset into a training set (70%), validation set (20%),
and test set (10%). Statistics of the images and annotations in each split are
shown in Table 2.

| Training | Validation | Test | Total
---|---|---|---|---
Positive Images | 1,433 | 415 | 205 | 2,053
Negative Images | 1,066 | 305 | 136 | 1,507
Total Images | 2,499 | 720 | 341 | 3,560
Ship Track Instances | 8,786 | 2,568 | 1,141 | 12,495
Table 2: Statistics of the training, validation, and test sets in CloudTracks.

##  3 Experiments

We ran several experiments on CloudTracks assessing the ship track
localization and counting performance of semantic segmentation and instance
segmentation models. For all models, we tuned the learning rate (1e-3, 1e-4,
1e-5) and optimizer (Adam with default parameters, SGD with a momentum of 0.9)
with weight decay of 0.0001 and a batch size of 2. We ran each experiment
three times with different random seeds and report the mean and standard
deviation of the results. All experiments were run on a single NVIDIA A4000
GPU.

###  3.1 Semantic Segmentation

We developed a variety of semantic segmentation models which input a satellite
image and produce a per-pixel classification indicating which pixels in the
image correspond to ship tracks. Before inputting the images into the models,
we resized the images to a 672 x 672 resolution to adhere to segmentation
architecture requirements and memory constraints. We used spatial
augmentations including random horizontal and vertical flips (each with 50%
chance), affine scaling along the x and y axes (up to 95% to 105% of the
original image sizes), affine translations (by -30% to +30% on the x and y
axes independently), and rotations of 90, 180, or 270 degrees. Furthermore, we
experimented with different loss functions including jaccard loss, binary
cross entropy (BCE), and a convex combination of the two with equal weight.
Our preliminary experiments explored well-established semantic segmentation
model architectures including DeepLabV3 (Chen et al., 2017), UNet (Ronneberger
et al., 2015), and Feature Pyramid Networks (FPNs) (Lin et al., 2017) as well
as various backbone architectures pre-trained on ImageNet (Deng et al., 2009)
including ResNets (ResNet18, ResNet34, ResNet50, ResNet101, ResNet152) (He et
al., 2016), ResNeXt (ResNeXt101) (Xie et al., 2017), DenseNets (DenseNet121
and DenseNet161) (Huang et al., 2017), and EfficientNet (EfficientNet-b7) (Tan
and Le, 2019). The best model was a UNet architecture with a EfficientNet-b7
backbone trained with a learning rate of 1e-4 with an Adam optimizer. We refer
to this as the “Best Semantic Segmentation” model. We compared this to a
reimplementation of the model in Watson-Parris et al. (2022) which uses a UNet
architecture with a ResNet152 backbone trained using an Adam optimizer with a
learning rate of 1e-2 on the original, uncorrected labels of the dataset using
the same splits. We evaluated both models on the test set with the corrected
labels.

###  3.2 Instance Segmentation

We developed instance segmentation models which input a satellite image and
generate a per-pixel classification indicating which pixels in the image
correspond to ship tracks as well as bounding boxes identifying separate
instances of ship tracks. During training, we used common data augmentations
for instance segmentation including random resizing to different image scales
(1333 x 800, 1333 x 768, 1333 x 736, 1333 x 704, 1333 x 672, and 1333 x 640),
random horizontal and vertical flips (each with a 50% flip ratio), and padding
(with a size divisor of 32). We experimented with two instance segmentation
architectures, namely Mask-RCNN (He et al., 2017) and SOLOv2 (Wang et al.,
2020) which uses an FPN with deformable convolutions (Zhu et al., 2019) and we
explored the use of two ResNet backbone encoders (ResNet50, ResNet101) pre-
trained on ImageNet. We used the default loss function for each model
architecture, namely cross entropy loss for the class and mask losses and L1
for the bounding box regression in Mask-RCNN, and dice loss for the mask loss
and focal loss for the class loss with SOLOv2. The best model was a SOLOv2
architecture with a ResNet101 backbone trained with a learning rate of 1e-3
with an SGD optimizer, and we refer to this model as the “Best Instance
Segmentation” model.

###  3.3 Evaluation

####  3.3.1 Localization

For both tasks, we evaluated the localization performance of the models using
Intersection over Union (IoU) on the images with ship tracks and pixel-level
precision, recall, F1 score, and specificity on all images. However, we
observed that ship track predictions can be high quality but still attain low
values of these metrics as ship tracks can be very narrow (Figure 5) and the
ground truth annotations are not always perfectly precise. To address this, we
evaluated with more relaxed performance metrics in the following way. We
considered any predicted ship track pixels within $N$ pixels of the annotated
ship tracks as true positives rather than false positives and any missed
annotated ship track pixels within $N$ pixels of the predicted ship tracks as
true positives rather than false negatives. Then we computed all metrics in
the usual way. We set $N=5$ because we use a width of 10 pixels when
generating the ship track annotations. We report the relaxed metrics in all
subsequent experiments, and report the original metrics for comparison in
Table 4 of the Appendix.

####  3.3.2 Instance Counting

We measured the effectiveness of the models to count the number of instances
using mean average error (MAE) between the predicted number and true number of
ship tracks in each image. To obtain instance predictions with the semantic
segmentation models, we followed Watson-Parris et al. (2022) and used
contouring to detect ship track polygons in the predicted mask. For the
baseline semantic segmentation model, we set the confidence level of the
contouring using their setting of 0.8. For our semantic segmentation model, we
tuned the confidence level of the contouring based on MAE on the validation
set and found that a 0.6 level works best. We also tuned the confidence
threshold for determining whether to keep or drop bounding boxes produced by
the instance segmentation by using the MAE on the validation set, and found
that using a threshold of 0.2 achieves the highest MAE (as well as IoU). These
settings were used for the models respectively when evaluating on the test
set.

##  4 Results

Model | IoU | Precision | Recall | F1 | Specificity
---|---|---|---|---|---
Watson-Parris et al. (2022) | 48.65 $\pm$ 2.28 | 63.57 $\pm$ 3.06 | 49.55 $\pm$ 3.74 | 27.83 $\pm$ 1.66 | 99.88 $\pm$ 0.01
Best Semantic Segmentation | 61.29 $\pm$ 1.11 | 77.63 $\pm$ 1.29 | 71.69 $\pm$ 1.98 | 37.26 $\pm$ 0.27 | 99.89 $\pm$ 0.01
Best Instance Segmentation | 57.55 $\pm$ 0.24 | 74.26 $\pm$ 1.02 | 67.84 $\pm$ 0.71 | 35.45 $\pm$ 0.06 | 99.88 $\pm$ 0.01

Table 3: Test set performance metrics of the semantic and instance segmentation models trained on CloudTracks. Error bars represent the standard deviation of three identical runs with different random seeds. |  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 2: Example images in the test set showcasing the patterns of
improvements of our models over the baseline semantic segmentation model. The
images from left to right are: False-colored satellite image, Ground-truth
annotation, Watson-Parris et al. (2022) prediction, Best Semantic Segmentation
prediction, and Best Instance Segmentation prediction. The first row shows
more contiguous tracks predicted by the improved models, second row shows an
example of increased sensitivity to ship tracks, and third shows less
extraneous predictions.

###  4.1 Localization

Our semantic and instance segmentation models performed comparably on ship
track localization and substantially outperformed the baseline model in
(Watson-Parris et al., 2022) across all measured evaluation metrics (Table 3).
Specifically, the best semantic segmentation model outperformed the baseline
by 12.64 IoU and 9.43 F1. Our best instance segmentation model underperformed
the best semantic segmentation model across all metrics, notably by 3.74 IoU
and 1.81 F1. This may be because the semantic segmentation model was solely
optimized for localization whereas the instance segmentation model was jointly
optimized for localization and instance detection. Still, the localization
performance of the instance segmentation model was close to that of the best
segmentation model. Predictions of both models were strong qualitatively
compared to the baseline (Figure 2). Both models produced longer, more
continuous tracks and fewer spurious short predictions (see the first row of
the figure). Both models were more sensitive to ship tracks, detecting many
more of the tracks in the images (for example the vertical ship tracks in the
second row). The increased sensitivity did not come at the cost of lower
precision; both of our models made fewer extraneous predictions than the
baseline which often made false positive predictions due to confounding cloud
features (see third row).

###  4.2 Instance Counting

The instance segmentation model performed best in identifying the correct
number of instances (Figure 3). Specifically, the instance segmentation model
achieved an MAE of 1.64 $\pm$ 0.05 compared to the best semantic segmentation
model’s performance of 1.90 $\pm$ 0.11 MAE and baseline of 4.99 $\pm$ 0.54
MAE. The baseline semantic segmentation model tended to predict too many ship
tracks compared to the ground truth, in some cases falsely identifying more
than 40 ship tracks. It also often produced a nonzero amount of predictions on
images without any ship tracks. The best semantic segmentation and instance
segmentation models, however, tended to slightly underpredict the amount of
ship tracks in the image. Both models did not ever predict more than 40 ship
tracks in an image, and the instance segmentation model almost always
identified the number of ship tracks within 10 tracks of the ground truth.
Both models also produced less false positive predictions on images without
ship tracks, with the instance segmentation model slightly outperforming the
semantic segmentation model on those images. Representative examples of
correct and incorrect instance predictions are shown in Figure 4. The model
often accurately localized long ship tracks and was able to identify
individual ship tracks in densely packed groups. The model struggled with
images that contain lots of overlap and crossings between ship tracks. The
model also produced some false positive predictions on features of clouds that
appear similarly to ship tracks.

(a) Watson-Parris et al. (2022)

(b) Best Semantic Segmentation

(c) Best Instance Segmentation

Figure 3: Instance counting performance of the three segmentation models on
the CloudTracks test set. The optimal line is shown in teal and best fit line
between the model predicted counts and ground truth counts is shown in pink.

(a) Long track detections

(b) Dense tracks

(c) Difficulty of crossings

(d) Ship track mimicking cloud features

Figure 4: Example instance segmentation predictions on the test set. Ground
truth ship track polygons are shown in red and ship track predicted masks are
shown in green with corresponding predicted bounding box and confidence score
in blue. The model often successfully identified and localized instances of
long ship tracks (a) and densely packed tracks (b). Common mistakes included
difficulties identifying single instances in cases where many ship tracks
cross and images with features that mimic ship tracks, leading to false
positive predictions.

##  5 Discussion

By leveraging CloudTracks, we achieved state-of-the-art ship track semantic
and instance segmentation performance. In addition to a ship track
localization improvement of 8.90 IoU and 18.29 F1 compared to previous work,
we improved upon ship track detection for key qualitative metrics, such as
prediction continuity and precision in difficult satellite images with
confusing objects. The additional granularity of separate ship track instances
makes the resulting dataset and model outputs substantially more useful for
climate science research.

###  5.1 Technical Challenges

Although the models we train on CloudTracks demonstrated superior performance
to prior methods, the dataset still presents multiple challenges to the
segmentation models. Thin, long, and continuous objects like ship tracks are
uncommon in large-scale image datasets used for pretraining, such as COCO (Lin
et al., 2014). This may explain why model architectures developed for that
dataset, and subsequent pretrained models fine-tuned on CloudTracks, struggle
to achieve excellent performance on CloudTracks. We found that this manifests
as discontinuous predictions and difficult localizing instances at crossings
and occlusions where ship tracks are densely packed. It is worthwhile for
future work to explore improving model performance using methods developed to
address discontinuity issues including approaches from boundary detection
(Wang et al., 2022) and deformable linear object detection (Keipour et al.,
2022), as well as methods to handle dense and overlapping objects (Goldman et
al., 2019; Chen et al., 2022). Such approaches could also help improve related
climate tasks such as detecting contrails (Ng et al., 2023).

###  5.2 Dataset Limitations

##### Geographic Diversity

As CloudTracks only contains satellite images in the Pacific Ocean, models
trained on CloudTracks may not generalize to new geographic locations not
represented in the dataset. Future work may benefit from obtaining satellite
images across a more globally distributed area.

##### Label Consistency

It can be difficult to consistently identify ship tracks in satellite imagery,
largely due to subjective judgements about when ship tracks terminate. While
we tried to maximize inter-rater agreement using clear rules for annotation
and an additional “uncertain” class to capture particularly difficult cases,
we acknowledge that the labels in CloudTracks still may not be perfect. We
suggest that users of the dataset are cautious about this, and although we did
not explore it in this work, we believe the use of the uncertainty labels are
also an interesting direction for future work. Furthermore, the ship track
labels were generated as fixed-width buffers of the sequences of points, but
we observed that ship tracks often manifest as varying-width objects, commonly
starting from a narrow point then widening outward through diffusion later in
the track. We used relaxed evaluation metrics to address this, but users of
the dataset may want to explore the use of different ship track widths which
is straightfoward to do from the released version of CloudTracks.

##### Climate Impacts

While our models and experiments are lightweight, we acknowledge the carbon
footprint associated with training deep learning models (Schwartz et al.,
2020). We recommend that future researchers are judicious about energy usage
when training their own models on CloudTracks and in general (Heguerte et al.,
2023).

##  6 Conclusion

We introduce CloudTracks, a new dataset containing 3,560 satellite images
hand-annotated with more than 12,000 ship track instances. We benchmarked
semantic and instance segmentation experiments on the dataset and find that
they achieve state-of-the-art performance on ship track localization. We hope
the dataset will stimulate novel machine learning approaches to improve
detection of thin, occluded, and intersecting objects in noisy geospatial
imagery, as well as advance research about anthropogenic aerosol effects on
clouds and climate change.

## Appendix A

Model | IoU | Precision | Recall | F1 | Specificity
---|---|---|---|---|---
Watson-Parris et al. (2022) | 17.30 $\pm$ 1.22 | 41.06 $\pm$ 1.40 | 31.30 $\pm$ 2.86 | 17.75 $\pm$ 1.18 | 99.80 $\pm$ 0.01
Best Semantic Segmentation | 26.44 $\pm$ 0.17 | 48.61 $\pm$ 1.02 | 51.10 $\pm$ 0.87 | 24.91 $\pm$ 0.11 | 99.76 $\pm$ 0.01
Best Instance Segmentation | 26.64 $\pm$ 0.84 | 49.08 $\pm$ 2.38 | 53.72 $\pm$ 1.21 | 25.62 $\pm$ 0.41 | 99.76 $\pm$ 0.03

Table 4: Original (unrelaxed) test set performance metrics of the semantic and
instance segmentation models trained on CloudTracks. Error bars represent the
standard deviation of three identical runs with different random seeds. Ground
truth maskPredictions overlaidBuffered ground truth Figure 5: Example
demonstrating the motivation to use buffering when computing relaxed
evaluation metrics. Importantly, using the original metrics without buffering,
an almost perfect prediction (middle figure) translated by a few pixels
achieves a very low IoU score.

## References

  * Ackerman et al. [2004] Andrew S. Ackerman, Michael P. Kirkpatrick, David E. Stevens, and Owen B. Toon.  The impact of humidity above stratiform clouds on indirect aerosol climate forcing.  _Nature_ , 432(7020):1014–1017, 12 2004.  ISSN 0028-0836.  doi: 10.1038/nature03174.
  * Albrecht [1989] Bruce A. Albrecht.  Aerosols, Cloud Microphysics, and Fractional Cloudiness.  _Science_ , 245(4923):1227–1230, 9 1989.  ISSN 0036-8075.  doi: 10.1126/science.245.4923.1227.
  * Bellouin et al. [2020] Nicolas Bellouin, Johannes Quaas, Edward Gryspeerdt, Stefan Kinne, Philip Stier, Duncan Watson-Parris, Olivier Boucher, Ken S Carslaw, Matthew Christensen, A-L Daniau, et al.  Bounding global aerosol radiative forcing of climate change.  _Reviews of Geophysics_ , 58(1):e2019RG000660, 2020.
  * Chen et al. [2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Chen et al. [2022] Long Chen, Yuli Wu, and Dorit Merhof.  Instance segmentation of dense and overlapping objects via layering.  _arXiv preprint arXiv:2210.03551_ , 2022.
  * Christensen and Stephens [2011] Matthew W. Christensen and Graeme L. Stephens.  Microphysical and macrophysical responses of marine stratocumulus polluted by underlying ships: Evidence of cloud deepening.  _Journal of Geophysical Research: Atmospheres_ , 116(D3), 2 2011.  ISSN 0148-0227.  doi: 10.1029/2010JD014638.
  * Christensen et al. [2009] Matthew W. Christensen, James A. Coakley Jr., and William R. Tahnk.  Morning-to-Afternoon Evolution of Marine Stratus Polluted by Underlying Ships: Implications for the Relative Lifetimes of Polluted and Unpolluted Clouds.  _Journal of the Atmospheric Sciences_ , 66(7):2097–2106, 7 2009.  ISSN 0022-4928.  doi: 10.1175/2009JAS2951.1.
  * Christensen et al. [2022] Matthew W Christensen, Andrew Gettelman, Jan Cermak, Guy Dagan, Michael Diamond, Alyson Douglas, Graham Feingold, Franziska Glassmeier, Tom Goren, Daniel P Grosvenor, et al.  Opportunistic experiments to constrain aerosol effective radiative forcing.  _Atmospheric chemistry and physics_ , 22(1):641–674, 2022.
  * Cohen et al. [2017] Aaron J Cohen, Michael Brauer, Richard Burnett, H Ross Anderson, Joseph Frostad, Kara Estep, Kalpana Balakrishnan, Bert Brunekreef, Lalit Dandona, Rakhi Dandona, et al.  Estimates and 25-year trends of the global burden of disease attributable to ambient air pollution: an analysis of data from the global burden of diseases study 2015.  _The lancet_ , 389(10082):1907–1918, 2017.
  * Conover [1966] John H. Conover.  Anomalous Cloud Lines.  _Journal of the Atmospheric Sciences_ , 23(6):778–785, 1966.  ISSN 0022-4928.  doi: 10.1175/1520-0469(1966)023¡0778:acl¿2.0.co;2.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Forster et al. [2021] P. Forster, T. Storelvmo, K. Armour, W. Collins, J. L. Dufresne, D. Frame, D. J. Lunt, T. Mauritsen, M. D. Palmer, M. Watanabe, M. Wild, and H. Zhang.  The Earth’s Energy Budget, Climate Feedbacks, and Climate Sensitivity.  Technical report, Cambridge University Press, 2021.
  * Goldman et al. [2019] Eran Goldman, Roei Herzig, Aviv Eisenschtat, Jacob Goldberger, and Tal Hassner.  Precise detection in densely packed scenes.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5227–5236, 2019.
  * Goren and Rosenfeld [2012] Tom Goren and Daniel Rosenfeld.  Satellite observations of ship emission induced transitions from broken to closed cell marine stratocumulus over large areas.  _Journal of Geophysical Research: Atmospheres_ , 117(D17):n/a–n/a, 9 2012.  ISSN 0148-0227.  doi: 10.1029/2012JD017981.
  * Gryspeerdt et al. [2019] Edward Gryspeerdt, Tristan W. P. Smith, Eoin O’Keeffe, Matthew W. Christensen, and Fraser W. Goldsworth.  The Impact of Ship Emission Controls Recorded by Cloud Properties.  _Geophysical Research Letters_ , 46(21):12547–12555, 11 2019.  ISSN 0094-8276.  doi: 10.1029/2019GL084700.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Heguerte et al. [2023] Lucia Bouza Heguerte, Aurélie Bugeau, and Loïc Lannelongue.  How to estimate carbon footprint when training deep learning models? a guide and review.  _arXiv preprint arXiv:2306.08323_ , 2023.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Keipour et al. [2022] Azarakhsh Keipour, Mohammadreza Mousaei, Maryam Bandari, Stefan Schaal, and Sebastian Scherer.  Detection and physical interaction with deformable linear objects.  _arXiv preprint arXiv:2205.08041_ , 2022.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2117–2125, 2017.
  * Ng et al. [2023] Joe Yue-Hei Ng, Kevin McCloskey, Jian Cui, Vincent R. Meijer, Erica Brand, Aaron Sarna, Nita Goyal, Christopher Van Arsdale, and Scott Geraedts.  Opencontrails: Benchmarking contrail detection on goes-16 abi, 2023.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Schwartz et al. [2020] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni.  Green ai.  _Communications of the ACM_ , 63(12):54–63, 2020.
  * Shindell and Smith [2019] Drew Shindell and Christopher J Smith.  Climate and air-quality benefits of a realistic phase-out of fossil fuels.  _Nature_ , 573(7774):408–411, 2019.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR, 2019.
  * Twomey et al. [1968] S Twomey, HB Howell, and TA Wojciechowski.  Comments on “anomalous cloud lines”.  _Journal of the Atmospheric Sciences_ , 25(2):333–334, 1968.
  * Wada [2018] Kentaro Wada.  Labelme: Image Polygonal Annotation with Python, 2018.  URL https://github.com/wkentaro/labelme.
  * Wang et al. [2022] Chi Wang, Yunke Zhang, Miaomiao Cui, Peiran Ren, Yin Yang, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, and Weiwei Xu.  Active boundary loss for semantic segmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 2397–2405, 2022.
  * Wang et al. [2020] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  Solov2: Dynamic and fast instance segmentation.  _Advances in Neural information processing systems_ , 33:17721–17732, 2020.
  * Watson-Parris and Smith [2022] Duncan Watson-Parris and Christopher J. Smith.  Large uncertainty in future warming due to aerosol forcing.  _Nature Climate Change_ , pages 1–3, 2022.  ISSN 1758-678X.  doi: 10.1038/s41558-022-01516-0.
  * Watson-Parris et al. [2022] Duncan Watson-Parris, Matthew W Christensen, Angus Laurenson, Daniel Clewley, Edward Gryspeerdt, and Philip Stier.  Shipping regulations lead to large reduction in cloud perturbations.  _Proceedings of the National Academy of Sciences_ , 119(41):e2206885119, 2022.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Yuan et al. [2022] Tianle Yuan, Hua Song, Robert Wood, Chenxi Wang, Lazaros Oreopoulos, Steven E. Platnick, Sophia von Hippel, Kerry Meyer, Siobhan Light, and Eric Wilcox.  Global reduction in ship-tracks from sulfur regulations for shipping fuel.  _Science Advances_ , 8(29):eabn7988, 2022.  doi: 10.1126/sciadv.abn7988.
  * Zhu et al. [2019] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9308–9316, 2019.
  * Ångström and Angstrom [1929] Anders Ångström and Anders Angstrom.  On the Atmospheric Transmission of Sun Radiation and on Dust in the Air.  _Geografiska Annaler_ , 11:156, 1929.  ISSN 1651-3215.  doi: 10.2307/519399.