# Ethical Dimensions of Visualization Research

Michael Correll  0001-7902-3907 Tableau ResearchSeattleWA<EMAIL_ADDRESS>
###### Abstract.

Visualizations have a potentially enormous influence on how data are used to
make decisions across all areas of human endeavor. However, it is not clear
how this power connects to ethical duties: what obligations do we have when it
comes to visualizations and visual analytics systems, beyond our duties as
scientists and engineers? Drawing on historical and contemporary examples, I
address the moral components of the design and use of visualizations, identify
some ongoing areas of visualization research with ethical dilemmas, and
propose a set of additional moral obligations that we have as designers,
builders, and researchers of visualizations.

Information Visualization, Visual Analytics, Ethics

††copyright: none††conference: ; ; ††journalyear: 2018††ccs: Human-centered
computing Visualization theory, concepts and paradigms††ccs: Security and
privacy Social aspects of security and privacy Figure 1. Visualizations
projects are often described or evaluated as though they are straightforward
paths from data collection to design to the intended user (solid outlines).
This neglects or makes invisible critical populations, impacts, and labor
(dashed outlines) that can contribute to the ethical character of a project.
We have an obligation, where possible, to make these invisible facets and
contributions visible.

##  1\. Introduction

In the wake of leaked information about the NSA’s spying program, Rogaway
wrote “The Moral Character of Cryptographic Work” (Rogaway, 2015). In that
paper, he argues that the work of academics and engineers in cryptography has
an inescapable moral character: it shifts power amongst social groups, and so
has an inherent political impact on society, for good or ill. Critical
movements in cartography (Crampton and Krygier, 2006; Crampton, 2011; Wood,
2010) and data science (boyd and Crawford, 2012; Dalton and Thatcher, 2014)
have begun to analyze how the the use (and abuse) of data shifts structures of
power. Visualization work has the same capacity, and so must also be analyzed
with respect to its moral character. As per Rogaway:

> I suspect that many of you see no real connection between social, political,
> and ethical values and what you work on. You don’t build bombs, experiment
> on people, or destroy the environment. You don’t spy on populations. You
> hack math and write papers. This doesn’t sound ethically laden. I want to
> show you that it is.

In this paper I will draw on the history of visualization and analytics to
illustrate that _all_ visualization research, no matter how superficially
apolitical or trivial, has a moral character. I will then illustrate how this
moral character is reflected in conflicts between virtues that arise in
current emerging areas of visualization research, and how they might be
balanced. My goal with this work is both to promote caution and contemplation
in visualization research (in that we should stop doing unethical work) but
also to present new opportunities for research and growth (in that we should
study the broader impact of our work and look for new areas to explore and
problems to solve).

In the first two sections of this paper I will address the common feeling that
data and data visualization, respectively, are apolitical or somehow ethically
neutral, and that therefore we lack moral obligations with regards to how data
are collected and visualized. It is this tendency to view our work as the mere
reporting or structuring of objective fact that is most dangerous to me.
Heidegger (Heidegger, 1954) specifically calls out the danger of this
perspective:

> Everywhere we remain unfree and chained to technology, whether we
> passionately affirm or deny it. But we are delivered over to it in the worst
> possible way when we regard it as something neutral.

In the final two sections of the paper, I will address current trends in
visualization research that appear to have ethical implications. I will then
use these case studies as a basis to propose additional obligations that
visualization researchers have in addition to their existing moral obligations
as scientists, teachers, and citizens.

##  2\. Against the Neutrality of Data

Figure 2. A table from Thomas Paine’s 1775 pamphlet _Common Sense_(Paine,
1776). These data were collected for an initial accounting purpose, but are
used by Paine to argue for the relative weakness and fragility of the English
Navy, and the potential strength of the American Navy, as part of an argument
for independence and revolution.

It is tempting to claim that visualization is an ethically neutral activity
because we are merely reporting the data, and data are just facts about the
world. It’s not our problem how these facts are collected, or who uses them.
We’re just the middle-man (or, more nefariously, the man in the middle
(Correll and Heer, 2017)) between a stakeholder and their data. Provided that
we did not introduce bias or intentionally deceive when presenting our data,
we completed our duties. However, data are not naturally occurring phenomenon.
The world does not spontaneously quantify, curate, or data-mine itself.
Rather, the process of observing the world and quantifying it is a political
act, and deserves ethical consideration (Barocas and boyd, 2017).

Heidegger identifies quantification as the hallmark of modern technology: the
turning of things (and people) into “standing reserves” of resources
(Heidegger, 1954). A river is not just a flowing thing to be admired, it’s a
certain amount of megawatts of power if connected to a hydroelectric dam. An
acre of forest is not just a scenic location, but a reserve of charcoal and
lumber and so on. Modern technological systems are not (just) alienating, but
an entire reframing of how we relate to the world around us in terms of
exploiting and utilizing resources. And people are, of course, no exception.
The collection of mass data about people is a way of turning them into a
standing reserve (of ad revenue, of content creators, of soldiers, of bodies).

This collection of data, and the distillation of people into data, has
tremendous political power. Gottfried Achenwall coined the term “ _Statistik_
” to be the “science of the state” in his 1752 work _Constitution of the
Present Leading European States_. The collection of vital statistics was
initially intended to be undertaken by the state for such organizational
purposes as determining the size of a tax base, or the amount of trees
available for naval vessels. This initial data collection was by no means
apolitical: the proper data set can help start wars (Fig. 2). Nor has this
centralized and political use and meaning of statistics disappeared in the
digital age: one of the first uses of computing machines to process
statistical population data were the machines that IBM’s subsidiary Dehomag
developed for the Nazi regime, which were used to expedite and support the
Final Solution (Black, 2001; Dillard, 2003). The relative emotional distance
of collecting and reporting on _data_ (as opposed to managing and reporting on
_people_) arguably contributed to the uniquely _bureaucratic_ horrors of the
Holocaust (Ward, 2016), and to what Hannah Arendt refers to as the “banality
of evil” (Arendt, 2006).

Conversely, _refraining_ from collecting data likewise has political and
ethical consequences. Within academia, the convenience sampling of so-called
WEIRD populations (Western, Educated, Industrialized, Rich, and Democratic)
constrains the broader applicability of findings (Henrich et al., 2010), and
excludes populations from consideration in later designs. This imbalance in
data collection can result in unequal outcomes, as with the example of the
over-representation of white faces in computer vision benchmarks resulting in
commercial products that fail to accurately detect or model the faces of
people with darker skin (Buolamwini, 2018). Absence of data can be engineered
for political ends: the Trump administration’s attempt to add a question about
citizenship to the U.S. census is likely an attempt to dissuade non-citizens
from answering the census in fear of retaliation (Wines, 2018), and so
therefore to guide the distribution of state resources away from areas with
larger immigrant populations.

There is no such thing as an objective view from nowhere: rather, knowledge is
_situated_ (Haraway, 1988) within our perspectives and circumscribed by the
limits of our experience. Therefore, data are not neutral and objective facts
about the world— there is no such thing as “raw” data (Gitelman, 2013). Data
are always collected or processed by _someone_ , for some aim. Often the work
that goes into collecting and structuring data is made invisible (D’Ignazio
and Klein, 2016; Day, 2017). Often, too, are the purposes for which these data
are collected and used given less importance than data as an abstract puzzle
to be solved or a collection of insights to be gathered. The emerging field of
“critical data science” (Dalton and Thatcher, 2014) seeks to examine how data
reinforce or challenge systems of power, and to “undo” (boyd, 2016)
assumptions that collecting more data inevitably results in an increase in
efficiency or a decrease in bias.

##  3\. Against the Neutrality of Visualization

(a) Source-destination map (Bundesarchiv, 1940)

(b) Flow map (Meyer and Blohm, 1942)

Figure 3. Two visualizations from the Nazi regime’s “Heim ins Reich” (Home to
the Reich) campaign. This campaign was meant to promote the resettlement of
ethnic Germans from other parts of Europe to newly conquered territories in
Poland. The first map merely mentions that the existing Polish and Jewish
population will be resettled; the second map does not mention them at all.
Also invisible are the original borders of the annexed Polish state.

Well-designed visualizations are often conceived of as clear depictions of
objective data. Drucker (Drucker, 2012) views this framing as particularly
dangerous:

> While it may seem like an extreme statement, I think the ideology of almost
> all current information visualization is anathema to humanistic thought,
> antipathetic to its aims and values. The persuasive and seductive rhetorical
> force of visualization performs such a powerful reification of information
> that graphics such as Google Maps are taken to be simply a presentation of
> “what is,” as if all critical thought had been precipitously and completely
> jettisoned.

In other words, visualizations often depict data as a _given_ , a collection
of facts about the world that brook no argument or disagreement.
Visualizations are often used as part of a rhetorical appeal to the authority
and expertise of the people communicating the data (Richards, 2003), and can
stifle critical or contradictory voices who do not have their own data sets to
point to. Even the language we use to discuss and critique visualizations can
echo implicit biases and inequalities present in society at large (Hill et
al., 2016). Designers often exclude representation of factors like the
uncertainty of the data or the variability of forecasts for reasons of
complexity, scope, or anticipated innumeracy in the audience (Boukhelifa and
Duke, 2009; Greis et al., 2017), which can contribute to the perception that
the data are immutable truths about the world, rather than designed artifacts
representing one flawed, incomplete, and potentially idiosyncratic set of
structured observations. The clean lines and structured layouts of traditional
visualizations communicate authority and certainty in implicit but measurable
ways (Kennedy et al., 2016). While visualizations can be used to promote
exploration and further questioning (as in the “martini glass” (Segel and
Heer, 2010) structured narrative visualization), often designers must use
unconventional designs to promote self-critique or skepticism (Wood et al.,
2012).

Another concern is that data visualization, by presenting the data (rather
than the people behind the data), can result in “cruel” and “inhuman” (Dragga
and Voss, 2001) charts. That is, by treating a chart of casualty figures as no
different qualitatively than a chart of employment statistics, visualizations
can hide the ethical and human suffering underlying the data. The infographics
of the Nazi regime are a particularly heinous (but by no means unique) example
of this erasure. For example Fig. 3 shows the colonization of conquered lands
in full detail while relegating information about the forced resettlement and
likely death of the original occupants to a caption.

Visualization creates an inherent separation between the people impacted by
the data and the people consuming the data. The abstraction, quantification,
and digital presentation creates what Baudrillard calls “virtualization”
(Baudrillard, 1995): an air of unreality about the needs and suffering of
people of concern. Likewise, Cairo (Zer-Aviv, 2015) mentions that “I am just
very skeptical to the idea that data visualization is a medium that can convey
(or even care about conveying) or increase ‘empathy’,” and recent experiments
by Boy et al. (Boy et al., 2017) suggest that even designs where the human
component of data are made more prominent can fail to significantly impact our
empathy with human suffering.

Figure 4. A visualization by W.E.B. Du Bois for the 1900 Paris Exhibition(Du
Bois, 1900). Despite being relatively straightforward charts without much
commentary, Du Bois intended that these visualizations depict the progress and
dangers of the African-American population(Lewis and Willis, 2010) for a moral
and political purpose.

All visualizations are rhetorical, and have the power to potentially persuade
(Pandey et al., 2014). Minor choices in how these charts are designed and
presented can control the message that people take away (Hullman and
Diakopoulos, 2011), occasionally without conscious knowledge: e.g., the
biasing title of a visualization may not be recalled, but can still measurably
impact the remembered contents of a chart (Kong et al., 2018).

Visualization researchers may attempt to sidestep the rhetorical power of
charts by separating visualizations into genres of infographics (that are
meant for general audiences and can be used for persuasion) and statistical
graphics (that are meant for experts and are actively discouraged from having
adornments or embellishments (Bateman et al., 2010)). However, relatively
unadorned visualizations in the style of statistical graphics have a long
history of use by politicians to bolster their arguments (as in Fig 5).

Likewise, visualizations do not have to be explicitly placed in a political or
argumentative context in order to be intended as persuasive. For instance,
while the chart in Fig. 4 may appear to be a statement of demographic fact,
its author, civil rights activist W.E.B. Du Bois, intended it to implicitly
function as part of an argument about the status and trajectory of African-
Americans (Du Bois et al., 1926):

> Thus all art is propaganda and ever must be, despite the wailing of the
> purists. I stand in utter shamelessness and say that whatever art I have for
> writing has been used always for propaganda for gaining the right of black
> folk to love and enjoy. I do not care a damn for any art that is not used
> for propaganda. But I do care when propaganda is confined to one side while
> the other is stripped and silent…

In our own work, the assumption that attempting to persuade with
visualizations is only the goal of the propagandist, and that scientific
visualization and statistical graphics are therefore above such
considerations, cedes rhetorical ground to the groups that do not have such
scruples.

(a) Ronald Reagan uses a chart as part of a public address from the Oval
Office in support of the Economic Recovery Tax Act of 1981, showing the
difference between the Republican and Democratic tax cut plans.

(b) Al Gore uses a chart (and a scissors lift) as part of his movie _An
Inconvenient Truth_ to show the connection between temperature and carbon
dioxide, and the unprecedented scale of recent increases.

Figure 5. Charts lend authority and the perception of objectivity to
arguments. Designers of all charts, not just infographics or educational
graphs, must be mindful of how visualizations can be used to persuade.

In response to these issues, and drawing on similar concerns and methods from
the “critical cartography” (Crampton and Krygier, 2006; Muehlenhaus, 2013)
movement in GIS, Dörk et al. have called for a “critical infovis” (Dörk et
al., 2013) movement with the goal of making explicit the values and politics
of visualizations. Likewise, D’Ignazio and Klein articulate the notion of
“data feminism” (D’Ignazio and Klein, [n. d.]) where the power imbalances in
the process of designing and deploying data visualizations are centered.

##  4\. Concerning Trends in Visualization Research

There are several emerging areas of interest in the visualization community
where work (or the lack of work) is a cause for concern. These areas are
places where _power_ and _responsibility_ are being allocated in ways that
could lead to unethical or irresponsible practice and outcomes. While a full
review of all topics of visualization research, and their associated ethical
considerations, is out of the scope of this paper, I selected these areas as
representing ongoing areas of research where there are values and virtues in
_conflict_. That is, emerging topics where there may not be a single clear
path forward (as in rule-based deontological ethics), but where researchers
will have to balance and cultivate opposing ethical principles (as in virtue
ethics (Hursthouse, 1999)). Virtue ethics does not generate prescriptive rules
to follow or objective measures of success (Louden, 1984). Rather, this
framing suggests mutual (occasionally conflicting) values to cultivate.

I conclude each topic with a list of _design dilemmas_ : open-ended
expressions of ethical implications that might arise from visualization
research in these areas.

###  4.1. Automated Analysis

Figure 6. Systems that seek to automatically locate “insights” in datasets can
save time for users, and assist users without strong backgrounds in
statistics. However, they can promote noise over signal, and lead to
unjustified conclusions. How do we empower users without supporting
potentially dangerous decision-making?

One primary goal of visualization is the affordance of “insights”: complex,
deep, qualitative, unexpected, and relevant (North, 2006) revelations. In
order to support insights, systems are beginning to explore the concept of
automatic recommendations and analyses (Vartak et al., 2015; Zhu and Porter,
2002). The promise of these methods is that analysts can instantly discover
important relationships in data, without having to spend many hours exploring
trivial or uninteresting patterns.

However, analytics systems (or their consumers) may lack the statistical tools
to _validate_ these insights. Therefore, analysts can frequently come away
with conclusions from visualizations that are empirically false or
statistically unsupported (Binnig et al., 2017; Zgraggen et al., 2018).
Automatic methods can exacerbate this problem (Binnig et al., 2017), and
create what Pu and Kay call “p-hacking machines” (Pu and Kay, 2018).

Unfortunately, “p-hacking machines” are alluring from an end-user perspective.
Finding _something_ is a better user experience than finding _nothing_. People
may lack the statistical expertise to properly make use of factors that might
contextualize the importance of patterns in visualizations, like confidence
intervals (Belia et al., 2005) or probability information (Micallef et al.,
2012). Very few visual analytics systems guide users not just to interesting
data, but also through the process of analyzing such findings statistically
(Wacharamanotham et al., 2015). Fewer still take into account decision-making
biases and attempt to correct for analytical paths not taken (Wall et al.,
2017).

An ethical concern with this research is therefore that we are enabling bad
behavior (unjustified, incorrect, and potentially damaging conclusions from
data) without adequate care for the people that can be harmed by decisions
based on these conclusions, or adequate understanding about the literacy and
capabilities of the people who we are empowering with these automated tools.

A further concern is that the our interactive systems only exacerbate the
“garden of forking paths”(Gelman and Loken, 2013) problem that has contributed
to the replication crisis in the sciences. Visualization research itself has
many of the same issues as problematic work in other fields (Kosara and Haroz,
2018). By not creating robust ways of visualizing findings we therefore risk
our own credibility as well.

However, automatic insights, by allowing people to quickly discover important
facets of their data, can empower people without the time or expertise to
discover these findings alone. Systems with excessive guidance or constraints
also reduce the agency of the user. There is therefore a potential conflict
between democratizing data analytics and promoting statistically sound
decision-making (Fig. 6).

Design Dilemmas: How much _guidance_ should analytics systems provide to
users? How _prescriptive_ should such systems be in forbidding or advising
against actions that are likely to lead to statistically spurious conclusions?

###  4.2. Machine Learning

Figure 7. Visualizing Machine Learning models creates a conflict between
_transparency_ in decision-making, and managing the _complexity_. This trade-
off also appears to come at the expense of _accuracy_. Is it more important to
have a more understandable model, or a more accurate one?

Machine learning methods are powerful tools for structuring and making
predictions with data, and are present in many critical areas of our society,
from finance to college admissions. The resulting models are often opaque, and
fail to gracefully allow appeals from people who have be wrongfully or
prejudicially categorized (O’Neil, 2016). We have a moral duty (and, in some
cases, a legal duty (Wachter et al., 2017)) to communicate decision-making
based on ML to the populations that are impacted by it. Communication in this
way provides much needed context for decisions that seem misguided or callous,
and allow those impacted by the decisions to appeal their decisions or seek
better outcomes (Lecher, 2018). Legal scholars such as Citron have argued for
the right to “Technological Due Process” (Citron, 2007) in the face of opaque
algorithmic decision-making.

Despite this obligation, and the historical positioning of visualization as a
way of presenting statistical information to wider audiences, much of the
prominent work on visualizing ML focuses on expert users (Talbot et al., 2009;
Wongsuphasawat et al., 2018). An ethical concern is that we are therefore
empowering the creators of ML models, but are not empowering the people
affected by these models. That is, we are not focused on transparently
communicating why a particular model made a choice (about one’s eligibility
for a loan, or eligibility for parole) to audiences without deep statistical
expertise.

Venues such as the 2018 Workshop on Visualization for AI Explainability
(http://visxai.io/) are beginning to collect scholarship in this area, and
online platforms such as Distill (https://distill.pub/) are beginning to
collect public-facing “explainers” of ML concepts, but currently there are no
standard methods and few success stories of visually communicating algorithmic
decisions to the general audience. Even the very definition of what it means
for an ML model to be “interpretable” is ill-defined and sometimes
contradictory (Lipton, 2016). On the ML side, work on explainability is often
considered in terms of numerically representing the contribution of particular
features (Lundberg and Lee, 2017; Ribeiro et al., 2016), despite the fact that
long lists of feature contributions may be difficult to interpret, or fail to
speak to the domain expertise of the analyst. Methods in common use in the
field, such as saliency maps (Kindermans et al., 2017) or model prototypes
(Kim et al., 2016), are often poor conceptual models of ML behavior.
Optimizing for human understanding of models often requires empirical testing
and optimization independent of the modeling itself (Chang et al., 2009).

Simple models may be more _explainable_ (Gleicher, 2013), but they are often
less _accurate_. The values of transparency and utility may therefore be in
conflict (Fig. 7). We want to give the people impacted by our models the
opportunity to correct errors, identify points of unfairness, and in general
have agency in the decisions that affect them. On the other hand, reducing
complexity to afford explainability could result in performance losses that
result in worse outcomes. Likewise, there are costs even for successful
explanations of ML models and decision-making. Bad actors can game the system
at the expense of those who are participating fairly, as with the “cabal”
(Jeong, 2018) of romance writers who engaged in a number of questionable
activities (such as self-plagiarizing and misleading advertising) in order to
consistently appear at the top of Amazon’s ranking algorithms. Full
transparency in models, especially in models built from demographic data, can
also compromise the privacy of those who have had their data collected (as in
social network data (Wang et al., 2018)).

Design Dilemmas: How much _abstraction_ or _approximation_ should we use when
communicating complex ML models? What _standards_ or _expectations_ should we
cultivate when choosing which parts of algorithmic decision-making to display?

###  4.3. Provenance

Figure 8. When we visualize the _end result_ of a visualization design, but
not the _process_ by which it was created, we risk propagating false,
misleading, or unreproducible findings. On the other hand, showing too many
extraneous details may weaken the rhetorical impact, and increase the
complexity, of visualizations. How do we use data to convince people, but
without taking away agency?

Visual analytics systems are increasing in both complexity and importance.
Combined with the “garden of forking paths” problem mentioned above, this
large number of potential actions means that it is becoming increasingly
difficult to articulate exactly what steps an analyst took in order to produce
a particular chart or to arrive at a particular conclusion. This need becomes
even more important as analytical systems become more tightly integrated with
machine learning, which can be non-deterministic in its output, or highly
dependent on hyperparameters in its input.

An unmet ethical challenge in visualization is therefore to visualize the
provenance of data and decision-making. Communicating the decisions that an
analyst took, and affording different decisions, is a key component of both
affording criticism and supporting transparency in data-driven decision-
making. Much of visualization work is instead focused on affording exploration
and analysis, rather than communication of how this exploration and analysis
was performed (Kosara and Mackinlay, 2013). There is initial work in
increasing the transparency of visualizations: systems like Vistrails
(Callahan et al., 2006) and Hindsight (Feng et al., 2017) represent initial
steps at visualizing scientific workflows and user viewing histories,
respectively. Similarly, “literate visualization” (Wood et al., 2018) has the
goal of making the design decisions that lead to a final visualization
documented and transparent. However, very few visual analytics systems are
built with the goal of analytical transparency in mind.

Notebook-style interfaces such as Jupyter and Observable and other literate
programming environments such as R markdown represent important ecosystems for
the transparent communication of analyses, but require coding or scripting
expertise to construct. In contrast, popular visual analytics systems (such as
Tableau, PowerBI, and Spotfire) heavily rely on GUIs and do not require coding
expertise to use. There is therefore a gap between ease of analysis and ease
of documentation, further muddying the waters between exploratory and
confirmatory analytics.

A connected challenge is the rhetorical one of how to convince viewers not to
be taken in by unreliable data or information (such as “fake news” (Allcott
and Gentzkow, 2017)) or how to mitigate cognitive biases in decision-making
(Dimara et al., 2018). Here there is an ethical balance between supporting the
agency and desires of the viewer (who may not appreciate being intentionally
guided away from the information they want to see) and the desire to
communicate information that is both correct and useful. Visualizing
provenance information and making explicit the analytical choices made by a
system is one way of navigating between these two competing values (Fig 8).

Design Dilemmas: How, and how many, _alternate design or analytical decisions_
should we surface to the user? Should we _audit_ or _structure_ the provenance
of a visualization in order to surface irregularities?

##  5\. What Are Our Obligations As Visualization Researchers?

Visualization, especially visualization research, operates at the intersection
of science, communication, and engineering. We have certain ethical
obligations as scientists (for instance, to avoid breaches of consent and
excesses of harm laid out in codes of research conduct such as the Declaration
of Helsinki (Association, 2013). Likewise, we have ethical obligations as
engineers (for instance, to avoid doing shoddy work or work dangerous to the
public good) as laid out in professional codes of conduct such as the ACM’s
code of ethics (Anderson, 1992). Lastly, insofar as we are presenting data to
the public, we have ethical obligations as journalists (for instance, to issue
corrections and disclose conflicts of interest), as laid out in codes of
conduct such as the SPJ’s code of ethics (Farley et al., 2014).

Beyond the obligation of the component parts of visualization praxis, we also
have obligations in that we have a great deal of power over how people
ultimately make use of data, both in the patterns they see and the conclusions
they draw (Correll and Heer, 2017). We are often the first and only contact a
person might have with an underlying store of data. This gives us special
access to impacted populations, and special responsibilities as we control the
curation, presentation, rhetorical content of the visualizations we create.
Visualizations systems we create also embody design principles concerning
democratization, transparency, clarity, and automation that lend their use a
unique moral signature.

In the following subsections I will present three ethical challenges of
visualization work, related to visibility, privacy, and power. I will briefly
describe how visualization work impacts these realms, and suggest some virtues
or related principles that can ameliorate the negative impact of visualization
work in these spheres. In the spirit of virtue ethics, I do not view these
principles as unimpeachable or absolute. Therefore, I end each section with a
list of potential caveats, where adherence to these principles can have
unwanted ethical impacts, or where there exist virtues whose cultivation may
directly conflict with the principles I propose.

###  5.1. Make the Invisible Visible

There are many potentially invisible aspects of a visualization (Fig. 1).
These non-visualized components such as the choices (and labor) that went into
collecting and curating the data, or the populations that could be impacted by
the decisions made by viewers of these visualizations, have a non-trivial
impact on the good (or harm) that a visualization can do. I echo the view of
Dörk et al. (Dörk et al., 2013) that we have a responsibility to make the
invisible visible:

We ought to visualize hidden labor. Properly acknowledging and rewarding
people for their labor is a key component of fairness. Certain kinds of labor
(especially those performed by marginalized groups) are under-represented or
under-valued in our current schemes of commodification or valuation. For
instance, the “emotional labor” (Hochshild, 1983) of people in service and
nursing-related professions is often overlooked. Echoing D’Ignazio and Klein
(D’Ignazio and Klein, 2016), I believe that the labor that goes into
collecting, curating, and archiving data, and the further work of analyzing
the data, is often invisible in visualizations. Visualizations are often
presented as finished products, with the steps in their construction (and
alternative steps not taken) hidden from view. Beyond fairness in attribution,
making this labor visible is also of benefit to the progress of the field as a
whole. Making the labor of analysis and data prep visible will contribute to
reproducibility and openness, and will also facilitate in the creation of
established standards. Making the work of design and user research visible
(including surfacing intermediate or failed prototypes) will serve as points
of inspiration for other designers, or warnings about potential unfruitful
avenues of effort.

We ought to visualize hidden uncertainty. Uncertainty is an inescapable
component of data collection and analysis, yet it is often hidden from the end
user for reasons of complexity or anticipated literacy. Recent work in novel
visualizations of uncertainty has specifically targeted general audiences
(Greis et al., 2017; Kay et al., 2016) and shown that the ability of the
general public to make use of uncertainty information can be quite high given
the right kind of visual presentation. Weather data, personal informatics, and
electoral polling are all examples where the general public is presented with
data with an inescapable component of uncertainty. The way that this
uncertainty is presented can have measurable impacts on decision-making. For
instance the perceived risk of hurricanes is impacted by how their paths are
depicted (Ruginski et al., 2016), and voter turnout in elections can be
impacted by the race’s perceived closeness (Ashworth et al., 2006), which in
turn can be impacted by how the polling data are presented (Correll and
Gleicher, 2014). To encourage better decision-making (or more uncertainty-
aware decision-making), we must investigate the design space of uncertainty
visualization and how to measure its impact on our audiences, just as we would
also wish to measure task speed or accuracy (Hullman et al., 2018).

We ought to visualize hidden impacts. The ACM Future of Computing Academy has
called for writers of academic papers to make the potential impacts
(especially the potential negative impacts) of academic work to be made
explicit (Hecht et al., 2018). Visualization work often focuses on the
positive aspects of a system (for instance, its ease of use, or the speed or
accuracy with which analysts conduct their tasks), but rarely on the potential
of these systems for harm or misuse. For instance, what harm could be done
using the new classes of insights afforded by the visualization system? What
groups or individuals may be impacted by the work but do not fit the data
model in use? Are there immoral or predatory domains that could make use of
the proposed techniques? Beyond these potentially extreme examples of negative
impacts, there are potential negative impacts even for visualizations that
have been judged to have succeeded in their goals. For instance, will people
be harmed if and when the system ceases to be supported (a common occurrence
for systems developed in academia)? What are the opportunity costs (in terms
of training, collaboration time, and student or developer labor) associated
with the system? Do all of the components of the ultimate value of the
visualization (Van Wijk, 2005) have a positive sign, from a consequentialist
perspective?

Caveats: Visualizations are already complex and multifaceted artifacts, and
designers must frequently struggle with the comprehensibility of their designs
and the literacy of their audience. Visualizing additional facets of the data,
even for laudatory reasons of transparency and accountability, only exacerbate
these issues. In addition to making visualizations more difficult to interpret
(and so limiting their audience), depicting uncertainty and counter-narratives
without proper care can weaken the rhetorical impact of visualizations.
Managing complexity is therefore a virtue in design that can be in direct
opposition with the desire to visualize the invisible.

###  5.2. Collect Data With Empathy

Ceglowski places many of the ills of the current internet at the hands of
“investor storytime”(Ceglowski, 2014): the fable that existing data models are
not quite up to the task of performing feats like microtargeting ads or
predicting user behavior, but that they will be _if only we collect more, and
more personal_ data. This goal results in an increasing pressure to collect as
much data as possible to improve models or build bigger pictures or discover
more context. The resulting ecosystem of omnipresent data collection means
that it is becoming easier and easier to breach the privacy of users with
relatively little effort (Vines et al., 2017). Visualization is well-
positioned to explicitly push against the pressure to collect more and more
data by better communicating data’s value and impact to analysts. We also have
the option of contextualizing and curating data in a way that is respectful of
our larger sets of shared values:

We ought to encourage “small data.” boyd and Crawford claim that “bigger data
are not always better data” (boyd and Crawford, 2012). The collection of
additional data is not just an expenditure of time and resources, it can also
intrusively erode the privacy and agency of the people who are subject to this
data collection. Even if a particular dataset is used for a population’s
benefit, merely setting or eroding expectations of how much data one “needs”
to perform analysis can have negative repercussions for vulnerable populations
in the future. Designers of visualizations and analytics systems should be
able to communicate how much data is “enough,” and condition analysts to
accept tradeoffs of accuracy or certainty in exchange for concision and
protection.

We ought to anthropomorphize data. Much of visualization’s power comes from
the power of abstraction, but this creates a gap between populations and how
they are represented in visualizations: quantization and virtualization of
human beings stymies empathy. Designers must attempt to cross this gap between
map and territory, especially for visualizations with high moral stakes such
as those concerning human suffering. Proposed solutions to recenter human
beings in data visualizations using person-shaped glyphs has not been shown to
produce any additional empathic response (Boy et al., 2017), but including
actual human beings in visualizations can help communicate complex phenomena
(Slobin, 2014) and contribute to human interest in, and memorability of,
visualizations (Borkin et al., 2013, 2016). Visualizations designers may have
to borrow techniques from journalism and rhetoric, and propose novel designs
or interventions, in order to foster empathy and spur action using
visualizations.

We ought to obfuscate data to protect privacy. Visualization designers often
have privileged access to sensitive datasets, and are then charged with
communicating these datasets to wider public spheres. The privacy and consent
of the people whose data we collect or measure is therefore of paramount
ethical importance. People are often unfamiliar with how much data is really
collected or collectible through public APIs or other services. These
inadvertent exposures of data, combined with the ability of visualizations to
highlight previously unseen patterns and trends, can result in severe breaches
of confidentiality and privacy, as with the recent use of fitness tracking
company Strava’s user heatmap and public API to identify the locations and
internal layouts of U.S. military bases (Hsu, 2018). Preserving the privacy of
the people in our dataset may involve novel designs (Dasgupta and Kosara,
2011) or interactive workflows (Wang et al., 2018). Both of these classes of
techniques involve aggregating, fuzzing, or otherwise restructuring data to
preserve privacy. A related component of this obfuscation is then
communicating the upper limit of accuracy or detail to analysts.

Caveats: Restricting the type and amount of data that we collect has a direct
impact on the quality and scope of our analyses. It may be laudatory to avoid
collecting unnecessary information, but these seemingly irrelevant fields can
result in serendipitous discoveries that might not have otherwise been
possible. Limiting the scope of data collection also entails a form of
selection bias that could result in biased or unjust conclusions arising from
a lack of context. Aggregation to preserve privacy also can result in
seemingly contradictory conclusions such as Simpson’s Paradox. Our obligation
to provide context and analytical power can therefore stand in direct
opposition to the empathic collection of data.

Likewise, our empathic judgments can be biased or otherwise fraught. For
instance, Nagel’s concept of _moral luck_ (Williams and Nagel, 1976) notes the
role that blind chance plays on our moral judgments. For instance, attempted
murderers receive lighter sentences than murderers who succeed in killing
their victims, even if their intent and actions were identical. Similarly,
Bloom (Bloom, 2017) argues that our visceral empathic reactions can be misused
in service of violent or discriminatory ends. Cultivating our _sympathy_ may
therefore cause us to come in conflict with institutional fairness. Patil et
al (Patil et al., 2018) argue for a checklist-based system of ethical
reflection to circumvent these complexities.

###  5.3. Challenge Structures of Power

Satirist Peter Finley Dunne suggested that one of the jobs of newspapers is to
“comfort the afflicted” and “afflict the comfortable.” Data visualizations do
not often achieve these goals. The process of collecting data still requires
money, time, access, and storage, which inherently gives the advantage and
priority to governments and corporations with access to those resources. Many
of the resources for performing academic research likewise originate from
powerful governments or corporations. The academic visualization focus on
esoteric or otherwise complex datasets means that the intended audience of a
visualization is often those with high scientific, mathematic, and visual
literacies rather than wider and more general audiences. Visualization work
should be concerned with imbalances in power, and focus on distributing power
in more equitable ways, and to more ethical ends:

We ought to support data “due process”. Citron’s (Citron, 2007) notion of a
technological due process highlights the danger that automated and data-driven
decision-making has on our norms of decision-making and procedure. Datasets
only imperfectly capture important information relevant to decsion-making, but
may then become reified by algorithms or visualizations into unappealable
assertions about the state of the world. People impacted by these systems have
the same rights as they do with other forms of decision-making, and deserve
some say in how they are (or are not) represented. Legal frameworks such as
the EU’s GDPR are beginning to codify rights such as “the right to
explanation,” “the right to privacy,” and the “right to be forgotten.”
However, these existing laws capture our intuitive notions about these rights
only imperfectly (Wachter et al., 2017). Our ethical obligations therefore may
or may not reflect the letter of the law. Compounding this issue is that many
of these algorithmic systems are of sufficient size or complexity that there
are no clear procedures for visualizing them, especially for audiences that
lack statistical expertise or extensive context about the domain. We therefore
also have design research and pedagogical responsibilities to ensure that we
are giving people agency and representation in ways that are useful and
understandable.

We ought to act as data advocates. Visualizations have rhetorical strength and
political power. Government agencies and corporations have explicit resources
dedicated to the design and publication of annual reports and data reporting.
Marginalized groups do not often have access to the same set of resources, and
so are under-represented in data-based conversations. Conversely, groups may
have financial or political interest in muddying the waters around debates of
fact such as climate change or humanitarian crises. Just as it is considered
laudatory to donate time or money to charitable causes, we should also donate
a portion of our expertise in the presentation of information to advocate or
amplify causes we believe in. This could be a relatively low-cost endeavor.
For instance, many visualization papers use a similar set of standard datasets
to illustrate or evaluate their designs. These datasets often have limited
relevance or importance (Kosara, 2018). Alternative datasets about issues of
current concern would also suffice to show that a system operates correctly,
but could increase the visibility of ongoing injustices.

We ought to pressure or slow unethical analytical behavior. In response to
abuses by U.S. Immigration and Customs Enforcement (ICE) agency, Amazon
employees circulated an internal memo asking CEO Jeff Bezos to cut ties with
the agency (Shaban, 2018). Google researcher Jack Poulson, _inter alia_ ,
resigned from Google over ethical concerns about the design of a search engine
that censors internet content in mainland China (Gallagher, 2018b; Poulson,
2018). Public resignations and dissent can surface perceived ethical lapses in
companies, but many may lack the financial or political security to engage in
such tactics. Likewise, organizational power may seek to circumvent or exclude
those with ethical concerns from decision-making: Google’s senior management
reportedly kept its censorship project secret from the internal teams that
typically engage with the ethical implications of Google’s work (Gallagher,
2018a). Even for those outside of such organizations, the very companies or
governments engaged in ethical lapses are also frequently the source of funds
and exposure for visualization research both in academia and industrial
research. In such cases where one is unwilling or unable to risk retaliation
to stop or speak out against unethical work, other options are the intentional
sabotage or slowdown of labor. Within the context of visualization work, this
could be overestimation of budgets of money and time, underestimation of
result quality, or unnecessary delays (say, for additional user testing or
ablation studies). Such sabotage may involve a conflict between _ethical_ ,
_professional_ and perhaps even _legal_ duties, and should not be undertaken
lightly.

Caveats: Conspiracy theorists, political extremists, and corporate interests
(such as the tobacco and oil industries) make use of the margins of discourse,
counter-narratives, and doubt to advance agendas that rely on the general
public discarding the opinion of experts. These bad-faith epistemologies
combined with the increasingly fractal nature of academic study has resulted
in what Nichols calls “the death of expertise” (Nichols, 2017): a ongoing and
increasing hostility to scientific and technocratic sources of knowledge.
While people and organizations collecting data may be in positions of power
compared to the general public, they might be at a further power disadvantage
compared to these organizations that do not want the public to have access to
or comprehension of particular information. The goal of promoting truth and
suppressing falsehood may require amplifying existing structures of expertise
and power, and suppressing conflicts for the sake of rhetorical impact.

##  6\. Conclusion

This work presents some of the pressing ethical considerations of
visualization work, but it functions as neither a complete survey of this
space nor an exhaustive and prescriptive decision criteria to guarantee that a
visualization was designed or deployed ethically (given the disagreements that
rational people can have over what constitutes a moral course of action, no
such criterion is likely to exist). Future work requires both developing a new
pedagogy for instilling the right values in visualization designers and
researchers as well as post-hoc studies of the ethical impact of visualization
work in the existing moral landscapes.

It is my intention that this work functions as a both a synthesis of existing
critical and ethical views of data and visualization as well as a call to
action to be mindful of the ethical implications of our work, and to cultivate
the right values and virtues in our work moving forward.

## References

  * (1)
  * Allcott and Gentzkow (2017) Hunt Allcott and Matthew Gentzkow. 2017.  Social media and fake news in the 2016 election.  _Journal of Economic Perspectives_ 31, 2 (2017), 211–36.
  * Anderson (1992) Ronald E Anderson. 1992\.  ACM code of ethics and professional conduct.  _Commun. ACM_ 35, 5 (1992), 94–99.
  * Arendt (2006) Hannah Arendt. 2006\.  _Eichmann in jerusalem_.  Penguin.
  * Ashworth et al. (2006) John Ashworth, Benny Geys, and Bruno Heyndels. 2006\.  Everyone likes a winner: An empirical test of the effect of electoral closeness on turnout in a context of expressive voting.  _Public Choice_ 128, 3-4 (2006), 383–405.
  * Association (2013) World Medical Association. 2013\.  World medical association declaration of helsinki: Ethical principles for medical research involving human subjects.  _JAMA_ 310, 20 (2013), 2191–2194.  https://doi.org/10.1001/jama.2013.281053 arXiv:/data/journals/jama/929397/jsc130006.pdf
  * Barocas and boyd (2017) Solon Barocas and danah boyd. 2017.  Engaging the ethics of data science in practice.  _Commun. ACM_ 60, 11 (2017), 23–25.
  * Bateman et al. (2010) Scott Bateman, Regan L Mandryk, Carl Gutwin, Aaron Genest, David McDine, and Christopher Brooks. 2010\.  Useful junk?: the effects of visual embellishment on comprehension and memorability of charts. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 2573–2582.
  * Baudrillard (1995) Jean Baudrillard. 1995\.  _The Gulf War did not take place_.  Indiana University Press.
  * Belia et al. (2005) Sarah Belia, Fiona Fidler, Jennifer Williams, and Geoff Cumming. 2005.  Researchers misunderstand confidence intervals and standard error bars.  _Psychological methods_ 10, 4 (2005), 389.
  * Binnig et al. (2017) Carsten Binnig, Lorenzo De Stefani, Tim Kraska, Eli Upfal, Emanuel Zgraggen, and Zheguang Zhao. 2017\.  Toward Sustainable Insights, or Why Polygamy is Bad for You.. In _Proceedings of the Conference on Innovative Data Systems Research (CIDR)_.
  * Black (2001) Edwin Black. 2001\.  _IBM and the Holocaust: The strategic alliance between Nazi Germany and America’s most powerful corporation_.  Random House Inc.
  * Bloom (2017) Paul Bloom. 2017\.  _Against empathy: The case for rational compassion_.  Random House.
  * Borkin et al. (2016) Michelle A Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. 2016.  Beyond memorability: Visualization recognition and recall.  _IEEE transactions on visualization and computer graphics_ 22, 1 (2016), 519–528.
  * Borkin et al. (2013) Michelle A Borkin, Azalea A Vo, Zoya Bylinskii, Phillip Isola, Shashank Sunkavalli, Aude Oliva, and Hanspeter Pfister. 2013\.  What makes a visualization memorable?  _IEEE Transactions on Visualization and Computer Graphics_ 19, 12 (2013), 2306–2315.
  * Boukhelifa and Duke (2009) Nadia Boukhelifa and David John Duke. 2009.  Uncertainty visualization: why might it fail?. In _CHI’09 Extended Abstracts on Human Factors in Computing Systems_. ACM, 4051–4056.
  * Boy et al. (2017) Jeremy Boy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. 2017.  Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data?. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, 5462–5474.
  * boyd (2016) danah boyd. 2016\.  Undoing the neutrality of big data.  _Florida Law Review_ 67 (2016), 226–232.
  * boyd and Crawford (2012) danah boyd and Kate Crawford. 2012.  Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon.  _Information, communication & society_ 15, 5 (2012), 662–679.
  * Bundesarchiv (1940) Bundesarchiv. 1940\.  Planung und Aufbau im Osten.  https://en.wikipedia.org/wiki/Heim_ins_Reich#/media/File:Bundesarchiv_R_49_Bild-0025,_Ausstellung_%22Planung_und_Aufbau_im_Osten%22 _Schautafel.jpg.  Photograph by M. Krajewsky.
  * Buolamwini (2018) Joy Buolamwini. 2018\.  When the Robot Doesn’t See Dark Skin.  _New York Times_ (Jun 2018).  https://www.nytimes.com/2018/06/21/opinion/facial-analysis-technology-bias.html
  * Callahan et al. (2006) Steven P Callahan, Juliana Freire, Emanuele Santos, Carlos E Scheidegger, Cláudio T Silva, and Huy T Vo. 2006.  VisTrails: visualization meets data management. In _Proceedings of the 2006 ACM SIGMOD international conference on Management of data_. ACM, 745–747.
  * Ceglowski (2014) Maciej Ceglowski. 2014\.  The Internet with a Human Face.  _talk at Beyond Tellerrand in Düsseldorf, Germany, on_ 20 (2014).
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. 2009\.  Reading tea leaves: How humans interpret topic models. In _Advances in neural information processing systems_. 288–296.
  * Citron (2007) Danielle Keats Citron. 2007\.  Technological due process.  _Wash. UL Rev._ 85 (2007), 1249.
  * Correll and Gleicher (2014) Michael Correll and Michael Gleicher. 2014.  Error bars considered harmful: Exploring alternate encodings for mean and error.  _IEEE transactions on visualization and computer graphics_ 20, 12 (2014), 2142–2151.
  * Correll and Heer (2017) Michael Correll and Jeffrey Heer. 2017.  Black hat visualization. In _Workshop on Dealing with Cognitive Biases in Visualisations (DECISIVe), IEEE VIS_.
  * Crampton (2011) Jeremy W Crampton. 2011\.  _Mapping: A critical introduction to cartography and GIS_. Vol. 11.  John Wiley & Sons.
  * Crampton and Krygier (2006) Jeremy W Crampton and John Krygier. 2006.  An introduction to critical cartography.  _ACME: an International E-journal for Critical Geographies_ 4, 1 (2006), 11–33.
  * Dalton and Thatcher (2014) Craig Dalton and Jim Thatcher. 2014.  What does a critical data studies look like, and why do we care? Seven points for a critical approach to ‘big data’.  _Society and Space_ 29 (2014).
  * Dasgupta and Kosara (2011) Aritra Dasgupta and Robert Kosara. 2011.  Adaptive privacy-preserving visualization using parallel coordinates.  _IEEE Transactions on Visualization and Computer Graphics_ 17, 12 (2011), 2241–2248.
  * Day (2017) Deanna Day. 2017\.  The History of Data is the History of Labor.  _The New Inquiry_ (Mar 2017).  https://thenewinquiry.com/blog/the-history-of-data-is-the-history-of-labor/
  * D’Ignazio and Klein ([n. d.]) Catherine D’Ignazio and Lauren Klein. [n. d.].  _Data Feminism_.  2018 Draft.
  * D’Ignazio and Klein (2016) Catherine D’Ignazio and Lauren F Klein. 2016.  Feminist data visualization. In _Workshop on Visualization for the Digital Humanities (VIS4DH), Baltimore. IEEE_.
  * Dillard (2003) Jesse F Dillard. 2003\.  Professional services, IBM, and the Holocaust.  _Journal of Information Systems_ 17, 2 (2003), 1–16.
  * Dimara et al. (2018) E. Dimara, G. Bailly, A. Bezerianos, and S. Franconeri. 2018\.  Mitigating the Attraction Effect with Visualizations.  _IEEE Transactions on Visualization and Computer Graphics_ (2018), 1–1.  https://doi.org/10.1109/TVCG.2018.2865233
  * Dörk et al. (2013) Marian Dörk, Patrick Feng, Christopher Collins, and Sheelagh Carpendale. 2013.  Critical InfoVis: exploring the politics of visualization. In _CHI’13 Extended Abstracts on Human Factors in Computing Systems_. ACM, 2189–2198.
  * Dragga and Voss (2001) Sam Dragga and Dan Voss. 2001.  Cruel pies: The inhumanity of technical illustrations.  _Technical communication_ 48, 3 (2001), 265–274.
  * Drucker (2012) Johanna Drucker. 2012\.  Humanistic theory and digital scholarship.  _Debates in the digital humanities_ (2012), 85–95.
  * Du Bois (1900) W.E.B. Du Bois. 1900\.  Proportion of freemen and slaves among American Negroes.  http://hdl.loc.gov/loc.pnp/ppmsca.33913.  A series of statistical charts illustrating the condition of the descendants of former African slaves now in residence in the United States of America.
  * Du Bois et al. (1926) W.E.B. Du Bois et al. 1926\.  Criteria of Negro art.  _Crisis_ 32, 6 (1926), 290–297.
  * Farley et al. (2014) Elizabeth Farley, Fiona Grady, Dean S Miller, Rory O’Connor, Howard Schneider, Michael Spikes, Constantia Constantinou, et al. 2014\.  SPJ Code of Ethics.  _The Power of Images_ (2014).
  * Feng et al. (2017) Mi Feng, Cheng Deng, Evan M Peck, and Lane Harrison. 2017\.  HindSight: Encouraging exploration through direct encoding of personal interaction history.  _IEEE transactions on visualization and computer graphics_ 23, 1 (2017), 351–360.
  * Gallagher (2018a) Ryan Gallagher. 2018a.  Google Shut Out Privacy and Security Teams From Secret China Project.  _The Intercept_ (Nov 2018).  https://theintercept.com/2018/11/29/google-china-censored-search/
  * Gallagher (2018b) Ryan Gallagher. 2018b.  Senior Google Scientist Resigns Over “Foreiture of our Values” in China.  _The Intercept_ (Sep 2018).  https://theintercept.com/2018/09/13/google-china-search-engine-employee-resigns/
  * Gelman and Loken (2013) Andrew Gelman and Eric Loken. 2013.  The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time.  _Department of Statistics, Columbia University_ (2013).
  * Gitelman (2013) Lisa Gitelman. 2013\.  _Raw data is an oxymoron_.  MIT Press.
  * Gleicher (2013) Michael Gleicher. 2013\.  Explainers: Expert explorations with crafted projections.  _IEEE Transactions on Visualization & Computer Graphics_ 12 (2013), 2042–2051.
  * Greis et al. (2017) Miriam Greis, Jessica Hullman, Michael Correll, Matthew Kay, and Orit Shaer. 2017.  Designing for Uncertainty in HCI: When Does Uncertainty Help?. In _Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems_. ACM, 593–600.
  * Haraway (1988) Donna Haraway. 1988\.  Situated knowledges: The science question in feminism and the privilege of partial perspective.  _Feminist studies_ 14, 3 (1988), 575–599.
  * Hecht et al. (2018) B. Hecht, L. Wilcox, J.P. Bigham, J. Schöning, E. Hoque, J. Ernst, Y. Bisk, L. De Russis, L. Yarosh, B. Anjum, D. Contractor, and C. Wu. 2018.  It’s Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process.  _ACM Future of Computing Blog_ (Mar 2018).  https://acm-fca.org/2018/03/29/negativeimpacts/
  * Heidegger (1954) Martin Heidegger. 1954\.  The question concerning technology.  _Technology and values: Essential readings_ 99 (1954), 113.
  * Henrich et al. (2010) Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010\.  Most people are not WEIRD.  _Nature_ 466, 7302 (2010), 29.
  * Hill et al. (2016) Rosemary Lucy Hill, Helen Kennedy, and Ysabel Gerrard. 2016\.  Visualizing junk: Big data visualizations and the need for feminist data studies.  _Journal of Communication Inquiry_ 40, 4 (2016), 331–350.
  * Hochshild (1983) Arlie Russell Hochshild. 1983\.  _The Managed Heart: Commercialization of Human Feeling_.  The University of California Press.
  * Hsu (2018) Jeremy Hsu. 2018\.  The Strava Heat Map and the End of Secrets.  _Wired_ (Jan 2018).  https://www.wired.com/story/strava-heat-map-military-bases-fitness-trackers-privacy/
  * Hullman and Diakopoulos (2011) Jessica Hullman and Nick Diakopoulos. 2011.  Visualization rhetoric: Framing effects in narrative visualization.  _IEEE transactions on visualization and computer graphics_ 17, 12 (2011), 2231–2240.
  * Hullman et al. (2018) Jessica Hullman, Xiaoli Qiao, Michael Correll, Alex Kale, and Matthew Kay. 2018.  In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation.  _IEEE transactions on visualization and computer graphics_ (2018).
  * Hursthouse (1999) Rosalind Hursthouse. 1999\.  _On virtue ethics_.  OUP Oxford.
  * Jeong (2018) Sarah Jeong. 2018\.  Bad Romance: How a cabal of authors profited by gaming Amazon’s Kindle Unlimited algorithm.  _The Verge_ (Jul 2018).  https://www.theverge.com/2018/7/16/17566276/
  * Kay et al. (2016) Matthew Kay, Tara Kola, Jessica R Hullman, and Sean A Munson. 2016\.  When (ish) is my bus?: User-centered visualizations of uncertainty in everyday, mobile predictive systems. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_. ACM, 5092–5103.
  * Kennedy et al. (2016) Helen Kennedy, Rosemary Lucy Hill, Giorgia Aiello, and William Allen. 2016.  The work that visualisation conventions do.  _Information, Communication & Society_ 19, 6 (2016), 715–735.
  * Kim et al. (2016) Been Kim, Rajiv Khanna, and Sanmi Koyejo. 2016.  Examples are not Enough, Learn to Criticize! Criticism for Interpretability. In _Advances in Neural Information Processing Systems_.
  * Kindermans et al. (2017) P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. 2017\.  The (Un)reliability of saliency methods.  _NIPS workshop on Explaining and Visualizing Deep Learning_ (2017).  arXiv:stat.ML/1711.00867
  * Kong et al. (2018) Ha-Kyung Kong, Zhicheng Liu, and Karrie Karahalios. 2018\.  Frames and Slants in Titles of Visualizations on Controversial Topics. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. ACM, 438\.
  * Kosara (2018) Robert Kosara. 2018\.  How to Get Excited About Standard Datasets.  https://eagereyes.org/blog/2018/how-to-get-excited-about-standard-datasets.
  * Kosara and Haroz (2018) Robert Kosara and Steve Haroz. 2018.  Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them. In _Proceedings of BELIV 2018: Evaluation and Beyond – Methodological Approaches for Visualization_.
  * Kosara and Mackinlay (2013) Robert Kosara and Jock Mackinlay. 2013.  Storytelling: The next step for visualization.  _Computer_ 46, 5 (2013), 44–50.
  * Lecher (2018) Colin Lecher. 2018\.  What Happens When An Algorithm Cuts Your Health Care.  _The Verge_ (Mar 2018).  https://www.theverge.com/2018/3/21/17144260/
  * Lewis and Willis (2010) David Levering Lewis and Deborah Willis. 2010.  _A small nation of people: W.E.B. Du Bois and African American portraits of progress_.  Zondervan.
  * Lipton (2016) Zachary C Lipton. 2016\.  The mythos of model interpretability.  _arXiv preprint arXiv:1606.03490_ (2016).
  * Louden (1984) Robert B Louden. 1984\.  On some vices of virtue ethics.  _American Philosophical Quarterly_ 21, 3 (1984), 227–236.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Advances in Neural Information Processing Systems_. 4765–4774.
  * Meyer and Blohm (1942) Konrad Meyer and Georg Blohm. 1942.  _Landvolk im Werden_.  Deutsche Landbuchhandlung.
  * Micallef et al. (2012) Luana Micallef, Pierre Dragicevic, and Jean-Daniel Fekete. 2012\.  Assessing the effect of visualizations on bayesian reasoning through crowdsourcing.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 12 (2012), 2536–2545.
  * Muehlenhaus (2013) Ian Muehlenhaus. 2013\.  The design and composition of persuasive maps.  _Cartography and Geographic Information Science_ 40, 5 (2013), 401–414.
  * Nichols (2017) Thomas M Nichols. 2017\.  _The death of expertise_.  Tantor Media, Incorporated.
  * North (2006) Chris North. 2006\.  Toward measuring visualization insight.  _IEEE computer graphics and applications_ 26, 3 (2006), 6–9.
  * O’Neil (2016) Cathy O’Neil. 2016\.  _Weapons of math destruction: How big data increases inequality and threatens democracy_.  Broadway Books.
  * Paine (1776) Thomas Paine. 1776\.  _Common sense_.
  * Pandey et al. (2014) Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite, and Enrico Bertini. 2014.  The persuasive power of data visualization.  _IEEE transactions on visualization and computer graphics_ 20, 12 (2014), 2211–2220.
  * Patil et al. (2018) DJ Patil, Hilary Mason, and Mike Loukides. 2018.  _Ethics and Data Science_.  O’Reilly.
  * Poulson (2018) Jack Poulson. 2018\.  I Quit Google Over Its Censored Chinese Search Engine. The Company Needs to Clarify Its Position on Human Rights.  _The Intercept_ (Sep 2018).  https://theintercept.com/2018/12/01/google-china-censorship-human-rights/
  * Pu and Kay (2018) Xiaoying Pu and Matthew Kay. 2018.  The garden of forking paths in visualization: A design space for reliable exploratory visual analytics. In _Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV)_.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  Why should i trust you?: Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. ACM, 1135–1144.
  * Richards (2003) Anne R Richards. 2003\.  Argument and authority in the visual representations of science.  _Technical Communication Quarterly_ 12, 2 (2003), 183–206.
  * Rogaway (2015) Phillip Rogaway. 2015\.  The Moral Character of Cryptographic Work.  _IACR Cryptology ePrint Archive_ 2015 (2015), 1162.
  * Ruginski et al. (2016) Ian T Ruginski, Alexander P Boone, Lace M Padilla, Le Liu, Nahal Heydari, Heidi S Kramer, Mary Hegarty, William B Thompson, Donald H House, and Sarah H Creem-Regehr. 2016.  Non-expert interpretations of hurricane forecast uncertainty visualizations.  _Spatial Cognition & Computation_ 16, 2 (2016), 154–172.
  * Segel and Heer (2010) Edward Segel and Jeffrey Heer. 2010.  Narrative visualization: Telling stories with data.  _IEEE transactions on visualization and computer graphics_ 16, 6 (2010), 1139–1148.
  * Shaban (2018) Hamza Shaban. 2018\.  Amazon employees demand company cut ties with ICE.  _Washington Post_ (June 2018).  https://www.washingtonpost.com/news/the-switch/wp/2018/06/22/amazon-employees-demand-company-cut-ties-with-ice/
  * Slobin (2014) Sarah Slobin. 2014\.  What If the Data Visualization is Actually People?  _Source_ (Apr 2014).  https://source.opennews.org/articles/what-if-data-visualization-actually-people/
  * Talbot et al. (2009) Justin Talbot, Bongshin Lee, Ashish Kapoor, and Desney S Tan. 2009\.  EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_. ACM, 1283–1292.
  * Van Wijk (2005) Jarke J Van Wijk. 2005\.  The value of visualization. In _Visualization, 2005. VIS 05. IEEE_. IEEE, 79–86.
  * Vartak et al. (2015) Manasi Vartak, Sajjadur Rahman, Samuel Madden, Aditya Parameswaran, and Neoklis Polyzotis. 2015\.  SeeDB: efficient data-driven visualization recommendations to support visual analytics.  _Proceedings of the VLDB Endowment_ 8, 13 (2015), 2182–2193.
  * Vines et al. (2017) Paul Vines, Franziska Roesner, and Tadayoshi Kohno. 2017\.  Exploring ADINT: Using Ad Targeting for Surveillance on a Budget-or-How Alice Can Buy Ads to Track Bob. In _Proceedings of the 2017 on Workshop on Privacy in the Electronic Society_. ACM, 153–164.
  * Wacharamanotham et al. (2015) Chat Wacharamanotham, Krishna Subramanian, Sarah Theres Völkel, and Jan Borchers. 2015\.  Statsplorer: Guiding novices in statistical analysis. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, 2693–2702.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017\.  Counterfactual explanations without opening the black box: Automated decisions and the GDPR.  _Harvard Journal of Law & Technology_ 32, 2 (2017).
  * Wall et al. (2017) Emily Wall, Leslie M Blaha, Lyndsey Franklin, and Alex Endert. 2017.  Warning, bias may occur: A proposed approach to detecting cognitive bias in interactive visual analytics. In _IEEE Conference on Visual Analytics Science and Technology (VAST)_.
  * Wang et al. (2018) Xumeng Wang, Wei Chen, Jia-Kai Chou, Chris Bryan, Huihua Guan, Wenlong Chen, Rusheng Pan, and Kwan-Liu Ma. 2018\.  GraphProtector: A Visual Interface for Employing and Assessing Multiple Privacy Preserving Graph Algorithms.  _IEEE transactions on visualization and computer graphics_ (2018).
  * Ward (2016) Mark Ward. 2016\.  _Deadly documents: Technical communication, organizational discourse, and the Holocaust: Lessons from the rhetorical work of everyday texts_.  Routledge.
  * Williams and Nagel (1976) Bernard AO Williams and Thomas Nagel. 1976.  Moral luck.  _Proceedings of the Aristotelian Society, Supplementary Volumes_ 50 (1976), 115–151.
  * Wines (2018) Mchael Wines. 2018\.  Why Was a Citizenship Question Put on the Census? ‘Bad Faith,’ a Judge Suggests.  _New York Times_ (Jul 2018).  https://www.nytimes.com/2018/07/10/us/citizenship-question-census.html
  * Wongsuphasawat et al. (2018) Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mané, Doug Fritz, Dilip Krishnan, Fernanda B Viégas, and Martin Wattenberg. 2018.  Visualizing dataflow graphs of deep learning models in TensorFlow.  _IEEE transactions on visualization and computer graphics_ 24, 1 (2018), 1–12.
  * Wood (2010) Denis Wood. 2010\.  _Rethinking the power of maps_.  Guilford Press.
  * Wood et al. (2012) Jo Wood, Petra Isenberg, Tobias Isenberg, Jason Dykes, Nadia Boukhelifa, and Aidan Slingsby. 2012\.  Sketchy rendering for information visualization.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 12 (2012), 2749–2758.
  * Wood et al. (2018) Jo Wood, Alexander Kachkaev, and Jason Dykes. 2018\.  Design Exposition with Literate Visualization.  _IEEE transactions on visualization and computer graphics_ (2018).
  * Zer-Aviv (2015) Mushon Zer-Aviv. 2015\.  DataViz–The UnEmpathetic Art.  https://responsibledata.io/2015/10/19/dataviz-the-unempathetic-art/.
  * Zgraggen et al. (2018) Emanuel Zgraggen, Zheguang Zhao, Robert Zeleznik, and Tim Kraska. 2018.  Investigating the Effect of the Multiple Comparisons Problem in Visual Analysis. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. ACM, 479.
  * Zhu and Porter (2002) Donghua Zhu and Alan L Porter. 2002.  Automated extraction and visualization of information for technological intelligence and forecasting.  _Technological forecasting and social change_ 69, 5 (2002), 495–506.


# Learning Multiscale Consistency for Self-supervised Electron Microscopy
Instance Segmentation

###### Abstract

Instance segmentation in electron microscopy (EM) volumes is tough due to
complex shapes and sparse annotations. Self-supervised learning helps but
still struggles with intricate visual patterns in EM. To address this, we
propose a pretraining framework that enhances multiscale consistency in EM
volumes. Our approach leverages a Siamese network architecture, integrating
both strong and weak data augmentations to effectively extract multiscale
features. We uphold voxel-level coherence by reconstructing the original input
data from these augmented instances. Furthermore, we incorporate cross-
attention mechanisms to facilitate fine-grained feature alignment between
these augmentations. Finally, we apply contrastive learning techniques across
a feature pyramid, allowing us to distill distinctive representations spanning
various scales. After pretraining on four large-scale EM datasets, our
framework significantly improves downstream tasks like neuron and mitochondria
segmentation, especially with limited finetuning data. It effectively captures
voxel and feature consistency, showing promise for learning transferable
representations for EM analysis.

Index Terms—  Self-supervised pretraining, Instance segmentation, Electron
microscopy volume

##  1 Introduction

Electron microscopy (EM) imaging plays a vital role in providing high-
resolution views of cellular tissue structures, especially in the nervous
system [1]. With the development of deep learning techniques, automatic
segmentation for EM volumes has become an indispensable tool for analyzing the
huge amount of data [2, 3]. However, existing fully supervised EM instance
segmentation methods are fragile due to a lack of sufficient annotations [4].

Self-supervised learning exploits the inherent structure and information
within data to acquire meaningful representations without the supervision of
manual annotations, which brings performance improvement in downstream tasks
[5, 6, 7]. However, existing self-supervised pretraining methods often rely on
simple and generic proxy tasks, such as image reconstruction [8, 9], which
cannot fully exploit the complex and diverse visual representations in EM
volumes.

To address this issue, we propose a multiscale consistency pretraining
framework tailored for EM volumes. The framework focuses on learning visual
representation consistency at different scales through joint training on
multiple tasks, including voxel-level reconstruction, soft feature matching,
and feature contrastive learning. To obtain robust EM visual representations,
we utilize a Siamese network architecture with a CNN or Transformer backbone
to extract features from strong and weak augmented EM volume inputs. To
maintain voxel-level consistency in the latent representations, we first
enforce the network to reconstruct the original volume from its augmented
counterparts. To achieve fine-grained feature-level consistency, we then adopt
a cross-attention mechanism for soft feature matching between strong and weak
augmentations. Moreover, we propose a contrastive learning scheme on the
feature pyramid, which utilizes a hierarchical approach to compute similarity
scores across different levels of the pyramid. In this way, we promote the
network to learn invariant features on multiple scales, thereby enhancing its
robustness to complex instance morphology in EM volumes.

Our contributions are as follows: 1) We propose a novel framework tailored for
EM volume pretraining, enforcing both voxel-level and feature-level
consistency in EM visual representations. 2) We utilize a Siamese network with
cross-attention to focus on features at different scales for fine-grained
feature matching. 3) We conduct extensive pretraining on four large-scale EM
datasets using both CNN and Transformer backbones and demonstrate the
effectiveness of our framework on downstream tasks of neuron segmentation and
mitochondria segmentation.

##  2 Methodology

Fig. 1: Our proposed method consists of two stages: pretraining and
finetuning. (a) illustrates the framework of pretraining, which aims to add
consistency constraints to both weak and strong augmented data. (b) describes
the method to extract multiscale features. (c) outlines our instance
segmentation method during the finetuning stage.

###  2.1 Siamese Network Design

To enhance the robustness of the network and improve its ability to extract
features at different scales, we feed both strong augmented and weak augmented
data into a Siamese network as shown in Figure 1(a). Strong augmentation
techniques include elastic deformation, Gaussian noise, gamma transformation,
cropping, splicing, scaling, and large-scale masking, while weak augmentation
techniques consist of simple transformations such as translation, rotation,
scaling, and flipping. We transform the original 3D pretraining data
$\mathcal{D}$ into
$\mathcal{D^{\prime}}=\\{\left(x_{w,1},x_{s,1}\right),\allowbreak\left(x_{w,2},x_{s,2}\right),\ldots,\allowbreak\left(x_{w,N},x_{s,N}\right)\\}$,
where ${x_{w,i},x_{s,i}}$ represent the weak and strong augmented inputs,
respectively, with ${x_{w,i},x_{s,i}}\in\mathbb{R}^{B\times C\times H\times
W\times D}$. Here, $B$ represents the batch size, $C$ represents the number of
channels, and $H$, $W$, and $D$ represent the dimensions of the volume. The
transformed data are then mapped into the latent space using a shared-weight
backbone, such as UNet or ViT [10], to generate features at different scales.
For example, given $x_{w,1}$ as input, the decoder of our main network
generates a series of encoded visual features
$\mathbf{W}_{1}=\\{\mathbf{w}_{1}^{1},\mathbf{w}_{1}^{2},....,\mathbf{w}_{1}^{N}\\}$,
where $\mathbf{w}_{i}^{j}\in\mathbb{R}^{B\times C_{1}\times H_{1}\times
W_{1}\times D_{1}}$. To ensure consistent output sizes, we pass the multiscale
features through an Adaptive Average Pooling layer, resulting in encoded
features with dimensions of $H_{i}=16,W_{i}=16,D_{i}=16$. Moreover, we
eliminate the skip connections in the UNet structure to stop the network from
producing degenerate solutions based on input features from the encoder [11].

###  2.2 Pretraining Method

####  2.2.1 Voxel-level Reconstruction.

Self-supervised reconstruction tasks can be used to learn visual
representations with good reconstruction performance [4]. By embedding input
data into a high-dimensional space and then reconstructing it back to the
original input, the model can learn a robust representation of the input data.
Therefore, we added a consistency reconstruction objective to the output of
the Siamese network, aimed at reconstructing the original input under strong
and weak data augmentation. The loss function of the reconstruction task can
be formulated as follows:

$\vspace{-0.2cm}\mathcal{L}_{{r}}=\frac{1}{2N}\sum_{i=1}^{N}(||\mathbf{w}_{i}^{N}-\mathbf{x}_{i}||_{2}^{2}+||\mathbf{s}_{i}^{N}-\mathbf{x}_{i}||_{2}^{2}),$
(1)

where $N$ signifies the total number of intermediate output features within
the network.

####  2.2.2 Soft Feature Matching based on Cross-attention.

Cross-attention based feature match does not require explicit feature
extraction and handles larger variations between input data [12]. Since our
Siamese network introduces complex deformations and perturbations, we
introduce a bidirectional cross-attention mechanism to match fine-grained
features. The specific technical details are as follows.

For each pair of strong-weak augmentation $(\mathbf{W_{i}},\mathbf{S_{i}})$,
we first project them into lower-dimensional embeddings using
$\allowbreak{\mathbf{W}}^{e}_{i}=mlp(\mathbf{w_{i}^{1}},\mathbf{w_{i}^{2}},....,\mathbf{w_{i}^{N}})$
and
${\allowbreak\mathbf{S}}^{e}_{i}=mlp(\mathbf{s_{i}^{1}},\mathbf{s_{i}^{2}},....,\allowbreak\mathbf{s_{i}^{N}})$.
To enhance the correlation between strong and weak augmentation tokens, we
propose to utilize a cross-attention mechanism to compute soft feature
matching between generated strong-weak augmentation tokens. Specifically, for
the $j$-th strong augmentation token embedding ${\mathbf{w}}_{i}^{j}$ in the
$i$-th strong-weak augmentation pair, we let ${\mathbf{w}}_{i}^{j}$ attend to
all weak augmentation token embeddings in ${\mathbf{S}}_{i}$. We first compute
bidirectional cross-attention,
$v_{i}^{j2k}=\text{softmax}\left[\frac{\left(Q{\mathbf{w}}_{i}^{j}\right)^{T}\left(K{\mathbf{s}}_{i}^{k}\right)}{\sqrt{d}}\right]$,$\allowbreak\omega_{i}^{j2k}=\text{softmax}\left[\frac{\left(Q{\mathbf{s}}_{i}^{j}\right)^{T}\left(K{\mathbf{w}}_{i}^{k}\right)}{\sqrt{d}}\right]$,
and then calculate their attention weights
$\mathbf{m}_{i}^{j}=\sum_{k=1}^{N}\allowbreak\left[v_{i}^{j2k}\left(V{\mathbf{s}}_{i}^{k}\right)\right]$
and
$\mathbf{n}_{i}^{j}=\sum_{k=1}^{N}\allowbreak\left[\omega_{i}^{j2k}\left(V{\mathbf{w}}_{i}^{k}\right)\right]$,
where K, Q, and V are learnable matrices. We introduce InfoNCE loss [5] to
pull the corresponding strong-weak token embeddings closer but push them away
from other token embeddings, and the resulting cross-attention loss is
calculated as follows:

$\mathcal{L}_{\text{cross}}=-\frac{1}{NB}\displaystyle\sum\limits_{i=1}^{N}\displaystyle\sum\limits_{j=1}^{B}\log\frac{e^{\mathbf{m_{j}^{i}}^{T}\cdot\mathbf{n_{j}^{i}}/\tau}}{e^{\mathbf{m_{j}^{i}}^{T}\cdot\mathbf{n_{j}^{i}}/\tau}+\sum_{k=1,k\neq
i}^{N}e^{\mathbf{m_{j}^{i}}^{T}\cdot\mathbf{n_{k}}/\tau}},$

(2)

where $\tau$ represents the temperature hyperparameter, we have set it to
0.07.

####  2.2.3 Multiscale Feature Contrastive Learning.

To further consider the importance of multiscale features in Electron
Microscopy (EM) volumes and reduce the computational overhead associated with
conventional contrastive learning, inspired by recent work like [11, 13], we
adopt a approach for Self-Supervised Learning (SSL) that minimizes distances
between feature pairs $\mathbf{w}_{i}^{j}$ and $\mathbf{s}_{i}^{j}$ at
different scales, encouraging the model to retain multiscale self-supervised
representations, as shown in Figure 1(b).

For a given feature pair $(\mathbf{w}_{i}^{j},\mathbf{s}_{i}^{j})$, we begin
by randomly selecting a single channel from each feature. Subsequently, we
employ adaptive average pooling to standardize the feature sizes, and
concatenate the features from different levels, resulting in
$\mathbf{w}_{i}^{cat}$ and $\mathbf{s}_{i}^{cat}$, and then flatten these
features to compute their similarity. To introduce some asymmetry and prevent
model collapse, as suggested in [14, 15, 16], we add an additional MLP
encoding layer to the vector produced by strong augmentation, resulting in
output vectors $\mathbf{w}_{i}^{cat}$ and $mlp(\mathbf{s}_{i}^{cat})$.
Subsequently, we compute the similarity of multiscale features from the
Siamese network using cosine similarity. We calculate the multiscale
similarity loss $\mathcal{L}_{sim}$ for the entire batch as follows:

$\mathcal{L}_{\text{sim}}=\displaystyle\sum\limits_{\begin{subarray}{c}i=1\end{subarray}}^{B}-\frac{1}{2}\left[\cos\left(\left(\mathbf{w}_{i}^{cat}\right),mlp\left(\mathbf{s}_{i}^{cat}\right)\right)\right.\left.+\cos\left(mlp\left(\mathbf{s}_{i}^{cat}\right),\left(\mathbf{w}_{i}^{cat}\right)\right)\right],$

(3)

where cos refers to cosine similarity, $N$ signifies the total number of
intermediate output features within the network, and $mlp$ denotes a fully
connected neural network.

Our pretraining method’s loss function is mainly composed of the three
aforementioned parts. Therefore, the final loss function can be described as
follows:

$\mathcal{L}_{{total}}=\alpha_{1}\mathcal{L}_{r}+\alpha_{2}\mathcal{L}_{cross}+\alpha_{3}\mathcal{L}_{sim},$
(4)

where $\alpha_{1}$, $\alpha_{2}$, and $\alpha_{3}$ represent the weighting
coefficients with values of 1, 0.1, and 0.1, respectively.

##  3 Experiments and Analysis

###  3.1 Dataset.

Our pretraining dataset comprises four large-scale electron microscopy (EM)
datasets, namely, the Full Adult Fly Brain (FAFB) [17], MitoEM [18], FIB-25
[19], and Kasthuri15 [20] datasets. These datasets originate from various
organisms, including Drosophila, mice, rats, and humans. To ensure a balanced
and diverse input for our network, we uniformly sample from these four
datasets with equal probabilities for each batch. The total size of our
pretraining dataset is approximately 400 GB.

To validate the effectiveness of our pretraining method, we conduct
experiments for neuron and mitochondria segmentation tasks in the finetuning
stage. For neuron segmentation, we utilize the CREMI A/B/C datasets [21],
training on 75 or 10 of the images and testing on the remaining 50 images. For
the mitochondria segmentation task, we adopt the MitoEM-R dataset [18],
training on the first 400 or 100 images and testing on the remaining 100
images.

###  3.2 Evaluation Metrics and Hyperparameters.

In the finetuning stage, we evaluate our pretraining method on two 3D instance
segmentation tasks: neuron segmentation and mitochondria segmentation. As
illustrated in Figure 1(c), following [22, 3], we predict the instance’s
affinity map or boundary map through the network and generate specific
instances via post-processing methods waterz [2]. We load the weights obtained
from pretraining to improve the final instance segmentation performance and
demonstrate the effectiveness of our pretraining method.

For neuron instance segmentation, we evaluate performance using the Variation
of Information (VOI) [23] and Adapted Rand (Arand) [24] metrics, while for
mitochondria segmentation, we use the average precision with an IoU threshold
of 0.75 (AP75) metric. Additionally, following [18], we provide evaluation
metrics for different sizes of mitochondria instances based on the number of
mitochondria voxels, with thresholds set at 5K and 15K for small, medium, and
large instances, respectively.

We employ the Adam optimizer with $\beta_{1}=0.9,\beta_{2}=0.999$ and set the
learning rate to $1\times 10^{-4}$ for both pretraining and finetuning.
Pretraining is conducted with a batch size of 16 on 8 NVIDIA RTX 3090s GPUs.
We randomly sample the four large-scale EM datasets with equal probability and
train the network for a total of $500$K iterations. In the finetuning phase,
we set the batch size to 4 on 2 NVIDIA RTX 3090s GPUs, and conduct experiments
on both neuron and mitochondria segmentation tasks, with $200$K iterations for
each task.

Table 1: Quantitative comparisons for neuron segmentation on the CREMI datasets. Methods | VOI $\downarrow$ | ARAND $\downarrow$
---|---|---
10 | 75 | 10 | 75
Random | 4.432 | 1.497 | 0.771 | 0.236
BYOL [25] | 2.971 | 1.445 | 0.559 | 0.229
SimSiam [26] | 2.544 | 1.413 | 0.467 | 0.213
BarlowTwins [15] | 2.605 | 1.415 | 0.546 | 0.186
PCRLv2 [11] | 2.496 | 1.316 | 0.460 | 0.196
Ours | 2.369 | 1.366 | 0.449 | 0.173

###  3.3 Neuron Segmentation.

We employ the ViT-base [10] model as an encoder and utilize the UNETR [27]
approach for neuron segmentation. Additionally, we have reproduced three
state-of-the-art self-supervised learning methods commonly used in natural
image analysis, namely BYOL [25], SimSiam [26], and BarlowTwins [15].
Moreover, we have incorporated the latest state-of-the-art algorithm designed
specifically for medical images, PCRLv2 [11]. The segmentation performance for
neuron detection is presented in Table 1, and visualization results are
available in Figure 2.

Our approach has demonstrated promising performance improvements in the task
of neuron segmentation based on ViT backbone networks. Table 1 presents the
average performance across the CREMI A/B/C datasets. We have observed that our
pretraining method excels when fine-tuning with a limited amount of data. For
instance, when employing the waterz method for post-processing, the VOI gain
averages 8.6% for fine-tuning with 75 images, while it impressively reaches
46.8% for fine-tuning with only 10 images. This underscores the exceptional
effectiveness of our approach in scenarios with constrained annotation
resources.

Fig. 2: Visual comparisons for neuron segmentation, where colored boxes
highlight the significant differences in the segmentation results.

###  3.4 Mitochondria Segmentation.

Fig. 3: Visual comparisons for mitochondria segmentation, where colored boxes
and arrows highlight the significant differences in 2D and 3D segmentation
results, respectively.

For the mitochondria segmentation task, we employed a resnet-50 as the
backbone and employed ResUNet [3] as the segmentation network, using the same
pretraining strategy as for the neuron segmentation task. We fine-tuned these
models on the first 400/100 images, respectively, and tested them on the last
100 images. The mitochondria segmentation results are presented in Table 2,
and detailed visualizations are shown in Figure 3.

Our experimental findings demonstrate a significant performance improvement
when dealing with challenging instances, specifically in small-volume
mitochondria segmentation. This suggests that our pretraining approach
substantially enhances segmentation performance on challenging examples.
Moreover, in our experimental design, even with the use of only 1/4 of the
data for fine-tuning, our method nearly matches the results obtained by fine-
tuning on the entire dataset. This underscores the considerable potential of
our pretraining approach in enhancing segmentation performance on difficult
samples.

Table 2: Quantitative comparisons for mitochondria segmentation on the MitoEM-R dataset. Methods | 400 | 100
---|---|---
small | medium | large | small | medium | large
Random | 0.355 | 0.821 | 0.914 | 0.192 | 0.776 | 0.793
BYOL [25] | 0.365 | 0.825 | 0.903 | 0.212 | 0.795 | 0.783
SimSiam [26] | 0.361 | 0.819 | 0.917 | 0.223 | 0.799 | 0.816
BarlowTwins [15] | 0.377 | 0.831 | 0.923 | 0.209 | 0.814 | 0.827
PCRLv2 [11] | 0.381 | 0.853 | 0.939 | 0.329 | 0.821 | 0.856
Ours | 0.392 | 0.859 | 0.931 | 0.357 | 0.842 | 0.881

###  3.5 Ablation Study.

We present ablation experimental results regarding the multi-task constraints
in Table 3. Specifically, we fine-tune our model using the first 75 images
from the CREMI C dataset with a ViT-base backbone and observe a maximum
performance improvement of up to 34.3% compared to the baseline method.
Additionally, we note that the soft feature matching within the cross-
attention mechanism and the reconstruction task exhibit more significant
improvements in downstream segmentation performance.

Table 3: Ablation study on the multi-task pretraining. Dataset | $L_{sim}$ | $L_{r}$ | $L_{cross}$ | VOI $\downarrow$ | Arand $\downarrow$
---|---|---|---|---|---
CREMI C |  |  |  | 1.924 | 0.192
|  | ✓ | 1.779(+0.145) | 0.204(-0.012)
| ✓ |  | 1.601(+0.323) | 0.178(+0.014)
| ✓ | ✓ | 1.444(+0.480) | 0.153(+0.039)
✓ | ✓ | ✓ | 1.433(+0.491) | 0.162(+0.030)

##  4 Conclusion

In this paper, we introduce a novel pretraining framework for EM volume
analysis that leverages voxel-level reconstruction, soft feature matching
based on cross-attention, and multiscale feature contrastive learning. By
imposing consistent output constraints on the strong-weak data augmentation
inputs of a Siamese network, our framework maintains both voxel-level and
feature-level consistency in the latent representations of EM volumes. We
conduct pretraining on four large-scale EM datasets to ensure data diversity
and balance. Compared to existing self-supervised frameworks, our approach
demonstrates promising performance improvements in neuron and mitochondria
instance segmentation, particularly in cases with complex instance morphology
and limited annotations. Moreover, our framework is adaptable to both CNN and
Transformer network architectures.

## References

  * [1] Arlo Sheridan, Tri M Nguyen, Diptodip Deb, Wei-Chung Allen Lee, Stephan Saalfeld, Srinivas C Turaga, Uri Manor, and Jan Funke,  “Local shape descriptors for neuron segmentation,”  Nature Methods, pp. 1–9, 2022.
  * [2] Jan Funke, Fabian Tschopp, William Grisaitis, and et al.,  “Large scale image segmentation with structured loss based deep learning for connectome reconstruction,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 7, pp. 1669–1680, 2018.
  * [3] Mingxing Li, Chang Chen, Xiaoyu Liu, Wei Huang, Yueyi Zhang, and Zhiwei Xiong,  “Advanced deep networks for 3d mitochondria instance segmentation,”  in ISBI. IEEE, 2022, pp. 1–5.
  * [4] Wei Huang, Chang Chen, Zhiwei Xiong, Yueyi Zhang, Xuejin Chen, Xiaoyan Sun, and Feng Wu,  “Semi-supervised neuron segmentation via reinforced consistency learning,”  IEEE Transactions on Medical Imaging, vol. 41, no. 11, pp. 3016–3028, 2022.
  * [5] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick,  “Momentum contrast for unsupervised visual representation learning,”  in CVPR, 2020, pp. 9729–9738.
  * [6] Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, César Quilodrán-Casas, and Rossella Arcucci,  “Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias,”  arXiv preprint arXiv:2305.19894, 2023.
  * [7] Che Liu, Sibo Cheng, Chen Chen, Mengyun Qiao, Weitong Zhang, Anand Shah, Wenjia Bai, and Rossella Arcucci,  “M-flag: Medical vision-language pre-training with frozen language models and latent space geometry optimization,”  arXiv preprint arXiv:2307.08347, 2023.
  * [8] Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna,  “Self pre-training with masked autoencoders for medical image analysis,”  arXiv preprint arXiv:2203.05573, 2022.
  * [9] Yinda Chen, Wei Huang, Shenglong Zhou, Qi Chen, and Zhiwei Xiong,  “Self-supervised neuron segmentation with multi-agent reinforcement learning,”  in IJCAI, 2023.
  * [10] Alexey Dosovitskiy, Lucas Beyer, and et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [11] Hong-Yu Zhou, Chixiang Lu, Chaoqi Chen, Sibei Yang, and Yizhou Yu,  “A unified visual information preservation framework for self-supervised pre-training in medical image analysis,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [12] Fuying Wang, Yuyin Zhou, Shujun Wang, and et al.,  “Multi-granularity cross-modal alignment for generalized medical visual representation learning,”  in NeurIPS, 2022.
  * [13] Shentong Mo, Zhun Sun, and Chao Li,  “Multi-level contrastive learning for self-supervised vision transformers,”  in WACV, 2023, pp. 2778–2787.
  * [14] Jun Li, Che Liu, Sibo Cheng, Rossella Arcucci, and Shenda Hong,  “Frozen language model helps ecg zero-shot learning,”  in MIDL, 2023.
  * [15] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny,  “Barlow twins: Self-supervised learning via redundancy reduction,”  in ICML. PMLR, 2021, pp. 12310–12320.
  * [16] Yinda Chen, Che Liu, Wei Huang, Sibo Cheng, Rossella Arcucci, and Zhiwei Xiong,  “Generative text-guided 3d vision-language pretraining for unified medical image segmentation,”  arXiv preprint arXiv:2306.04811, 2023.
  * [17] Philipp Schlegel, Alexander S Bates, Tejal Parag, Gregory SXE Jefferis, and Davi D Bock,  “Automatic detection of synaptic partners in a whole-brain drosophila em dataset,”  Nature Methods, vol. 18, no. 8, pp. 877–884, 2021.
  * [18] Donglai Wei, Zudi Lin, Daniel Franco-Barranco, Nils Wendt, Xingyu Liu, and et al.,  “Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images,”  in MICCAI. Springer, 2020, pp. 66–76.
  * [19] Shin-ya Takemura, Yoshinori Aso, Toshihide Hige, Allan Wong, Zhiyuan Lu, C Shan Xu, Patricia K Rivlin, Harald Hess, Ting Zhao, Toufiq Parag, et al.,  “A connectome of a learning and memory center in the adult drosophila brain,”  Elife, vol. 6, pp. e26975, 2017.
  * [20] Narayanan Kasthuri, Kenneth Jeffrey Hayworth, Daniel Raimund Berger, Richard Lee Schalek, José Angel Conchello, Seymour Knowles-Barley, Dongil Lee, Amelio Vázquez-Reina, Verena Kaynig, Thouis Raymond Jones, et al.,  “Saturated reconstruction of a volume of neocortex,”  Cell, vol. 162, no. 3, pp. 648–661, 2015.
  * [21] J Funke, S Saalfeld, DD Bock, SC Turaga, and E Perlman,  “Miccai challenge on circuit reconstruction from electron microscopy images,” 2016.
  * [22] Wei Huang, Shiyu Deng, Chang Chen, Xueyang Fu, and Zhiwei Xiong,  “Learning to model pixel-embedded affinity for homogeneous instance segmentation,”  in AAAI, 2022, vol. 36, pp. 1007–1015.
  * [23] Juan Nunez-Iglesias, Ryan Kennedy, Toufiq Parag, Jianbo Shi, and Dmitri B Chklovskii,  “Machine learning of hierarchical clustering to segment 2d and 3d images,”  PloS one, vol. 8, no. 8, pp. e71715, 2013.
  * [24] Ignacio Arganda-Carreras, Srinivas C Turaga, and et al.,  “Crowdsourcing the creation of image segmentation algorithms for connectomics,”  Frontiers in Neuroanatomy, p. 142, 2015.
  * [25] Jean-Bastien Grill, Florian Strub, Florent Altché, and et al.,  “Bootstrap your own latent-a new approach to self-supervised learning,”  in NeurIPS, 2020, vol. 33, pp. 21271–21284.
  * [26] Xinlei Chen and Kaiming He,  “Exploring simple siamese representation learning,”  in CVPR, 2021, pp. 15750–15758.
  * [27] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, and et al.,  “Unetr: Transformers for 3d medical image segmentation,”  in WACV, 2022, pp. 574–584.