# Statistical Methods for Auditing the Quality of Manual Content Reviews

Xuan Yang, Andrew Smart, Daniel Theron
Google LLC
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large technology firms face the problem of moderating content on their online
platforms for compliance with laws and policies. To accomplish this at the
scale of billions of pieces of content per day, a combination of human and
machine review are necessary to label content. Subjective judgement and human
bias are of concern to both human annotated content as well as to auditors who
may be employed to evaluate the quality of such annotations in conformance
with law and/or policy. To address this concern, this paper presents a novel
application of statistical analysis methods to identify human error and these
sources of audit risk.

##  1 Background and Problem

Content moderation has been defined as ‘governance mechanisms that structure
participation in a community to facilitate cooperation and prevent abuse,’
(Gorwa et al., 2020). Online platforms may use a combination of algorithmic
and manual review methods to handle the volume of content they host (Binns et
al., 2017). Audits of such methods may be required to ensure that they conform
to regulatory, policy or other requirements (MacCarthy, 2020). During these
audits, auditors check the accuracy of the moderation decisions against
agreed-upon policies. The problem of inter-rater reliability during these
audits may result in an increased source of risk against consistency and
neutrality requirements, as human review is naturally subjective and it may be
difficult to maintain or measure consistency and equality among independent
reviewers (Keller et al., 2020; Neal, 2022). Legal risks may also arise if
reviews are not systematically equal in some contexts (Ángel Díaz & Hecht-
Felella, 2021). This paper evaluates quantitative methods to measure and to
minimize audit risks as a result of human reviews via statistical analysis.

##  2 Methodology: Prospective Analysis

Consider a scenario where there is a need to determine if certain products are
being marketed towards appropriate demographics, as some may require
additional scrutiny during internal review. During experimentation, we
obtained a synthetic data set consisting of three reviewers and their answers
regarding a nine question rubric on 1,528 products (see the data and code in
appendix). Our measure of reviewer consistency is the agreement rate: the
percentage of products for which there is complete reviewer consensus,
assuming there are multiple reviewers reviewing the same content with multiple
criteria questions.

By calculating the agreement rate, we can determine which questions reviewers
have difficulty reaching a consensus on and may represent a source of bias.
One of the approaches we can use when there are multiple rubric review
questions is Fleiss Kappa (McHugh, 2012), which is a statistical measure for
assessing the reliability of agreement between a fixed number of raters. This
is accomplished by analyzing the distribution of the reviewers’ results, and
the correlation of each rubric review question to the human labeled result.
This approach can be applied at both the question level and sampled group
level. For example, based on the Table 1, the increased level of Fleiss Kappa
represents an increased level of agreement between reviewers on each question,
thus question 3 has a systematically higher disagreement rate. This may
indicate some experimental design error for this question, and allows us to
quantify the level of dissimilarity of subject matter expert opinions, rather
than relying on conventional subjective audit reviews.

Table 1: Fleiss Kappa result for subset of sampled review group Rubric Question |  Fleiss Kappa |  Overall Fleiss Kappa
---|---|---
Question 1 | 0.649255 | 0.475072
Question 2 | 0.545445 |
Question 3 | 0.093997 |
Question 4 | 0.399111 |

We wish to determine if there is a systematic relationship between each
criterion or rubric question and the final reviewer classification. This can
be done through a chi-square test. Based on our risk tolerance and generally
accepted industry standards (Broderick, 1974), we test at the 5% alpha level,
which if met would signify that there is 95% confidence to conclude that there
is a relationship between a question’s answer and the reviewer classification.

We also wish to determine if there is a systematic difference between review
teams and the ground truth result for different products. In scenarios where
there are only two review teams to compare across, this can be done through
hypothesis testing with a t-test. In scenarios where there are more than two
comparison groups, an ANOVA test is more appropriate to avoid multiple
pairwise comparisons. This analysis allows us to determine if the difference
between sampling error rates are systematically different between groups.

##  3 Methodology: Retrospective Analysis

Consider another scenario where reviewers conduct reviews sequentially instead
of simultaneously. This may result in one reviewer being able to see previous
reviewers’ review results. There is existing research to suggest human error
exists in single blind vs double blind settings (A Tomkins, 2017) even in
simultaneous review cases. We want to evaluate whether sequential review
scenarios and their blinding effects would also affect review quality.

We can design an experiment among two groups of reviewers where one group can
see the previous reviewers’ result and the other one cannot by running an A/B
test. However, many audit scenarios might not allow for direct interdiction
into the process being audited, or may be resource constrained. In such a
scenario, a method called Difference-in-Difference may be employed, as it uses
historical data to approximate the effects of review changes (Anders
Fredriksson, 2019).

Consider two groups: pre and post review change. One of the groups has been
subject to the review change, while the other has not. One can assume that
these two groups would have behaved roughly identically were it not for the
review change, and that any systematic differences can be attributed to the
review change itself. By observing the difference in performance, we can
quantify how big of a difference the review change had in these groups’
behavior. Refer to Figure 1 in the appendix, where the solid lines represent
actual, historical trends, and the dotted line refers to a hypothetical
performance (the counterfactual) from the treatment group where no changes to
review structure had occurred. The difference between the actual vs
hypothetical performance in the treatment group represents the treatment
effect.

##  4 Conclusion

The statistical techniques presented here help auditors identify high risk
groups, question design choices, and review methods that are more likely to
result in human review error. For example, these techniques allow auditors to
identify which questions are most likely to have a high disagreement rate
(Question 3), which reviewers are most likely subject to bias (Reviewer 1),
the relationship between rubric questions and the final classification, and
the agreement rate for the overall group (0.47). These insights are unlikely
to be gleaned from traditional, more subjective audit methodologies, which
would have considered all questions equally well formulated. This paves the
way for remediation suggestions more precisely targeted to reduce human error
and sources of experimental risk (Gino & Coffman, 2021). Through these
methods, we are able to use statistical analysis and experimental design to
quantitatively and pinpoint sources of human error in audit procedures.

### URM Statement

The authors acknowledge that at least one key author of this work meets the
URM criteria of ICLR 2023 Tiny Papers Track.

## References

  * A Tomkins (2017) WD Heavlin A Tomkins, M Zhang.  Reviewer bias in single- versus double-blind peer review.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114, 2017.
  * Anders Fredriksson (2019) Gustavo Magalhães de Oliveira Anders Fredriksson.  Impact evaluation using difference-in-differences.  _RAUSP Management Journal_ , 54(4):519–532, 2019\.
  * Binns et al. (2017) Reuben Binns, Michael Veale, Max Van Kleek, and Nigel Shadbolt.  Like trainer, like bot? inheritance of bias in algorithmic content moderation.  In _Social Informatics: 9th International Conference, SocInfo 2017, Oxford, UK, September 13-15, 2017, Proceedings, Part II 9_ , pp. 405–415. Springer, 2017.
  * Broderick (1974) John C. Broderick.  Setting standards for statistical sampling in auditing.  _Contemporary Auditing Problems_ , pp. 77–84, 1974.
  * Gino & Coffman (2021) Francesca Gino and Katherine Coffman.  Unconscious bias training that works.  _Harvard Business Review_ , 2021.
  * Gorwa et al. (2020) Robert Gorwa, Reuben Binns, and Christian Katzenbach.  Algorithmic content moderation: Technical and political challenges in the automation of platform governance.  _Big Data & Society_, 7(1):2053951719897945, 2020.
  * Keller et al. (2020) Daphne Keller, Paddy Leerssen, et al.  Facts and where to find them: Empirical research on internet platforms and content moderation.  _Social media and democracy: The state of the field and prospects for reform_ , 220:224, 2020.
  * MacCarthy (2020) Mark MacCarthy.  Transparency requirements for digital social media platforms: Recommendations for policy makers and industry.  _Transatlantic Working Group_ , 2020.
  * McHugh (2012) Mary L. McHugh.  Interrater reliability: the kappa statistic.  _Biochemia Medica_ , 22(3):276–282, 2012.
  * Neal (2022) et al Neal, T. M. S.  A general model of cognitive bias in human judgment and systematic review specific to forensic mental health.  _Law and Human Behavior_ , 46(2):99–120, 2022\.
  * Ángel Díaz & Hecht-Felella (2021) By Ángel Díaz and Laura Hecht-Felella.  Double standards in social media content moderation.  _Brennan Center for Justice_ , 2021.

##  Appendix A Appendix

1) The code and data for the prospective analysis discussed in this paper:
https://github.com/xuanyang0607/openreviewpaper

2) Inter-rater reliability: is the level of consistency among individual
raters. In this paper, we measure the inter-rater reliability through Fleiss’
kappa.

3) Sample group represents the sampled products that are being reviewed during
experimentation. Rubric questions represent the review criteria that the
product is being reviewed upon.

4) Fleiss Kappa: Statistical measure for assessing the reliability of
agreement between a fixed number of raters. Fleiss Kappa is defined as:

$\kappa=\frac{\bar{p}-\bar{p_{e}}}{1-\bar{p_{e}}}$

Where $\bar{p}$ is the average reviewers’ agreement and $\bar{p_{e}}$ is the
sum of squares of the proportion of assignment to the index category.

5) Chi Square Equation: It is a statistical hypothesis testing that helps to
identify if two categorical variables have significant relationship or not.
Hypotheses:

$H_{o}$: Null hypothesis. Eg: p1 = p0

$H_{a}$: Alternative hypothesis. Eg: p1 ¡¿ p0

Null hypothesis represents the statement of a test that intend to show no
effect or no relationships between the tested variables.

Alternative hypothesis is the statement the opposite of the null hypothesis
which represents the statement that intends to show effect or relationship
between the tested variables.

$\tilde{\chi}^{2}=\frac{1}{d}\sum_{k=1}^{n}\frac{(O_{k}-E_{k})^{2}}{E_{k}}$

Where:

d represents the degrees of freedom

O represents the observed value

E represents the expected value

Comparing the value with the chi-square distribution to assess whether to
reject or fail to reject the null hypothesis statement.

6) T-test: A statistical test to determine if there is a significant
differences between two groups.

Hypotheses:

$H_{o}$: Null hypothesis.

$H_{a}$: Alternative hypothesis.

$H_{o}:\mu\geq\mu_{o}$ vs $H_{a}:\mu<\mu_{o}$ (one-tail test, lower-tail)

$H_{o}:\mu\leq\mu_{o}$ vs $H_{a}:\mu>\mu_{o}$ (one-tail test, upper-tail)

$H_{o}:\mu=\mu_{o}$ vs $H_{a}:\mu\neq\mu_{o}$ (two-tail test)

Test Statistic:

Case 1: $\sigma^{2}$ is known

$z=\frac{\overline{x}-\mu_{o}}{\frac{\sigma}{\sqrt{n}}}\sim N(0,1)$

Case 2: $\sigma^{2}$ is unknown

$t=\frac{\overline{x}-\mu_{o}}{\frac{s}{\sqrt{n}}}\sim t_{n-1}$

7) ANOVA testing: A statistical test to determine if there are significant
differences between two or more groups.

Hypotheses:

$H_{o}:\mu_{1}=\mu_{2}=...=\mu_{k}$

$H_{a}:\mu_{i}\neq\mu_{j}$ for $i\neq j$

Test Statistic:

$F_{obs}=\frac{\textrm{MSTr}}{\textrm{MSE}}\sim F_{v_{1},v_{2}}$ |  $v1=df(\textrm{SSTr})=k-1$
---|---
|  $v2=df(\textrm{SSE})=n-k$
Reject $H_{o}$ if $F_{obs}\geq F_{\alpha,v_{1},v_{2}}$ |

SSTr |  = $\sum_{i=1}^{k}\sum_{j=1}^{n_{i}}(\overline{y}_{i\cdot}-\overline{y}_{\cdot\cdot})^{2}=\sum_{i=1}^{k}\frac{1}{n_{i}}y_{i\cdot}^{2}-\frac{1}{n}y_{\cdot\cdot}^{2}$
---|---
SSE |  = $\sum_{i=1}^{k}\sum_{j=1}^{n_{i}}(y_{ij}-\overline{y}_{i\cdot})^{2}=\sum_{i=1}^{k}\sum_{j=1}^{n_{i}}y_{ij}^{2}-\sum_{i=1}^{k}\frac{y_{i\cdot}^{2}}{n_{i}}=\sum_{i=1}^{k}(n_{i}-1)s_{i}^{2}$

8) Further Analysis for Prospective Analysis:

It may also useful to extrapolate the sampling error rate to the population by
calculating the confidence interval of the error rate using the binomial
distribution. This helps auditors quantify the impact of human reviewer bias,
and identify high risk groups of products for further investigation.

Lastly, in order to help identify to what extent various factors might affect
reviewer bias, we can use both a regression model, as well as a classification
model to determine which metrics (ex.: product metadata or rubric criterion)
have the most influence on the classification results by leveraging the beta
coefficients. In general, a classification models should give similar metrics
as regression models regarding the cause of bias.

9) Figure 1 Difference in Difference approach

Figure 1: Difference in Difference approach

10) Figure 2 Methodology Flowchart

Figure 2: Methodology Flowchart


capbtabboxtable[][]

11institutetext: Vanderbilt University, Dept. of Electrical Engineering and
Computer Science22institutetext: Vanderbilt University, Dept. of Biomedical
Engineering, Nashville, TN, USA

# LIFE: A Generalizable Autodidactic Pipeline for 3D OCT-A Vessel Segmentation

Dewei Hu 11 Can Cui 11 Hao Li 11 Kathleen E. Larson 22 Yuankai K. Tao 22 Ipek
Oguz 11

###### Abstract

Optical coherence tomography (OCT) is a non-invasive imaging technique widely
used for ophthalmology. It can be extended to OCT angiography (OCT-A), which
reveals the retinal vasculature with improved contrast. Recent deep learning
algorithms produced promising vascular segmentation results; however, 3D
retinal vessel segmentation remains difficult due to the lack of manually
annotated training data. We propose a learning-based method that is only
supervised by a self-synthesized modality named local intensity fusion (LIF).
LIF is a capillary-enhanced volume computed directly from the input OCT-A. We
then construct the local intensity fusion encoder (LIFE) to map a given OCT-A
volume and its LIF counterpart to a shared latent space. The latent space of
LIFE has the same dimensions as the input data and it contains features common
to both modalities. By binarizing this latent space, we obtain a volumetric
vessel segmentation. Our method is evaluated in a human fovea OCT-A and three
zebrafish OCT-A volumes with manual labels. It yields a Dice score of 0.7736
on human data and $0.8594\pm 0.0275$ on zebrafish data, a dramatic improvement
over existing unsupervised algorithms.

###### Keywords:

OCT angiography self-supervised vessel segmentation

##  1 Introduction

Optical coherence tomography (OCT) is a non-invasive imaging technique that
provides high-resolution volumetric visualization of the retina [19]. However,
it offers poor contrast between vessels and nerve tissue layers [9]. This can
be overcome by decoupling the dynamic blood flow within vessels from
stationary nerve tissue by decorrelating multiple cross-sectional images
(B-scans) taken at the same spatial location. By computing the variance of
these repeated B-scans, we obtain an OCT angiography (OCT-A) volume that has
better visualization of retinal vasculature than traditional OCT [15]. In
contrast to other techniques such as fluorescein angiography (FA), OCT-A is
advantageous because it both provides depth-resolved information in 3D and is
free of risks related to dye leakage or potential allergic reaction [9]. OCT-A
is popular for studying various retinal pathologies [4, 14]. Recent usage of
the vascular plexus density as a disease severity indicator [12] highlights
the need for vessel segmentation in OCT-A.

Unlike magnetic resonance angiography (MRA) and computed tomography
angiography (CTA), OCT-A suffers from severe speckle noise, which induces poor
contrast and vessel discontinuity. Consequently, unsupervised vessel
segmentation approaches [3, 33, 2, 22, 28] developed for other modalities do
not translate well to OCT-A. Denoising OCT/OCT-A images has thus been an
active topic of research [24, 13, 5]. The noise is compounded in OCT-A due to
the unpredictable patterns of blood flow as well as artifacts caused by
residual registration errors, which lead to insufficient suppression of
stationary tissue. This severe noise level, coupled with the intricate detail
of the retinal capillaries, leads to a fundamental roadblock to 3D
segmentation of the retinal blood vessels: the task is too challenging for
unsupervised methods, and yet, obtaining manual segmentations to train
supervised models is prohibitively expensive. For instance, a single patch
capturing only about 5% of the whole fovea (Fig. 4f) took approximately 30
hours to manually segment. The large inter-subject variability and the vast
inter-rater variability which is inevitable in such a detailed task make the
creation of a suitably large manual training dataset intractable.

As a workaround, retinal vessel segmentation attempts have been largely
limited to 2D images with better SNR, such as the depth-projection of the
OCT-A [10]. This only produces a single 2D segmentation out of a whole 3D
volume, evidently sacrificing the 3D depth information. Similar approaches to
segment inherently 2D data such as fundus images have also been reported [17].
Recently, Liu et al. [20] proposed an unsupervised 2D vessel segmentation
method using two registered modalities from two different imaging devices.
Unfortunately, multiple scans of a single subject are not typically available
in practice. Further, the extension to 3D can be problematic due to inaccurate
volumetric registration between modalities. Zhang et al. proposed the optimal
oriented flux [18] (OOF) for 3D OCT-A segmentation [32], but, as their focus
is shape analysis, neither a detailed discussion nor any numerical evaluation
on segmentation are provided.

We propose the local intensity fusion encoder (LIFE), a self-supervised method
to segment 3D retinal vasculature from OCT-A. LIFE requires neither manual
delineation nor multiple acquisition devices. To our best knowledge, it is the
first label-free learning method with quantitative validation of 3D OCT-A
vessel segmentation. Fig. 1 summarizes the pipeline. Our novel contributions
are:

  * •

An _en-face_ denoising method for OCT-A images via local intensity fusion
(LIF) as a new modality (Sec. 2.1)

  * •

A variational auto-encoder network, the local intensity fusion encoder (LIFE),
that considers the original OCT-A images and the LIF modality to estimate the
latent space which contains the retinal vasculature (Sec. 2.2)

  * •

Quantitative and qualitative evaluation on human and zebrafish data (Sec. 3)

##  2 Methods

Figure 1: Overall pipeline. * indicates LIF and CE-LIF provide supervision for
Dn-Net and LIFE in training process respectively.

###  2.1 Local intensity fusion: LIF

Small capillaries have low intensity in OCT-A since they have slower blood
flow, and are therefore hard to distinguish from the ubiquitous speckle noise.
We exploit the similarity of vasculature between consecutive _en-face_ OCT-A
slices to improve the image quality. This local intensity fusion (LIF)
technique derives from the Joint Label Fusion [29] and related synthesis
methods [7, 27, 24].

Joint label fusion (JLF) [29] is a well-known multi-atlas label fusion method
for segmentation. In JLF, a library of $K$ atlases with known segmentations
$(\bm{X}_{k},\bm{S}_{k})$ is deformably registered to the target image
$\bm{Y}$ to obtain $(\hat{\bm{X}_{k}},\hat{\bm{S}_{k}})$. Locally varying
weight maps are computed for each atlas based on the local residual
registration error between $\bm{Y}$ and $\hat{\bm{X}_{k}}$. The weighted sum
of the $\hat{\bm{S}}_{k}$ provides the consensus segmentation $\bm{S}$ on the
target image.

JLF has been extended to joint intensity fusion (JIF), an image synthesis
method that does not require atlas segmentations. JIF has been used for lesion
in-painting [7] and cross-modality synthesis [27]. Here, we propose a JIF
variant, LIF, performing fusion between the 2D _en-face_ slices of a 3D OCT-A
volume.

Instead of an external group of atlases, for each 2D _en-face_ slice $\bm{X}$
of a 3D OCT-A volume, adjacent slices within an R-neighborhood
$\\{\bm{X}_{-R},\ldots,\bm{X}_{+R}\\}$ are regarded as our group of ’atlases’
for $\bm{X}$. Note that the atlas $X_{0}$ is the target $X$ itself,
represented as the image with a red rim in Fig. 1. We perform registration
using the _greedy_ software [31]. While closely related, we note that the
self-fusion method reported in [24] for tissue layer enhancement is not
suitable for vessel enhancement as it tends to substantially blur and distort
blood vessels [13].

|  |
---|---|---
(a) original | (b) LIF | (c) CE-LIF
Figure 2: Modalities of _en-face_ OCT-A. Large red box highlights improvement
in capillary visibility. Small red box points out a phantom vessel.

Similar to a 1-D Gaussian filter along the depth axis, LIF has a blurring
effect that improves the homogeneity of vessels without dilating their
thickness in the _en-face_ image. Further, it can also smooth the speckle
noise in the background while raising the overall intensity level, as shown in
Fig. 2a/2b. In order to make vessels stand out better, we introduce the
contrast enhanced local intensity fusion (CE-
LIF)111https://pillow.readthedocs.io/en/stable/reference/ImageEnhance.html in
Fig. 2c. However, intensity fusion of _en-face_ images sacrifices the accuracy
of vessel diameter in the depth direction. Specifically, some vessels existing
exclusively in neighboring images are inadvertently projected on the target
slice. For example, the small red box in Fig. 2 highlights a phantom vessel
caused by incorrect fusion. As a result, LIF and CE-LIF are not appropriate
for direct use in application, in spite of the desirable improvement they
offer in visibility of capillaries (e.g., large red box). In the following
section, we propose a novel method that allows us to leverage LIF as an
auxiliary modality for feature extraction during which these excessive
projections will be filtered out.

###  2.2 Cross-modality feature extraction: LIFE

Liu et al. [20] introduced an important concept for unsupervised feature
extraction. Two depth-projected 2D OCT-A images, $\bm{M}_{1}$ and
$\bm{M}_{2}$, are acquired using different devices on the same retina. If they
are well aligned, then aside from noise and difference in style, the majority
of the anatomical structure would be the same. A variational autoencoder (VAE)
is set as a pix2pix translator from $\bm{M}_{1}$ to $\bm{M}_{2}$ in which the
latent space $\bm{L}_{12}$ keeps full resolution.

$\bm{L}_{12}=f_{e}(\bm{M}_{1})\qquad\mathrm{and}\qquad\bm{M}_{2}^{{}^{\prime}}=f_{d}(\bm{L}_{12})$
(1)

If $\bm{M}_{2}$ is well reconstructed
($\bm{M}_{2}^{{}^{\prime}}\approx\bm{M}_{2}$), then the latent feature map
$\bm{L}_{12}$ can be regarded as the common features between $\bm{M}_{1}$ and
$\bm{M}_{2}$, namely, vasculature. The encoder $f_{e}$ is considered a
segmentation network (Seg-Net) and the decoder $f_{d}$ a synthesis network
(Syn-Net).

Unfortunately, this method has several drawbacks in practice. Imaging the same
retina with different devices is rarely possible even in research settings and
unrealistic in clinical practice. Furthermore, the 3D extension does not
appear straightforward due to the differences in image spacing between OCT
devices and the difficulty of volumetric registration in these very noisy
images. In contrast, we propose to use a single OCT-A volume $X$ and its LIF,
$X_{LIF}$, as the two modalities. This removes the need for multiple devices
or registration, and allows us to produce a 3D segmentation by operating on
individual _en-face_ OCT-A slices rather than a single depth-projection image.
We call the new translator network local intensity fusion encoder (LIFE).

Figure 3: Network architecture

Fig. 3 shows the network architecture. To reduce the influence of speckle
noise, we train a residual U-Net as a denoising network (Dn-Net), supervised
by LIF. For the encoder, we implement a more complex model (R2U-Net) [1] than
Liu et al. [20], supervised by CE-LIF. As the decoder we use a shallow,
residual U-Net to balance computational power and segmentation performance.
The reparameterization trick enables gradient back propagation when sampling
is involved in a deep network [16]. This sampling is achieved by
$\bm{S}=\bm{\mu}+\bm{\sigma}\cdot\bm{\epsilon}$, where
$\bm{\epsilon}\in\mathcal{N}(0,1)$, and $\bm{\mu}$ and $\bm{\sigma}$ are mean
and standard deviation of the latent space. The intensity ranges of all images
are normalized to $[0,255]$. To introduce some blurring effect, both $L_{1}$
and $L_{2}$ norm are added to the VAE loss function:

$Loss=a\sum_{i,j}|\bm{Y}(i,j)-\bm{Y}^{{}^{\prime}}(i,j)|+\frac{b}{N}\sum_{i,j}(\bm{Y}(i,j)-\bm{Y}^{{}^{\prime}}(i,j))^{2}$
(2)

where (i,j) are pixel coordinates, N is the number of pixels, $\bf{Y}$ is CE-
LIF and $\bf{Y}^{{}^{\prime}}$ is the output of Syn-Net. $a=1$ and $b=0.05$
are hyperparameters. Eq. 2 is also used as the loss for the Dn-Net, with the
LIF image as $\bf{Y}$, $a=1$ and $b=0.01$.

As discussed above, LIF enhances the appearance of blood vessels but also
introduces phantom vessels because of fusion. The set of input vessel features
$\mathcal{V}$ in $X$ will thus be a subset of $\mathcal{V}_{LIF}$ in
$X_{LIF}$. Because LIFE works to extract $\mathcal{V}\cap\mathcal{V}_{LIF}$,
the phantom features that exist only in $\mathcal{V}_{LIF}$ will be cancelled
out as long as the model is properly trained without suffering from
overfitting.

###  2.3 Experimental details

Preprocessing for motion artifact removal. Decorrelation allows OCT-A to
emphasize vessels while other tissue types get suppressed (Fig. 4a). However,
this requires the repeated OCT B-scans to be precisely aligned. Any
registration errors cause motion artifacts, such that stationary tissue is not
properly suppressed (Fig. 4b). These appear as horizontal artifacts in _en-
face_ images (Fig. 4c). We remove these artifacts by matching the histogram of
the artifact B-scan to its closest well-decorrelated neighbor (Fig. 4d).

|
---
(a) B-scan without artifact
(c) original _en-face_ image
|
---
(b) B-scan with artifact
(d) cleaned _en-face_ image
|
---
(e) whole plexus
(f) ROI
Figure 4: (a-d) Horizontal motion artifacts and their removal. (e,f) Three
manually labelled vessel plexuses. Only the branches contained within the ROI
were fully segmented; any branches outside the ROI and all other trees were
omitted for brevity.

Binarization. To binarize the latent space $L_{12}$ estimated by LIFE, we
apply the $2^{nd}$ Perona-Malik diffusion equation [26] followed by the global
Otsu threshold [25]. Any islands smaller than 30 voxels are removed.

Dataset. The OCT volumes were acquired with $2560\times 500\times 400\times 4$
pix. (spectral $\times$ lines $\times$ frames $\times$ repeated frames) [23,
6]. OCT-A is performed on motion-corrected [11] OCT volumes using singular
value decomposition. We manually crop the volume to only retain the depth
slices that contain most of the vessels near the fovea, between the ganglion
cell layer (GCL) and inner plexiform layer (IPL). Three fovea volumes are used
for training and one for testing. As the number of slices between GCL and IPL
is limited, we aggressively augment the dataset by randomly cropping and
flipping 10 windows of size $[320,320]$ for each _en-face_ image. To evaluate
on vessels differing in size, we labeled 3 interacting plexus near the fovea,
displayed in Fig. 4e. A smaller ROI ($120\times 120\times 17$) cropped in the
center (Fig. 4f) is used for numerical evaluation.

To further evaluate the method, we train and test our model on OCT-A of
zebrafish eyes, which have a simple vessel structure ideal for easy manual
labeling. This also allows us to test the generalizability of our method to
images from different species. Furthermore, the fish dataset contains stronger
speckle noise than the human data, which allows us to test the robustness of
the method to high noise. 3 volumes ($480\times 480\times 25\times 5$ each)
are labeled for testing and 5 volumes are used for training. All manual
labelling is done on ITKSnap [30].

Baseline methods. Due to the lack of labeled data, no supervised learning
method is applicable. Similar to our approach that follows the enhance +
binarize pattern, we apply Frangi’s multi-scale vesselness filter [8] and
optimally oriented flux (OOF) [32, 18] respectively to enhance the artifact-
removed original image, then use the same binarization steps described above.
We also present results using Otsu thresholding and k-means clustering.

Implementation details. All networks are trained on an NVIDIA RTX 2080TI 11GB
GPU for 50 epochs with batch size set to 2. For the first 3 epochs, the entire
network uses the same Adam optimizer with learning rate of 0.001. After that,
LIFE and decoder are separately optimized with starting learning rates of
0.002 and 0.0001 respectively in order to distribute more workload on the
LIFE. Both networks decay every 3 epochs with at a rate of 0.5.

|  |  |
---|---|---|---
human input | human latent | fish input | fish latent
Figure 5: The latent image $L_{12}$ from LIFE considerably improves vessel
appearance.

##  3 Results

2D slice |  |  |  |  |  |
---|---|---|---|---|---|---
3D, FN |  |  |  |  |  |
3D, FP |  |  |  |  |  |
| Manual | LIFE | Otsu | K-mean | Frangi | OOF
Figure 6: 2D slice and 3D marching cubes rendering of segmentation results on human retina, with Gaussian smoothing, $\sigma=0.70$. Red, green, blue show three different branches; yellow highlights false positives. LIFE is the only method that can recover the 3D structure and connectivity of the capillaries outside the largest vessels without causing excessive FP (yellow). White boxes highlight LIFE’s improved sensitivity.  |  |  |  |  |
---|---|---|---|---|---
Manual | LIFE | Otsu | k-mean | Frangi | OOF
Figure 7: segmentation result of zebrafish retina with the same rendering
setting in Fig.6.

Fig. 5 displays examples of extracted latent images. It is visually evident
that LIFE successfully highlights the vasculature. Compared with the raw
input, even delicate capillaries show improved homogeneity and separability
from the background. Fig. 6 illustrates 2D segmentation results within the
manually segmented ROI, where LIFE can be seen to have better sensitivity and
connectivity than the baseline methods. Fig. 6 also shows a 3D rendering (via
marching cubes [21]) of each method. In the middle row, we filtered out the
false positives (FP) to highlight the false negatives (FN). These omitted FP
areas are highlighted in yellow in the bottom row. It is easy to see that
these FPs are often distributed along horizontal lines, caused by unresolved
motion artifacts. Hessian-based methods appear especially sensitive to motion
artifacts and noise; hence Frangi’s method and OOF introduce excessive FP.
Clearly, LIFE achieves the best preservation in small capillaries, such as the
areas highlighted in white boxes, without introducing too many FPs.

Fig. 7 shows that LIFE has superior performance on the zebrafish data. The
white boxes highlight that only LIFE can capture smaller branches.

Fig. 8 shows quantitative evaluation across B-scans, and Table 1 across the
whole volume. Consistent with our qualitative assessments, LIFE significantly
($p\ll 0.05$) and dramatically (over 0.20 Dice gain) outperforms the baseline
methods on both human and fish data.

Finally, we directly binarize LIF and CE-LIF as additional baselines. The Dice
scores on the human data are 0.5293 and 0.4892, well below LIFE (0.7736).

Figure 8: Quantitative result evaluation for (left) human and (right)
zebrafish data. TPR: true positive rate, FPR: false positive rate, Acc:
accuracy.

Algorithm | TPR | FPR | Accuracy | Dice
---|---|---|---|---
Human | Fish | Human | Fish | Human | Fish | Human | Fish
k-means | 0.3633 | 0.4167 | $\bm{0.0042}$ | 0.0303 | 0.9440 | 0.9228 | 0.5152 | 0.6249
Otsu | 0.4403 | 0.4356 | 0.0076 | 0.0346 | 0.9472 | 0.9191 | 0.5772 | 0.6399
Frangi+bin | 0.4900 | 0.2117 | 0.0419 | 0.0002 | 0.9198 | $\bm{0.9489}$ | 0.5002 | 0.4212
OOF+bin | $\bm{0.6826}$ | 0.3775 | 0.0748 | 0.0165 | 0.9053 | 0.9350 | 0.5414 | 0.6247
LIFE+bin | 0.6613 | $\bm{0.4999}$ | 0.0104 | $\bm{0.0153}$ | $\bm{0.9627}$ | 0.9386 | $\bm{0.7736}$ | $\bm{0.8594}$

Table 1: Quantitative evaluation of human and zebrafish segmentation. TPR:
true positive rate, FPR: false positive rate. Bold indicates the best score
per column.

##  4 Discussion and Conclusion

We proposed a method for 3D segmentation of fovea vessels and capillaries from
OCT-A volumes that requires neither manual annotation nor multiple image
acquisitions to train. The introduction of the LIF modality brings many
benefits for the method. Since LIF is directly computed from the input data,
no inter-volume registration is needed between the two modalities input to
LIFE. Further, rather than purely depending on image intensity, LIF exploits
local structural information to enhance small features like capillaries.
Still, there are some disadvantages to overcome in future research. For
instance, LIFE cannot directly provide a binarized output and hence the crude
thresholding method used for binarization influences the segmentation
performance.

#### Acknowledgements.

This work is supported by NIH R01EY031769, NIH R01EY030490 and Vanderbilt
University Discovery Grant Program.

## References

  * [1] Alom, M.Z., Hasan, M., Yakopcic, C., Taha, T.M., Asari, V.K.: Recurrent residual CNN based on u-net (r2u-net) for medical image segmentation. arXiv preprint arXiv:1802.06955 (2018)
  * [2] Aylward, S.R., Bullitt, E.: Initialization, noise, singularities, and scale in height ridge traversal for tubular object centerline extraction. IEEE Transactions on Medical Imaging 21(2), 61–75 (2002). https://doi.org/10.1109/42.993126
  * [3] Bozkurt, F., Köse, C., Sarı, A.: A texture-based 3d region growing approach for segmentation of ica through the skull base in cta. Multimedia Tools and Applications 79(43), 33253–33278 (2020)
  * [4] Burke, T.R., Chu, C.J., Salvatore, S., Bailey, C., Dick, A.D., Lee, R.W., Ross, A.H., Carreño, E.: Application of oct-angiography to characterise the evolution of chorioretinal lesions in acute posterior multifocal placoid pigment epitheliopathy. Eye 31(10), 1399–1408 (2017)
  * [5] Devalla, S.K., Subramanian, G., Pham, T.H., Wang, X., Perera, S., Tun, T.A., Aung, T., Schmetterer, L., Thiéry, A.H., Girard, M.J.: A deep learning approach to denoise OCT images of the optic nerve head. Scientific reports 9(1), 1–13 (2019)
  * [6] El-Haddad, M.T., Bozic, I., Tao, Y.K.: Spectrally encoded coherence tomography and reflectometry: Simultaneous en face and cross-sectional imaging at 2 gigapixels per second. Journal of biophotonics 11(4), e201700268 (2018)
  * [7] Fleishman, G.M., Valcarcel, A., Pham, D.L., Roy, S., Calabresi, P.A., Yushkevich, P., Shinohara, R.T., Oguz, I.: Joint intensity fusion image synthesis applied to MS lesion segmentation. In: MICCAI BrainLes Workshop. pp. 43–54 (2017)
  * [8] Frangi, A.F., Niessen, W.J., Vincken, K.L., Viergever, M.A.: Multiscale vessel enhancement filtering. In: MICCAI. pp. 130–137. Springer (1998)
  * [9] Gao, S., Jia, Y., Zhang, M., Su, J., Liu, G., Hwang, T., Bailey, S., Huang, D.: Optical coherence tomography angiography. IOVS 57(9), OCT27–OCT36 (2016)
  * [10] Giarratano, Y., Bianchi, E., Gray, C., Morris, A., MacGillivray, T., Dhillon, B., Bernabeu, M.O.: Automated and network structure preserving segmentation of optical coherence tomography angiograms. arXiv preprint arXiv:1912.09978 (2019)
  * [11] Guizar-Sicairos, M., Thurman, S.T., Fienup, J.R.: Efficient subpixel image registration algorithms. Optics letters 33(2), 156–158 (2008)
  * [12] Holló, G.: Comparison of peripapillary oct angiography vessel density and retinal nerve fiber layer thickness measurements for their ability to detect progression in glaucoma. Journal of glaucoma 27(3), 302–305 (2018)
  * [13] Hu, D., Malone, J., Atay, Y., Tao, Y., Oguz, I.: Retinal OCT Denoising with Pseudo-Multimodal Fusion Network. In: MICCAI OMIA. pp. 125–135 (2020)
  * [14] Ishibazawa, A., Nagaoka, T., Takahashi, A., Omae, T., Tani, T., Sogawa, K., Yokota, H., Yoshida, A.: OCT angiography in diabetic retinopathy: a prospective pilot study. American journal of ophthalmology 160(1), 35–44 (2015)
  * [15] Jia, Y., Tan, O., Tokayer, J., Potsaid, B., Wang, Y., Liu, J.J., Kraus, M.F., Subhash, H., Fujimoto, J.G., Hornegger, J., et al.: Split-spectrum amplitude-decorrelation angiography with optical coherence tomography. Optics express 20(4), 4710–4725 (2012)
  * [16] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [17] Lahiri, A., Roy, A.G., Sheet, D., Biswas, P.K.: Deep neural ensemble for retinal vessel segmentation in fundus images towards achieving label-free angiography. In: IEEE EMBC. pp. 1340–1343. IEEE (2016)
  * [18] Law, M.W., Chung, A.C.: Three dimensional curvilinear structure detection using optimally oriented flux. In: European conference on computer vision. pp. 368–382. Springer (2008)
  * [19] Li, M., Idoughi, R., Choudhury, B., Heidrich, W.: Statistical model for oct image denoising. Biomedical Optics Express 8(9), 3903–3917 (2017)
  * [20] Liu, Y., Zuo, L., Carass, A., He, Y., Filippatou, A., Solomon, S.D., Saidha, S., Calabresi, P.A., Prince, J.L.: Variational intensity cross channel encoder for unsupervised vessel segmentation on oct angiography. In: SPIE Medical Imaging 2020: Image Processing. vol. 11313, p. 113130Y (2020)
  * [21] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. SIGGRAPH Comput. Graph. 21(4), 163–169 (Aug 1987). https://doi.org/10.1145/37402.37422, https://doi.org/10.1145/37402.37422
  * [22] Lorigo, L.M., Faugeras, O.D., Grimson, W.E., Keriven, R., Kikinis, R., Nabavi, A., Westin, C.F.: CURVES: curve evolution for vessel segmentation. Medical Image Analysis 5(3), 195–206 (Sep 2001)
  * [23] Malone, J.D., El-Haddad, M.T., Yerramreddy, S.S., Oguz, I., Tao, Y.K.: Handheld spectrally encoded coherence tomography and reflectometry for motion-corrected ophthalmic OCT and OCT-A. Neurophotonics 6(4), 041102 (2019)
  * [24] Oguz, I., Malone, J.D., Atay, Y., Tao, Y.K.: Self-fusion for OCT noise reduction. In: SPIE Medical Imaging 2020: Image Processing. vol. 11313, p. 113130C (2020)
  * [25] Otsu, N.: A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics 9(1), 62–66 (1979)
  * [26] Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. on pattern analysis and machine intelligence 12(7), 629–639 (1990)
  * [27] Ufford, K., Vandekar, S., Oguz, I.: Joint intensity fusion with normalized cross-correlation metric for cross-modality MRI synthesis. In: SPIE Medical Imaging 2020: Image Processing. vol. 11313 (2020)
  * [28] Vasilevskiy, A., Siddiqi, K.: Flux maximizing geometric flows. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 1565–1578 (2001)
  * [29] Wang, H., Suh, J.W., Das, S.R., Pluta, J.B., Craige, C., Yushkevich, P.A.: Multi-atlas segmentation with joint label fusion. IEEE PAMI 35(3), 611–623 (2012)
  * [30] Yushkevich, P.A., Piven, J., Cody Hazlett, H., Gimpel Smith, R., Ho, S., Gee, J.C., Gerig, G.: User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage 31(3), 1116–1128 (2006)
  * [31] Yushkevich, P.A., Pluta, J., Wang, H., Wisse, L.E., Das, S., Wolk, D.: Fast automatic segmentation of hippocampal subfields and medial temporal lobe subregions in 3T and 7T T2-weighted MRI. Alzheimer’s & Dementia 7(12), P126–P127 (2016)
  * [32] Zhang, J., Qiao, Y., Sarabi, M.S., Khansari, M.M., Gahm, J.K., Kashani, A.H., Shi, Y.: 3d shape modeling and analysis of retinal microvasculature in oct-angiography images. IEEE TMI 39(5), 1335–1346 (2020)
  * [33] Zhao, S., Tian, Y., Wang, X., Xu, P., Deng, Q., Zhou, M.: Vascular extraction using mra statistics and gradient information. Mathematical Problems in Engineering 2018 (2018)