DNA-Graphene Interactions During Translocation Through Nanogaps

Hiral N. Patel1, Ian Carroll1, Rodolfo Lopez, Jr.1, Sandeep Sankararaman1,
Charles Etienne1, Subba Ramaiah Kodigala1, Mark R. Paul2, Henk W.Ch. Postma1,*

1 Department of Physics and Astronomy, California State University Northridge,
Northridge, California, United States of America

2 Department of Mechanical Engineering, Virginia Polytechnic Institute and
State University, Blacksburg, Virginia, United States of America

*<EMAIL_ADDRESS>
This manuscript is published in PLOS One, see
doi.org/10.1371/journal.pone.0171505

## Abstract

We study how double-stranded DNA translocates through graphene nanogaps.
Nanogaps are fabricated with a novel capillary-force induced graphene nanogap
formation technique. DNA translocation signatures for nanogaps are
qualitatively different from those obtained with circular nanopores, owing to
the distinct shape of the gaps discussed here. Translocation time and
conductance values vary by $\sim 100$%, which we suggest are caused by local
gap width variations. We also observe exponentially relaxing current traces.
We suggest that slow relaxation of the graphene membrane following DNA
translocation may be responsible. We conclude that DNA-graphene interactions
are important, and need to be considered for graphene-nanogap based devices.
This work further opens up new avenues for direct read of single molecule
activitities, and possibly sequencing.

## Introduction

Solid-state and biological nanopores hold great promise as analytical single-
molecule tools [1]. They enable study of folding dynamics [2], enzyme activity
[3], direct detection of DNA knots [4], and detection of single-nucleotide
polymorphisms [5]. They may even enable direct-read single-molecule sequencing
[6]. Solid-state nanopores may be fabricated with a focused ion beam [7],
atomic force microscope [8], transmission electron microscope [9], or
dielectric breakdown [10, 11, 12]. Graphene has especially advantageous
properties as a material for nanopore studies [13, 14, 15]. First results for
the MiniIon nanopore sequencer are promising, but show a relatively high error
rate [16]. While this may be improved by repeated sequencing of identical
molecules, this means there is still an unmet need in single-molecule de novo
sequencing. Graphene nanogaps are a promising candidate for such a sequencing
device [17, 18, 19, 20].

Here, we present the first studies of double-stranded DNA (dsDNA)
translocating through graphene nanogaps. The translocation signatures differ
significantly from those found in other solid-state and biological nanopores,
owing to the unique properties of these graphene nanogaps, and unique DNA-
graphene nanogap interactions. Our nanogap formation procedure is based on
capillary-force-induced breaking that can be controlled down to a few
nanometers.

## Methods

We fabricate graphene nanogaps and demonstrate that DNA is able to translocate
through them (Fig 1).

Fig 1:  Fabrication procedure for the graphene nanogap devices studied here.
(A) Diagram of graphene placed by wedge transfer on top of prefabricated
trench in SiO2 substrate. A nanogap with width $w$ is formed. A micropore is
fabricated by electron-beam lithography over the nanogap location. (B) Optical
microscope image of a graphene nanogap device with micropore at the same
fabrication stage as Fig 1A, placed on an etched trench (yellow) in a $300$ nm
thick SiO2 substrate (purple). Contrast has been enhanced. The fabricated
micropore changes the color of the SiO2 substrate and trench, making the
trench appear brighter yellow and the fraction of uncovered unetched SiO2 more
purple. The image has 50% increased contrast and 30% reduced brightness. (C)
Diagram of a graphene nanogap transferred to a measurement device with a $\sim
25\;\upmu\mathrm{m}$ hole that the nanogap/micropore assembly is centered on.
(D) Optical microscope image with enhanced contrast of a graphene nanogap
device ready for transverse conductance measurements at the fabrication stage
shown in Fig 1C.

Graphene is deposited on Si/SiO2 wafers as described before [21]. A $5-10$ mm
piece of a graphenium flake (manufacturer: NGS Naturgraphit GmbH) is
mechanically exfoliated with Blue Nitto tape (manufacturer: Nitto Denko, SPV
224LB-PE). The flakes are deposited on a Si wafer with a thermally-grown $285$
nm thick SiO2 layer that has been treated with a $5$ min long O2 plasma to
promote graphene adhesion. A single-layer graphene sheet is identified by its
contrast in an optical microscope.

Si/SiO2 wafers with trenches and buried metal alignment markers are fabricated
using electron-beam lithography (EBL), processing, and etching. Briefly, PMMA
is patterned by writing with an electron beam and subsequent development in a
1:3 mixture of Methyl Isobutyl Ketone (MIBK) and Isopropyl Alcohol (IPA). We
thereby created a $20\times 20\;\upmu\mathrm{m}^{2}$ square mask, which we
etch for $30$ s in buffered oxide etchant (BHF, manufacturer: J.T. Baker). A
$25$ nm thin layer of Au is evaporated into the squares after a $3$ nm Cr
adhesion layer, followed by liftoff in $80\;^{\circ}$C acetone. A second layer
of PMMA resist is applied, followed by EBL-definition of $1\times
100\;\upmu\mathrm{m}^{2}$ trenches and development. The trenches are made
$\sim 150$ nm deep by subsequent etching in BHF.

Exfoliated single-layer graphene sheets are transferred onto the trenches
using a wedge-transfer technique (Fig 1A,B) developed by Schneider et al.[21].
Graphene sheets are covered in cellulose acetate butyrate (CAB) in ethyl
acetate (EtAc) and wedged by water. They are positioned over the trench, after
which the water level is lowered. The CAB membrane is dissolved in EtAc, and
the solvent is exchanged for IPA, after which the sample is dried with N2 gas.
The capillary forces of the drying IPA break the graphene sheet inside of the
trench, thus forming a graphene nanogap [22]. The size of the gap is
determined by the balance between the bending energy of the graphene sheet and
its adhesion to the underlying SiO2 substrate, and can be controlled down to a
few nanometers, as we describe below.

To create a narrow channel to guide the DNA to the nanogap, and to block any
other potential holes that may have been accidentally created, a
$\sim\upmu\mathrm{m}$-size pore in PMMA, “micropore”, is fabricated on top of
the graphene nanogap (Fig 1A,B). The coordinates of the graphene nanogap with
respect to the previously defined Au square markers are determined in the
optical microscope. A PMMA film is deposited, and the micropore is exposed
over the graphene nanogap using custom-written EBL software and development
(Fig 1A,B). The PMMA film containing the micropore over the graphene nanogap
is then transferred to a prefabricated hole in a SiN membrane (Fig 1C,D).

The assembly is mounted in a fluid cell with a saline solution of $10$ mM Tris
acetate, $1$ mM Ethylenediaminetetraacetic acid (EDTA) and $0.1$ M KCl.
Ag/AgCl electrodes are mounted on either side, and a $\sim 20\mbox{ mV}$ bias
voltage is applied. The nanogaps used here can be expected to have a larger
cross section and a corresponding larger conductance than the graphene
nanopores reported earlier [13, 14, 15]. In order to avoid saturating our
current amplifier, we used a lower salt concentration than the $1-3$ M in
previously-reported graphene nanopore studies [13, 14, 15]. The ion current is
amplified with a patch-clamp amplifier (Axopatch 200B) and recorded at a
sample rate of $1.25$ MHz with custom software [23].

Double-stranded DNA ($\uplambda$-DNA, $48$ kbp long, Promega) is introduced on
the cis side, and upon translocation through the nanogap, causes abrupt
changes in ion-current.

### Nanogap Size Calculations

We calculate the expected size of the nanogap by balancing the bending energy
cost $E_{B}$ with a surface binding energy gain $E_{S}$ (Fig 2). Graphene
adheres well to SiO2 surfaces, but for a part of the graphene sheet to adhere
to the underlying surface, the sheet has to be bent. The bending energy is
stored in the bent portion of the graphene sheet between the support and the
point of contact with the bottom of the trench,

$E_{B}=\frac{EI}{2}\int_{x}\left[\left(\frac{\partial x}{\partial
l}\right)^{2}\frac{\partial^{2}y}{\partial x^{2}}+\frac{\partial y}{\partial
x}\frac{\partial^{2}x}{\partial l^{2}}\right]^{2}\frac{\partial x}{\partial
l}\mathrm{d}x\qquad,$

where $E=1$ TPa is the Young’s modulus [24], $I=gt^{3}/12$ is the moment of
inertia, $t=0.34$ nm is graphene’s thickness, and $g$ is the width of the
graphene sheet. At equilibrium, the bending energy is equal to the surface
binding energy $E_{S}=\varepsilon gs$, where $s$ is the length of the graphene
sheet that is adhering to the SiO2 substrate, and $\varepsilon=E_{S}/A$ is the
surface adhesion energy. We model the nanogap size for both the SiO2 adhesion
energy of $\varepsilon=2.81$ eV/nm2 of graphene to SiO2 [25] and the graphene
exfoliation energy $\varepsilon=0.50\;\mathrm{eV/nm}$2 [26]. We numerically
solve these equations to deduce the nanogap width as a function of suspended
length $L$ (Fig 2B).

Fig 2:  Model for breaking a graphene sheet. (A) Breaking a graphene sheet of
thickness $t$ over a trench of width $L$ and depth $h$. The graphene shape is
estimated by balancing the bending energy $E_{B}$ up to the point $p$ with the
surface binding energy $E_{S}$, leading to a nanogap width $w$. (B) Expected
nanogap width $w$ as a function of suspended length $L$ for two values of
surface adhesion energy $\varepsilon$ (see text).

In order to break the graphene sheet, we need to exceed the yield strength of
graphene. In the experiments presented here, we broke the sheet in a trench
with $h=150$ nm and $L=1\;\upmu\mathrm{m}$. We numerically deduce $\Delta
L/L=49\%$ in this geometry, well above the failure strain of graphene of
$25\%$ [27]. Upon wedge transfer to the final device for the translocation
studies presented here, we release it from the SiO2 surface. Therefore, the
graphene sheet can be expected to relax and $w$ to become smaller.

### Scanning Electron Microscope Nonlinear Correction

The scanning electron microscope (JEOL JSM-840) is used to make a micropore in
the PMMA above the graphene sheet. In order to locate the sheet with high
accuracy, we calibrated the SEM beam deflection as a function of applied
deflector voltage. A pattern of prefabricated Au square markers was imaged in
an optical microscope (Fig 3A) and the SEM (Fig 3B). The $x$ and $y$
coordinates of markers in both optical and SEM images are recorded as complex
numbers $z=x+\imath y$ (red circles). A linear fit of $z_{\mbox{\tiny SEM}}$
vs $z_{\mbox{\tiny OPT}}$ yields a best fit linear transform $z_{\mbox{\tiny
SEM}}=az_{\mbox{\tiny OPT}}+b$. This fit simultaneously records offset $b$,
scale $|a|$, and rotation $\arg(a)$. The error after the fit $z_{\mbox{\tiny
SEM}}-az_{\mbox{\tiny OPT}}-b$ is analyzed (Fig 3C,D). A small error remains
that is linearly proportional to the $x$ and $y$ SEM coordinates. The
correction is quadrupolar in nature; it is positive in the $x$ direction, and
negative in the $y$ direction. We assume the most significant contribution to
it is due to the electron optics in the SEM. The error can amount to several
$\upmu\mathrm{m}$, and neglecting it would cause the micropore to not be
located exactly over the trench. This quadrupolar correction was applied to
our SEM micropore fabrication procedure.

Fig 3:  Nonlinear correction of electron-beam deflection. (A) Optical image of
Au alignment markers and selected markers with enhanced contrast for
coordinate analysis (red circles). (B) SEM image of same device as A and
selected markers with enhanced contrast (red circles). (C) Residual error in
$x$ coordinate of SEM beam deflection (black circles) that scales linearly
with the SEM $x$ coordinate (red line). (D) Residual error (black circles) in
SEM $y$ coordinate scaling linearly with the SEM $y$ coordinate (red line).

### Control Experiments, Data Processing, and Event Detection

Control current traces are recorded before introduction of DNA, and traces
with DNA are recorded as described above (Fig 4A). Translocation events are
identified and analyzed in a three-step process as follows. First, the raw
current traces are FFT filtered to remove spurious interference from
instrumentation and environment (Fig 4B). Next, a slowly varying background is
extracted from the signal by computing a running average using a one-second
length. This background is subtracted from the signal, and the RMS value of
the trace $\sigma$ is calculated. Candidate events are determined as either
positive or negative peaks exceeding $5\sigma$. The area under the peak is
calculated to determine the statistical significance of the candidate event.
The threshold for calling it an ‘event’ is adjusted until the control
experiments do not yield false positives anymore. We include particularly
noisy control traces such as the black trace in Fig 4A to make sure that
obviously-apparent intereference and spikes and dips do not yield any false
positives either. Short dips such as that marked $\times$ in Fig 4A and Fig 4B
are thereby excluded. Third, the unfiltered candidate events are fitted by a
least squares method to both an exponentially decaying function and a
rectangular function. We fit the unfiltered data to avoid distortions to the
event shape by the FFT filtering and background subtraction. The squared
difference between fit and data for both is used to determine whether the
event better fits a rectangular or exponential event, and it is classified as
such. The resulting fits are recorded and zoomed plots are generated (Fig 4C).
All fits are visually confirmed.

Fig 4:  Event detection and classification. (A) Raw current traces are
recorded as described without DNA (black) and with DNA (red, blue). Eventual
determination of rectangular and exponential events are indicated by blue
square and red triangle, respectively. A dip in the control experiment
($\times$) is discarded as a possible event as described in the text. (B)
Narrow-band noise is removed from the traces and eventual event position is
indicated. (C) Events are fitted with rectangular (blue) or exponential (red)
traces depending on which fits better statistically.

## Results

Upon introduction of dsDNA on the cis side of the chamber and application of a
bias voltage between the two Ag/AgCl electrodes, brief changes in the ion
current are observed (Fig 5A). These events only occur after introduction of
the dsDNA, we therefore attribute them to translocation of dsDNA through the
graphene nanogap.

Fig 5:  Recorded translocation of dsDNA through graphene nanogap with
resistance of $\sim 36\;\mathrm{M}\Omega$. (A) Typical rectangular (blue, top)
and exponential (red, bottom) translocation event. (B) Analysis of rectangular
(blue squares) and exponential translocation events (red triangles, pointing
up for $\Delta G>0$ and down for $\Delta G<0$). The solid lines are fits of
the data to $\Delta G\propto 1/\tau$. The error bars indicate the range of
geometrical standard deviation.

We observe two types of events. Firstly, we observe rectangular events, where
the current briefly changes from the baseline current (Fig 5A, top). The
average change in conductance during such a blockade event, $\Delta G$, is $3$
nS, while the events have an average duration $\tau=0.7$ ms (Fig 5B). These
events have a geometric standard deviation
$\sigma_{g}(x)=\exp(\mathrm{std}(\log(x)))$ in both $\Delta G$ and $\tau$ of
$\sigma_{g}\sim 2$. For all rectangular events, we find that the conductance
increases during the event, i.e. $\Delta G>0$.

Secondly, we observe short decay events, where the current rapidly changes
from the baseline current and relaxes exponentially back with a decay constant
$\tau$ ranging from $\sim 1$ to $100$ ms (Fig 5A, bottom). The maximum
conductance change $\Delta G$ during such exponential events is larger than
for rectangular events with an average value of $\Delta G=20$ nS. These events
have a geometric standard deviation in both $\Delta G$ and $\tau$ of
$\sigma_{g}\sim 6$. We find that during exponential events, the conductance
change can be either positive or negative, with no discernible correlation
between $\tau$ and the sign of $\Delta G$, nor between the magnitude of
$\Delta G$ and the sign of $\Delta G$ (Fig 5B). For both types of events, the
data is not following the behavior $\Delta G\propto 1/\tau$ that has been
observed for circular graphene nanopores [13, 14, 15] (solid lines), as we
find $\Delta G\propto\tau^{\alpha}$ with $\alpha=0.04\pm 0.34$ and
$\alpha=0.09\pm 0.27$ for rectangular and exponential events, respectively.

## Discussion

The rectangular events have similar $\tau$ and $\Delta G$ average values to
events commonly observed in solid-state nanopores [7, 28]. They are signatures
of DNA translocation; the DNA temporarily occupies the nanopore and
consequently changes the ion current through the pore.

We attribute the positive nature of these translocation events ($\Delta G>0$)
to the low salt concentration that we employed in these experiments. In Si-
based semiconducting nanopores, a crossover between $\Delta G>0$ to $\Delta
G<0$ occurs at a salt concentration of $0.4\;\mathrm{M}$ [29]. At low salt
concentrations, the conductance during an event is dominated by the counterion
current along the DNA molecule, while at high concentrations, the conductance
is dominated by blocking of the pore cross section by the DNA molecule. Our
observation that $\Delta G>0$ at a concentration of $0.1$ M is consistent with
that picture and leads us to conclude that the conductance change is dominated
by counterion current along the DNA molecule.

We attribute the large range in $\tau$ and $\Delta G$ to the unique properties
of the rectangular graphene nanogaps studied here. In circular graphene
nanopores, the ion current density profile of a graphene nanopore depends on
the distance from the pore wall. The $\Delta G$ is therefore a function of
which part of the pore’s cross section is blocked by the DNA during
translocation. However, in these pores, the point of translocation through the
nanopore is approximately equal for all translocation events, and consequently
$\Delta G$ does not vary much. The typical geometric standard deviation for
graphene nanopores is $\sigma_{g}\sim 1.2$, or, equivalently, $\Delta G$ only
varies by $\sim 20$% [13, 14]. In contrast, the graphene nanogaps studied here
can be assumed to have edges that are serrated on a nanometer scale, causing
the local width at the DNA’s translocation point to vary (Fig 6A). The current
density that is blocked by the DNA during translocation therefore varies more,
and we find a larger $\sigma_{g}\sim 2$, a variation of $100$%. It was
recently reported that an increase of nanopore diameter by only a factor $2$
causes an increase in translocation rate by an order of magnitude due to
decreasing nanopore-DNA interaction strength [30]. Due to the serrated nature
of the nanogap edge, the effective width at the local point of translocation
can be expected to parametrically alter the DNA-gap interaction strength, and
thereby the translocation speed (Fig 6B,C). Since both $\Delta G$ and $\tau$
vary more than for circular graphene nanopores, the process for the events
described here is fundamentally different, and it is not surprising that the
trend $\Delta G\propto 1/\tau$ is not followed. Finally, the interaction
between the graphene edge and the DNA molecule may also be affected by the
nature of the edge, and harnessing this interaction may enhance performance of
DNA sequencing devices built on this principle [18, 31, 32].

Fig 6:  Effect of local nanogap geometry on translocation events, showing gaps
and associated event data. (A) Illustration of continuously changing gap width
over the length of the gap. As DNA translocates through the gap, its passage
is characterized by the gap width at its point of traversal. (B) DNA passes
through the upper gap, causing a rectangular conductance change. (C) DNA
passes through the middle gap more slowly due to the narrower gap diameter.
(D) DNA forces the edges of the lower gap to bend outward, which then relax to
their equilibrium position once it has passed, causing an exponential decay
event.

The exponential decay events reported here have not been reported for circular
nanopores, nor do we see them in our control experiments. They could therefore
be due to the unique geometry of the nanogaps presented here. One possible
mechanism responsible could be reversible mechanical changes to the graphene
upon interaction with DNA. In such a case, the DNA may arrive at a part of the
nanogap where the width is too small to go through. The membrane could be
forced to flex, allowing passage of the DNA molecule, after which the membrane
relaxes (Fig 6D). As the translocation event does not require a complete
unfolding of the molecule, the translocation event itself may be much shorter
than events that require unfolding of the molecule (rectangular events).
Indeed, we observe a very short exponential increase of the current, followed
by a longer exponential decay. We therefore establish an upper limit to the
unfolded translocation time of $\sim 15\;\upmu\mathrm{s}$, the temporal
resolution of the experiment.

The exponential relaxation after the DNA has passed has an average geometric
relaxation time of $50$ ms; it is surprisingly large. Micro and nanoscale
elastic objects placed in a viscous fluid can often be modeled as a simple
harmonic oscillator [33, 34]. For nanoscale elastic objects in a viscous fluid
the dynamics are essentially inertialess and can be described as an overdamped
simple harmonic oscillator which yields long relaxation times. To accurately
model this mechanical relaxation, a detailed treatment of viscoelastic drag of
counterions on the graphene sheet is required, and is beyond the scope of this
paper.

### Modeling the Slow Exponential Decay Events

We will show that the long relaxation times measured in our experiments are
consistent with the dynamics that would occur for a graphene sheet returning
to equilibrium after being initially displaced by the DNA passing through the
gap. We represent the motion of the fundamental mode of oscillation of the
graphene membrane as a simple harmonic oscillator as

$m_{f}\ddot{x}(t)+\gamma\dot{x}(t)+kx(t)=0,$ (1)

where $x$ is the displacement of the membrane, $m_{f}$ is the equivalent mass
which includes the mass of the graphene membrane and the added mass of the
fluid that is in motion ($m_{f}\gg m$ where $m$ is the mass of the graphene
alone), $\gamma$ in the viscous damping acting on the graphene by water, $k$
is the equivalent spring constant of the graphene, and $t$ is time.

For this system, it is straightforward to show that the Reynolds number of the
fluid flow is very small $\text{Re}\ll 1$ which indicates the dominance of
viscous effects over inertia. In light of this, we expect the graphene
membrane to act as an overdamped oscillator after being displaced which has
been demonstrated for nanoscale cantilevers in fluid [33].

If we assume that the initial displacement of the graphene membrane is
$x_{0}=F_{0}/k$ where $F_{0}$ is an applied force, the return to equilibrium
of the graphene membrane can be expressed as [34]

$x(t)=x_{0}\left(\frac{\lambda_{+}}{\lambda_{+}-\lambda_{-}}e^{\lambda_{-}t}+\frac{\lambda_{-}}{\lambda_{-}-\lambda_{+}}e^{\lambda_{+}t}\right)$

where

$\lambda_{\pm}=\omega_{f}\left(-\frac{1}{2Q}\pm\sqrt{\frac{1}{4Q^{2}}-1}\right)\sim-\omega_{f}Q^{\pm
1}.$

In these expressions $Q=m_{f}\omega_{f}/\gamma$ is the quality factor and
$\omega_{f}$ is the resonant frequency of the graphene membrane when immersed
in the fluid. For nanoscale oscillating objects in fluid
$\omega_{f}\ll\omega_{0}$ where $\omega_{0}$ is the resonant frequency in a
vacuum.

For small $Q$ we have $|\lambda_{-}|\gg|\lambda_{+}|$ where
$\lambda_{+}\sim-\omega_{f}Q$ and $\lambda_{-}\sim-\omega_{f}/Q$ and the
large-$t$ response is dominated by $\lambda_{+}$. The solution can then be
represented as $x(t)=x_{0}e^{\lambda_{+}t}=x_{0}e^{-t/\tau}$ where $\tau$ is
the relaxation time and is given by $\tau=(\omega_{f}Q)^{-1}$.

If we assume that the membrane dynamics are similar to that of a wide elastic
cantilever in fluid we can make the following analytical predictions to
suggest the order of magnitude of the expected response for the graphene
membrane. Using the approach described in Paul et al.[34] we can predict the
values of $Q$, $\omega_{f}$ (and therefore $\tau$) given only the values of
frequency parameter $R_{0}$, mass loading parameter $T_{0}$, and the resonant
frequency in vacuum $\omega_{0}$.

For the graphene membrane we will use a Young’s modulus of $E=1\times 10^{12}$
N/m2 and a density of $\rho_{g}=2000$ kg/m3. For water we will use a density
of $\rho_{f}=1000$ kg/m3 and a dynamic viscosity of $\mu_{f}=1\times 10^{-3}$
$\text{kg}/\text{m}\cdot\text{s}$. We next assume the membrane can be
represented as a cantilever with a length $L\approx 1\;\upmu$m, width
$W\approx L/2$, and a thickness of $H\approx 0.3$ nm. Using these values
yields an equivalent spring constant of $k\approx 3.4~{}\upmu\text{N/m}$ and a
resonant frequency in vacuum of $\omega_{0}\approx 6.8\times
10^{6}\text{rad/s}$.

These values yield a frequency parameter
$R_{0}=\rho_{f}\omega_{0}W^{2}/(4\mu_{f})=0.43$ and a mass loading parameter
of $T_{0}=\pi\rho_{f}W/(4\rho H)=655$. Given these values for $R_{0}$,
$T_{0}$, and $\omega_{0}$ the analytical expressions of Paul et al.[34] yield
$Q\approx 0.17$ and $\omega_{f}\approx 1694$ rad/s. Using this with our
expression for the decay time yields $\tau\approx 3$ ms. This is commensurate
with the long relaxation times that are measured in the experiments. We
emphasize that these estimates are approximate and a more accurate analysis
would require numerical simulations for the precise conditions of the
experiment. In addition, the predicted value of the relaxation time from our
approximate analysis is sensitive to the chosen geometry of the membrane. For
example, as the length or width of the membrane becomes larger the decay time
will increase.

## Conclusion

We have demonstrated translocation of DNA molecules through graphene nanogaps
and observed signatures that have features different from those observed with
circular solid-state nanopores. We argue that these derive from the DNA
interacting with the unique nanogap geometry and we conclude that DNA-graphene
gap interactions are important and need to be included in a realistic design
of graphene-nanogap based sequencing devices. This study opens up new
advancement in single molecule genomic screening devices, and DNA sequencing.

## Acknowledgments

We acknowledge support from the National Science Foundation under award
DMR-1034937. We thank Richard Kaner and Jaime Torres for discussions. We thank
Cees Dekker and Greg Schneider for useful discussions and their hospitality
while helping us learn their wedge-transfer technique.

## References

  * 1. Muthukumar M, Plesa C, Dekker C.  Single-molecule sensing with nanopores.  Physics Today. 2015;68(8):40–46.  doi:10.1063/PT.3.2881.
  * 2. Bundschuh R, Gerland U.  Coupled Dynamics of RNA Folding and Nanopore Translocation.  Physical Review Letters. 2005;95(20):208104.  doi:10.1103/PhysRevLett.95.208104.
  * 3. Fennouri A, Przybylski C, Pastoriza-Gallego M, Bacri L, Auvray L, Daniel R.  Single molecule detection of glycosaminoglycan hyaluronic acid oligosaccharides and depolymerization enzyme activity using a protein nanopore.  ACS nano. 2012;6(11):9672–9678.
  * 4. Plesa C, Verschueren D, Pud S, van der Torre J, Ruitenberg JW, Witteveen MJ, et al.  Direct observation of DNA knots using a solid-state nanopore.  Nature Nanotechnology. 2016;advance online publication.  doi:10.1038/nnano.2016.153.
  * 5. Zhao Q, Sigalov G, Dimitrov V, Dorvel B, Mirsaidov U, Sligar S, et al.  Detecting SNPs Using a Synthetic Nanopore.  Nano Letters. 2007;7(6):1680–1685.  doi:10.1021/nl070668c.
  * 6. Branton D, Deamer DW, Marziali A, Bayley H, Benner SA, Butler T, et al.  The potential and challenges of nanopore sequencing.  Nat Biotech. 2008;26(10):1146–1153.  doi:10.1038/nbt.1495.
  * 7. Li J, Stein D, McMullan C, Branton D, Aziz MJ, Golovchenko JA.  Ion-beam sculpting at nanometre length scales.  Nature. 2001;412(6843):166–169.  doi:10.1038/35084037.
  * 8. Held R, Heinzel T, Studerus P, Ensslin K.  Nanolithography by local anodic oxidation of metal films using an atomic force microscope.  Physica E. 1998;2(1-4).
  * 9. Storm AJ, Chen JH, Ling XS, Zandbergen HW, Dekker C.  Fabrication of solid-state nanopores with single-nanometre precision.  Nat Mater. 2003;2(8):537–540.  doi:10.1038/nmat941.
  * 10. Kwok H, Briggs K, Tabard-Cossa V.  Nanopore Fabrication by Controlled Dielectric Breakdown.  PLoS ONE. 2014;9(3):e92880.  doi:10.1371/journal.pone.0092880.
  * 11. Yanagi I, Akahori R, Hatano T, Takeda Ki.  Fabricating nanopores with diameters of sub-1 nm to 3 nm using multilevel pulse-voltage injection.  Scientific Reports. 2014;4.  doi:10.1038/srep05000.
  * 12. Kuan AT, Lu B, Xie P, Szalay T, Golovchenko JA.  Electrical pulse fabrication of graphene nanopores in electrolyte solution.  Applied Physics Letters. 2015;106(20):203109.  doi:10.1063/1.4921620.
  * 13. Schneider GF, Kowalczyk SW, Calado VE, Pandraud G, Zandbergen HW, Vandersypen LMK, et al.  DNA Translocation through Graphene Nanopores.  Nano Letters. 2010;10(8):3163–3167.  doi:10.1021/nl102069z.
  * 14. Merchant CA, Healy K, Wanunu M, Ray V, Peterman N, Bartel J, et al.  DNA Translocation through Graphene Nanopores.  Nano Letters. 2010;10(8):2915–2921.  doi:10.1021/nl101046t.
  * 15. Garaj S, Hubbard W, Reina A, Kong J, Branton D, Golovchenko JA.  Graphene as a subnanometre trans-electrode membrane.  Nature. 2010;467(7312):190–193.  doi:10.1038/nature09379.
  * 16. Jain M, Fiddes IT, Miga KH, Olsen HE, Paten B, Akeson M.  Improved data analysis for the MinION nanopore sequencer.  Nature Methods. 2015;12(4):351–356.  doi:10.1038/nmeth.3290.
  * 17. Postma HWC.  Rapid Sequencing of Individual DNA Molecules in Graphene Nanogaps.  Nano Letters. 2010;10(2):420–425.  doi:10.1021/nl9029237.
  * 18. He Y, Scheicher RH, Grigoriev A, Ahuja R, Long S, Huo Z, et al.  Enhanced DNA Sequencing Performance Through Edge-Hydrogenation of Graphene Electrodes.  Advanced Functional Materials. 2011;21(14):2674–2679.
  * 19. Prasongkit J, Grigoriev A, Pathak B, Ahuja R, Scheicher RH.  Transverse Conductance of DNA Nucleotides in a Graphene Nanogap from First Principles.  Nano Letters. 2011;11(5):1941–1945.  doi:10.1021/nl200147x.
  * 20. Heerema SJ, Dekker C.  Graphene nanodevices for DNA sequencing.  Nature Nanotechnology. 2016;11(2):127–136.  doi:10.1038/nnano.2015.307.
  * 21. Schneider GF, Calado VE, Zandbergen H, Vandersypen LMK, Dekker C.  Wedging Transfer of Nanostructures.  Nano Letters. 2010;10(5):1912–1916.  doi:10.1021/nl1008037.
  * 22. Wang HM, Zheng Z, Wang YY, Qiu JJ, Guo ZB, Shen ZX, et al.  Fabrication of graphene nanogap with crystallographically matching edges and its electron emission properties.  Applied Physics Letters. 2010;96(2):023106–023106.
  * 23. Postma HWC. DAQ: Data AcQuisition; 2012.  Available from: https://code.launchpad.net/~henkpm/daq/quickly_trunk.
  * 24. Bunch JS, van der Zande AM, Verbridge SS, Frank IW, Tanenbaum DM, Parpia JM, et al.  Electromechanical Resonators from Graphene Sheets.  Science. 2007;315(5811):490–493.  doi:10.1126/science.1136836.
  * 25. Koenig SP, Boddeti NG, Dunn ML, Bunch JS.  Ultrastrong adhesion of graphene membranes.  Nat Nano. 2011;6:543–546.  doi:10.1038/nnano.2011.123.
  * 26. Zacharia R, Ulbricht H, Hertel T.  Interlayer cohesive energy of graphite from thermal desorption of polyaromatic hydrocarbons.  Physical Review B. 2004;69(15):155406.
  * 27. Wei X, Fragneaud B, Marianetti CA, Kysar JW.  Nonlinear elastic behavior of graphene: Ab initio calculations to continuum description.  Physical Review B. 2009;80(20):205407.  doi:10.1103/PhysRevB.80.205407.
  * 28. Storm AJ, Storm C, Chen J, Zandbergen H, Joanny JF, Dekker C.  Fast DNA Translocation through a Solid-State Nanopore.  Nano Letters. 2005;5(7):1193–1197.
  * 29. Smeets RMM, Keyser UF, Krapf D, Wu MY, Dekker NH, Dekker C.  Salt Dependence of Ion Transport and DNA Translocation through Solid-State Nanopores.  Nano Letters. 2006;6(1):89–95.  doi:10.1021/nl052107w.
  * 30. Wanunu M, Sutin J, McNally B, Chow A, Meller A.  DNA Translocation Governed by Interactions with Solid-State Nanopores.  Biophysical Journal. 2008;95(10):4716–4725.  doi:10.1529/biophysj.108.140475.
  * 31. Amorim RG, Rocha AR, Scheicher RH.  Boosting DNA Recognition Sensitivity of Graphene Nanogaps through Nitrogen Edge Functionalization.  The Journal of Physical Chemistry C. 2016;doi:10.1021/acs.jpcc.6b04683.
  * 32. He H, Scheicher RH, Pandey R, Rocha† AR, , Sanvito S, et al.  Functionalized Nanopore-Embedded Electrodes for Rapid DNA Sequencing.  J Phys Chem C,. 2008;112:3456–3459.
  * 33. Paul MR, Cross MC.  Stochastic Dynamics of Nanoscale Mechanical Oscillators Immersed in a Viscous Fluid.  Physical Review Letters. 2004;92(23):235501.  doi:10.1103/PhysRevLett.92.235501.
  * 34. Paul MR, Clark MT, Cross MC.  The stochastic dynamics of micron and nanoscale elastic cantilevers in fluid: fluctuations from dissipation.  Nanotechnology. 2006;17(17):4502.  doi:10.1088/0957-4484/17/17/037.

## Supporting information

S1 File. Source data. This is the data described in the manuscript.


11institutetext: Healthcare Group, Baidu
11email<EMAIL_ADDRESS>

# Opinions Vary? Diagnosis First!

Junde Wu  Huihui Fang  Dalu Yang  Zhaowei Wang  Wenshuo Zhou  Fangxin Shang
Yehui Yang  Yanwu Xu (✉)

###### Abstract

With the advancement of deep learning techniques, an increasing number of
methods have been proposed for optic disc and cup (OD/OC) segmentation from
the fundus images. Clinically, OD/OC segmentation is often annotated by
multiple clinical experts to mitigate the personal bias. However, it is hard
to train the automated deep learning models on multiple labels. A common
practice to tackle the issue is majority vote, e.g., taking the average of
multiple labels. However such a strategy ignores the different expertness of
medical experts. Motivated by the observation that OD/OC segmentation is often
used for the glaucoma diagnosis clinically, in this paper, we propose a novel
strategy to fuse the multi-rater OD/OC segmentation labels via the glaucoma
diagnosis performance. Specifically, we assess the expertness of each rater
through an attentive glaucoma diagnosis network. For each rater, its
contribution for the diagnosis will be reflected as an expertness map. To
ensure the expertness maps are general for different glaucoma diagnosis
models, we further propose an Expertness Generator (ExpG) to eliminate the
high-frequency components in the optimization process. Based on the obtained
expertness maps, the multi-rater labels can be fused as a single ground-truth
which we dubbed as Diagnosis First Ground-truth ($DiagFirst$GT). Experimental
results show that by using $DiagFirst$GT as ground-truth, OD/OC segmentation
networks will predict the masks with superior glaucoma diagnosis performance.

###### Keywords:

Multi-rater learning Optic disc/cup segmentation Glaucoma diagnosis

##  1 Introduction

Accurate optic disc and cup (OD/OC) segmentation on fundus images is important
for the clinical assessment of glaucoma. It has been increasingly popular to
develop automated OD/OC segmentation [13, 14]ß, which is especially
accelerated by the exciting breakthroughs of deep neural networks and publicly
available datasets [2, 7, 24]. Different from the natural image dataset, OD/OC
segmentation datasets are often annotated by multiple clinical experts (e.g.,
RIGA [2], REFUGE-2 [7]) to mitigate the subjective bias of particular rater.
However, this multi-rater scenario also brings challenges to the deployment of
deep learning models, since they are commonly trained by a single ground-
truth. For training the neural network in multi-rater scenario, a common
practice is to take the average of multiple labels, i.e., majority vote.
Although this fusion strategy is simple and easy to implement, it comes at the
cost of ignoring the different expertness of multiple experts [9].

It is thus necessary to estimate the rater expertness for the fusion of multi-
rater annotations. A popular way is to estimate the expertness based on the
raw image prior [5, 25, 18]. However, we note that OD/OC segmentation is
clinically established for the glaucoma diagnosis. Specifically, the Cup-to-
Disc Ratio (CDR) calculated from the segmentation masks is an essential
clinical parameter for the glaucoma diagnosis [10]. This inspires us to take
glaucoma diagnosis as a gold standard to estimate the multi-rater expertness
and fuse the multi-rater labels. The fused label then can be used as the
ground-truth for the segmentation training.

Specifically, we use a glaucoma diagnosis network to evaluate the multi-rater
expertness. The evaluation diagnosis network is implemented by a segmentation
attentive diagnosis network, following [31]. We optimize an expertness map for
each rater to maximize the glaucoma diagnosis performance of the network. The
ground-truth fused according to this multi-rater expertness maps is named as
Diagnosis First Ground-truth ($DiagFirst$GT). In order to make $DiagFirst$GT
general for different diagnosis networks, we further propose Expertness
Generator (ExpG) to generate the expertness maps. ExpG helps to constrain the
high-frequency components by the continuity nature of neural network, so as to
promise the effectiveness of $DiagFirst$GT across different diagnosis
networks.

In brief, three major contributions are made with this paper. First, we
propose a novel strategy to fuse the ground-truth from multiple OD/OC
segmentation labels. The obtained ground-truth is called $DiagFirst$GT, which
can significantly facilitate the glaucoma diagnosis. Secondly, in order to
reinforce the generalization of $DiagFirst$GT, we further propose ExpG to
generate the expertness maps in the optimization process. ExpG eliminates the
high-frequency components by the continuity of neural network. Finally, the
experimental results show the OD/OC segmentation models trained on
$DiagFirst$GT estimates the masks with a clear 3% AUC improvement on standard
glaucoma diagnosis network. Such a method can be freely applied to any model
architecture and gains general improvement without external data.

##  2 Methodology

###  2.1 Motivation

In clinical research, a better segmentation often lead to a better diagnosis.
In the neural network models, many prior work [31, 4, 27] also show that
lesions segmentation masks can bring diagnosis networks with solid
improvement. Move a step further, a natural question is that whether the
segmentation masks with better qualities are better for the disease diagnosis?

Figure 1: Diagnosis performance of segmentation masks with different
qualities. It shows the OD/OC mask quality and glaucoma diagnosis performance
are positively correlated.

We design a simple experiment on fundus images to answer this question. We
train ResNet50 [12] to diagnose glaucoma from the fundus images, with OD/OC
segmentation masks as the auxiliary information. The fundus image and its
segmentation mask are concatenated as the input of the network. The network is
supervised by the glaucoma classification labels on a private dataset with
1600 samples. Segmentation masks with different qualities are collected from
early-stop training, which are denoted as Bad (average IOU of segmentation is
0.3), Medium (= 0.5), Good (= 0.7), and Annotation. We use these masks to
train the corresponding diagnosis networks, denoted as Bad Model, Medium
Model, Good Model, and Annotation Model. Then the segmentation masks with
different qualities are given to each model for the inference. The final
diagnosis performance measured by AUC score (%) is shown in Figure 1. In the
figure, NoMask Model denotes training the diagnosis network only on the raw
images.

It is clear to see that better segmentation masks promote the diagnosis
performance more than the worse. The diagnosis networks trained on better
segmentation masks show better average diagnosis performance. In the
inference, the diagnosis performance is also gradually improved with the
improvement of the segmentation quality. This result suggests, like what in
clinical research, better OD/OC segmentation is also more conducive to the
glaucoma diagnosis in the neural network models.

###  2.2 Learning $DiagFirst$GT

Figure 2: Overall flow of proposed method. Blue denotes static parameters.
Orange denotes learnable parameters.

Motivated by this relationship between OD/OC segmentation and glaucoma
diagnosis, we want to find the optimal OD/OC segmentation label toward the
glaucoma diagnosis when multiple labels are collected. Such a label, based on
the results above, may be closer to the potential gold segmentation. To fuse
multiple labels to one ground-truth, we weighted sum the multiple labels by
the expertness of each rater:

$\textrm{Groundtruth}=s\odot m=\sum_{i=1}^{n}s_{i}*m_{i},$ (1)

where $*$ denotes the element-wise multiplication, $s_{i}$ and $m_{i}$ are the
annotation and the expertness map of the rater $i$, respectively.

We then assess the rater expertness through the glaucoma diagnosis. In other
words, the rater contributes more to the correct diagnosis would be given
higher expertness. Toward that end, we take the multi-rater expertness maps as
the learnable variables to maximize the diagnosis performance of a OD/OC
attentive diagnosis network. The network is pre-trained on the same data so as
to correctly evaluate the diagnosis contribution. The optimization is solved
by the gradient descent to activate the target class in diagnosis network. We
dubbed the label fused as $DiagFirst$GT.

The overflow of learning $DiagFirst$GT is shown in Figure 2 (a). Formally,
consider each raw image $x\in\mathbb{R}^{h\times w\times c}$ is annotated by
$n$ raters, resulting in $n$ segmentation masks $s\in\mathbb{R}^{h\times
w\times n}$, and the image is diagnosed as $y\in[0,1]$. Let $\theta$ denotes
the set of diagnosis model parameters. $L(\theta,x,y)$ denotes the loss
function of a standard classification task. Our goal is to find an optimal
expertness map $m\in\mathbb{R}^{h\times w\times n}$ by solving the following
optimization problem:

$m^{*}=\mathop{\arg\min}\limits_{m}L(\theta,x\oplus[s\odot m],y),$ (2)

where $\oplus$ denotes concatenation operation, $m^{*}$ denotes the optimal
expertness maps. Note that we applied Softmax on $m$ to normalize the weights.
According to Eqn. (2) , we are actually finding the expertness maps which can
minimize the diagnosis loss. We name the fused ground-truth under these
expertness maps $s\odot m^{*}$ as $DiagFirst$GT.

Gradient descent is then adopted to solve Eqn. (2)

$m^{t+1}=m^{t}+\alpha\nabla_{m}L(\theta,x\oplus[s\odot m],y),$ (3)

where $\alpha$ is the learning rate. The fusion ground-truth optimized in this
way can improve the diagnosis performance to a very high level, but since it
is optimized toward one specific diagnosis network, it is not general enough.
The visualization results also show it suffers heavily from high-frequency
noises. A visualized example is shown in Figure 3. The latest findings suggest
that these high-frequency components have close relationship with the
generalization capability [26, 23]. Specifically, high-frequency components
are generated by the repeated grid effect of the transposed convolution [22]
when we backprop gradients through each convolution layer. Thus they are very
specific to the network architecture, like the number of the layers, the
stride of the convolution, etc.

###  2.3 ExpG

Therefore, a possible approach to improve generalization of $DiagFirst$GT is
to constrain the high-frequency components in the optimization process. We
tried several methods to achieve the goal, including Transformation Robustness
(TransRob), Fourier Transform (Fourier) and proposed Expertness Generator
(ExpG). Transformation Robustness constrains high-frequency gradients by
applying small transformations to the expertness map before optimization. In
practice, we rotate, scale and jitter the maps. Fourier Transform transforms
the expertness map parameters to frequency domain, thus decorrelated the
relationship between the neighbour pixels.

ExpG constrains the high-frequency components by the continuity nature of the
neural network, which is implemented by a tiny CNN based pixel generator. An
illustration of ExpG is shown in Figure 2 (b). The network scans one pixel at
a time. For each pixel it predicts the pixel value given the position of the
pixel. The input of ExpG is the coordinate vector $(i,j)$ and the output is
the pixel value. ExpG is optimized to generate expertness maps which are able
to minimize the diagnosis loss. Denote the parameters of ExpG as $\phi$, our
goal is to solve:

$\mathop{\arg\min}\limits_{\phi}L(\theta,x\oplus[s\odot\\{\rm{ExpG}_{\phi}(i,j)\\}_{i=1\sim
h}^{j=1\sim w}],y),$ (4)

where $\\{\rm{ExpG}_{\phi}(i,j)\\}_{i=1\sim h}^{j=1\sim w}$ denotes generated
expertness map with size $h\times w\times n$ by ExpG. Since the continuity of
the neural network mapping function, similar inputs tend to cause similar
outputs, which lead the element values in expertness maps variant smoothly
between the positions and thus eliminate the high-frequency components.

Figure 3: Visualized results of various high-frequency elimination methods and
ExpG. From left to right are raw fundus image, original high-frequency
results, TransRob, Fourier, ExpG low-frequency results and activation results
of ExpG.

The visualized results of all high-frequency elimination methods are shown in
Figure 3. The first line is a non-glaucoma example and the second line is a
glaucoma example. We can see the visualized effects of TransRob and Fourier
are obviously improved comparing with high-frequency examples. But ExpG
achieves much better visualized effects than both of them. Thus, we adopt ExpG
to eliminate the high-frequency components in the experiments.

##  3 Experiment

###  3.1 Experimental settings

All the experiments are implemented with the PyTorch platform and
trained/tested on 4 Tesla P40 GPU with 24GB of memory. All images are
uniformly resized to the dimension of 256$\times$256 pixels. The networks are
trained in an end-to-end manner using Adam optimizer with a mini-batch of 16
for 80 epochs. The learning rate is initially set to 1 $\times 10^{-4}$. The
detailed hyper-parameters and architectures can be found in the code.

We conduct the experiments on REFUGE-2 [7] dataset. REFUGE-2 is a publicly
available dataset for retinal cup and disc segmentation annotated by multiple
glaucoma experts. REFUGE-2 contains in total 2000 color fundus images,
including three sets each with 1200 images for training, 400 for validation,
and 400 for testing. Seven glaucoma experts from different organizations
labeled the optic cup and disc contour masks manually for the REFUGE-2
benchmark. 200 samples correspond to glaucomatous subjects, and the others
correspond to non-glaucomatous subjects. The glaucomatous subjects are
distributed equally to the training, validation, and test set. It is worth
noting that the diagnosis network is pre-trained on REFUGE-2 training set to
generate $DiagFirst$GTs, thus no external data is used.

###  3.2 Comparing with SOTA

To verify the glaucoma diagnosis effect of $DiagFirst$GT, we compare it with
state-of-the-art (SOTA) multi-rater fusion strategies on REFUGE-2 dataset.
Specifically, we train standard UNet on various multi-rater fusion ground-
truths, including random fusion (Random), majority vote (MV), STAPLE [22],
AggNet [1] and Max-Mig [5]. We then evaluate the diagnosis performance of
their estimated masks on the test set. The glaucoma diagnosis performance is
evaluated by a range of OD/OC segmentation-assisted glaucoma prediction
methods, including which based on vertical Cup-to-Disc Ratio (vCDR) [6, 19]
and which based on deep learning models [15, 17, 27, 28]. The obtained
diagnosis performance measured by AUC score (%) is shown in Table
LABEL:Table:GoldDiag.

Table 1: Comparison of $DiagFirst$GT and SOTA multi-rater fused ground-truths
on a range of diagnosis models (a) and segmentation models (b).

| MV |  $DiagFirst$GT
---|---|---
| $\mathcal{D}_{disc}$ | $\mathcal{D}_{cup}$ | AUC | $\mathcal{D}_{disc}$ | $\mathcal{D}_{cup}$ | AUC
AGNet | 91.30 | 77.38 | 80.18 | 91.65 | 75.14 | 81.58
CENet | 91.70 | 80.34 | 80.46 | 91.05 | 78.63 | 83.41
pOSAL | 94.52 | 82.42 | 81.53 | 93.64 | 82.32 | 84.07
BEAL | 93.84 | 83.51 | 81.66 | 93.88 | 81.93 | 84.35
ATTNet | 96.10 | 84.91 | 81.75 | 95.78 | 84.65 | 84.82

(b) Comparison of majority vote (MV) and $DiagFirst$GT ground-truth on various
segmentation models measured by Dice (%) and AUC (%). Although $DiagFirst$GT
is more difficult to learn, it supervises the segmentation models to learn
more glaucoma diagnosis knowledge.

We can see $DiagFirst$GT significantly improves the diagnosis performance on
all of the methods, no matter it is vCDR based [19, 6], attentive CNN based
[15, 28, 17] or transfer learning CNN based [27]. On vCDR based, attentive
network based and transfer learning based methods, it outperforms the previous
best ground-truths (MV, STAPLE and LFC) by a 1.82%, 2.68% and 2.12% AUC,
respectively. This demonstrates $DiagFirst$GT is a general and effective
ground-truth for the glaucoma diagnosis algorithms, and also indicates its
significant potential value in clinical diagnosis of glaucoma.

To verify $DiagFirst$GT is also an effective ground-truth for various
segmentation models. We train SOTA segmentation models on $DiagFirst$GT, and
compare it with traditional majority vote ground-truth (MV). The SOTA
segmentation models include AGNet [29], CENet [11], pOSAL [21], BEAL [20] and
ATTNet [30]. The segmentation performance measured by optic disc Dice score
($\mathcal{D}_{diusc}$) and optic cup Dice score ($\mathcal{D}_{cup}$) is
shown in Table 1(b). We can see the segmentation models perform better on MV
than $DiagFirst$GT. That is because MV is a simple average of multiple labels
which is easy to learn, while the fusion of $DiagFirst$GT is based on the
diagnosis prior knowledge that is more difficult to learn. We also report the
glaucoma diagnosis performance of their estimated masks measured by AUC (%).
The diagnosis performance is evaluated by [31] pre-trained on a large glaucoma
detection dataset (LAG [15]). It is shown although the segmentation models
perform better on MV, the diagnosis performance of the estimated masks is
worse than that of $DiagFirst$GT. With the improvement of the segmentation
capabilities, the glaucoma diagnosis AUC on $DiagFirst$GT increase about 3%,
also outperforms the counterpart on MV (about 1%), indicating the supervision
of $DiagFirst$GT can improve the diagnosis performance of estimated masks to a
larger extent.

###  3.3 Ablation study on ExpG

ExpG is proposed in the paper to improve generalization of $DiagFirst$GT. In
order to verify the effectiveness of ExpG, we compare the $DiagFirst$GT before
and after the application of ExpG on REFUGE-2 dataset. The diagnosis
performance measured by AUC (%) is shown in Table 2. Experiment is conducted
on a range of OD/OC-assisted diagnosis models to evaluate the generalization
capability. Six different models are selected, including DualStage [3], DENet
[8], AGCNN [15], ColNet [31], L2T-KT [27], and Swin Transformer[16] with mask-
concated input.

As shown in Table 2, comparing with original high-frequency $DiagFirst$GT,
ExpG generated $DiagFirst$GT consistently achieves higher performance on
different diagnosis methods, especially on stronger networks. Concretely, ExpG
generated $DiagFirst$GT outperforms the counterpart by a 3.95% AUC on DENet, a
4.93% AUC on AGCNN, a 6.06% AUC on Swin-cat and a 7.94% AUC on L2T-KT. It is
obvious that ExpG generated $DiagFirst$GT is more robust for different
diagnosis networks.

Table 2: The ablation study of ExpG. High-frequency $DiagFirst$GT and ExpG generated low-frequency $DiagFirst$GT are compared on different diagnosis backbones. The diagnosis performance is evaluated by AUC (%).  | Original | ExpG
---|---|---
DualStage [3] | 79.36 | 80.78
DENet [8] | 81.21 | 85.16
AGCNN [15] | 79.35 | 84.28
Swin-cat [16] | 78.66 | 84.72
ColNet [31] | 81.69 | 87.34
L2T-KT [27] | 83.67 | 91.61

##  4 Conclusion

OD/OC segmentation is often annotated by multiple experts. For training
standard deep learning models, the multiple labels need to be fused to one
ground-truth. The existed fusion methods often ignore the different expertness
of each rater. In the paper, we take glaucoma diagnosis as the gold standard
to assess the multi-rater expertness, which enable the fused ground-truth
$DiagFirst$GT to facilitate the glaucoma diagnosis. Detailed experiments
revealed that by training on $DiagFirst$GT, the OD/OC segmentation networks
can produce the masks with high glaucoma diagnosis performance. The results
are verified on a variety of glaucoma diagnosis networks to promise the
generalization. Since OD/OC segmentation is often used to assist the glaucoma
diagnosis clinically, we believe the proposed method is in line with the
original intention of OD/OC segmentation task and has great significance for
the clinical glaucoma diagnosis.

## References

  * [1] Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., Navab, N.: Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE transactions on medical imaging 35(5), 1313–1321 (2016)
  * [2] Almazroa, A., Alodhayb, S., Osman, E., Ramadan, E., Hummadi, M., Dlaim, M., Alkatee, M., Raahemifar, K., Lakshminarayanan, V.: Retinal fundus images for glaucoma analysis: The RIGA dataset. In: SPIE Conference on Medical Imaging (2018)
  * [3] Bajwa, M.N., Malik, M.I., Siddiqui, S.A., Dengel, A., Shafait, F., Neumeier, W., Ahmed, S.: Two-stage framework for optic disc localization and glaucoma classification in retinal fundus images using deep learning. BMC medical informatics and decision making 19(1), 1–16 (2019)
  * [4] Bechar, M.E.A., Settouti, N., Barra, V., Chikh, M.A.: Semi-supervised superpixel classification for medical images segmentation: application to detection of glaucoma disease. Multidimensional Systems and Signal Processing 29(3), 979–998 (2018)
  * [5] Cao, P., Xu, Y., Kong, Y., Wang, Y.: Max-mig: an information theoretic approach for joint learning from crowds. arXiv preprint arXiv:1905.13436 (2019)
  * [6] Chandrika, S., Nirmala, K.: Analysis of cdr detection for glaucoma diagnosis. International Journal of Engineering Research and Application 2(4), 23–27 (2013)
  * [7] Fang, H., Li, F., Fu, H., Sun, X., Cao, X., Son, J., Yu, S., Zhang, M., Yuan, C., Bian, C., et al.: Refuge2 challenge: Treasure for multi-domain learning in glaucoma assessment. arXiv preprint arXiv:2202.08994 (2022)
  * [8] Fu, H., Cheng, J., Xu, Y., Zhang, C., Wong, D.W.K., Liu, J., Cao, X.: Disc-aware ensemble network for glaucoma screening from fundus image. IEEE transactions on medical imaging 37(11), 2493–2501 (2018)
  * [9] Fu, H., Li, F., Xu, Y., Liao, J., Xiong, J., Shen, J., Liu, J., Zhang, X., iChallenge-GON study group, et al.: A retrospective comparison of deep learning to manual annotations for optic disc and optic cup segmentation in fundus photographs. Translational vision science & technology 9(2), 33–33 (2020)
  * [10] Garway-Heath, D.F., Ruben, S.T., Viswanathan, A., Hitchings, R.A.: Vertical cup/disc ratio in relation to optic disc size: its value in the assessment of the glaucoma suspect. British Journal of Ophthalmology 82(10), 1118–1124 (1998)
  * [11] Gu, Z., Cheng, J., Fu, H., Zhou, K., Hao, H., Zhao, Y., Zhang, T., Gao, S., Liu, J.: Ce-net: Context encoder network for 2d medical image segmentation. IEEE transactions on medical imaging 38(10), 2281–2292 (2019)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [13] Held, K., Kops, E.R., Krause, B.J., Wells, W.M., Kikinis, R., Muller-Gartner, H.W.: Markov random field segmentation of brain mr images. IEEE transactions on medical imaging 16(6), 878–886 (1997)
  * [14] Ji, W., Yu, S., Wu, J., Ma, K., Bian, C., Bi, Q., Li, J., Liu, H., Cheng, L., Zheng, Y.: Learning calibrated medical image segmentation via multi-rater agreement modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12341–12351 (2021)
  * [15] Li, L., Xu, M., Wang, X., Jiang, L., Liu, H.: Attention based glaucoma detection: A large-scale database and cnn model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10571–10580 (2019)
  * [16] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)
  * [17] Luo, Y., Huang, Q., Li, X.: Segmentation information with attention integration for classification of breast tumor in ultrasound image. Pattern Recognition p. 108427 (2021)
  * [18] Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. Journal of Machine Learning Research 11(Apr), 1297–1322 (2010)
  * [19] Thangaraj, V., Natarajan, V.: Glaucoma diagnosis using support vector machine. In: 2017 International Conference on Intelligent Computing and Control Systems (ICICCS). pp. 394–399. IEEE (2017)
  * [20] Wang, S., Yu, L., Li, K., Yang, X., Fu, C.W., Heng, P.A.: Boundary and entropy-driven adversarial learning for fundus image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 102–110. Springer (2019)
  * [21] Wang, S., Yu, L., Yang, X., Fu, C.W., Heng, P.A.: Patch-based output space adversarial learning for joint optic disc and cup segmentation. IEEE transactions on medical imaging 38(11), 2485–2495 (2019)
  * [22] Warfield, S.K., Zou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation. IEEE transactions on medical imaging 23(7), 903–921 (2004)
  * [23] Wojna, Z., Ferrari, V., Guadarrama, S., Silberman, N., Chen, L.C., Fathi, A., Uijlings, J.: The devil is in the decoder. In: British Machine Vision Conference 2017, BMVC 2017. pp. 1–13. BMVA Press (2017)
  * [24] Wu, J., Fang, H., Li, F., Fu, H., Lin, F., Li, J., Huang, L., Yu, Q., Song, S., Xu, X., et al.: Gamma challenge: glaucoma grading from multi-modality images. arXiv preprint arXiv:2202.06511 (2022)
  * [25] Wu, J., Fang, H., Shang, F., Wang, Z., Yang, D., Zhou, W., Yang, Y., Xu, Y.: Learning self-calibrated optic disc and cup segmentation from multi-rater annotations (2022)
  * [26] Wu, J., Fu, R.: Universal, transferable and targeted adversarial attacks. arXiv preprint arXiv:2109.07217 (2019)
  * [27] Wu, J., Yu, S., Chen, W., Ma, K., Fu, R., Liu, H., Di, X., Zheng, Y.: Leveraging undiagnosed data for glaucoma classification with teacher-student learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 731–740. Springer (2020)
  * [28] Zhang, J., Xie, Y., Xia, Y., Shen, C.: Attention residual learning for skin lesion classification. IEEE transactions on medical imaging 38(9), 2092–2103 (2019)
  * [29] Zhang, S., Fu, H., Yan, Y., Zhang, Y., Wu, Q., Yang, M., Tan, M., Xu, Y.: Attention guided network for retinal image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 797–805. Springer (2019)
  * [30] Zhao, X., Wang, S., Zhao, J., Wei, H., Xiao, M., Ta, N.: Application of an attention u-net incorporating transfer learning for optic disc and cup segmentation. Signal, Image and Video Processing 15(5), 913–921 (2021)
  * [31] Zhou, Y., He, X., Huang, L., Liu, L., Zhu, F., Cui, S., Shao, L.: Collaborative learning of semi-supervised segmentation and classification for medical images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2079–2088 (2019)