A Detailed View of the Broad Line Region in NGC 3783 from Velocity-Resolved Reverberation Mapping

0000-0002-2816-5398]Misty C. Bentz
Department of Physics and Astronomy,
Georgia State University,
Atlanta, GA 30303, USA

0000-0002-4645-6578]Peter R. Williams
Department of Physics and Astronomy,
University of California,
Los Angeles, CA 90095, USA

0000-0001-6279-0552]Rachel Street
LCOGT, 6740 Cortona Drive, Suite 102,
Goleta, CA 93117, USA

0000-0003-0017-349X]Christopher A. Onken
Research School of Astronomy and Astrophysics,
Australian National University,
Canberra, ACT 2611, Australia

0000-0002-6257-2341]Monica Valluri
Department of Astronomy,
University of Michigan,
Ann Arbor, MI, 48104, USA

0000-0002-8460-0390]Tommaso Treu
Packard Fellow
Department of Physics and Astronomy,
University of California,
Los Angeles, CA 90095, USA

We have modeled the full velocity-resolved reverberation response of the H$\beta$ and He2 optical broad emission lines in NGC 3783 to constrain the geometry and kinematics of the low-ionization and high-ionization broad line region.  The geometry is found to be a thick disk that is nearly face on, inclined at $\sim18\degr$ to our line of sight, and exhibiting clear ionization stratification, with an extended H$\beta$-emitting region ($r_{\rm median}=10.07^{+1.10}_{-1.12}$ light days) and a more compact and centrally-located He2-emitting region ($r_{\rm median}=1.33^{+0.34}_{-0.42}$ light days). In the H$\beta$-emitting region, the kinematics are dominated by near-circular Keplerian orbits, but with $\sim 40$% of the orbits inflowing.  The more compact He2-emitting region, on the other hand, appears to be dominated by outflowing orbits. The black hole mass is constrained to be $=2.82^{+1.55}_{-0.63}\times10^7$ , which is consistent with the simple reverberation constraint on the mass based on a mean time delay, line width, and scale factor of $\langle f \rangle=4.82$.  The difference in kinematics between the H$\beta$- and He2-emitting regions of the BLR is intriguing given the recent history of large changes in the ionizing luminosity of NGC 3783 and evidence for possible changes in the BLR structure as a result.

§ INTRODUCTION

Black holes continue to capture our imaginations centuries after the concept was first recorded in a letter written by a country clergyman [Michell, 1784]. Today, not only are black holes a recurrent feature in science fiction, but they have become securely ensconced in science fact.  We now know that supermassive ($=10^5-10^{10}$ ) black holes exist, that they inhabit the centers of most (all?) massive galaxies, and that their masses scale with several measurable properties of their host galaxies, including the bulge stellar velocity dispersion and bulge luminosity (e.g., Magorrian et al., 1998, Ferrarese & Merritt, 2000, Gebhardt et al., 2000, Gültekin et al., 2009, Kormendy & Ho, 2013).

Only a handful of methods are able to directly constrain the mass of a central, supermassive black hole through its gravitational effects on luminous matter.  In the case of the Milky Way, astrometric monitoring of individual stars in the central few parsecs has resulted in a constraint on the mass of Sagittarius A* of $=(4.1\pm0.6)\times10^6$ [Ghez et al., 2000, Genzel et al., 2000, Ghez et al., 2008], while relativistic modeling of the emission from gas just outside the event horizon has constrained the mass of Pōwehi, the central black hole in M87, to $=(6.5\pm0.7)\times10^9$ [Event Horizon Telescope Collaboration et al., 2019].  Most other galaxies are not able to be studied with similar methods because we lack the necessary spatial resolution.  However, many nearby galaxies ($D\lesssim100$ Mpc) may still be studied through spatially-resolved observations of the bulk nuclear gas or stellar kinematics on scales of $\sim$tens of parsecs (e.g., Gültekin et al., 2009, Kormendy & Ho, 2013).  Reverberation mapping is notable among black hole mass measurement techniques because it relies on time resolution rather than angular resolution.  By monitoring the spectrophotometric variability of an active galactic nucleus (AGN), the black hole mass, among other properties, may be constrained for a local Seyfert or a distant quasar (for a recent review, see Cackett et al., 2021).

Reverberation mapping makes use of the response of photoionized gas in the broad emission-line region (BLR) to variations in the continuum luminosity, a technique that was first proposed by Bahcall et al., 1972.  As it is generally implemented, reverberation mapping constrains an average responsivity-weighted radius for the BLR in an AGN.  Combining the radius with a measure of the line-of-sight velocity of the BLR gas via the virial theorem constrains  [Peterson & Wandel, 1999, Peterson & Wandel, 2000], modulo a scale factor that accounts for the generally unknown BLR geometry and kinematics (e.g., Onken et al., 2004, Park et al., 2012, Grier et al., 2013, Batiste et al., 2017).  However, high quality spectrophotometric monitoring data contain information about the gas response as a function of line-of-sight velocity, thus providing constraints on the emissivity and position of photoionized gas in a spatially-unresolved source [Blandford & McKee, 1982].  Velocity-resolved reverberation mapping, as it has come to be known, is thus able to directly constrain the BLR geometry and the black hole mass, thus avoiding the need to apply a scale factor.

The analysis of velocity-resolved reverberation mapping data can be approached as an ill-posed inverse problem, in which the goal is to recover the transfer function that describes the time delay distribution as a function of velocity across a broad emission line (e.g., Horne, 1994, Skielboe et al., 2015, Anderson et al., 2021).  Or it can be approached through direct modeling, in which a framework of fully self-consistent models is built and an exploration of the available parameter space yields the family of models that best match the observational constraints (e.g., Pancoast et al., 2011).  Direct modeling has the advantage that it is relatively simple to interpret the results, however its ability to match complicated data sets is limited by the phenomenology that is included and how it is parametrized.  Recovery of the transfer function, on the other hand, takes advantage of the full range of details present in the observations but is nontrivial to interpret.

While the promise of velocity-resolved reverberation mapping has long been understood, it was only within the last decade or so that improvements in the quality of reverberation mapping data (e.g., Bentz et al., 2008, Bentz et al., 2009, Denney et al., 2009, Grier et al., 2012) have finally allowed the BLR structure and kinematics to be explored in detail for a handful of AGNs [Pancoast et al., 2014, Grier et al., 2017, Williams et al., 2018, Williams et al., 2020].  In general, direct modeling has found many similarities across objects, although the exact details vary: the low-ionization BLR is arranged in a thick disk-like structure at low to moderate inclination to our line of sight, and with kinematics that are dominated by near-circular Keplerian orbits but with a contribution from inflow (although Williams et al., 2018 find evidence for outflow, rather than inflow, in some of their sample).  The high-ionization BLR is less well studied, and Williams et al., 2020 find several key differences in not just the kinematics but also the geometry of the low- and high-ionization BLR gas in NGC 5548.  Studies that have focused on the recovery of the transfer function have generally drawn similar conclusions about the BLR structure and kinematics [Bentz et al., 2010, Horne et al., 2021].  A key finding of all these studies is that the black hole masses derived from a more simplistic reverberation analysis, involving a mean time delay and line width and an adopted scale factor of $\langle f \rangle \approx 5$, are generally in good agreement within their uncertainties with the masses derived from modeling.  As expected, the largest differences are generally found for those AGNs where direct modeling derives an inclination of the BLR that is $\lesssim 15^{\circ}$ to our line of sight (cf. Figure 14 of Williams et al., 2018).  Very low inclinations result in small observed line-of-sight velocities, which bias the simplistic mass estimates to low values.

We recently conducted a new reverberation mapping program focusing on the bright Southern Seyfert, NGC 3783, with the intent of improving the constraints on the black hole mass.  A nearly face-on barred spiral galaxy at $z=0.0097$, NGC 3783 is one of the most well-studied AGNs in the sky.  It is one of a few Seyfert 1s that may be studied in detail with VLT GRAVITY observations on spatial scales that resolve the dust torus and outer broad line region [Gravity Collaboration et al., 2021], thus it is a critical target for informing our understanding of both feeding and feedback.  Furthermore, NGC 3783 is also one of a small number of Seyfert 1 galaxies that are near enough to allow a reverberation-based mass to be directly compared with masses constrained through dynamical methods.  The comparison of reverberation and dynamical masses is the only independent check that we can use to investigate the reliability of the entire black hole mass scale that we currently apply across cosmic history, an important point given the different systematic biases that are inherent in each black hole mass measurement technique.

An initial assessment of the monitoring data constrained a reverberation-based black hole mass of $M_{\rm BH} =(2.3\pm0.4)\times 10^7$ M$_{\odot}$ [Bentz et al., 2021], assuming a scale factor of $\langle f \rangle = 4.82$ [Batiste et al., 2017].  However, variations in the time delay as a function of velocity across  H$\beta$ and other optical emission lines were also seen in the spectra, with longer time delays observed near the line center and shorter time delays in the line wings.  These initial results indicated that direct modeling would be likely to provide strong constraints on the BLR geometry and kinematics in NGC 3783, and that we might be able to probe both the low-ionization BLR through the broad H$\beta$ emission line as well as the high-ionization BLR through the He2 $\lambda4686$ broad line.  Here, we present the results of that modeling and a new direct constraint on the black hole mass in NGC 3783.

§ DATA

A detailed description of the photometric and spectroscopic monitoring data are provided by Bentz et al., 2021.  In summary, $V-$band photometric monitoring was carried out with the Las Cumbres Observatory global telescope (LCOGT) network of 1-m telescopes from 12 February to 30 June 2020. Notwithstanding the sudden onset of a global pandemic and the shutdown of several observatories, 209 images were acquired over this period with a median temporal sampling of 0.4 days.  Spectroscopic monitoring with the robotic FLOYDS spectrograph on the 2-m Faulkes Telescope South was carried out over the same period, with 50 spectra acquired between 27 February and 26 June 2020, with a median temporal sampling of 1.7 days.

Example spectrum of NGC 3783 in black with the ULySS fit to the continuum and [O3] $\lambda\lambda 4959,5007$ doublet overplotted in red, and the continuum and O3-subtracted spectrum in blue. The vertical dotted lines mark the limits of the regions that were modeled for the H$\beta$  (4816$-$5025 Å) and He2 (4653$-$4816 Å) emission lines. With the continuum subtracted, low-level Fe2 emission is visible on the blue side of He2 and the red side of the [O3] doublet, but the analysis of Bentz et al., 2021 shows that Fe2 was not variable at a detectable level in these data.

The images and spectra were reduced in IRAF[IRAF is distributed by the National Optical Astronomy Observatories, which are operated by the Association of Universities for Research in Astronomy, Inc., under cooperative agreement with the National Science Foundation.] following standard procedures.  The spectra were intercalibrated using the [O3] $\lambda\lambda 4959,5007$ emission lines, which are constant in flux on timescales of a few months [Peterson et al., 2013], thus providing a correction for small wavelength shifts, differences in resolution, and offsets in flux calibration from night to night.   Image subtraction methods [Alard & Lupton, 1998, Alard, 2000] were used to isolate the variable AGN point source from the constant host-galaxy emission in the images, providing a well-sampled and well-calibrated light curve of the AGN optical continuum emission.  This was merged with the flux-calibrated continuum light curve measured at $5100\times(1+z)$ Å in the spectra, with data points taken within 0.25 days binned together for the final continuum light curve.

Before modeling the reverberation response, the continuum and [O3]  emission lines were subtracted from each spectrum to allow the broad emission to be isolated.  This was accomplished by modeling the spectral components in the high signal-to-noise mean spectrum with ULySS [Koleva et al., 2009] and then slightly adjusting that model to create a best fit for each individual spectrum before subtracting the desired model components.  The continuum was fit with a powerlaw, representing the AGN continuum contribution, and a host-galaxy component parameterized by the Vazdekis models derived from the MILES library of empirical stellar spectra [Vazdekis et al., 2010].  Emission lines were fit with multiple Gaussian profiles, with 4 Gaussians needed to match each of the H$\beta$ and [O3] doublet lines and $1-4$ Gaussians needed to match other emission features in the spectrum.  Once a best fit was achieved for the mean spectrum, the individual spectra were then fit one at a time, with the host-galaxy component held fixed to the best-fit template but allowed to vary in flux contribution, and with the power law and the emission-line components allowed to vary but with initial values matching their best-fit values.  Once a best fit was found, the host-galaxy and power law continua and the [O3] components were then subtracted from each spectrum.  Figure <ref> shows an example spectrum from a single night of observations in black, with the best-fit continuum and [O3] emission in red, and the spectrum after subtraction of those components in blue.

The H$\beta$ region was then isolated for modeling between observed wavelengths $4816-5025$ Å  with the narrow emission line peak at 4910 Å, while the He2 region was isolated between $4653-4816$ Å with the narrow emission line peak observed at 4735 Å.   Throughout the campaign, the rest-frame equivalent width of broad H$\beta$ relative to the starlight-corrected AGN continuum has a mean value of 139.9 Å with a median  of 130.5 Å and a standard deviation of 22.4 Å.   For He2, the mean rest-frame equivalent width is 15.8 Å with a median of 15.1 Å and a standard deviation of 5.4 Å.
While the blue spectra also cover the H$\gamma$ and H$\delta$ broad emission lines, and the red spectra cover the H$\alpha$ emission line, Bentz et al., 2021 described the difficulties in accurately calibrating the red spectra and the short wavelength end of the blue spectra.  The integrated light curves for these emission lines clearly demonstrate significant excess noise, so we do not attempt to model them here.

§ BLR MODELS

Modeling of the BLR for H$\beta$ and for He2 was carried out with CARAMEL, a phenomenological modeling code that is described in detail by Pancoast et al., 2014. CARAMEL is capable of constraining both the geometry and kinematics of the BLR using the reverberation response across the profile of a broad emission line throughout a monitoring campaign.  Here, we summarize the main components of the model.

CARAMEL represents the BLR as a large collection of massless point particles that are distributed in position and velocity space,  surrounding a massive black hole whose gravity dominates the region.  Each point particle processes incident continuum flux instantaneously, and the observed time delay profile of the BLR depends on the spatial distribution of point particles while the broad line wavelength profile depends on the velocity distribution of point particles.

The spatial distribution of particles is parametrized with angular and radial distributions. The radial positions of particles are drawn from a gamma distribution

\begin{equation}
p(x|\alpha,\theta) \propto x^{\alpha - 1}\exp{\left( - \frac{x}{\theta} \right)}
\end{equation}

that provides the flexibility to represent a Gaussian  ($\alpha>1$), an exponential ($\alpha=1$), or a cuspier profile ($0<\alpha<1$).  The gamma distribution of particles is shifted away from the location of the black hole by the Schwarzschild radius, $R_s = 2GM/c^2$, plus a minimum radius $r_{\rm min}$. To assist with interpretation of the modeling results, a change of variables is performed so that parametrization is given in terms of ($\mu$, $\beta$, $F$):

\begin{equation}
\mu = r_{\rm min} + \alpha \theta,
\end{equation}

\begin{equation}
\beta = \frac{1}{\sqrt{\alpha}},
\end{equation}

\begin{equation}
F = \frac{r_{\rm min}}{r_{\rm min} + \alpha \theta},
\end{equation}

where $\mu$ is the mean radius, $\beta$ is the shape parameter, and $F$ is $r_{\rm min}$ in units of $\mu$.  The standard deviation of the shifted gamma profile is given by $\sigma_r = \mu \beta (1-F)$, and the BLR is truncated at an outer radius of $r_{\rm out} = c \Delta t_{\rm data}/2$, where $\Delta t_{\rm data}$ is the time difference between the first point in the modeled continuum light curve and the first point in the emission-line light curve.  This truncation assumes that the total length of the monitoring campaign is sufficient to track reverberation signals throughout the entire BLR.

The angular distribution of the particles is then arranged in a disk with a thickness that is set by an opening angle $\theta_o$, where $\theta_o=0\degr$ is a thin disk and $\theta_o=90\degr$ is a sphere.  The inclination of the disk to the observer's line of sight is set by $\theta_i$, where $\theta_i=0\degr$ is viewed face on and $\theta_i=90\degr$ is viewed edge on.  The strength of line emission from different depths within the disk is parametrized by the distribution of particles as a function of depth.  For a single particle, the angle of displacement from the disk midplane is given by

\begin{equation}
\theta_{d,N} = \arccos (\cos \theta_o + (1-\cos \theta_o)\times U^{\gamma})
\end{equation}

where $U$ is a random number drawn uniformly between 0 and 1.  The value of $\gamma$ ranges from 1, where particles are distributed uniformly throughout the thickness of the disk, to 5, where particles are clustered at the disk face and therefore emission is preferentially from the outer skin of the BLR.  An additional asymmetry parameter, $\xi$ allows for the possibility of obscuration along the midplane of the disk, where $\xi \rightarrow 0$ causes the entire back half of the disk to be obscured and $\xi = 1$ has no midplane obscuration.  The final asymmetry parameter $\kappa$ is related to the weight of a particle

\begin{equation}
W(\phi) = \frac{1}{2} + \kappa \cos \phi
\end{equation}

where $W$ is the fraction of continuum flux that is reradiated back towards the observer as line flux  and $\phi$ is the angle between the observer's line of sight to the source and the particle's line of sight to the source. The value of $\kappa$ ranges from $-0.5$, where particles preferentially emit back towards the ionizing source, to $0.5$, where particles preferentially radiate away from the ionizing source.  In the case of $\kappa=-0.5$, an observer would see preferential emission from the far side of the disk, while preferential emission from the near side would be observed in the case of $\kappa=0.5$.

Histograms displaying the posterior distributions of the BLR model parameters for H$\beta$ (blue) and He2 (red).

Tabulated values are the median and 68% confidence intervals.

The velocity distribution of particles includes radial and tangential distributions.  A fraction of the particles, $f_{\rm ellip}$, have near-circular orbits within the Keplerian potential of the central black hole with mass .  The remaining particles ($1-f_{\rm ellip}$) are either inflowing ($f_{\rm flow}<0.5$) or outflowing ($f_{\rm flow}>0.5$). Whether these orbits are generally bound or unbound is determined by the parameter $\theta_e$.  For a plane defined by the possible values of the radial and tangential velocities, $\theta_e$  describes the angle of the velocity components away from the escape velocity and towards the circular velocity.  If $\theta_e =0$ degrees then the  orbits are drawn from a Gaussian distribution centered on the escape velocity. As $\theta_e \rightarrow 90\degr$, the inflowing or outflowing orbits approach the parameter space occupied by near-circular orbits.  Thus high values of $\theta_e$ indicate inflowing or outflowing orbits that are very nearly circular, $\theta_e\approx45\degr$ indicates that most of the inflowing or outflowing orbits are highly eccentric but still bound, and low values of $\theta_e$ indicate that most particles are near the escape velocity and  unbound.

A contribution from macroturbulence is included in the line-of-sight component of the velocity vector for each point particle as

\begin{equation}
v_{\rm turb} = \mathcal{N} (0,\sigma_{\rm turb})|v_{\rm circ}|,
\end{equation}

where $v_{\rm circ}$ is the circular velocity and $\mathcal{N}(0,\sigma_{\rm turb})$ is a normal distribution centered on 0 and with standard deviation $\sigma_{\rm turb}$.

With the spatial and velocity distributions of the particles parametrized, the emission-line profile can then be calculated for each continuum flux measurement, assuming that the continuum flux tracks the ionizing flux from a central point source. A nonvariable narrow emission-line component is included in the modeled emission-line profiles, as is a smoothing parameter to account for the small differences in spectral resolution that arise from variable seeing conditions throughout the monitoring campaign.

To explore the full range of possible time delays arising from the BLR geometry and to properly compare the modeled emission line profiles with the measured profiles, the continuum light curve must be interpolated.  CARAMEL uses Gaussian processes to both interpolate between continuum flux measurements and to extrapolate the continuum light curve beyond the beginning and end of the monitoring campaign to extend the range of time delays that may be probed.  The uncertainties on the Gaussian process model parameters are included in the determination of the BLR model parameters, thus capturing the effect of the uncertainties that arise from interpolating and extrapolating the continuum data.

For each model realization, we include 2000 individual point particles to represent the BLR.  The continuum light curve is interpolated and model emission-line profiles are calculated for each epoch at which an emission-line measurement was acquired.  A Gaussian likelihood function compares the modeled spectra against the measured spectra and adjusts the model parameters accordingly. CARAMEL utilizes a diffusive nested sampling code, with the latest version employing DNEST4 [Brewer & Foreman-Mackey, 2018], to efficiently explore the model parameter space.  DNEST4 allows for the use of a likelihood softening parameter, or statistical temperature $T$, which has the effect of increasing the measurement uncertainties.  This parameter can account for underestimated measurement uncertainties or for the inability of the simplified model to capture all of the real details in the measurements.  The value of $T$ is determined in the post analysis by examining the distributions of the model parameters and choosing the largest value of $T$ for which the distributions remain smooth and generally unimodal.

Finally, to check that convergence had been reached, we compared the constrained values of the model parameters from the first half of the model runs to the second half of the model runs, with the total number of model runs being 10,000.  There was no discernible difference between the parameters constrained during the first half or second half of the model runs for either H$\beta$ or He2.

§ RESULTS

The top three panels display the data, one possible model, and residuals (data$-$model) for the H$\beta$ spectra, with epochs 1 and 13 and their model fits displayed immediately below to exemplify a low flux spectrum (magenta curve) and a high flux spectrum (cyan curve). The bottom two panels display the continuum and integrated H$\beta$ light curves as data points with model fits overlaid.  The full ranges of the models are displayed in light turquoise with the example model corresponding to the top four panels overlaid in dark turquoise.  Flux densities ($F_{\lambda}$) are in units of $10^{-15}$ erg s$^{-1}$ cm$^{-2}$ Å$^{-1}$ while integrated flux ($F$) is in units of $10^{-13}$ erg s$^{-1}$ cm$^{-2}$. Across the six panels, it is evident that most of the gross characteristics of the data are captured by the models, although some of the finer details are not. Furthermore, the continuum model is less well constrained during time periods with multi-day gaps between the measurements.  Unfortunately, these gaps resulted from the shutdown of numerous observatories in response to the global coronavirus pandemic and could not be avoided.

Modeling of the H$\beta$ emission line in NGC 3783 provides constraints on the low ionization BLR, while modeling of the He2 emission line constrains the high ionization BLR.  Figure <ref> compares the posterior probability distribution functions for all the parameters of the BLR models for both H$\beta$ and He2, while the median and 68% confidence intervals for each parameter are listed in Table <ref>.  We describe the resultant set of models for each emission line below.

§.§ H$\beta$

The models for H$\beta$ require a likelihood softening of $T=125$, which amounts to increasing the uncertainties on the data by a factor of $\sqrt{T} = 11.2$.  Figure <ref> displays the continuum and integrated H$\beta$ emission-line light curves and the observed H$\beta$ line profiles along with model fits to all of these.  In general, the emission line profiles are well-fit by the modeled profiles as are the gross flux variations of the integrated emission-line light curve, although some of the finer details of the data are not captured by the models.  The small disagreements between the data and the models could be the result of uncertainties that are still underestimated for some data points, or they could signal that the models are too simplistic and do not have the full flexibility needed to match all of the real variations, or both.

The geometry of the H$\beta$-emitting BLR is found to be a relatively face-on thick disk with an opening angle of $\theta_o=34.7^{+6.2}_{-9.9}$ degrees and an inclination to our line of sight of $\theta_i=17.9^{+5.3}_{-6.1}$ degrees.  The disk has an inner minimum radius of $r_{\rm min}=3.25^{+1.13}_{-1.54}$ light days with a median radius of $r_{\rm median}=10.07^{+1.10}_{-1.21}$ light days and a width of $\sigma_r=10.47^{+15.44}_{-3.82}$ light days.  The disk emission is distributed radially in a near-exponential profile ($\beta=0.95^{+0.25}_{-0.25}$), and is distributed throughout the thickness of the disk with a slight preference for stronger emission near the face of the disk ($\gamma=1.84^{+1.48}_{-0.67}$) and strong but not total obscuration along the midplane ($\xi=0.23^{+0.24}_{-0.15}$).  The line emission direction is rather unconstrained, with the median value centered around isotropic emission but having large uncertainties that do not discriminate between a preference for radiation towards or away from the central source ($\kappa=0.04^{+0.31}_{-0.30}$).  Figure <ref> displays a representative geometric model for the H$\beta$ response in the BLR of NGC 3783, drawn from the posterior probability distribution.

Representative geometric model for the H$\beta$ response in the broad line region of NGC 3783, drawn from the posterior probability distribution.  The left panel is oriented edge on, with an Earth-based observer on the +x axis, while the right panel shows the Earth-based observer's view.  The transparency of each point represents the relative response of the gas to continuum fluctuations at each position, with more opaque points responsible for a stronger response. This effect is most easily viewed in the right panel, where there is less overlap between points.

Transfer function $\Psi(\lambda,\tau)$ for the example H$\beta$ model displayed in Figure <ref>.  Integrating the transfer function over wavelength gives the one-dimensional lag profile $\Psi(\tau)$, which is shown on the right.  Integrating the transfer function over time delay gives $\Psi(\lambda)$, or the variable emission-line profile, which is shown immediately under the transfer function.  The bottom panel displays the average lag as a function of wavelength across the emission line, with the turquoise crosses showing the average time delay for 5 velocity bins across the H$\beta$ profile from Figure 6 of Bentz et al., 2021.

The associated mean and median time delays for H$\beta$ are found to be $\tau_{\rm mean}=9.05^{+0.68}_{-0.64}$ days and $\tau_{\rm median}=7.42^{+0.70}_{-0.74}$ days, which agree well with the average H$\beta$ time delay reported by Bentz et al., 2021 of $\tau_{\rm cent}=9.60^{+0.65}_{-0.72}$ days.  Figure <ref> displays the  transfer function, $\Psi(\lambda,\tau)$, for a representative model.  Also referred to as the velocity-delay map, the transfer function displays the range of H$\beta$ responsivities as a function of time delay and velocity (or wavelength) across the broad emission line profile.  The shape of the transfer function generally agrees with the cross-correlation time delays computed for different velocity bins of the H$\beta$ profile by Bentz et al., 2021, displayed here as the turquoise crosses in the bottom panel of Figure <ref>.

The black hole mass is constrained to be $\log_{10} (M_{\rm BH}/M_{\odot})=7.51^{+0.26}_{-0.13}$.  Roughly 60% of the particle orbits are near circular ($f_{\rm ellip}=0.60^{+0.09}_{-0.15})$, with the other 40% strongly preferring inflow ($f_{\rm flow}<0.5$).  With a low value of $\theta_e=16.1^{+18.6}_{-11.0}$ degrees, most of these are truly inflowing orbits rather than highly elliptical bound orbits.  There is also a small but non-zero contribution to the kinematics from turbulence ($\sigma_{\rm turb}=0.024^{+0.050}_{-0.021}$).

§.§ He2

Same as Figure <ref>, but for He2.

The models for He2 require a likelihood softening of $T=145$, which amounts to increasing the uncertainties on the data by a factor of $\sqrt{T} = 12.0$.  Figure <ref> displays the continuum and integrated He2 emission-line light curves and the observed He2 line profiles along with model fits to all of these.  In general, the modeled emission line profiles fit the main features of the observations, however the lower integrated flux and larger uncertainties compared to H$\beta$ do lead to somewhat less agreement between the observations and the models. The gross flux variations of the integrated emission-line light curve also seem to be mostly captured by the models.

The geometry of the He2-emitting BLR is again found to be a relatively face-on thick disk  with an opening angle of $\theta_o=23.5^{+11.8}_{-8.0}$ degrees and an inclination to our line of sight of $\theta_i=19.1^{+10.3}_{-7.0}$ degrees.  The disk has an inner minimum radius of $r_{\rm min}=1.00^{+0.46}_{-0.42}$ light days with a median radius of $r_{\rm median}=1.33^{+0.34}_{-0.42}$ light days and a width of $\sigma_r=0.17^{+0.34}_{-0.13}$ light days.  The disk emission is distributed radially in a Gaussian profile ($\beta=0.67^{+0.83}_{-0.45}$), although the constraints on this parameter are quite weak.  The distribution of emission throughout the thickness of the disk is also not well constrained ($\gamma=2.77^{+1.55}_{-1.23}$), but there is a preference for strong obscuration along the midplane ($\xi=0.08^{+0.23}_{-0.06}$).  The line emission slightly prefers radiation back towards the central source ($\kappa=-0.20^{+0.45}_{-0.24}$).
Figure <ref> displays a representative model for the He2 response in the BLR of NGC 3783, drawn from the posterior probability distribution.  As expected, it is significantly more compact than H$\beta$.

Same as Figure <ref>, but for He2.

Same as Figure <ref>, but for He2.

The associated mean and median time delays for He2 are found to be $\tau_{\rm mean}=1.19^{+0.28}_{-0.30}$ days and $\tau_{\rm median}=1.16^{+0.29}_{-0.32}$ days, which are a bit more compact but agree within the uncertainties with the average He2 time delay reported by Bentz et al., 2021 of $\tau_{\rm cent}=1.95^{+1.02}_{-0.98}$ days.  Figure <ref> displays the transfer function for a representative model.  The shape is much more asymmetric than was found for H$\beta$, with a heavier response in the blue wing and very little response in the red wing.

The black hole mass is constrained to be $\log_{10} (M_{\rm BH}/M_{\odot})=7.13^{+0.43}_{-0.37}$.  Only $1/5$ of the particle orbits are near circular ($f_{\rm ellip}=0.22^{+0.19}_{-0.16}$), while the rest of the orbits strongly preferring outflow ($f_{\rm flow}>0.5$).  With a low value of $\theta_e=14.6^{+11.8}_{-10.3}$ degrees, most of these are truly outflowing orbits rather than highly elliptical bound orbits.  Finally, there is again a small but non-zero contribution to the kinematics from turbulence ($\sigma_{\rm turb}=0.013^{+0.044}_{-0.011}$).

Constraints on the black hole mass in NGC 3783 from H$\beta$ (blue), He2 (red), and the joint inference using results from both emission lines (black).

Posterior distributions of the BLR model parameters for H$\beta$ before (turquoise, “unweighted”) and after (black, “weighted”) selecting only those models that agree with the joint constraint on . The unweighted distributions in turquoise are the same as the results for H$\beta$ in Figure <ref>, but are effectively smoothed with a Gaussian kernel for easier comparison with the weighted constraints.  The vertical dotted lines mark the median values, while the dashed vertical lines mark the 68% confidence intervals.  The parameters are generally unchanged when models that agree with the joint constraint on  are preferred.

Same as Figure <ref> but for He2. The most significant changes are for the BLR radius, which shifts to larger values, and the fraction of near-circular orbits, which decreases to smaller values.

§ DISCUSSION

While both emission lines were modeled independently, they arise from the same AGN and should agree on some parameters while possibly differing for others.   Comparing and contrasting the results for H$\beta$ and He2 in the context of other studies may thus shed additional light on the Seyfert nucleus of NGC 3783.

§.§ Black Hole Mass

The black hole mass of NGC 3783 is expected to be the same for both H$\beta$ and He2.  And indeed, we see that there is significant overlap between the two in the top left panel of Figure <ref>.  We investigated the joint inference on the black hole mass following the method described by Williams et al., 2020.  We first approximated the posterior probability distribution functions of each with a Gaussian kernel density estimate and then multiplied them together.  The result is shown in Figure <ref> and gives $\log_{10} (M_{\rm BH}/M_{\odot})=7.45^{+0.19}_{-0.11}$, or $=2.82^{+1.55}_{-0.63}\times10^7$ .  This is consistent with the simple reverberation constraint on the mass, $M_{\rm BH} = 2.34^{+0.43}_{-0.43} \times 10^7$ , or $\log_{10} (M_{\rm BH}/M_{\odot})=7.37^{+0.07}_{-0.09}$, which is based on the mean H$\beta$ time delay and line width and an assumed scale factor of $\langle f \rangle=4.82$.  We note that the uncertainties quoted for the simple mass constraint include only the measurement uncertainties on the time delay and line width, and do not include other potential uncertainties such as the object-to-object variation in the scale factor.

With a black hole mass constraint from the BLR models, we can infer a specific value of $f=6.0^{+3.5}_{-1.8}$ for NGC 3783 using the mean time delay and line width for H$\beta$.  Previous investigations [Pancoast et al., 2014, Grier et al., 2017, Williams et al., 2018] have found that $f$ scales most strongly with the inclination of the system, as expected because only the line of sight velocity component is measured.  NGC 3783 seems to follow the same trend that has previously been seen for other Seyferts, as the inclination angle constrained by the models together with the linear regression results of Williams et al., 2018 predict $f=6^{+16}_{-4}$, or $\log_{10} (f)=0.75^{+0.59}_{-0.61}$.  Thus, the good agreement between our mass constraint and the simple reverberation constraint arises from the inclination of NGC 3783 being close to the mean inclination value for the sample of local Seyferts, and so having an individual $f$ factor that is similar to the population average.

We can also investigate any changes to the distributions of model parameters that may arise from selecting only those models that agree with the joint H$\beta$ and He2 constraint on .  Figure <ref> shows the constraints on the model parameters for H$\beta$ before and after selecting only those models that agree with the joint  constraint.  The results are quite similar, which is unsurprising since the H$\beta$ models provided the strongest initial constraint on .  Figure <ref> shows the same but for He2.  In this case, we find that models that favor the joint constraint on  also favor a slightly larger radius, which makes sense since the joint constraint on  was at the upper end of the  mass distribution  for He2, and also favors an even smaller fraction ($\sim15$%) of bound near-circular orbits with the rest of the orbits outflowing.  No additional changes are seen in the distributions of the model parameters if we similarly constrain the inclination angle in addition to , in which case we find a joint constraint of $\theta_i=18.2^{+3.6}_{-5.5}$ degrees.

§.§ Geometry and Kinematics

The similarities between the inclinations and opening angle constraints for H$\beta$ and He2 support the interpretation that both emission lines are probing different regions of the same thick disk of gas.  While the median values of the opening angles might suggest that the H$\beta$ emitting region is more “puffed up” than the He2 emitting region, as might be expected for a bowl-shaped model of the BLR like that proposed by Goad et al., 2012, the large uncertainties on the He2 opening angle mean that the two values formally agree.  As expected from the differences in their mean time delays reported by Bentz et al., 2021, the He2-emitting region is significantly more compact and close to the central ionizing source than the H$\beta$-emitting region, demonstrating clear ionization stratification within the BLR  (e.g., Peterson, 1993 and references therein).  There is little to no overlap between the two, with $r_{\rm min}=3.25^{+1.13}_{-1.54}$ light days for H$\beta$ compared to $r_{\rm mean}=1.40^{+0.31}_{-0.42}$ light days and $\sigma_{\rm r}=0.17^{+0.34}_{-0.13}$ light days for He2 (see Figure <ref>).

The two regions of the BLR appear, however, to be dominated by different kinematics.  In the case of the H$\beta$ emitting region, the kinematics are dominated by near-circular orbits with some infall, whereas the He2 emitting region is dominated by outflow.  Repeated studies of the same AGN, such as NGC 5548 [Pancoast et al., 2014, Williams et al., 2020], have found that the best-fit kinematics can change from one reverberation dataset to another, so the different kinematics may not be indicating structural differences between the inner and outer BLR in NGC 3783, but rather transient effects (“weather”).  On the other hand, Korista & Goad, 2004 find that photoionization models predict He2 $\lambda 4686$ is preferentially emitted in the presence of an ionizing photon flux that is $\sim 300$ times stronger than H$\beta$.  While H$\beta$-emitting gas in the BLR has been shown to be fairly stable against radiation pressure [Netzer & Marziani, 2010], He2 is preferentially emitted from lower density gas [Korista & Goad, 2004] and may be more susceptible to radiation pressure forces.

It may be that a combination of weather and photoionization physics  explains the difference in kinematics between H$\beta$ and He2.  NGC 3783 has demonstrated possible evidence for changes in the structure of the BLR in the recent past.  Kriss et al., 2019 obtained UV spectra of NGC 3783 shortly after the discovery of a strong soft X-ray obscuring event was detected in 2016.  They interpret changes in the UV broad emission lines of NGC 3783 together with the appearance of new broad absorption lines as evidence that the BLR scale height may have collapsed following a period of low ionizing luminosity that began in 2013 and continued to 2016.  By late 2016, the luminosity had increased significantly and remained high through at least January 2018 [Kaastra et al., 2018], and could thus begin to drive changes in the structure of the BLR on the dynamical timescale ($\sim0.3$ years at a BLR radius of 2.0 light days, or 3 years at a radius of 10 light days).  The luminosity of NGC 3783 between early 2018 and early 2020, when our observing campaign began, is unknown, but the BLR may have still been in the process of recovering from the extended low-luminosity period observed in $2013-2016$.   And indeed, a rough comparison of the broad H$\beta$ profile in 2020 with the profiles observed in 2011 and 2016 (Fig. 18; Kriss et al., 2019) suggests that much of the flux deficit observed in the line core in 2016 has filled in, although the line profile has not fully returned to its 2011 state.  Further multiwavelength monitoring coupled with  velocity-resolved reverberation analyses could help to inform our understanding of structural changes in the BLR as a result of large changes in the ionizing luminosity.

Combined representative geometric models for the H$\beta$ response (blue) and He2 response (red) in the broad line region of NGC 3783.  The left panel is oriented edge on, with an Earth-based observer on the +x axis, while the right panel shows the Earth-based observer's view.  The transparency of each point represents the relative response of the gas to continuum fluctuations at each position, with more opaque points responsible for a stronger response.

Several studies of NGC 3783 have focused on attempts to model the accretion disk using the Fe K$\alpha$ emission line or the continuum emission [Brenneman et al., 2011, Patrick et al., 2011, Capellupo et al., 2017] and have found similar relatively face-on inclinations for the inner accretion disk, even when they disagree on other components of the models (such as the black hole spin).  A similar inclination angle has also been found by modeling the three-dimensional structure of the spatially-resolved narrow line region on parsec scales [Fischer et al., 2013]. The consistency in inclination angles from the innermost regions of the accretion disk through the broad line region and the outermost narrow line region suggests that the spin axis of the central black hole has been stable for quite some time.  With no evidence for large torques on the spin, and with the black hole spin axis apparently matching the rotation axis of this relatively face-on galaxy, the recent evolution of the supermassive black hole appears to be dominated by secular processes that are aligned with the disk of the galaxy.

The best-fit models that we find for H$\beta$ also agree well with recent interferometry results for NGC 3783 from GRAVITY [Gravity Collaboration et al., 2021], in which measurements of the broad Br$\gamma$ emission are best described by a rotating thick disk inclined at $\sim 20^{\circ}$ to our line of sight and surrounding a black hole with $\log_{10} (M_{\rm BH}/M_{\odot})=7.68^{+0.45}_{-0.43}$.  Additionally, the radial extent of the Br$\gamma$-emitting region ($r_{\rm mean} = 16^{+12}_{-5}$ light days assuming $D=38.5$ Mpc) is in good agreement with the radial extent of the H$\beta$ emitting region ($r_{\rm mean}=11.4^{+1.1}_{-1.1}$ light days;  see Table <ref>).  A joint analysis of the GRAVITY observations with the continuum and integrated H$\beta$ light curves from Bentz et al., 2021 confirms and improves upon the results, with $M_{\rm BH}=2.54^{+0.90}_{-0.72}\times10^7$ , or $\log_{10} (M_{\rm BH}/M_{\odot})=7.40^{+0.13}_{-0.14}$, and $r_{\rm median}=16.2^{+2.8}_{-1.8}$ light days [Gravity Collaboration et al., 2021]. The black hole mass is still in excellent agreement with our findings, while the stronger constraints on the BLR radius in the joint analysis are somewhat in tension with the size of the BLR reported here ($r_{\rm median}=10.07^{+1.10}_{-1.21}$ light days). It is important to recognize that the GRAVITY results depend on the distance to NGC 3783 which is somewhat uncertain (recent studies suggest values of $35-50$ Mpc; Kourkchi et al., 2020, Robinson et al., 2021), reverberation mapping measures a responsivity-weighted radius while interferometry measures a flux-weighted radius, and photoionization effects (which are ignored in both our models and those employed in the analysis of the GRAVITY data) are known to cause different reverberation time delays for different Hydrogen recombination lines (e.g., Bentz et al., 2010).  Despite these complicating factors, the good agreement between the results lends additional confidence to both.
Future work will investigate the joint constraints that may be derived from an analysis of the velocity-resolved reverberation data that we have presented here in tandem with the GRAVITY observations.

§ SUMMARY

We have modeled the full velocity-resolved response of the broad H$\beta$ and He2 emission lines in NGC 3783.  The results give a black hole mass constraint that is independent of any scaling factor, and a joint analysis of the results for the two emission lines prefers $=2.82^{+1.55}_{-0.63}\times10^7$ .  The geometry of the BLR is found to be a thick disk that is close to face on ($\theta_i\approx 18^{\circ}$) and exhibiting clear ionization stratification, with H$\beta$ arising from an extended region of $~\sim 3-20$ light-days, while He2 arises from a significantly more compact and centralized region of $1-2$ light days.  The kinematics of the outer BLR probed by H$\beta$ are dominated by near-circular orbits with a contribution from infall, whereas the kinematics of the inner BLR probed by He2 are dominated by an unbound outflow.  Given the recent history of a deficit of ionizing radiation in NGC 3783 that was observed from $2013-2016$, and the hypothesis that the BLR height collapsed as a result, it is possible that we may be seeing the BLR undergoing structural changes as it recovers.

We thank the anonymous referee for suggestions that improved the presentation of this work.
We also thank Kate Grier for helpful conversations about CARAMEL.
MCB gratefully acknowledges support from the NSF through grant AST-2009230.
TT and PRW acknowledge support by the Packard Foundation through a Packard Research Fellowship to TT and from NSF through grant NSF-AST-1907208. PRW acknowledges support from the UCLA graduate division through a Dissertation Year Fellowship.

ULySS [Koleva et al., 2011], CARAMEL [Pancoast et al., 2014]

[Alard, 2000]
Alard, C. 2000, , 144, 363, 10.1051/aas:2000214

[Alard & Lupton, 1998]
Alard, C., & Lupton, R. H. 1998, , 503, 325, 10.1086/305984

[Anderson et al., 2021]
Anderson, M. D., Baron, F., & Bentz, M. C. 2021, ,

[Bahcall et al., 1972]
Bahcall, J. N., Kozlovsky, B.-Z., & Salpeter, E. E. 1972, , 171,
467, 10.1086/151300

[Batiste et al., 2017]
Batiste, M., Bentz, M. C., Raimundo, S. I., Vestergaard, M., &
Onken, C. A. 2017, , 838, L10, 10.3847/2041-8213/aa6571

[Bentz et al., 2010]
Bentz, M. C., Horne, K., Barth, A. J., et al. 2010a,
, 720, L46, 10.1088/2041-8205/720/1/L46

[Bentz et al., 2021]
Bentz, M. C., Street, R., Onken, C. A., & Valluri, M. 2021, , 906,
50, 10.3847/1538-4357/abccd4

[Bentz et al., 2009]
Bentz, M. C., Walsh, J. L., Barth, A. J., et al. 2009, , 705, 199,

[Bentz et al., 2010]
—. 2010b, , 716, 993, 10.1088/0004-637X/716/2/993

[Bentz et al., 2008]
Bentz, M. C., et al. 2008, , 689, L21, 10.1086/595719

[Blandford & McKee, 1982]
Blandford, R. D., & McKee, C. F. 1982, , 255, 419

[Brenneman et al., 2011]
Brenneman, L. W., Reynolds, C. S., Nowak, M. A., et al. 2011, ,
736, 103, 10.1088/0004-637X/736/2/103

[Brewer & Foreman-Mackey, 2018]
Brewer, B. J., & Foreman-Mackey, D. 2018, Journal of Statistical Software,
Articles, 86, 1, 10.18637/jss.v086.i07

[Cackett et al., 2021]
Cackett, E. M., Bentz, M. C., & Kara, E. 2021, iScience, 24, 102557,

[Capellupo et al., 2017]
Capellupo, D. M., Wafflard-Fernandez, G., & Haggard, D. 2017, ,
836, L8, 10.3847/2041-8213/aa5cac

[Denney et al., 2009]
Denney, K. D., Peterson, B. M., Pogge, R. W., et al. 2009, , 704,
L80, 10.1088/0004-637X/704/2/L80

[Event Horizon Telescope Collaboration et al., 2019]
Event Horizon Telescope Collaboration, Akiyama, K., Alberdi, A., et al.
2019, , 875, L6, 10.3847/2041-8213/ab1141

[Ferrarese & Merritt, 2000]
Ferrarese, L., & Merritt, D. 2000, , 539, L9

[Fischer et al., 2013]
Fischer, T. C., Crenshaw, D. M., Kraemer, S. B., & Schmitt, H. R.
2013, , 209, 1, 10.1088/0067-0049/209/1/1

[Gebhardt et al., 2000]
Gebhardt, K., Bender, R., Bower, G., et al. 2000, , 539, L13

[Genzel et al., 2000]
Genzel, R., Pichon, C., Eckart, A., Gerhard, O. E., & Ott, T. 2000,
, 317, 348, 10.1046/j.1365-8711.2000.03582.x

[Ghez et al., 2000]
Ghez, A. M., Morris, M., Becklin, E. E., Tanner, A., & Kremenek, T.
2000, , 407, 349, 10.1038/407349a

[Ghez et al., 2008]
Ghez, A. M., Salim, S., Weinberg, N. N., et al. 2008, , 689, 1044,

[Goad et al., 2012]
Goad, M. R., Korista, K. T., & Ruff, A. J. 2012, , 426, 3086,

[Gravity Collaboration et al., 2021]
Gravity Collaboration, Amorim, A., Bauböck, M., et al.
2021a, , 648, A117, 10.1051/0004-6361/202040061

[Gravity Collaboration et al., 2021]
—. 2021b, , submitted

[Grier et al., 2013]
Grier, C. J., Martini, P., Watson, L. C., et al. 2013, , 773, 90,

[Grier et al., 2017]
Grier, C. J., Pancoast, A., Barth, A. J., et al. 2017, , 849, 146,

[Grier et al., 2012]
Grier, C. J., Peterson, B. M., Pogge, R. W., et al. 2012, , 755,
60, 10.1088/0004-637X/755/1/60

[Gültekin et al., 2009]
Gültekin, K., Richstone, D. O., Gebhardt, K., et al. 2009, ,
698, 198, 10.1088/0004-637X/698/1/198

[Horne, 1994]
Horne, K. 1994, in Astronomical Society of the Pacific Conference Series,
Vol. 69, Reverberation Mapping of the Broad-Line Region in Active Galactic
Nuclei, ed. P. M. Gondhalekar, K. Horne, & B. M. Peterson, 23

[Horne et al., 2021]
Horne, K., De Rosa, G., Peterson, B. M., et al. 2021, , 907, 76,

[Kaastra et al., 2018]
Kaastra, J. S., Mehdipour, M., Behar, E., et al. 2018, , 619, A112,

[Koleva et al., 2009]
Koleva, M., Prugniel, P., Bouchard, A., & Wu, Y. 2009, , 501,
1269, 10.1051/0004-6361/200811467

[Koleva et al., 2011]
—. 2011, ULySS: A Full Spectrum Fitting Package.

[Korista & Goad, 2004]
Korista, K. T., & Goad, M. R. 2004, , 606, 749

[Kormendy & Ho, 2013]
Kormendy, J., & Ho, L. C. 2013, , 51, 511,

[Kourkchi et al., 2020]
Kourkchi, E., Courtois, H. M., Graziani, R., et al. 2020, , 159, 67,

[Kriss et al., 2019]
Kriss, G. A., Mehdipour, M., Kaastra, J. S., et al. 2019, , 621,
A12, 10.1051/0004-6361/201834326

[Magorrian et al., 1998]
Magorrian, J., Tremaine, S., Richstone, D., et al. 1998, , 115, 2285

[Michell, 1784]
Michell, J. 1784, Philosophical Transactions of the Royal Society of London
Series I, 74, 35

[Netzer & Marziani, 2010]
Netzer, H., & Marziani, P. 2010, , 724, 318,

[Onken et al., 2004]
Onken, C. A., Ferrarese, L., Merritt, D., et al. 2004, , 615, 645,

[Pancoast et al., 2011]
Pancoast, A., Brewer, B. J., & Treu, T. 2011, , 730, 139,

[Pancoast et al., 2014]
—. 2014a, , 445, 3055, 10.1093/mnras/stu1809

[Pancoast et al., 2014]
Pancoast, A., Brewer, B. J., Treu, T., et al. 2014b,
, 445, 3073, 10.1093/mnras/stu1419

[Park et al., 2012]
Park, D., Kelly, B. C., Woo, J.-H., & Treu, T. 2012, , 203, 6,

[Patrick et al., 2011]
Patrick, A. R., Reeves, J. N., Lobban, A. P., Porquet, D., &
Markowitz, A. G. 2011, , 416, 2725,

[Peterson, 1993]
Peterson, B. M. 1993, , 105, 247

[Peterson et al., 2013]
Peterson, B. M., Denney, K. D., De Rosa, G., et al. 2013, , 779,
109, 10.1088/0004-637X/779/2/109

[Peterson & Wandel, 1999]
Peterson, B. M., & Wandel, A. 1999, , 521, L95

[Peterson & Wandel, 2000]
—. 2000, , 540, L13

[Robinson et al., 2021]
Robinson, J. H., Bentz, M. C., Courtois, H. M., et al. 2021, , 912,
160, 10.3847/1538-4357/abedaa

[Skielboe et al., 2015]
Skielboe, A., Pancoast, A., Treu, T., et al. 2015, , 454, 144,

[Vazdekis et al., 2010]
Vazdekis, A., Sánchez-Blázquez, P., Falcón-Barroso, J.,
et al. 2010, , 404, 1639, 10.1111/j.1365-2966.2010.16407.x

[Williams et al., 2018]
Williams, P. R., Pancoast, A., Treu, T., et al. 2018, , 866, 75,

[Williams et al., 2020]
—. 2020, , 902, 74, 10.3847/1538-4357/abbad7


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

Human-in-the-loop data curation workshop at ACM CIKM 2022, Oct 17–21, 2022,
Atlanta, GA

<EMAIL_ADDRESS>] [1] [1]

<EMAIL_ADDRESS>] [1] [1]

[1]Corresponding author. [1]KBM and RS contributed equally to the majority of
this research. Additional authors contributed to specific aspects including
initial models, data sets and project or platform development.

# From fat droplets to floating forests: cross-domain transfer learning using
a PatchGAN-based segmentation model

Kameswara Bharadwaj Mantha  School of Physics & Astronomy, University of
Minnesota, Twin Cities, 116 Church St SE, Minneapolis, MN, 55455  Ramanakumar
Sankar  Yuping Zheng  Lucy Fortson  Thomas Pengo  University of Minnesota
Informatics Institute, 2231 6th St SE, Minneapolis, MN, 55455  Douglas Mashek
Medical School, University of Minnesota, Twin Cities, 420 Delaware Street SE,
Minneapolis, MN, 55455  Mark Sanders  Trace Christensen  Jeffrey Salisbury
Mayo Clinic, 200 First Street SW, Rochester, MN, 55905  Laura Trouille  Adler
Planetarium, 1300 S DuSable Lake Shore Dr., Chicago, IL 60605  Jarrett E. K.
Byrnes  Isaac Rosenthal  Department of Biology, University of Massachusetts
Boston 100 Morrissey Blvd; Boston, MA, 02125  Henry Houskeeper  Kyle Cavanaugh
Department of Geography, University of California Los Angeles, Los Angeles, CA
90095

(2022)

###### Abstract

Many scientific domains gather sufficient labels to train machine algorithms
through human-in-the-loop techniques provided by the Zooniverse.org citizen
science platform. As the range of projects, task types and data rates
increase, acceleration of model training is of paramount concern to focus
volunteer effort where most needed. The application of Transfer Learning (TL)
between Zooniverse projects holds promise as a solution. However,
understanding the effectiveness of TL approaches that pretrain on large-scale
generic image sets vs. images with similar characteristics possibly from
similar tasks is an open challenge. We apply a generative segmentation model
on two Zooniverse project-based data sets: (1) to identify fat droplets in
liver cells (FatChecker; FC) and (2) the identification of kelp beds in
satellite images (Floating Forests; FF) through transfer learning from the
first project. We compare and contrast its performance with a TL model based
on the COCO image set, and subsequently with baseline counterparts. We find
that both the FC and COCO TL models perform better than the baseline cases
when using $>75\%$ of the original training sample size. The COCO-based TL
model generally performs better than the FC-based one, likely due to its
generalized features. Our investigations provide important insights into usage
of TL approaches on multi-domain data hosted across different Zooniverse
projects, enabling future projects to accelerate task completion.

###### keywords:

datasets generative adversarial neural networks UNET generator patch-based
discriminator focal tversky loss transfer learning

##  1 Introduction

Citizen Science has established itself as a valuable method for distributed
data analysis enabling research teams from diverse domains to solve problems
involving large quantities of data with complexity levels requiring human
pattern recognition capabilities [1, 2]. As the largest citizen science
platform, Zooniverse.org has enabled over 2.5 million volunteers to provide
over half a billion annotations on hundreds of projects across the sciences
and humanities. Many of these projects use the resulting labels to train
machine learning algorithms typically training models from scratch e.g., [3,
4, 5, 6, 7, 8, 9, 10, 11]. To accelerate labeling efficiencies across the
platform, the Zooniverse human-machine system should take advantage of
transfer learning techniques, especially when volunteer engagement is at a
premium. When applying transfer learning, a new project would require fewer
labels from volunteers to achieve the same performance as training a model
from scratch. Volunteer labelers would thus be able to focus on tasks more
suited to humans such as anomaly detection e.g., [12].

Transfer learning (TL) is an established approach, where the feature space
from a pretrained model can be transferred to another framework and fine tuned
to perform analogous or different tasks. Feature extraction is typically
performed using Deep Convolutional Neural Networks (CNNs) such as [13, 14].
Transfer learning generally uses models trained on data that is either “out-
of-domain” (i.e., training data characteristics are different from data at
hand) or “in-domain” (data that are similar or closely relatable to the data
at hand). Quantifying the gains provided by these different TL approaches is
an active area of research, where studies find several factors to be at play
that govern its effectiveness: Accuracy and architecture choice of the
pretrained model [15], robustness of model to input adversarial noise [16],
and type of task to which the TL is being applied [17]. Recent works (e.g.,
[12, 8]) have demonstrated that transfer learning from a model pretrained on
in-domain data performs better than transfer learning from out-of-domain data.
On the other hand, some studies find that TL models based on out-of-domain
data (e.g., ImageNet or COCO datasets) perform on par with or better than the
in-domain TL models [18, 19].

In order to leverage the Zooniverse’s large library of image-label pairs
across multiple domains, there is thus a clear need to better understand the
effectiveness of cross-domain transfer learning. In particular, we are
interested in the application of transfer learning specifically to projects
that share task similarity across a wide range of domains. For example, image
segmentation tasks vary across vastly different disciplines, from cell biology
to satellite imagery. Frameworks such as the U-Net [20], Recurrent
Convolutional Networks such as Mask-RCNNs [21], and Generative Adversarial
Networks (GANs; e.g., [22, 23]) have been used to perform such object
segmentation across multiple domains and data sets. However, robust learning
of such segmentation models from scratch often requires large annotated
training samples that may not be available (e.g., medical imaging), which can
lead to poor generalizability of the learnt features to newer data, even in
related domains. While Zooniverse can provide these large annotation sets per
project, this comes at the cost of volunteer effort which we seek to optimize.

In an effort to increase project completion rates, this study investigates
potential machine performance gains through transfer learning across domains
by leveraging the shared task similarity between Zooniverse projects. We use a
PatchGAN-based [23] segmentation
model111https://github.com/ramanakumars/patchGAN/ to investigate the
effectiveness of segmenting kelp beds from satellite images. Particularly, we
test transfer learning from the COCO dataset (i.e., out-of-domain) and
microscopy imaging of lipid droplets in liver cells (pseudo-in-domain) and
compare them to their corresponding “trained from scratch” counterparts.

##  2 Methods

In this section, we detail our PatchGAN architecture [23], the training and
testing data and its preparation, and the description of the five models
analyzed in our work.

###  2.1 PatchGAN Framework

The implemented PatchGAN framework is inherited from the Pix2Pix GAN
architecture in [23], which is a conditional GAN for realizing paired image-
to-image translation. The PatchGAN architecture consists of a Generator ($G$)
and Discriminator ($D$):

The generator is composed of a U-Net [20], a U-shaped encoder-decoder neural
network, with skip connections across the bottleneck layer (Figure 1). The
encoder (decoder) comprises of $6$ downsampling (upsampling) blocks, each
consisting of $4\times 4$ convolution (transposed convolution), Leaky ReLU
activation, and a batch normalization layer. All the blocks in the inner
layers of the network also include a dropout layer which omits 50% of the
extracted features during training. The outputs of the transposed convolutions
are also concatenated with the corresponding skip connection feature map from
the encoder block.

Figure 1: U-Net Generator (top) and Discriminator (bottom) of our PatchGAN
framework.

The discriminator is a patch-wise binary classifier that takes a concatenation
of the input image and its corresponding ground truth or generated mask and
outputs a $30\times 30$ probability matrix. Each unit of this matrix
represents a 70 $\times$ 70 patch of the input image, and provides the
probability that the patch is real.

###  2.2 Data

For this study, we use three sources for our image-mask pairs: the Floating
Forests dataset, Etch-a-Cell dataset and the COCO-stuff. The former two are
Zooniverse projects focusing on image segmentation, while the latter
represents a generic image dataset that is used in computer vision,
representing an out-of-domain dataset compared to the former two. These three
data sources represent a diverse feature set on which to perform our transfer
learning experiment. Figure 2 shows an example of an image-mask pair from each
dataset.

####  2.2.1 Floating Forests ($FF$)

Floating Forests is an ecology-based citizen science project hosted on
Zooniverse.org222https://www.zooniverse.org/projects/zooniverse/floating-
forests/ to identify kelp beds in Landsat imagery. The project presents
segments of Landsat data to Zooniverse volunteers, who draw outlines around
the kelp beds. These annotations are aggregated using a pixel-by-pixel
consensus to create masks of the kelp beds in the corresponding Landsat
segments. We use 4 channels from the Landsat data (Blue, Green, Red and near
Infrared) to train the patchGAN on the image-mask pairs. This FF data
comprises 6,967 ($350\times 350$ pix) image-mask pairs. We pre-process these
data such that each pair is cropped into four $256\times 256$ overlapping
cutouts, and augment each crop 5 times (rotation and flipping). This resulted
in $118,440$ training and $4180$ testing images.

####  2.2.2 Etch-a-Cell: Fat Checker ($FC$)

Etch-a-Cell: Fat Checker is a cell biology project hosted on
Zooniverse.org333https://www.zooniverse.org/projects/dwright04/etch-a-cell-
fat-checker to identify lipid droplets in electron microscopy data. The
Zooniverse project presents 2D slices of the data to volunteers who annotate
the outline of the lipid droplet. The lipid mask is generated by aggregating
the annotations by multiple volunteers based on consensus. The data set
consists of $2341$ image-mask pairs and each image is $1200\times 1200\,{\rm
pix}$ in shape, with 3 channels. We split the sample into $2,106$ training and
$235$ testing sets. We transform these images and masks to work with our
PatchGAN framework by resizing them to $512\times 512$ pix and generating five
crops (four corners and one center crop). We further augment them by applying
three rotations ($90,180,270\,{\rm deg}$) per image, yielding augmented
training and testing samples of $42120$ and $4700$ images, respectively.

####  2.2.3 COCO-Stuff

The Common Objects in COntext (COCO; [24]) is a large collection of several
real-world images with objects set in various simple to complex scenes, which
are annotated by outlines444https://github.com/nightrome/cocostuff. [25]
further processed the COCO data set to produce dense pixel-wise annotations
for them (the COCO-Stuff data set; hereafter COCO). These images and annotated
masks vary widely in their shapes, and therefore, we standardize these images
by resizing them to a $256\times 256$ pix shape. For our PatchGAN training, we
limit the training and testing data to those that host the ‘person’ class.
This amounts to $63785$ training and $2673$ testing image-mask pairs.

Figure 2: Visualization of example input image, truth mask, and patchGAN
predicted output mask.

###  2.3 Experimental Design

In this work, we investigate the potential of cross-domain transfer learning
by training $5$ models. The first $3$ models are trained from scratch –
$\Lambda_{FF}$, $\Lambda_{FC}$, and $\Lambda_{COCO}$ – using $100\%$ of their
corresponding data sets $FF$, $FC$, and $COCO$, respectively. Next, we train
the $\Lambda_{FC\rightarrow FF}$ and $\Lambda_{COCO\rightarrow FF}$ by
transferring the weights from the trained $\Lambda_{FC}$ and $\Lambda_{COCO}$
models to the $\Lambda_{FF}$. By comparing between the baseline $\Lambda_{FF}$
to the transfer learnt models $\Lambda_{FC\rightarrow FF}$ and
$\Lambda_{COCO\rightarrow FF}$, we quantify the impact of performing transfer
learning on the accelerated learning of the $\Lambda_{FF}$ model from two
distinct feature initializations. During this transfer learning exercise, we
also vary the amount of training data used from $10\%$-$100\%$.

##  3 Training & Results

In this section, we outline the training strategy and provide details of the
hyper parameters. We also present the results of our training and discuss the
outcomes of our transfer learning exercise.

###  3.1 Training Strategy

Our $\Lambda_{FF}$, $\Lambda_{FC}$, and $\Lambda_{COCO}$ models have been
trained for $50$ epochs. For the generator, we use the Focal Tversky Loss
(FTL; [26]), which is a generalized version of the Tversky Loss (TL) defined
in terms of the Tversky Index (TI) as:

$\begin{split}TI=\frac{TP}{TP+\alpha FN+\beta FP}\rightarrow
TL=(1-TI)\rightarrow FTL=(TL)^{\gamma},\end{split}$ (1)

For our training, we use $\alpha=0.7$ and $\beta=0.3$. The $\gamma$ parameter
controls the non-linearity of the TL with respect to the $TI$, enabling the
learning to focus on easier ($\gamma<1$) vs. harder ($\gamma>1$) examples. We
use $\gamma=0.75$ during our training. For the discriminator optimization, we
use the Binary Cross-Entropy (BCE) loss. Specifically, our total discriminator
loss is the average of two components: the discriminator applied on the
generated mask (i.e., against a fake label), and applied on the true mask
(i.e., the real label). For both the generator and discriminator, we use the
Adam optimizer with an initial learning rate $5\times 10^{-4}$ and $1\times
10^{-4}$ respectively, decayed exponentially by $\tau=0.95$, applied every 5
epochs.

Figure 3: Comparison of generated mask from different model runs on the
Floating Forests data, showing different performance gains from transfer
learning.

###  3.2 Transfer learning strategy

For our transfer learning based model training of $\Lambda_{FC\rightarrow FF}$
and $\Lambda_{COCO\rightarrow FF}$, we load the weights of the $\Lambda_{FC}$
and $\Lambda_{COCO}$ models into the freshly initialized $\Lambda_{FF}$ model
architecture. To account for the $3$ vs $4$ channel mismatch between the
$\Lambda_{COCO}$, $\Lambda_{FC}$ and $\Lambda_{FF}$, we load model layer
parameters excluding the input layer. For each model, we train 5 different
versions, using random subsets of $10\%,25\%,50\%,75\%$ and $100\%$ of the
full Floating Forests data, to compare TL efficiency gains from having a
smaller dataset. For these experiments, we also use only the first 6,967 un-
augmented images for re-training. We train the $\Lambda_{FC\rightarrow FF}$
and $\Lambda_{COCO\rightarrow FF}$ models with the same hyper-parameter
settings as the aforementioned “from scratch” models for $50$ epochs.

Figure 4: Comparison of mean final loss on Floating Forests validation data
across the different models.

###  3.3 Results and discussion

We find that our $\Lambda_{FF}$, $\Lambda_{FC}$ and $\Lambda_{COCO}$ generally
predict the annotation masks reasonably well (Figure 2), qualitatively
matching with the ground truths. Figures 3 and 4 show our transfer learning
results. In Figure 4, we show our average validation loss for the different
model training runs. As expected, larger training samples provide much better
performance, but we also find that the model pretrained on the COCO dataset
provides noticeably better performance on the Floating Forests data, compared
to both $\Lambda_{FC\rightarrow FF}$ and also $\Lambda_{FF}$. In fact, the
$\Lambda_{COCO\rightarrow FF}$ is able to match the performance of the
$\Lambda_{FF}$ model with between 50-75% of the training Floating Forests
dataset.

In Figure 3, we show examples highlighting the difference between the
generated masks from $\Lambda_{FF}$ and corresponding masks from
$\Lambda_{FC\rightarrow FF}$ and $\Lambda_{COCO\rightarrow FF}$. The sharpness
of the kelp beds is poorly reconstructed by the $\Lambda_{FF}$ model but is
well captured by the transfer learnt models (particularly when training
$\Lambda_{COCO\rightarrow FF}$ with more than 75% of the original data). The
transfer learnt models are also better at capturing kelp beds not identified
in the original consensus data. For example, both the ground truth and
$\Lambda_{FF}$ fail to reveal the kelp beds in the top left of the image, but
these are picked up well by the transfer learnt models.

This is likely due to the large diversity of the features in the COCO dataset,
making it a much more robust feature extraction network to transfer learn
from. Indeed, compared to $\Lambda_{FC\rightarrow FF}$, the
$\Lambda_{COCO\rightarrow FF}$ model-detected kelp beds are qualitatively
better visually (e.g., Figure 3), especially at lower training data sizes.
This is likely compounded with the lower feature diversity in both the
Floating Forest and Fat Checker data sets, given the fewer number of samples
in the training data and low variety in target classes.

####  3.3.1 Transfer learning approaches for citizen science datasets

For the Zooniverse platform, this study provides an avenue to build quick
access for projects to use machine learning frameworks for simple tasks (e.g.,
image segmentation), by transfer learning from existing models on a small
sample of volunteer annotated data sets. However, despite the results
presented here, there are still several key questions which need to be
answered:

Domain dependency: It is unclear how much of the performance gained from COCO
was a ‘global truth’. That is, whether COCO (or similarly diverse datasets)
are immediately applicable to out-of-domain data, for all domains, or if there
are domain-specific restrictions which allow these performance gains to occur
on data such as Floating Forests. This requires more experiments with
increasingly different data sets on Zooniverse to investigate the range of
performance gains possible.

Task dependency:  Previous studies on transfer learning across domains show
significant variations in performance across different task types. For
example, image classification tasks (e.g., [12, 17]) show lower gains than
image segmentation based tasks (e.g., [18]). We need to further investigate
the inherent difficulty associated with different tasks on Zooniverse
projects, and how effectively they can be transferred between domains. [12],
for example, show that significant boosts to performance is only provided by
using in-domain transfer learning.

Target data purity:  For Zooniverse projects, data labels are generally
provided by volunteers and are aggregated based on volunteer consensus. In
this study, we found that transfer learning can help mitigate data purity
effects, since transfer learnt feature extraction models are generally robust
to mislabeled data. The extent to which transfer learning models are sensitive
to data purity effects needs to be further investigated.

In conclusion, we find that transfer learning can provide a significant boost
to projects that contain similar tasks on Zooniverse. However, the extent to
which this can be generalized across the full Zooniverse ecosystem is a
question of ongoing study.

## Acknowledgements

The authors would like to thank the Zooniverse volunteers without whom this
work would not have been possible. RS, KM, LF, YZ, LT would like to
acknowledge partial support from the National Science Foundation under grant
numbers IIS $2006894$ and OAC $1835530$. Partial support by RS, KM, LF, TP,
MS, TC, JS is acknowledged through Minnesota Partnership MNP IF#119.09.

## References

  * Trouille et al. [2019] L. Trouille, C. J. Lintott, L. F. Fortson,  Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human–machine systems,  Proceedings of the National Academy of Sciences 116 (2019) 1902–1909. URL: https://www.pnas.org/content/116/6/1902. doi:10.1073/pnas.1807190116.
  * Fortson et al. [2018] L. Fortson, D. Wright, C. Lintott, L. Trouille,  Optimizing the human-machine partnership with Zooniverse,  in: CI 2018: ACM Collective Intelligence, ACM, http://arxiv.org/abs/1809.09738, 2018\. URL: http://arxiv.org/abs/1809.09738. arXiv:1809.09738.
  * Beaumont et al. [2014] C. N. Beaumont, A. A. Goodman, S. Kendrew, J. P. Williams, R. Simpson,  The Milky Way Project: Leveraging Citizen Science and Machine Learning to Detect Interstellar Bubbles,  ApJS 214 (2014) 3\. doi:10.1088/0067-0049/214/1/3. arXiv:1406.2692.
  * Zevin et al. [2017] M. Zevin, S. Coughlin, S. Bahaadini, E. Besler, N. Rohani, S. Allen, M. Cabero, K. Crowston, A. K. Katsaggelos, S. L. Larson, et al.,  Gravity Spy: integrating advanced ligo detector characterization, machine learning, and citizen science,  Classical and Quantum Gravity 34 (2017) 064003.
  * Norouzzadeh et al. [2017] M. Norouzzadeh, A. Nguyen, M. Kosmala, A. Swanson, C. Packer, J. Clune,  Automatically identifying wild animals in camera trap images with deep learning,  arXiv preprint arXiv:1703.05830 (2017).
  * Wright et al. [2017] D. Wright, C. Lintott, S. Smartt, K. Smith, L. Fortson, L. Trouille, C. Allen, M. Beck, M. Bouslog, A. Boyer, K. Chambers, H. Flewelling, W. Granger, E. Magnier, A. McMaster, G. Miller, J. O’Donnell, B. Simmons, H. Spiers, J. Tonry, M. Veldthuis, R. Wainscoat, C. Waters, M. Willman, Z. Wolfenbarger, D. Young,  A transient search using combined human and machine classifications,  Monthly Notices of the Royal Astronomical Society 472 (2017) 1315–1323. URL: http://dx.doi.org/10.1093/mnras/stx1812. doi:10.1093/mnras/stx1812.
  * Domínguez Sánchez et al. [2018] H. Domínguez Sánchez, M. Huertas-Company, M. Bernardi, D. Tuccillo, J. L. Fischer,  Improving galaxy morphologies for SDSS with Deep Learning,  Monthly Notices of the Royal Astronomical Society 476 (2018) 3661–3676. URL: https://doi.org/10.1093/mnras/sty338. doi:10.1093/mnras/sty338.
  * Willi et al. [2019] M. Willi, R. T. Pitman, A. W. Cardoso, C. Locke, A. Swanson, A. Boyer, M. Veldthuis, L. Fortson,  Identifying animal species in camera trap images using deep learning and citizen science,  Methods in Ecology and Evolution 10 (2019) 80–91.
  * Laraia et al. [2019] M. Laraia, D. Wright, H. Dickinson, A. Simenstad, K. Flanagan, S. Serjeant, L. Fortson, VERITAS Collaboration,  Muon Hunter 2.0: efficient crowdsourcing of labels for IACT image analysis,  in: 36th International Cosmic Ray Conference (ICRC2019), volume 36 of International Cosmic Ray Conference, 2019, p. 678.
  * Ranadive et al. [2020] O. Ranadive, S. van der Lee, V. Tang, K. Chao,  Applying Machine Learning to Crowd-sourced Data from Earthquake Detective,  arXiv e-prints (2020) arXiv:2011.04740. arXiv:2011.04740.
  * Spiers et al. [2020] H. Spiers, H. Songhurst, L. Nightingale, J. de Folter, R. Hutchings, C. J. Peddie, A. Weston, A. Strange, S. Hindmarsh, C. Lintott, L. M. Collinson, M. L. Jones,  Citizen science, cells and cnns – deep learning for automatic segmentation of the nuclear envelope in electron microscopy data, trained with volunteer segmentations,  bioRxiv (2020). URL: https://www.biorxiv.org/content/early/2020/07/29/2020.07.28.223024. doi:10.1101/2020.07.28.223024.
  * Walmsley et al. [2022] M. Walmsley, A. M. M. Scaife, C. Lintott, M. Lochner, V. Etsebeth, T. Géron, H. Dickinson, L. Fortson, S. Kruk, K. L. Masters, K. B. Mantha, B. D. Simmons,  Practical galaxy morphology tools from deep supervised representation learning,  Monthly Notices of the Royal Astronomical Society 513 (2022) 1581–1599. doi:10.1093/mnras/stac525. arXiv:2110.12735.
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * Simonyan and Zisserman [2014] K. Simonyan, A. Zisserman,  Very deep convolutional networks for large-scale image recognition,  arXiv preprint arXiv:1409.1556 (2014).
  * Hosseinzadeh Kassani et al. [2022] S. Hosseinzadeh Kassani, P. Hosseinzadeh Kassani, M. J. Wesolowski, K. A. Schneider, R. Deters,  Deep transfer learning based model for colorectal cancer histopathology segmentation: A comparative study of deep pre-trained models,  International Journal of Medical Informatics 159 (2022) 104669. URL: https://www.sciencedirect.com/science/article/pii/S1386505621002951. doi:https://doi.org/10.1016/j.ijmedinf.2021.104669.
  * Salman et al. [2020] H. Salman, A. Ilyas, L. Engstrom, A. Kapoor, A. Madry,  Do adversarially robust imagenet models transfer better?,  in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 3533–3545. URL: https://proceedings.neurips.cc/paper/2020/file/24357dd085d2c4b1a88a7e0692e60294-Paper.pdf.
  * Thenmozhi and Reddy [2019] K. Thenmozhi, U. S. Reddy,  Crop pest classification based on deep convolutional neural network and transfer learning,  Computers and Electronics in Agriculture 164 (2019) 104906.
  * Majurski et al. [2019] M. Majurski, P. Manescu, S. Padi, N. Schaub, N. Hotaling, C. Simon Jr, P. Bajcsy,  Cell image segmentation using generative adversarial networks, transfer learning, and augmentations,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0–0.
  * Ma et al. [2022] J. Ma, L. Bao, Q. Lou, D. Kong,  Transfer learning for automatic joint segmentation of thyroid and breast lesions from ultrasound images,  International Journal of Computer Assisted Radiology and Surgery 17 (2022) 363–372.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, T. Brox,  U-net: Convolutional networks for biomedical image segmentation,  in: International Conference on Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234–241.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, R. Girshick,  Mask r-cnn,  in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961–2969.
  * Huo et al. [2018] Y. Huo, Z. Xu, S. Bao, A. Assad, R. G. Abramson, B. A. Landman,  Adversarial synthesis learning enables segmentation without target modality ground truth,  in: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), IEEE, 2018, pp. 1217–1220.
  * Isola et al. [2017] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros,  Image-to-image translation with conditional adversarial networks,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125–1134.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick,  Microsoft coco: Common objects in context,  in: European conference on computer vision, Springer, 2014, pp. 740–755.
  * Caesar et al. [2018] H. Caesar, J. Uijlings, V. Ferrari,  Coco-stuff: Thing and stuff classes in context,  in: Computer vision and pattern recognition (CVPR), 2018 IEEE conference on, IEEE, 2018\.
  * Abraham and Mefraz Khan [2018] N. Abraham, N. Mefraz Khan,  A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentation,  arXiv e-prints (2018) arXiv:1810.07842. arXiv:1810.07842.