# Spatial Propagation of Weak Lensing Shear Response Corrections

T. D. Kitching1 N. Tessore2 P. L. Taylor3,4,5 1Mullard Space Science
Laboratory, University College London, Holmbury St Mary, Dorking, Surrey RH5
6NT, UK
2Department of Physics & Astronomy, University College London, Gower Street,
London WC1E 6BT, UK
3Center for Cosmology and AstroParticle Physics (CCAPP), The Ohio State
University, Columbus, OH 43210, USA
4Department of Physics, The Ohio State University, Columbus, OH 43210, USA
5Department of Astronomy, The Ohio State University, Columbus, OH 43210, USA

###### Abstract

In this paper we show how response function corrections to shear measurements
(e.g. as required by Metacalibration) propagate into cosmic shear power
spectra. We investigate a 2-sphere pixel (also known as ‘HEALpixels’)
correction and a forward-modelling approach using simple Gaussian simulations.
In the 2-sphere pixel-correction approach we find a free parameter that is the
tolerated condition number of the local response matrices: if this is too
large then this can cause an amplification of the shot noise power spectrum,
if too small it can lead to a loss of area (and a possible selection bias). In
contrast by forward-modelling the power spectrum this choice can be avoided.
This also applies to map-based inference methods using shear-response
calibrated maps.

##  1 Introduction

Weak lensing is the effect whereby the apparent observed ellipticity (third
flattening, or third eccentricity) of galaxies is altered by the presence of
matter along the line-of-sight. The effect can be approximated by an
additional ellipticity added to the unlensed (intrinsic) ellipticity that is
known as shear. Measurement of the weak lensing effect from data, to infer the
shear, can be biased by several effects e.g. inaccuracies in the algorithms
used (Heymans et al., 2006; Massey et al., 2007; Bridle et al., 2010; Kitching
et al., 2012; Mandelbaum et al., 2015), detector effects (Antilogus et al.,
2014), the size of the point spread function (Hoekstra et al., 2017; Kannawadi
et al., 2019; Gatti et al., 2021), and detection effects (Hoekstra et al.,
2015; Hoekstra, 2021). Kitching et al. (2019b, 2020) demonstrated how biases,
parameterised by multiplicative and additive terms that modify the observed
ellipticity (unlensed ellipticity plus shear), affect the observed power
spectrum of weak lensing data (known as cosmic shear).

However, in several methods such as lensfit (Miller et al., 2007) and
Metacalibration (MetaCal; Sheldon & Huff, 2017; Huff & Mandelbaum, 2017),
there is the concept of an additional multiplicative response matrix that
multiplies only the shear and not the unlensed ellipticity measurement. In
particular to account for such a term in MetaCal a local pixel-level
correction has been proposed, but the effect of the local noise on these
corrections, and alternatively the propagation of the response terms through
to the cosmic shear power spectrum has not been shown, and this is what we
address in this paper. In Section 2 we present the methodology, in Section 3
we present results of testing on simulations, and in Section 4 we discuss
conclusions.

##  2 Method

The MetaCal (Huff & Mandelbaum, 2017; Sheldon & Huff, 2017) correction can be
written as a local transformation of the ellipticity field like

$[\widetilde{e}_{1}+{\rm i}\widetilde{e}_{2}]_{p}=[e_{1}+{\rm
i}e_{2}+(R_{11}\gamma_{1}+R_{12}\gamma_{2})+{\rm
i}(R_{21}\gamma_{1}+R_{22}\gamma_{2})]_{p},$ (1)

where $R_{ij}$ are elements of the $2\times 2$ response matrix $R$,
$\gamma_{i}$ are the true shear, $\widetilde{e}$ are the observed
ellipticities, and $e_{i}$ are the unlensed uncorrelated intrinsic ellipticity
component (i.e. the shot noise term). This is a locally defined transform
within a angular pixel $p$ where $i=\\{1,2\\}$ refer to local distortions
relative to a Cartesian tangent plane, where $i=1$ are distortions along the
axes and $i=2$ are distortions at $45$ degrees to the axes. Locally, the
elements of the response matrix can be related to spin-0 and spin-4
multiplicative biases to the shear where

$\displaystyle[\widetilde{e}_{1}+{\rm i}\widetilde{e}_{2}]_{p}$
$\displaystyle=$ $\displaystyle[e_{1}+{\rm i}e_{2}+r_{0}(\gamma_{1}+{\rm
i}\gamma_{2})+r_{4}(\gamma_{1}-{\rm i}\gamma_{2})]_{p}$ $\displaystyle r_{0}$
$\displaystyle=$ $\displaystyle\frac{1}{2}[(R_{11}+R_{22})+{\rm
i}(R_{21}-R_{12})]$ $\displaystyle r_{4}$ $\displaystyle=$
$\displaystyle\frac{1}{2}[(R_{11}-R_{22})+{\rm i}(R_{21}+R_{12})].$ (2)

This can be compared to a global expression on the Celestial sphere in which

$\widetilde{\mathbf{e}}(\mathbf{\Omega})={\mathbf{e}}(\mathbf{\Omega})+r_{0}(\mathbf{\Omega}){\mathbf{\gamma}}(\mathbf{\Omega})+r_{4}(\mathbf{\Omega}){\mathbf{\gamma}^{*}}(\mathbf{\Omega})$
(3)

where ${\mathbf{x}}(\mathbf{\Omega})=x_{E}(\mathbf{\Omega})+{\rm
i}x_{B}(\mathbf{\Omega})$, where each field is defined relative to an $E$ and
$B$-mode field on the sphere, where $\mathbf{\Omega}$ are angular coordinates.
$r_{0}$ is a spin-0 term and $r_{4}$ is a spin-4 term, where the total spin-2
nature of all terms is conserved.

###  2.1 Power Spectra

The power spectra estimates for the measured ellipticity can now be computed
by taking the correlation of the spherical harmonic coefficients, computed
using a spin-weight spherical harmonic transform for a spin-2 field, where

$\displaystyle\widetilde{C}^{GH}_{\ell,ij}$ $\displaystyle\equiv$
$\displaystyle\frac{1}{2\ell+1}\sum_{m}\widetilde{e}^{G}_{\ell
m,i}\widetilde{e}^{H,*}_{\ell m,j}$ (4)

for $G=\\{E,B\\}$ and $H=\\{E,B\\}$, where $i$ and $j$ labels for tomographic
bins delineating galaxy populations defined by redshift or colour (Kitching et
al., 2019a). We will assume that the true $EB$ and $BE$ power spectra are zero
$C^{EB}_{\ell,ij}=C^{BE}_{\ell,ij}=0$, which should be the case in all but the
most exotic dark energy models that cause parity-violating modes (Amendola et
al., 2013). Given this assumption, the estimated $EE$ power spectra is given
by

$\displaystyle\widetilde{C}^{EE}_{\ell,ij}=\left[{\sum_{\ell^{\prime}}}{\mathcal{M}}^{++}_{\ell\ell^{\prime},ij}C^{EE}_{\ell^{\prime},ij}+{\mathcal{M}}^{--}_{\ell\ell^{\prime},ij}C^{BB}_{\ell^{\prime},ij}\right]+N_{\ell},\,\,\,\,\,\,\,{\rm
and}\,\,\,\,\,\,\,\widetilde{C}^{BB}_{\ell,ij}=\left[{\sum_{\ell^{\prime}}}{\mathcal{M}}^{+-}_{\ell\ell^{\prime},ij}C^{BB}_{\ell^{\prime},ij}+{\mathcal{M}}^{-+}_{\ell\ell^{\prime},ij}C^{EE}_{\ell^{\prime},ij}\right]+N_{\ell}$
(5)

where $N_{\ell}=\sigma^{2}_{e}/N_{\rm gal}$ is the shot noise power spectrum
(where $N_{\rm gal}$ is the number of galaxies used to compute the power
spectrum and $\sigma^{2}_{e}$ is the variance of the unlensed ellipticity),
and $C^{EE}_{\ell,ij}$ and $C^{BB}_{\ell^{\prime},ij}$ are the $EE$ and $BB$
power spectra of shear. If the data is masked or there is a further
multiplicative bias then a further mixing of modes would occur to the
$\widetilde{C}^{EE}_{\ell,ij}$ and $\widetilde{C}^{BB}_{\ell,ij}$, as
described in Kitching et al. (2020); these additional mixing matrices would
affect both the signal and the noise terms.

Following Brown et al. (2005) and Appendix A the calculation of mixing
matrices can be written like

$\displaystyle{\mathcal{M}}^{+\pm}_{\ell\ell^{\prime},ij}$ $\displaystyle=$
$\displaystyle\frac{2\ell^{\prime}+1}{8\pi}\sum_{\ell^{\prime\prime}}(2\ell^{\prime\prime}+1)[1+(-1)^{\ell+\ell^{\prime}+\ell^{\prime\prime}}]\left[C^{r_{0}r_{0}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
-2&2&0\\\
\end{pmatrix}^{2}+C^{r_{4}r_{4}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
2&2&-4\\\ \end{pmatrix}^{2}\pm
2C^{r_{0}r_{4}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
-2&2&0\\\
\end{pmatrix}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
2&2&-4\\\ \end{pmatrix}\right]$
$\displaystyle{\mathcal{M}}^{-\pm}_{\ell\ell^{\prime},ij}$ $\displaystyle=$
$\displaystyle\frac{2\ell^{\prime}+1}{8\pi}\sum_{\ell^{\prime\prime}}(2\ell^{\prime\prime}+1)[(-1)^{\ell+\ell^{\prime}+\ell^{\prime\prime}}-1]\left[C^{r_{0}r_{0}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
-2&2&0\\\
\end{pmatrix}^{2}+C^{r_{4}r_{4}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
2&2&-4\\\ \end{pmatrix}^{2}\pm
2C^{r_{0}r_{4}}_{\ell^{\prime\prime},ij}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
-2&2&0\\\
\end{pmatrix}\begin{pmatrix}\ell&\ell^{\prime}&\ell^{\prime\prime}\\\
2&2&-4\\\ \end{pmatrix}\right]$ (6)

where $C^{r_{X}r_{Y}}_{\ell,ij}$ is the (cross) power spectrum of $r_{X}$ and
$r_{Y}$, where $X$ and $Y=\\{0,4\\}$, and the matrices are Wigner-3$j$
symbols. It is noted that, since the values of $R_{ij}$ are not small compared
to the shear, all terms including cross-terms need to be included.

###  2.2 Pixel Correction

An alternative is to correct the observed ellipticity field directly. In this
case we label quantities with a subscript $p$ to mean an angular/2-sphere
pixels (we use pixelisation of the 2-sphere defined in McEwen & Wiaux, 2011)
on the sky e.g. ${\mathbf{e}}_{p}$. A correction for the response function
then can be constructed as $\hat{\mathbf{\gamma}}_{p}\simeq
R^{-1}_{p}{\mathbf{e}}_{p}+{\mathbf{\gamma}}_{p}$, where
$\hat{\mathbf{\gamma}}_{p}$ is the estimated shear after correction, which in
the case of no noise (${\mathbf{e}}\rightarrow 0$) is equal to the shear.
However, the first term in the correction $R^{-1}_{p}{\mathbf{e}}$ represents
a local amplification of the shot noise; this is true even if one only applies
an average correct by computing the mean of the inverse-response over the sky.
This is equivalent of applying a matrix transformation to the unlensed
ellipticity where

$\displaystyle[\widehat{\gamma}_{1}+{\rm i}\widehat{\gamma}_{2}]_{p}$
$\displaystyle=$ $\displaystyle[\gamma_{1}+{\rm
i}\gamma_{2}+r^{\prime}_{0}(e_{1}+{\rm i}e_{2})+r^{\prime}_{4}(e_{1}-{\rm
i}e_{2})]_{p}$ $\displaystyle r^{\prime}_{0}$ $\displaystyle=$
$\displaystyle\frac{1}{2}[(R^{-1}_{11}+R^{-1}_{22})+{\rm
i}(R^{-1}_{21}-R^{-1}_{12})]$ $\displaystyle r^{\prime}_{4}$ $\displaystyle=$
$\displaystyle\frac{1}{2}[(R^{-1}_{11}-R^{-1}_{22})+{\rm
i}(R^{-1}_{21}+R^{-1}_{12})],$ (7)

where $R^{-1}_{ij}$ are elements of the locally-inversed $R$ field on the
sphere (i.e. that field which is constructed by computing a pixel-by-pixel
inverse of the $R$ field). Therefore when taking the power spectrum the noise-
amplification term needs to be corrected. Taking the power spectrum of
equation (2.2) one finds a analogous equation to before where

$\displaystyle
C^{EE}_{\ell,ij}=\widehat{C}^{EE}_{\ell,ij}-\left[{\sum_{\ell^{\prime}}}{\mathcal{N}}^{++}_{\ell\ell^{\prime},ij}N^{EE}_{\ell^{\prime},ij}+{\mathcal{N}}^{--}_{\ell\ell^{\prime},ij}N^{BB}_{\ell^{\prime},ij}\right],\,\,\,\,\,\,\,{\rm
and}\,\,\,\,\,\,\,C^{BB}_{\ell,ij}=\widehat{C}^{BB}_{\ell,ij}-\left[{\sum_{\ell^{\prime}}}{\mathcal{N}}^{+-}_{\ell\ell^{\prime},ij}N^{BB}_{\ell^{\prime},ij}+{\mathcal{N}}^{-+}_{\ell\ell^{\prime},ij}N^{EE}_{\ell^{\prime},ij}\right]$
(8)

where the ${\mathcal{N}}^{+\pm}_{\ell\ell^{\prime},ij}$ and
${\mathcal{N}}^{-\pm}_{\ell\ell^{\prime},ij}$ are defined in the same way as
equation (2.1) except using the $r^{\prime}_{0}$ and $r^{\prime}_{4}$ fields
defined in equation (2.2). We note that $N^{BB}_{\ell,ij}\simeq
N^{EE}_{\ell,ij}$ so some terms will cancel.

Thus we find that a local pixel correction requires a correction for the noise
amplification effect at the power spectrum level. We note that in doing such a
pixel-correction an inverse of the local $R$ matrices is required, which for
close-to-singular matrices may lead to an ill-conditioned computational
procedure; and since the distribution of $R_{ij}$ values is broad and can
cross $R_{ij}=0$ it is possible the matrices may be singular. Hence a free
parameter in such an approach is the acceptable condition number allowed,
below which a pixel would be masked (thus decreasing the usable area of the
survey); we define the conditional number111See
https://numpy.org/doc/stable/reference/generated/numpy.linalg.cond.html. for
$x$ as the norm of $x$ multiplied by the norm of the inverse of $x$, i.e.
$c_{R}=|x|/|x^{-1}|$. To explain further: in the pixel-correction case one
needs to invert the $R_{ij}$ matrices locally, the matrices might not be
numerically invertible everywhere, we use the condition number to determine
what is invertible, we consider a pixel unobserved if its matrix is not
invertible. In the next Section we create simple simulations to test the
forward-modelling approach, the pixel-correction approach, and an approach of
taking the mean response over the sky.

##  3 Tests on Simulations

We model $\gamma(\mathbf{\Omega})$ as a Gaussian random field (generateed
using the massmappy code Wallis et al., 2017), assuming a DES Year 1 cosmology
(Flaugher et al., 2015; Abbott et al., 2018; Morganson et al., 2018) to
compute the EE cosmic shear power spectrum. We assume the the Limber (Limber,
1953; Kitching et al., 2017; Lemos et al., 2017), reduced shear (Deshpande &
Kitching, 2020), flat-Universe (Taylor et al., 2018), flat-sky (Kamionkowski
et al., 1998) and prefactor-unity (Kitching et al., 2017) approximations.
Therefore the EE power spectrum is given by:

$C^{EE}_{\ell}=\int_{0}^{\chi_{\rm H}}{\rm
d}\chi\frac{q^{2}(\chi)}{\chi^{2}}P_{\delta}\left(\frac{\ell+1/2}{\chi},\chi\right),\,\,\,\,\,\,\,{\rm
where}\,\,\,\,\,\,\,q(\chi)=\frac{3}{2}\Omega_{\rm
M}\frac{H^{2}_{0}}{c^{2}}\frac{\chi}{a(\chi)}\int^{\chi_{\rm H}}_{\chi}{\rm
d}\chi^{\prime}\,n(\chi^{\prime})\,\frac{\chi^{\prime}-\chi}{\chi};$ (9)

where $P_{\delta}$ is the power spectrum of matter overdensities that we
calculate using CAMB (Lewis et al., 2000) (we include the corrections from
Mead et al., 2015, for the non-linear corrections). $H_{0}$ is the Hubble
constant, $\chi$ and $\chi_{\rm H}$ are the comoving distance and comoving
distance to the horizon respectively (calculated using the astropy package
Astropy Collaboration et al., 2018, 2013), $a$ is the scale factor of the
Universe, $\Omega_{\rm M}$ is the dimensionless total matter density of the
Universe, and $c$ is the speed of light in a vacuum. $n(\chi)$ is the galaxy
distribution function of the survey (where we use the photometric DES Year 1
galaxy distribution. Abbott et al., 2018), and we assume only a single
tomographic bin. We assume that $EB$ and $BB$ shear power spectra are zero.

Figure 1: We show the observed power spectrum, that includes the effects of
the response matrix and noise (blue lines); and shear-only power spectrum that
does not include noise or the response matrix terms (red lines). We then
compare this the pixel-correction approach (equation 2.2, green lines) that
should reproduce the shear-only power spectrum, and the forward modelling
approach (equation 5; blue lines) that should produce the observed power
spectrum. The left-hand plot is for a condition number limit of $c_{R}\leq
10$, and the right-hand for $c_{R}\leq 100$ in the pixel-corrected approach.

We also make a mask that removes data within $20^{\circ}$ in both the galactic
and ecliptic planes; and also $20\%$ of pixels at random, to represent an all-
sky mask with random patches removed, resulting in a total observed sky
fraction of $f_{\rm sky}=0.4$. Finally we assume $\sigma_{e}=0.3$, and $N_{\rm
gal}=30f_{\rm sky}3600(4\pi[180/\pi]^{2})$ for the shot noise modelling. This
is is similar to upcoming Stage-IV surveys (Albrecht et al., 2006). We then
model the response matrix elements as Gaussian random fields with a white-
noise power spectrum with mean $\mu=1$ and a standard deviation of
$\sigma=0.6$ for all $R_{ij}$; which approximates the amplitude observed in
real shape measurement methods (Huff & Mandelbaum, 2017). This is meant as the
simplest test of the approaches outlined in this paper, in future much more
realistic values should be used that also correlate with galaxies, instrument
and telescope properties.

In Figure 1 we show the observed power spectrum, that includes the effects of
the response matrix and noise (blue lines); and shear-only power spectrum that
does not include noise or the response matrix terms (red lines). We then
compare this the pixel-correction approach (equation 2.2) that should
reproduce the shear-only power spectrum, and the forward modelling approach
(equation 5) that should produce the observed power spectrum. We test the
pixel-correction approach for two cases where the condition number of the
response matrices is limited to $c_{R}\leq 10$ and $c_{R}\leq 100$; where
$c_{R}$ is the condition number of the local response matrices, defined using
the numpy function numpy.linalg.cond1. We find that both approaches reproduce
the expected results, however in the pixel-correction approach setting a
condition number limit on the matrices that is too large can lead to an
amplification of the noise term in equation (2.2) and hence a deviation from
the shear-only power spectrum. Thus there is a trade-off in the pixel-
correction approach between accuracy (a tight limit on the condition number)
and area coverage (i.e. pixels not included because they are excluded by the
limit). In our simple simulation we find a 20% change in area caused by going
from $c_{R}\leq 100$ to $c_{R}\leq 10$. Such a trade-off is not required in
the forward modelling approach. In the case of tomographic binning the shot-
noise amplification will be larger because there will be fewer galaxies in
each pixel.

Finally a different approach could instead use a global correction, by
dividing all observed ellipticities by the mean of the response function over
the sky, hoping to derive the true shear power spectrum. In this case, since
in our simulations the mean is unity, this would result in an error between
the inferred shear power spectrum and the true shear power spectrum that was
equivalent to the difference between the uncorrected case (blue lines) and the
true case (red lines). Clearly in such an approach there would be a large
bias.

##  4 Conclusions

In this paper we show how response functions propagate into cosmic shear power
spectra computed from pixelised maps, and we investigate pixel-correction and
forward-modelling approaches using simulations. In the pixel-correction
approach there is a free parameter that is the tolerated condition number of
the local response matrices – if this is too large then this can cause an
amplification of the shot noise power spectrum, if too small it can lead to a
loss of area. In contrast forward-modelling the power spectrum avoids this
choice. Forward-modelling involves measuring the response of each galaxy as a
function of local conditions at the map level and the propagation of angular
variation through to the power spectrum. In more complex approaches one could
also draw from the probability distribution function of the response-function
map to propagate uncertainties in these measurements into the power spectrum.
Alternatively one could infer the power spectrum directly, rather than via a
map, but in this case the angular variation of the response function would
still need to be accounted for.

###### Acknowledgements.

_Acknowledgements:_ We thank the developers of SSHT, massmappy, astropy and
CAMB. NT is supported by UK Space Agency grants ST/W002574/1 and ST/X00208X/1.

## References

  * Abbott et al. (2018) Abbott T. M. C., et al., 2018, http://dx.doi.org/10.3847/1538-4365/aae9f0 ApJS, https://ui.adsabs.harvard.edu/abs/2018ApJS..239…18A 239, 18
  * Albrecht et al. (2006) Albrecht A., et al., 2006, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2006astro.ph..9591A pp astro–ph/0609591
  * Amendola et al. (2013) Amendola L., et al., 2013, http://dx.doi.org/10.12942/lrr-2013-6 Living Reviews in Relativity, https://ui.adsabs.harvard.edu/abs/2013LRR….16….6A 16, 6
  * Antilogus et al. (2014) Antilogus P., Astier P., Doherty P., Guyonnet A., Regnault N., 2014, http://dx.doi.org/10.1088/1748-0221/9/03/C03048 Journal of Instrumentation, https://ui.adsabs.harvard.edu/abs/2014JInst…9C3048A 9, C03048
  * Astropy Collaboration et al. (2013) Astropy Collaboration et al., 2013, http://dx.doi.org/10.1051/0004-6361/201322068 A&A, https://ui.adsabs.harvard.edu/abs/2013A&A…558A..33A 558, A33
  * Astropy Collaboration et al. (2018) Astropy Collaboration et al., 2018, http://dx.doi.org/10.3847/1538-3881/aabc4f AJ, https://ui.adsabs.harvard.edu/abs/2018AJ….156..123A 156, 123
  * Bridle et al. (2010) Bridle S., et al., 2010, http://dx.doi.org/10.1111/j.1365-2966.2010.16598.x MNRAS, http://adsabs.harvard.edu/abs/2010MNRAS.405.2044B 405, 2044
  * Brown et al. (2005) Brown M. L., Castro P. G., Taylor A. N., 2005, http://dx.doi.org/10.1111/j.1365-2966.2005.09111.x MNRAS, http://adsabs.harvard.edu/abs/2005MNRAS.360.1262B 360, 1262
  * DLMF (2022) DLMF 2022, NIST Digital Library of Mathematical Functions, http://dlmf.nist.gov/, Release 1.1.8 of 2022-12-15, http://dlmf.nist.gov/
  * Deshpande & Kitching (2020) Deshpande A. C., Kitching T. D., 2020, http://dx.doi.org/10.1103/PhysRevD.101.103531 Phys. Rev. D, https://ui.adsabs.harvard.edu/abs/2020PhRvD.101j3531D 101, 103531
  * Flaugher et al. (2015) Flaugher B., et al., 2015, http://dx.doi.org/10.1088/0004-6256/150/5/150 AJ, https://ui.adsabs.harvard.edu/abs/2015AJ….150..150F 150, 150
  * Gatti et al. (2021) Gatti M., et al., 2021, http://dx.doi.org/10.1093/mnras/stab918 MNRAS, https://ui.adsabs.harvard.edu/abs/2021MNRAS.504.4312G 504, 4312
  * Heymans et al. (2006) Heymans C., et al., 2006, http://dx.doi.org/10.1111/j.1365-2966.2006.10198.x MNRAS, http://adsabs.harvard.edu/abs/2006MNRAS.368.1323H 368, 1323
  * Hoekstra (2021) Hoekstra H., 2021, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2021arXiv210810057H p. arXiv:2108.10057
  * Hoekstra et al. (2015) Hoekstra H., Herbonnet R., Muzzin A., Babul A., Mahdavi A., Viola M., Cacciato M., 2015, http://dx.doi.org/10.1093/mnras/stv275 MNRAS, http://adsabs.harvard.edu/abs/2015MNRAS.449..685H 449, 685
  * Hoekstra et al. (2017) Hoekstra H., Viola M., Herbonnet R., 2017, http://dx.doi.org/10.1093/mnras/stx724 MNRAS, https://ui.adsabs.harvard.edu/abs/2017MNRAS.468.3295H 468, 3295
  * Huff & Mandelbaum (2017) Huff E., Mandelbaum R., 2017, arXiv e-prints, https://ui.adsabs.harvard.edu/#abs/2017arXiv170202600H p. arXiv:1702.02600
  * Kamionkowski et al. (1998) Kamionkowski M., Babul A., Cress C. M., Refregier A., 1998, http://dx.doi.org/10.1046/j.1365-8711.1998.02054.x Monthly Notices of the Royal Astronomical Society, 301, 1064
  * Kannawadi et al. (2019) Kannawadi A., et al., 2019, http://dx.doi.org/10.1051/0004-6361/201834819 A&A, https://ui.adsabs.harvard.edu/abs/2019A&A…624A..92K 624, A92
  * Kitching et al. (2012) Kitching T. D., et al., 2012, http://dx.doi.org/10.1111/j.1365-2966.2012.21095.x MNRAS, http://adsabs.harvard.edu/abs/2012MNRAS.423.3163K 423, 3163
  * Kitching et al. (2017) Kitching T. D., Alsing J., Heavens A. F., Jimenez R., McEwen J. D., Verde L., 2017, http://dx.doi.org/10.1093/mnras/stx1039 MNRAS, https://ui.adsabs.harvard.edu/abs/2017MNRAS.469.2737K 469, 2737
  * Kitching et al. (2019a) Kitching T. D., Taylor P. L., Capak P., Masters D., Hoekstra H., 2019a, arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2019arXiv190106495K p. arXiv:1901.06495
  * Kitching et al. (2019b) Kitching T. D., Paykari P., Hoekstra H., Cropper M., 2019b, http://dx.doi.org/10.21105/astro.1904.07173 The Open Journal of Astrophysics, https://ui.adsabs.harvard.edu/abs/2019OJAp….2E…5K 2, 5
  * Kitching et al. (2020) Kitching T. D., Deshpande A. C., Taylor P. L., 2020, http://dx.doi.org/10.21105/astro.2010.07749 The Open Journal of Astrophysics, https://ui.adsabs.harvard.edu/abs/2020OJAp….3E..14K 3, 14
  * Lemos et al. (2017) Lemos P., Challinor A., Efstathiou G., 2017, http://dx.doi.org/10.1088/1475-7516/2017/05/014 J. Cosmology Astropart. Phys, https://ui.adsabs.harvard.edu/abs/2017JCAP…05..014L 2017, 014
  * Lewis et al. (2000) Lewis A., Challinor A., Lasenby A., 2000, http://dx.doi.org/10.1086/309179 ApJ, https://ui.adsabs.harvard.edu/abs/2000ApJ…538..473L 538, 473
  * Limber (1953) Limber D. N., 1953, http://dx.doi.org/10.1086/145672 ApJ, https://ui.adsabs.harvard.edu/abs/1953ApJ…117..134L 117, 134
  * Mandelbaum et al. (2015) Mandelbaum R., et al., 2015, http://dx.doi.org/10.1093/mnras/stv781 MNRAS, http://adsabs.harvard.edu/abs/2015MNRAS.450.2963M 450, 2963
  * Massey et al. (2007) Massey R., et al., 2007, http://dx.doi.org/10.1111/j.1365-2966.2006.11315.x MNRAS, http://adsabs.harvard.edu/abs/2007MNRAS.376…13M 376, 13
  * McEwen & Wiaux (2011) McEwen J. D., Wiaux Y., 2011, http://dx.doi.org/10.1109/TSP.2011.2166394 IEEE Transactions on Signal Processing, http://adsabs.harvard.edu/abs/2011ITSP…59.5876M 59, 5876
  * Mead et al. (2015) Mead A. J., Peacock J. A., Heymans C., Joudaki S., Heavens A. F., 2015, http://dx.doi.org/10.1093/mnras/stv2036 MNRAS, https://ui.adsabs.harvard.edu/abs/2015MNRAS.454.1958M 454, 1958
  * Miller et al. (2007) Miller L., Kitching T. D., Heymans C., Heavens A. F., van Waerbeke L., 2007, http://dx.doi.org/10.1111/j.1365-2966.2007.12363.x MNRAS, https://ui.adsabs.harvard.edu/abs/2007MNRAS.382..315M 382, 315
  * Morganson et al. (2018) Morganson E., et al., 2018, http://dx.doi.org/10.1088/1538-3873/aab4ef PASP, https://ui.adsabs.harvard.edu/abs/2018PASP..130g4501M 130, 074501
  * Sheldon & Huff (2017) Sheldon E. S., Huff E. M., 2017, http://dx.doi.org/10.3847/1538-4357/aa704b ApJ, https://ui.adsabs.harvard.edu/abs/2017ApJ…841…24S 841, 24
  * Taylor et al. (2018) Taylor P. L., Kitching T. D., McEwen J. D., Tram T., 2018, http://dx.doi.org/10.1103/PhysRevD.98.023522 Phys. Rev. D, https://ui.adsabs.harvard.edu/abs/2018PhRvD..98b3522T 98, 023522
  * Tessore (2019) Tessore N., 2019, http://dx.doi.org/10.48550/arXiv.1904.09973 arXiv e-prints, https://ui.adsabs.harvard.edu/abs/2019arXiv190409973T p. arXiv:1904.09973
  * Wallis et al. (2017) Wallis C. G. R., McEwen J. D., Kitching T. D., Leistedt B., Plouviez A., 2017, preprint, http://adsabs.harvard.edu/abs/2017arXiv170309233W (http://arxiv.org/abs/1703.09233 arXiv:1703.09233)

## Appendix A: Product of Spherical Spin-Weight Functions on the Sphere

Here we follow DLMF (2022) Section 34.3 and Tessore (2019). Let
$f({\mathbf{\Omega}})$ and $g({\mathbf{\Omega}})$ be spin-weighted spherical
functions with respective spins $s_{f}$ and $s_{g}$. The product
$h({\mathbf{\Omega}})=f({\mathbf{\Omega}})g({\mathbf{\Omega}})$ has spin
weight $s=s_{f}+s_{g}$. The spherical harmonic transform of
$h({\mathbf{\Omega}})$ can be written in terms of spin-weighted spherical
harmonic functions ${}_{s}Y_{\ell m}(\mathbf{\Omega})$

$h_{\ell
m}=\sum_{\ell_{f}m_{f}}\sum_{\ell_{g}m_{g}}f_{\ell_{f}m_{f}}g_{\ell_{g}m_{g}}\int{\rm
d}{\mathbf{\Omega}}\,\,_{s_{f}}Y_{\ell_{f}m_{f}}(\mathbf{\Omega})_{s_{g}}Y_{\ell_{g}m_{g}}(\mathbf{\Omega})_{s}Y^{*}_{\ell
m}(\mathbf{\Omega}),$ (10)

where $f_{\ell_{f}m_{f}}$ and $g_{\ell_{f}m_{f}}$ are the spherical harmonic
transforms of $f$ and $g$; $\ell$ and $m$ are wavenumbers. The integral can be
rewritten so that

$h_{\ell
m}=\sum_{\ell_{f}m_{f}}\sum_{\ell_{g}m_{g}}(-1)^{s+m}f_{\ell_{f}m_{f}}g_{\ell_{g}m_{g}}\left(\frac{(2\ell_{f}+1)(2\ell_{g}+1)(2\ell+1)}{4\pi}\right)^{1/2}\begin{pmatrix}\ell_{f}&\ell_{g}&\ell\\\
-s_{f}&-s_{g}&s\\\ \end{pmatrix}\begin{pmatrix}\ell_{f}&\ell_{g}&\ell\\\
m_{f}&m_{g}&-m\\\ \end{pmatrix},$ (11)

this is the spherical harmonic convolution theorem for products of functions,
where the matrices are Wigner-$3j$ symbols. To compute the effect of the
convolution on the angular power spectrum we consider the products $h=fg$ and
$h^{\prime}=f^{\prime}g^{\prime}$. In general it is not possible to express
the cross-power spectrum of $h$ and $h^{\prime}$, $C^{hh^{\prime}}_{\ell}$ in
terms of combinations of the power spectrum of the individual fields. However
if $f$ and $f^{\prime}$ are jointly homogeneous spherical random fields,
independent of $g$ and $g^{\prime}$, we can write down an expression for the
expectation of the angular power spectrum $\langle
C^{hh^{\prime}}_{\ell}\rangle$; since the differently-numbered modes of any
jointly homogeneous random fields are uncorrelated i.e. $\langle f_{\ell
m}(f^{\prime}_{\ell^{\prime}m^{\prime}})^{*}\rangle=\delta^{K}_{\ell\ell^{\prime}}\delta^{K}_{mm^{\prime}}\langle
C^{ff^{\prime}}_{\ell}\rangle$, where $\delta^{K}_{xx^{\prime}}$ is a
Kronecker delta. Using this, and the orthogonality relation for Wigner-$3j$
symbols we can write down the final result as the convolution

$\displaystyle\langle
C^{hh^{\prime}}_{\ell}\rangle=\sum_{\ell_{f}\ell_{g}}(-1)^{s+s^{\prime}}\frac{(2\ell_{f}+1)(2\ell_{g}+1)}{4\pi}\langle
C^{ff^{\prime}}_{\ell_{f}}\rangle
C^{gg^{\prime}}_{\ell_{g}}\begin{pmatrix}\ell_{f}&\ell_{g}&\ell\\\
-s_{f}&-s_{g}&s\\\ \end{pmatrix}\begin{pmatrix}\ell_{f}&\ell_{g}&\ell\\\
-s^{\prime}_{f}&-s^{\prime}_{g}&s^{\prime}\\\ \end{pmatrix}.$ (12)

This can be written as a matrix multiplication $\langle
C^{hh^{\prime}}_{\ell}\rangle=\sum_{\ell_{f}}{\mathcal{M}}^{gg^{\prime}}_{\ell\ell_{f}}\langle
C^{ff^{\prime}}_{\ell_{f}}\rangle$ where
${\mathcal{M}}^{gg^{\prime}}_{\ell\ell_{f}}$ can be inferred from equation
(12).


# FragNet: Writer Identification using
Deep Fragment Networks

Sheng He, Lambert Schomaker  S.He is with the Boston Children’s Hospital and
Harvard Medical School, Harvard University, 300 Longwood Ave., Boston, MA, USA
(e-mail: heshengxgd@gmail.com)L.Schomaker is with the Bernoulli Institute for
Mathematics, Computer Science and Artificial Intelligence, University of
Groningen, Groningen, PO Box 407, 9700 AK, The Netherlands (e-mail:
l.r.b.schomaker@rug.nl).

###### Abstract

Writer identification based on a small amount of text is a challenging
problem. In this paper, we propose a new benchmark study for writer
identification based on word or text block images which approximately contain
one word. In order to extract powerful features on these word images, a deep
neural network, named FragNet, is proposed. The FragNet has two pathways:
feature pyramid which is used to extract feature maps and fragment pathway
which is trained to predict the writer identity based on fragments extracted
from the input image and the feature maps on the feature pyramid. We conduct
experiments on four benchmark datasets, which show that our proposed method
can generate efficient and robust deep representations for writer
identification based on both word and page images.

###### Index Terms:

FragNet, Fragment segmentation, writer identification, convolutional neural
networks.

##  I Introduction

Writer identification is a typical pattern recognition problem in forensic
science and has been studied for many years. It has potential applications in
historical [1] and forensic [2] document analysis. Forensic document examiners
usually use certain features of handwritten texts to determine the authorship.
Therefore, traditional methods focus on extracting handcrafted features to
capture the writing style information of a given handwritten document [3, 4,
5]. Handcrafted features usually have been designed to describe specific
attributes of the natural handwriting style of a writer, such as slant [3] and
curvature [4]. Due to the fact that handcrafted features capture the
statistical information of handwriting style which need more information to
obtain a good performance, a system using handcrafted features for writer
identification usually requires a large amount of handwritten text, such as a
paragraph or a text block with several sentences [3, 6].

The traditional writer identification methods focus on extracting handcrafted
or deep learned features on the whole document images. Each writer usually has
two samples: one is used for training and the other is used for evaluation.
However, as discussed in our previous work [7], in the digital era, a real-
world writer identification system should recognize the author based on a very
small amount of handwritten text since people are preferring typing on
keyboards instead of writing on papers. Thus, we propose a new protocol for
writer identification based on the traditional benchmark datasets: identifying
the writer based on word or text-block images segmented on the whole
handwritten documents. Following the traditional writer identification
protocol, word or text-block images segmented on one page of each writer are
used for modeling the handwriting style and text-block images segmented on
other documents from the same writer are used for testing. Therefore, it can
be used for both word-based and page-based writer identification tasks. Word-
based writer identification is the problem of identifying the writer based on
word images while page-based writer identification relies on global features
computed by aggregating local features extracted on word or text-block images.

Writer identification based on word images is more challenging than writer
identification based on page images since the writer-related style information
in word images is limited for modeling the writer’s handwriting style. Our
previous work [7] studies on writer identification based on word images using
Convolutional neural networks (CNNs) with two branches for different tasks,
which requires extra annotations for the auxiliary task. Therefore, it can be
only applied on these word images which carry word labels. In this paper, we
propose a new method, which recognizes the writer identity based on word
images using deep learning [8, 9] without any extra labels. CNNs can learn
deep abstract and high level features on a small amount of text, such as word
images or text blocks which contain several characters. Therefore, most
methods extract deep local features on character images and their sub-regions
[10] or image patches [11]. These local features are aggregated together for
computing the global feature of each handwritten page for writer
identification [10, 11].

Although CNNs can provide a good performance, there are two drawbacks to apply
CNNs for writer identification based on word images: (1) the decision made by
the CNN is hard to be interpreted and (2) the detailed handwriting style
information on word images is not fully explored. In order to solve these
problems, we propose the FragNet method for writer identification based on
word or text-block images.

###  I-A FragNet can be interpreted.

The proposed FragNet is inspired by Fraglets [3] and BagNet [12]. Fraglets [3]
is the grapheme-based method, which extracts the basic graphemes (sub or
supra-allographic fragments) from handwritten ink traces which are
characteristics of each writer. The big advantage of Fraglets is that it can
be visualized to end users. BagNet [12] provides an alternative way to
understand deep learned features which are extracted on patches in input
images and they are averaged for the image-level prediction, inspired by the
ensemble learning [13]. The advantage of BagNet is that the importance of each
part to the final classification can be visualized. Our FragNet method can be
considered as an extension of Fraglets and BagNet, which builds the neural
network on each fragment in word images. Therefore, the FragNet inherits all
the advantages of Fraglets and BagNet.

Different from Fraglets [3], which segments the fragment on input images, our
proposed FragNet method extracts fragments (a patch with a square size) on
different levels of the featurized image pyramids [14] computed by deep
convolutional neural networks. The writer evidence of each fragment is
computed by another neural network and the average of responses from fragments
on the same word image is considered as the word-level writer evidence used
for writer identification.

###  I-B FragNet can explore the detailed writer style information.

Our experimental results show that the deep features learned by FragNet are
much more effective for writer identification and retrieval than the deep
features learned from the whole word image. The reason is that the trained
deep networks mainly focus on the most discriminative parts from the input
images and ignore others [15]. The discriminative part is named as “spotting
region” (effective receptive field of the network) in this paper which is
similar to the attention area [16]. When training on word images, the network
makes a decision based on the information from the whole image and the
information in the spotting region is easy to be recognized first, resulting
in a small loss during training. For example, when training the network to
recognize the writer in Fig. 1, the network is mainly dominated by one salient
shape (the red circle means the spotting region) and other writing shapes are
ignored since this salient shape yields a small loss to the network. However,
other shapes also contain the writing style information which is useful for
writer identification, thus combining their evidences properly can improve the
performance. An efficient way to explore all shape information in the input
word image is to classify and ensemble local patches, inspired by BagNet [13].
As shown in Fig. 1, aggregating local patches results in multiple spotting
regions in the input image, making the class evidence more robust. Based on
this observation, our proposed FragNet receives fragments segmented in the
input image and it forces the network to learn the handwriting style
information or find the “spotting region” in each fragment. Combining the
results of these fragments makes the network to have multiple spotting
regions, avoiding overfitting on a specific part in the input image and
yielding a high performance.

Figure 1: The red circle shows the “spotting region” (effective receptive
field) of the neural network. FragNet has multiple spotting regions and the
number of spotting regions is determined by the number of fragments.
Therefore, FragNet can capture the detailed handwriting style information and
provide a good performance for writer identification based on word images.

The rest of this paper is organized as follows. Section II provides a brief
summary of previous works about writer identification. We describe the
proposed FragNet in Section III and conduct experiments in Section IV. The
conclusion is given in Section V.

##  II Related Work

This section provides a brief review of several typical writer identification
methods. A comprehensive survey of writer identification can be found in [17].

Before the deep learning era, handcrafted features are widely used for writer
identification, which can be roughly divided into two groups: textural-based
and grapheme-based features. Based on the assumption that the writer’s
handwriting can be considered as a texture [18], textural features are widely
used. The Gabor filters are used in [18] to extract writer-specific textures.
In [19], the Gabor and XGabor filters are used to extract features on
handwritten patterns for Persian writer identification. Newell and Griffin [5]
propose to use a bank of six Derivative-of-Gaussian filters to extract texture
features at two scales. The RootSIFT descriptors [20] are used to extract
local features on handwritten images and Gaussian Mixture Models (GMM) are
used for local feature encoding in [21, 6]. After obtaining the global feature
representation, Christlein et al. [21] propose to use Exemplar-SVMs for writer
identification while Khan et al. [6] introduce the concept of similarity and
dissimilarity GMM with different descriptors to compute the writing style
similarity between different document images.

Model-based methods have also been widely studied for writer identification.
In [22], the beta-elliptic model is used to generate the grapheme codebook
without training and the performance is evaluated on the Arabic handwritten
documents. Contour-based graphemes are also used in [23, 24]. Our previous
work [25] uses the junction as the basic grapheme, which contains the
handwriting style information. Christlein et al. [26] extract local
descriptors on a small image patch and use a Gaussian mixture model to encode
the extracted local features into a common space for handwriting similarity
measurement. Later, they propose an unsupervised feature learning method [27],
which learns deep features with the pseudo-label generated by $k$-means.

Several writer identification methods combine the texture and allograph
features to improve the performance. Schomaker and Bulacu [24] propose a
writer identification method on the uppercase Western script using the
combined features of the edge-direction and connect-component contours. This
method has been extended to the Hinge and Fraglets features in [3] and it has
been shown in [3] that the combined Hinge and Fraglets features provide better
performance than the individual features. Siddiqi and Vincent [4] propose two
feature extraction methods: codebook-based and contour-based features. The
small fragments of the handwritten text are extracted and mapped into a common
space to generate the global feature vector. The contour-based features
describe the orientation and curvature information of the handwritten text.
Instead of combining different types of features, the joint distribution of
different attributes on writing contours also provides a good writer
identification performance. Brink et al. [1] propose the Quill feature which
is the joint distribution of the ink direction and the ink width. The COLD
feature which is the joint distribution of the relation between orientation
and length of line segments computed from ink contours is also applied for
writer identification [28].

Writer identification based on a small amount of data set has also been
studied. Adak and Chaudhuri [29] study the writer identification based on the
isolated characters and numerals. Aubin et al. [30] propose a writer
identification method based on the simple graphemes or single strokes. These
methods extract several handcrafted descriptors on characters or strokes and
SVM is used for recognizing the writer identity.

Convolutional neural networks have also been studied for writer
identification. For example, methods [31, 32] extract features from the last
fully-connected layers in a trained neural network. A multi-stream CNN is
proposed in [33], which can leverage the spatial relationship between
handwritten image patches. Nguyen et.al [10] train a neural network to capture
local features in the whole character images and their sub-regions. The local
features extracted from tuples of images are aggregated to form the global
feature of page images. In [34], a denoising network is used to extract deep
features on small patches. A transfer deep learning from ImageNet is used in
[35] where deep features are extracted on small image patches and then fed to
a SVM classifier. Keglevic et al. [36] apply a triplet network to learn a
similarity measure for image patches and the global feature is computed as the
vector of locally aggregated image patch descriptors. Our previous work in [7]
uses a deep adaptive learning method which learns deep features using a two-
stream neural network for different tasks for writer identification based on
single-word images.

##  III Approach

Given a set of word or text block images with writer identity, the naive way
is to train a neural network on these images for writer prediction. However,
as mentioned before, the neural network usually has one spotting region in the
input image, which dominates the training and the information in other regions
are ignored. In order to explore all writing style information contained in
the whole word image, we force the network to learn the useful information on
the fragments segmented on word images and feature maps computed from CNNs and
combine the evidences of all fragments to make the final decision.

###  III-A Fragment segmentation in Neural Networks

Fragments segmented from handwritten texts usually contain the individual’s
writing style information and are widely used for writer identification. The
basic assumption, as mentioned in [3], is that “the writer acts as a
stochastic generator of ink-blob shapes, or graphemes”. The distribution of
these segmented fragments can be used for writing style description, based on
the classic bag-of-words model. The traditional fragment-based methods need
word or text segmentation. For example, Bulacu and Schomaker [3] segment the
word image at the minima in the lower contour of ink-trace. Siddiqi and
Vincent [4] use a square window to segment fragments on handwritten strokes.

In this paper, we propose a different fragment segmentation method: segment
the fragment in feature maps of the neural network, which contain semantic and
high-level information of the handwriting style. Our method is inspired by the
featurized image pyramids [14], which enable a network to segment fragments on
different scales by scanning on different pyramid levels. The segmentation
processing is the cutting operation, which cuts out a continuous region from a
feature map of a convolutional layer, similar to the region of interest (RoI)
pooling [37]. Fig. 2 shows an example, in which the fragment with the size of
$h\times w\times D$ is segmented on the position $p(x,y)$ given the feature
map with the size of $H\times W\times D$ ($h\ll H$ and $w\ll W$).
Mathematically, each fragment $A$ is defined by:

$A=\text{Crop}\big{(}T,\theta=(x,y,h,w)\big{)}$ (1)

where $T$ is the input tensor and the four-tuple $\theta=(x,y,h,w)$ is the set
of parameters about the position and size of the fragment cropped on the input
tensor $T$. Noted that we only segment on the spatial space and different
fragments can be obtained given the different position $p(x,y)$. The input
image can be considered as a special feature map of the neural network.
Therefore, fragments can also be obtained from input images.

Figure 2: Fragment segmentation in a feature map space with the size: $H\times
W\times D$. The fragment with the size of $h\times w\times D$ is cropped on
the position $p(x,y)$.

###  III-B FragNet networks

Figure 3: A FragNet network has two pathways: feature pyramid (blue color)
which accepts the whole word image as the input and Fragment pathway (green
color) which accepts the fragment as the input. (P)-CBR means the sequences of
P: max-pooling, C: convolutional, B: batch normalization and R: ReLU layers. C
with circle is the concatenate operation. x2 means that two blocks stacked
together. $G_{i}$ and $F_{i}$ are the $i$-th feature maps in the feature
pyramid and fragment pathway, respectively.

The generic architecture of FragNet has a feature pyramid and a fragment
pathway, which are fused by lateral connections. Fig. 3 illustrates the
concept of the FragNet network.

####  III-B1 Feature pyramid.

The feature pyramid (the blue pathway in Fig. 3) is the traditional
convolutional neural network [9] which computes the feature maps hierarchy in
different scales. It contains four $(P)$-$CBR$ blocks, in which $P$ is the
max-pooling layer, $C$ is the convolutional layer, $B$ is the batch
normalization layer and $R$ is the Rectified Linear Units (ReLU) layer. The
kernel size of each convolutional layer is $3\times 3$ and the stride step is
1. The max-pooling with a kernel size of $2\times 2$ and stride step of 2 is
applied after every two convolutional layers in order to reduce the spatial
size of the feature maps and obtain translation invariance. The input of the
feature pyramid is the whole input image, such as the word image in this
paper. Four feature maps, $G_{i},i\in\\{1,2,3,4\\}$ as shown in Fig. 3, are
obtained in the feature pyramid after every two convolutional layers:

$G^{n}_{i}=\circledast(G^{m}_{i-1},W^{m\times 3\times 3\times
n}_{i},W^{b}_{i})$ (2)

where $\circledast$ denotes of the $P$-$CBR$ blocks, $G^{1}_{0}$ is the input
image with the channel size of 1 and $W^{m\times 3\times 3\times n}_{i}$ is
the kernel with the size of $m\times 3\times 3\times n$ in convolutional
layers ($m$ is the channel size of the input feature map and $n$ is the
channel size of the output feature map and $W^{b}_{i}$ is the set of
parameters in the batch normalization layer. The channel numbers of four
feature maps are set to $[64,128,256,512]$, respectively.

####  III-B2 Fragment pathway.

Fragments are segmented from the input image and feature maps on the feature
pyramid. We denote a fragment as $A_{i}$ which is segmented on the $i$-th
feature map $G_{i}$ in the feature pyramid by:

$A_{i}=\text{Crop}\big{(}G_{i},\theta=(x,y,h,w)\big{)}$ (3)

Specially, $A_{0}$ is the fragment segmented from the input image $G_{0}$. The
feature map of the fragment is denoted as $F_{i}$, which is defined as:

$\begin{split}M^{n}_{i}&=\circledast(F^{m}_{i-1},W^{m\times 3\times 3\times
n}_{i},W^{b}_{i})\\\ F_{i}&=[M_{i},A_{i}]\\\ \end{split}$ (4)

where $M_{i}$ is computed by $\circledast$, representing two convolutional
layers $C$, followed by the Batch Normalization layer $B$ and the ReLU layer
$R$. The feature map of the fragment is denoted as $F_{i}$, which is the
concatenation $[]$ of $M_{i}$ and $A_{i}$. Note that $F_{0}=A_{0}$. More
details are shown in Fig. 3.

Specifically, the fragment pathway fuses the information from two directions:
one is from the previous convolutional layer in the fragment pathway and one
is from the fragment segmented from the corresponding convolutional layer in
the feature pyramid. Similar to DenseNet [38], the concatenate operation is
used for merging these two different information. The fragments are segmented
in different feature maps ( $G_{1},G_{2},G_{3},G_{4}$ in Fig. 3), including
the input image $G_{0}$, by the method described in Section III-A. In order to
keep the segmented fragment consistency in the spatial space, the fragment is
cropped in the same space with respect to the input image. For example,
assuming that the segment position is $p_{i}(x,y)$ with the size of $(h,w)$ in
the feature map $G_{i}$ of the feature pyramid, the segment position in the
next feature scale $G_{i+1}$ is $p_{i+1}(x/2,y/2)$ with the size of
$(h/2,w/2)$ when the stride of the max-pooling is 2.

Finally, a global average pooling is used on the fragment pathway’s output to
compute the global feature vector, followed by a fully-connected classifier
layer for writer prediction. Given the input image, fragments can be segmented
in different positions, yielding different fragments and predictions. As
suggested by BagNet [12], the predictions from all fragments in one input
image are averaged to infer the word-level writer evidence, which is inspired
by the bagging ensemble learning [13]. Note that the parameters of the
fragment pathway are shared between different fragments from the same input
image. Because there is no prior knowledge about the locations of the
informative fragments, we use a sliding window strategy to segment all
possible fragments on the input image and feature maps of the feature pyramid.
The stride of the sliding window on $G_{0}$ (the input image) is set to 16 in
the vertical and horizontal directions and reduced to half after each max-
pooling layer on feature maps of the feature pyramid.

One important parameter for the fragment segmentation is the size of fragment
corresponding to the input image. In this paper, we use a square window with
the size of $q\times q$ to cut the fragment in the input image. The resulting
architecture is denoted as FragNet-$q$ with the fragment size of $q\times q$
in the input image. For each fragment, the loss is defined as the cross
entropy loss between the prediction and the ground-truth, which is defined as:

$L_{i}=-\sum_{i}^{M}g_{i}\cdot\text{log}(p_{i})$ (5)

where $M$ is the number of writers in the training set, $g_{i}$ is the ground-
truth of the fragment and $p_{i}$ is the output of the neural network after
softmax. Finally, the total training loss: $L=\sum_{i}^{N}L_{i}$ where $L_{i}$
is the loss of the $i$-th fragment and $N$ is the number of fragments in one
word. During testing, the writer evidence $P$ of the input word image is
computed as the average response of all segmented fragments:
$P=\frac{1}{N}\sum p_{i}$.

Our previous work in [7] for writer identification is also based on single
word images. It has two branches and the information between these two
branches interact at the same stages in the network. The proposed FragNet
method is different from our previous work [7] in several aspects: (1) The aim
of our previous work [7] is to use the adaptive learning to transfer features
learned from different tasks while the goal of FragNet is to explore the
detailed writer style information from fragments of the input word image and
deep feature maps; (2) In [7] the two branches learn different features from
different tasks while the two branches of FragNet learn features from only one
task with different effective receptive fields; (3) The input of the two
branches in [7] is the same word image while the input of the feature pyramid
of FragNet is the whole word image while the input of the fragment pathway is
fragments segmented from the input image and deep feature maps, forcing the
network of the fragment path to learn the detailed handwriting style within
the fragment.

##  IV Experiments

In this section, we provide the experimental results of the proposed FragNet
on four benchmark datasets for writer identification based on word and page
images. Unlike our previous work [7] which randomly splits word images into
training and testing sets, we carefully split all word or text block images
into training and testing subsets to make sure that the word images from one
page appear only in the training or the testing set, which making it possible
for writer identification based on page-level images following the traditional
writer identification protocol.

The word or text blocks used in this experiments are available on the author’s
website 111https://www.ai.rug.nl/~sheng/writeridataset.html, which can be
considered as new benchmark datasets for writer identification based on word
or text block images.

###  IV-A Databases

The proposed method is evaluated on four datasets: IAM [39], CVL [40],
Firemaker [41] and CERUG-EN [25].

Table I: The number of training and testing word images on each data set. Data set | #Writers | #Training | #Testing
---|---|---|---
IAM [39] | 657 | 56,432 | 25,827
CVL [40] | 310 | 62,406 | 34,564
Firemaker [41] | 250 | 25,256 | 11,595
CERUG-EN [25] | 105 | 5,702 | 5,127

IAM [39] is the widely used dataset for writer identification. There are 1,452
pages, which are written by 657 different writers and each writer contributes
a variable number of handwritten pages. For writers who contribute more than
one page, we randomly select one page for testing and the rest pages are used
for training. For writers which contribute only one page, we randomly split
text lines into training and testing sets. The bounding boxes of word images
are provided in this dataset. Therefore, we collect the word images from the
training pages to form the training set and word images from the testing pages
are used for testing.

CVL [40] contains 310 writers and each writer contributes at least 5 pages (27
writers wrote 7 pages). In this paper, the first three pages are used for
training and the rest pages are used for testing. Similar to the IAM dataset,
word images are also available on this dataset.

The Firemaker dataset [41] contains handwritten documents from 250 Dutch
subjects and each writer contributes four pages. Similar to [3], we use the
text images in Page 1 for training and the text images in Page 4 for testing.
The CERUG-EN dataset [25] contains handwritten documents from 105 Chinese
subjects with two paragraphs in English. In this paper, we use the first
paragraph for training and the second paragraph for testing. Since there is no
word box existed in the Firemaker and CERUG-EN datasets, we roughly segment
these documents into text blocks (most text blocks contain only one word).

Figure 4: Several examples of training samples on each data set.

All word images are resized to a fixed size $(64,128)$ by keeping the aspect
ratio without distortions with white-pixel padding until reaching the pre-
determined size. No augmentation method is used during training. Table I shows
the number of training and testing samples and Fig. 4 shows several training
samples on each dataset. From the figure we can see that our training samples
are based on word regions, which could contain at least two characters.

###  IV-B Implementation details

Neural networks are trained by the Adam optimizer [42]. The mini-batch size is
set to 10 due to the limitation of the GPU memory. The initial learning rate
is set to 0.0001 and reduce to half at the epoch 10 and 20 and the network is
fine-tuned with a learning rate of 0.00001 at the last 5 epochs. The whole
training takes 30 epochs. In this paper, we evaluate FragNet-$q$ with
$q\in[64,32,16]$.

All experiments are conducted on one GPU card (Tesla V100-SXM2) with 16GB RAM
using Tensorflow. We use the FLOPs (floating point operations) to measure the
computational complexity. The FLOPs of a convolutional operation is
$c_{i}\times h\times w\times k_{i}\times k_{j}\times c_{o}$ where $c_{i}$ and
$c_{o}$ are the dimensions of input and output tensors, $h$ and $w$ are the
height and width of the tensor, $k_{i}$ and $k_{j}$ are the kernel size. There
are eight convolutional layers on the feature pyramid and the dimensions of
feature maps are [64, 128, 256, 512] in different scales. Thus, the FLOPs of
WordImgNet is around 1.05G. The FLOPs of FragNet-64, FragNet-32 and FragNet-16
are around 7.14G, 7.41G and 3.90G, respectively.

###  IV-C Performance of writer identification based on word images

In this section, we evaluate the proposed method for writer identification
based on word or text block images. We compare FragNet with two baselines:
WordImgNet and ResNet18 [43]. WordImgNet is the network with the same
structure as the fragment pathway of FragNet, receiving the whole word image
as the input and producing one global prediction. For ResNet18 [43], we use
the kernel size of 3$\times$3 at the first convolutional layer and remove the
max-pooling layer to keep the detailed writing style information on the
earlier stage. Our proposed FragNet, however, receives the whole word images
and fragments segmented from the input image and feature maps of the feature
pyramid. For each word image, several fragments are segmented and fed into the
trained FragNet to identify the writer. In the evaluation, we take the average
response of different fragments segmented from the input word image as the
final prediction of the same input word image. The training configurations of
WordImgNet, ResNet18 and FragNet are the same for fair comparison.

####  IV-C1 Comparison with baselines

Table II shows the writer identification performance of two baselines:
WordImgNet, ResNet18 [43] and the proposed FragNet method. From the table we
can see that FragNet-$q$ with a larger fragment size ($q$) provides better
results on the four datasets, which is similar to the conclusion found in
BagNet [12]. In addition, the proposed FragNet-$64$ provides better results
than the baselines WordImgNet and ResNet18 in terms of the Top-1 performance.
The FragNet-$32$ gives better results than the baseline WordImgNet in the IAM
and CVL data sets while worse results in the Firemaker and CERUG-EN data sets.
The results suggest the effectiveness of the proposed FragNet-$64$ for writer
identification based on word images.

Table II: Performance of writer identification with different networks based
on word images.

Net | IAM | CVL | Firemaker | CERUG-EN
---|---|---|---|---
Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5
ResNet18 [43] | 83.2 | 94.3 | 88.5 | 96.7 | 63.9 | 86.4 | 70.6 | 94.0
WordImgNet | 81.8 | 94.1 | 88.6 | 96.8 | 67.9 | 88.1 | 77.3 | 96.4
FragNet-$16$ | 79.8 | 93.3 | 89.0 | 97.2 | 59.6 | 83.2 | 60.6 | 90.3
FragNet-$32$ | 83.6 | 94.8 | 89.0 | 97.3 | 65.0 | 86.8 | 62.3 | 90.1
FragNet-$64$ | 85.1 | 95.0 | 90.2 | 97.5 | 69.0 | 88.5 | 77.5 | 95.6

####  IV-C2 Performance of writer identification with different word lengths

Fig. 5 shows the Top-1 performance of writer identification with different
word lengths on the IAM and CVL datasets since each word image in these two
data sets carries the word label. Similar to our previous work [7], word
images with only two characters contain less texts and thus the performance of
both FragNet-$64$ and WordImgNet is lower than the performance of other words.
The performance of word images with more then three letters is usually high
because there are more allograhpic information in these words which are
sufficient for training. Moreover, FragNet-$64$ provides better results than
WordImgNet with different word lengths, especially on the word images with
only two letters.

Figure 5: Performance of writer identification (Top-1) for FragNet-$64$ and
WordImgNet with different word lengths on the IAM and CVL data sets.

####  IV-C3 Accuracy distribution between FragNet-$64$ and WordImgNet

In Fig. 6, we plot the Top-1 performance within each writer in the four data
sets of FragNet-$64$ against the performance of WordImgNet. We can see that
the accuracy distribution is fairly consistent between FragNet-$64$ and
WordImgNet. The performance of FragNet-$64$ is low on the writers whose
performance of WordImgNet is also low. However, FragNet-$64$ can improve the
performance and provide a high performance (the red boxes in Fig. 6) on these
writers whose performance given by WordImgNet is low, especially on the IAM
dataset.

Figure 6: Scatter plots of the writer-conditional Top-1 accuracy of
FragNet-$64$ against WordImgNet in four datasets. Note that the number of dots
in these figures equals to the number of writers in the corresponding dataset.
The range of both the x- and y-axis are [0,100]. The red boxes show the most
writers with low performance given by WordImgNet and their performance is
improved by the proposed FragNet-$64$.

####  IV-C4 Visualization of the informative fragments

Since the writer evidence is from the input image and fragments, which making
it possible to visualize which fragment combined with the word image
contributes most to the writer identity. In this section, we display these
heatmaps of FragNet-$16$ and FragNet-$32$ for the predicted writer evidence on
word images from the IAM dataset in Fig. 7. Highlighting the most informative
fragments is very useful for explanation.

Figure 7: The word images and the corresponding heatmaps of FragNet-$16$ and
FragNet-$32$ on the IAM dataset. The rectangle in each word image is the most
predictive fragment. The heatmap of each word shows the writer evidence
extracted from each fragment. The spatial sum over the evidence is the total
writer evidence. Failure cases (last column in Fig.(a) with the cyan box):
Note that the prediction will not only rely on fragments segmented in the
input image, but also rely on fragments segmented in feature maps of the
feature pyramid. Therefore, some fragments which contain a small part of texts
or no text can still provide a high evidence for writer identification since
they also receive the global information on the feature pyramid.

####  IV-C5 Comparison with the handcrafted features

We evaluate the effectiveness of features learned by the WordImgNet and
FragNet-$64$ networks for writer identification, comparing with the
traditional handcrafted features. For FragNet, the features of fragments on
the input word image are averaged to form the final feature representation.
Given features extracted on word images, the writer model is built by the
averaging of word features in the training set from the same writer. The
writer identification is performed by the nearest neighbor method, which is
widely used in writer identification [3, 4]. Table III shows the results of
writer identification based on word images. From the table we can see that the
performance of the handcrafted features for writer identification based on
word images is very low since these textural features capture the statistics
information of handwritten text, which usually requires a large amount of text
to obtain a stable representation. However, the features learned by neural
networks provide much better results than the handcrafted features and the
proposed FragNet-$64$ gives the best performance.

Table III: The comparison of writer identification performance based on word
images using different features.

Method | IAM | CVL | Firemaker | CERUG-EN
---|---|---|---|---
Top1 | Top5 | Top1 | Top5 | Top1 | Top5 | Top1 | Top5
Hinge [3] | 13.8 | 28.3 | 13.6 | 29.7 | 19.6 | 40.0 | 14.4 | 32.8
Quill [1] | 23.8 | 44.0 | 23.8 | 46.7 | 21.7 | 43.7 | 24.5 | 51.9
CoHinge [44] | 19.4 | 34.1 | 18.2 | 34.2 | 27.4 | 48.2 | 17.7 | 34.0
QuadHinge [44] | 20.9 | 37.4 | 17.8 | 35.5 | 26.5 | 47.4 | 17.0 | 36.0
COLD [28] | 12.3 | 28.3 | 12.4 | 29.0 | 22.7 | 45.1 | 17.3 | 42.2
Chain Code Pairs [4] | 12.4 | 27.1 | 13.5 | 30.3 | 17.5 | 36.8 | 14.5 | 33.0
Chain Code Triplets [4] | 16.9 | 33.0 | 17.2 | 35.4 | 22.9 | 43.8 | 17.8 | 38.0
WordImgNet | 52.4 | 70.9 | 62.5 | 82.0 | 50.2 | 75.8 | 74.3 | 94.6
FragNet-$64$ | 72.2 | 88.0 | 79.2 | 93.3 | 57.5 | 80.8 | 75.9 | 94.7

####  IV-C6 Performance of writer retrieval

We evaluate the effectiveness of the deep learned features by WordImgNet and
FragNet-$64$ for writer retrieval on the test set. Writer retrieval is similar
to the writer identification problem, which aims to find the word images which
are written by the same writer of the query. If there is only one ground truth
existed in the data set given the query, writer retrieval is exactly the same
as the writer identification problem using the “leave-one-out” strategy [3,
4]. Each word image on the test set is tested against all remaining ones on
the test set and the performance of writer retrieval is evaluated by the mean
average precision (mAP) and Top-1 rate, similar to [21, 45].

Table IV shows the results of writer retrieval using deep features extracted
from WordImgNet and FragNet-$64$. The proposed FragNet-$64$ provides better
results than WordImgNet, which demonstrates that the proposed FragNet-$64$ can
learn a robust and effective deep feature representation for writer retrieval.

Table IV: Performance of writer retrieval with features extracted on different
networks based on word images.

Net | IAM | CVL | Firemaker | CERUG-EN
---|---|---|---|---
Top-1 | mAP | Top-1 | mAP | Top-1 | mAP | Top-1 | mAP
WordImgNet | 79.4 | 0.3011 | 80.9 | 0.2813 | 60.7 | 0.2386 | 80.0 | 0.5017
FragNet-$64$ | 86.5 | 0.4947 | 87.1 | 0.4939 | 65.4 | 0.3310 | 80.0 | 0.5544

Table V: Performance of writer identification with different networks based on
page images.

Net | IAM | CVL | Firemaker | CERUG-EN
---|---|---|---|---
Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5 | Top-1 | Top-5
WordImgNet | 95.8 | 98.0 | 98.8 | 99.4 | 97.6 | 98.8 | 97.1 | 100
FragNet-$16$ | 94.2 | 97.4 | 98.5 | 99.4 | 92.8 | 98.0 | 79.0 | 97.1
FragNet-$32$ | 95.3 | 98.0 | 98.6 | 99.4 | 96.0 | 99.2 | 84.7 | 97.1
FragNet-$64$ | 96.3 | 98.0 | 99.1 | 99.4 | 97.6 | 99.6 | 98.1 | 100

Figure 8: The performance of writer identification based on page images by
features extracted from neural networks trained on different data sets.

###  IV-D Performance of writer identification based on page images

Following the traditional methods which perform writer identification based on
page images, we also evaluate the proposed method for writer identification
based on the global feature extracted on the whole page. We evaluate the
proposed method in two scenarios: within-dataset and cross-datasets
evaluation.

####  IV-D1 Within-dataset evaluation

In this section, we conduct the within-dataset evaluation for writer
identification. The writer evidence of each test page is computed by averagely
aggregating word features extracted by the network trained with training
samples from the same dataset:

$P_{page}=\frac{1}{N}\sum_{w\in page}^{N}P(w)$ (6)

where $P_{page}$ is the writer probability on the $page$ and $P(w)$ is the
writer probability of the word $w$ from the network. $N$ is the total number
of words on the $page$. Table V shows the performance of deep features
extracted from WordImgNet and FragNet for writer identification based on the
page level. Compared to Table II, the performance of writer identification on
page images is much better than the performance on word images. In addition,
the proposed FragNet-$64$ provides slight better results than others in these
four datasets in terms of the Top-1 performance.

####  IV-D2 Cross-datasets evaluation

Given a trained network, the global feature vector of one page is the average
aggregation of deep features of all word images within the page. Once feature
vectors of all pages on one dataset are extracted, writer identification is
performed by the widely used “leave-one-out” strategy. There are many
different methods for extracting the global features by aggregating word image
features in one page. However, as found in [10], the average aggregation
provides the best performance. Therefore, in this paper, we apply the average
aggregation of all word images in one page to compute the global feature for
writer identification. The dimension of the feature vector is equal to the
number of writers on each data set since we extract the deep feature on the
last classification layer.

We provide the performance of deep learned features extracted from networks
trained using different datasets for writer identification based on page
images. The global feature of each page on one dataset is computed as the
average of word features extracted from the network trained on other datasets.
Fig. 8 shows the experimental results of WordImgNet and FragNet-$64$ on
different datasets. For example, the top-left panel shows the performance of
writer identification on the IAM dataset, using deep features trained on the
training set of the IAM, CVL, Firemaker and CERUG-EN datasets. From the figure
we can see that the best performance on each dataset is given by deep features
extracted on the neural network trained on training samples from the same
dataset. The reason is that the handwriting styles of different datasets are
quite different and thus the trained neural network can only capture the
handwriting style which appears in the training dataset. In addition, the
proposed FragNet-$64$ provides better results than WordImgNet, which indicates
that the FragNet can capture the detailed handwriting style information which
does not appear in the training set.

Moreover, the handwriting style on the IAM and CERUG-EN datasets are similar
since features extracted from the network trained on the CERUG-EN dataset also
provide a good performance on the IAM dataset and vice versa. However, the
handwriting style on the Firemaker dataset whose handwritten documents are
wrote in Dutch is quite different from others.

Table VI: Comparison of state-of-the-art methods on the IAM dataset.

Reference | #Writer | Feature | Top-1(%)
---|---|---|---
Siddiqi and Vincent [4] | 650 | Contour and codebook features | 91.0
He and Schomaker [44] | 650 | Best results among 17 handcraft features | 93.2
Khalifa et al. [46] | 650 | Graphemes with codebook | 92.0
Hadjadji and Chibani [47] | 657 | LPQ, RL and oBIF with OC-K-Means | 94.5
Wu et al. [48] | 657 | Scale invariant feature transform | 98.5
Khan et al. [6] | 650 | SIFT+RootSIFT | 97.8
Nguyen et. al [10] | 650 | CNN with sub-images of size $64\times 64$ | 93.1
WordImgNet | 657 | CNN with word images | 95.8
FragNet-$64$ | 657 | CNN with word images and fragments | 96.3

Table VII: Comparison of state-of-the-art methods on the CVL dataset.

Reference | #Writer | Feature | Top-1(%)
---|---|---|---
Fiel and Sablatnig [49] | 309 | SIFT with GMM | 97.8
Tang and Wu [32] | 310 | CNN with joint Bayesian | 99.7
Christlein et al. [21] | 310 | SIFT with GMM and Examplar-SVMs | 99.2
Khan et al. [6] | 310 | SIFT+RootSIFT | 99.0
WordImgNet | 310 | CNN with word images | 98.8
FragNet-$64$ | 310 | CNN with word images and fragments | 99.1

Table VIII: Comparison of state-of-the-art methods on the Firemaker dataset.

Reference | #Writer | Feature | Top-1(%)
---|---|---|---
He and Schomaker [44] | 250 | Best results among 17 handcraft features | 92.2
Wu et al. [48] | 250 | Scale invariant feature transform | 92.4
Nguyen et. al [10] | 250 | CNN with sub-images of size $64\times 64$ | 93.6
Khan et al. [6] | 250 | SIFT+RootSIFT | 98.0
WordImgNet | 250 | CNN with word images | 97.6
FragNet-$64$ | 250 | CNN with word images and fragments | 97.6

Table IX: Comparison of state-of-the-art methods on the CERUG-EN dataset.

Reference | #Writer | Feature | Top-1(%)
---|---|---|---
He and Schomaker [44] | 105 | Best results among 17 handcraft features | 97.1
WordImgNet | 105 | CNN with word images | 98.1
FragNet-$64$ | 105 | CNN with word images and fragments | 100.0

####  IV-D3 Comparison with other studies

In this section, we summarize state-of-the-art methods about writer
identification in the literature in Tables VI, VII, VIII, IX. Please note that
these methods are not comparable to our method because they follow different
experimental protocols. Nevertheless, we can draw some conclusions. Generally,
deep learned features provide better results than the best handcrafted
features shown in [44]. Our proposed method is comparable to other state-of-
the-art methods. Especially, our method is better than the system [10], which
also extracts deep features on handwritten patches with a size of $64\times
64$. The difference is that the patches in [10] are randomly selected in
handwritten documents while fragments in the proposed method are uniformly
segmented from word images and feature maps on the feature pyramid.

##  V Conclusion

This paper proposed a FragNet network for writer identification based on
fragments segmented on the input image and feature pyramid maps in the
convolutional neural network. Writer identification is evaluated based on word
or text block images and page images. We shown that networks trained with word
images and fragments (the proposed FragNet) can provide better performance
than networks trained with only word images (the baseline WordImgNet) for the
word-based and page-based writer identification on four benchmark datasets.
Generally, deep features learned from fragments with a large size provide
better performance. Specifically, FragNet-$64$ provides much better results
than WordImgNet on word images which contains only two letters (Fig. 5) and it
can improve the performance on these writers whose performance given by
WordImgNet is low (Fig. 6). More importantly, the decision made by FragNet is
based on input images and fragments, thus the most informative fragments can
be visualized. Note that the aim of the proposed method is not to achieve
state-of-the-art performance, but to show that the fragment based network can
improve the performance and can be visualized for end users. In addition, the
proposed FragNet could be used for on-line writer identification with a
sliding window strategy.

One of the limitations of the proposed FragNet is that it needs word image or
region segmentation, which is challenging on the highly cursive writing
documents. This limitation leads to a direction of future works, focusing on
extending FragNet for writer identification on any handwritten document (such
as historical documents [50]) without segmentation.

## References

  * [1] A. Brink, J. Smit, M. Bulacu, and L. Schomaker, “Writer identification using directional ink-trace width measurements,” _Pattern Recognition_ , vol. 45, no. 1, pp. 162–171, 2012.
  * [2] V. Pervouchine and G. Leedham, “Extraction and analysis of forensic document examiner features used for writer identification,” _Pattern Recognition_ , vol. 40, no. 3, pp. 1004–1013, 2007.
  * [3] M. Bulacu and L. Schomaker, “Text-independent writer identification and verification using textural and allographic features,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 4, pp. 701–717, 2007.
  * [4] I. Siddiqi and N. Vincent, “Text independent writer recognition using redundant writing patterns with contour-based orientation and curvature features,” _Pattern Recognition_ , vol. 43, no. 11, pp. 3853–3865, 2010\.
  * [5] A. J. Newell and L. D. Griffin, “Writer identification using oriented basic image features and the delta encoding,” _Pattern Recognition_ , vol. 47, no. 6, pp. 2255–2265, 2014.
  * [6] F. Khan, F. Khelifi, M. Tahir, and A. Bouridane, “Dissimilarity Gaussian mixture models for efficient offline handwritten text-independent identification using SIFT and RootSIFT descriptors,” _IEEE Transactions on Information Forensics and Security_ , 2018.
  * [7] S. He and L. Schomaker, “Deep adaptive learning for writer identification based on single handwritten word images,” _Pattern Recognition_ , vol. 88, pp. 64–74, 2019.
  * [8] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [9] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [10] H. T. Nguyen, C. T. Nguyen, T. Ino, B. Indurkhya, and M. Nakagawa, “Text-independent writer identification using convolutional neural network,” _Pattern Recognition Letters_ , 2018.
  * [11] S. Chen, Y. Wang, C.-T. Lin, W. Ding, and Z. Cao, “Semi-supervised feature learning for improving writer identification,” _Information Sciences_ , vol. 482, pp. 156–170, 2019.
  * [12] W. Brendel and M. Bethge, “Approximating cnns with bag-of-local-features models works surprisingly well on imagenet,” _arXiv preprint arXiv:1904.00760_ , 2019.
  * [13] L. Breiman, “Bagging predictors,” _Machine learning_ , vol. 24, no. 2, pp. 123–140, 1996.
  * [14] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2117–2125.
  * [15] H. Yao, S. Zhang, R. Hong, Y. Zhang, C. Xu, and Q. Tian, “Deep representation learning with part loss for person re-identification,” _IEEE Transactions on Image Processing_ , 2019.
  * [16] S. Jetley, N. A. Lord, N. Lee, and P. H. Torr, “Learn to pay attention,” _arXiv preprint arXiv:1804.02391_ , 2018.
  * [17] S. Dargan and M. Kumar, “Writer identification system for Indic and Non-Indic scripts: State-of-the-art survey,” _Archives of Computational Methods in Engineering_ , pp. 1–29, 2019.
  * [18] H. E. Said, T. N. Tan, and K. D. Baker, “Personal identification based on handwriting,” _Pattern Recognition_ , vol. 33, no. 1, pp. 149–160, 2000\.
  * [19] B. Helli and M. E. Moghaddam, “A text-independent Persian writer identification based on feature relation graph (FRG),” _Pattern Recognition_ , vol. 43, no. 6, pp. 2199–2209, 2010.
  * [20] R. Arandjelovic and A. Zisserman, “Three things everyone should know to improve object retrieval,” in _Computer Vision and Pattern Recognition (CVPR)_ , 2012, pp. 2911–2918.
  * [21] V. Christlein, D. Bernecker, F. Hönig, A. Maier, and E. Angelopoulou, “Writer identification using GMM supervectors and exemplar-SVMs,” _Pattern Recognition_ , vol. 63, pp. 258–267, 2017.
  * [22] M. N. Abdi and M. Khemakhem, “A model-based approach to offline text-independent Arabic writer identification and verification,” _Pattern Recognition_ , vol. 48, no. 5, pp. 1890–1903, 2015.
  * [23] G. Ghiasi and R. Safabakhsh, “Offline text-independent writer identification using codebook and efficient code extraction methods,” _Image and Vision Computing_ , vol. 31, no. 5, pp. 379–391, 2013.
  * [24] L. Schomaker and M. Bulacu, “Automatic writer identification using connected-component contours and edge-based features of uppercase western script,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 26, no. 6, pp. 787–798, 2004.
  * [25] S. He, M. Wiering, and L. Schomaker, “Junction detection in handwritten documents and its application to writer identification,” _Pattern Recognition_ , vol. 48, no. 12, pp. 4036–4048, 2015.
  * [26] V. Christlein, D. Bernecker, A. Maier, and E. Angelopoulou, “Offline writer identification using convolutional neural network activation features,” in _German Conference on Pattern Recognition_. Springer, 2015, pp. 540–552.
  * [27] V. Christlein, M. Gropp, S. Fiel, and A. Maier, “Unsupervised feature learning for writer identification and writer retrieval,” in _International Conference on Document Analysis and Recognition (ICDAR),_ , vol. 1, 2017, pp. 991–997.
  * [28] S. He and L. Schomaker, “Writer identification using curvature-free features,” _Pattern Recognition_ , vol. 63, pp. 451–464, 2017.
  * [29] C. Adak and B. B. Chaudhuri, “Writer identification from offline isolated Bangla characters and numerals,” in _International Conference on Document Analysis and Recognition (ICDAR)_ , 2015, pp. 486–490.
  * [30] V. Aubin, M. Mora, and M. Santos-Peñas, “Off-line writer verification based on simple graphemes,” _Pattern Recognition_ , vol. 79, pp. 414–426, 2018.
  * [31] S. Fiel and R. Sablatnig, “Writer identification and retrieval using a convolutional neural network,” in _International Conference on Computer Analysis of Images and Patterns_. Springer, 2015, pp. 26–37.
  * [32] Y. Tang and X. Wu, “Text-independent writer identification via CNN features and joint Bayesian,” in _International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2016, pp. 566–571.
  * [33] L. Xing and Y. Qiao, “Deepwriter: A multi-stream deep CNN for text-independent writer identification,” in _International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2016, pp. 584–589.
  * [34] K. Ni, P. Callier, and B. Hatch, “Writer identification in noisy handwritten documents,” in _Winter Conference on Applications of Computer Vision (WACV)_ , 2017, pp. 1177–1186.
  * [35] A. Rehman, S. Naz, M. I. Razzak, and I. A. Hameed, “Automatic visual features for writer identification: a deep learning approach,” _IEEE Access_ , vol. 7, pp. 17 149–17 157, 2019.
  * [36] M. Keglevic, S. Fiel, and R. Sablatnig, “Learning features for writer retrieval and identification using triplet CNNs,” in _International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2018, pp. 211–216.
  * [37] R. Girshick, “Fast R-CNN,” in _International Conference on Computer Vision (ICCV)_ , 2015, pp. 1440–1448.
  * [38] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4700–4708.
  * [39] U.-V. Marti and H. Bunke, “The IAM-database: an English sentence database for offline handwriting recognition,” _International Journal on Document Analysis and Recognition_ , vol. 5, no. 1, pp. 39–46, 2002.
  * [40] F. Kleber, S. Fiel, M. Diem, and R. Sablatnig, “CVL-database: An off-line database for writer retrieval, writer identification and word spotting,” in _International Conference on Document Analysis and Recognition (ICDAR)_ , pp. 560–564.
  * [41] L. Schomaker and L. Vuurpijl, “Forensic writer identification: A benchmark data set and a comparison of two systems [internal report for the $\\{$Netherlands Forensic Institute$\\}$],” 2000.
  * [42] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [43] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Computer vision and pattern recognition (CVPR)_ , 2016, pp. 770–778.
  * [44] S. He and L. Schomaker, “Beyond OCR: Multi-faceted understanding of handwritten document characteristics,” _Pattern Recognition_ , vol. 63, pp. 321–333, 2017.
  * [45] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” in _International Conference on Computer Vision (ICCV)_ , 2015, pp. 1116–1124.
  * [46] E. Khalifa, S. Al-Maadeed, M. A. Tahir, A. Bouridane, and A. Jamshed, “Off-line writer identification using an ensemble of grapheme codebook features,” _Pattern Recognition Letters_ , vol. 59, pp. 18–25, 2015.
  * [47] B. Hadjadji and Y. Chibani, “Two combination stages of clustered one-class classifiers for writer identification from text fragments,” _Pattern Recognition_ , vol. 82, pp. 147–162, 2018.
  * [48] X. Wu, Y. Tang, and W. Bu, “Offline text-independent writer identification based on scale invariant feature transform,” _IEEE Transactions on Information Forensics and Security_ , vol. 9, no. 3, pp. 526–536, 2014.
  * [49] S. Fiel and R. Sablatnig, “Writer identification and writer retrieval using the fisher vector on visual vocabularies,” in _International Conference on Document Analysis and Recognition (ICDAR)_ , 2013, pp. 545–549.
  * [50] S. Fiel, F. Kleber, M. Diem, V. Christlein, G. Louloudis, S. Nikos, and B. Gatos, “ICDAR2017 competition on historical document writer identification (historical-wi),” in _International Conference on Document Analysis and Recognition (ICDAR)_ , vol. 1, 2017, pp. 1377–1382.