# A weighted analysis to improve the X-ray polarization sensitivity of IXPE

Alessandro Di Marco INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Enrico Costa INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Fabio Muleri INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Paolo Soffitta INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Sergio Fabiani INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Fabio La Monaca INAF – IAPS, via Fosso del Cavaliere, 100, Rome,
Italy I-00133  John Rankin INAF – IAPS, via Fosso del Cavaliere, 100, Rome,
Italy I-00133  Università di Roma “La Sapienza”, Dipartimento di Fisica,
Piazzale Aldo Moro 2, Rome, Italy I-00185  Università di Roma ”Tor Vergata”,
Dipartimento di Fisica’, Via della Ricerca Scientifica, 1, I-00133 Roma, Italy
Fei Xie INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy I-00133  Matteo
Bachetti INAF-OAC, Via della Scienza 5, I-09047 Selargius (CA), Italy  Luca
Baldini Università di Pisa, Dipartimento di Ingegneria dell’Informazione, Via
G. Caruso 16, I-56122 Pisa, Italy  INFN-Pisa, Largo B. Pontecorvo 3, I-56127
Pisa, Italy  Wayne Baumgartner  NASA Marshall Space Flight Ctr., Huntsville,
AL 35812, USA  Ronaldo Bellazzini INFN-Pisa, Largo B. Pontecorvo 3, I-56127
Pisa, Italy  Alessandro Brez INFN-Pisa, Largo B. Pontecorvo 3, I-56127 Pisa,
Italy  Simone Castellano INFN-Pisa, Largo B. Pontecorvo 3, I-56127 Pisa, Italy
Ettore Del Monte INAF – IAPS, via Fosso del Cavaliere, 100, Rome, Italy
I-00133  Niccolò Di Lalla W.W. Hansen Experimental Physics Laboratory, Kavli
Institute for Particle Astrophysics and Cosmology, Department of Physics and
SLAC National AcceleratorLaboratory, Stanford University, Stanford, CA 94305,
USA  Riccardo Ferrazzoli INAF – IAPS, via Fosso del Cavaliere, 100, Rome,
Italy I-00133  Università di Roma “La Sapienza”, Dipartimento di Fisica,
Piazzale Aldo Moro 2, Rome, Italy I-00185  Università di Roma ”Tor Vergata”,
Dipartimento di Fisica’, Via della Ricerca Scientifica, 1, I-00133 Roma, Italy
Luca Latronico INFN-Torino, Via P. Giuria, 1, I-10125 Torino, Italy  Simone
Maldera INFN-Torino, Via P. Giuria, 1, I-10125 Torino, Italy  Alberto Manfreda
INFN-Pisa, Largo B. Pontecorvo 3, I-56127 Pisa, Italy  Stephen L. O’Dell NASA
Marshall Space Flight Ctr., Huntsville, AL 35812, USA  Matteo Perri INAF-OAR,
Via Frascati 33, I-00040, Monte Porzio Catone (RM)  Melissa Pesce-Rollins
INFN-Pisa, Largo B. Pontecorvo 3, I-56127 Pisa, Italy  Simonetta Puccetti ASI,
Via del Politecnico snc, I-00133 Roma, Italy  Brian D. Ramsey  NASA Marshall
Space Flight Ctr., Huntsville, AL 35812, USA  Ajay Ratheesh INAF – IAPS, via
Fosso del Cavaliere, 100, Rome, Italy I-00133  Università di Roma “La
Sapienza”, Dipartimento di Fisica, Piazzale Aldo Moro 2, Rome, Italy I-00185
Università di Roma ”Tor Vergata”, Dipartimento di Fisica’, Via della Ricerca
Scientifica, 1, I-00133 Roma, Italy  Carmelo Sgrò INFN-Pisa, Largo B.
Pontecorvo 3, I-56127 Pisa, Italy  Gloria Spandre INFN-Pisa, Largo B.
Pontecorvo 3, I-56127 Pisa, Italy  Allyn F. Tennant  NASA Marshall Space
Flight Ctr., Huntsville, AL 35812, USA  Antonino Tobia INAF – IAPS, via Fosso
del Cavaliere, 100, Rome, Italy I-00133  Alessio Trois INAF-OAC, Via della
Scienza 5, I-09047 Selargius (CA), Italy  Martin C. Weisskopf NASA Marshall
Space Flight Ctr., Huntsville, AL 35812, USA

(Revised November 25, 2021)

###### Abstract

IXPE is a Small Explorer mission that was launched at the end of 2021 to
measure the polarization of X-ray emission from tens of astronomical sources.
Its focal plane detectors are based on the Gas Pixel Detector, which measures
the polarization by imaging photoelectron tracks in a gas mixture and
reconstructing their initial directions. The quality of the single track, and
then the capability of correctly determining the original direction of the
photoelectron, depends on many factors, e.g., whether the photoelectron is
emitted at low or high inclination with respect to the collection plane or the
occurrence of a large Coulomb scattering close to the generation point. The
reconstruction algorithm used by IXPE to obtain the photoelectron emission
direction, also calculates several properties of the shape of the tracks which
characterize the process. In this paper we compare several such properties and
identify the best one to weight each track on the basis of the reconstruction
accuracy. We demonstrate that significant improvement in sensitivity can be
achieved with this approach and for this reason it will be the baseline for
IXPE data analysis.

X-rays — polarimetry — gas detectors — IXPE — data analysis — statistycal
methods — astrophysics

††journal: AJ

##  1 Introduction

X-ray astronomy had a substantial break-through with the introduction of X-ray
optics mated with imaging detectors in the focal plane. Beside the capability
to image extended sources, resolve complex fields and localize with high
precision point-like sources, the optics had a dramatic impact on the
measurement sensitivity. In experiments with collimators, included the
improvements of modulation collimators and coded masks, the photons from the
source are compared with the fluctuations of the background counts, measured
on the same surface. By means of a telescope the photons collected on the
mirrors area, projected on the optical axis, are focused on a focal plane spot
of very small surface where a very few background counts are expected. In
terms of astrophysics the revolution of the optics, achieved for the first
time with HEAO-2/Einstein satellite, enabled imaging of galaxy clusters,
supernova remnants and black hole jets and detection of a huge number of
extra-galactic sources, introducing X-ray astronomy as a new subject of
cosmology.

Polarimetry was not able to follow this evolution and the main reason was the
instrument technology. Any polarimeter is based on a large modulation, namely
a high dependence of the response to polarization, but an effective
exploitation of the optics in terms of both imaging and background needs also
a good localization capability. The only viable polarimeters in the early
stage of X-ray astronomy were based on Bragg diffraction at 45∘ or Compton
scattering around 90∘. Neither technique localizes the interaction point of
the impinging photon. The situation evolved with the discovery that, thanks to
the new technologies of microelectronics, a finely subdivided gas filled
detector could be used as a polarimeter based upon the photoelectric effect.

The Gas Pixel Detector (GPD) serves as an imaging photoelectric polarimeter,
which has been developed in Italy by the Istituto Nazionale di
Astrofisica/Istituto di Astrofisica e Planetologia Spaziali (INAF-IAPS) and
Istituto Nazionale di Fisica Nucleare (INFN) over the past $\sim$20 years
(Costa et al., 2001; Bellazzini et al., 2003a, 2006, 2007; Soffitta et al.,
2021; Baldini et al., 2021a). The Imaging X-ray Polarimetry Explorer (IXPE,
Weisskopf et al. (2016); Soffitta (2017); O’Dell et al. (2019); Soffitta et
al. (2020)) carries three X-ray telescopes, each with a grazing-incidence
mirror assembly and a GPD focal-plane detector (Soffitta et al., 2020, 2021;
Baldini et al., 2021a) to conduct imaging X-ray polarimetry of cosmic sources.

In photoelectric polarimeters as the GPD, the $s$ atomic orbital electrons are
ejected from the atom with an azimuthal distribution of cos${}^{2}\phi$,
peaked on the direction of the electric vector of the photon. If the azimuthal
angles $\phi$ of the initial direction of the photoelectrons were measured
perfectly, this polarimeter would have a modulation factor of 100%. In
reality, there are several effects that limit the ideal response of the
instrument. Photoelectron path in the gas is traced by the ionization charges
produced along the way, which have to drift to the collection plane; they are
multiplied and eventually read-out on a pixellated plane with finite pixel
size. The quantization of the linear energy loss, the finite pixel dimensions
and the transverse diffusion result in an highly blurred image of the track.
Moreover, photoelectrons may be scattered close to their generation points,
making it difficult to reconstruct their original directions. In the IXPE GPD,
photoelectron tracks are analyzed by a custom reconstruction algorithm
developed in-house (Bellazzini et al., 2003a, b; Fabiani & Muleri, 2014;
Baldini et al., 2021a), which estimates the initial direction of the
photoelectron and the point of the photon absorption. Such an algorithm
features several steps: i) a clustering algorithm to distinguish the
contiguous physical track from isolated noisy pixels; ii) simple calculations
on the moments of the collected charge distribution to distinguish the initial
and final part of the track; iii) a refinement of the calculated quantities
with a higher weight for the pixels at the beginning of the track. The
algorithm calculates also a number of track properties as the track size, the
total energy, etc. The purpose of this paper is to explore such parameters to
determine which correlate better with the response to linear polarization.
These results will be used to weight conveniently the tracks on which a more
accurate reconstruction is possible and verify if a better sensitivity can be
achieved. In the next section the statistical treatment of an event-by-event
weighted analysis is presented. In Section 3 a brief explanation of the track
reconstruction is given and some different tracks properties are considered to
identify the best one to be used as a weight to improve IXPE sensitivity. In
section 4 and 5 the weighted analysis is applied to existing IXPE calibration
data and to estimate the reachable sensitivity on a reference observation.

##  2 Weighted polarization estimation

A convenient approach to the weighting of data collected by X-ray polarimeters
was defined by Kislat et al. (2015). The method was originally developed for
weighting the response of instruments with non-uniform acceptance, but it can
be extended also to increase the contribution of tracks which are better
reconstructed because of higher quality. Here we summarize the statistical
method presented in Kislat et al. (2015). We remark that in this Section and
in the following, equations for $\mathcal{U}$ and $\mathcal{Q}$ Stokes
parameters are multiplied by a factor 2 with respect to the usual definition.
This choice is due to the fact that, as shown in Kislat et al. (2015), for
photoelectric polarimeters when the usual Stokes parameters definition is
applied the polarization degree is
$\mathcal{P}=\frac{2}{\mu}\sqrt{Q^{2}+U^{2}}$ while the adopted definition
allows to obtain the Equation 6 for $\mathcal{P}$ as expected.

In photoelectric polarimeters, the (normalized) modulation Stokes parameters
of a single event are calculated from the the photoelectron emission angle of
the k-th event $\phi_{k}$ and they need to be corrected for the detector
response in a later stage analysis

$\displaystyle i_{k}$ $\displaystyle=$ $\displaystyle 1$ $\displaystyle q_{k}$
$\displaystyle=$ $\displaystyle 2\cos 2\phi_{k}$ (1) $\displaystyle u_{k}$
$\displaystyle=$ $\displaystyle 2\sin 2\phi_{k}$

following an angular distribution Kislat et al. (2015):

$f(\phi)=\frac{1}{2\pi}\left(1+\mathcal{P}\mu\cos(2(\phi-\phi_{0}))\right)$
(2)

with $\mathcal{P}$ being the polarization degree, $\phi_{0}$ the polarization
angle and $\mu$ is the modulation factor, that is, the amplitude of the
instrumental response to 100% polarized radiation.

Weights can be introduced, as done in Kislat et al. (2015), applying a
multiplication factor of the single-event Stokes parameters:

$\displaystyle i_{k}$ $\displaystyle=$ $\displaystyle w_{k}$ $\displaystyle
q_{k}$ $\displaystyle=$ $\displaystyle 2w_{k}\cos 2\phi_{k}$ (3)
$\displaystyle u_{k}$ $\displaystyle=$ $\displaystyle 2w_{k}\sin 2\phi_{k}.$

The overall Stokes parameters of a measurement of $N$ events are obtained as

$\displaystyle\mathcal{I}$ $\displaystyle=$ $\displaystyle\sum_{k=1}^{N}w_{k}$
$\displaystyle\mathcal{Q}$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{N}2w_{k}\cos 2\phi_{k}$ (4)
$\displaystyle\mathcal{U}$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{N}2w_{k}\sin 2\phi_{k}.$

For the $\phi$ distribution of Eq. 2, the expected $\mathcal{U}$ and
$\mathcal{Q}$ values can be estimated

$\displaystyle\langle\mathcal{Q}\rangle$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{N}2w_{k}\int_{0}^{2\pi}\cos 2\phi
f(\phi)d\phi=\mathcal{I}\mathcal{P}\mu\cos(2\phi_{0})$ (5)
$\displaystyle\langle\mathcal{U}\rangle$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{N}2w_{k}\int_{0}^{2\pi}\sin 2\phi
f(\phi)d\phi=\mathcal{I}\mathcal{P}\mu\sin(2\phi_{0})$

then the polarization degree is

$\mathcal{P}=\frac{\sqrt{\mathcal{U}^{2}+\mathcal{Q}^{2}}}{\mu\mathcal{I}}=\frac{\sqrt{Q^{2}+U^{2}}}{\mu},$
(6)

where $Q$ and $U$ are the $\mathcal{Q}$ and $\mathcal{U}$ Stokes parameters
normalized by $\mathcal{I}$ and $\mu$ is the modulation factor, that is, the
amplitude of the instrumental response to 100% polarized radiation.

Following a similar approach, the expected variances for the Stokes parameters
can be determined:

$\displaystyle Var(\mathcal{Q})$ $\displaystyle=$ $\displaystyle
W_{2}\int_{0}^{2\pi}\left(2\cos
2\phi-\mathcal{P}\mu\cos(2\phi_{0})\right)^{2}f(\phi)d\phi=$ (7)
$\displaystyle=$ $\displaystyle
W_{2}\left[2-\mu^{2}\mathcal{P}^{2}\cos^{2}(2\phi_{0})\right]$ $\displaystyle
Var(\mathcal{U})$ $\displaystyle=$ $\displaystyle
W_{2}\left[2-\mu^{2}\mathcal{P}^{2}\sin^{2}(2\phi_{0})\right]$

where we introduced the following quantity:

$W_{2}=\sum_{k=1}^{N}w_{k}^{2}.$ (8)

From the previous equations, we can determine the reconstructed Stokes
parameters to compare with results from theory or other experimental values as
$Q_{r}=Q/\mu$ and $U_{r}=U/\mu$. For these latter values we can obtain the
following uncertainties in case of a large data set

$\displaystyle\sigma_{Q_{r}}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{W_{2}}{\mathcal{I}^{2}}\left(\frac{2}{\mu^{2}}-Q_{r}^{2}\right)}$
(9) $\displaystyle\sigma_{U_{r}}$ $\displaystyle=$
$\displaystyle\sqrt{\frac{W_{2}}{\mathcal{I}^{2}}\left(\frac{2}{\mu^{2}}-U_{r}^{2}\right)}.$

Following the same approach the covariance of $\mathcal{Q}$ and $\mathcal{U}$
can be obtained:

$\displaystyle Cov(\mathcal{Q},\mathcal{U})$ $\displaystyle=$
$\displaystyle-\frac{W_{2}}{2}\mathcal{P}^{2}\mu^{2}\sin(4\phi_{0})$ (10)

and from this latter one we obtain

$Cov(Q_{r},U_{r})=-\frac{W_{2}}{2\mathcal{I}^{2}}\mathcal{P}_{r}^{2}\sin(4\phi_{r})$
(11)

and the Pearson correlation coefficient is

$\rho(Q_{r},U_{r})=-\frac{\mathcal{P}_{r}^{2}\sin(4\phi_{r})}{\sqrt{16-8\mathcal{P}_{r}^{2}+\mathcal{P}_{r}^{4}\sin^{2}(2\phi_{r})}}$
(12)

From these latter relations it is possible to observe that a correlation
between U and Q is present for higher values of polarization degree and $\mu$,
in case of low polarization degree this correlation goes to zero.

In Kislat et al. (2015) a joint error posterior distribution for the
polarization degree and angle estimators has been derived to include the
correlation between the Stokes parameters

$\displaystyle P(\hat{\mathcal{P}},\hat{\phi_{0}}|\mathcal{P},\phi_{0})$
$\displaystyle=$
$\displaystyle\frac{\sqrt{N_{eff}}\hat{\mathcal{P}}^{2}\mu^{2}}{2\pi\sigma}\times$
(13) $\displaystyle
exp\left(-\frac{\mu^{2}}{4\sigma^{2}}\left[\hat{\mathcal{P}}^{2}+\mathcal{P}^{2}-2\hat{\mathcal{P}}\mathcal{P}\cos(2(\phi_{0}-\hat{\phi_{0}}))-\frac{\hat{\mathcal{P}}^{2}\mathcal{P}^{2}\mu^{2}}{2}\sin^{2}(2(\phi_{0}-\hat{\phi_{0}}))\right]\right)$

with $\hat{\mathcal{P}}$ and $\hat{\phi_{0}}$ obtained values of polarization
degree and angle respectively and

$\sigma=\sqrt{\frac{1}{N_{eff}}\left(1-\frac{\mathcal{P}^{2}\mu^{2}}{2}\right)},$
(14)

where we define the “effective” number of counts of the measurement
$N_{\textrm{eff}}=\mathcal{I}^{2}/W_{2}$.

This distribution, when the correlation term is negligible, can be reduced to
the Rice distribution as shown by Vaillancourt (2006). In cases with
$\mathcal{P}$ and $\mu$ not close to 0, the Gaussian approximation for the
marginalized errors can be used as stated in Kislat et al. (2015):

$\sigma^{2}_{\mathcal{P}}\simeq\frac{2-\mathcal{P}^{2}\mu^{2}}{(N_{eff}-1)\mu^{2}}.$
(15)

in case the correlation term can be neglected, it reduces after
marginalization to

$\sigma_{\mathcal{P}}^{2}\simeq\frac{2}{\mu^{2}N_{eff}}.$ (16)

From this equation it is possible to see that minimizing
$\sigma_{\mathcal{P}}$ corresponds to maximizing the modulation factor.

The Minimum Detectable Polarization, MDP99, quantifies the polarization
sensitivity at 99% confidence level. For the weighted analysis it is

$\textrm{MDP}_{99}\approx\frac{4.29}{\mu_{weighted}}\sqrt{\frac{W_{2}}{\mathcal{I}^{2}}}=\frac{4.29}{\mu_{weighted}\sqrt{N_{\textrm{eff}}}},$
(17)

in analogy of the corresponding expression for the unweighted analysis:

$\textrm{MDP}_{99,\textrm{unweighted}}\approx{\mathbf{\frac{missing}{missing}}{4.29}{\mu_{unweighted}\sqrt{N}}}.$
(18)

An older, alternative analysis method is based on a cut analysis, the so-
called “standard cuts” (Muleri et al., 2016). This one is carried out applying
a two step selection of the events where 20% of them are removed. The first
group of events is removed applying an energy cut: the energy spectrum is
fitted with a Gaussian and events outside $\pm 3\sigma$ from the peak center
are removed. The events surviving the energy selection are ordered as a
function of their “eccentricity” and lower-eccentricity tracks are removed up
to a threshold which removes 20% of the initial events including the ones
removed from the energy selection.

This method and every kind of cut/selection analysis is similar to a weighted
analysis, by the fact that data selection is a simple kind of weights where
good events have $w_{k}=1$ and the bad ones $w_{k}=0$. In this case, the
effective number of events is equal to the number of the good ones, while bad
ones are removed by the analysis.

“Standard cuts” were developed for the analysis of monochromatic laboratory
sources and they cannot be applied to observations of astrophysical sources
with continuum spectra, which will be carried out with IXPE. As an alternative
approach is needed to obtain a suitable sensitivity, we propose this weighted
analysis. It is worth noting that, when weights are introduced,
$N_{\textrm{eff}}<N$ such that a better sensitivity is achieved only if the
decrease in the square root of counts’ effective number is balanced by a
sufficient increase of modulation factor.

##  3 Determination of event-by-event best-weight parameter

Photoelectron tracks collected by the GPD on-board IXPE are analyzed by a
custom algorithm to extract relevant information. In this algorithm the
photoelectron track direction is determined on the basis of a two-step moment
analysis that has been refined over the years (Bellazzini et al., 2003a, b;
Fabiani & Muleri, 2014; Baldini et al., 2021a). In the following we summarize
such an algorithm to provide context for the following analysis. In the first
step the barycenter of the charge distribution (as the one of Figure 1) is
calculated by

$\displaystyle x_{B}=\frac{\sum_{i}q_{i}x_{i}}{\sum_{i}q_{i}}$ (19)
$\displaystyle y_{B}=\frac{\sum_{i}q_{i}y_{i}}{\sum_{i}q_{i}},$

where $x_{i}$ and $y_{i}$ are the coordinates of pixels in the image and
$q_{i}$ is the charge collected in it.

Figure 1: Example of ionization track resulting from the absorption of a
5.9-keV X-ray in the GPD. The reconstructed photoelectron direction and impact
point are shown (Sgrò, 2017).

Centering the track in $(x_{B},y_{B})$, the second moment of the charge
distribution is

$M_{2}(\psi)=\frac{\sum_{i}q_{i}\left[(x_{i}-x_{B})\cos(\psi)+(y_{i}-y_{B})\sin(\psi)\right]^{2}}{\sum_{i}q_{i}},$
(20)

where the angle $\psi$ is between the track direction and the $x$ axis. This
is used to determine the axis for the charge distribution of minimal or
maximal extension: in fact, the angle which maximizes or minimizes $M_{2}$ is
obtained imposing $dM/d\psi=0$. The maximum and minimum angles ($\psi_{max}$
and $\psi_{min}$ respectively) are 90∘ apart and the allow to determine the
longitudinal and transverse second moment (corresponding to Track Length, TL,
and Track Width, TW, respectively). At this point $\psi_{max}$ defines the
angle between the track direction and the $x$-axis. The third moment estimated
for $\psi_{max}$

$M_{3}(\psi_{max})=\frac{\sum_{i}q_{i}\left[(x_{i}-x_{B})\cos(\psi_{max})+(y_{i}-y_{B})\sin(\psi_{max})\right]^{3}}{\sum_{i}q_{i}},$
(21)

is used to determine the X-ray impact point position with respect to the
$(x_{B},y_{B})$ coordinates. In fact, this is the less dense part, as the
photoelectron loses more and more energy as it slows down, forming the so-
called Bragg peak. In Figure 1 it is possible to observe that the Bragg peak
region is more dense than the initial part of the track: however the initial
photoelectron emission direction in this part of the track is lost because of
the photoelectron scattering within the gas cell.

In the second step the distance between each pixel and the $(x_{B},y_{B})$
coordinates is estimated in $M_{2}$ units, pixels that have a sign of the
distance different with respect to the one of the $M_{3}(\psi_{max})$ are
removed, pixels having a distance value within a horseshoe region centered at
$(x_{B},y_{B})$ with a minimum and maximum radius defined by 1.5 and 3.5,
respectively, are selected and used to determine the impact point coordinates,
as explained in the following. This method allows to exclude pixels due to the
Bragg peak or Auger electrons. To take into account the energy loss in the gas
cell, pixel charges are weighted by $w=e^{-\frac{D}{w_{0}}}$, with $D$
distance from the impact point and $w_{0}=0.05$. The weighted charges in each
pixel are then used to estimate the $M_{2}$ in the horseshoe region and to
obtain the the final photoelectron track direction.

In Figure 2 we report the horseshoe region, the direction determined in the
first and the second step, but also the true one obtained from a Monte Carlo
simulation. It is possible to observe that in cases where the initial part can
be well identified with respect to the Bragg peak region, the initial
direction of the photoelectron can be reconstructed with a better accuracy, as
the tracks are of better quality (Figure 2-$right$). In case, as in Figure
2-$left$, where the Bragg peak is not well identified, the track direction is
not well determined.

Figure 2: Two examples of simulated tracks at about 4 keV from a Gas Pixel
Detector: $Left$ a circular track with $\alpha\simeq 0.01$; $Right$ a
quasi–linear track with $\alpha\simeq 0.7$. In these plots, the green line
represents the Monte Carlo true direction; the dotted-blue line, the first-
step reconstructed direction; and the dotted-black line, the final
reconstructed direction after Bragg removal (see text).

We compared several quantities calculated by the GPD track-analysis algorithm
to obtain an estimator of the track quality, to be used as a weight in the
subsequent analysis. In the following we present only some of them which are
related to the topological properties of the tracks, in particular:

  * •

Track Length (TL): longitudinal second moment of the track;

  * •

Track Width (TW): transverse second moment of the track;

  * •

Track Size (TS): number of pixel over threshold in the track;

  * •

Track Elongation (TE): ratio of Track Length and Track Width;

  * •

Ellipticity, defined as

$\alpha=\frac{TL-TW}{TL+TW},$ (22)

which quantifies the track’s elongation. It is defined in the interval [0;1],
being 0 for circular tracks and 1 for linear tracks;

  * •

Skewness of the track, defined by the third moment.

To relate these parameters with the response to polarization, we used the
IXPESIM Monte Carlo tool developed by the IXPE team (Baldini et al., 2021b)
and based on Geant4 to simulate a large data set. In particular, we generated
$\simeq 10^{7}$ events produced by 100% polarized source and distributed in
energy with a power-law spectrum with photon index $-2$. To obtain an
approximate but quick evaluation, we assumed that the best quantity is the one
that provides the smallest uncertainty on the degree of polarization, defined
in the previous section, that means to maximize the modulation factor.

In Marshall (2021) it has also been demonstrated that in weighted analysis,
the modulation factor itself is the best weight parameter. This means that we
need to find in the following the parameter which allows to minimize the
$\sigma_{\mathcal{P}}$ and that is also a good proxy for the modulation
factor. We notice that we could have used also the uncertainty on the angle
(as in Equation 37 of Kislat et al. (2015)) but our choice is consistent with
the use of the MDP99 (and not on the error on the angle) as the primary factor
of merit of a polarimeter. Moreover both approaches converge, in practice, to
the selection of the modulation factor as the driver.

The distributions of the parameters under study are shown in Figure 3. The
data have been divided in 20 bins with the same number of events. For each
bin, the modulation factor $\mu_{j}$ is calculated, as shown in Figure 3 and
we estimated the $\sigma^{2}$ on the whole data set as
$\sigma^{2}_{\mu}=\frac{1}{\sum_{j=1}^{20}\frac{1}{\sigma_{\mu_{j}}^{2}}}$;
$\sigma^{2}$ values are reported in Table 1.

---
Figure 3:  Distribution of event’s TL, TW, TS, TE, Ellipticity and Skewness
absolute value for simulated data (left) and modulation factor as a function
of the considered parameter (right).

From this analysis it is evident that the Track Ellipticity provides the
lowest $\sigma_{\mathcal{P}}$ value and is also linearly related with the
modulation factor.

Table 1: Modulation factor variance for each of the parameter considered as a weight. The data set is obtained by Monte Carlo simulations for a power-law energy spectrum in the energy range 2–8 keV and 500000 events. Parameter | $\sigma^{2}(\times 10^{-5})$
---|---
Track Length | 1.98
Track Width | 4.40
Track Size | 2.59
Track Elongation | 2.02
Track Ellipticity | 1.90
Abs. Track Skewness | 4.23

The results obtained above with Monte Carlo simulations are confirmed with a
representative measurement carried out during the calibration of the IXPE
focal-plane detector with a 3.69 keV polarized source, described in Muleri et
al. (2022) as shown by the results in Table 2. The numerical values of Table 1
and Table 2 refer to different data sets with a different energy spectrum and
a different number of events, but they show the same best-weight parameter —
namely, $\alpha$.

Table 2: Modulation Factor variance for each parameter considered as a weight for ground calibration data measured at 3.69 keV. Parameter | $\sigma^{2}(\times 10^{-5})$
---|---
Track Length | 3.06
Track Width | 3.33
Track Size | 3.51
Track Elongation | 2.78
Track Ellipticity | 2.77
Abs. Track Skewness | 3.00

Both elongation and ellipticity provide good performance, but we choose the
latter because its interval is conveniently between 0 and 1 and the modulation
factor grows linearly with the ellipticity parameter following the function
$\mu=0.03+0.804\alpha$.

It is worth noting that $\alpha$ is not a mere proxy of the energy. In Figure
4 the $\alpha$ distributions at 2 keV, 4 keV and 6 keV have been estimated by
Monte Carlo simulations. One can see that, for a monochromatic source,
$\alpha$ is distributed over a wide range of values.

---
Figure 4:  Distribution of Ellipticity, $\alpha$, at 2 keV (blue line), 4 keV
(orange line), 6 keV (green line).

Being that ellipticity is the best parameter for the weighted analysis, we
tried to further improve the sensitivity using as a weight some function of
$\alpha$ — namely, $\alpha^{\lambda}$. To choose the best value of $\lambda$,
we applied it to the simulated data set and calculated for different values of
$\lambda$ the following quantities:

  * •

modulation factor as a function of the energy;

  * •

reduction of the effective-count fraction, $N_{eff}/\mathcal{I}$, as a
function of the energy;

  * •

MDP99 calculated as the mean value of 10 Monte Carlo simulations. From this
latter plot we conclude that MDP99 has a minimum for $\lambda\simeq 0.75$;

The results are reported in Figure 5 and 6. As we increase the value of
$\lambda$, the weighting of data is more and more important with the effect
that the number of effective counts decreases but the modulation factor
increases. When we combine the two to obtain the sensitivity expressed with
the MDP99, the best value for sensitivity – that is the minimum in MDP99 – is
obtained for $\lambda=0.75$.

---
Figure 5:  $N_{eff}/\mathcal{I}$ ($left$) and modulation factor ($right$) as a
function of the energy at different $\lambda$ values, as obtained by Monte
Carlo simulations. See the text.
---
Figure 6:  MDP99 values as a function of $\lambda$ for 10 different Monte
Carlo simulations. All these distributions have a minimum around
$\lambda=0.75$

##  4 Application of the weighted analysis to IXPE ground calibration data

In this section we applied the weighting approach to calibration data obtained
with IXPE Detector Units (DUs) during ground calibrations (Fabiani et al.,
2021). As results are consistent for all the IXPE DUs, we present, as a
representative case, results for DU-FM2.

Data analysis has been performed for both unpolarized and polarized sources to
derive the response to completely polarized and unpolarized radiation. In
particular, in this section we derive the modulation factor and the spurious
modulation amplitude — that is the detector response to unpolarized radiation
— by using the following analyses:

  * •

unweighted/uncut: Analysis without cuts or weights, where all events are
included in the analysis;

  * •

weighted/uncut: Analysis without cuts but with an event-by-event weight equal
to $\alpha^{0.75}$.

  * •

std_cut/unweighted: Analysis with “standard cuts” to remove 20% of the data
but without weights.

Figure 7a shows the spurious modulation as a function of energy as measured
(dots). Note that the unweighted and weighted analysis give similar results,
smaller than the ones obtained with “standard cuts”. This component is removed
in the analysis with an algorithm described by Rankin et al. (2021).

(a) | (b) | (c)
---|---|---
|  |
Figure 7:  Comparison as a function of energy for three different analyses
(unweighted, weighted and “standard cuts”) displaying: (a) measured spurious
modulation; (b) measured modulation factor; (c) relative quality factor.
Although values are shown for the representative case of IXPE DU-FM2, similar
behaviors are obtained for the other DUs aboard IXPE.

The modulation factor as a function of the energy is reported in Figure 7b.
Modulation factor with weights and “standard cuts” is higher than the value
obtained with the unweighted analysis.

The guideline for these analysis approaches is always the maximization of the
so-called “Factor of Quality” (Pacciani et al., 2003). This is the product of
the modulation factor by the square root of the efficiency, which had been
already used effectively for the selection of the design parameters of the
detectors. Thus, to have an overall idea of the improvement in sensitivity,
taking into account the $N_{eff}$ contribution with experimental data, the
quality factor is given by $\mu\sqrt{f_{w}}$, where $f_{w}$ is the fraction of
events $N_{eff}/N$ after cuts/weights analysis. In Figure 7c the quality
factor for each analysis is normalized to the unweighted case, showing that
the weighted analysis provides the best improvement in sensitivity over the
entire IXPE energy range.

##  5 Sensitivity improvement on a simulated IXPE observation

---
Figure 8:  Distribution of MDP99 values for 200 IXPE simulated observations of
a point-like source with energy spectrum described by a power-law with number
index -2, a flux 10-11 erg/cm2/s in the energy range 2–8 keV and 10 days of
integration time. MDP99 values are estimated for weighted ($Left$) and
unweighted ($Right$) analysis. See text.

Above we evaluated the improvement achieved by applying the weighted analysis
to a monochromatic beam, albeit for different energies. However IXPE (and any
other polarimetry mission except the Bragg-diffraction polarimeters)
integrates photons within a continuous energy band, in order to achieve
adequate sensitivity. Depending upon the celestial source modeling or its
intensity, it could be the whole energy range of the instrument. We want to
know how the improvements detected at single energy values convert into an
improved detection capability for a continuum source.

To estimate the improvement of the weighted analysis with respect to the
unweighted one, we simulated an observation with IXPE of a reference point-
like source with a power-law spectrum (number index -2), a flux of 10-11
erg/cm2/s in the energy range 2–8 keV and 10 days of integration time. In the
simulation, we considered the GPD quantum efficiency at launch (following the
model described by Baldini et al. (2021a) and applying to each IXPE DU the
expected DME pressure at launch), the UV filters transparency and the optics
effective area as measured at NASA Marshall Space Flight Center during ground
calibrations of the optics. We simulated 200 data sets in IXPESIM obtaining
MDP99 values in the range (5.2–5.7)% for the weighted analysis and (5.9-6.6)%
for the unweighted one. Figure 8 shows the distribution of these values, which
are compatible with a Gaussian distribution with mean values 5.5% and 6.2% for
the weighted and the unweighted analysis, respectively.

This kind of analysis cannot be performed with the “standard cuts” approach
because the energy spectrum of the source is continuous. This result shows
that the weighted analysis by using $\alpha^{0.75}$ as an optimal weight can
allow a significant improvements in polarization sensitivity. The weighted
analysis achieves a sensitivity within the IXPE scientific requirement that
MDP99 not exceed 5.5% for the reference source.

##  6 Conclusions

In this paper we applied the weighted analysis described in Kislat et al.
(2015) to data collected by the focal-plane X-ray polarimeter on-board IXPE,
which are the GPDs. Starting from quantities calculated using the dedicated
track reconstruction analysis developed for the GPD, we identified a best-
weight parameter for both simulated and real data. The weighted analysis
improves the IXPE sensitivity.

In this analysis we showed that the modulation factor is affected
significantly by weights and its value depends upon the “strength” of the
weighting. However, the gain on sensitivity, expressed as a reduction of
MDP99, is partially offset by a reduction in the effective number of events
($N_{eff}$). Among different choices for the weighting, we used the track
ellipticity, which provides the best sensitivity and has the advantage that
its range is [0,1].

In such an assumption, when the analysis is applied to calibration data we
conclude the following:

  * •

Application of weights reduces spurious modulation with respect to values
obtained with “standard cuts” used in IXPE calibration analysis;

  * •

Modulation factor is larger than the value obtained with “standard cuts” or
unweighted/uncut analysis.

We evaluated by means of Monte Carlo simulations the improvement in
sensitivity considering a Crab-like spectrum source. Our analysis showed that,
with respect to an unweighted scheme, the weighted scheme improves the
sensitivity by 13%. This translates in 30% reduction in observing time which
is like adding a fourth telescope to the three already available. The weighted
scheme for track analysis is now considered the baseline for IXPE data
analysis.

IXPE data are analyzed on ground, as briefly explained in Soffitta et al.
(2021). Binary telemetry files are converted into FITS standard format
containing all the information provided by the Instrument and other relevant
information from the spacecraft. In particular, they contain track images that
are analyzed to estimate the Stokes parameters and $\alpha^{0.75}$ that is the
best weight available to be used, as explained in this paper. IXPE data will
be publically released to the scientific community with a format including for
each event a weigjt to apply, given by $\alpha^{0.75}$.

##  7 Acknowledgments

The Italian contribution to the IXPE mission is supported by the Italian Space
Agency (ASI) through the contract ASI-OHBI-2017-12-I.0, the agreements ASI-
INAF-2017-12-H0 and ASI-INFN-2017.13-H0, and its Space Science Data Center
(SSDC), and by the Istituto Nazionale di Astrofisica (INAF) and the Istituto
Nazionale di Fisica Nucleare (INFN) in Italy.

IXPE is a NASA Astrophysics Small Explorers (SMEX) mission, managed by MSFC
and overseen by the Explorers Program Office at GSFC.

## References

  * Baldini et al. (2021a) Baldini, L., Barbanera, M., Bellazzini, R., et al. 2021a, Astroparticle Physics, 133, 102628, doi: https://doi.org/10.1016/j.astropartphys.2021.102628
  * Baldini et al. (2021b) Baldini, L., Bellazzini, R., Castellano, S., et al. 2021b, In preparation
  * Bellazzini et al. (2003a) Bellazzini, R., Angelini, F., Baldini, L., et al. 2003a, in Polarimetry in Astronomy, ed. S. Fineschi, Vol. 4843, International Society for Optics and Photonics (SPIE), 372 – 382, doi: 10.1117/12.459380
  * Bellazzini et al. (2003b) Bellazzini, R., Angelini, F., Baldini, L., et al. 2003b, in Polarimetry in Astronomy, ed. S. Fineschi, Vol. 4843, International Society for Optics and Photonics (SPIE), 383 – 393, doi: 10.1117/12.459381
  * Bellazzini et al. (2006) Bellazzini, R., Spandre, G., Minuti, M., et al. 2006, Nuclear Instruments and Methods in Physics Research A, 566, 552, doi: 10.1016/j.nima.2006.07.036
  * Bellazzini et al. (2007) —. 2007, Nuclear Instruments and Methods in Physics Research A, 579, 853, doi: 10.1016/j.nima.2007.05.304
  * Costa et al. (2001) Costa, E., Soffitta, P., Bellazzini, R., et al. 2001, Nature, 411, 662, doi: 10.1038/35079508
  * Fabiani & Muleri (2014) Fabiani, S., & Muleri, F. 2014, Astronomical X-Ray Polarimetry, Astronomia e astrofisica (Aracne).  https://books.google.it/books?id=K_MFogEACAAJ
  * Fabiani et al. (2021) Fabiani, S., Muleri, F., Di Marco, A., et al. 2021, in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, S. Nikzad, & K. Nakazawa, Vol. 11444, International Society for Optics and Photonics (SPIE), 1029 – 1037, doi: 10.1117/12.2566998
  * Kislat et al. (2015) Kislat, F., Clark, B., Beilicke, M., & Krawczynski, H. 2015, Astroparticle Physics, 68, 45, doi: https://doi.org/10.1016/j.astropartphys.2015.02.007
  * Marshall (2021) Marshall, H. L. 2021, The Astronomical Journal, 162, 134, doi: 10.3847/1538-3881/ac173d
  * Muleri et al. (2016) Muleri, F., Soffitta, P., Baldini, L., et al. 2016, in Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, T. Takahashi, & M. Bautz, Vol. 9905, International Society for Optics and Photonics (SPIE), 1401 – 1407.  https://doi.org/10.1117/12.2233206
  * Muleri et al. (2022) Muleri, F., Piazzolla, R., Di Marco, A., et al. 2022, Astroparticle Physics, 136, 102658, doi: https://doi.org/10.1016/j.astropartphys.2021.102658
  * O’Dell et al. (2019) O’Dell, S. L., Attinà, P., Baldini, L., et al. 2019, in UV, X-Ray, and Gamma-Ray Space Instrumentation for Astronomy XXI, ed. O. H. Siegmund, Vol. 11118, International Society for Optics and Photonics (SPIE), 248 – 261, doi: 10.1117/12.2530646
  * Pacciani et al. (2003) Pacciani, L., Costa, E., Persio, G. D., et al. 2003, in Polarimetry in Astronomy, ed. S. Fineschi, Vol. 4843, International Society for Optics and Photonics (SPIE), 394 – 405.  https://doi.org/10.1117/12.459275
  * Rankin et al. (2021) Rankin, J., Muleri, F., Tennant, A. F., et al. 2021, In preparation
  * Sgrò (2017) Sgrò, C. 2017, in UV, X-Ray, and Gamma-Ray Space Instrumentation for Astronomy XX, ed. O. H. Siegmund, Vol. 10397, International Society for Optics and Photonics (SPIE), 104 – 110, doi: 10.1117/12.2273922
  * Soffitta (2017) Soffitta, P. 2017, in UV, X-Ray, and Gamma-Ray Space Instrumentation for Astronomy XX, ed. O. H. Siegmund, Vol. 10397, International Society for Optics and Photonics (SPIE), 127 – 135, doi: 10.1117/12.2275485
  * Soffitta et al. (2020) Soffitta, P., Attinà, P., Baldini, L., et al. 2020, in Space Telescopes and Instrumentation 2020: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, S. Nikzad, & K. Nakazawa, Vol. 11444, International Society for Optics and Photonics (SPIE), 1017 – 1028, doi: 10.1117/12.2567001
  * Soffitta et al. (2021) Soffitta, P., Baldini, L., Bellazzini, R., et al. 2021, The Instrument of the Imaging X-ray Polarimetry Explorer.  https://arxiv.org/abs/2108.00284
  * Vaillancourt (2006) Vaillancourt, J. E. 2006, Publications of the Astronomical Society of the Pacific, 118, 1340, doi: 10.1086/507472
  * Weisskopf et al. (2016) Weisskopf, M. C., Ramsey, B., O’Dell, S., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9905, Space Telescopes and Instrumentation 2016: Ultraviolet to Gamma Ray, ed. J.-W. A. den Herder, T. Takahashi, & M. Bautz, 990517, doi: 10.1117/12.2235240


11institutetext: Kanishka Misra and Julia Taylor Rayz 22institutetext: Purdue
University, West Lafayette, IN 47906, USA
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Finding Fuzziness in Neural Network Models of Language Processing

Kanishka Misra and Julia Taylor Rayz

###### Abstract

Humans often communicate by using imprecise language, suggesting that fuzzy
concepts with unclear boundaries are prevalent in language use. In this paper,
we test the extent to which models trained to capture the distributional
statistics of language show correspondence to fuzzy-membership patterns. Using
the task of natural language inference, we test a recent state of the art
model on the classical case of temperature, by examining its mapping of
temperature data to fuzzy-perceptions such as cool, hot, etc. We find the
model to show patterns that are similar to classical fuzzy-set theoretic
formulations of linguistic hedges, albeit with a substantial amount of noise,
suggesting that models trained solely on language show promise in encoding
fuzziness.

##  1 Introduction

Fuzziness of variables such as hot, tall, etc. is often reflected in language
use. Humans tend to use vague constructions such as “Joe is quite tall” or
“it’s very cold outside” in their everyday language and are still able to
successfully communicate their intent. Fuzzy sets zadeh1965fuzzy  provide us
with a calculus for handling such imprecise expressions, and have been used in
numerous engineering applications, but have yet to be fully adopted by the
Natural Language Processing (NLP) community carvalho2012critical . Computing
with Words (CWW) zadeh1999fuzzy  initiatives followed a similar trend: they
are described by engineers and computer scientists and used in some
applications, but are not trending in publications of natural language
processing (NLP) fields.

A prevalent hypothesis in the computational linguistics and natural language
processing communities is that the distributional statistics of language use
can guide the process of acquiring word meaning — that “You shall know the
meaning of a word by the company it keeps” firth1957synopsis . This
distributional hypothesis has been the basis of a number of prominent language
representation architectures, ranging from early neural network-based vector
space models such as word2vec mikolov2013distributed  to more recent models
that fall under the category of “pre-trained language models,” such as BERT
devlin-etal-2019-bert . A key factor in models that follow the distributional
hypothesis is that their knowledge is acquired solely from linguistic input,
and reflects the extent to which the statistics present in the language can
inform the model about the world. Indeed, recent studies have shown models
such as BERT to show great feasibility in capturing factual
petroni2019language  and categorical knowledge weir2020probing  by
investigating their outputs on word prediction prompts such as “A robin is a
.” While these analyses shed light on how well prediction in such models
reflects what is generally known to be true in the world (in a bivalent
setting of strict truth and false), much is unknown about the manifestation of
vagueness within the semantic knowledge that they capture.

We argue in this paper that models that are built to interpret and act upon
language should also be able to account for vagueness in categories and
predicates, which have borderline cases where the exact truth value is
unclear. This is especially important in the case of systems involving
rational human interaction, where, for example, a computer can communicate to
the user how icy a road is on a particular wintry night by approximating it
from precise data van2012not , and for this to be made possible, our models of
language processing must encode aspects vagueness or fuzziness. To test this,
we investigate a state of the art NLP model (RoBERTa${}_{\textit{large}}$
liu2019roberta ) in its ability to encode fuzziness of language, specifically
in this case, language about a classical fuzzy concept: temperature. We
formulate the probing experiment as a Natural Language Inference (NLI) task
where the model is supplied with precise descriptions about temperature data
and the model is evaluated based on the extent to which that data maps to
opportunistically selected fuzzy categories such as freezing, warm, etc.

##  2 The concept of vagueness and the vagueness of concepts

We define vagueness to be the phenomenon underlying the uncertainty in
demarcating concepts, i.e., the lack of clear-cut boundaries separating
membership to one concept with the other.

###  2.1 Psychological significance

Experimentally, vagueness has been of interest in the psychological study of
concepts. The seminal work of Rosch rosch1975cognitive  showed consistently
that people perceived category membership as a graded phenomenon, as opposed
to being a matter of crisp boundaries. Labov labov1973boundaries  experimented
on the boundaries of commonplace concepts such as cup, mug, and bowl, by
presenting human subjects with exemplar drawing of the objects. He found
subjects to be consistent in attributing names to object drawings with typical
visual characteristics, while showing large inconsistencies between each other
while naming drawings where the object was visually intermediate between
categories (for instance being too tall to be a member of bowl but too wide to
be a cup), indicating the close relationship of lack of consensus with
vagueness. Additionally, the work of McCloskey and Glucksberg
mccloskey1978natural  involved experiments where human subjects had to
explicitly rate whether a particular category (e.g. fruit) applied to multiple
items in consideration (e.g. apple, banana). The authors found that when the
same group of people were asked to perform the experiment again, after a few
weeks, the subjects tended to be highly inconsistent with their original
judgements for borderline cases (e.g. tomato), suggesting that vagueness also
involves individual uncertainty.

###  2.2 Handling vagueness using fuzzy interpretations of linguistic hedges

Linguistic hedges lakoff1975hedges , such as sort of, technically, etc., are
central to vagueness in language. Zadeh’s seminal work zadeh1965fuzzy
proposes a calculus to handle such fuzziness in scalar adjectives such as
tall, old, etc. Subsequent work zadeh1972fuzzy  proposed to explicitly treat
hedges, such as very, more-or-less, slightly as operations on pre-defined
fuzzy sets. Usually, the definition of the membership functions and fuzzy sets
is left to the experts, as it is one of the most difficult tasks in a
successful fuzzy expert systems martine2002linguistic . For instance, Zadeh
proposed that for a given fuzzy set $F=\\{(x,\mu_{F}(x))\\}$, hedges such as
very, extremely, etc. can be modeled using “concentration” functions that pull
the curve of memberships for $F$ inward, such as $\textit{very
}F=\mu_{F}^{2}(x)$. Meanwhile, hedges such as slightly, almost were modeled
using “dilation” functions that spread out the values of the membership
function, such as $\mu_{F}^{1/2}(x)$. Zadeh also proposed more complex
interpretation of hedges, by employing the use of “intensifiers.” An example
of applying such recommendations to the concept of tall is shown in fig. 1.
Whether or not Zadeh’s proposal for fuzzy-interpretations of linguistic hedges
is correct, they provide a useful method for handling vague predicates in
semantics and serve as inspiration for some of our analyses.

t]

Figure 1: Possible membership transformations for linguistic hedges of _tall_
(with membership function $\mu(h)$, generated using a general bell curve): (1)
slightly tall – $\mu^{1/2}(h)$; (2) very tall – $\mu^{2}(h)$; and (3)
_extremely tall_ – $\mu^{4}(h)$, where $h$ is height. Note that this is solely
for illustrative purposes and is not an exhaustive list of hedges of tall.

While fuzzy sets in general, and linguistic hedges—modeled as operators—are
defined by experts to model data, we wonder whether very large amount of data
can model such linguistics hedges. Given a membership function, and some point
$x$, one can determine whether this point $x$ is a member of a fuzzy set, and
to what degree. What we are interested in is an answer to this question: given
a point $x$, and approximate membership value of some fuzzy set, is it
possible to reconstruct a fuzzy membership function for $x\in X$ where $|X|$
is very large.

In other words, we can assume that if somebody says that it is 90∘F outside,
we find a membership degree of hot, very hot, slightly hot, etc. using an
expert-determined fuzzy membership function. What is less clear is whether an
oracle (let us assume that we have such a thing) would be able to map 90∘F to
some relative position of hot, very-hot, etc. without knowing the actual
equations for the membership functions.

More specifically, we wonder if the notion of fuzzy sets and hedges—modeled as
proposed by Zadeh—can be reconstructed from very large data. If such
reconstruction can be done, we should see graphs that are qualitatively
similar to that of fig. 1. However, it should be noted that the linguistic
hedges are indeed linguistic in nature, and thus, they can allow synonym
substitutions. In particular, somebody very very tall may be called a giant,
and our data may show correlation with that word choice, but not with “very
very tall.” While this raises the complexity of analysis, we note that it is
essential in order to facilitate conversational interactions between computers
and humans, but leave the fine-grained details for future work.

##  3 Pre-trained Language Models

Recent advances in the field of NLP have lead to the rise of a class of highly
parameterized neural network-based models called pre-trained language models
(PLMs). These models are trained on vast amounts of text using the language
modeling objective — estimating probabilities of missing words in
context.111we assume a general form of the language modeling objective which
also includes its bidirectional sense A common architecture that governs the
computation in these models is the transformer architecture
vaswani2017attention . The attention module in the transformer architecture
allows PLMs to ‘remember’ all words in a given sentence context, while
predicting the missing word, thereby making its internal representations more
finely attuned to the entire context. Though originally proposed for ranking
sentences (by assigning them probabilities), the language modeling objective
guides PLMs into learning general-purpose language representations which can
be fine-tuned to any supervised learning task end-to-end, i.e., all
representations get updated in order to optimize for a given task(s). Their
expressiveness and adaptability has enabled PLMs to achieve state of the art
results on several high-level NLP tasks, such as Question Answering, Natural
Language Inference, Reading Comprehension, etc. PLMs come in two main
variants: (1) Incremental PLMs, which are trained to predict words when
conditioned on a left context, e.g., “I went to the library to  ”; and (2)
Masked PLMs, which have bidirectional context while predicting tokens, e.g.,
“I went to the  to read.” Examples of Incremental PLMs include GPT2
radford2019language  while those of Masked PLMs include BERT devlin-
etal-2019-bert  and RoBERTa liu2019roberta .

##  4 Vagueness in Natural Language Inference Models

In this section, we present our behavioral experiment targeting the extent to
which NLP models, trained on very large data, are sensitive to
fuzziness/vagueness of concepts. We specifically consider the classical fuzzy
concept of temperature, where, for instance, the transition from cool to warm
is gradual, and there is no clear-cut separation between when someone
perceives their surrounding to fall under either of the two categories. To
probe for how models encode the fuzziness of temperature, we rely on the task
of natural language inference (NLI) bowman2015large .

NLI, also referred to as “recognizing textual entailment” (RTE), is the task
of predicting the logical entailment relation between pairs of sentences.
Specifically, a model trained to do this task is supplied with two sentences,
a premise and a hypothesis, and it has to predict whether the premise entails
the hypothesis. For instance, the sentence “a dog is running towards the man”
logically entails “an animal is moving towards a human”, while “the boy is
mowing the lawn” contradicts “the boy is sleeping”. Note that it is assumed
that the premise and the hypothesis must refer to the same event in order for
this task to be carried out. NLI features prominently in many language
processing benchmarks wang2018glue ; wang2019superglue , highlighting its
importance in progress towards achieving true natural language understanding,
and by extension, an understanding of the world.

We argue that the task of NLI is suitable in our endeavor of exploring
fuzziness in models due to the amount of flexibility it offers. Models trained
to perform NLI are capable of making—in theory—an infinite number of
inferences for a given premise, thereby enabling the testing of a number of
pre-defined phenomena of interest. For instance, Wang et al. wang2018glue
formulate many natural language understanding phenomena such as lexical
entailment (dog entails animal), monotonic reasoning (I do not have a blue
book does not entail I have no books), symmetry (John married Josh entails
Josh married John but Larry likes Sally does not entail Sally likes Larry),
etc. as NLI tasks in order to test a range of models. A caveat in
investigating semantic phenomena by studying model predictions on such a task
is that the logical relation between the premise and the hypothesis is pre-
defined, making it easy for the researcher to relate the model’s predictions
with its knowledge about the specific phenomena of interest. However, when it
comes to the study of vagueness or fuzziness—which is arguably an integral
component of understanding the world, (see section 2)—knowing exactly that a
temperature of 50∘ F entails cool or warm is un-achievable due to the
epistemic uncertainty surrounding the concept of vagueness itself
williamson1994vagueness . However, the exploration of vagueness within the
models’ behavioral responses in an NLI setting can still be possible if one
supplies multiple plausible hypotheses for a given premise, by relying on
commonsense knowledge and intuition. Comparing the behavior of the model in
its predictions on the different hypotheses, then, facilitates roughly
exploring the model’s encoding of the fuzziness of the given concept.
Revisiting the earlier example, one can compare the extent to which NLI models
prefer attributing an entailment relation to a temperature of 50∘ F with a
number of fuzzy categories that may encompass the concept of temperature, such
as warm, cool, moderate, etc. In our demonstrative experiments, described
below, we test an existing NLI model to classify a natural language premise
describing the temperature into five arbitrarily chosen fuzzy categories:
freezing, cold, cool, warm, hot.

###  4.1 Methodology

#### Model investigated

We perform our experiments on RoBERTa${}_{\textit{large}}$ liu2019roberta ,
fine-tuned on the MNLI dataset williams2018broad . The RoBERTa model’s core
architecture is similar to that of BERT devlin-etal-2019-bert , i.e., it uses
a transformer trained to bidirectionally predict missing words in context.
Unlike BERT, it does not have the next sentence prediction task, and is
trained on 10-times the amount of text data (160 GB) as compared to BERT (16
GB). To fine-tune on MNLI, the model accepts two sentences, a premise and a
hypothesis, in the following format:

> <cls> Premise </s></s> Hypothesis </s>,

where <cls> is a classification token, and </s> acts as a sentence separator.
The model estimates scores (in the form of log-probabilities) for the three
classes: entailment, contradiction, and neutral, by applying a linear layer
that accepts the representation for the <cls> token and produces a three-
dimensional vector with scores for each label. We follow this format of
representing the input to the model in our experiments.

Table 1: Configurations used to generate the analytical dataset. Note: The columns “Location Phrase” and “Fuzzy Category” encompass all rows. Unit |  | Temp.
---
Range
Location Phrase |  | Fuzzy
---
Category
Total Instances
No unit | $[-50,122]$ |  No-Location, in the bedroom, in the living room, in the basement, outside, inside. |  freezing, cold, cool, warm, hot | 5190
Fahrenheit (∘F) | $[-50,122]$ | 5190
Celsius (∘C) | $[-50,50]$ | 3030

#### Stimuli

Like any standard NLI input, our stimuli are composed of pairs of premise and
hypothesis sentences. The premise sentences are natural language descriptions
of temperature measurements, with the option of a specified location as well
as the unit of measurement. For example, “It is $n$ degrees outside.” For
every premise, we construct a hypothesis about each of the plausible fuzzy
category to be inferred from the temperature value specified in the premise —
freezing, cold, cool, warm, hot. For example, “It is warm in the basement.”
Our premise and hypothesis templates are shown below, with text in ‘{}’
indicating an optional element:

> Premise Template: It is [temperature] degrees {units} [location-phrase].
> Hypothesis Template: It is [fuzzy-category] [location-phrase].

In summary, we vary between three different unit options, six different
location phrases, and five different fuzzy sets in generating our premise-
hypothesis pairs, amounting to 54 different settings for every degree of
measurement. The configurations we apply to generate our analytical dataset is
presented in table 1. The location phrases are selected opportunistically. In
total we test on 13,410 different sentence pairs.

#### Analysis and Results

To examine the model’s behavior in making fuzzy inferences from a given
natural language description of a temperature setting, we compute the
probability the model assigns to the “entailment” label for every premise-
hypothesis pair. Formally, for a given pair $\langle P_{i},H_{ij}\rangle$,
where $i$ is the temperature in the premise, and $j$ is the perceived fuzzy
category specified in the hypothesis, we compute the entailment score
$\mathcal{E}$ as:

$\displaystyle\mathcal{E}=p(\text{``entailment''}\mid\langle
P_{i},H_{ij}\rangle).$ (1)

Since this computation is probabilistic, we cannot translate it directly into
a membership function. We instead focus on the patterns the model shows in its
entailment scores for all the fuzzy categories we consider, in order to
characterize its behavior in terms of how well it can mimic qualitative
properties of membership functions, both typical and atypical. Table 2 shows
two randomly selected premises, their corresponding hypotheses, and
$\mathcal{E}$-values computed using the model that we study.

The results of our experiments are collectively shown in fig. 2. Each plot in
fig. 2 represents the model’s $\mathcal{E}$-values for each of the five
hypotheses types in a given unit of measurement and location setting. To
better understand general patterns in each setting, we smooth out the noise in
our measurements by fitting a Generalized Additive Model (GAM)
hastie1990generalized . The first thing to note from our results is that,
overall, model shows noisy but qualitatively sound numeric knowledge patterns
based on how the raw temperature values relate to the model’s synthetic
perception — large values typically show greater $\mathcal{E}$-values for the
warm and hot hypotheses than for hypotheses representing freezing, cold, and
cool. This is especially noticeable upon comparing no-unit (fig. 2a) and
Fahrenheit (fig. 2b) conditions with that of Celsius (fig. 2c), we see that
the model has approximately grasped what is considered to be “cold” in the
Celsius condition — the model shows almost zero-membership values for warm and
hot when the temperature specified in the premise is below 0∘C and gradually
goes up to positive $\mathcal{E}$-values as the temperature increases. In the
Fahrenheit condition, however, the model is unable to show exactly the same
pattern (0∘C $=$ 32∘F), suggesting a slight lack of internal consistency.

Table 2: Examples of the extent to which RoBERTa${}_{\textit{large}}$—fine-tuned on the MNLI dataset williams2018broad —is able to infer one of five arbitrarily chosen temperature categories when given the temperature of a given room (no unit specified). Note: “Entailment Scores” are not the same as fuzzy membership values, and are instead used to perform the behavioral analysis of the model. Premise | Hypotheses |  Entailment Score ($\mathcal{E}$)
---|---|---
It is 0 degrees in the bedroom. | It is freezing in the bedroom. | 0.956
It is cold in the bedroom. | 0.961
It is cool in the bedroom. | 0.962
It is warm in the bedroom. | 0.009
It is hot in the bedroom. | 0.004
It is 70 degrees in the living room. | It is freezing in the living room. | <0.001
It is cold in the living room. | 0.002
It is cool in the living room. | 0.928
It is warm in the living room. | 0.902
It is hot in the living room. | 0.713

(a)  (b)  (c)

Figure 2: Entailment scores (y-axis) from RoBERTa${}_{\textit{large}}$ for
various fuzzy categories, given premise with specific temperature (x-axis):
(a) No unit; (b) Fahrenheit; and (c) Celsius. Scores are smoothed using a
generalized additive model (with default parameters, in R).

Revisiting the Celsius condition, the model is rather noisy, as compared to
its no-unit and Fahrenheit counterparts — it shows similar
$\mathcal{E}$-values for cool, cold, warm, and hot categories in this
condition, deviating slightly from intuitive perceptions about temperature.
For instance, at 50∘C, the perception of the model leans more towards cold
than hot. The model’s patterns when no unit of measurement is specified in the
premise show strong correspondence with that in the Fahrenheit condition
(RMSE222root mean-squared error between No-unit and Fahrenheit is 0.09, while
that between No-unit and Celsius is 0.16). This suggests that when no
information is available, the model defaults to inferring Fahrenheit as the
unit of measuring temperature. We hypothesize that this is likely because the
model is primarily trained on American-influenced text (e.g. Reddit).

In the context of computational treatment of fuzziness, we find the model to
show strikingly noticeable similarities to fuzzy hedges lakoff1975hedges ;
zadeh1972fuzzy . More prominently, the $\mathcal{E}$-values for hot bear
qualitative resemblance to a ‘concentration’ transformation of the
$\mathcal{E}$-values for warm. That is, if we take the membership of warm and
raise it to the power of some value greater than 1 (say, $\lambda$), we should
get a hedged version of warm ($H=$ very, extremely, etc.) that could
potentially be mapped to the hot category, pulling the curve inward, as
proposed by Zadeh zadeh1972fuzzy :

$\displaystyle\textit{hot}\approx H\textit{ warm}=\int_{U}\mu^{\lambda}(x)$
($\lambda>1$)

Empirically, we perform a naïve search between $[1,8]$ (100 values) and
minimize the RMSE between $\mathcal{E}_{\textit{hot}}$ and
$\mathcal{E}^{\lambda}_{\text{warm}}$ to find the best value for $\lambda$ as
per the above equation, and show results in fig. 3. We find the best $\lambda$
value to be close to 2 in all three cases (with Fahrenheit showing the
greatest deviation), suggesting that hot is approximately equal to very warm
(if Zadeh’s recommendation in zadeh1972fuzzy  is to be followed for very). We
also find the $\mathcal{E}$-value for warm to be greater than that of hot
approximately 77.64%, 75.33%, and 98.51% of the time for the no-unit,
Fahrenheit, and Celsius conditions, respectively. This empirically supports
the above notion that highlights the possibility of hedging warm to produce
hot, and also largely follows the ‘fuzzy’ interpretation of semantic
entailment ($P$ (hot or hedged-warm) entails $Q$ (warm) iff
$\mu_{P}(x)\leq\mu_{Q}(x)$), as noted by lakoff1975hedges , with its strongest
effect being included in the Celsius condition. Finally, we also observe a
tendency of the hot and warm curves to show a decrease in extremely hot
temperatures, suggesting the existence of another fuzzy set that is applicable
to more extreme temperatures. This is likely to be an antonym of freezing or
something that is synonymous to very hot. However, since there isn’t a clear
word with comparable frequency,333we experimented with scorching, blistering,
and balmy, but all three have considerably greater differences in frequency
with freezing, as compared to that between cold-hot and cool-warm. This can be
verified by using a public resource such as Google’s n-gram viewer. we leave
this analysis for future work.

Figure 3: Points representing the best $\lambda$ values for minimizing
$\textsc{rmse}(\mathcal{E}_{\textit{hot}},\mathcal{E}^{\lambda}_{\textit{warm}})$
using a naïve search among 100 values in $[1,8]$. Dashed line represents
$\lambda$ corresponding to very as per Zadeh zadeh1972fuzzy .

##  5 General Discussion and Conclusion

More often than not, natural language expressions have a graded—rather than
absolute—measurement of truth and falsehood lakoff1975hedges . This graded
property of language owes itself to the vague boundaries in concepts such as
tall, fruit, etc., that do not have a clear-cut criteria for membership. The
development of fuzzy set theory zadeh1965fuzzy  has provided us with a useful
toolkit to handle imprecision in language, e.g. “the coffee is very warm,” by
formulating membership functions to model data. We argue in this paper that
computational models—such as PLMs—that learn language representations must
account for the fuzziness contained in language use. To this end, we
formulated a demonstrative experiment by investigating a current state of the
art model (RoBERTa${}_{\textit{large}}$) for the classical fuzzy case of
temperature—to what extent can it map natural language premises such as “it is
40∘F inside” to hypotheses carrying fuzzy categories such as “it is cold
inside” Our tests follow the task of natural language inference
bowman2015large , which allows broad-coverage testing of a variety of semantic
phenomena encoded in models. Furthermore, the model we test was fine-tuned to
perform the NLI task, making our investigation a natural environment for the
model. We primarily focus on our model’s behavioral responses to the supplied
premise-hypotheses pairs by examining its entailment-scores (the degree to
which the premise entails the hypothesis, according to the model) for various
fuzzy categories mentioned in the hypotheses, and analyse their patterns.
Overall, we found the model to show a noisy-but-sound grasp of temperature
perception, with substantial differences between cold and warm temperature
categories towards the extreme ends of the scale. While the model showed sound
patterns of temperature perception across our unit-conditions, we found it to
show slightly lower correspondence with intuition when the temperature
mentioned in the premise was in Celsius. This, together with the fact that its
patterns on the Fahrenheit condition are found to be strongly similar to that
in the no-unit condition suggest that the model has a bias towards American
English. In our main analyses, the model showed similar patterns in its
entailment scores across the fuzzy categories as standard membership functions
formulated for showing hedging effects. For instance, the model’s entailment
scores for the category of hot mimicked the entailment-scores of a
concentrated version of warm, pulling its curve inward. However this was not
the case in the colder temperatures, suggesting a lack of consistency in this
behavior. This indicates, tentatively, that state of the art models show
noticeable signs of approximately showing patterns akin to that of fuzzy
membership functions in graded cases such as temperature. We therefore use
this paper as a call towards a systematic study of the manifestation of
vagueness in state of the art language models, as well as how multi-valued
logic such as fuzzy logic can be incorporated within them in order to better
handle imprecise language.

Disclaimer: This is a preprint of the accepted manuscript: Kanishka Misra and
Julia Taylor Rayz, Finding fuzziness in Neural Network Models of Language
Processing, to be presented at NAFIPS 2021, whose proceedings will be
published in Explainable AI and Other Applications of Fuzzy Techniques, edited
by Julia Taylor Rayz, Victor Raskin, Scott Dick, and Vladik Kreinovich,
reproduced with permission of Springer Nature Switzerland AG. The final
authenticated version is available online at: url TBD.

Reproducibility: The code to reproduce experiments reported in this paper can
be found at https://github.com/kanishkamisra/nafips2021.

## References

  * (1) Bowman, S., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference.  In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642 (2015)
  * (2) Carvalho, J.P., Batista, F., Coheur, L.: A critical survey on the use of fuzzy sets in speech and natural language processing.  In: 2012 IEEE international conference on fuzzy systems, pp. 1–8. IEEE (2012)
  * (3) Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding.  In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019)
  * (4) Firth, J.R.: A synopsis of linguistic theory, 1930-1955.  Studies in linguistic analysis (1957)
  * (5) Hastie, T.J., Tibshirani, R.J.: Generalized additive models, vol. 43.  CRC press (1990)
  * (6) Labov, W., Bailey, C., Shuy, R.: The boundaries of words and their meanings.  1973 pp. 340–73 (1973)
  * (7) Lakoff, G.: Hedges: A study in meaning criteria and the logic of fuzzy concepts.  In: Contemporary research in philosophical logic and linguistic semantics, pp. 221–271. Springer (1975)
  * (8) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A Robustly Optimized BERT Pretraining Approach.  arXiv preprint arXiv:1907.11692 (2019)
  * (9) Martine, D.: Linguistic hedges: a quantifier based approach.  Soft Computing Systems: Design, Management and Applications 87, 142 (2002)
  * (10) McCloskey, M.E., Glucksberg, S.: Natural categories: Well defined or fuzzy sets?  Memory & Cognition 6(4), 462–472 (1978)
  * (11) Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality.  In: Advances in Neural Information Processing Systems, vol. 26, pp. 3111–3119 (2013)
  * (12) Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.: Language models as knowledge bases?  In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473 (2019)
  * (13) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners.  OpenAI blog 1(8), 9 (2019)
  * (14) Rosch, E.: Cognitive representations of semantic categories.  Journal of experimental psychology: General 104(3), 192 (1975)
  * (15) Van Deemter, K.: Not exactly: In praise of vagueness.  Oxford University Press (2012)
  * (16) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need.  Advances in Neural Information Processing Systems 30, 5998–6008 (2017)
  * (17) Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Superglue: A stickier benchmark for general-purpose language understanding systems.  In: Advances in Neural Information Processing Systems, pp. 3266–3280 (2019)
  * (18) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., Bowman, S.R.: Glue: A multi-task benchmark and analysis platform for natural language understanding.  In: International Conference on Learning Representations (2018)
  * (19) Weir, N., Poliak, A., Van Durme, B.: Probing neural language models for human tacit assumptions.  In: S. Denison, M. Mack, Y. Xu, B.C. Armstrong (eds.) Proceedings of the 42nd Annual Conference of the Cognitive Science Society, pp. 377–383. Cognitive Science Society (2020)
  * (20) Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference.  In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122. Association for Computational Linguistics (2018)
  * (21) Williamson, T.: Vagueness.  Routledge (1994)
  * (22) Zadeh, L.A.: Fuzzy sets.  Information and Control 8, 338–353 (1965)
  * (23) Zadeh, L.A.: A fuzzy-set-theoretic interpretation of linguistic hedges (1972)
  * (24) Zadeh, L.A.: Fuzzy logic = computing with words.  In: Computing with Words in Information/Intelligent Systems 1, pp. 3–23. Springer (1999)