# Using fluorescent beads to emulate single flurophores

Luis A. Alemán-Castañeda,1,* Sherry Yi-Ting Feng,2 Rodrigo Gutiérrez-Cuevas,3
Isael Herrera,1 Thomas G. Brown,2 Sophie Brasselet,1 Miguel A. Alonso1,2,**

###### Abstract

We study the conditions under which fluorescent beads can be used to emulate
single fluorescent molecules in the calibration of optical microscopes.
Although beads are widely used due to their brightness and easy manipulation,
there can be notable differences between the point spread functions (PSFs)
they produce and those for single-molecule fluorophores, caused by their
different emission pattern and their size. We study theoretically these
differences for various scenarios, e.g. with or without polarization channel
splitting, to determine the conditions under which the use of beads as a model
for single molecules is valid. We also propose methods to model the blurring
due to the size difference and compensate for it to produce PSFs that are more
similar to those for single molecules.

1Aix Marseille Univ, CNRS, Centrale Marseille, Institut Fresnel, Marseille
13013, France
2The Institute of Optics, University of Rochester, Rochester, NY 14627, USA
3Institut Langevin, ESPCI Paris, Université PSL, CNRS, 75005 Paris, France

*<EMAIL_ADDRESS>**<EMAIL_ADDRESS>
##  1 Introduction

Fluorescence microscopy is a widely used technique for bio-imaging
applications, not only due to its intrinsic optical contrast and/or
specificity [1] but also for its compatibility with super-resolution
techniques[2, 3, 4]. For example, single-molecule localization microscopy
(SMLM) can achieve a spatial resolution of up to a few nanometers [5, 4, 6,
7]. Additionally, when combined with polarization measurements, fluorescence
microscopy can offer information about the sample’s structural properties, in
particular, the average orientation and the order/wobbling of the molecules,
both on the ensemble and the single-molecule regimes [8]. This information is
key not only for biological applications but also in a broader sense, since
not considering the fluorophore’s orientation can bias localization
measurements[9, 10]. The simultaneous estimation of the emitters’ 3D
localization and 3D orientational behavior is called single-molecule
orientation localization microscopy (SMOLM). Common SMOLM techniques include
polarization channel splitting [11, 12, 13] and point spread function (PSF)
engineering [14, 15, 16, 17, 18, 19].

Fluorescence from a single molecule can be modeled as dipolar radiation [20,
8], resulting in a pattern at the pupil plane (or far field) that is not
uniform in intensity nor in polarization, and it is strongly dependent on the
dipole’s 3D orientation [21]. The resulting PSF at the detector plane can be
predicted from this distribution. However, microscopes are typically imperfect
optical systems that have optical aberrations and/or polarization distortions
that bias both localization and orientation estimations. Furthermore, these
distortions may change across the field of view. Therefore, it is important to
characterize the imaging system properly, and one way to do so is by imaging
fluorescent beads, given their brightness (significantly larger than that of
single fluorophores) and easy manipulation [22, 23, 24, 25, 26, 27].

There are two main differences, however, between the PSFs for beads and those
for actual single molecule emitters. First, while beads can be quite small (a
few tens of nanometers), they are still much larger than single molecules and
this spatial extension induces a blurring effect on the measured PSF. Second,
beads are composed of many fluorophores with different orientations [22, 23],
making their emission pattern quite different from that of a single dipole:
the light emitted by a bead is in principle unpolarized and therefore fills
fairly uniformly the pupil of the system, in contrast to single-molecule
fluorophores that have specific polarization and intensity distributions.

In this paper, we study the circumstances under which fluorescent beads, with
the help of polarizers or other optical components at the pupil plane, can be
used to produce PSFs that mimic those for single fluorophores with different
orientations. We start by discussing in Section 2 the emission of a fixed
dipole and in Section 3 that of an unpolarized point-like emitter. The effect
of bead size are then studied in Section 4, both through numeric and semi-
analytic approaches. We consider two simple models for the bead emission. In
particular, the semi-analytic approach offers a series of corrections that can
be used to counter the bead size effects on the PSF in image post-processing.
Finally, the validity of the use of fluorescent beads supplemented with
polarizers to mimic single fluorophores is discussed in Section 5, and some
conclusions are drawn in Section 6. The analysis presented here follows
directly from several theoretical advances due to Prof. Emil Wolf, such as the
Richards-Wolf method for modeling high numerical aperture systems [28] and the
theory for partial spatial coherence and polarization [29], so we think this
work is appropriate for this special issue celebrating his hundredth birthday.

##  2 Radiation from a fixed dipole

In this section we describe the forward model based on the Richards-Wolf
formalism [28, 30, 16] used to calculate the PSF of a single dipole at a
distance $-z_{0}$ from the coverslip and for a system in which the location of
the focal plane with respect the coverslip is given by $z_{f}$, as shown in
Fig. 1. We assume that the dipole’s orientation is fixed and aligned with the
unit vector $\boldsymbol{\hat{\mu}}^{\rm T}=(\mu_{x},\mu_{y},\mu_{z})$. We
ignore for now transverse ($x,y$) displacements, since they correspond simply
to a linear phase factor at the pupil plane, and in turn to a transverse
displacement at the detector plane. Let us denote as
$\mathbf{E}_{\mathrm{p}}(\mathbf{u})$ the field at the system’s pupil plane,
where $\mathbf{u}$ is a dimensionless pupil position with polar coordinates
$(u,\varphi)$, normalized so that the edge of the pupil corresponds to
$u_{\mathrm{max}}=\mathrm{NA}$, the numerical aperture of the objective. For
an ideal aplanatic system, the field at the pupil plane is given by

$\mathbf{E}_{\mathrm{p}}(\mathbf{u};z_{0},z_{f})=e^{\mathrm{i}kn_{f}z_{f}\gamma(u,n_{f})}e^{-\mathrm{i}kn_{0}z_{0}\gamma(u,n_{0})}~{}\mathbb{K}(\mathbf{u})\cdot\boldsymbol{\hat{\mu}},$
(1)

where $n_{0}$ is the refractive index of the medium embedding the fluorophore,
$n_{f}$ is that of the immersion liquid of the objective (assumed here to be
the same as that of the coverslip), and $\gamma(u,n)=\sqrt{1-(u/n)^{2}}$ [28].
Here, $\mathbb{K}(\mathbf{u})$ is a $2\times 3$ matrix that determines the
amplitude/polarization distribution at the pupil plane (where the field is
paraxial and hence requires only two components) generated by each of the
three Cartesian components of the dipole source, according to

$\displaystyle\mathbb{K}(\mathbf{u})$
$\displaystyle=\begin{pmatrix}g_{0}(u)+g_{2}(u)\cos 2\varphi&g_{2}(u)\sin
2\varphi&g_{1}(u)\cos\varphi\\\ g_{2}(u)\sin 2\varphi&g_{0}(u)-g_{2}(u)\cos
2\varphi&g_{1}(u)\sin\varphi\end{pmatrix},$ (2)

where

$\displaystyle g_{0,2}(u)$
$\displaystyle=P(u;n_{f},n_{0})\left[t_{p}(u)\gamma(u,n_{0})\pm
t_{s}(u)\right],$ (3a) $\displaystyle g_{1}(u)$
$\displaystyle=2P(u;n_{f},n_{0})~{}\frac{u}{n_{0}}~{}t_{p}(u),$ (3b)
$\displaystyle P(u;n_{f},n_{0})$
$\displaystyle=\frac{1}{2}\frac{n_{f}}{n_{0}}\frac{\sqrt{\gamma(u,n_{f})}}{\gamma(u,n_{0})},$
(3c)

with $t_{p}(u)$ and $t_{s}(u)$ being the transmission coefficients for the
radial (transverse-magnetic) and azimuthal (transverse-electric) components of
the field, respectively.

Figure 1: Sketch of the optical system collecting light from the fluorophore
to the system’s pupil. The fluorophore (black dot), located at an axial
position $z_{0}$, is embedded in a medium with refractive index $n_{0}$. This
medium has a planar interface at $z=0$ with a coverslip, whose refractive
index $n_{f}$ is assumed to match that of the immersion liquid of the
objective. The curved surface represents the collimation effect of the
aplanatic objective, whose focal plane is at $z_{f}$. The radial coordinate of
a ray at the back focal plane (BFP), which coincides with the pupil plane, is
given by $f_{\mathrm{obj}}~{}u/n_{f}$.

Figure 2 shows the intensity distribution for a horizontal dipole embedded
either in water (a,b) or in air (c), and at different distances from the
coverslip. Notice that we observe supercritical angle fluorescence (SAF) if
the NA of the system is larger than the refractive index of the embedding
medium. The strength of SAF depends greatly on the distance between the dipole
and the coverslip, $|z_{0}|$, being significant only for $|z_{0}|<\lambda$.
Figure 3 shows the polarization distribution at the pupil for three orthogonal
dipole orientations in air. Polarization within the pupil is everywhere
linearly polarized except in the annular region occupied by SAF (outside the
red circle), in which $t_{p}$ and $t_{s}$ become complex and introduce a
relative phase between the radial and azimuthal field components. For a
longitudinal dipole, however, polarization remains linear even in the SAF
region since only the radial component contributes. Note that for transverse
dipoles, polarization at the pupil is largely in the dipole direction for low
NA, but less so for higher NA.

Figure 2: Intensity distribution at the pupil generated by a horizontal dipole
embedded in (a,b) water ($n_{0}=1.33$), and (c) air ($n_{0}=1$). For all
$n_{f}=1.51$. The dashed green circles denote (from smaller to larger)
$\mathrm{NA}=0.6,~{}0.9,~{}1.2$ and $1.5$, while the red circle indicates
$\mathrm{NA}=n_{0}$, which is the radial lower bound of the annular region
corresponding to SAF. In (a) and (c) SAF is important since the dipole is
close to the coverslip, $|z_{0}|=\lambda/10$, while in (b) SAF is negligible
since the dipole is further away, $|z_{0}|=\lambda$.  Figure 3: Polarization
distribution at the pupil generated by a (a) transverse horizontal, (b)
transverse vertical, and (c) longitudinal dipole embedded in air and with
$n_{f}=1.51$. Polarization ellipses are plotted on top of the respective
amplitude distributions, with cyan denoting left-handedness and magenta right-
handedness. The dashed green circles denote (from smaller to larger)
$\mathrm{NA}=0.6,~{}0.9,~{}1.2$ and $1.5$, while the red circle indicates
$\mathrm{NA}=n_{0}$, where SAF begins. Intensity distributions are normalized
with their respective maxima.

The field from the pupil is focused on the detector, which in the ideal case
corresponds to a Fourier transform of the form

$\mathbf{E}_{\mathrm{det}}(\boldsymbol{\bar{\rho}};z_{0},z_{f})=\frac{f}{M^{2}\lambda}\int\mathbb{J}(\mathbf{u})\cdot\mathbf{E}_{\mathrm{p}}(\mathbf{u};z_{0},z_{f})\exp\left(-\mathrm{i}k\mathbf{u}\cdot\frac{\boldsymbol{\bar{\rho}}}{M}\right)d^{2}\mathbf{u},$
(4)

where $M$ is the optical system’s magnification, $f$ is the focal length of
the focusing system, and $\boldsymbol{\bar{\rho}}$ is the transverse position
variable at the detector. We employ barred variables to distinguish the
detector coordinates from the object space of the fluorophores. Here,
$\mathbb{J}(\mathbf{u})$ represents a Jones matrix describing any manipulation
on the field performed after the pupil plane, ranging from a simple uniform
analyzer before the detector to a more elaborate mask with varying amplitude,
phase or birefringence, like those used in many PSF engineering techniques
[25, 26, 27, 16, 18, 19]. In what follows we use the shorthand $\mathcal{F}$
for this Fourier transform including the corresponding physical parameters.
The final intensity at the detector is given by

$I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};z_{0},z_{f})=\left[\mathcal{F}\left\\{\mathbb{J}(\mathbf{u})\cdot\mathbf{E}_{\mathrm{p}}(\mathbf{u};z_{0},z_{f})\right\\}\right]^{\dagger}\left[\mathcal{F}\left\\{\mathbb{J}(\mathbf{u})\cdot\mathbf{E}_{\mathrm{p}}(\mathbf{u};z_{0},z_{f})\right\\}\right],$
(5)

where † denotes the conjugate transpose operation.

We now study the variation of the PSF with the axial position $z_{0}$ of the
dipole. For this purpose we expand in a Taylor series in $z_{0}$ the intensity
at the detector plane. This expansion will let us gain physical insight and
compare different sources such as fixed dipoles, unpolarized point-like
emitters, and fluorescent beads. It will also be the basis for a semi-analytic
model for accounting for bead size discussed in Section 4. Let this Taylor
expansion be centered around some value $z_{0}=d$:

$I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};z_{0},z_{f})=\sum_{m=0}^{\infty}\frac{1}{m!}(z_{0}-d)^{m}\frac{\partial^{m}I_{\mathrm{det}}}{\partial
z_{0}^{m}}(\boldsymbol{\bar{\rho}};d,z_{f})=\sum_{m=0}^{\infty}\frac{1}{m!}(z_{0}-d)^{m}\mathcal{I}_{m}(\boldsymbol{\bar{\rho}};d,z_{f}),$
(6)

where in the last step we introduced the notation
$\mathcal{I}_{m}(\boldsymbol{\bar{\rho}};d,z_{m})$ for the $m$th derivative in
$z_{0}$ of the intensity. While written here as a sum over all non-negative
integers, in practice the sum in $m$ will be truncated at some modest value.
Since the only dependence in $z_{0}$ of the field $\mathbf{E}_{\mathrm{det}}$
is within the complex phase factor, we have

$\frac{\partial^{m}\mathbf{E}_{\mathrm{det}}}{\partial
z_{0}^{m}}(\boldsymbol{\bar{\rho}};d,z_{f})=\mathcal{F}\left\\{\mathbb{J}(\mathbf{u})\cdot\mathbf{E}_{\mathrm{p}}(\mathbf{u};d,z_{f})[-\mathrm{i}kn_{0}\gamma(u,n_{0})]^{m}\right\\},$
(7)

so that the coefficients of the series become

$\displaystyle\mathcal{I}_{m}(\boldsymbol{\bar{\rho}};d,z_{f})=\sum_{l=0}^{m}\frac{m!}{l!~{}(m-l)!}\frac{\partial^{l}\mathbf{E}_{\mathrm{det}}^{\dagger}}{\partial
z_{0}^{l}}(\boldsymbol{\bar{\rho}};d,z_{f})\frac{\partial^{m-l}\mathbf{E}_{\mathrm{det}}}{\partial
z_{0}^{m-l}}(\boldsymbol{\bar{\rho}};d,z_{f}).$ (8)

Note that for $m>0$, $\mathcal{I}_{m}$ are not intensities but rather changes
in the intensity with $z_{0}$, so they can be positive or negative, although
their individual integrals must vanish due to energy conservation. To
illustrate the usefulness of this expansion, let us consider the simple case
of index-matching (a common strategy in microscopy to avoid optical
aberrations introduced by the interface), i.e. $n_{0}=n_{f}$, for which there
is no SAF. The first four columns of Fig. 4 show the distributions for
$\mathcal{I}_{m}$ at the nominal focal distance ($z_{0}=z_{f}$) for different
constant choices for the Jones matrix $\mathbb{J}$, namely the identity (no
manipulation before focusing), and pairs of orthogonal polarization
projections before the detector: horizontal/vertical and circular left/right.
The fifth column shows the PSFs at the defocused plane
$z_{f}-z_{0}=\lambda/3$, calculated as a weighted superposition of the first
four columns following Eq. (6). Notice that the PSF is slightly elongated in
the dipole direction (horizontally). Given the absence of SAF contributions
and aberrations, the focal spot is symmetric in $z$ around the nominal image
plane, so that the odd $m$ contributions vanish in Fig. 4(a-c). Nevertheless,
as shown in Fig. 4(d,e) when a circular analyzer is used, ${\cal I}_{1}$ does
not vanish, hence breaking the symmetry, and causing the circularly-polarized
PSFs to counter-rotate with respect to each other with defocus, as shown in
the last column of Fig. 4(d,e). This could serve as the basis of a SMOLM
technique for the estimation of transverse dipoles’ localization and
orientation, consisting of forming separately the PSF of each circular
component: for perfectly in-focus dipoles the elongation reports on the dipole
direction, while for defocused dipoles the amount of counter-rotation between
the PSFs reports on the defocus. This behavior is reminiscent of that of the
PSFs in a technique proposed recently [16]. However, the signature of the
elongation and rotation of the PSFs in Fig. 4(d,e) may not be sufficiently
strong to remain detectable in the presence of noise and camera pixelation.

Figure 4: First four columns are the PSFs ${\cal I}_{0}$ and the subsequent
Taylor components ${\cal I}_{m}$ for a system with NA$=1.3$ (no SAF) imaging a
horizontal dipole in water in an index-matching scenario ($n_{0}=n_{f}\approx
1.33$), for several polarization projections following the pupil: (a) no
polarizer, (b) horizontal, (c) vertical, (d) left-circular and (e) right-
circular polarizers. The last column shows the corresponding simulated
defocused PSFs at $z_{f}-z_{0}=\lambda/3$. The elements of each column are
normalized to the largest value in the column, but the results in (c) are
scaled by a factor of 128 and those in (d) and (e) are scaled by a factor of
2. Here $\lambda=525$nm and squares are of the length of $1\mu$m on the object
side.

This index-matching example shows some of the insights gained by this Taylor
expansion approach. In particular, it illustrates that in order to break the
insensitivity to the sign of defocus of the measured PSFs, it is not
imperative to have optical aberrations or use complicated Fourier
manipulations (a common belief), but one can simply use the correct
polarization projections. Note that the distributions for $\mathcal{I}_{m}$
when the refractive indices are not matched ($n_{0}\neq n_{f}$) can also be
computed, even when SAF contributions are present. These contributions would
disrupt the symmetry around the focal plane so that $\mathcal{I}_{1}$ would
not vanish. In fact, the choice of a focal plane would no longer be unique,
and some criterion (e.g. paraxial focus, minimmun spot size, minimum rms
wavefront error, etc.) would have to be adopted for its choice. Nonetheless,
the insight gained in the index-matching case remains valid, and the PSFs
resulting from circular polarization projection would still counter-rotating
with defocus.

##  3 Radiation from an unpolarized emitter.

An unpolarized (in the 3D sense) point-like emitter can be understood as the
incoherent sum of the contributions from three orthogonal dipoles of equal
magnitude. Without loss of generality, we can use three dipoles aligned with
the coordinate axes $x$ (transverse horizontal), $y$ (transverse vertical),
and $z$ (longitudinal). Therefore, the intensity distribution at the pupil can
be written as

$I_{\mathrm{pupil,unp}}=|\mathbb{K}\cdot\mathbf{\hat{x}}|^{2}+|\mathbb{K}\cdot\mathbf{\hat{y}}|^{2}+|\mathbb{K}\cdot\mathbf{\hat{z}}|^{2}=\mathrm{Tr}\left\\{\mathbb{K}^{\dagger}\mathbb{K}\right\\},$
(9)

where $\mathbf{\hat{x}}$, $\mathbf{\hat{y}}$ and $\mathbf{\hat{z}}$ are 3D
unit vectors in these three directions, and $\mathrm{Tr}\\{\\}$ denotes the
trace. It can be shown that this intensity distribution reduces to the
radially symmetric function

$I_{\mathrm{pupil,unp}}(u)=|P(u;n_{f},n_{0})|^{2}F(u;z_{0})\left\\{\mathcal{S}|t_{p}(u)|^{2}+|t_{s}(u)|^{2}\right\\},$
(10)

where
$F(u,z_{0})=\mathrm{exp}[-2~{}\mathrm{Im}\,\\{kn_{0}|z_{0}|\gamma(u,n_{0})\\}]$
and $\mathcal{S}=|\gamma(u,n_{0})|^{2}+\left(u/n_{0}\right)^{2}$, with $z_{0}$
the distance to the coverslip and $\mathrm{Im}\,\\{\\}$ denoting the imaginary
part. Note that outside of the SAF region, i.e. for $u\leq n_{0}$,
$P(u;n_{f},n_{0})=F(u,z_{0})=1$. Figure 5 shows this intensity distribution
and the one resulting from passing through a horizontal polarizer, as well as
the one for a horizontal dipole. Note that the latter two are similar (i.e.
fairly constant) for low NA but their differences increase considerably for
higher NA. In particular, the distribution in Fig. 5(b) is maximal at the two
intersections of the horizontal with the circle of radius $n_{0}$, and these
points corresponds to the minima in Fig. 5(c).

Figure 5: Intensity distributions at the pupil for (a) an unpolarized emitter,
(b) an unpolarized emitter filtered by a horizontal polarizer at the pupil,
and (c) a horizontal dipole, all embedded in air, with a distance to the
coverslip of $d=\lambda/10$, and for a system with NA$=1.5$. The dashed green
circles denote (from smaller to larger) $\mathrm{NA}=0.6,~{}0.9,~{}1.2$ and
$1.5$, while the red circle indicates ${\rm NA}=n_{0}$ where SAF begins.

Let us now look at the polarization distribution at the pupil plane for an
unpolarized point emitter. The distribution of the normalized Stokes
parameters, i.e. $s_{i}(\mathbf{u})=S_{i}(\mathbf{u})/S_{0}(\mathbf{u})$ for
$i=1,2,3$ where $S_{0}(\mathbf{u})=I_{\mathrm{pupil,unp}}(\mathbf{u})$, are
found to be given by

$\displaystyle s_{1}(\mathbf{u})$
$\displaystyle=\frac{\mathcal{S}|t_{p}(u)|^{2}-|t_{s}(u)|^{2}}{\mathcal{S}|t_{p}(u)|^{2}+|t_{s}(u)|^{2}}\cos
2\varphi,$ (11a) $\displaystyle s_{2}(\mathbf{u})$
$\displaystyle=\frac{\mathcal{S}|t_{p}(u)|^{2}-|t_{s}(u)|^{2}}{\mathcal{S}|t_{p}(u)|^{2}+|t_{s}(u)|^{2}}\sin
2\varphi,$ (11b) $\displaystyle s_{3}(\mathbf{u})$ $\displaystyle=0,$ (11c)

which (unlike $I_{\mathrm{pupil,unp}}$) depend on $\varphi$ although as a
simple sinusoidal modulation. The degree of polarization is rotationally
symmetric:

$\mathrm{DOP}(u)=\sqrt{\sum_{i=1}^{3}s_{i}^{2}(\mathbf{u})}=\frac{\left|\mathcal{S}|t_{p}(u)|^{2}-|t_{s}(u)|^{2}\right|}{\mathcal{S}|t_{p}(u)|^{2}+|t_{s}(u)|^{2}}.$
(12)

That is, the field at the pupil plane is fully unpolarized ($\mathrm{DOP}=0$)
only at the center, and as one approaches the border of the pupil the
difference between the Fresnel transmission coefficients introduces more
polarization. This degree of polarization can be particularly strong for the
SAF contribution. This is illustrated in Fig. 6, which shows the pupil
distributions of the normalized Stokes parameters as well as the degree of
polarization of an unpolarized emitter close to the coverslip.

Figure 6: Normalized Stokes parameters at the pupil (a) $s_{1}$ and (b)
$s_{2}$, and (c) $\mathrm{DOP_{\mathrm{p}}}$, over the pupil for an
unpolarized emitter embedded in air, with a distance to the coverslip of
$d=\lambda/10$, and for a system with NA$=1.5$. The dashed green circles
denote (from smaller to larger) $\mathrm{NA}=0.6,~{}0.9,~{}1.2$ and $1.5$,
while the red circle indicates ${\rm NA}=n_{0}$ where SAF begins.

Like the pupil distributions, the PSFs can be constructed as the incoherent
sum of the intensity contributions at the detector of the three orthogonal
dipoles (horizontal, vertical, and longitudinal). Note that, given the
linearity of the intensity, the Taylor expansion for the total intensity is
simply the sum of the corresponding contributions
$\mathcal{I}_{m}(\boldsymbol{\bar{\rho}};z_{0})$ for each of these three
orthogonal dipoles. Figure 7 shows the distribution of $\mathcal{I}_{m}$ at
the nominal focal distance, $z_{0}=z_{f}$, in an index-matching scenario,
$n_{0}=n_{f}$, for different constant choices for $\mathbb{J}$, namely the
identity (no manipulation before focusing), and pairs of orthogonal
polarization projections before the detector: horizontal/vertical and
left/right circular. For horizontal and vertical polarization projections, the
PSFs become elongated, whereas they retain rotational symmetry when no
polarization projection takes place or for circular polarization projections.
Note that, given the symmetry of the focused field around the nominal image
plane in the absence of aberrations, the odd derivative terms vanish in Fig.
7.

Figure 7: First four columns are the PSFs ${\cal I}_{0}$ and the subsequent
Taylor components ${\cal I}_{m}$ for a system with NA$=1.3$ imaging an
unpolarized emitter in water in a index-matching scenario ($n_{f}=n_{0}\approx
1.33$), for several polarization projections following the pupil: (a) no
polarizer, (b) horizontal, (c) vertical, (d) left-circular and (e) right-
circular polarizers. The last column shows the corresponding simulated
defocused PSFs at $z_{f}-z_{0}=\lambda/3$. The elements of each column are
normalized to the largest value in the column, but a factor of 2 is applied in
rows (b-e) with respect to (a). Here $\lambda=525$nm and squares are of length
of $1\mu$m on the object side.

##  4 Modeling the effects of bead size

As mentioned in the introduction, we can envision several simple models for
the emission and geometry of a fluorescent bead, which include: (1) a
transparent solid sphere, in which all points inside the bead emit light, and
this light is able to emerge from the bead; (2) a transparent shell, in which
only the points at the surface of the sphere emit, but the light from all
these points is visible in any direction; (3) an opaque shell, where only the
light from points at the surface in line of sight can be detected, perhaps
following a Lambertian emission model. Given their transparency, fluorescent
beads most likely are best described by the first model, particularly when
their index is well matched to that of its surrounding medium. However, we
also describe the second model in what follows, given how similar its
mathematical description is to that of the first. The third model will not be
discussed further, since it is harder to model and would be relevant, for
example, if the refractive index of the bead had an important imaginary part.

To model an extended incoherent source we could follow several approaches. One
would be to superpose incoherently the contributions from randomly oriented
dipoles whose positions within the desired volume are also chosen randomly.
This stochastic approach can be easily implemented for simulating fluorophore
distributions of arbitrary shapes. However, it is computationally slow, and it
does not offer much physical insight. We therefore opt instead for a
deterministic approach that assumes the same emission at all points, hence
allowing the use of numerical or semi-analytic approximations.

###  4.1 Transparent solid sphere

Let us assume that the intensity distribution at the detector due to a source
at $(x,y,z)$ is $I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x,y,z)$ (where again
coordinates without a bar describe the space of the bead, and barred
coordinates correspond to the detector plane). Note that we omit the
dependence on $z_{f}$ for brevity. Consider a transparent fluorescent bead of
radius $R$ centered at $(x_{0},y_{0},z_{0})$. The measured intensity can be
calculated as a superposition of 2D convolutions:

$I_{\mathrm{tot}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0},R)=\sigma_{V}\int^{R}_{-R}I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0}+z)\ast
D(\boldsymbol{\bar{\rho}};R,z)\,dz,$ (13)

where $\sigma_{V}$ is the volumetric density of fluorophores, $\ast$ denotes a
2D convolution in the detector coordinates, and
$D(\boldsymbol{\bar{\rho}};R,z)$ is a disk function representing the magnified
section of the bead at $z$, where the transverse magnification is given by
$M$, so

$D(\boldsymbol{\bar{\rho}};R,z)=\mathrm{circ}\left(\frac{1}{R}\sqrt{\left|\frac{\boldsymbol{\bar{\rho}}}{M}\right|^{2}+z^{2}}~{}\right)=\mathrm{circ}\left(\frac{|\boldsymbol{\bar{\rho}}/M|}{\sqrt{R^{2}-z^{2}}}\right).$
(14)

For numerical purposes, the integral in $z$ can be approximated through a
discrete sum of $N$ slices, i.e.

$I_{\mathrm{tot}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0},R)\approx\sigma_{V}\sum_{j=1}^{N}I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0}+z_{j})\ast
D(\boldsymbol{\bar{\rho}};R,z_{j})\,\Delta z,$ (15)

where

$\displaystyle\Delta z$ $\displaystyle=\frac{2R}{N},$ $\displaystyle z_{j}$
$\displaystyle=\left(j-\frac{1}{2}\right)\Delta z-R.$ (16)

This approximate model can then be implemented as

$I_{\mathrm{tot}}=\sigma_{V}~{}\mathcal{F}^{-1}_{\mathbf{u}\rightarrow\boldsymbol{\bar{\rho}}}\left\\{\sum_{j=1}^{N}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0}+z_{j})\right\\}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{D(\boldsymbol{\bar{\rho}};R,z_{j})\right\\}\right\\}\Delta
z,$ (17)

where

$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{D(\boldsymbol{\bar{\rho}};R,z_{j})\right\\}(u)=2\pi
M^{2}R_{j}^{2}~{}\frac{J_{1}(kuR_{j})}{kuR_{j}},$ (18)

with $R_{j}=\sqrt{R^{2}-z_{j}^{2}}$, i.e. the section of the bead at $z_{j}$.
Note that the way the sum is implemented, using $N=1$ (i.e. one slice through
the center of the bead) corresponds to a convolution of the PSF with a disk of
the same nominal radius as the bead, giving a very simple model for the
blurring effect.

Alternatively, a semi-analytic approach can be used based on a Taylor
expansion in $z$ around $z_{0}$ of $I_{\mathrm{det}}$ like that in Eq. (6),
namely,

$I_{\mathrm{tot}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0},R)=\sigma_{V}\sum_{m=0}^{\infty}\frac{1}{m!}\mathcal{I}_{\mathrm{det},m}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0})\ast
G_{m}(\boldsymbol{\bar{\rho}};R),$ (19)

where, for sufficiently small beads, a good approximation can be obtained by
truncating the sum at a low value of $m$. The function
$G_{m}(\boldsymbol{\bar{\rho}};R)$ is defined as the easily-solvable integral

$G_{m}(\boldsymbol{\bar{\rho}};R)=\int^{R}_{-R}z^{m}D(\boldsymbol{\bar{\rho}};R,z)dz=\int^{\ell/2}_{-\ell/2}z^{m}dz=\left\\{\begin{array}[]{cc}\frac{1}{2^{m}(m+1)}\ell^{m+1},&m\,{\rm
even,}\\\ 0,&m\,{\rm odd}.\end{array}\right.$ (20)

where $\ell(\boldsymbol{\bar{\rho}};R)$ is the length in $z$ of the bead at a
given transverse position. In terms of the detector coordinates, this length
is given by

$\ell({\bar{\rho}};R)=2\sqrt{R^{2}-\left|\frac{{\bar{\rho}}}{M}\right|^{2}}.$
(21)

Note that $G_{m}=0$ for odd $m$ due to the symmetry of the bead, so the sum in
Eq. (19) includes only contributions for even $m$, even when $n_{0}\neq
n_{f}$. We therefore substitute $m=2\mu$ for $\mu=0,1,...$ in what follows. By
using the convolution property of Fourier transforms, we can rewrite Eq. (19)
as

$I_{\mathrm{tot}}=\sigma_{V}~{}\mathcal{F}^{-1}_{\mathbf{u}\rightarrow\boldsymbol{\bar{\rho}}}\left\\{\sum_{\mu=0}^{\infty}\frac{1}{(2\mu)!}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{\mathcal{I}_{\mathrm{det},2\mu}\right\\}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{2\mu}\\}\right\\}$
(22)

where we dropped the arguments for brevity. Since $G_{2\mu}$ are radially
symmetric functions, the Hankel transform can be used to obtain their Fourier
transforms:

$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{2\mu}\\}(u)=4\pi~{}M^{2\mu+2}R^{2\mu+3}(2\mu-1)!!~{}\frac{j_{\mu+1}(kuR)}{(kuR)^{\mu+1}},$
(23)

where $n!!=(n)(n-2)\dots$ is the double factorial, and $j_{n}$ denotes a
spherical Bessel function of order $n$.

Figure 8(a) shows the PSF for the particular case of a bead with diameter
$2R=200$nm embedded in water and index matched, imaged by a system with
NA$=1.3$. The PSF was obtained using the discrete sum of convolutions in Eq.
(15) with $N=21$. On the other hand, Fig. 8(b-c) shows the difference between
our ground truth, the PSF in Fig. 8(a) and the PSFs obtained using (b) the
same method but for $N=1$, namely a simple convolution with a uniform disk,
and (c,d) the semi-analytical approach in Eq. (22) with the sum truncated at
(c) $\mu=0$, namely a simple convolution with the apodized disk $G_{0}$, and
(d) $\mu=1$, that is, including an extra correction. We take as ground truth
the result shown in Fig. 8(a). We see that, for large NAs the two models
including only one contribution, namely the convolution with a uniform disk
$D(\boldsymbol{\bar{\rho}})$ (b) and with an apodized disk
$G_{0}(\boldsymbol{\bar{\rho}})$ (c), give results with similar levels of
error, whereas the semi-analytic result including a correction term gives an
error 20 times smaller. Note that the errors in Fig. 8(b) and (c) have
opposite behavior, due to the fact that $G_{0}(\boldsymbol{\bar{\rho}})$
weights more heavily the center compared to the borders, while
$D(\boldsymbol{\bar{\rho}})$ is uniform.

Figure 8: (a) Simulated PSFs for a 200nm diameter fluorescent bead using
numerical integration with 21 slices. Difference between the numerical
integration result and (b) one slice, i.e. uniform disk model, (c) semi-
analytic model with $\mu=0$, and (d) semi-analytic model with $\mu=1$. Note
that (b) and (c) are normalized to $1\%$ of the peak value of the PSF, whereas
(d) is normalized to $0.05\%$ of the maximum value recorded for (a). Here
$\mathrm{NA}=1.3$, and the index-matching condition ($n_{f}=n_{0}\approx
1.33$) and squares are of the length of $1\mu$m on the object side.

In order to compare the models more thoroughly, the RMS error for a range of
NAs and bead radii was computed using again as ground truth the results of Eq.
(15) with $N=15$. Figure 9 shows the logarithm of the RMS error. Notice that
for small NA the convolution with the apodized disk,
$G_{0}(\boldsymbol{\bar{\rho}})$, is much better than the convolution with a
uniform disk, $D(\boldsymbol{\bar{\rho}})$, but for larger NA they have
similar performance, and the uniform disk even outperforms the apodized one.
An interesting difference between these two simple models is that convolving
with $G_{0}(\boldsymbol{\bar{\rho}})$ leads to an estimate whose error grows
with both NA and $R$, while convolving with $D(\boldsymbol{\bar{\rho}})$ leads
to an error that is roughly independent of NA. The semi-analytical model
becomes again roughly independent of NA for small beads when the leading
correction is included.

Figure 9: RMS error, for varying NA and bead radius, between the numerically
integrated PSF and the PSFs obtained by using: (a) Eq. (15) with
$N=1$(convolution with a flat disk), (b) Eq. (22) truncated at $\mu=0$
(convolution with an apodized disk), and (c) Eq. (22) truncated at $\mu=1$.
Here $\mathrm{NA}=1.3$ and the embedding medium is water ($n_{0}\approx
1.33$).

A key result, particularly for small NA systems, is that the factor for the
zeroth-order, namely

$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{0}\\}(u)=4\pi~{}M^{2}R^{3}~{}\frac{j_{1}(kuR)}{(kuR)},$
(24)

can be used to directly deconvolve the measured intensity at the detector to
obtain a more realistic PSF, i.e.

$I^{\prime}=\mathcal{F}^{-1}_{\mathbf{u}\rightarrow\boldsymbol{\bar{\rho}}}\left\\{\frac{\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{I_{\mathrm{measured}}\\}}{\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{0}\\}}\right\\}.$
(25)

This deconvolution is reliable under appropriate signal-to-noise ratio
conditions, and requires that the zeros of
$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{0}\\}$ lie
well outside the incoherent pupil radius of 2NA. The first zero of
$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{0}\\}$ is at
$u^{\prime}\approx 1.43\pi/(kR)$, giving the safety condition for the bead’s
diameter $2R<0.715\lambda/\mathrm{NA}$. In a similar manner, the uniform disk
model obtained form considering a single slice in Eq. (17) provides an
alternative to perform the deconvolution. Here, the location of the first zero
of $\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{D\\}$
restricts the parameters to satisfy the slightly more restrictive condition
$2R<0.61\lambda/\mathrm{NA}$.

###  4.2 Transparent shell

Although we do not expect this model to be a faithful representation of a
fluorescent bead, we include it given its simplicity, and in case it can be
useful in a different application. Following the same procedure as in the
previous section, let us now assume a fluorescent empty shell of radius $R$
centered at $(x_{0},y_{0},z_{0})$. Again, the measured intensity can be
calculated as a superposition of 2D convolutions:

$I_{\mathrm{tot}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0},R)=\sigma_{S}\int^{R}_{-R}I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0}+z)\ast
A(\boldsymbol{\bar{\rho}};R,z)\,dz,$ (26)

where $\sigma_{S}$ is the surface density of fluorophores,
$A(\boldsymbol{\bar{\rho}};R,z)$ is an impulse ring function representing the
magnified perimeter of the sphere at $z$, i.e.

$A(\boldsymbol{\bar{\rho}};R,z)=\delta\left(\sqrt{\frac{\bar{\rho}^{2}}{M^{2}}+z^{2}}-R\right)=\frac{\delta(z-\ell/2)+\delta(z+\ell/2)}{|z|/R},$
(27)

where $\ell$ is defined in Eq. (21). Numerically, this integral in $z$ can be
computed through a discrete sum of $N$ slices as

$I_{\mathrm{tot}}=\sigma_{S}~{}\mathcal{F}^{-1}_{\mathbf{u}\rightarrow\boldsymbol{\bar{\rho}}}\left\\{\sum_{j=1}^{N}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{I_{\mathrm{det}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0}+z_{j})\right\\}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{A(\boldsymbol{\bar{\rho}};R,z_{j})\right\\}\right\\}\Delta
z,$ (28)

where $\Delta z$ and $z_{j}$ are defined as in Eq. (16) and

$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{A(\boldsymbol{\bar{\rho}};R,z_{j})\right\\}(u)=\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{\frac{MR}{\bar{\rho}}\delta\left(\frac{\bar{\rho}}{M}-R_{j}\right)\right\\}(u)=2\pi
M^{2}R~{}J_{0}(kuR_{j}),$ (29)

in which once more $R_{j}=\sqrt{R^{2}-z_{j}^{2}}$, i.e. it is the radius of
the bead at $z_{j}$.

On the other hand, as in the previous section, a semi-analytic approach can be
used based on a Taylor expansion in $z$ around $z_{0}$ of $I_{\mathrm{det}}$,
namely,

$I_{\mathrm{tot}}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0},R)=\sigma_{S}\sum_{m=0}^{\infty}\frac{1}{m!}\mathcal{I}_{\mathrm{det},m}(\boldsymbol{\bar{\rho}};x_{0},y_{0},z_{0})\ast
H_{m}(\boldsymbol{\bar{\rho}};R),$ (30)

where the function $H_{m}(\boldsymbol{\bar{\rho}};R)$ is defined as the
easily-solvable integral

$H_{m}(\boldsymbol{\bar{\rho}};R)=\int^{R}_{-R}z^{m}A(\boldsymbol{\bar{\rho}};R,z)dz=\left\\{\begin{array}[]{cc}R~{}\frac{\ell^{m-1}}{2^{m-2}},&m\,{\rm
even,}\\\ 0,&m\,{\rm odd}.\end{array}\right.$ (31)

Note that in general $H_{m}=(m-1)RG_{m-2}$, where $G_{m}$ is given in Eq.
(20), and so the total intensity can be expressed (by again substituting
$m=2\mu$) as

$I_{\mathrm{tot}}=R\sigma_{S}~{}\mathcal{F}^{-1}_{\mathbf{u}\rightarrow\boldsymbol{\bar{\rho}}}\left\\{\sum_{\mu=0}^{\infty}\frac{(2\mu-1)}{(2\mu)!}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\left\\{\mathcal{I}_{\mathrm{det},2\mu}\right\\}\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{2(\mu-1)}\\}\right\\},$
(32)

in which
$\mathcal{F}_{\boldsymbol{\bar{\rho}}\rightarrow\mathbf{u}}\\{G_{2\mu}\\}$ is
given in Eq. (23).

##  5 Mimicking transverse dipoles with beads

We now evaluate how well a flourescent bead supplemented with a polarizer at
the pupil can be used to mimic the PSF of a transverse point dipole. For of a
transverse dipole, the intensity distribution at the pupil is fairly uniform
except near the pupil edge and is similar to an unpolarized emitter.
Therefore, by inserting a linear polarizer at the pupil, one can make the PSF
of an unpolarized emitter mimic the one for a transverse dipole [24, 16]. For
systems with small NA, this should be a good approximation, while for higher
NAs differences become more evident at the edge of the pupil distribution,
both in intensity and polarization, causing discrepancies between the
corresponding PSFs. In this section we first study how a point-like emitter
can mimic a real transversal dipole, and then we incorporate the effects of
bead size using the results of the previous section. Particularly we highlight
how the function $G_{0}$ can be used to deconvolve the measured PSFs to
correct the blurring introduced by the bead size.

###  5.1 Mimicking transverse dipoles using a point-like source

Let us, without loss of generality, consider a horizontal dipole (aligned with
the $x$ axis). The Jones matrix at the pupil is then written as

$\mathbb{J}(\mathbf{u})=\mathbb{J}^{\prime}(\mathbf{u})\mathbb{P}_{x},$ (33)

where
$\mathbb{P}_{x}=\boldsymbol{\hat{\rho}_{x}}\boldsymbol{\hat{\rho}_{x}}^{\dagger}$
is a horizontal projection matrix representing a horizontal polarizer, with
$\boldsymbol{\hat{\rho}_{x}}$ being a 2D unit vector in the horizontal
direction. That is, the Jones matrix representing all the manipulations after
the pupil can be chosen to represent the cascade of a horizontal linear
polarizer and other possible manipulations represented by the Jones matrix
$\mathbb{J}^{\prime}(\mathbf{u})$. Let us for now set this second Jones matrix
to the identity. It can be shown that the intensity distribution at the pupil
just after the polarizer becomes

$\displaystyle I_{\mathrm{p,mimic}}(\mathbf{u})=$
$\displaystyle\left(|g_{0}(u)|^{2}+|g_{2}(u)|^{2}+2~{}\mathrm{Re}\,\\{g_{0}^{*}(u)g_{2}(u)\\}\cos
2\varphi\right.$ (34)
$\displaystyle\left.+|g_{1}(u)|^{2}\cos^{2}\varphi\right)F(u;z_{0}),$

whose only difference with that for a true horizontal dipole is the last term,
$|g_{1}(u)|^{2}\cos^{2}\varphi$, meaning that the difference is zero only at
the central axial line perpendicular to the polarization.

Perhaps of more consequence to the final PSFs are the differences in spatial
coherence and polarization over the pupil. For a pure horizontal dipole
source, the cross-spectral density matrix is

$\mathbb{W}_{x}({\bf u}_{1},{\bf u}_{2})=\mathbb{K}({\bf
u}_{1})~{}\mathbf{\hat{x}}\mathbf{\hat{x}}^{\dagger}~{}\mathbb{K}^{\dagger}({\bf
u}_{2}),$ (35)

where $\mathbf{\hat{x}}\mathbf{\hat{x}}^{\dagger}$ is a 3D projection in the
$x$ direction (as opposed to $\mathbb{P}_{x}$ which enacts a 2D projection).
Note that $\mathbb{W}_{x}$ is simply the outer product of
$\mathbb{K}~{}\mathbf{\hat{x}}$ and its conjugate (each evaluated at a
different pupil point), so there is no statistical decorrelation between any
two pupil points. This is not the case, on the other hand, for the mimicked
dipole based on an unpolarized emitter followed by a polarizer at the pupil,
for which all three orthogonal dipole directions contribute to the $x$
component at the pupil, each with a different distribution (two of them being
important mainly at the edges of the pupil). These contributions are mutually
incoherent. Therefore, the cross-spectral density matrix at the pupil right
after the polarizer due to an unpolarized emitter at the nominal source plane
is given by

$\mathbb{W}_{\mathrm{p,mimic}}({\bf u}_{1},{\bf
u}_{2})=\mathbb{P}_{x}\mathbb{K}({\bf u}_{1})\mathbb{K}^{\dagger}({\bf
u}_{2})\mathbb{P}_{x}$ (36)

whose only nonzero element is the $xx$ one. The degree of coherence for this
nonzero component is then

$\mu_{\mathrm{p,mimic}}({\bf u}_{1},{\bf
u}_{2})=\frac{\\{\mathbb{W}_{\mathrm{p,mimic}}({\bf u}_{1},{\bf
u}_{2})\\}_{xx}}{\sqrt{\\{\mathbb{W}_{\mathrm{p,mimic}}({\bf u}_{1},{\bf
u}_{1})\\}_{xx}\\{\mathbb{W}_{\mathrm{p,mimic}}({\bf u}_{2},{\bf
u}_{2})\\}_{xx}}}.$ (37)

Note that for an emitter at the origin and in the absence of aberrations or
SAF contributions, this degree of polarization is real. Because
$\\{\mathbb{W}_{\mathrm{p,mimic}}\\}_{xx}$ is not a separable function of
$\mathbf{u}_{1}$ and $\mathbf{u}_{2}$, $\mu_{\mathrm{p,mimic}}$ typically
differs from unity for $\mathbf{u}_{1}\neq\mathbf{u}_{2}$. This is illustrated
in Fig. 10 for different slices of the pupil: two in which
$\mathbf{u}_{1}\parallel\mathbf{u}_{2}$, and one in which
$\mathbf{u}_{1}\perp\mathbf{u}_{2}$. Note in particular that for sufficiently
separated points along the horizontal line, the degree of coherence vanishes
and then becomes negative, due to the predominance at those points of the
radially polarized contribution introduced by the longitudinal dipole. On the
other hand, for any two pupil points along a vertical line
($\varphi_{1}=\varphi_{2}=\pi/2$), the polarizer eliminates the contributions
from the vertical and longitudinal components, so that
$\mu_{\mathrm{p,mimic}}=1$. In general, if both points are near the center of
the pupil, the correlation approaches unity.

Figure 10: Degree of coherence $\mu_{\mathrm{p,mimic}}$ for (a)
$\varphi_{1}=\varphi_{2}=0$, (b) $\varphi_{1}=\varphi_{2}=\pi/4$ and (c)
$\varphi_{1}=0$ and $\varphi_{2}=\pi/2$. Here the unpolarized emitter is
assumed to be in the water ($n_{0}\approx 1.33$), $n_{f}=1.51$, and we only
show $u_{1,2}$ within the non-SAF regions, since for SAF the degree of
coherence becomes complex.

A quantitative measure of the similarity of both pupil distributions that
accounts for amplitude, coherence and polarization is given by the following
normalized projection:

$\displaystyle C({\rm NA})$ $\displaystyle=\left\\{\frac{\iint{\rm
Tr}[\mathbb{W}_{x}({\bf u}_{1},{\bf u}_{2})\mathbb{W}_{\mathrm{p,mimic}}({\bf
u}_{2},{\bf u}_{1})]\,d^{2}{\bf u}_{1}\,d^{2}{\bf u}_{2}}{\int{\rm
Tr}[\mathbb{W}_{x}({\bf u},{\bf u})]\,d^{2}{\bf u}\,\int{\rm
Tr}[\mathbb{W}_{\mathrm{p,mimic}}({\bf u},{\bf u})]\,d^{2}{\bf
u}}\right\\}^{1/2}.$ (38)

This measure achieves its maximum value of unity only if the two cross-
spectral density matrices are identical. Given Parseval’s identity, this
measure also quantifies the difference in amplitude, coherence and
polarization of the corresponding fields at the detector plane; given
appropriate normalization, such RMS difference is given by $\sqrt{1-C^{2}}$.
However, the main interest in this work is in differences in PSF, namely only
in intensity. Therefore, it is expected that $\sqrt{1-C^{2}}$ will
overestimate (and hence provide an upper bound for) the RMS errors in PSF. It
is convenient to use polar coordinates to evaluate this measure, and for the
case considered here the integrals in the angular variables turn out to be
solvable in closed form, leading to the expression

$\displaystyle C({\rm NA})$
$\displaystyle=\frac{\mathcal{G}_{0}+\mathcal{G}_{2}/2}{\sqrt{(\mathcal{G}_{0}+\mathcal{G}_{2})(\mathcal{G}_{0}+\mathcal{G}_{1}/2+\mathcal{G}_{2})}},$
(39)

where $\mathcal{G}_{j}=\int|g_{j}(u)|^{2}F(u,z_{0})\,u\,du$. Note that the
dependence in $z_{0}$ remains for the SAF region.

So far we have considered the similarity of the PSFs for a transverse dipole
and an unpolarized source (supplemented with a polarizer), both at the focal
plane. Let us consider now the effect of a small amount of defocus. The Taylor
expansion approach is useful for these purposes. Figure 7(b) shows the
distributions of $\mathcal{I}_{m}$ at the nominal focal distance for an
unpolarized emitter followed by a horizontal polarizer. Given their uniform
polarization, these PSF components maintain their distributions (except for a
scaling factor) when projected onto any other polarization. Therefore, unlike
a single horizontal dipole, as shown in Fig. 4(d,e), $\mathcal{I}_{m}$ remains
zero for odd $m$ even when projecting onto a circular polarization, showing
that the behavior of a transverse dipole under defocus is not being fully
mimicked.

###  5.2 Mimicking transverse dipoles using finite-size beads

Let us finish by comparing the PSF of a true horizontal dipole to that of a
mimicked one using a fluorescent bead and a linear polarizer at the pupil
plane of the imaging system. For the latter, we use the approximation in Eq.
(15) with $N=15$. We also test the deconvolution method discussed in the
previous section to compensate for the blurring introduced by the bead’s size.
Figure 11 shows the resulting simulated PSFs for a high NA system with no SAF
and with an index matching scenario (NA$<n_{0}=n_{f}$) for two cases: a
standard imaging system with no aberrations, and a PSF engineering technique
called CHIDO [16] based on a spatially-varying birefringent mask and circular
polarization filtering.

Figure 11: Simulation of the PSFs of (a,d) true horizontal dipoles, (b,e)
mimicked ones using a 200nm diameter fluorescent bead and a polarizer, and
(c,f) the deconvolution of the results in (b) and (e) with an apodized disk
$G_{0}$. These PSFs are (a-c) for no polarization filtering, and (d-f) for an
engineered PSF [16]. Here NA$=1.3$, there is index matching and the embedding
medium is water ($n_{f}=n_{0}\approx 1.33$). Squares represent $1.5\mu$m in
the object space.

We also computed the RMS error between the PSFs of an emulated fluorophore
(with and without deconvolution) and a true horizontal dipole using different
bead radii and various values of NA; these are shown in Fig. 12. Naturally,
the error between the raw PSFs increases with both NA and bead radius, as
shown in Fig. 12(a). Note from Fig. 12(b), though, that the deconvolution
using the apodized disk $G_{0}$ eliminates almost completely the error’s
dependence on the radius (since the contours are essentially horizontal).
Surprisingly, on the other hand, the deconvolution with a uniform disk, shown
in Fig. 12(c) also accounts well for the size mismatch, and can give an error
that is smaller for larger beads. We performed the same computations for
CHIDO’s engineered PSFs and obtained almost identical behavior, only with
slightly higher RMS errors (approximately shifted up by $1\%$).

Figure 12: RMS error, for varying NA and bead radius, between the PSF of a
true transverse dipole and the mimicked one using a fluorescent bead for: (a)
no treatment, (b) deconvolution with the apodized disc $G_{0}$, and (c)
deconvolution with a uniform disk. Here $\mathrm{NA}=1.3$ and the embedding
medium is water ($n_{0}=1.33$).

The reason why the effect of particle size is not necessarily negative can be
understood from the coherence analysis. Due to the van Cittert-Zernike
theorem, the effect of the spatial extension of the bead is to include an
extra factor in the cross-spectral density matrix at the pupil given in Eq.
(36). This factor turns out to take a simple analytic form for the two bead
models discussed here: for the solid sphere it is given by
$3\sigma_{V}\,j_{1}(\eta)/\eta$, while for the empty shell it is given by
$\sigma_{S}{\rm sinc}(\eta)$, where for an index matching scenario

$\eta({\bf u}_{1},{\bf
u}_{2})=knR\sqrt{u_{1}^{2}+u_{2}^{2}-2u_{1}u_{2}\cos(\varphi_{2}-\varphi_{1})+[\gamma^{*}(u_{1},n)-\gamma(u_{2},n)]^{2}}.$
(40)

Note that for $R\to 0$ the two factors tend to a constant equal to
$\sigma_{V}$ or $\sigma_{S}$. For $R>0$, on the other hand, these factors
decrease as $u_{1}$ and $u_{2}$ increase, meaning that the contributions from
the edges of the pupil are partially suppressed. These are precisely the
regions where the radiation pattern of a transverse dipole differs most from
that of an unpolarized source supplemented with a linear polarizer, and
therefore this suppression is not necessarily detrimental to the fit. As shown
by the results shown in Fig. 12(c), increasing the radius can even lead to
possibly better matches, particularly after post-processing of the PSFs.

##  6 Conclusions

In this work we studied how and under which conditions fluorescent beads can
be used to mimic transverse dipoles. We did this by developing models that
take into account the three-dimensional size and the incoherent and isotropic
emission pattern of fluorescent beads. Of particular importance is the
development of a semi-analytic method to model the blurring due the bead size.
This method is based on a Taylor expansion around the bead’s central plane and
provides different orders of corrections for the final PSF which are
implemented through two-dimensional convolutions. It is therefore much easier
to implement than a full three-dimensional convolution, and it converges
rapidly, with the first two terms providing already an estimate of the
blurring that is almost indistinguishable from the exact result for typical
bead sizes and NAs. Additionally, we showed that the difference in size
between a point-like incoherent source and a fluorescent bead can be corrected
satisfactorily by 2D deconvolution with the apodized disk function given in
Eq. (20), hence reducing drastically the computational resources needed for
post-processing.

The spatial extent of the bead is not the only factor that makes its PSF
differ from that of a point dipole. The second important difference is in
their radiation patterns. The pattern for the bead can be made more similar to
that of the dipole by placing at the pupil a polarizer plane that is aligned
with the direction of the dipole, assumed here to transverse to the axis of
the microscope. For systems with small NA this leads to very good agreement,
but for larger NA there are notable differences in the intensity, polarization
direction, and even the spatial coherence of the pupil distributions of the
point dipole and the bead followed by a polarizer. These differences result in
a difference in the corresponding PSFs, particularly in their high spatial
frequency components. As mentioned earlier, the difference due to the blurring
caused by bead size can be partially removed through deconvolution.
Surprisingly, the resulting error can sometimes be smaller for larger beads.
This can be explained by the fact that the spatial blurring suppresses the
contributions at the pupil plane from the edges of the pupil (i.e., the PSF’s
high spatial frequencies). The deconvolution process, which aims to reinstate
these suppressed contributions, does it in a way that can be sometimes more
consistent with the pattern for a point dipole than the suppressed
contributions were.

The current manuscript focuses mostly on transverse dipoles, but future work
will extend this study to ways of emulating longitudinal and/or oblique
dipoles by using beads and appropriate polarization plates.

## Funding

This research received funding from the 3DPolariSR and 3Dpol ANR grants
(ANR-20-CE42-0003, ANR-21-CE24-0014). R. G. C. acknowledges funding from the
Labex WIFI (ANR-10-LABX-24, ANR-10-IDEX-0001-02 PSL*).

## Disclosures

The authors declare no conflicts of interest.

## Data Availability Statement

The code used is available from the corresponding authors upon reasonable
request.

## References

  * [1] J. Mertz, “Nonlinear microscopy: new techniques and applications,” Current Opinion in Neurobiology, vol. 14, no. 5, pp. 610–616, 2004.
  * [2] S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett., vol. 19, pp. 780–782, Jun 1994.
  * [3] M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm),” Nature Methods, vol. 3, no. 10, pp. 793–796, 2006.
  * [4] S. T. Hess, T. P. K. Girirajan, and M. D. Mason, “Ultra-high resolution imaging by fluorescence photoactivation localization microscopy,” Biophysical Journal, vol. 91, pp. 4258–4272, 2022/08/07 2006.
  * [5] H. Kao and A. Verkman, “Tracking of single fluorescent particles in three dimensions: use of cylindrical optics to encode particle position,” Biophysical Journal, vol. 67, no. 3, pp. 1291–1300, 1994.
  * [6] B. Huang, W. Wang, M. Bates, and X. Zhuang, “Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy,” Science, vol. 319, no. 5864, pp. 810–813, 2008.
  * [7] S. Liu, E. B. Kromann, W. D. Krueger, J. Bewersdorf, and K. A. Lidke, “Three dimensional single molecule localization using a phase retrieved pupil function,” Opt. Express, vol. 21, pp. 29462–29487, Dec 2013.
  * [8] S. Brasselet, “Polarization-resolved nonlinear microscopy: application to structural molecular and biological imaging,” Adv. Opt. Photon., vol. 3, no. 3, p. 205, 2011.
  * [9] J. Enderlein, E. Toprak, and P. R. Selvin, “Polarization effect on position accuracy of fluorophore localization,” Opt. Express, vol. 14, pp. 8111–8120, Sep 2006.
  * [10] M. P. Backlund, M. D. Lew, A. S. Backer, S. J. Sahl, G. Grover, A. Agrawal, R. Piestun, and M. W. E., “Simultaneous, accurate measurement of the 3D position and orientation of single molecules,” Proc. Nat. Acad. Sci., vol. 109, pp. 19087–92, 2012.
  * [11] S. Stallinga and B. Rieger, “Position and orientation estimation of fixed dipole emitters using an effective hermite point spread function model,” Opt. Express, vol. 20, pp. 5896–5921, Mar 2012.
  * [12] C. A. Valades Cruz, H. A. Shaban, A. Kress, N. Bertaux, S. Monneret, M. Mavrakis, J. Savatier, and S. Brasselet, “Quantitative nanoscale imaging of orientational order in biological filaments by polarized superresolution microscopy,” Proceedings of the National Academy of Sciences, vol. 113, no. 7, pp. E820–E828, 2016.
  * [13] C. V. Rimoli, C. A. Valades-Cruz, V. Curcio, M. Mavrakis, and S. Brasselet, “4polar-storm polarized super-resolution imaging of actin filament organization in cells,” Nat. Commun., vol. 13, no. 1, p. 301, 2022.
  * [14] A. S. Backer, M. P. Backlund, A. R. von Diezmann, S. J. Sahl, and W. E. Moerner, “A bisected pupil for studying single-molecule orientational dynamics and its application to three-dimensional super-resolution microscopy,” Applied Physics Letters, vol. 104, no. 19, p. 193701, 2014\.
  * [15] O. Zhang, J. Lu, T. Ding, and M. D. Lew, “Imaging the three-dimensional orientation and rotational mobility of fluorescent emitters using the tri-spot point spread function,” Applied Physics Letters, vol. 113, no. 3, p. 031103, 2018.
  * [16] V. Curcio, L. A. Alemán-Castañeda, T. G. Brown, S. Brasselet, and M. A. Alonso, “Birefringent fourier filtering for single molecule coordinate and height super-resolution imaging with dithering and orientation,” Nat. Commun., vol. 11, no. 1, p. 5307, 2020.
  * [17] C. N. Hulleman, R. Ø. Thorsen, E. Kim, C. Dekker, S. Stallinga, and B. Rieger, “Simultaneous orientation and 3d localization microscopy with a vortex point spread function,” Nature Communications, vol. 12, no. 1, p. 5934, 2021.
  * [18] T. Ding and M. D. Lew, “Single-molecule localization microscopy of 3d orientation and anisotropic wobble using a polarized vortex point spread function,” The Journal of Physical Chemistry B, vol. 125, pp. 12718–12729, 11 2021.
  * [19] T. Wu, J. Lu, and M. D. Lew, “Dipole-spread-function engineering for simultaneously measuring the 3d orientations and 3d positions of fluorescent molecules,” Optica, vol. 9, pp. 505–511, May 2022.
  * [20] M. Böhmer and J. Enderlein, “Orientation imaging of single molecules by wide-field epifluorescence microscopy,” J. Opt. Soc. Am. B, vol. 20, pp. 554–559, Mar 2003.
  * [21] M. A. Lieb, J. M. Zavislan, and L. Novotny, “Single-molecule orientations determined by direct emission pattern imaging,” J. Opt. Soc. Am. B, vol. 21, pp. 1210–1215, Jun 2004.
  * [22] R. Luchowski, Z. Gryczynski, Z. Földes-Papp, A. Chang, J. Borejdo, P. Sarkar, and I. Gryczynski, “Polarized fluorescent nanospheres,” Opt. Express, vol. 18, pp. 4289–4299, Mar 2010.
  * [23] L. Neumann, J. van ’t Oever, and N. F. van Hulst, “A Resonant Scanning Dipole-Antenna Probe for Enhanced Nanoscale Imaging,” Nano Letters, vol. 13, pp. 5070–5074, Nov. 2013.  Publisher: American Chemical Society.
  * [24] K. Zhanghao, L. Chen, X.-S. Yang, M.-Y. Wang, Z.-L. Jing, H.-B. Han, M. Q. Zhang, D. Jin, J.-T. Gao, and P. Xi, “Super-resolution dipole orientation mapping via polarization demodulation,” Light: Science & Applications, vol. 5, pp. e16166–e16166, Oct. 2016.
  * [25] P. N. Petrov, Y. Shechtman, and W. E. Moerner, “Measurement-based estimation of global pupil functions in 3d localization microscopy,” Opt. Express, vol. 25, pp. 7945–7959, Apr 2017.
  * [26] Y. Li, Y.-L. Wu, P. Hoess, M. Mund, and J. Ries, “Depth-dependent psf calibration and aberration correction for 3d single-molecule localization,” Biomed. Opt. Express, vol. 10, pp. 2708–2718, Jun 2019.
  * [27] T. Yan, C. J. Richardson, M. Zhang, and A. Gahlmann, “Computational correction of spatially variant optical aberrations in 3d single-molecule localization microscopy,” Opt. Express, vol. 27, pp. 12582–12599, Apr 2019.
  * [28] B. Richards, E. Wolf, and D. Gabor, “Electromagnetic diffraction in optical systems, ii. structure of the image field in an aplanatic system,” Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, vol. 253, no. 1274, pp. 358–379, 1959.
  * [29] E. Wolf, Introduction to the theory of coherence and polarization.  Cambridge University Press, 2007.
  * [30] L. Novotny and B. Hecht, Principles of Nano-Optics.  Cambridge University Press, 2006.


# Cosine Similarity of Multimodal Content Vectors for TV Programmes

Saba Nazir  Taner Cagali  Chris Newell  Mehrnoosh Sadrzadeh

###### Abstract

Multimodal information originates from a variety of sources: audiovisual
files, textual descriptions, and metadata. We show how one can represent the
content encoded by each individual source using vectors, how to combine the
vectors via middle and late fusion techniques, and how to compute the semantic
similarities between the contents. Our vectorial representations are built
from spectral features and Bags of Audio Words, for audio, LSI topics and
Doc2vec embeddings for subtitles, and the categorical features, for metadata.
We implement our model on a dataset of BBC TV programmes and evaluate the
fused representations to provide recommendations. The late fused similarity
matrices significantly improve the precision and diversity of recommendations.

Vector Representation, Multimodal Data, Audio, Subscripts, Metadata, Fusion,
Recommendation

##  1 Introduction

Ideas put forwards by Firth and Harris in the 1930’s led to the development of
vector representations for words. The original word vectors represented
context of textual use and the cosine distances between them, semantic
similarities (Turney & Pantel, 2010). Subsequent research extended the vector
representation methods from words to sentences and documents; nowadays, these
vectors are learnt using neural networks, for a survey see (Jurafsky & Martin,
2019). Recently, the textual vector representations have been enriched by
other modes of information, such audio-visual and cognitive (E. Bruni &
Baroni, 2014; Kiela & Clark, 2017). The enriched representations are also used
in multimodal content-based and hybrid recommendation systems for problems
such as cold-start (Oramas et al., 2017), (Barkan et al., 2019), e-commerce
assortment (Iqbal et al., 2018b) and genre classification (Ekenel & Semela,
2013). A large body of work exists here, with none as extensive or conclusive:
e.g. (Yang et al., 2007), only considers tags and titles for textual data,
(Bougiatiotis & Giannakopoulos, 2018) does use full subtitles but does not
improve on the metadata-only recommendations.

This paper works with multimodal vector representations for audio and text,
and investigates their application to TV recommendations, based on a dataset
of 145 BBC TV programmes. On the methodological side, this is the first time
that neural (Doc2Vec) and topical (LSI) document vectors are combined with
audio (BoAW) vectors and vector representations of metadata. On the
experimental side, the late fused representations significantly improve the
precision and diversity of the recommendations.

##  2 Multimodal Content Vectors

Figure 1:  Multimodal Content Recommendations Framework

Our dataset contains 145 BBC TV programmes with their subtitle and audiovisual
files and metadata information.

Subtitle Vectors. Latent Semantic Indexing (LSI) (Papadimitriou et al., 2000),
a topic modelling technique, was applied to the subtitle files. LSI is a two-
step procedure. Firstly, a document- term matrix is generated via a low-rank
approximation obtained from the term vector space projections of the Bag of
Words vectors. Secondly, Singular Value Decomposition (SVD) is applied to the
document-term matrix, where the newly created eigenvectors represent the
concepts within the latent space. We worked with 50 dimensional spaces. LSI
improves on the term-document matrices, but does not take word order into
account. To deal with this, we worked with neural semantics embeddings Doc2vec
(Le & Mikolov, 2014). Doc2vec is an extension of the neural semantic word
embeddings Word2vec (T. Mikolov & Dean, 2013). We worked with Paragraph Vector
Distributed Memory (PV- DM), which concatenates the unique document ID with
the context words with respect to the specified context window over the text
and preserves the order of words.

Audio Vectors. We extracted acoustic features including MFCCs, Spectral
Centroid, Zero Crossing Rate, Spectral Flatness and Root Mean Square using
LibROSA (McFee et al., 2015), keeping audio sampling rate of 22050 Hz and hop
length of 512 samples, with variable lengths of audio tracks averaging on
about 30 mins each for a detailed analysis. The extracted multiple acoustic
features are concatenated, normalised and then used as audio vectors for each
audio.We then followed (Kiela & Clark, 2017) and used a Bag of Audio Words
(BoAW) model to learn abstract audio vector representations. BoAW is used in
audio information retrieval recognition (Liu et al., 2010; Pancoast & Akbacak,
2012; Rawat et al., 2013) and acoustic event detection (Plinge et al., 2014;
Grzeszick et al., 2015; Lim et al., 2015). They are learnt via mini-batch
K-means clustering with $K=50$.

Metadata Vectors. Metadata representations are based on the editorially-
assigned attributes of the programmes. We worked with hierarchical genre
information, e.g. ”factual/scienceandnature/natureandenvironment”, where a
match can occur at any level. A categorical feature vector is created for each
attribute by traversing the trees.

Fusion. Individual content vectors are ranked using cosine similarities and
are fused with middle and late fusion techniques (Kiela & Clark, 2017; Atrey
et al., 2010; Zhu et al., 2006). In middle fusion, we concatenated the
different vectors representations. In late fusion, we first computed the
cosine similarities of pairwise vectors, resulting in 3 symmetric $145\times
145$ similarity matrices; then combined these with each other by weighted
averaging. Figure 1 shows our late fusion framework.

Table 1: Singular and fused model evaluations. The acronyms LSI, D2V, Aud, MD, and Fus are used for Latent Semantic Indexing, Doc2vec, Audios, metadata, respectively. User is the user-based behavioural similarity that we are trying to estimate. Model | MAP@10 | ILD@10 | MAP@20 | ILD@20
---|---|---|---|---
LSI | 11.30 | 69.89 | 13.40 | 76.79
D2V | 11.76 | 77.20 | 13.88 | 80.37
Aud | 06.67 | 77.96 | 8.11 | 81.38
MD | 10.78 | 35.52 | 12.77 | 52.72
Fus | 14.98 | 61.29 | 17.45 | 70.00
User | 15.60 | 79.73 | 18.51 | 80.90

##  3 Evaluation and Results

Performance of the singular and fused vectors is evaluated by a personalised
Python-based recommender system evaluation framework, developed using
MyMediaLite library (MyMediaLite, ). We calculated the Mean Average Precision
(MAP) and Intra-list diversity (ILD) of the recommendations at ranks 10 and 20
obtained from cosine similarities, and compared these with a metadata-only
recommendation system (MD) and the behavioural similarity measure obtained
from users’ viewings (USER). Our aim was to increase ILD while maintaining or
increasing MAP. Even individually, LSI and Doc2Vec outperformed metadata in
both MAP and ILD; audios (AUD) showed the highest ILD and lowest MAP. The best
performance was by the multimodal model late fusion (FUS) with the weights 0.7
LSI, 1.5 D2V, 0.2 AUD, 0.65 MD. It increased MAP and ILD, at both ranks 10 and
20, see Table 1. It also came very close to the gold standard user behaviour.

Analysis. We analysed individual features of a subset of programmes to find
out the roles various features played in measuring the similarities. We worked
in three groups: Group (I). closely correlated programmes of the same genre,
e.g. Eastenders, Doctors, Waterloo Road. Group (II). uncorrelated programmes,
of different genres, e.g. Eastenders, Football League Show, University
Challenge. Group (III). correlated programmes, of different genres, e.g.
Eastenders (soap drama), Jamie Private School Girl (comedy), Notorious Betty
Page (autobiographical drama). In Group (I), the genre similarities were
nicely reflected by the audio and textual features. In Group (II) ,
uncorrelatedness of the programmes were manifested in the textual and audio
features. In Group (III), a genre-only analysis failed but a multi modal one
succeeded. The similarities between the different-genre programmes clearly
showed themselves in the textual and audio features. The most efficient
differentiating vectors were BoAW model and LSI.

##  4 Future Work

Our framework can easily be extended to other modalities. We worked with
standard Python packages for sentiment and writing style, with format and
service metadata, and with 200 scene images extracted from the video files of
programmes, but did not obtain improvements. Working with more sophisticated
attributes, and larger number of images and jointly learning multimodal
representations, as in (Iqbal et al., 2018a), via neural nets is work in
progress.

## References

  * Atrey et al. (2010) Atrey, P. K., Hossain, M. A., El Saddik, A., and Kankanhalli, M. S.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ , 16(6):345–379, 2010.
  * Barkan et al. (2019) Barkan, O., Koenigstein, N., Yogev, E., and Katz, O.  Cb2cf: a neural multiview content-to-collaborative filtering model for completely cold item recommendations.  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , pp. 228–236, 2019.
  * Bougiatiotis & Giannakopoulos (2018) Bougiatiotis, K. and Giannakopoulos, T.  Enhanced movie content similarity based on textual, auditory and visual information.  _Expert Systems with Applications_ , 96:86 – 102, 2018\.
  * E. Bruni & Baroni (2014) E. Bruni, N. T. and Baroni, M.  Multimodal distributional semantics.  _Journal of Artifical Intelligence Research_ , 49:1–47, 2014.
  * Ekenel & Semela (2013) Ekenel, H. K. and Semela, T.  Multimodal genre classification of tv programs and youtube videos.  _Multimedia tools and applications_ , 63(2):547–567, 2013.
  * Grzeszick et al. (2015) Grzeszick, R., Plinge, A., and Fink, G. A.  Temporal acoustic words for online acoustic event detection.  In _German Conference on Pattern Recognition_ , pp. 142–153. Springer, 2015.
  * Iqbal et al. (2018a) Iqbal, M., Kovac, A., and Aryafar, K.  A multimodal recommender system for large-scale assortment generation in e-commerce.  In _The SIGIR 2018 Workshop On eCommerce co-located with the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, USA, July 12, 2018_ , 2018a.
  * Iqbal et al. (2018b) Iqbal, M., Kovac, A., and Aryafar, K.  A multimodal recommender system for large-scale assortment generation in e-commerce.  _arXiv preprint arXiv:1806.11226_ , 2018b.
  * Jurafsky & Martin (2019) Jurafsky, D. and Martin, J. H.  _Speech and Language Processing_.  https://web.stanford.edu/ jurafsky/slp3/, 2019.
  * Kiela & Clark (2017) Kiela, D. and Clark, S.  Learning neural audio embeddings for grounding semantics in auditory perception.  _Journal of Artificial Intelligence Research_ , 60:1003–1030, 2017.
  * Le & Mikolov (2014) Le, Q. and Mikolov, T.  Distributed representations of sentences and documents.  In _International conference on machine learning_ , pp. 1188–1196, 2014.
  * Lim et al. (2015) Lim, H., Kim, M. J., and Kim, H.  Robust sound event classification using lbp-hog based bag-of-audio-words feature representation.  In _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015.
  * Liu et al. (2010) Liu, Y., Zhao, W.-L., Ngo, C.-W., Xu, C.-S., and Lu, H.-Q.  Coherent bag-of audio words model for efficient large-scale video copy detection.  In _Proceedings of the ACM international conference on image and video retrieval_ , pp. 89–96, 2010.
  * McFee et al. (2015) McFee, B., Raffel, C., Liang, D., Ellis, D. P., McVicar, M., Battenberg, E., and Nieto, O.  librosa: Audio and music signal analysis in python.  In _Proceedings of the 14th python in science conference_ , volume 8, 2015.
  * (15) MyMediaLite.  Mymedialite recommender system library.  http://www.mymedialite.net/.  (Accessed on 02/20/2020).
  * Oramas et al. (2017) Oramas, S., Nieto, O., Sordo, M., and Serra, X.  A deep multimodal approach for cold-start music recommendation.  In _Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems_ , pp. 32–37, 2017.
  * Pancoast & Akbacak (2012) Pancoast, S. and Akbacak, M.  Bag-of-audio-words approach for multimedia event classification.  In _Thirteenth Annual Conference of the International Speech Communication Association_ , 2012.
  * Papadimitriou et al. (2000) Papadimitriou, C. H., Raghavan, P., Tamaki, H., and Vempala, S.  Latent semantic indexing: A probabilistic analysis.  _Journal of Computer and System Sciences_ , 61(2):217–235, 2000.
  * Plinge et al. (2014) Plinge, A., Grzeszick, R., and Fink, G. A.  A bag-of-features approach to acoustic event detection.  In _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 3704–3708. IEEE, 2014.
  * Rawat et al. (2013) Rawat, S., Schulam, P. F., Burger, S., Ding, D., Wang, Y., and Metze, F.  Robust audio-codebooks for large-scale event detection in consumer videos.  In _INTERSPEECH_ , pp. 2929–2933, 2013.
  * T. Mikolov & Dean (2013) T. Mikolov, K. Chen, G. C. and Dean, J.  Efficient estimation of word representations in vector space.  In _Proceedings of ICLR_ , Scottsdale, AZ, 2013.
  * Turney & Pantel (2010) Turney, P. and Pantel, P.  From frequency to meaning: vector space models of semantics.  _Journal of Artifical Intelligence Research_ , 37:141–188, 2010.
  * Yang et al. (2007) Yang, B., Mei, T., Hua, X.-S., Yang, L., Yang, S.-Q., and Li, M.  Online video recommendation based on multimodal fusion and relevance feedback.  In _Proceedings of the 6th ACM International Conference on Image and Video Retrieval_ , CIVR ’07, pp. 73–80, New York, NY, USA, 2007. Association for Computing Machinery.
  * Zhu et al. (2006) Zhu, Q., Yeh, M.-C., and Cheng, K.-T.  Multimodal fusion using learned text concepts for image categorization.  In _Proceedings of the 14th ACM international conference on Multimedia_ , pp. 211–220, 2006.