# S-polarized light-sheets improve resolution and light-efficiency in oblique
plane microscopy

Jon-Richard Sommernes  Department of Physics and Technology, UiT The Arctic
University of Norway, Tromsø, Norway  Alfred Millett-Sikking  Calico Life
Sciences LLC, South San Francisco, CA, USA  Florian Ströhl  Department of
Physics and Technology, UiT The Arctic University of Norway, Tromsø, Norway
<EMAIL_ADDRESS>
###### Abstract

Oblique plane microscopy (OPM) offers 3D optically sectioned imaging with high
spatial- and temporal-resolution while enabling conventional sample mounting.
The technique uses a concatenation of three microscopes, two for remote
focusing and a tilted tertiary microscope, often including an immersion
objective, to image an oblique sample plane. This design induces Fresnel
reflections and a reduced effective aperture, thus impacting the resolution
and light efficiency of the system. Using vectorial diffraction simulations,
the system performance was characterized based on illumination angle and
polarization, signal to noise ratio, and refractive index of the tertiary
objective immersion. We show that for samples with high fluorescent
anisotropy, s-polarized light-sheets yield higher average resolution for all
system configurations, as well as higher light-efficiency. We also provide a
tool for performance characterization of arbitrary light-sheet imaging
systems.

## Introduction

Volumetric imaging is an important tool for studying biological phenomena, and
has thus given rise to a broad set of techniques with various benefits and
drawbacks[1]. One of these techniques is light-sheet fluorescent microscopy
(LSFM)[2, 3, 4, 5, 6, 7, 8]. LSFM offers a combination of high spatial- and
temporal-resolution, optical sectioning, and low phototoxicity, which makes
the technique highly suitable for live cell imaging. LSFM works by having two
orthogonally oriented objectives with a shared foci. One objective illuminates
the focus plane of the second objective with a light-sheet during imaging.
This technique has given rise to several designs over the last decades [3, 4,
5, 6, 7]. However, because the objectives are placed orthogonal to each other
while maintaining a shared foci, the combined Numerical Aperture (NA) of the
two objectives is spatially limited. Additionally, the physical design of the
two-objective setup generally prevents the system from utilizing conventional
sample mounting. This makes the technique challenging for certain sample types
like slides, dishes, and multiwell plates.

Figure 1: Schematic setup of an OPM system. O and T represent objectives and
tube lenses respectively. The lens train from O1 to O2 forms a perfect 3D
imaging system, and O3 and T3 forms a tertiary tilted microscope. The sample
is placed above O1, and illuminated by a light-sheet (blue) inclined by a tilt
$\upalpha$. The illuminated area is relayed onto the focus of O3, and re-
imaged onto the camera.

Oblique Plane Microscopy (OPM)[9, 10, 11, 12, 13, 14, 15] is a technique that
uses the same objective lens for both illumination and imaging, thus
overcoming the main limitations of LSFM. A schematic OPM setup is shown in
Figure 1. In this setup, the light-sheet is formed by illuminating the pupil
of the imaging objective off center. The center point of the pupil
illumination will determine the inclination of the light-sheet, while the
extent of the illumination will determine the thickness, width, and length of
the focused sheet. This will generate a light-sheet at an oblique angle to the
focal plane of O1. Where O represents a microscope objective, and T represents
a tube lens.

Due to the oblique illumination, most of the illuminated area will be out of
the traditional focus plane. To get this plane in focus, the sample volume
needs to be relayed to a remote image space using a perfect 3D imaging
system[16], seen in Figure 1 as the lens train from O1 to O2. To avoid
aberrations and distortions, the magnification of a perfect imaging system
should be close to unity[17]. The remote image space is then re-imaged by a
third microscope, seen in Figure 1 as O3 and T3. This microscope is oriented
so the remote image plane is perfectly in the traditional focus plane of O3.

Here, we assume the effective pupil function of this system will then be
limited by the overlap of the O2 and O3 pupil. This effective pupil will limit
the achievable resolution of the system. To maximize the effective pupil, a
high NA O3 can be used. However, due to the tilted geometry of the tertiary
microscope, an immersion medium is needed to fit a high NA O3. A specialized
glass immersion objective (P/N: AMS-AGY), often called “snouty”[18], has been
created for this purpose, although water immersion objectives using glass
cover slips have also been used[19]. This change in RI from O2 to O3 can make
the light transmission highly polarization dependent. This dependency will be
especially noticeable for samples with high fluorescent anisotropy[20].

Fluorescent anisotropy is a measure of how much of the excitation polarization
is preserved in the emission. In this paper we model fluorophores as ensembles
of rotating dipoles. If a dipole has low rotational diffusion, e.g. from a
chemical bond or high mass, the anisotropy of the dipole is increased. In this
paper, we explore the extreme case of low rotational diffusion, corresponding
to maximum retention of the excitation polarization.

## Methods

In order to investigate the impact of polarization on the system, a simulation
software was developed to model the vectorial diffraction of the system as
outlined by Kim, et al[21]. In the simulations, an emission source was
modelled as a point-like dipole and placed in the focal point of O1. This was
done to immitate the emission pattern of a small fluorophore. The electric
field of the dipole emission was then characterized at discreet points within
the collection cone of O1. In order to trace the fields through the optical
system, each optical component was modelled as a 3$\times$3 Jones matrix. To
find the complete transformation of the system, the matrices were then
multiplied to find a system transformation. For our case, the system model can
be described as:

$\displaystyle\textbf{T}_{sys}(\theta_{1},\phi,\alpha)=$
$\displaystyle\textbf{R}_{\textbf{z}}^{-1}(\phi_{\alpha})\textbf{M}_{3}(\theta_{3})\textbf{R}_{\textbf{y}}(\theta_{3})\textbf{F}_{\textbf{T}}\textbf{R}_{\textbf{y}}(-\theta_{2})\textbf{R}_{\textbf{z}}(\phi_{\alpha})\textbf{R}_{\textbf{x}}(\alpha)\textbf{R}_{\textbf{z}}^{-1}(\phi)\overline{\textbf{M}}_{2}(\theta_{2})\textbf{M}_{1}(\theta_{1})\textbf{R}_{\textbf{z}}(\phi)\underline{E}_{0}$
(1) $\displaystyle\textbf{M}_{i}(\theta_{i})=$
$\displaystyle\textbf{T}_{i}(\theta_{i}^{\prime})\textbf{O}_{i}(\theta_{i})\boldsymbol{\Gamma}_{i}(\theta_{i})A_{i}(\theta_{i})\text{\quad,
\quad}\overline{\textbf{M}}_{i}(\theta_{i})=\textbf{O}_{i}(\theta_{i})\textbf{T}_{i}(\theta_{i}^{\prime})\boldsymbol{\Gamma}_{i}(\theta_{i})A_{i}(\theta_{i})$
(2)

where $\textbf{R}_{\textbf{i}}$ denotes the rotation matrix around axis i, O
and T are both lens matrices but separated to distinguish between tube lenses
and objectives, $\boldsymbol{\Gamma}_{i}$ is the light transmission of the
objective at $\theta_{i}$, $A_{i}$ is the apodization of the microscope at
$\theta_{i}$, $\textbf{F}_{\textbf{T}}$ is the Fresnel transmission of the
refractive index (RI) change, and $\underline{E}_{0}$ is the electric field of
the dipole emission. The Fresnel transmission is only applied if the
transmission to a new immersion medium is not altered by a anti-reflection
coating. A full description of the matrices can be found in the supplementary
information.

For the glass immersion objective (AMS-AGY v1.0), and the dry objective used
for O2, the transmission was measured experimentally using a single-axis stage
and a power meter. A laser, beam expander, tube lens, polarizer, and half-wave
plate was built into a single unit and mounted on a translation stage. The
laser beam was expanded to fill a tube lens, then focused onto the back focal
plane (BFP) of the objective. By moving the focus of the beam radially from
the center of the BFP, the output angle of the beam was changed. The power of
the beam was then measured at a range of angles for both p- and s-polarized
light. The results were then curve fitted to 5th degree polynomials. This
polynomial can be assumed to be a taylor approximation of the true
transmission function. The function was then used to make a transmission mask
for the corresponding objective. The data used for this mask is found in
Figure 2.

Figure 2: The top panel shows the light transmission for a 0.95NA dry
objective (CFI Plan Apochromat Lambda D 40X), while the bottom panel shows the
light transmission for a 1NA glass immersion objective (AMS-AGY v1.0). The
blue and red lines indicate p- and s-polarized light respectively. The
transmission of the objectives was normalized to the maximum value and curve
fitted using 5th degree polynomials.

Further, for every microscope the lenses are assumed to share an identical
BFP. Assuming all lenses follow the abbe sine condition, the angles of the
simulated rays can be related using[22]:

$\theta_{i}^{\prime}=\arcsin\left(\frac{\text{NA}^{\prime}}{\text{NA}}n\sin(\theta_{i})\right)$
(3)

where NA is the Numerical Aperture of the objective, $\text{NA}^{\prime}$ is
the Numerical Aperture of the tube lens, $\theta_{i}$ is the angles of rays at
the objective, $\theta_{i}^{\prime}$ is the anlges of the rays at the tube
lens, and $n$ is the refractive index of the objective immersion medium.

For two lenses sharing an image space, the lenses are assumed to share an
identical focal point. To get the input angles of the collimating lens, the
output angles of the focusing lens is altered using either Snell’s law for a
immersion medium change, or a change in the optical axis for the transition
from O2 to O3.

To ensure the validity of the model, some assumptions were made for the
system: the dipole emitter is located perfectly in the focal plane of the
first objective, the lenses are rotationally symmetric and centered on the
optical axis, and there are no spherical or chromatic aberrations.

When the electric field has been traced through the system, we end up with the
final electric field converging towards the focal point. This electric field
can then be evaluated using a version of the Debye integral[23]. Owing to the
computational inefficiency of numerical integration, the integral was
evaluated using a Fourier transform version of the Debye integral. The version
of the Debye integral used here was based on the method outlined by
Leutenegger et al.[24], then modified and implemented by James Manton[25]. To
get sufficient sampling of our signal, the pixel size, $v$, of our camera
needs to be sufficiently small. To fulfill this condition, Nyquist sampling is
normally sufficient. For an in depth derivation of this condition, see the
supplementary information.

Having set a suitable pixel size, we need to find a sampling volume that is
large enough to avoid significant PSF truncation. As the maximum extent of the
PSF is in the axial direction, we can find the sampling volume by the axial
extent of the PSF. Assuming the PSF can be modeled as an Airy disc, the PSF
will extent until the signal is below the noise floor. Although the
mathematical formulation of the Airy disc can be used to determine this, it is
more practical to test different sampling volumes until the results converge.
For our simulations, a sampling size of 240$\upmu$m in each axis was found to
be mostly sufficient.

Now we have set constrains on the pixel size and count. The remaining
simulation parameters are only constrained by the specific setup, and can thus
be freely chosen without impacting the validity of the simulation. The
remaining simulation parameters are: the light-sheet inclination and
polarization, the fluorescent anisotropy and brightness of the sample, the
excitation and emission wavelength, the lenses used, and the size of the
dipole ensemble used for the PSF calculation. The light-sheet thickness is
determined by its opening angle. As we are, in this paper, interested in the
limits of each system configuration, we assume the light-sheet is made using
the maximum available aperture. The parameters used for the simulations will
be given in the results section, alongside the corresponding result.

To predict the Point Spread Function (PSF) that will be read from a camera,
the noise-less PSF was normalized to achieve the desired photon count. Photon
shot noise was then added by passing the voxels through a Poisson
distribution, followed by the addition of Gaussian noise to simulate camera
readout noise. Once noise was added, a bias offset was included and the voxels
were discretized.

To determine the Optical Transfer Function (OTF), the PSF was Fourier
transformed. The OTF contains the resolvable frequencies of an image, and by
identifying the maximum extent of the OTF in each axis, the corresponding
resolution limit can be defined. Assuming a PSF with shot noise, Gaussian
readout noise, and a bias offset, the background of the OTF can be
approximated using:

$|\hat{\eta}_{X}|=\sqrt{DC_{\hat{X}}+\sigma_{RMS}^{2}-b}$ (4)

where $|\hat{\eta}_{X}|$ is the average background power of the Fourier
transform, $DC_{\hat{X}}$ is the DC component of the Fourier transform,
$\sigma_{RMS}$ is the RMS of the readout noise, and $b$ is the bias offset.
For a rigorous derivation of this background, see the supplementary
information.

Subtracting this background from the OTF, the cutoff frequency is given by the
first zero crossing of the modified OTF. The resolution limit is then found
using the cutoff frequency and the base frequency of the Fourier transform.
However, in $\hat{z}$-direction the OTF will have two different cutoff
frequencies, one along the $\hat{z}$-axis and a second for the maximum extent
in the $\hat{z}$ direction. The cutoff along the $\hat{z}$-axis will give a
sectioning thickness, while the maximum extent will give the axial resolution
limit. Then, assuming the PSF is an ellipsoid, we can find the in-focus area
and volume of the PSF. For this paper, we will use the sectioning thickness to
calculate the PSF volume.

## Results

Figure 3: Panels a) and b) shows the PSF volume and area respectively for all system configurations using an AMS-AGY v1.0 O3. The blue and red bars represent p- and s-polarized light-sheets respectively. The left axis represents a range of light-sheet inclinations (angle from focus plane to light-sheet plane), while the right axis represents the photon count at the brightest pixel. The PSF area and volume were calculated assuming an elipsoidal PSF. All results are derived from ten simulations with identical system configuration and an ensemble of 100 dipoles. Table 1: Resolution limits, sectioning, and light-sheet length for all tested system configurations | x [nm] | y [nm] | z [nm] | S [nm] |  $\mathrm{L}_{ls}$ [$\text{\,}\mathrm{\SIUnitSymbolMicro m}$]
---|---|---|---|---|---
$\upalpha$ | p | s | p | s | p/s | p/s |
20∘ | 188 | 185 | 216 | 205 | 571 | 1714 | 57.3
25∘ | 185 | 182 | 212 | 209 | 457 | 980 | 14.4
30∘ | 185 | 187 | 222 | 214 | 403 | 623 | 6.5
35∘ | 189 | 186 | 223 | 219 | 343 | 490 | 3.7
40∘ | 198 | 188 | 235 | 231 | 286 | 403 | 2.4

Figure 4: The OTF of a system configuration using 30° light-sheet inclination,
1.25NA water immersion O1, 0.95NA dry O2, 1.0 NA glass immersion O3, and
SNR=100. Panels a), c), and e) show the OTF for an s-polarized light-sheet,
while panels b), d), and f) show the OTF for a p-polarized light-sheet. Panels
a) and b) show the XY cross section of the OTF, c) and d) show the YZ cross
section, and e) and f) show a 3D rendering of the OTF.

Figure 5: Optical efficiency of OPM. a) shows the amount of light emitted by
the fluorophore that is transmitted through the system, normalized to the
maximum transmission. The black lines represent a 1NA dry O3 (physically
unrealized, but simulated to remove Fresnel reflections). The blue lines
represent an AMS-AGY v1.0 style O3. The yellow lines represent water immersion
O3 using a 1.5 RI cover slip. The green lines show the transmission of a
collimated laser transmitted through O2 and O3, measured experimentally. The
dotted lines represent p-polarized input light, and solid lines represent
s-polarized light. b) shows pictogram representing the excitation and emission
of the dipole emitters. The light-sheet (blue) can be p-polarized or
s-polarized, indicated by the axes drawn on the bottom left and top right of
the light-sheet respectively. The green shapes above the matrix represent the
emission pattern of a dipole oriented along the z-, x- and y-axis going from
left to right. The acceptance cone of the objective is indicated by the dark-
shaded area. In the matrix, each cell consist of two elements. On the left of
the cell is the excitation efficiency of the light-sheet on the dipole, and on
the right is the collection efficiency of the dipole emission. A red cross
represent low efficiency, and a green check mark represent high efficiency.

Figure 3 shows the PSF volume and is using a 1.35NA silicone immersion O1,
0.95NA dry O2, and a 1NA glass immersion O3. The results use an ensemble of
100 dipoles to make the effective PSF. To minimize the effects of the random
noise added to the PSF, the OTF was calculated for ten noise samples drawn
from the same distribution. The resolution limits were then found for all ten
noise samples, and averaged to find a single value. The axis specific
resolution limits, as well as the sectioning thickness and length of the
light-sheet, are given in Table 1. This table shows the resolution limits for
the highest simulated SNR value (using $10^{4}$ photons at the brightest
pixel). The results show an increase in PSF area and a decrease in PSF volume
when increasing the light-sheet inclination. The results also show an effect
on both PSF area and volume with respect to photon count. Depending on the
system configuration and SNR, s-polarized light can give anything from 2% to
30% decreased PSF area and volume.

Figure 4 shows an OTF for a system configuration using a 30° light-sheet
inclination, 1.25NA water immersion O1, 0.95NA dry O2, AMS-AGY v1.0 O3, and
SNR=100. The left panels (a, c, and e) show the OTF for an s-polarized light-
sheet, while the right panels (b, d, and f) show the OTF for a p-polarized
light-sheet. From the images, we can see a ring feature in the OTF for a
p-polarized light-sheet that is not present in the s-polarized case.

Figure 5, panel a) shows the optical efficiency for different system
configurations. All system configurations use a 1.35NA silicone immersion O1,
0.95NA dry O2, and a 1NA O3. The black lines represent a system with a dry O3.
Albeit a 1.0 NA dry objective does not exist, it is included to show the
theoretically optimal case. The yellow lines represent a water immersion O3
with a 1.5 RI glass coverslip to hold in the water[19]. The blue lines
represent an AMS-AGY v1.0 O3. From this, we see that an AMS-AGY v1.0 objective
gives higher optical efficiency than its water immersion counterpart thanks to
its anti-reflection coating. We also find that s-polarized light-sheets have a
higher optical efficiency than p-polarized light for most light-sheet
inclinations. Only at extreme tilt angles do p-polarized light-sheets
outperform s-polarized illumination in terms of light-efficiency.

Panel b) in Figure 5 shows a pictogram describing the light-sheet excitation
and dipole emission. The light-sheet (blue) can be p-polarized or s-polarized,
indicated by the axes drawn on the light-sheet. The green shapes above the
matrix represent the emission pattern of a dipole in various orientations. The
excitation efficiency of the light-sheet on the dipole is given as a inner
product of the polarization of the dipole and light-sheet. The collection
efficiency of the dipole emission is given as the overlap between the emission
spectrum and the acceptance cone of the objective lens. For a system
configuration to be light efficient, both the excitation and collection
efficiency must be good. As shown in the figure, this is only achieved for
s-polarized light-sheets, with a dipole oriented along the y-axis.

## Discussion

In the results shown in Figure 3, we see an increase in PSF area with
increased angle. This is expected, as the overlap of the O2 and O3 acceptance
cone will decrease with increased angle. This results in a decrease in the
effective aperture of the system, thus increasing the PSF area. The PSF
volume, however, is seen to decrease with increased angle. As the light-sheet
inclination increases, the aperture available for making the light-sheet
increases, thinning the light-sheet, and thus decreasing the effective PSF
volume. However, a system can use a fixed light-sheet thickness for any
inclination. In this case, the length of the light-sheet will be contant. This
allows for imaging deeper into a sample, but at the cost of volumetric
resolution. The light-sheet volume will then follow the trend of the simulated
PSF area.

The effect of light-sheet polarization is seen in Figure 4. Here, we see that
p-polarized light-sheets give rise to side lobes of low power in the OTF.
These side lobes are likely a result of a sign change in the OTF and are prone
to blend into the background more quickly. This effect is not present for
s-polarized light-sheets, making the higher frequency components of the OTF
more visible for s-polarized light-sheets. Also found in Figure 4 are
artifacts from the Fourier transform along the kz axis (seen in panels c) and
d)). These artifacts are a result of PSF truncation, and can be minimized by
increasing the sampling volume.

The results shown in Figure 3 are simulated using an AMS-AGY v1.0 style O3. As
seen in Figure 2, the light throughput of this objective is minimally affected
by the polarization, which is not the case for a water immersion O3 with
coverslip. Note that an anti-reflection coating could be applied to a
coverslip to provide similar resilience to polarization and improved
transmission performance. In this case, the light throughput will be highly
dependent on the polarization of the light-sheet. This will likely alter the
results shown in Figure 3.

Further, the immersion medium of O3 will affect the light transmission of the
system, as seen in Figure 5. Here we see that s-polarized light-sheets have a
higher optical efficiency than p-polarized light-sheets. This effect is
illustrated in panel b) of Figure 5. As an s-polarized light-sheet is the most
effective at exciting a dipole oriented along the y-axis, the collection
efficiency of the emission will be higher for s-polarized light-sheets.
However, due to better transmission of p-polarized light after O1, as seen in
Figure 2, this gap in optical efficiency might decrease for certain system
configurations. This increased transmission is not, however, sufficient to
overcome the increased collection efficiency of s-polarized light-sheets. This
means that s-polarized light-sheets will almost always have a higher optical
efficiency than p-polarized light-sheets. This effect has only been simulated
for a non-scattering medium. In scattering samples, other effects might come
into play. For instance, p-polarized light is more likely to be scattered in
the plane of the light-sheet, thus reducing unwanted background light
excitation in dense samples.

The results also show that a AMS-AGY v1.0 style O3 gives lower light-loss than
the Fresnel reflections caused by a water immersion O3 using a glass
coverslip. This can be explained by the anti-reflection coating on the AMS-AGY
v1.0. Further, the light transmission of the AMS-AGY v1.0 has been measured
experimentally, while the water immersion O3 has only been simulated using
Fresnel transmissions. This means the transmission loss will likely be higher
for the water immersion O3 than shown in Figure 5, which would further
increase the gap in optical efficiency between the two objectives. If no anti-
reflection coated coverslip is used with a water-immersion lens, we conclude
that the AMS-AGY O3 is the better choice for most OPM systems

## Conclusion

In this paper we have found that s-polarized light-sheets offer a more stable
OTF, resulting in up to 30% reduction in PSF volume for samples with high
fluorescent anisotropy. We have also shown s-polarized excitation light to
give up to 10% increased optical efficiency for certain system configurations.
We have found that s-polarized light-sheets increase optical efficiency for
inclinations up to approximately 45∘. This increase in both achievable
resolution and optical efficiency makes s-polarized light-sheets the better
choise for most OPM setups.

We also provide a software tool for simulating any arbitrary light-sheet
imaging system (see Additional information). This software will allow users to
find the PSF, OTF, and light transmission of any light-sheet imaging system as
given in this paper.

## References

  * [1] Fischer, R. S., Wu, Y., Kanchanawong, P., Shroff, H. & Waterman, C. M.  Microscopy in 3d: a biologist’s toolbox.  _Trends in cell biology_ 21, 682–691 (2011).
  * [2] Mappes, T. _et al._ The invention of immersion ultramicroscopy in 1912—the birth of nanotechnology?  _Angewandte Chemie International Edition_ 51, 11208–11212 (2012).
  * [3] Voie, A. H., Burns, D. & Spelman, F.  Orthogonal-plane fluorescence optical sectioning: Three-dimensional imaging of macroscopic biological specimens.  _Journal of microscopy_ 170, 229–236 (1993).
  * [4] Stelzer, E. H. & Lindek, S.  Fundamental reduction of the observation volume in far-field light microscopy by detection orthogonal to the illumination axis: confocal theta microscopy.  _Optics Communications_ 111, 536–547 (1994).
  * [5] Stelzer, E. _et al._ A new tool for the observation of embryos and other large specimens: confocal theta fluorescence microscopy.  _Journal of microscopy_ 179, 1–10 (1995).
  * [6] Chen, B.-C. _et al._ Lattice light-sheet microscopy: imaging molecules to embryos at high spatiotemporal resolution.  _Science_ 346, 1257998 (2014).
  * [7] Huisken, J. & Stainier, D. Y. R.  Selective plane illumination microscopy techniques in developmental biology.  _Development_ 136, 1963–1975, DOI: 10.1242/dev.022426 (2009).
  * [8] Santi, P. A.  Light sheet fluorescence microscopy: a review.  _Journal of Histochemistry & Cytochemistry_ 59, 129–138 (2011).
  * [9] Dunsby, C.  Optically sectioned imaging by oblique plane microscopy.  _Optics express_ 16, 20306–20316 (2008).
  * [10] Bouchard, M. B. _et al._ Swept confocally-aligned planar excitation (scape) microscopy for high-speed volumetric imaging of behaving organisms.  _Nature photonics_ 9, 113–119 (2015).
  * [11] Kumar, M., Kishore, S., Nasenbeny, J., McLean, D. L. & Kozorovitskiy, Y.  Integrated one-and two-photon scanned oblique plane illumination (sopi) microscopy for rapid volumetric imaging.  _Optics express_ 26, 13027–13041 (2018).
  * [12] Sparks, H. _et al._ Dual-view oblique plane microscopy (dopm).  _Biomedical optics express_ 11, 7204–7220 (2020).
  * [13] Glaser, A. K. _et al._ A hybrid open-top light-sheet microscope for versatile multi-scale imaging of cleared tissues.  _Nature Methods_ 19, 613–619 (2022).
  * [14] Yang, B. _et al._ Daxi—high-resolution, large imaging volume and multi-view single-objective light-sheet microscopy.  _Nature methods_ 19, 461–469 (2022).
  * [15] Sapoznik, E. _et al._ A versatile oblique plane microscope for large-scale and high-resolution imaging of subcellular dynamics.  _Elife_ 9, e57681 (2020).
  * [16] Botcherby, E. J., Juškaitis, R., Booth, M. J. & Wilson, T.  An optical technique for remote focusing in microscopy.  _Optics Communications_ 281, 880–887 (2008).
  * [17] Mohanan, S. & Corbett, A. D.  Sensitivity of remote focusing microscopes to magnification mismatch.  _Journal of Microscopy_ 288, 95–105 (2022).
  * [18] Millett-Sikking, A. & York, A.  Andrewgyork/high_na_single_objective_ lightsheet:work-in-progress, DOI: 10.5281/zenodo.3376243 (2019).
  * [19] Yang, B. _et al._ Epi-illumination spim for volumetric imaging with high spatial-temporal resolution.  _Nature methods_ 16, 501–504 (2019).
  * [20] Lakowicz, J. R.  _Principles of fluorescence spectroscopy_ (Springer, 2006).
  * [21] Kim, J., Wang, Y. & Zhang, X.  Calculation of vectorial diffraction in optical systems.  _JOSA A_ 35, 526–535 (2018).
  * [22] Gu, M.  _Advanced optical imaging theory_ , vol. 75 (Springer Science & Business Media, 2000).
  * [23] Debye, P.  Das verhalten von lichtwellen in der n¨ahe eines brennpunktes oder einer brennlinie.  _Annals of Physics_ 335, 755–776 (1909).
  * [24] Leutenegger, M., Rao, R., Leitgeb, R. A. & Lasser, T.  Fast focus field calculations.  _Optics express_ 14, 11277–11291 (2006).
  * [25] Manton, J.  Debye diffraction code.  https://github.com/jdmanton/debye_diffraction_code (2022).

## Author contributions statement

F.S. and J.R.S. conceived the simulations, A.M.S. and J.R.S conducted physical
measurements, and J.R.S. wrote the simulations and analysed the results. All
authors reviewed the manuscript.

## Additional information

Data Availability Simulation software including user interface is found at
https://doi.org/10.18710/YAYNNL.

Competing interests Calico holds IP on the AMS-AGY objective. A.M.S. is an
employee of Calico Life Sciences LLC, and one of the inventors of the AMS-AGY.
The other authors declare that there are no competing interests.

Funding The Research Council of Norway (#314546) and the Centre for Digital
Life Norway.

The corresponding author is responsible for submitting a competing interests
statement on behalf of all authors of the paper. This statement must be
included in the submitted article file.


# Learning long-term music representations via hierarchical contextual
constraints

###### Abstract

Learning symbolic music representations, especially disentangled
representations with probabilistic interpretations, has been shown to benefit
both music understanding and generation. However, most models are only
applicable to short-term music, while learning long-term music representations
remains a challenging task. We have seen several studies attempting to learn
hierarchical representations directly in an end-to-end manner, but these
models have not been able to achieve the desired results and the training
process is not stable. In this paper, we propose a novel approach to learn
long-term symbolic music representations through contextual constraints.
First, we use contrastive learning to pre-train a long-term representation by
constraining its difference from the short-term representation (extracted by
an off-the-shelf model). Then, we fine-tune the long-term representation by a
hierarchical prediction model such that a good long-term representation (e.g.,
an 8-bar representation) can reconstruct the corresponding short-term ones
(e.g., the 2-bar representations within the 8-bar range). Experiments show
that our method stabilizes the training and the fine-tuning steps. In
addition, the designed contextual constraints benefit both reconstruction and
disentanglement, significantly outperforming the baselines.

##  1 Introduction

Deep music representation learning have been proven to be a powerful tool for
high-quality symbolic music generation [1]. The learned representations can be
directly fed into downstream predictive models such as LSTMs[2] and
Transformers[3] to achieve more coherent results than note-based or event-
based generation [4, 5, 6] Furthermore, when a representation learning model
has a probability interpretation, the representation can then be easily
interpolated or resampled to create new music pieces. Recently, several
studies further disentangle music representations into interpretable factors
(such as pitch, rhythm, chord and texture) to achieve a more controllable and
interactive music generation [7, 5, 8]. For example, we can keep the pitch
factor of a melody while resampling its rhythm factor to achieve theme
variation. We can also interpolate the pitch factor for a smooth music
morphing [7].

Despite the above mentioned progress [9, 10, 11], most existing work applies
only to short music segments with a length of several beats, while learning
long-term representations remains a challenging task. In particular, studies
have shown that even for monophonic melodies, "flat" model designs (e.g.,
using long-range sequential encoders) have difficulty remembering a complete
music phrase at once. Some other studies have attempted to solve this problem
by building another layer of hierarchy on top of short-range flat models,
learning short-term and long-term representations simultaneously in an end-to-
end manner [1, 12]. However, as the model expressivity increases with the
number of layers, models also become much more difficult to train.

We argue that the main problem with current methods is the lack of proper
inductive bias, and in this paper we propose a new method for learning long-
term, phrase-level symbolic music representations through contextual
constraints. The method consists of two stages pre-training and fine-tuning,
with two steps in each stage. In the pre-training stage, we first adopt
EC2-VAE [7] to learn bar-level, disentangled latent pitch and rhythm
representations. Then, we apply the same model to learn phrase-level
representation but with contrastive losses to constrain the difference between
phrase-level and bar-level representations. It is indeed difficult to learn
phrase-level representations directly using bar-level models, but the
additional contrastive constraint can serve as a useful inductive bias to help
find a reasonable solution that can subsequently be improved by fine-tuning.
During the fine-tuning stage, we replace the pre-trained decoder with a
hierarchical prediction model that forces the phrase-level representation to
reconstruct the bar-level ones. This is achieved by first tuning only the new
hierarchical decoder (while fixing the pre-trained encoder) and then tuning
the whole network. During these two steps, structured contrastive loss is
applied to stabilize the learning process.

Experiments show that the proposed method significantly outperforms the
baselines and successfully learns disentangled pitch and rhythm
representations for 8-bar long phrases (32 beats in 4/4 meter) without
increasing the latent dimensionality. To our knowledge, this is also the first
generative model that achieves phrase-level composition style transfer, latent
factor interpolation, and theme variation. In sum, our contributions are as
follows:

  * •

We demonstrate the importance of structured contextual constraints in learning
long-term disentangled representations. Our approach only requires reasonable
amount of data to train and could learn compact latent representation.

  * •

We show that the proposed Structured InfoNCE loss effectively expresses the
contextual constraints, stabilizes the training of long-range models and helps
the model converge faster.

  * •

Our model achieves phrase-level music style transfer, latent factor
interpolation, and theme variation.

##  2 Related Work

We review two realms of research related to our work on long-term music-
representation learning: contrastive learning, which is the main method to
stabilize the training process, and hierarchical music modeling, which is
related to our fine-tuning model.

###  2.1 Contrastive Learning

Contrastive learning (CL) is an efficient method in self-supervised
learning[13, 14, 15], serving as regularization to latent representations. For
example, NCE-based contrastive losses[16, 17] have been widely used and
achieved good results in natural language processing. Contrastive predictive
coding (CPC) [18] and Deep Infomax (DIM) [19] explore the relation between
minimizing a contrastive learning loss and maximizing a lower bound of the
mutual information. In DIM, global feature is connected with local feature to
learn more abstract and informative representations.

###  2.2 Hierarchical Music Representation Learning

The hierarchical nature of music has been studied for a long time[20, 21, 22,
23]. Recently, we see some efforts on learning long-term music representations
using hierarchical modeling[24, 25, 12]. The basic idea is that since a flat
model design can only effectively learn shot-term representations, we can
stack more layers on top of the short-term representations module for long-
term representations. Existing works include MusicVAE [1], Music Transformer
VAE [12], Jukebox [26], etc. However, experiments show that unless we have a
huge amount of data, the model is in general very difficult to train. In this
study, we provide a two-stage algorithm with contrastive loss as a better
learning strategy. Also, no model so far has achieved disentanglement for
long-term representation as done in this study.

##  3 Methodology

In this section, we introduce our algorithm in detail. Conceptually, it
consists of two stages, each with two steps.The first stage is pre-training:

  * •

In step 1, we simply adopt EC2-VAE [7], an existing music representation
disentanglement model, to extract short-term pitch and rhythm representations.

  * •

In step 2, we build Long-EC2-VAE, a long-term version of the model and train
it with an extra contextual constraint using the proposed Structured InfoNCE
loss. Intuitively, this loss prevents the learned long-term representations
from deviating too far from corresponding well-trained short-term
representations.

The second stage is fine-tuning, in which we build a hierarchical
representation-learning model by combining the encoder of Long-EC2-VAE with a
hierarchical decoder. We name this model after Hierarchical-EC2-VAE.

  * •

In step 1, we only train the hierarchical decoder to ensure the predictive
power of the long-term representation.

  * •

In step 2, we train the whole hierarchical network for a better long-term
pitch-rhythm disentanglement.

###  3.1 Pre-training by Contrastive Learning

The model of the pre-training stage, Long-EC2-VAE, is shown in Figure 1. It is
built upon an off-the-shelf music representation model, EC2-VAE [7], which can
effectively disentangle pitch and rhythm factors for short music segments by
cutting the latent representation into two parts and pairing one part with a
local rhythm decoder.

Figure 1: The model architecture of Long-EC2-VAE in the pre-training stage,
where the right-hand-side is short-term model with parameter fixed and the
left-hand-side is the long-term model. The dotted lines denote contrastive
losses, whose weighting matrices are joined optimized with the parameters on
the left-hand-side networks. Figure 2: The model architecture of Hierarchical-
EC2-VAE in the fine-tuning stage. The training follows two steps.

In Figure 1, the right-hand-side part is a literal copy of the EC2-VAE encoder
(with parameters fixed) to extract short-term representations, while the left-
hand-side part is a simple adaptation of EC2-VAE for long-term music by
lengthening its temporal receptive field. Note that the left part alone is not
able to learn long-term representations, and our goal is to assist it using
contrastive learning. Formally, the loss function of Long-EC2-VAE is:

$\mathcal{L}=\mathcal{L}_{\rm Long\text{-}EC^{2}\text{-VAE}}+\mathcal{L}_{\rm
Structured\,InfoNCE},$ (1)

where $\mathcal{L}_{\rm Long\text{-}EC^{2}\text{-VAE}}$ is the same as in the
original EC2-VAE model (which contains the KL loss, the rhythm loss and the
overall reconstruction loss). The Structured InfoNCE loss expresses the
contextual constraint. It is developed from InfoNCE [18] loss, and it is
structured since the compared representation pairs are extracted from music
segments of different length, one is long term and the other is short term.
Formally:

$\displaystyle\mathcal{L}_{\text{Structured InfoNCE}}=$ (2)
$\displaystyle-\ln\frac{\exp{\left(z^{T}_{\text{L},f}W\hat{z}^{+}_{\text{
S},f}/\tau\right)}}{\exp{\left(z^{T}_{\text{L},f}W\hat{z}^{+}_{\text{
S},f}/\tau\right)}+\sum_{i=1}^{K}\exp{\left(z^{T}_{\text{L},f}W\hat{z}^{-}_{\text{S},f}/\tau\right)}},$

where $z_{\text{L},f}$ and $W$ are the normalized long-term representations
and weighing matrix we need to learn. $f=\\{\rm{p},\rm{r}\\}$ indicates
whether it is the pitch or rhythm factor. Likewise, we use
$\hat{z}_{\text{S},f}$ to denote the short-term representations extracted by
right-hand-side model. $K$ and $\tau$ are hyper-parameters. $K$ is the amount
negative samples and $\tau$ is the temperature parameter.

In specific, the short-term melodies are half as long as long-term ones. The
positive samples $\hat{z}^{+}_{\text{ S},f}$ are in the cases that the
corresponding short-term melody is a part of the long-term melody and $f$
takes the same value as in ${z}_{\text{L},f}$, while the negative samples are
not in this case. Also, the long-term and short-term representations share the
same dimensionality.

###  3.2 Fine-tuning with Hierarchical Generation

Figure 2 shows the architecture of the fine-tuning model, Hierarchical-
EC2-VAE, where the two subfigures illustrate the two training steps. Here, the
encoder design is the same as in the Long-EC2-VAE model, while the decoder is
a hierarchical predictive model with three layers. The first two layers are
new designed and the last layer is an aggregation of several EC2-VAE decoders
sharing the same parameters. Given the disentangled long-term (phrase-level)
representations, it first decodes intermediate-level representations, then
decodes bar-level representations, and finally reconstructs concrete rhythm
and music tokens.

Compared to the phrase-level representation, the temporal receptive fields of
the intermediate-level representations all shrink to a half, but at the same
time their number doubles in order to cover the same range of music. The same
relationship holds between intermediate and bar-level representations. In
particular, a phrase means 8 bar (in 4/4 meter, 32 beats) in our design, so
that the intermediate-level and bar-level mean 4-bar and 2-bar melody segments
(a length which the original EC2-VAE model can handle), respectively. All
levels of latent representations share the same dimensionality.

In the first step of training (Figure 2(a)), the encoder is a literal copy
from the Long-EC2-VAE model and we only train the hierarchical decoder.
Formally, the loss function is:

$\displaystyle\mathcal{L}_{\rm step1}=\mathcal{L}_{\rm Hierarchical-
EC^{2}\text{-VAE Decoder}}+\mathcal{L}_{\rm InfoNCE},$ (3)

where the first term refers to the reconstruction losses adopted from the
EC2-VAE model, and the second term is defined as:

$\displaystyle\mathcal{L}_{\rm InfoNCE}=$ (4)
$\displaystyle-\ln\frac{\exp{\left(z^{T}_{l,f}W\hat{z}^{+}_{l,f}/\tau\right)}}{\exp{\left(z_{l,f}^{T}W\hat{z}^{+}_{l,f}/\tau\right)}+\sum_{i=1}^{K}\exp{\left(z^{T}_{l,f}W\hat{z}^{-}_{l,f}/\tau\right)}},$

where $z_{l,f}$ are the normalized hierarchical representations we need to
learn with $l=\\{\rm{intermediate},\rm{bar}\\}$ indicating the level of
representation and other notations follow the same meaning as in Eq.(2). Here,
both positive $\hat{z}^{+}_{l,f}$ and negative $\hat{z}^{-}_{l,f}$ samples are
normalized representations computed from a pre-trained EC2-VAE, in which the
positive samples are in the cases that the $\hat{z}^{+}_{l,f}$ and $z_{l,f}$
are computed based on the same music segment and have the same value of $l$
and $f$, while $\hat{z}^{-}_{l,f}$ are not in this case.

After the first step achieves a reasonable accuracy, we proceed to step 2
(Figure 2(b)), unfreezing the encoder and training the whole hierarchical
representation-learning model with:

$\displaystyle\mathcal{L}_{\rm step2}=\mathcal{L}_{\rm step1}$
$\displaystyle+\mathcal{L}_{\rm Structured\,InfoNCE}$ (5)
$\displaystyle+\beta\mathcal{L}_{\rm KL\,phrase},$

where the first two terms are defined in Eq. (3) and Eq. (2) respectively.
$L_{\rm KL\,phrase}$ is KL divergence to only regularize the phrase-level
representations by a normal distribution. The value $\beta$ controls the
degree of KL divergence penalty.

##  4 Experiments

###  4.1 Dataset and data format

We train our model on Nottingham Database[27] and POP909 database [28]. Our
dataset contains 2154 melodies (at song level) in total. We randomly split
these pieces into 2 subsets: $90\%$ pieces for training and for $10\%$ pieces
for test. The data format is designed as the same as in [7] in which 4 bar or
8 bar melodies are formalized as sequences of 130-dimensional one-hot
embedding vectors and 16-beat and 32-beat rhythm pattern is represented by a
sequence of 3-dimensional one-hot embedding vectors. Each vector in the melody
sequence denotes a $\frac{1}{4}$-beat unit. The first 128 dimensions of this
vector denote 128 MIDI-format pitches from $0$ to $127$, the $129^{\rm th}$
dimension is the holding state for longer note duration, and the last
dimension is kept for rest. The three dimensions of rhythm pattern vectors
represent the onset of any pitch, a holding state, and rest, respectively.

###  4.2 Implementation Details

All of our models are trained using Adam optimizer [29] with a scheduled
learning rate from 1e-3 to 1e-5. The batch size is 128 in the pre-training
stage and is 64 in the fine-tuning stage. We do normalization on
representations in Eq.(2) and (4) to make the training process more stable.
The representations fed into decoders are original representation without
normalization.

####  4.2.1 Pre-training

In the pre-training stage, we simply adopt the structure of EC2-VAE [7] to
model 4 bar and 8 bar EC2-VAE. Each model comprises an encoder with a bi-
directional GRU layer, a rhythm decoder with a GRU layer, and a global decoder
with a GRU layer. We set the hidden dimension of the GRU in the encoder and
decoders to 2048. The latent dimension is 128 for disentangled pitch
representations and 128 for disentangled rhythm representations for each range
model. For $\mathcal{L}_{\text{Structured InfoNCE}}$ depicted in Eq. (2), we
set $K$ to 512 and $\tau$ to 1. The positive samples for Eq. (2) and Eq. (4)
are the representations of 1-4th, 3-6th and 5-8th bar from well-trained 4 bar
EC2-VAE. Actually, even when training the 4-bar EC2-VAE (right-hand side of
Figure 1), we use a similar constrastive loss as in Eq. (2) where the positive
samples are representations of 1-2th, 2-3th, 3-4th bar from well-trained 2 bar
(original) EC2-VAE [7] .

####  4.2.2 Fine-tuning

Hierarchical-EC2-VAE model consists of a long-term (8 bar) EC2-VAE encoder, 4
GRU layers, and an aggregation of 2 bar EC2-VAE decoders. We first train the
hierarchical model with fixed 8 bar EC2-VAE encoder from pre-trained stage for
around 25 epochs. Then we train the whole model without fixing parameters. We
set the hidden dimension of 4 GRU layers to 1024. We set $K$ to 256 and $\tau$
to 1 for both Structured InfoNCE loss and InfoNCE loss and set $\beta$ to 0.1
in Eq. (5).

###  4.3 Objective Evaluation

We objectively evaluate the model in terms of reconstruction accuracy,
training stability, and disentanglement.

####  4.3.1 Reconstruction Accuracy

Table 1 shows that the reconstruction accuracy of the proposed models (2nd an
3rd rows) significantly outperform the baseline, a vanilla EC2-VAE applied to
8-bar melody (first row). The last two rows show the results of two ablation
settings of Hierarchical-EC2-VAE: one without the contrastive loss and the
other without first fixing the parameters of encoder and directly train the
model end-to-end. We see that the proposed Structured InfoNCE or InfoNCE
losses play a vital role for an accurate reconstruction and the two-step
training strategy improves the result marginally.

Method | Recon.Acc | Rhythm Recon.Acc
---|---|---
Baseline | 0.772 | 0.847
Long-EC2-VAE | 0.992 | 0.995
H-EC2-VAE(ours) | 0.995 | 0.995
H-EC2-VAE(w/o CL) | 0.584 | 0.599
H-EC2-VAE(w/o fixed) | 0.991 | 0.989
Table 1: A comparison on reconstruction accuracy of different models.

####  4.3.2 Training stability

Figure 3: Experimental results of overall reconstruction and rhythm accuracy
on the test set.

Comparing the accuracy curves of the proposed Long-EC2-VAE with the baseline
as illustrated in Figure 3, we find that the proposed Long-EC2-VAE converges
more quickly during training. This indicates that the proposed training
strategy leads to a better initialization and makes the performance of the
model fluctuate less during training.

####  4.3.3 Disentanglement Evaluation

We evaluate the disentanglement performance of models using a disentanglement
evaluation method adopted in [7] and [30]. The method randomly transposes all
the notes of the input data by $i(i\in[1,12])$ semitones while keeping the
rhythm and underlying chord unchanged and then measures the variation of
disentangled representations.We denote $\Sigma|\Delta z_{\text{p}}|$ and
$\Sigma|\Delta z_{\text{r}}|$ as the variation of $z_{\text{p}}$ and
$z_{\text{r}}$ .

(a) Baseline model (8 bar)

(b) Long-EC2-VAE model (8 bar)

(c) Hirarchical-EC2-VAE model (8 bar)

Figure 4: The comparison between $\Sigma|\Delta z_{\rm p}|$ and $\Sigma|\Delta
z_{\rm r}|$ after transposition. The numbers show the pitch augmented by 12
semitones in each sub-figure from left to right.

As shown in Figure 4, values of $\Sigma|\Delta z_{\text{p}}|$ of the proposed
Hierarchical-EC2-VAE are relatively high while $\Sigma|\Delta z_{\text{r}}|$
maintains in a significantly low level. This indicates that the pitch and
rhythm representations of the proposed Hierarchical-EC2-VAE are well-
disentangled as the change of notes has a tiny impact on $z_{\text{r}}$.
Similarly, we can intuitively find in the figure that the disentanglement
performance of the proposed Hierarchical-EC2-VAE is much better than the
baseline and also outperforms the proposed Long-EC2-VAE.

###  4.4 Music generative examples

In this section, we show some music generation results by manipulating the
disentangled phrase-level pitch and rhythm representations in three different
ways: style transfer via swapping the representation, rhythm morphing via
interpolating the representation, and theme variation via representation
posterior sampling.

####  4.4.1 Phrase-level composition style transfer

We cross-swap the disentangled pitch and rhythm factors $z_{\rm p}$ and
$z_{\rm r}$ of two 8-bar melodies A and B and then obtain generative pieces C
and D. The results are shown in Figure 5, in which we see that both of the two
generative pieces perfectly inherit target rhythm patterns. Besides, these
generative melodies vary slightly from the source melody and these variations
tend to sound creative, i.e. the appearance of embellished notes.

Figure 5: Style transfer examples by hierarchical-EC2-VAE model.

####  4.4.2 Latent $z_{\rm r}$ interpolation

We interpolate rhythm representations $z_{\text{r}}$ of two phrases using
SLERP [31] while keeping the pitch and chord unchanged. The interpolated
latent representations can then be “re-synthesized” using Hierarchical-
EC2-VAE.

Figure 6: Interpolation examples.

As shown in Figure 6, we interpolate $z_{\text{r}}$ of the piece A and B with
different SLERP weights. The results exhibit a surprising sense of coherence
of pitch and rhythm in generative melodies, even in the transition between
consecutive bars.This implies that a longer-term representation is also adept
at modeling short-term generation and even contains more global harmonic
information than a short-term representation.

####  4.4.3 Theme variation

We can also achieve theme variation by adding a Gaussian noise to
$z_{\text{r}}$ while keeping $z_{\text{p}}$ unchanged . As a sample shown in
Figure 7, we find that as the variance of the noise grows larger, the pitch
and rhythm of the generative melody are still reasonable smooth, implying that
the long-term representations contain the coherence of contextual information
and can “control” the generation process.

Figure 7: Rhythm representation posterior sampling examples.

###  4.5 Subjective Evaluation

One may wonder what are the advantages of learning long-term representations
since we can always generate the music bar by bar using short-term models and
just concatenate the generated samples together. One merit lies in the
coherency in controlled music generation. For example, when sampling the long-
term rhythm representation, the overall rhythm pattern of a phrase is
controlled as an organic whole, while individually sampling the rhythm of
different bar may easily lose the rhythm coherency. To better illustrate this
idea,we conduct a survey on theme variation (as in Section 4.4.3) to compare
the performance of the proposed 8-bar Hierarchical-EC2-VAE and baseline 2-bar
EC2-VAE.

####  4.5.1 Survey Configuration

In our survey, each subject is given 5 groups of pieces. Each group contains
three 8 bar pieces: a human-composed piece from Nottingham dataset and 2 theme
variations generated by a 2-bar EC2-VAE and Hierarchical-EC2-VAE,
respectively. In each group, the generated pieces use $z_{\rm p}$ of the
human-composed piece and the sampled $z_{\rm r}$.

Each subject listens to five randomly arranged groups in turn and is required
to rate each melody ranging from 1 (very low) to 5 (very high) according to
three aspects: _creativity_ , _naturalness_ (how human-like the composition
is) and overall _musicality_.

####  4.5.2 Results

A total of 29 subjects ( 18 females and 11 males) participated in the survey.
Experimental results depicted in Figure 8 demonstrate that people prefer
melodies generated by the proposed Hierarchical-EC2-VAE to those generated by
the 2 bar EC2-VAE[7], implying the effects of a long-term coherence learned by
our model. The heights of bars represent means of the ratings and the error
bars represent the MSEs computed via within-subject ANOVA [32]. The results
show that our model performs significantly better than the 2 bar EC2-VAE in
terms of all three dimensions(p < 0.05). Besides, the qualities of melodies
generated by the proposed Hierarchical-EC2-VAE reach a competitive standard
compared to the human-composed pieces, especially in _creativity_.

Figure 8: The results of the subjective evaluation.

##  5 Conclusion

In conclusion, we contribute a pipeline of algorithms to learn long-term and
disentangled music representations. The main novelty lies in the proposed two
inductive biases which constrain the long-term representations using
contextual information. The first one requires long-term representation to be
not too different from the short-term ones which represent a part of the long-
term sequence, and we demonstrate contrastive loss is well-suited for such
rough constraint. The second inductive bias is that a good long-term
representation should be able to reconstruct the corresponding short-term
ones, and we use a hierarchical predictive model to realize this constraint.
Unlike most hierarchical models, our purpose is not prediction for its own
sake, but rather to leverage the prediction power to learn a well-disentangled
long-term representation. Experimental results show that our approach is quite
successful, capable of disentangling pitch and rhythmic factors for phrase-
level (32 beats) melody without increasing the dimensionality of latent
representation compared to bar-level models. Moreover, the learned
representations enable high-quality phrase-level style transfer via
representation swapping and theme variation by representation interpolation
and posterior sampling.

## References

  * [1] A. Roberts, J. H. Engel, C. Raffel, C. Hawthorne, and D. Eck, “A hierarchical latent vector model for learning long-term structure in music,” in _Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden_ , 2018, pp. 4361–4370.
  * [2] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA_ , 2017, pp. 5998–6008.
  * [4] K. Chen, G. Xia, and S. Dubnov, “Continuous melody generation via disentangled short-term representations and structural conditions,” in _14th International Conference on Semantic Computing, San Diego, CA, USA_ , 2020, pp. 128–135.
  * [5] Z. Wang, D. Wang, Y. Zhang, and G. Xia, “Learning interpretable representation for controllable polyphonic music generation,” in _Proceedings of the 21th International Society for Music Information Retrieval Conference, Montreal, Canada_ , 2020, pp. 662–669.
  * [6] A. Pati, A. Lerch, and G. Hadjeres, “Learning to traverse latent spaces for musical score inpainting,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands_ , 2019, pp. 343–351.
  * [7] R. Yang, D. Wang, Z. Wang, T. Chen, J. Jiang, and G. Xia, “Deep music analogy via latent representation disentanglement,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands_ , 2019, pp. 596–603.
  * [8] Y. Huang and Y. Yang, “Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions,” in _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA_ , 2020, pp. 1180–1188.
  * [9] G. Brunner, A. Konrad, Y. Wang, and R. Wattenhofer, “MIDI-VAE: modeling dynamics and instrumentation of music with applications to style transfer,” in _Proceedings of the 19th International Society for Music Information Retrieval Conference, Paris, France_ , 2018, pp. 747–754.
  * [10] T. Akama, “Controlling symbolic music generation based on concept learning from domain knowledge,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands_ , 2019, pp. 816–823.
  * [11] Y. Luo, K. Agres, and D. Herremans, “Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders,” in _Proceedings of the 20th International Society for Music Information Retrieval Conference, Delft, The Netherlands_ , 2019, pp. 746–753.
  * [12] J. Jiang, G. Xia, D. B. Carlton, C. N. Anderson, and R. H. Miyakawa, “Transformer VAE: A hierarchical model for structure-aware and interpretable music representation learning,” in _International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain_ , 2020, pp. 516–520.
  * [13] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proceedings of the 37th International Conference on Machine Learning_ , 2020, pp. 1597–1607.
  * [14] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA_ , 2020, pp. 9726–9735.
  * [15] X. Chen, H. Fan, R. B. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _CoRR_ , vol. abs/2003.04297, 2020.
  * [16] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , ser. JMLR Proceedings, vol. 9, 2010, pp. 297–304.
  * [17] A. Mnih and K. Kavukcuoglu, “Learning word embeddings efficiently with noise-contrastive estimation,” in _Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems, Lake Tahoe, Nevada, United States_ , 2013, pp. 2265–2273.
  * [18] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _CoRR_ , vol. abs/1807.03748, 2018.
  * [19] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” in _7th International Conference on Learning Representations,New Orleans, LA_ , 2019.
  * [20] F. Lerdahl and R. Jackendoff, “A generative theory of tonal music,” _Journal of Aesthetics and Art Criticism_ , vol. 9, no. 1, pp. 72–73, 1996\.
  * [21] W. Rothstein, O. Jonas, and J. Rothgeb, “Introduction to the theory of heinrich schenker: The nature of the musical work of art,” _Journal of Music Theory_ , vol. 27, no. 2, 1983.
  * [22] M. Hamanaka, K. Hirata, and S. Tojo, “Implementing "a generative theory of tonal music",” _Journal of New Music Research_ , vol. 35, no. 4, pp. 249–277, 2006.
  * [23] A. Marsden, “Schenkerian analysis by computer: A proof of concept,” _Journal of New Music Research_ , vol. 39, no. 3, pp. 269–289, 2010.
  * [24] H. H. Tan and D. Herremans, “Music fadernets: Controllable music generation based on high-level features via low-level feature modelling,” in _Proceedings of the 21th International Society for Music Information Retrieval Conference, Montreal, Canada_ , 2020, pp. 109–116.
  * [25] G. Hadjeres and L. Crestel, “Vector quantized contrastive predictive coding for template-based music generation,” _CoRR_ , vol. abs/2004.10120, 2020\.
  * [26] P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” _CoRR_ , vol. abs/2005.00341, 2020.
  * [27] E. Foxley, “Nottingham database,” 2011.
  * [28] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, and G. Xia, “POP909: A pop-song dataset for music arrangement generation,” in _Proceedings of the 21th International Society for Music Information Retrieval Conference, Montreal, Canada_ , 2020, pp. 38–45.
  * [29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [30] H. Kim and A. Mnih, “Disentangling by factorising,” in _Proceedings of the 35th International Conference on Machine Learning_ , 2018, pp. 2654–2663.
  * [31] S. Hollasch, “Advanced animation and rendering techniques: By alan watt and mark watt, acm press,” _Computers & Graphics_, vol. 18, no. 2, p. 249, 1994\.
  * [32] H. Scheffé, “The analysis of variance,” in _Architectural Institute of Japan_ , 1999.