# Transverse-mode coupling effects in scanning cavity microscopy

Julia Benedikter1,2, Thea Moosmayer2, Matthias Mader1,3, Thomas Hümmer1,3,
David Hunger2 1Fakultät für Physik, Ludwig-Maximilians-Universität,
Schellingstraße 4, 80799 München, Germany  2Karlsruher Institut für
Technologie, Physikalisches Institut, Wolfgang-Gaede-Str. 1, 76131 Karlsruhe,
Germany  3Max-Planck-Institut für Quantenoptik, Hans-Kopfermann-Str. 1, 85748
Garching, Germany<EMAIL_ADDRESS>
###### Abstract

Tunable open-access Fabry-Pérot microcavities enable the combination of cavity
enhancement with high resolution imaging. To assess the limits of this
technique originating from background variations, we perform high-finesse
scanning cavity microscopy of pristine planar mirrors. We observe spatially
localized features of strong cavity transmission reduction for certain cavity
mode orders, and periodic background patterns with high spatial frequency. We
show in detailed measurements that the localized structures originate from
resonant transverse-mode coupling and arise from the topography of the planar
mirror surface, in particular its local curvature and gradient. We further
examine the background patterns and find that they derive from non-resonant
mode coupling, and we attribute it to the micro roughness of the mirror. Our
measurements and analysis elucidate the impact of imperfect mirrors and reveal
the influence of their microscopic topography. This is crucial for the
interpretation of scanning cavity images, and could provide relevant insight
for precision applications such as gravitational wave detectors, laser
gyroscopes, and reference cavities.

Keywords: Fabry-Perot resonators, fiber cavities, mode coupling, scanning
cavity microscopy

##  1 Introduction

Optical microcavities are a powerful tool to enhance light-matter interactions
for a variety of applications [1]. With its capability to combine cavity
enhancement with optical microscopy, scanning cavity microscopy (SCM) is
emerging as a powerful technique for single-particle sensing and spectroscopy
of heterogeneous nanomaterials [2, 3, 4], and for the realization of efficient
light-matter interfaces for solid-state quantum emitters [5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16]. The basic functionality derives from a fully tunable,
open-access Fabry-Perot cavity with micron-scale mode size, which is typically
realized by the combination of a micro-machined concave and a macroscopic
planar mirror, the latter carrying the sample to be studied. One of the two
mirrors can be raster-scanned laterally to obtain spatially resolved
measurements. For the example of sensing and spectroscopy, small changes in
the resonant cavity transmission are recorded to obtain information about
sample-induced scattering and absorption [2, 3, 4]. In consequence, any
background variations in transmission originating from the bare cavity alone
need to be as small as possible to enable the resolution of small signals.

In different experiments [2, 4, 12] using various mirror coatings, cavity
geometries, and wavelengths, we consistently observe two classes of artefacts
in SCM images: spatially localized contour lines with deteriorated cavity
performance for certain longitudinal mode orders, and weak periodic background
patterns.

To understand and quantify the effects, we perform extensive SCM measurements
of pristine planar dielectric mirrors with high reflectivity. We find that
both types of artefacts are related to transverse mode coupling, which is
present due to imperfect mirror shapes [17].

Different techniques have been developed to produce concave, near-spherical
profiles as micro-mirror substrates, including CO2 laser machining [18, 8, 19,
20, 21], chemical etching [22, 23], focused ion beam milling [24, 25], and
thermal reflow [26, 27]. While the achieved shape accuracy and surface
roughness is at a remarkable level [19, 28, 29], even nano-scale deviations of
the surface from the ideal spherical shape affect the mode structure. It has
been shown that such imperfect mirror shape and finite mirror size can lead to
loss, mode deformation, and shifted resonance frequencies at particular mirror
separations [17]. The behavior is explained by resonant coupling between
different transverse modes of the cavity and mode-dependent diffraction loss.
The effects can be quantified by a model based on resonant state expansion
[30] that takes the measured mirror profile into account.

Here, we show that mode coupling is also the origin of artefacts in SCM images
due to imperfections of superpolished planar mirrors. In the first class of
artefacts, the localized structures correspond to a spatially varying
resonance condition for transverse mode coupling. It can be traced back to the
surface topography of the planar mirror and its local curvature and gradient,
which both lead to a spatial variation of the effective radius of curvature
the mode samples, either on the nominally planar or on the concave micro-
mirror. For the second class of artefacts, analysis of the background
patterns’ Fourier components reveals non-resonant admixture of specific
higher-order modes to the fundamental cavity mode. Nano-scale roughness of the
planar mirror leads to a spatial variation of the admixture which results in
the observed high spatial frequency patterns.

These artefacts can have significant impact on SCM measurements, and we
discuss ways to avoid the effects and also point towards their possibly
beneficial use.

Figure 1:  (a) Schematic scanning cavity setup showing the machined fiber, the
plane mirror, and the optical mode. (b) Schematic cavity transmission spectrum
showing how the fundamental mode (blue) can become resonant with a higher
order transverse mode (red) of the neighboring longitudinal mode order. (c)
Upper panel: Fiber profile section in $x$ and $y$ direction. Dashed lines:
Parabolic fit. Lower panel: Residuals of the fits. (d) Simulated frequency
detuning with respect to the fundamental mode (black horizontal lines)
normalized to one free spectral range as a function of longitudinal mode order
$q$.Modes are shown for transversal mode orders $\tilde{m}=1$ (red),
$\tilde{m}=2$ (yellow), … , $\tilde{m}=7$ (purple). (e) Resonant cavity
transmission scan of a $100\times 100~{}\mathnormal{\mu m}$ area at $q=15$.
(i) Loss introduced by a nano-scale particle yielding the Gaussian point
spread function of the fundamental cavity mode. (ii) Contour line of resonant
mode coupling. (iii) Periodic background pattern. Black square: scan area of
Fig. 3 (d)-(f).

##  2 Experimental setup and observations

The cavity design studied here is depicted schematically in Fig. 1 (a): the
resonator consists of a macroscopic plane mirror and a concave micromirror on
the end-facet of a single mode optical fiber machined with a CO2 laser. The
super-polished plane mirror substrate (fused silica, rms roughness $<0.2$nm)
is coated with a Bragg mirror for a center wavelength at 780 nm terminated
with a $\lambda/4$ layer of SiO2, yielding a transmission of $T_{1}=60$ ppm
and a combined absorption and scattering loss $L_{1}=20$ ppm. The fiber tip is
coated for the same center wavelength with $T_{2}=12$ ppm and $L_{2}=12$ ppm.
At short mirror separation, a cavity built from the two mirrors achieves a
measured finesse of $\mathcal{F}=60000$, consistent with the above numbers.
Both mirrors are characterized with white light interferometry to obtain the
surface topography with a vertical resolution of $<0.2~{}$nm and a diffraction
limited lateral resolution of $~{}450~{}$nm. Figure 1 (c) shows cuts through
the measured fiber mirror profile along two orthogonal axes together with
parabolic fits, yielding radii of curvature of $r_{c}^{x}=161~{}\mu$m and
$r_{c}^{y}=201~{}\mu$m. The residual shows deviations on a few-nm scale, which
are the origin of transverse-mode coupling effects.

The procedure described in [17] based on resonant state expansion [30] is used
to calculate the mode frequencies $\nu_{qmn}$ of the studied cavity based on
the measured surface profile, where $q$ is the longitudinal, and $m,n$ the
transverse mode number of Hermite-Gaussian modes. Figure 1b) shows
schematically the transverse mode spectrum for a given mirror separation,
indicating the situation where a higher-order mode becomes (near-) degenerate
with a fundamental mode of a neighboring longitudinal mode order. Figure 1 (d)
shows a calculation of the mode resonance spectra as a function of the mirror
separation, where the free spectral range is normalized such that the
fundamental modes appear at fixed frequencies. The different transverse-mode
families $\tilde{m}=m+n$ are color coded, and their sub-modes are non-
degenerate due to the presence of mirror ellipticity [21]. For a significant
fraction of longitudinal modes, the fundamental mode becomes near-resonant
with higher order modes.

In the experiment, the light of a grating-stabilized diode laser at 780 nm is
coupled into the cavity through the fiber, and light transmitted through the
plane mirror is detected with an avalanche photo diode. The laser frequency is
fixed and we modulate the cavity length to record the resonance of the
fundamental mode. We laterally raster-scan the planar mirror with a step size
of 200 nm to record SCM images of the resonant cavity transmission for the
fundamental mode at different longitudinal mode orders $q$. Figure 1 (e) shows
a measurement on a mostly clean mirror that shows three types of structures:
(i) circularly symmetric Gaussian shaped features, (ii) often closed, sharp
contour lines, and (iii) a periodic background pattern. The Gaussian shapes
(i) originate from loss introduced by contaminants with a size much smaller
than the cavity mode, such that the feature shape is given by the intensity
distribution of the fundamental cavity mode as long as the introduced loss is
small. This typically represents the signal of interest e.g. for cavity-
enhanced extinction microscopy, and the mode size directly yields the spatial
resolution of the measurement. Structures (ii) and (iii) are the focus of this
study. It is apparent that they are much more localized than (i) and thus
contain much higher spatial frequencies than the fundamental mode, such that
they cannot originate from the fundamental mode alone. We proceed by
discussing the artefacts (ii) and (iii) in detail.

Figure 2:  (a) - (d) Resonant cavity transmission scans for mode orders
$q=11,34,132,158$. The transmission is normalized to the peak value. (e)
Cavity finesse as a function of longitudinal mode order, deduced from taking
the predominant (solid red) and the average (light red) value of each
transmission scan and converted to finesse knowing the mirror losses. Black
dots: mode orders of scans (a) to (d). Grey dots: mode orders of scans (b) to
(e) in Fig. 6. (f) Overlay of transmission scans of longitudinal mode orders
$q=11,12,14,16,22,24,28,38$ showing isocontours of the penetration depth into
the dielectric layer stack of the mirror. Arbitrary units. Color scale: see
(a).

##  3 Transverse-mode resonance isocontours

To obtain a more detailed understanding of the properties of the contour lines
(ii), we perform analogous measurements as shown in Figure 1 (e) on a defect-
free, $30\times 30~{}\mu$m area of a pristine planar mirror for all accessible
longitudinal mode orders $q$ of the cavity. The complete data set is shown in
the supplement. Figure 2 (e) shows the evaluated predominant (solid red) or
the average (light red) finesse value as a function of $q$ from this dataset.
Specific mode orders show a reduced finesse, which in the latter case is
related to the fraction of the area that is affected by the contour lines.
Figures 2 (a-d) show example measurements at $q=(11,34,132,158)$, which are
also indicated in Fig. 2 (e).

It can be observed that the shape of certain contours is very similar across
different measurements, and that the width of the contour lines increases
continuously with increasing $q$. Figure 2 (f) shows the superposition of
measurements taken at mode orders $q=11,12,14,16,22,24,28,38$. Remarkably, the
individual measurements combine to a consistent isocontour map.

Figure 3:  (a) SCM transmission measurement at $q=132$ with a close up higher
resolution scan. Pixel size: 0.5 and 0.2 µm. The black line indicates a
$5.4~{}\mu$m long path along which spectra shown in (b) are taken. (b) Cavity
transmission spectra around the fundamental mode for different lateral
positions on the path in (a) showing an avoided crossing with mode $(6,0)$.
The minimal frequency separation is found at the darkest spot on the path in
(a). (c) Maximal amplitude of the spectra in (b) (blue) and frequency
separation $\Delta\nu_{00,60}$ of the two modes $(0,0)$ and $(6,0)$ along the
path. (d),(e) Transmission scans at $q=15$. Changing the wavelength from
$\lambda=780.1400~{}$nm (d) to $\lambda=780.2328~{}$nm (e) increases the size
of the contour. (f) Scans (d) and (e) added up for comparison.

###  3.1 Analysis

A closer look at the cavity resonances in the surrounding of a contour line
shows that mode degeneracy occurs at these spatial locations. In Figure 3 (a)
we show an example transmission map taken at $q=132$. We follow the
fundamental cavity resonance across a linear path perpendicular to a contour
line, indicated in the inset. Figure 3 (b) shows cavity transmission spectra
for each position on the mirror. We observe that in addition to the
fundamental mode TEM00, a higher order mode TEM60 is visible, whose frequency
detuning and strength changes with the lateral mirror position in the manner
of an avoided level crossing. The frequency axis of the data is centered on
the average frequency of the two modes to make this apparent. The higher order
mode can be assigned unambiguously from the knowledge of $q$ and the mode
frequency simulation shown in Figure 1, as well as from counting the higher-
order modes in the cavity transmission spectrum. At the location of the
contour line, the mode separation is minimal, and the two modes have equal
amplitude. Figure 3 (c) shows the quantitative evaluation of the data in (b),
yielding a variation of the frequency separation $\Delta\nu_{00,60}\approx\pm
1~{}$GHz or equivalently $\pm 20$ cavity linewidths. This evidences that the
contour lines result from a spatially varying transverse-mode resonance, where
the minimal mode separation on the contour line directly yields the inter-mode
coupling strength. This observation implies that the transverse modes
experience different frequency shifts on different locations on the mirror.

The origin of the differential frequency shift can be constrained by
considering the expression for the resonance frequency for a mode with
longitudinal ($q$) and transverse $(m,n)$ order,

$\nu_{qmn}=\nu_{\mathrm{FSR}}\left(q+\frac{m+1/2}{\pi}\zeta^{x}+\frac{n+1/2}{\pi}\zeta^{y}\right),$
(1)

where $\nu_{\mathrm{FSR}}=c/(2d)$ is the free spectral range,

$\zeta^{x,y}=\arccos\sqrt{\left(1-\frac{d}{r_{c}^{x,y}}\right)\left(1-\frac{d}{r_{c,\mathrm{pl}}^{x,y}}\right)}$
(2)

the Gouy phase, $d=q\frac{\lambda}{2}+d_{\mathrm{pen}}$ is the effective
cavity length, $r_{c}^{x,y}$ is the fiber mirror radius of curvature along the
profile’s major axes ($x,y$), $r_{c,\mathrm{pl}}^{x,y}$ the local radius of
curvature of the nominally planar mirror, and
$d_{\mathrm{pen}}=d_{\mathrm{pen}}^{\mathrm{fiber}}+d_{\mathrm{pen}}^{\mathrm{plan}}$
the optical field penetration depth into the dielectric mirror. For a quarter-
wave Bragg mirror at the design wavelength as used for the fiber here, it is
given by

$d_{pen}^{H}=\frac{\lambda}{4\left(n_{H}-n_{L}\right)},$ (3)

while for the low-refractive index terminated planar mirror, the field
penetration is larger according to [31]

$d_{pen}^{L}=d_{pen}^{H}\bar{n}^{2},$ (4)

with $\bar{n}=2(1/n_{L}+1/n_{H})^{-1}$. One can see that the mode frequency
difference

$\Delta\nu=\nu_{qmn}-\nu_{(q+1)00}=\nu_{\mathrm{FSR}}\left[\frac{m}{\pi}\zeta^{x}+\frac{n}{\pi}\zeta^{y}-1\right]$
(5)

of a mode pair with a given set of $(q,m,n)$, can be varied by a change of
$q$, the laser wavelength $\lambda$, the effective mirror radii of curvature
$r_{c}^{x,y},r_{c,\mathrm{pl}}^{x,y}$ the modes experience [17], and a change
of the field penetration depth $d_{pen}$, which itself depends on the layer
thickness and the refractive indices $n_{H},n_{L}$. Note that the free
spectral range $\nu_{\mathrm{FSR}}=c/(2d)$ depends on the penetration depth,
such that the given expression for $\Delta\nu$ is an approximation in the case
of varying $d_{pen}$.

###  3.2 Spatial width of mode resonances

The observed overall behavior of an increasing width of spatial mode
resonances for increasing $q$ as seen in Fig. 2 (a-d) can be explained by the
dispersion of the modes involved. From Equations 1, 5 and the condition of
degeneracy, one can estimate the mode order leading to degeneracy at a given
mirror separation by

$\tilde{m}\approx\frac{\pi}{\mathrm{arccos}\sqrt{1-d/r_{c}}}.$ (6)

More accurately, the mode admixture to the fundamental mode can be calculated
with the simulation of the full cavity mode structure [17]. Figure 6 (a) shows
the contributions of transverse modes to the fundamental mode as a function of
the longitudinal mode order. For short cavities, transverse modes with large
$\tilde{m}$ become degenerate with the fundamental mode. These modes have a
dispersion that differs largely from the fundamental mode and lead to a mode
crossing under a large angle $\propto\tilde{m}$, see Fig. 1(d). This means
that small variations lead to a large frequency detuning between the modes,
restricting resonant mixing to small spatial regions and thus the observed
sharp features. For long cavities, in contrast, the transverse-mode spacing is
much larger, and modes with small $\tilde{m}$ become resonant with the
fundamental mode. The difference between the modes’ dispersion
$\propto\tilde{m}$ is consequently smaller such that the resonant coupling
region is large, consistent with the much broader features observed for longer
cavities.

###  3.3 Wavelength dependence

Next, we can directly evidence the dependence of the mode degeneracy location
on $\lambda$ by taking SCM images at different laser wavelengths. Figure 3
(d,e) show two examples where we measure at $q=15$ and change the wavelength
from $\lambda=780.1400~{}$nm in (d) to $\lambda=780.2328~{}$nm in (e), which
changes the size of the mode degeneracy contour by about its spatial width. A
wavelength change can thus globally change the observed contour pattern and
could be used to map out contours across the entire mirror, but is not the
origin of the structure. For all other measurements we have ensured a stable
laser wavelength.

###  3.4 Variation of local surface gradients

We now consider a spatial change in the effective radius of curvature of the
fiber mirror $r_{c}$ as a possible origin of the spatial variation of the mode
degeneracy condition. Unevenness of the planar mirror leads to a spatially
varying orientation of the cavity’s optical axis, such that the cavity mode
samples different areas on the micro mirror. Since the mirror shape is non-
spherical, this translates into a change in $r_{c}$. Figure 4 (b) shows this
situation schematically. To analyze the effect, we have taken interferometric
images of the planar mirror topography on several locations to infer the
typical variation of surface inclination. Figure 4 (a) shows an example of a
measured surface profile $h(x,y)$, and (c) shows the calculated modulus of the
local surface angle $\alpha=\arctan(\sqrt{(\Delta h/\Delta x)^{2}+(\Delta
h/\Delta y)^{2}})$ on the same area after smoothing the data with a Gaussian
filter of a $4.2~{}\mu$m $1/e^{2}$ radius that matches the mode radius $w_{0}$
on the mirror. We observe typical height variations of up to $\Delta h\sim 1$
nm on length scales of $\Delta x\sim 10~{}\mu$m, yielding angles of up to
$\alpha=0.005^{\circ}$. Assuming that the cavity mode will form on the curved
mirror where the two local surface angles match, we can calculate the change
in the effective radius of curvature e.g. for a Gaussian profile
$h(x)=-t\exp[-x^{2}/a^{2}]$ by solving for $x_{\alpha}$ for which
$\theta=\arctan(\mathrm{d}h(x)/\mathrm{d}x)|_{x_{\alpha}}=\alpha$ and
evaluating
$r_{c}(x_{\alpha})=[1+(\mathrm{d}h/\mathrm{d}x)^{2}]^{3/2}/(\mathrm{d}^{2}h/\mathrm{d}x^{2})$.
For a perfectly aligned cavity, this leads to relative changes in $r_{c}$ of
$\Delta r_{c}/r_{c}\approx 10^{-6}$. Figure 4 (d) shows a calculation of the
resulting frequency shift $\Delta\nu_{00,60}$ as a function of $\alpha$
(yellow solid line), which is too small to explain the observed frequency
shifts shown in Figure 3 (c) (indicated by dashed gray line in Fig. 4 (d)).
However, when the cavity is slightly misaligned, the change in $r_{c}$ with
$x$ is much larger, and for an initial misalignment of $1.5^{\circ}$, the
calculation shows that the observed range of frequency shifts is expected for
the measured range of surface angles $\alpha$, see blue line in Fig. 4 (d).
Misalignments of this order of magnitude are reasonable for the experiment
reported here. This effect is thus a relevant contribution to the observations
and can also be dominating.

Figure 4:  (a) White light interferometric measurement of the mirror surface.
(b) Schematic illustration of the effect of local gradients of the planar
mirror. (c) Calculated modulus of the local surface angle and (e) average
curvature of the area shown in (a). (d) Change of $\Delta\nu_{00,60}$ as a
function of surface angle at $q=132$ in units of $\kappa$. Yellow solid line:
Frequency shift due to changing $r_{c}$ for an aligned cavity. Yellow dashed
line: Shift due to changing $r_{c}$ and changing penetration depth. Blue line:
Frequency shift due to $r_{c}$ for an initial misalignment of
$\alpha_{0}=1.5^{\circ}$. Grey dashed line: Observed maximal shift. (f)
Calculated change of transverse-mode frequency separation $\Delta\nu_{00,60}$
as a function of the maximal local curvature in units of $\kappa$ over the
range of values observed in (a). Grey dashed line: Maximal shift observed in
Fig. 3.

###  3.5 Variation of field penetration

In principle, an additional contribution could arise from a spatial variation
of the field penetration depth on the planar mirror. From the observed
variation of $\Delta\nu$ we can quantify the necessary spatial variation of
either the refractive index contrast or the layer thickness (or a combination
of both). The observed $\Delta\nu\approx\pm 1~{}$GHz corresponds to a change
in the penetration depth of $28$ nm, which is about $3\%$ of the value for
$d_{pen}^{L}$ of the planar mirror. This could be caused by a variation of the
layer thickness by $3\%$, or alternatively by a change in the refractive index
contrast by $7\times 10^{-3}$. Both values are at least an order of magnitude
too large to be realistic.

We also consider a variation of field penetration at the fiber mirror due to
local gradients at the planar mirror: Due to the partial directionality of the
coating process [29], the coating layer thickness will depend on the local
surface orientation, and the coating on the fiber will be thinner
$\propto\cos(\theta+\theta_{0})/\cos(\theta_{0})$ at the inclined parts of the
curved mirror. Here, $\theta_{0}$ is the deposition direction. For our fiber
geometry, a maximal thickness variation up to $\pm 3\%$ is expected for e.g. a
fully directional deposition under $\theta_{0}=30^{\circ}$. Figure 4 (d)
(yellow dashed line) shows that this effect is comparable in magnitude to the
one due to the changing $r_{c}$ for a perfectly aligned cavity. However, the
thickness variation is almost independent of $\alpha_{0}$ and thus does not
contribute significantly for a tilted cavity.

###  3.6 Variation of local curvature

We now turn to the influence of the local curvature of the planar mirror due
to surface imperfections. Figure 4 (e) shows the calculated average curvature
on the area of Figure 4 (a). Therefore, we fit two-dimensional parabolas to
each datapoint over an $8\mu$m area and take the average of the maximal and
minimal curvature found along orthogonal eigenaxes. We find values that
correspond to radii of curvature $r_{c,\mathrm{pl}}$ as small as 70 mm, which
thus significantly affects the Gouy phase. The corresponding shift in mode
frequency difference $\Delta\nu$ is shown in Figure 4 (f) as a function of
maximal curvature. The values obtained from the surface curvature are
comparable and even larger than the observed frequency variations. This
suggests that this is a dominating effect in our experiments.

Figure 5:  (a) White light interferometric height measurement of a laser-
induced buckle created on the planar mirror. (b) Cut through the data shown in
(a) (black datapoints) together with a low-pass filtered copy. (c) Local
curvature ($1/r_{c,\textrm{pl}}$) calculated from the low-pass filtered
topography. (d) SCM measurement of an area with several buckles. (e) Cavity
transmission as a function of the cavity length for different points across a
buckle. (f) Height profile (black datapoints) evaluated from the position of
the fundamental mode in (e). Red solid line: Interpolated and smoothed
profile. (g) Measured Gouy phase difference to the fundamental mode for the
TEM01 (black datapoints) and TEM01 mode (red datapoints). Solid lines:
Calculated

To further support this picture, we have prepared a planar mirror with a
microscopic surface profile that is much taller than the imperfections of the
bare superpolished mirror, and that can be clearly identified with SCM
measurements. Therefore we irradiate the mirror with a pulse of a focused CO2
laser, leading to local delamination of the coating from the substrate and the
formation of a dome. Figure 5 (a) shows a white light interferometric image of
the structure together with a cut (b) and the calculated local curvature (c).
The peak curvature is more than two orders of magnitude larger than on the
pristine mirror, and its effect on the Gouy phase is thus unambiguously
observable. We perform SCM measurements of such profiles, see Figure 5 (d),
and observe circular isocontours of low cavity transmission as expected for
mode mixing resonances. For these experiments, we use a different cavity
operating at 860 nm with a fiber with $r_{c}^{x,y}=(280,295)~{}\mu$m. We use
the measured relative cavity length at which the fundamental cavity mode
appears to resolve the height profile of the dome. Figure 5 (e) shows
transmission spectra taken along a linear path across one dome, and Figure 5
(f) shows the evaluated height. Compared to Figure 5 b) the profile is less
sharp, consistent with the expected convolution of the cavity mode with the
profile. In addition, we obtain the position of the modes TEM01 and TEM10 and
calculate their repsective Gouy phase difference relative to the fundamental
mode from the cavity length difference, see datapoints in Figure 5 (g). We
observe an increase of $\zeta$ at the outer part of the profile and a
reduction in the center, which fits well to the positive (concave) and
negative (convex) curvature at the respective locations. In contrast, only a
small influence of the surface gradient is observable, which would lead to an
asymmetry of the Gouy phase profiles. We can also calculate the expected Gouy
phase from the measured height profile and its curvature. The solid lines in
Figure 5 (g) are the results of the calculation with $r_{c}^{x,y}$ taken as
free fit parameters. We obtain $r_{c}^{x}=275~{}\mu$m and
$r_{c}^{y}=380~{}\mu$m. While $r_{c}^{x}$ fits well to the measured value for
this fiber, $r_{c}^{y}$ is larger, consistent with an angular misalignment of
the fiber along this direction. This misalignment is also apparent from the
different height of the TEM01 and TEM10 which originates from mode matching.
The difference of the frequency variation of the two modes can be traced back
to the orientation of the modes and the surface curvature. While the TEM01
mode is oriented approximately along the scan direction and thus experiences
both positive and negative curvature along its axis, the orthogonally oriented
TEM10 mode does to first order not experience a positive curvature and is only
affected by the isotropic negative curvature at the dome’s center.

Overall, the good matching of the measured and calculated Gouy phase across
the dome confirms our interpretation, and we identify local curvature as the
dominant effect also for pristine planar mirrors.

##  4 Periodic background patterns

The second type of artefacts observed in SCM measurements are periodic
patterns in the resonant cavity transmission. In Fig. 1(e), such a pattern
(iii) is apparent over the entire measurement area. It contains higher spatial
frequencies than contained by the fundamental mode, giving first evidence for
the relevance of higher order transverse modes. Depending on $q$, the contrast
of such patterns ranges between $<1\%$ and $30\%$, indicating that non-
resonant coupling is present. This is further confirmed by the observed
insensitivity of the pattern on wavelength changes. Even for a wavelength
change of more than 30 nm, the pattern does not change significantly.

We investigate the background patterns for different $q$ and analyze their
spatial frequency spectrum. Figures 6 (b)-(e) show SCM transmission
measurements and corresponding two-dimensional (2D) spatial Fourier spectra.
White circles indicate the spatial and spectral extent of the fundamental
mode. The patterns show an orientation along the principal axes of the
Hermite-Gaussian modes of the cavity, which are defined by the profile
ellipticity of the fiber mirror. The Fourier spectra show localized maxima at
high frequency, where the $q$ dependence of the observed frequency can be
explained by the mode degeneracy condition approximated by eq. 6 and the
respective spatial frequency spectrum of the dominant TEMm,n mode. A Hermite-
Gaussian mode contains spatial frequencies up to $f_{m}=f_{0}\sqrt{m}$, where
$f_{0}$ is the $1/e^{2}$ frequency of the fundamental mode. This results from
the increase of the number of transverse field nodes $\propto m$ and the
increase in mode size $\propto\sqrt{m+1}$.

Figure 6:  (a) Contributions $c_{nm}$ of Hermite-Gaussian modes to the
fundamental mode shown as $\log\left|c_{nm}\right|^{2}$ for different
longitudinal mode orders. Each horizontal box separated by grey lines
corresponds to $m+n+1$ modes of order $m+n$ ranging from $(0,m+n)$ to
$(m+n,0)$. (b) - (d) Lower panels: Transmission scans showing checker board
and stripe background artefacts at different longitudinal mode orders. White
circles indicate the $1/e^{2}$-radii of the Gaussian modes at the respective
cavity length. Upper panels: Spatial Fourier transforms. White circles:
$1/e^{2}$-radii of Fourier-transformed Gaussian modes. Dashed lines: Assumed
principal axes of Hermite-Gaussian modes. (e) (i),(ii) Same as in (b) - (d)
for $q=35$. (iv) H.-G. mode TEM9,1 with its Fourier transform (iii). (vi)
Convolution of normally distributed random numbers with mode TEM9,1 and
Fourier transform of the image (v). (f) Radial integration of Fourier
transformed scans in (e) and (c) (green) together with the Gaussian mode
(yellow) and a sum of modes of the order, which is contributing most to the
ground mode: $\sum_{i=0}^{\tilde{m}}(i,m+n-i)$ (blue), where $\tilde{m}=10$
for $q=35$ and $\tilde{m}=5$ for $q=147$.

Figure 6 (f) shows a radial integration of the 2D Fourier spectrum and
compares it with the spectrum of the Gaussian fundamental mode and the
spectrum of the mode family that dominantly contributes to mode mixing as
identified from the simulation shown in Fig. 6 (a). While the observed low
frequency part matches well to the expected Gaussian mode spectrum for the
particular longitudinal mode order, additional high frequency components agree
with the frequencies of dominantly admixing modes. In particular, the
localized peaks observed in the 2D Fourier spectra occur at the peak frequency
of the most strongly admixing higher order mode. Components at higher
frequency do not show any structure and can be attributed to uncorrelated
measurement noise. The overall scaling of observed and expected frequencies
with $q$ can be clearly seen from the comparison of spectra for $q=35$ and
$q=147$ (top and bottom panel of Fig. 6 f).

It is thus clear that a spatially varying non-resonant admixture of a few
dominant higher-order modes leads to the observed patterns. We propose that
the variation of the mode admixture originates from surface micro-roughness on
the planar mirror. To support this hypothesis, we compare a measured structure
at $q=35$ and its Fourier transform with the pattern formed by convoluting
Gaussian white noise with the expected dominant admixing mode TEM9,1 as
obtained from the simulation shown in Figure 6 (a). Figure 6 (e) shows the
measured transmission map, the TEM9,1 mode, and the convolution of the TEM9,1
mode with noise in the lower row, and the corresponding 2D Fourier spectra in
the upper row. A similar texture with comparable length scales can be
identified. However, the simulated Fourier spectrum shows more frequency
components at intermediate frequencies $f_{m}>f>f_{0}$. The difference could
be a sign for destructive interference with additionally contributing modes,
which are indeed expected to be relevant from the simulation of admixing
modes.

##  5 Conclusion

In summary, we have analyzed artefacts that regularly appear in scanning
cavity microscopy and found consistent explanations that can describe the
observations. We have identified two types of effects that can be traced back
to variations of the local gradient and curvature of the surface topography
that leads to resonant mode coupling, and to microscopic surface roughness
which modulates the weak, non-resonant admixture of higher order modes. Both
effects arise from transverse-mode coupling, which is significant due to
imperfect shapes of both mirrors. In particular, even for superpolished
mirrors, surface imperfections lead to a variation of the effective local
radius of curvature that causes significant relative shifts of mode
frequencies.

For quantitative measurements of e.g. the extinction contrast, the artefacts
limit the achievable sensitivity [2]. The observed effects are also relevant
for cavity QED experiments, in particular with solid state emitters. E.g. for
the case of color centers in diamond, where thin diamond membranes are
integrated into open-access cavities [9, 13, 14, 32], mode coupling due to
imperfect membrane surfaces can additionally affect the mode structure and
limit the operation conditions for emitter-cavity coupling experiments.

In general, this coupling can be reduced by minimizing deviations of the
planar and curved mirror’s surface profile from the respective ideal shape.
Using e.g. multi-shot laser machining [28] or precise Focused-Ion-Beam milling
[29] potentially with CO2 post-treatment [33], concave profiles can be
optimized to minimize mode coupling. In addition, longitudinal mode orders
with weak mode coupling can be selected to fully avoid effect (ii) and largely
suppress effect (iii). Also, our calculations indicate that for a well aligned
cavity, the effect of local surface gradients on the planar mirror can be
negligible. Finally, by using differential techniques e.g. with different
wavelengths or with broadband illumination of the cavity, mode mixing can be
subtracted or averaged out very efficiently.

In contrast, the amplifying effect of resonant mode coupling could be
harnessed to sensitively characterize super mirrors for precision
applications. E.g. smallest surface distortions and scattering could be
analyzed with high spatial resolution, enabling new insight for challenging
applications such as laser gyroscopes, high-finesse reference cavities, or
gravitational wave detectors.

## Acknowledgments

This work was partially funded by the European Union H2020 research and
innovation programme under grant agreement No. 712721 (NanOQTech), and the DFG
Cluster of Excellence Nanosystems Initiative Munich. We thank Theodor W.
Hänsch for helpful discussions.

## Bibliography

## References

  * [1] Vahala, K. J.  Optical microcavities.  _Nature_ 424, 839–846 (2003).
  * [2] Mader, M., Reichel, J., Hänsch, T. W. & Hunger, D.  A scanning cavity microscope.  _Nature Communications_ 6, 7249 (2015).
  * [3] Kelkar, H. _et al._ Sensing Nanoparticles with a Cantilever-Based Scannable Optical Cavity of Low Finesse and Sub-${\lambda}^{3}$ Volume.  _Phys. Rev. Applied_ 4, 054010 (2015).
  * [4] Hümmer, T. _et al._ Cavity-enhanced Raman microscopy of individual carbon nanotubes.  _Nature Communications_ 7, 12155 (2016).
  * [5] Toninelli, C. _et al._ A scanning microcavity for in situ control of single-molecule emission.  _Appl. Phys. Lett._ 97, 021107 (2010).
  * [6] Kaupp, H. _et al._ Scaling laws of the cavity enhancement for nitrogen-vacancy centers in diamond.  _Phys. Rev. A_ 88, 053812 (2013).
  * [7] Albrecht, R., Bommer, A., Deutsch, C., Reichel, J. & Becher, C.  Coupling of a single NV-center in diamond to a fiber-based microcavity.  _Phys. Rev. Lett._ 110, 243602 (2013).
  * [8] Greuter, L. _et al._ A small volume tunable microcavity: Development and characterization.  _Appl. Phys. Lett._ 105, 121105 (2014).
  * [9] Janitz, E. _et al._ Fabry-Perot microcavity for diamond-based photonics.  _Phys. Rev. A_ 92, 043844 (2015).
  * [10] Johnson, S. _et al._ Tunable cavity coupling of the zero phonon line of a nitrogen-vacancy defect in diamond.  _New Journal of Physics_ 17, 122003 (2015).
  * [11] Kaupp, H. _et al._ Purcell-Enhanced Single-Photon Emission from Nitrogen-Vacancy Centers Coupled to a Tunable Microcavity.  _Phys. Rev. Applied_ 6, 054010 (2016).
  * [12] Benedikter, J. _et al._ Cavity-Enhanced Single-Photon Source Based on the Silicon-Vacancy Center in Diamond.  _Phys. Rev. Applied_ 7, 024031 (2017).
  * [13] Riedel, D. _et al._ Deterministic Enhancement of Coherent Photon Generation from a Nitrogen-Vacancy Center in Ultrapure Diamond.  _Phys. Rev. X_ 7, 031040 (2017).
  * [14] Bogdanović, S. _et al._ Design and low-temperature characterization of a tunable microcavity for diamond-based quantum networks.  _Appl. Phys. Lett._ 110, 171103 (2017).
  * [15] Wang, D. _et al._ Coherent Coupling of a Single Molecule to a Scanning Fabry-Perot Microcavity.  _Phys. Rev. X_ 7, 021014 (2017).
  * [16] Casabone, B. _et al._ Cavity-enhanced spectroscopy of a few-ion ensemble in Eu3+:Y2O3.  _New Journal of Physics_ 20, 095006 (2018).
  * [17] Benedikter, J. _et al._ Transverse-mode mixing and diffraction loss in tunable Fabry-Perot microcavities.  _New J. Phys._ 17, 053051 (2015).
  * [18] Hunger, D., Deutsch, C., Barbour, R. J., Warburton, R. J. & Reichel, J.  Laser micro-fabrication of concave, low-roughness features in silica.  _AIP Advances_ 2, 012119 (2012).
  * [19] Takahashi, H. _et al._ Novel laser machining of optical fibers for long cavities with low birefringence.  _Optics Express_ 22, 31317 (2014).
  * [20] Muller, A., Flagg, E. B., Metcalfe, M., Lawall, J. & Solomon, G. S.  Coupling an epitaxial quantum dot to a fiber-based external-mirror microcavity.  _Appl. Phys. Lett._ 95, 173101 (2009).
  * [21] Uphoff, M., Brekenfeld, M., Rempe, G. & Ritter, S.  Frequency splitting of polarization eigenmodes in microscopic Fabry-Perot cavities.  _New J. Phys._ 17, 013053 (2015).
  * [22] Trupke, M. _et al._ Microfabricated high-finesse optical cavity with open access and small mode volume.  _Appl. Phys. Lett._ 87, 211106 (2005).
  * [23] Biedermann, G. W. _et al._ Ultrasmooth microfabricated mirrors for quantum information.  _Appl. Phys. Lett._ 97, 181110 (2010).
  * [24] Dolan, P. R., Hughes, G. M., Grazioso, F., Patton, B. R. & Smith, J. M.  Femtoliter tunable optical cavity arrays.  _Optics Letters_ 35, 3556 (2010).
  * [25] Albrecht, R. _et al._ Narrow-band single photon emission at room temperature based on a single nitrogen-vacancy center coupled to an all-fiber-cavity.  _Appl. Phys. Lett._ 105, 073113 (2014).
  * [26] Cui, G., Hannigan, J. M., Loeckenhoff, R., Matinaga, F. M. & Raymer, M. G.  A hemispherical, high-solid-angle optical microcavity for cavity QED studies.  _Optics Express_ 14, 2289 (2006).
  * [27] Roy, A. & Barrett, M. D.  Fabrication of glass micro-cavities for cavity quantum electrodynamics experiments.  _Appl. Phys. Lett._ 99, 171112 (2011).
  * [28] Ott, K. _et al._ Millimeter-long fiber Fabry-Perot cavities.  _Optics Express_ 29, 9839 (2016).
  * [29] Trichet, A. A. P., Dolan, P. R., Coles, D. M., Hughes, G. M. & Smith, J. M.  Topographic control of open-access microcavities at the nanometer scale.  _Optics Express_ 23, 17205 (2015).
  * [30] Kleckner, D., Irvine, W. T. M., Oemrawsingh, S. S. R. & Bouwmeester, D.  Diffraction-limited high-finesse optical cavities.  _Phys. Rev. A_ 81, 043814 (2010).
  * [31] Brovelli, L. R. & Keller, U.  Simple analytical expressions for the reflectivity and the penetration depth of a Bragg mirror between arbitrary media.  _Optics Communications_ 116, 343–350 (1995).
  * [32] Häußler, S. _et al._ A Diamond-Photonics Platform Based on Silicon-Vacancy Centers in a Single Crystal Diamond Membrane and a Fiber-Cavity 1812.02426 (2018).
  * [33] Walker, B. T. _et al._ Driven-dissipative non-equilibrium Bose–Einstein condensation of less than ten photons.  _Nature Physics_ 14, 1173–1177 (2018).


# Constructions Are So Difficult That Even Large Language Models Get Them
Right for the Wrong Reasons

###### Abstract

In this paper, we make a contribution that can be understood from two
perspectives: from an NLP perspective, we introduce a small challenge dataset
for NLI with large lexical overlap, which minimises the possibility of models
discerning entailment solely based on token distinctions, and show that GPT-4
and Llama 2 fail it with strong bias. We then create further challenging sub-
tasks in an effort to explain this failure. From a Computational Linguistics
perspective, we identify a group of constructions with three classes of
adjectives which cannot be distinguished by surface features. This enables us
to probe for LLM’s understanding of these constructions in various ways, and
we find that they fail in a variety of ways to distinguish between them,
suggesting that they don’t adequately represent their meaning or capture the
lexical properties of phrasal heads.

Keywords: LLMs, construction grammar, semantics, natural language inference,
prompting

Constructions Are So Difficult That Even Large Language Models Get Them Right
for the Wrong Reasons

Shijia Zhou1, Leonie Weissweiler1,3, Taiqi He2,
---
Hinrich Schütze1,3, David R. Mortensen2, Lori Levin2
1LMU Munich, 2LTI, Carnegie Mellon University, 3Munich Center for Machine
Learning
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In this paper, we test the ability of large language models (LLMs) to identify
different meanings in sentences that are superficially similar. The three
sentences shown in Figure 1 each contain the intensifier so, an adjective that
heads an adjective phrase, and a clausal complement, a clause that is a
dependent in the adjective phrase. In spite of their surface similarity, the
three sentences have different semantic properties. In the first sentence, I
was so certain that I saw you, there are no causal connections between the
adjective and the clausal complement. Seeing you did not make me certain, and
being certain did not make me see you. In the second sentence, I was so happy
that I was freed, there is a causal connection. Being freed caused me to be
happy. In the third sentence, It was so big that it fell over, there is also a
causal connection, but it is the reverse, being excessively big caused it to
fall over. This is an example of the Causal Excess Construction as studied by
Kay and Sag (2012) and others.

We examine these sentences from the perspective of Construction Grammar (CxG,
Goldberg, 2006; Fillmore et al., 1988; Croft, 2001), an approach to grammar in
which arbitrary connections of form and meaning are not limited to the
lexicon. In CxG meaning bearing units can contain multiple words, morphemes,
and syntactic relations. The purpose of this paper is to test LLMs for their
ability to differentiate between the three constructions in Figure 1, despite
their surface similarity.

Epistemic Adjective Phrase (EAP) Iwasso certainthat I saw youA Affective
Adjective Phrase (AAP) Iwasso happythat I was freedA Causal Excess
Construction (CEC) Itwasso bigthat it fell over causality causality Figure 1:
Examples of the clausal complement and causal excess constructions.
Constructions involving intensifier, adjective, and clausal complement

The differences between the constructions can be observed semantically in
terms of the presence of causality and the direction of causality as shown in
Figure 1. The Causal Excess Construction (CEC) also differs from the others in
underlying lexico-syntactic properties. This can be seen by removing the word
so from each sentence. I was certain that I saw you and I was happy that I was
freed are grammatical sentences. In contrast, *It was big that it fell over is
not grammatical. The difference is explained in terms of licensing. Every
phrase or clause in a sentence must be licensed by a head that selects it. A
simple example of licensing is that transitive verbs license direct objects
(The student heard the lecture) whereas intransitive verbs do not (*The
student yawned the lecture). In the examples we are dealing with here,
epistemic and affective adjectives license complement clauses just as
transitive verbs license direct objects. Adjectives like big on the other
hand, do not license clausal complements. In the CEC the clausal complement is
licensed by the presence of so, or more accurately, by the constellation of
elements in the CEC construction (Kay and Sag, 2012). Therefore the clausal
complement cannot be present when so is not present.

Further underlying syntactic properties of the three constructions make our
work more interesting. Our experiments rely heavily on the fact that epistemic
and affective adjectives can appear in the CEC as in I was so certain that I
didn’t plan for the alternative and I was so happy that I cried. In these
sentences, the excess of certainty caused me not to plan for the alternative
and the excess of happiness caused me to cry. Epistemic and Affective
adjective constructions in the CEC may license two clausal complements, the
first licensed by the adjective and the second licensed by the CEC: I was so
certain that I was right that I didn’t plan for the alternative, I was so
happy that I was freed that I cried.

Importantly, for the experiments we present, any sentence without so cannot be
CEC. So although so happy that I cried can be CEC, happy that I cried cannot
be CEC—crying is interpreted as the cause of happiness in the latter example.

In this work, we ask how well LLMs differentiate between affective adjective
phrases (AAP), epistemic adjective phrases (EAP) and CEC. These offer an ideal
testbed for linguistic probing, since they are structurally identical, with no
exploitable surface cues. Above-baseline performance would have to stem from
(1) knowledge about what each adjective can license, (2) knowledge about
whether causality is typically associated with the adjective, and (3)
understanding the direction of causality (adjective causes clause or clause
causes adjective). Specifically, we investigate GPT-3.5 OpenAI (2021), GPT-4
OpenAI (2023), OpenAI’s ada-002 and Llama 2 Touvron et al. (2023) with various
questions, for each using both a prompt and a probing classifier. 111See
Appendix Table 7 for hyperparameters We observe that LLMs exhibit limited
capability to effectively discriminate between these constructions and display
a strong bias towards CEC, meaning LLMs tend to judge sentences containing so…
that… as causal and the adjective being the reason for the clausal complement.
Generally, Llama 2 demonstrates superior performance compared to both GPT-3.5
and GPT-4.

## 2\. Related Work

Our work is situated in the framework of CxG Croft (2001); Fillmore et al.
(1988); Goldberg (1995, 2006); Hoffmann and Trousdale (2013). In this work, we
define a construction as a pairing of form and meaning. We are considering
AAP, EAP and CEC to be three different constructions. They differ in meaning
(affective, epistemic, and excessive states) and in form (a clause that is
licensed by a lexical head and a clause that is licensed by the intensifier
so).

Recent studies use CxG to probe the inner workings of LLMs (Weissweiler et
al., 2022; Chronis et al., 2023; Mahowald, 2023), and make general
observations about the compatibility of the theory of CxG with the recent
successes of LLMs (Goldberg, To appear; Weissweiler et al., 2023). Most
related to our work, McCoy et al. (2019) construct a challenging dataset for
Natural Language Inference (NLI) that uses pairs of sentences with high
lexical overlap. They show that the surface similarity of words almost always
fools BERT into assuming that one sentence entails the other. Recent work Si
et al. (2022); Basmov et al. (2023) suggests that the performance of recent
LLMs on the McCoy et al. (2019) data has improved, though it is still far from
perfect. We are therefore motivated to provide another challenging dataset to
this line of inquiry.

## 3\. Dataset

Our data collection process takes advantage of Universal Dependency (Nivre et
al., 2020) annotations, which we use for prefiltering before manual
annotation. We use SPIKE (Shlain et al., 2020) to access a parsed Wikipedia
corpus and a parsed Amazon Reviews corpus. We establish that the parse trees
of all CEC, AAP and EAP constructions have an edge labelled ccomp from the
adjective to the head verb of the complement clause.

We extract all sentences matching so… that… pattern with a structural search
string The bookshelf was so adj:[tag]tall that it verb:[tag]toppled in SPIKE,
and group the sentences by adjective. Manually, where possible, we extract a
sentence pair where one is CEC and one either AAP or EAP resulting in 111 such
pairs. For the adjectives that cannot license a clausal complement, we extract
101 sentences, each with a different adjective. We call this class OCE (only
Causal Excess). Our set of CEC sentences excludes any OCE adjectives.

In total, we collect 323 sentences with 212 different adjectives. 222Data and
code are available at https://github.com/shijiazh/Constructions-Are-So-
Difficult An example of each sentence type is given in the first row of Table
1.

Type | Transformation |  OCE |  CEC |  AAP |  EAP
---|---|---|---|---|---
O | Original |  It was so big that it fell over. |  I was so happy that I cried. |  I was so happy that I was freed. |  I was so certain that I saw you.
DS |  $-$ ‘so’ |  It was {} big that it fell over . |  I was {} happy that I cried. |  I was {} happy that I was freed. |  I was {} certain that I saw you.
DT |  $-$ ‘that’ |  It was so big {} it fell over . |  I was so happy {} I cried. |  I was so happy {} I was freed. |  I was so certain {} I saw you.
DST |  $-$ ‘so’ & ‘that’ |  It was {} big {} it fell over . |  I was {} happy {} I cried. |  I was {} happy {} I was freed. |  I was {} certain {} I saw you.
AN |  $+$ ‘not’ |  It was not so big that it fell over . |  I was not so happy that I cried. |  I was not so happy that I was freed. |  I was not so certain that I saw you.
P1 | main clause |  It was so big. |  It was so happy. |  I was so happy. |  I was so certain.
P2 | sub. clause |  It fell over. |  I cried. |  I was freed. |  I saw you.
Y-N | yes–no question |  Did it fall over? |  Did I cry? |  Was I freed? |  Did I see you?
Table 1: Transformations of the dataset with examples

## 4\. Natural Language Inference

As shown in Table 2 (Prompts 1-1 to 1-4), we design four prompt variants to
test whether LLMs can detect significant changes in entailment when so is
removed from CEC sentences (DS type in Table 1). For example, I was so happy
that I cried does not automatically entail I was happy that I cried, whereas I
was so happy that I was freed entails I was happy that I was freed.

The results shown in Figure 2 are striking: For CEC sentences, models achieve
accuracy below 10%, while demonstrating 90% accuracy for AAP and EAP. It
indicates a pronounced bias towards entailment in the models, which replicates
a behaviour shown for BERT: “assume that a premise entails all hypotheses
constructed from words in the premise” by McCoy et al. (2019). The following
subsections investigate two hypotheses about why LLMs overestimate
entailments: (i) LLMs are unable to identify causality in sentences (§4.2);
(ii) LLMs do not recognize the change in the direction of causality (§4.3).

Figure 2: Performance of CEC, AAP, EAP on the central NLI task. Corresponding gold labels are no entailment, no entailment, entailment, and entailment. All models have a strong bias to answer entailment. For OCE verbs, DS (delete so) is ungrammatical, therefore, we did not prompt on the OCE type. No. |  Template
---|---
1 | 1 |  premise: O \n hypothesis: DS \n Classify as entailment, no entailment, or contradiction.
2 |  premise: O \n hypothesis: DS \n Classify as entailment, no entailment, or contradiction.
3 |  O Can we infer that “DS"? \n Answer with yes, no or uncertain.
4 |  O Can we infer that “DS"? \n Answer with yes, no or uncertain.
3 | 1 |  O \n Is there a causal relationship between the main clause and the subordinate clause? \n Answer with yes, no or uncertain.
2 |  O \n Part1: P1 \n Part2: P2 \n Is there a causal relationship between part 1 and part 2? \n Answer with yes, no or uncertain.
4 | 1 |  premise: O \n hypothesis: P2 \n Classify as entailment, no entailment, or contradiction.
2 |  O \n Y-N \n Answer with yes, no or uncertain.
3 |  O \n Part1: P1 \n Part2: P2 \n Can we infer that Part1 is the cause of Part2? \n Answer with yes, no or uncertain.
4 |  O \n Part1: P1 \n Part2: P2 \n Can we infer that Part2 is the cause of Part1? \n Answer with yes, no or uncertain.
5 |  O \n This entails one of two options. \n 1) P1 because P2 \n 2) P2 because P1 \n Answer with the correct number.
Table 2: Prompt templates of the central NLI task. To test the stability of
model responses, we design variants of each prompt, removing only so from
premise as hypothesis or removing both so and that.

### 4.1. Experimental Setup

We conduct our experiments with two methods. The first approach involves the
development of both implicit and explicit prompts. In the second approach, we
extract the last-layer embeddings from LLMs and then apply perceptron-based
classification to these embeddings, to assess how well the CEC, OCE, AAP and
EAP categories are internally represented in the models.

We employ a perceptron classifier to test the final layer embeddings of Llama
2, and ada-002 sentence embeddings. For Llama 2, we use the mean over token
embeddings as a sentence embedding. We hypothesise that the contextual
representation of the adjective itself will encode the presence or direction
of causality, and therefore also test its embedding separately. When the two
input classes are imbalanced, we upsample the smaller class. We group
sentences containing the same adjectives together and train the perceptron
using cross-validation over adjectives, meaning that the adjective itself is
no longer an exploitable feature. A Bag-of-Words (BoW) model serves as the
baseline.

### 4.2. Identifying Causality

#### Prompting

As depicted in Table 3, we devise two prompts to assess the models’ capability
to identify causal relationships. The first simply asks about a causal
relationship between the main and the subordinate clause, while the second
additionally provides the pre-segmented parts. In the EAP category, all models
have accuracy below 20%, suggesting a predisposition to infer causality in
sentences containing so… that… even when the adjective is epistemic. Llama 2
displays a stronger bias, attributing causality to over 90% of sentences in
all categories. Combining the previous observation that Llama 2 tends to
categorise every sentence containing so and that as grammatically correct,
along with its sensitivity to the absence of so in CEC, this suggests a
limited grasp of semantic nuances and an overreliance on simple lexical cues.
GPT models both struggle about equally, with less than 50% accuracy in AAP
instances. Even more perplexing, EAP sentences are classified as causal at a
significantly higher rate than CEC and OCE.

No. | Model |  OCE |  CEC |  AAP |  EAP | Gold Lab.
---|---|---|---|---|---|---
3-1 | GPT-3.5 |  67.33 |  60.90 |  41.68 |  18.57 | Y|Y|Y|N
GPT-4 |  63.37 |  58.74 |  41.20 |  15.00
Llama 2 |  95.05 |  98.65 |  95.18 |  08.93
3-2 | GPT-3.5 |  54.46 |  64.14 |  49.15 |  06.43 | Y|Y|Y|N
GPT-4 |  57.03 |  65.95 |  46.02 |  04.28
Llama 2 |  95.54 |  99.10 |  92.78 |  08.93
Table 3: Accuracy of the task of identifying causality with different prompts
Figure 3:  Accuracy of perceptrons trained with different embeddings across
three tasks. In all sub-tasks involving CEC structures, we attempt to replace
CEC with OCE. OCE adjectives are mutually exclusive with those in EAP and AAP.

#### Probing Classifier

The classifiers for CEC vs EAP and AAP vs EAP serve to assess the models’
representation of causality. As illustrated in Figure 3, on CEC vs EAP,
perceptrons trained on sentence embeddings beat the baseline while those
trained on adjective embeddings do not, suggesting that causality is encoded,
but not necessarily in the adjective. Interestingly, the adjective perceptron
beats the baseline on the AAP vs EAP test, even though the sets of adjectives
are mutually exclusive and we perform cross-validation over them, meaning that
the only source of information left would be a deeper commonality between
them. This suggests that the models may have learned common features for
affective and epistemic adjectives, respectively. Furthermore, the result also
demonstrates that the distinction between EAP and CEC is more pronounced in
the model’s perspective compared to the distinction between EAP and AAP,
especially for Llama 2 sentence embeddings.

### 4.3. Direction of Causality

#### Prompting

The negator not before so influences the truth value of the subclause for CEC
but has no influence in AAP. For instance, He was not so big that he fell
(CEC) does not imply that he fell, while He was not so happy that he went.
(AAP) suggests that he went. Asking the model to distinguish between these is
equivalent to distinguishing the direction of causality.

Therefore, we devise an explicit NLI prompt 4-1 (Webson and Pavlick, 2022) and
an implicit prompt 4-2 (AN + Y-N) to explore these effects. Accurate answers
depend on the models’ precise understanding of the causal direction. We also
introduce prompt 4-3, where P1 and P2 are provided, and the question is
whether P1 is the cause of P2. Prompt 4-4 maintains the same structure but now
inquires whether P2 is the cause of P1. Prompt 4-5 is structured as a
multiple-choice question between the two directions.

As can be seen in Table 4, results for prompts 4-1 and 4-2 suggest that they
bias all models towards answering yes for any sentence. By contrast, prompts
4-3 and 4-4 show a clearer picture, still biased, but Llama 2 scores clearly
correlate with the gold label. By comparing these two sets of prompts, we can
discern that Llama 2’s responses are grounded in an analysis of P1 and P2,
rather than simply providing uniform yes or no answers. Interestingly, prompt
4-5 elicit better performance from the GPT models in contrast, suggesting that
it might be more suited to the multiple-choice answer format. We conclude that
all models have some representation of the direction of causality, but it is
far from perfect.

Type |  Model | OCE | CEC | AAP | Gold Label
---|---|---|---|---|---
4-1 |  GPT-3.5 | 28.71 | 29.19 | 62.41 | N|N|Y
GPT-4 | 26.93 | 29.01 | 62.65
Llama 2 | 39.60 | 49.10 | 53.62
4-2 |  GPT-3.5 | 4.55 | 2.52 | 60.24 | N|N|Y
GPT-4 | 4.95 | 2.16 | 60.72
Llama 2 | 16.34 | 18.47 | 40.97
4-3 |  GPT-3.5 | 74.46 | 82.88 | 13.25 | Y|Y|N
GPT-4 | 77.03 | 83.96 | 8.91
Llama 2 | 87.13 | 93.69 | 46.99
4-4 |  GPT-3.5 | 50.69 | 42.70 | 44.09 | N|N|Y
GPT-4 | 48.31 | 40.18 | 47.95
Llama 2 | 77.23 | 71.17 | 81.92
4-5 |  GPT-3.5 | 61.19 | 60.72 | 77.59 | 2)|2)|1)
GPT-4 | 60.80 | 54.78 | 79.27
Llama 2 | 51.98 | 45.49 | 78.31
Table 4: Accuracy of direction of causality task with different prompts. Y:
yes/entailment, N: no/contradiction.

#### Probing Classifier

The classifier for CEC vs AAP serve to assess the models’ capability to
differentiate the direction of causality. The results in Figure 3 are similar
to those for identifying causality in Section 4.2, with the notable exception
of the OCE test set, which is easiest for the adjective classifiers with no
obvious explanation.

Figure 3 displays that on CEC vs AAP, similar to CEC vs EAP, models trained
with sentence embeddings outperform the baseline, while those trained with
adjective embeddings slightly lag behind. Additionally, the perceptron attains
the highest accuracy on the OCE test set.

As adjectives in OCE can only appear in CEC, while adjectives in CEC can also
occur in AAP or EAP, this can be interpreted as OCE’s adjective embeddings
being more easily identified as belonging to the CEC structure than those of
CEC.

## 5\. Grammatical Acceptability

We perform experiments on grammatical acceptability to see whether LLMs are
sensitive to the difference between embedded clauses that are lexically
licensed by an adjective and embedded clauses that are licensed by the CEC
construction. Following Mahowald (2023), we prompt LLMs for grammaticality
judgements by first presenting eight pairs of sentences from the CoLA corpus
(Warstadt et al., 2019), which consists of minimal pairs of grammatical and
ungrammatical sentences extracted from linguistics literature. Then the target
sentence is inserted, and the model is asked to generate one token, good or
bad.

No. | Model |  OCE |  CEC |  AAP |  EAP | Gold Label
---|---|---|---|---|---|---
O | GPT-3.5 |  89.31 |  92.43 |  80.96 |  73.57 | G|G|G|G
GPT-4 |  89.31 |  92.97 |  81.93 |  73.57
Llama 2 |  100.00 |  100.00 |  100.00 |  100.00
DS | GPT-3.5 |  36.83 |  84.69 |  87.23 |  79.28 | B|G|G|G
GPT-4 |  36.43 |  84.32 |  88.92 |  78.57
Llama 2 |  39.58 |  76.30 |  80.62 |  83.34
DT | GPT-3.5 |  80.40 |  89.55 |  72.29 |  60.00 | G|G|G|G
GPT-4 |  80.40 |  88.83 |  70.60 |  56.43
Llama 2 |  100.00 |  100.00 |  100.00 |  100.00
DST | GPT-3.5 |  57.83 |  67.39 |  67.95 |  65.71 | B|G|G|G
GPT-4 |  57.83 |  66.67 |  68.92 |  65.72
Llama 2 |  49.77 |  49.97 |  60.50 |  76.46
AN | GPT-3.5 |  83.76 |  90.63 |  74.46 |  69.29 | G|G|G|G
GPT-4 |  84.36 |  90.45 |  76.14 |  74.29
Llama 2 |  100.00 |  100.00 |  100.00 |  100.00
Table 5: Accuracy of the grammaticality task. Bold font indicates the models
with the highest accuracy for a type and transformation. G: good, B: bad.

We test the original sentences (O) and four transformations DS, DT, DST, and
AN, as mentioned in Table 1. Deleting that (DT) or adding not (AN) will not
affect the grammaticality of CEC, EAP and AAP, whereas removing so (DS) from
OCE makes them ungrammatical.

As can be seen in Table 5, compared to CEC OCE is more likely to be rated as
bad by both GPT-3.5 and GPT-4, regardless of the transformation.

It demonstrates that GPT models indeed detecte the distinction between OCE and
CEC especially regarding their reliance on the grammaticality with so.
Notably, for GPT models, removing that from AAP and EAP sentences results in
more good, whereas removing that from DS sentences tends to yield more bad
ratings even though it has no effect on grammaticality.

Llama 2’s answers deviate from that of GPT models. It rates all O, DT, and AN
sentences as good, which is exactly the gold label, signifying its robust
inclination to not only consider sentences featuring so… that… as acceptable,
but also acknowledge the possibility of omitting that in such contexts.
However, its performance on DS and DST is far from perfect.

Additionally, GPT4 performs better on the DS variants of AAP.333A probing
classifier for this prompting task would not be well-defined.

## 6\. Conclusion

Overall, our most striking result remains that no LLM performed adequately on
our NLI task, and that this result is not sufficiently explained by the
mediocre but better-than-baseline performance on the sub-tasks. Llama 2
performed better in those than the GPT models, but generally, prompting
results are often consistently below random and probing classifier results
only slightly above baseline. Interestingly, GPT-4 does not perform
significantly better than GPT-3.5 at any task.

Both in the central NLI task, and the sub-tasks, all LLMs show bias to offer
positive answers. Llama 2 demonstrates a more comprehensive understanding of
the grammatical structures within CEC, an enhanced ability to identify
causality within sentences, and a greater proficiency in ascertaining the
direction of causality compared to GPT models. These findings align with our
initial observations in the central NLI task.

We exclude the following from the current work: (1) Extraposition from clausal
subject (That I left was so bad/It was so bad that I left) (2) CEC meanings
with other intensifiers such as enough and non-finite clauses (big enough to
fall over) (3) CEC headed by nouns (so many people that the police came). We
have also not investigated the CEC in sentences other than copula sentences,
or when the CEC adjective is part of a noun phrase (I met many people so short
that they couldn’t reach the counter).

## Bibliographical References

##

  * Basmov et al. (2023) Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2023.  ChatGPT and simple linguistic inferences: Blind spots and blinds.
  * Chronis et al. (2023) Gabriella Chronis, Kyle Mahowald, and Katrin Erk. 2023.  A method for studying semantic construal in grammatical constructions with interpretable contextual embedding spaces.  _arXiv preprint arXiv:2305.18598_.
  * Croft (2001) William Croft. 2001.  _Radical Construction Grammar: Syntactic theory in typological perspective_.  Oxford University Press, Oxford, UK.
  * Fillmore et al. (1988) Charles J. Fillmore, Paul Kay, and Mary Catherine O’Connor. 1988.  Regularity and idiomaticity in grammatical constructions: the case of ‘let alone’.  _Language_ , 64(3):501–538.
  * Goldberg (1995) Adele E. Goldberg. 1995.  _Constructions: a construction grammar approach to argument structure_.  University of Chicago Press, Chicago.
  * Goldberg (2006) Adele E. Goldberg. 2006.  _Constructions at Work: The Nature of Generalization in Language_.  Oxford University Press, Oxford.
  * Goldberg (To appear) Adele E. Goldberg. To appear.  A chat about constructionist approaches and LLMs.  _Constructions and Frames_.
  * Hoffmann and Trousdale (2013) Thomas Hoffmann and Graeme Trousdale. 2013.  _The Oxford handbook of construction grammar_.  Oxford University Press.
  * Kay and Sag (2012) Paul Kay and Ivan A Sag. 2012.  Cleaning up the big mess: Discontinuous dependencies and complex determiners.  In _Sign-based construction grammar_ , chapter 5, pages 229–256. Citeseer.
  * Mahowald (2023) Kyle Mahowald. 2023.  A discerning several thousand judgments: GPT-3 rates the article + adjective + numeral + noun construction.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 265–273, Dubrovnik, Croatia. Association for Computational Linguistics.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal Dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4034–4043, Marseille, France. European Language Resources Association.
  * OpenAI (2021) OpenAI. 2021.  GPT-3.5: The Third-and-a-Half-Generation Language Model.  https://api.openai.com/docs/gpt-3.5.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Shlain et al. (2020) Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020.  Syntactic search by example.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 17–23, Online. Association for Computational Linguistics.
  * Si et al. (2022) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2022.  Prompting gpt-3 to be reliable.  In _The Eleventh International Conference on Learning Representations_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do prompt-based models really understand the meaning of their prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2300–2344, Seattle, United States. Association for Computational Linguistics.
  * Weissweiler et al. (2023) Leonie Weissweiler, Taiqi He, Naoki Otani, David R. Mortensen, Lori Levin, and Hinrich Schütze. 2023.  Construction grammar provides unique insight into neural language models.  In _Proceedings of the First International Workshop on Construction Grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023)_ , pages 85–95, Washington, D.C. Association for Computational Linguistics.
  * Weissweiler et al. (2022) Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022.  The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10859–10882, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

## Appendix

Now we are going to say which sentences are
---
acceptable (i.e., grammatical) and which are not.
Sentence: Flosa has often seen Marn.
Answer: good
Sentence: Chardon sees often Kuru.
Answer: bad
Sentence: Bob walk.
Answer: bad
Sentence: Malevolent floral candy is delicious.
Answer: good
Sentence: The bone chewed the dog.
Answer: good
Sentence: The bone dog the chewed.
Answer: bad
Sentence: I wonder you ate how much.
Answer: bad
Sentence: The fragrant orangutan sings loudest at
Easter.
Answer: good
Sentence: [TEST SENTENCE GOES HERE]
Answer:
Table 6: Few-shot CoLA prompts template created by Mahowald (2023). We tested 5 types of sentence: O, DS, DT, DST and AN. | GPT-3.5/GPT-4 | Llama 2
---|---|---
temperature | 1 | 0.7
top_p | 1 | 0.95
top_k | - | 40
max_tokens | null | 512
frequency_penalty | 0 | -
presence_penalty | 0 | -
Table 7: Hyperparameters for GPT-3.5, GPT-4, OpenAI’s ada-002 and Llama 2. For each prompt, we repeat the mean over six runs of the experiment. Adjective | Type |  Sentence
---|---|---
| CEC |  In one XFM show , he became so frustrated that he left the room before Karl finished the segment .
frustrated | AAP |  I am so frustrated that a $ 500 purchase brought such short lived joy .
| CEC |  Mandhata had dominated the whole planet and he became so proud that he wanted to rule heaven also .
proud | AAP |  My dad was so proud that his son made " aliyah " .
| CEC |  One man was so afraid that he camped in the middle of his flock , hoping to evade patrolling cowboys .
afraid | EAP |  He was so afraid that rival loyalist inmates wished to kill him inside the prison .
| CEC |  Like Napoleon , Hitler was so optimistic that he falsely believed he ’d make it to Moscow before Winter .
optimistic | EAP |  I am so optimistic that I made the best choice .
abrupt | OCE |  The growth was so abrupt that a village sprang .
beautiful | OCE |  The palace was so beautiful that the king of Mengwi heard of Tan Cin Jin .
Table 8: Examples from the collected database. CEC represents causal excess
construction, where the adjective is interpreted as the cause of the
complement. AAP stands for affective adjective phrases, which usually trigger
an inference that the complement caused the feeling expressed by the
adjective. EAP stands for epistemic adjective phrases, which lexically
liscence non-causal complement.