# Harnessing speckle for a sub-femtometre resolved broadband wavemeter and
laser stabilization

Nikolaus Klaus Metzger  SUPA, School of Physics and Astronomy, University of
St Andrews, KY16 9SS, UK.  Roman Spesyvtsev  SUPA, School of Physics and
Astronomy, University of St Andrews, KY16 9SS, UK.  Graham D. Bruce  SUPA,
School of Physics and Astronomy, University of St Andrews, KY16 9SS, UK.  Bill
Miller  M Squared Lasers Ltd, Venture Building, 1 Kelvin Campus, West of
Scotland Science Park, Glasgow, G20 0SP, UK  Gareth T. Maker  M Squared Lasers
Ltd, Venture Building, 1 Kelvin Campus, West of Scotland Science Park,
Glasgow, G20 0SP, UK  Graeme Malcolm  M Squared Lasers Ltd, Venture Building,
1 Kelvin Campus, West of Scotland Science Park, Glasgow, G20 0SP, UK  Michael
Mazilu  SUPA, School of Physics and Astronomy, University of St Andrews, KY16
9SS, UK.  Kishan Dholakia<EMAIL_ADDRESS>SUPA, School of Physics and
Astronomy, University of St Andrews, KY16 9SS, UK.

###### Abstract

The accurate determination and control of the wavelength of light is
fundamental to many fields of science. Speckle patterns resulting from the
interference of multiple reflections in disordered media are well-known to
scramble the information content of light by complex but linear processes.
However, these patterns are, in fact, exceptionally rich in information about
the illuminating source. We use a fibre-coupled integrating sphere to generate
wavelength-dependent speckle patterns, in combination with algorithms based on
the transmission matrix method and principal component analysis, to realize a
broadband and sensitive wavemeter. We demonstrate sub-femtometre wavelength
resolution at a centre wavelength of $780\text{\,}\mathrm{nm}$ and a broad
calibrated measurement range from 488 to $1064\text{\,}\mathrm{nm}$. This is
comparable with or exceeding the performance of conventional wavemeters. Using
this speckle wavemeter as part of a feedback loop, we stabilize a
$780\text{\,}\mathrm{nm}$ diode laser to achieve a linewidth better than
$1\text{\,}\mathrm{MHz}$.

##  1 Introduction

Light propagation in disordered media is often regarded as a randomization
process by which the information contained within an optical field is
destroyed, and the speckle pattern that results from repeated scattering and
interference is commonly deemed detrimental to an optical system. However,
although highly complex, multiple scattering is nonetheless a linear,
deterministic and reversible process [1]. A coherent beam propagating in a
disordered medium yields a unique speckle pattern that depends on the spatial
and temporal characteristics of the incident light field. In the spatial
domain [2, 3], light scattering in complex (disordered) media can be harnessed
for imaging at depth [4], novel endoscopy [5] and even sub-wavelength focusing
[6]. In a spatially-static system, the temporal variations of the speckle
pattern can be used to identify characteristics of the incident light, in
particular as a spectrometer or wavemeter [7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20], with state-of-the-art demonstrations of wavelength
resolution at the picometre level and operating ranges of over
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$ covering the visible and near-
infrared spectrum.

In this letter, we use an integrating sphere as the random medium for
wavelength detection. This acts like a highly sensitive and complex
interferometer due to internal path-length differences on the metre scale,
delivering a different speckle pattern for each wavelength in a system with a
small footprint. The resultant wavemeter is demonstrated over a wide spectral
operating range of approximately $600\text{\,}\mathrm{nm}$ in the visible and
near-infrared spectrum, with wavelength resolution at the sub-femtometre level
($0.3\text{\,}\mathrm{fm}$ measured at a centre wavelength of
$780\text{\,}\mathrm{nm}$). This resolution is up to two orders of magnitude
better than all previous spectrometers using complex media [16, 12, 23].
Through the subsequent use of appropriate electronic feedback, we realize an
alternative approach for laser frequency stabilization. We are able to
stabilize an external cavity diode laser (ECDL) over $10\text{\,}\mathrm{s}$,
(a timescale suitable for, e.g., laser cooling of Rb atoms) and observe
linewidth narrowing of the ECDL by a factor of four. As this approach of using
a speckle-based stabilization scheme is not based on atomic absorption, it has
the advantage of being able to lock at an arbitrary wavelength, as with Fabry-
Perot-based stabilization schemes [24].

##  2 Results

### Wavemeter Setup

Figure 1: Experimental Setup. (a) Laser light is delivered into an integrating
sphere S of diameter $D_{\textrm{sphere}}$ via an angled single mode fibre
(SMF) where it undergoes numerous scattering events. A camera C located at a
distance $L$ from the integrating sphere observation port captures the speckle
patterns. A wavemeter or Fabry-Perot interferometer REF is used to benchmark
the speckle wavemeter. (b) An example of a speckle pattern captured by the
camera at a wavelength of $780\text{\,}\mathrm{nm}$ with an image size of
$512\times 512$ pixels (with individual pixels of
$4.5\times$4.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$$). The white scale bar
denotes $200\text{\,}\mathrm{\SIUnitSymbolMicro m}$

.

All data in this letter were acquired with the setup shown in Fig. 1. The
laser beam is split into two paths. The first path is used for reference
measurements of the laser wavelength with a commercial wavemeter or a Fabry-
Perot interferometer. The second path is coupled into a spectralon-coated
integrating sphere through an angle-cleaved single-mode fibre to ensure a
spatially-static input beam and eradicate back reflections which could
otherwise hinder stable laser operation. The speckle pattern produced inside
the sphere is measured by a camera placed at a distance $L$ from the exit port
of the integrating sphere. In contrast to interferometer-based wavemeters, no
accurate alignment of the individual components is necessary. For example, the
camera and fibre can be offset by several degrees without perturbing the
measurement results as long as stability is maintained after the calibration
step.

The mean free path in the sphere plays a prominent role in increasing the
resolution of the system, analogous to the spectral to spatial mapping
principle of a spectrometer. Thus, both the sphere diameter
$D_{\textrm{sphere}}$ and propagation distance $L$ from sphere to camera
dictate the resolution and range of the wavemeter, as detailed in the
‘Methods’ section. Unless otherwise stated, here
$D_{\textrm{sphere}}=$5\text{\,}\mathrm{cm}$$ and
$L=$20\text{\,}\mathrm{cm}$$. The acquisition rate of the wavemeter depends on
the camera speed and can be above $200\text{\,}\mathrm{kHz}$ for a small
detector area. The total power sent to the sphere was below
$1\text{\,}\mathrm{mW}$ for highest frame rates and can be lowered when a
short exposure time and high frame rate are not required. Further details of
the experimental setup can be found in the ‘Methods’ section.

### Detection Algorithm

Figure 2: Wavelength measurement with the speckle wavemeter. (a) Confusion
matrix showing the probabilistic measurement of different set wavelengths of
an external cavity diode laser, analyzed with the transmission matrix method.
(b) Measured wavelength using the same external cavity diode laser as in a
(dark red), plus a broadly tunable Titanium-Sapphire laser (pink) and assorted
other single-wavelength sources, across the broad operational range of 488 to
$1064\text{\,}\mathrm{nm}$. (c) Principal component analysis is used to
determine the wavelength to sub-picometre accuracy. The graphs show the
measured wavelength of a modulated external cavity diode laser, obtained both
from our speckle pattern reconstruction (blue) and a reference trace obtained
by a commercial wavemeter (red). Here, $\Delta\lambda=0$ corresponds to a
wavelength of $779.9431\text{\,}\mathrm{nm}$. The inset shows the error
distribution of the measurement with respect to the reference wavemeter with a
standard deviation of $7.5\text{\,}\mathrm{fm}$. The data were taken within 10
minutes of the initial wavelength calibration.

The core of the speckle wavemeter is the detection algorithm used to associate
wavelength changes with the speckle pattern variations. To achieve a broad
operating range, a coarse analysis using the transmission matrix method (TMM)
follows standard approaches [7, 8, 9, 10, 11, 12, 13] for wavelength
determination with a nanometre resolution, as shown in Fig. 2a-b. The
intensity distribution recorded by the camera is $I=TS$, where $S$ is the
input spectrum and $T$ the transmission matrix. This is constructed in a
calibration stage, from speckle patterns recorded using established and well-
known laser sources across the required wavelength range with a step of
$1\text{\,}\mathrm{nm}$. Subsequently, the unknown spectrum of incident light
can be reconstructed from a speckle pattern, at the resolution of
$1\text{\,}\mathrm{nm}$, by performing left-inversion of the transmission
matrix. However, noise can corrupt the reconstruction, we therefore use the
truncated singular value decomposition to find a pseudo inverse transmission
matrix, as in [11]. At this stage, a nonlinear search algorithm [8, 9, 10, 11,
12, 13] could be employed to further lower the noise in the retrieval of the
spectrum and enable characterization of broadband light in a spectrometer
configuration [7, 13]. This part of our algorithm is exclusively used to
determine the wavelength of an unknown light field with a precision of 1 nm as
shown in Fig. 2a-b and no further attempt has been made to retrieve a broad
spectrum or increase the resolution. The demonstrated operational range of 488
to $1064\text{\,}\mathrm{nm}$ is currently limited by the spectral response of
the camera used.

To achieve a high resolution we use a different classification algorithm
capable of accurately distinguishing small speckle changes caused by
wavelength changes below $1\text{\,}\mathrm{nm}$. This algorithm is based on
principal component analysis (PCA) and is further detailed in the ‘Methods’
section. Principal component analysis (or singular value decomposition) is an
established tool to decorrelate multivariate data. In the field of optics PCA
has, for example, been applied to wavelength detection [16], detector noise
analysis [25], to distinguish surface scattering processes [26] and is
commonly used in the spectral band recognition of mixed colourants [27] as
well as in Raman analysis [28]. For our wavelength meter we project the data
set (speckle images) onto its principal components and thereby retrieve the
underlying wavelength modulation [16]. An initial calibration data set is
acquired and used to correlate the speckle patterns to wavelength, and sets
the measurement range of the high-resolution wavemeter. The density of the
calibration set aids the accuracy of the wavemeter [16]. Subsequently, all
measurements are projected against this calibration set to extract the unknown
wavelength. Importantly, the wavelength-dependent changes in the speckle
pattern that are captured by PCA are subtle in comparison to the TMM. A
comparison of both approaches is shown in the ‘Methods’ section. An example of
the retrieved wavelength trace (blue) is shown in Fig. 2c together with the
reference wavemeter reading (red). We note that, in order to measure an
unknown wavelength to high precision, these two algorithms can be used
sequentially. All subsequent measurements in this paper were performed close
to $780\text{\,}\mathrm{nm}$, but could in principle be performed at any
wavelength within the operational range of the camera.

### Wavemeter Resolution

Figure 3: Resolution of the speckle wavemeter. Wavelength modulation of an
external cavity diode laser at $f=$21\text{\,}\mathrm{Hz}$$ and an amplitude
of (a) $60\text{\,}\mathrm{fm}$ and (b) $0.3\text{\,}\mathrm{fm}$, measured
with the speckle wavemeter (blue) and the reference wavemeter (red).
$\Delta\lambda$ = 0 corresponds to an absolute wavelength of
$780.2437\text{\,}\mathrm{nm}$. The modulation at $0.3\text{\,}\mathrm{fm}$ is
below the resolution of commercial wavemeters.

After the calibration of the desired operating range, we test our speckle
wavemeter by measuring the change in wavelength from an ECDL centred at
$780.2437\text{\,}\mathrm{nm}$. We apply a controlled wavelength modulation by
varying the current to the laser diode at $21\text{\,}\mathrm{Hz}$, as shown
in Fig. 3.

For the case of relatively large wavelength modulation of 60 fm, Fig. 3a shows
the good agreement between the wavelength measured using the speckle
wavemeter, where we set $L=$2\text{\,}\mathrm{cm}$$, and a reference
wavemeter. To explore the resolution limit of our approach, we increase the
distance to the camera to $L=$20\text{\,}\mathrm{cm}$$ and decrease the
wavelength modulation amplitude by a factor of 200. These measurements are
below the wavelength resolution of our commercial wavemeter, but interpolating
over five runs gives agreement within $10\%$ of interpolated Fabry-Perot
interferometer measurements of the wavelength. In Fig. 3 the centre wavelength
has been subtracted from the speckle wavemeter wavelength readings and
displayed as $\Delta\lambda$ in femtometres, where $\Delta\lambda$ = 0
corresponds to an absolute wavelength of $780.2437\text{\,}\mathrm{nm}$.
Clearly, with a high resolution commercial wavemeter a more accurate
calibration would be possible and an absolute wavelength reading could be
presented. Further reduction of the modulation amplitude was not possible due
to the limited resolution of the modulation signal source. A more detailed
analysis of the range and resolution of the speckle wavemeter and its
dependence on experimental design parameters is presented in the ‘Methods’
section.

Figure 4: Measurement of laser dither tone. (a) The hyperfine structure of
rubidium, with the $160\text{\,}\mathrm{fm}$ separation of two crossover peaks
indicated. The laser is locked to the largest peak at a dither frequency of
$24\text{\,}\mathrm{kHz}$ with an amplitude of $1.3\text{\,}\mathrm{fm}$. (b)
Measurement of the dither tone while the laser is locked. To achieve a high
sampling rate the image size is reduced to almost a line with $2048\times 4$
pixels. The points show individual acquisitions while the dashed line provides
a guide to the eye. (c) Single-sided amplitude spectrum retrieving the lock
dither tone frequency as $23.9(7)\text{\,}\mathrm{kHz}$, where the error is
the half-width at half maximum of the peak.

Using the speckle wavemeter as a detector within a calibrated rubidium
absorption spectroscopy setup, we also resolve the wavelength changes of a
stabilized Titanium-Sapphire laser (M Squared Lasers Ltd SolsTiS). The laser
is stabilized by top-of-fringe locking [29] to the crossover between the
$F=2\rightarrow F^{\prime}=2$ and $F=2\rightarrow F^{\prime}=3$ hyperfine
transitions in the D2-line of 87Rb (see Fig. 4a). Importantly this locking
technique uses a dither tone at a frequency of 24 kHz with an amplitude of
$1.3\text{\,}\mathrm{fm}$. This modulation signal was wavelength calibrated
and is utilized as a calibration standard for our wavemeter. The speckle
wavemeter is able to correctly resolve this dither tone as shown in Fig. 4b-c.

A detailed comparison of our wavemeter to current commercial and research-
grade wavemeters and spectrometers can be found in Supplementary Table 2.

### Laser Stabilization

Figure 5: Laser Stabilization. (a) Experimental setup of the control loop:
Laser light is generated using an external cavity diode laser ECDL comprising
laser diode LD, diffraction grating G and a mirror M. The integrating sphere
wavemeter (sphere S and camera C) is employed as a detector of wavelength
perturbations. The speckle image is analyzed and a control algorithm hosted on
a desktop computer generates an error signal which is used to regulate the
laser wavelength by adjusting the DC current to the laser diode LD. A
controllable perturbation to the wavelength is applied by adjusting the cavity
length (double-arrow). (b) Wavelength trace of the experiment: Initially the
system is calibrated by linearly changing the drive current, defining the
capture range of the speckle lock to $\pm 30\text{\,}\mathrm{fm}$ or $\pm
15\text{\,}\mathrm{MHz}$ at a centre wavelength of
$780.24\text{\,}\mathrm{nm}$. The midpoint of the calibration range is the
wavelength setpoint (red dashed line). The laser is locked between 4 s and 18
s (shaded region) and the linewidth of the laser narrows from
$3.2\text{\,}\mathrm{MHz}$ to $0.82\text{\,}\mathrm{MHz}$. (c) Zoom of the
lock period. We apply a perturbation via the grating, indicated by the error
signal in red. The wavemeter output (blue) shows deviations only at points of
rapid change outside the response time of the lock.

Having bench tested our wavelength meter, we also incorporate it within the
wavelength stabilization control loop of the ECDL, as depicted in Fig. 5a.
Here, the speckle wavemeter is once again used as a detector: the speckle
image is analyzed using the PCA algorithm to determine the emitted wavelength.
A subsequent proportional-integral-derivative algorithm calculates an error
signal from the measured wavelength to a predefined wavelength setpoint. The
error signal regulates the ECDL diode drive current and thus the laser
wavelength. Initially the wavemeter is calibrated across $\pm
30\text{\,}\mathrm{fm}$ ($\pm 15\text{\,}\mathrm{MHz}$) from the stabilization
setpoint which sets the capture range of the speckle lock. The grating of the
ECDL is employed as an external controllable wavelength perturbation, which we
use to demonstrate the stabilization performance of the speckle lock.

Fig. 5b shows a wavelength trace captured by the external wavemeter during the
stabilization. The experiment starts at 2.5 seconds with the
$60\text{\,}\mathrm{fm}$-wide calibration ramp, then the laser is locked to
the central speckle image (at $780.24\text{\,}\mathrm{nm}$) from 4 to 18
seconds. From 18 seconds onward the ECDL is free running without feedback
control. During stabilization we achieve a linewidth narrowing from
$3.2\text{\,}\mathrm{MHz}$ (rms) to $0.82\text{\,}\mathrm{MHz}$ (rms) of the
ECDL over a 10 seconds observation time (for both measurements), which is
ample for applications such as the efficient laser cooling of atoms.

As shown in Fig. 5c, between 13 and $15\text{\,}\mathrm{s}$ we apply a
sawtooth modulation to the ECDL grating which corresponds to a 50 fm variation
of the unstabilized laser wavelength. The resultant error signal shows the
compensation applied to the locked laser and the wavemeter only detects
wavelength changes at the start and end of the ramp, when the perturbation is
largest. Increasing the update rate of the control loop, which is currently
limited to $200\text{\,}\mathrm{Hz}$, would minimize the amplitude of these
spikes.

##  3 Discussion

This work presents a practical speckle-based wavemeter with sub-femtometre
resolution, which retains a broad operating range. A central element of the
operating principle of the device is the use of an integrating sphere to
create speckle patterns with a high sensitivity to wavelength changes through
a compact and near lossless propagation path. A two step algorithm based on
the transmission matrix method and principal component analysis was applied to
resolve wavelengths within the range 488 to $1064\text{\,}\mathrm{nm}$ and
changes in wavelength below $0.3\text{\,}\mathrm{fm}$ respectively. A full
specification of the present performance of the wavemeter is presented in
Supplementary Table 1.

In comparison to the use of an integrating sphere, fibre-based [12, 9, 13]
speckle wavemeters have realized picometre resolution to date. Progress
towards attometre-level resolution of our wavemeter is currently limited by
our 16-bit amplitude modulation source and the availability of an accurate
wavelength reference to calibrate the variability of the principal components.
In theory the selectivity of the integrating sphere coating (Spectralon)
supports an operational range from 0.2 to
$2.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$ at high efficiency (reflectivity
$>$90%). In practice, this operational range was limited by the response of
the CMOS chip of our cameras.

Our present system is limited in its long-term stability (see Supplementary
Note 1 and Supplementary Figure 1), but mechanical and thermal stabilization
engineering could form part of a future study. Alternatively, a regular
recalibration step, as can be found in many commercial wavemeters, can be used
to maintain accuracy over longer periods of time.

The speckle-based wavemeter was also used as part of a laser stabilization
setup. Encouragingly, the system achieved laser stability over several seconds
without any thermal or vibration management, and reduced the linewidth of a
diode laser by a factor of four. It is important to note that the frequency
stability of the speckle-locked laser is presently not compared to any
reference laser and thus the measurement was limited to the accuracy of the
wavelength meter. The demonstrated capture range of $60\text{\,}\mathrm{fm}$
(corresponding to $30\text{\,}\mathrm{MHz}$ at $780\text{\,}\mathrm{nm}$) is
currently small compared to other approaches such as polarization spectroscopy
[30] which can achieve GHz capture range for lasers with a comparable
linewidth to those discussed here.

The locking technique described here could aid the development of portable
atom-based quantum technologies [31] due to the small footprint and
insensitivity to misalignment. The speckle-based stabilization is a convenient
method to lock at arbitrary detuning from an atomic resonance, such as the
tens of GHz typically required in atom interferometry experiments [32].
Furthermore, compared to dither locking of the laser to an atomic saturated
absorption feature, speckle-based stabilization has the advantage of
eliminating any frequency modulation of the laser. Finally, the extension of
our approach to the simultaneous detection and measurement of multiple
wavelengths [33] could also allow a single device to stabilize all of the
various lasers (cooling, repumper, Raman and trapping) required in quantum
technologies based on cold atoms.

##  4 Methods

### Details of the experimental setup

For the wavemeter study, several well known and established laser systems were
employed to determine the broadband range we were able to achieve with our
approach. For the resolution and stabilization studies, an ECDL (Sacher
Lasertechnik Lion TE-520, $780\text{\,}\mathrm{nm}$ and linewidth of
$3\text{\,}\mathrm{MHz}$ over 10 seconds) was used as a wavelength tunable
source. A commercial wavemeter (HighFinesse WS7, resolution
$5\text{\,}\mathrm{MHz}$) and a Fabry-Perot interferometer (Burleigh, free
spectral range $2\text{\,}\mathrm{GHz}$, finesse 200,
$\sim$$10\text{\,}\mathrm{MHz}$ linewidth resolution) were used for wavelength
reference to benchmark our system. The speckle patterns were captured using a
xiQ - USB3 Vision Camera for the wavelength measurements. A fast camera with a
maximum frame rate above 200 kHz (Mikrotron monochrome EoSens 4CXP, 4
megapixel CMOS camera with an Active Silicon FireBird CoaXPress frame grabber)
was used for the laser stabilization experiments. For all measurements, the
camera exposure and optical power were adjusted to avoid image saturation with
the aid of an image histogram as shown in Supplementary Figure 2. The speckle
patterns were subsequently analyzed on a $2.4\text{\,}\mathrm{GHz}$ dual core
personal computer.

### Speckle image analysis algorithm

Figure 6: Comparison of Transmission Matrix Method and Principal Component
Analysis. (a) The Transmission Matrix Method calibration matrix that converts
between spatial patterns and input frequencies for wavelengths of 769 to 791
nm. The image pixels associated with each wavelength form the column vectors
in the matrix. (b) Spectral correlation function of a. (c) Principal Component
Analysis training set that converts between spatial patterns and input
frequencies with a wavelength variability of 80 fm. (d) Spectral correlation
function of c. The corresponding pixel intensity for a and c is shown in the
right hand colour bar.

In the integrating sphere wavemeter every speckle pattern serves as a marker
for a wavelength. An initial training phase is used to correlate training
speckle patterns to wavelengths. In a second step every new acquired frame is
projected against the initial training set and the current wavelength is
retrieved. In the transmission matrix method (TMM) algorithm, for our training
set is a series of speckle images at known wavelengths with a step size of 1
nm. The image pixels associated with each wavelength form the column vectors
in our calibration matrix, that converts between spatial patterns and input
frequencies. Fig. 6a shows an example of a calibration matrix for wavelengths
of 769 to 791 nm. The average correlation over the image pixels, shown in Fig.
6b, is close to zero for a wavelength shift of 1 nm.

An example training set that converts between spatial patterns and input
frequencies at the femtometre level is shown in Fig. 6c. The spectral
correlation function in Fig. 6d shows strong correlation over a range of at
least 300 fm. Comparison of both calibration matrices in Fig. 6a and 6c makes
it evident that at the femtometre scale the overall speckle variability has
decreased significantly. Thus, to correlate the speckle pattern to wavelength
variations of less than one nanometre, we have implemented the PCA approach
using Matlab and in C++ for the laser stabilization experiment using the
standard template library for linear algebra. The PCA is applied to speckle
images $y_{i}$ which are acquired for a varying wavelength $\lambda$
(subscript $i$ denotes the pixel number). In Fig. 7 we plot the projections of
the measured speckle patterns $y_{i}(\lambda)$ in PC space, PC$j=U_{ji}\cdot
y_{i}$ where $U_{ji}$ corresponds to the principal components with index
$j\leq\,4$. The values on the $y$-axis are a relative marker of the variation
as the variance of the speckle pattern is multiplied by the coordinate of the
corresponding eigenvector $U$, with values between -1 and 1. The PCs for
wavelength modulations of the ECDL at $60$ fm (Fig. 7a) and $0.3$ fm (Fig. 7b)
are therefore represented relative to the sign of the eigenvector and can be
inverted. (Here, the speckle image size is $256\times 16$ pixels, the camera
distance $L=2$ cm and the integrating sphere diameter $D_{\text{sphere}}=5$ cm
for the $60$ fm modulation and $D_{\text{sphere}}=10$ cm for the $0.3$ fm
modulation.)

We observe that each PC shows an oscillatory behaviour that we further analyze
by Fourier transforming (using the single-sided amplitude spectrum) the PC
traces (Fig. 7c-d). We denote the Fourier transform of PC1 as
$F(\textrm{PC1})$. We observe that PC1 to 4 are showing analogous
oscillations. Here, PC1 is capturing wavelength modulation as generated by the
laser diode drive current modulation, while PC2 detects the second order
variations of the speckle pattern. Similarly, higher order PCs detect higher
order speckle pattern variations that are each orthogonal and uncorrelated to
each other. This orthogonality manifests itself also in the single-sided
amplitude spectrum of the PCs time series where we observe that PC3 and PC4
exhibit modulations corresponding to the third and fourth harmonic of the
wavelength modulation frequency $f=21$ Hz as shown in Fig. 7c.

### Measurement Range and Resolution.

Figure 7: Principal Component Analysis. Principal components (PC) 1-4 for a sinusoidal (at $f=21$ Hz) wavelength modulation of (a) $60$ fm and (b) $0.3$ fm. (c) and (d) Single-sided amplitude spectrum obtained by Fourier transform of the PCs presented in a and b respectively. The peak amplitude $\left|F(\textrm{PC1})\right|$, which is used to extract the resolution and range of the analysis in Fig. 8, is highlighted in c. Figure 8: Optimization of range and resolution. (a) Detected modulation peak $|F(\textrm{PC1})|$ as a function of wavelength modulation at $f=21$ Hz. Error bars indicate the standard deviation over five measurements. Different experimental parameters are investigated (see table 1): Blue and red are studies with image sizes of 256 x 16 and 256 x 128 pixels, respectively. Blue is used as a reference and subsequent measurements were compared to the results. Green indicates an increased distance $L$ from 2 to 40 cm in comparison to blue. Black is a study with an increased diameter of the integrating sphere $D_{\textrm{sphere}}$ in comparison to blue: from $5\text{\,}\mathrm{cm}$ to $10\text{\,}\mathrm{cm}$. (b) The bandwidth limit of the principal component analysis is the point at which the dependence of $\left|\textrm{F(PC1)}\right|$ on wavelength modulation deviates from linearity. For two different speckle image sizes of 256 x 16 pixels (blue) and 256 x 128 pixels (red), this limit is 16 fm and 60 fm respectively. The bandwidth increases with increased image size. Image size (pixels) | $D_{\textrm{sphere}}$ | $L$ | Resolution | Range |
---|---|---|---|---|---
$2336\times 1024$ | 2.5 cm | 2 cm | 80 fm | 8 nm |
$256\times 16$ | 5 cm | 2 cm | 3.2 fm | 16 fm |
$2336\times 1728$ | 5 cm | 2 cm | 1.2 fm | 1 nm |
$256\times 16$ | 5 cm | 20 cm |  $\leq$ 0.3 fm | 16 fm |
$256\times 16$ | 5 cm | 40 cm |  $\leq$ 0.3 fm | 16 fm |
$256\times 16$ | 10 cm | 2 cm |  $\leq$ 0.3 fm | 16 fm |
Table 1: Optimization of range and resolution. Resolution limit and
measurement range of a speckle based wavemeter dependent on key experimental
parameters at 780 nm.

The PCA algorithm is a powerful tool to detect small changes in the overall
speckle pattern. In this section we investigate the range and resolution of
the PCA solely.

One method to quantify the resolution and range limits of our wavemeter is to
determine the amplitude of the modulation peak $\left|F(\textrm{PC1})\right|$
of the single-sided amplitude spectrum of the first PC, as indicated in Fig.
7c. The height of this peak correlates directly with the wavelength modulation
and should follow a linear dependency. Deviations from this at low (high)
wavelength modulation indicate an incorrect retrieval of the modulation and
hence the resolution limit (range). This method has been experimentally
verified for several setups as a method to quantify the wavelength resolution.
In the lower limit, this corresponds to sub-pixel changes in the speckle
pattern, and in the upper limit, to changes occurring outside the finite image
range. Fig. 8 collates $\left|F(\textrm{PC1})\right|$ as a function of
wavelength modulation amplitude. This is investigated for different
experimental parameters (shown in Fig. 8) for small wavelength modulations of
0.3 to 10 fm. In future work, the theoretical model of our configuration which
is presented in Supplementary Note 2 and Supplementary Figures 2-3 could guide
the optimal design parameters of subsequent generations of our wavemeter.

A reference line (shown in blue) is measured for a $256\times 16$ pixel image
at $L=2$ cm from an integrating sphere with $D_{\textrm{sphere}}=5$ cm, which
gives a resolution of 3.2 fm and range of 16 fm. First, we studied the
influence of the total number of pixels. We observed that an eight times
larger speckle image size (red line, $256\times 128$ pixels) enhances the
outcome of the PCA and decreases the resolution limit from 3.2 fm to 1.2 fm
and increases the range to 60 fm. This can be explained by considering that a
bigger detection array is able to capture smaller distributed perturbations.

Another important parameter is the propagation distance $L$ of the speckle
pattern after the output from the sphere. Indeed, finer speckle structures can
be resolved by the camera as this distance increases. The upper spatial
frequency limit contained within a speckle pattern [34] is given by
$\xi_{\textrm{cutoff}}=\frac{D_{\textrm{port}}}{\lambda L}$ where
$D_{\textrm{port}}$ is the integrating sphere port diameter, $\lambda$ is the
wavelength and $L$ the distance between the port and the camera detector
array. For the detector array the Nyquist frequency can be calculated from
half the element spacing [35, 36] of $7\text{\,}\mathrm{\SIUnitSymbolMicro m}$
to 71.4 cycles mm-1. To avoid aliasing, the maximum spatial frequency
contained in the patterns should be equal to the spatial Nyquist frequency.
With a port diameter of 1 cm the camera should be approximately 18 cm away
from the sphere. Here, increased distances $L$ of 20 cm and 40 cm were
investigated and resolution was found to increase for $L\geq$18 cm (depicted
for $L$= 40 cm in Fig. 8a, green) to 0.3 fm.

Additionally, to enhance the spatial resolution of the system one can increase
the mean free path within the sphere. When the sphere diameter is increased
from 5 to 10 cm (depicted in Fig. 8a $D=10$ cm in black) a resolution of 0.3
fm can be achieved. Table 1 summarizes the parameters in detail and shows the
resolvable minimum wavelength modulation amplitude as well as the range.

The resolution of the method depends on the changes in the speckle pattern
with respect to the incoming wavelength. When the wavelength changes, the
speckle decorrelates. In previous work [7, 12], the half-width at half maximum
(HWHM) of the correlation function was used as an indicator for the ultimate
resolution of the device. However, this method depends on the reconstruction
algorithm and the ratio of the speckle variation to the detection noise [12].
Therefore, the actual resolution is determined by the ability to discern
speckle patterns from two closely spaced spectral lines. In this work we are
able to achieve resolution well below the HWHM of the speckle correlation. In
the presented ECDL setup we are driving a piezo to tune the wavelength. In
order to eliminate hysteresis effects due to the piezo, we used a sinusoidal
modulation signal and the frequency retrieval as a marker for the resolution
of the system.

## Data Availability

The data that support the findings of this study are available from the
University of St Andrews Research Data Repository at
doi:10.17630/9448553f-a806-464b-a41c-7e2f0c80522b.

## References

  * [1] Dainty J. C., ed. _Laser Speckle and Related Phenomena_ (Springer, 1984)
  * [2] George, N. and Jain, A. Space and wavelength dependence of speckle intensity Appl. Phys. 4, 201–212 (1974).
  * [3] Jones, R. and Wykes, C. Holographic and Speckle Interferometry (Cambridge University Press, 2012).
  * [4] Mosk, A. P., Lagendijk, A., Lerosey, G., & Fink, M. Controlling waves in space and time for imaging and focusing in complex media. Nature Photonics 6, 283-–292 (2012).
  * [5] Čižmár, T. & Dholakia, K. Exploiting multimode waveguides for pure fibre-based imaging. Nat. Commun. 3, 1027 (2012).
  * [6] Park, J. H. et al. Subwavelength light focusing using random nanoparticles. Nature Photon. 7, 454–458 (2013).
  * [7] Redding, B., Liew, S., Sarma, R. & Cao, H. Compact spectrometer based on a disordered photonic chip. Nature Photon. 7, 746–751 (2013).
  * [8] Xu, Z., et al. Multimodal multiplex spectroscopy using photonic crystals. Opt. Express 11, 2126–2133 (2003).
  * [9] Hang, Q., Ung, B., Syed, I., Guo, N. & Skorobogatiy, M. Photonic bandgap fiber bundle spectrometer. Appl. Opt. 49, 4791–4800 (2010).
  * [10] Redding, B. & Cao, H. Using a multimode fiber as high-resolution, low-loss spectrometer. Opt. Lett. 37, 3384-3386 (2012).
  * [11] Redding, B., Popoff, S. M. & Cao, H. All-fiber spectrometer based on speckle pattern reconstruction. Opt. Express 21, 6584-6600 (2013).
  * [12] Redding, B., Alam, M., Seifert, M. & Cao, H. High-resolution and broadband all-fiber spectrometers. Optica 1, 175–180 (2014).
  * [13] Wan, N. H., et al. High-resolution optical spectroscopy using multimode interference in a compact tapered fibre. Nat. Commun. 6, 7762 (2015).
  * [14] Feller, S. D., et al. Multiple order coded aperture (MOCA) spectrometer. Opt. Express 15, 5625–5630 (2007).
  * [15] Kohlgraf-Owens, T. W. & Dogariu, A. Transmission matrices of random media: means for spectral polarimetric measurements. Opt. Lett. 13, 2236–2238 (2010).
  * [16] Mazilu, M., Vettenburg, T., Di Falco, A. & Dholakia, K. Random super-prism wavelength meter. Opt. Lett. 39, 96–99 (2014).
  * [17] Wang, P. & Menon, R. Computational spectrometer based on a broadband diffractive optic. Opt. Express 22, 14575–14587, (2014).
  * [18] Bao, J. & Bawendi, M. G. A colloidal quantum dot spectrometer. Nature 523, 67–70 (2015).
  * [19] Liew, S. F., Redding, B., Choma, M. A., Tagare, H. T., & Cao, H. Broadband multimode fiber spectrometer. Opt. Lett. 41, 2029–2032 (2016)
  * [20] Redding, B., Liew, S., Bromberg, Y., Sarma, R. & Cao, H. Evanescently coupled multimode spiral spectrometer. Optica 3, 956–962 (2016).
  * [21] Shlens, J. A tutorial on principal component analysis. Preprint at http://arXiv.org/abs/1404.1100 (2014).
  * [22] Brand, M. Fast low-rank modifications of the thin singular value decomposition. Linear Algebra Appl. 415, 20–30 (2006).
  * [23] Chakrabarti, M., Jakobsen, M. L. & Hanson, S. G. Speckle-based spectrometer. Opt. Lett. 40, 3264–3267 (2015).
  * [24] Brachmann J. F. S., Kinder, T. & Dieckmann, K. Calibrating an interferometric laser frequency stabilization to megahertz precision. Appl. Opt. 51, 5517–5521 (2012).
  * [25] Ferrero, A., Alda, J., Campos, J., López-Alonso, J. M. & Pons, A. Principal components analysis of the photoresponse nonuniformity of a matrix detector. Appl. Opt. 46, 9–17 (2007).
  * [26] Ferrero, A., et al. Principal components analysis on the spectral bidirectional reflectance distribution function of ceramic colour standards. Opt. Express 19, 19199–19211 (2011).
  * [27] Fairman, M. H. & Brill, H. S. The principal components of reflectances. Color Res. Appl. 29, 104–110 (2004).
  * [28] Jess, P. R. T., et al. Early detection of cervical neoplasia by Raman spectroscopy. Int. J. Cancer 121, 2723–2728 (2007).
  * [29] Hänsch, T. W., Levenson, M. D. & Schawlow, A. L. Complete hyperfine structure of a molecular iodine line. Phys. Rev. Lett. 26, 946–949 (1971).
  * [30] Pearman, C. P., et al. Polarization spectroscopy of a closed atomic transition: applications to laser frequency locking. J. Phys. B 35, 5141–5151 (2002).
  * [31] Bongs, K., et al. The UK National Quantum Technologies Hub in sensors and metrology. Proc. SPIE 9900, Quantum Optics, 990009 (2016).
  * [32] Cronin, A. D., Schmiedmayer, J. and Pritchard, D. E. Optics and interferometry with atoms and molecules. Rev. Mod. Phys. 81, 1051–1129 (2009).
  * [33] Coluccelli, N., et al. The optical frequency comb fibre spectrometer. Nat. Commun. 7, 12995 (2016).
  * [34] Boreman, G. D., Centore, A. B. & Sun, Y. Generation of laser speckle with an integrating sphere. Opt. Eng. 29, 339–342 (1990).
  * [35] Pozo A. M. & Rubiño, M. Comparative analysis of techniques for measuring the modulation transfer functions of charge-coupled devices based on the generation of laser speckle. Appl. Opt. 44, 1543–1547 (2005).
  * [36] Pozo, A. M., Ferrero, A., Rubiño, M., Campos, M. J., & Pons, A. Improvements for determining the modulation transfer function of charge-coupled devices by the speckle method. Opt. Express 14, 5928–5936 (2006).

## Acknowledgements

N.K. Metzger acknowledges support from Dr. C.T.A. Brown and thanks Dr T.J.
Edwards and Dr S. Jiang for fruitful discussions. The authors acknowledge
funding from EPSRC through an Impact Acceleration Account (IAA) Case Study
award and grants EP/J01771X/1 and EP/M000869/1.

## Author contributions

NKM, MM and KD developed the project. NKM developed the algorithms and
conducted the experiments. NKM and GDB analyzed the data. RS implemented the
C++ real-time control algorithm, NKM developed the C++ feedback control loop.
BM, GTM and GM supported the work with a laser loan and technical advice. NKM,
GDB and KD wrote the paper with contributions from all the authors. KD
supervised the study.

## Competing financial interests

The intellectual property based around this work is owned by the University of
St Andrews and licensed to M Squared Lasers Ltd.


# Neuro-symbolic Commonsense Social Reasoning

David Chanin  Anthony Hunter
Department of Computer Science
University College London
{david.chanin.22<EMAIL_ADDRESS>
###### Abstract

Social norms underlie all human social interactions, yet formalizing and
reasoning with them remains a major challenge for AI systems. We present a
novel system for taking social rules of thumb (ROTs) in natural language from
the Social Chemistry 101 dataset and converting them to first-order logic
where reasoning is performed using a neuro-symbolic theorem prover. We
accomplish this in several steps. First, ROTs are converted into Abstract
Meaning Representation (AMR), which is a graphical representation of the
concepts in a sentence, and align the AMR with RoBERTa embeddings. We then
generate alternate simplified versions of the AMR via a novel algorithm,
recombining and merging embeddings for added robustness against different
wordings of text, and incorrect AMR parses. The AMR is then converted into
first-order logic, and is queried with a neuro-symbolic theorem prover. The
goal of this paper is to develop and evaluate a neuro-symbolic method which
performs explicit reasoning about social situations in a logical form.

##  1 Introduction

Deep learning approaches have seen success on a wide range of natural language
understanding (NLU) and natural language processing (NLP) tasks (?). However,
these models are largely black boxes and it is difficult to understand how
these models arrive at conclusions. This is especially problematic when
dealing with controversial social norms, where current deep learning models
have been shown to learn a range of problematic racial and gender biases (?;
?; ?). It is not hard to envision how this bias will lead to social harm as we
rely more and more on language models to generate content and power systems
that make decisions that affect people’s lives.

While interpretability methods exist that can be applied to deep learning
models (?; ?; ?), these methods at best provide hints into what sorts of
features influence a model’s predictions, but cannot directly explain the
reasoning process the model took internally, and do not provide a way to
directly change problematic model behavior.

Symbolic logic approaches, on the other hand, reason with explicitly defined
rules which gives a number of benefits for interpretability, multi-hop
reasoning, and directly changing model behavior. It is possible to trace
through inference steps in a symbolic logic system to determine how a
conclusion was derived, and tweak any problematic logical formulae as
necessary. Ideally, we should be able to vet the logical rules that a model is
using during inference and delete or modify any undesirable rules as needed.

Symbolic logic, however, tends to be brittle and struggles with small
inconsistencies in input data, as symbols in logical formulae cannot represent
semantically similar concepts like “dad” and “father” without every possible
alternate symbol and relation being explicitly added to the system. This
brittleness is not a problem for deep learning, however, as the entire system
learns to simply maximize output over noisy inputs. These contrasting
strengths motivate neuro-symbolic approaches which can combine the strengths
of both symbolic logic and deep learning.

The goal of this work is to develop a neuro-symbolic social reasoning system
to do logical reasoning using social rules of thumb described in natural
language. We accomplish this in several steps. First, we leverage Abstract
Meaning Representation (AMR) (?) by converting natural language social rules-
of-thumb into AMR and then align them with RoBERTa embeddings. We then
generate alternate simplified versions of the AMR for robustness against
different wordings of “semantically equivalent” sentences and AMR parse
errors. The AMR is then converted into first-order logic, and is queried with
a neuro-symbolic theorem prover (?; ?).

We use natural language social rules-of-thumb from the Social Chemistry 101
dataset (?) as a source of rules for the reasoning system. Social Chemistry
101 is a dataset containing over 290,000 rules of thumb (ROTs) for evaluating
people’s behavior in everyday social situations. Social Chemistry ROTs are
drawn from Reddit’s r/confessions and r/amitheasshole subreddits, the
ROCStories corpus (?), and from the Dear Abby advice column web archives
111https://www.uexpress.com/life/dearabby/archives. These rules of thumb cover
a wide range of common situations, with ROTs like “You should be there for
your child’s birthday” or “It is wrong to damage someone else’s property on
purpose”. ROTs consist of a short sentence containing a judgement like “It’s
wrong to …” or “It’s reasonable to …” followed by a description of the
situation being judged. In addition, the dataset contains a social situation
text (SST) for each ROT which gives an example that the ROT applies to. A
sample of Social Chemistry 101 data is shown in Table 1.

These ROTs have the potential to be a great source of commonsense knowledge
for a reasoning system, if they can be parsed into a logical form that makes
it possible to do reasoning. This paper attempts to do exactly this, by
turning Social Chemistry ROTs into first-order logic, as will be described in
more detail in section 4.

Rule of Thumb |  Social Situation Text
---|---
You shouldn’t put dog poop in front of someone’s door |  putting my neighbor’s dog’s poop in front of their door
It is wrong to be jealous of your partner |  being jealous of my gf
It’s expected that you’ll buy Christmas presents for the kids in your life |  not buying my roommate’s kids Christmas presents this year
It’s bad to be an enabler of harmful behavior |  not wanting to enable my grandparents hurting themselves
Table 1: Sample Social Chemistry ROTs and corresponding SSTs

We evaluate the performance of our method on Social Chemistry 101 by measuring
the degree to which the method is able to match each ROT in the dataset to its
corresponding SST via logical inference, and verify as well that we should not
be able to conclude a match via logical inference when applying each ROT to a
different, randomly chosen SST from the dataset. For example, we would expect
the ROT “It is wrong to be jealous of your partner” to apply to the SST “being
jealous of my girlfriend”, but it should not apply to the SST “Not buying my
roommates’ kids Christmas presents”.

We proceed as follows: In Section 2, we give a overview on AMR and converting
AMR into logical formulae; In Section 3, we give a background overview on
neuro-symbolic theorem proving; In Section 4, we detail our social reasoning
system; In Section 5 we evaluate our technique against the Social Chemistry
dataset; In Section 6, we discuss related work; In Section 7 we discuss future
directions for this work. The code for this paper is available online
222https://github.com/chanind/amr-social-chemistry-reasoner.

##  2 Abstract meaning representation

Abstract meaning representation (AMR) (?) is a powerful abstraction which can
help simplify and standardize sentences for semantic meaning. AMR represents
text as a rooted, directed acyclic graph drawing predicate senses and semantic
roles from the OntoNotes project (?). An example is shown in Figure 1.

        (w / want-01
            :ARG0 (b / boy)
            :ARG1 (b2 / believe-01
                :ARG0 (g / girl)
                :ARG1 b))

Figure 1: AMR for the sentence “The boy wants the girl to believe him”

AMR has first-class support for negation, which is particularly valuable when
translating AMR into first-order logic. This is demonstrated below for the
sentence “The boy does not go”, where negation is represented via the
:polarity relation.

    (g / go-02
        :ARG0 (b / boy)
        :polarity -)

The numbers after the instance name in AMR (e.g. go-02 above) refers to a
specific OntoNotes or PropBank (?) semantic frame. These frames can take
different parameters, but in general :ARG0 refers to the subject and :ARG1
refers to the object. In our work, however, we find that these frame numbers
(the -02 in go-02) tend to get mixed-up by existing AMR parsers, so we discard
these frame numbers when they appear in AMR. For the rest of this paper, we
will remove frame numbers in AMR.

AMR makes a number of simplifying assumptions in order to represent
semantically similar sentences using identical structure. AMR does not
differentiate between verbs and nouns, and cannot represent verb tenses. It
does not represent singular and plural, quotations, or articles. For instance,
the AMR in Figure 1 could refer to any of the following English sentences,
among many others: “The boy desires the girl to believe him.”, “The boy will
desire the girl to believe him.”, “A boy desires a girl to believe him.”, “The
boy has a desire to be believed by the girl.”, etc… For this research, the
simplification of English that AMR performs is generally beneficial since it
removes information from English sentences which is less relevant to the task
at hand.

AMR was also chosen due to the availability of high-quality off-the-shelf AMR
parsers (?; ?) and tooling like the Python Penman library (?).

AMR can be represented as either a tree or a graph, with the tree directly
mapping to the AMR text format (?). An example of showing the same AMR
represented as text, a tree, and a graph is shown in 2. It is always possible
to generate the graph representation from the tree representation, but the
reverse is not always possible as there are multiple possible trees that map
to the same graph. In the rest of this paper, we always use the tree
representation rather than the graph representation when working with AMR.

Figure 2: AMR represented as text, a tree, and a graph, from the Python Penman
library.

AMR also benefits from existing work on converting AMR to first-order logic.
(?) describes an algorithm for converting an AMR graph into existentially-
quantified first-order logic joining all predicates by an implicit
conjunction. An example of this is shown below, where the AMR for “The boy
does not go” from Figure LABEL:fig:negamr is converted into logical formulae
as follows:

$\neg\exists G.(\textrm{go}(G)\land\exists
B.(\textrm{:ARG0}(G,B)\land\textrm{boy}(B)))$

Another example of AMR and its converted logical formulae from (?) is show in
Figure 3.

    (e / dry
        :ARG0 (x / person
            :named "Mr Krupp")
        :ARG1 x)

$\displaystyle\exists X.(\textrm{person}(X)\land\textrm{named}(X,\textrm{``Mr
Krupp"})$ $\displaystyle\hskip 14.22636pt\land\;\exists
E.(\textrm{dry}(X)\land\textrm{:ARG0}(E,X)\land\textrm{:ARG1}(E,X)))$

Figure 3: AMR and corresponding logical formulae representation of the AMR for
“Mr Krupp dries himself”

There is not space to reproduce the full algorithm of (?) here, but we refer
the reader to the original paper for details. In addition, we implemented this
algorithm in an open-source Python library called “Amr Logic Converter”
333https://github.com/chanind/amr-logic-converter which is used in this paper.

##  3 Resolution with non-binary unification

Traditional theorem provers unify predicates using an exact match on the name
of the predicate (?). For instance, the statement father(Homer, Bart) would
unify against father(Homer, Y) since they both have the same predicate
father/2, and all constants match when the variable $Y$ is grounded to the
constant Bart. However, dad(Homer,Y) would not unify since dad/2 is not an
exact string match with father/2, despite them having the same semantic
meaning.

Issues where semantically similar predicates have different wording is common
in natural language, so we require a theorem prover which can work with a non-
binary unification function based on a similarity score rather than exact
string matches.

$a_{1}\lor\ldots a_{n}\lor b$ $\neg c\lor d_{1}\lor\ldots d_{m}$
$\textrm{unify}(b,c)$ $a_{1}\lor\ldots\lor a_{n}\lor d_{1}\lor\ldots\lor
d_{m}$

Figure 4: Resolution proof rule where each
$a_{i}\in\\{a_{1},\ldots,a_{n}\\},d_{i}\in\\{d_{1},\ldots,d_{n}\\}$ refer to a
grounded logical literal, and $b$ and $c$ are positive literals. The condition
unify(b,c) holds if the predicates for $b$ and $c$ match, and all constants in
all terms also match after grounding variables.

The traditional resolution rule (?) for first-order logic is shown in Figure
4. A unify function is shown in Algorithm 1. For simplicity, the algorithm
shown assumes that the variable substitution map is provided as input to the
unify function, although typically the substitution map is calculated along
with the unification. In addition, the algorithm shown does not include
function symbols, as those are not used in this paper.

The unify function in Algorithm 1 uses a similarity function, simFunc, which
returns a value between 0 and 1, and a threshold $\tau$, where the unification
succeeds if the minimum similarity of all similarity checks in the unification
is above $\tau$. When simFunc is a string comparison which returns 1 if the
strings are identical and 0 if not, and $\tau$ is 0.5, then this reduces to
traditional binary unification.

For example, using the example of father(Homer, Bart) and father(Homer, Y)
from above, unification succeeds with pred1,terms1 = father, [Homer,Bart],
pred2,term2 = father,[Homer,Y] and substitutions = {Y / Bart}.

Algorithm 1 Non-binary unify

Input: pred1, terms1
Input: pred2, terms2
Input: substitutions
Input: simFunc, $\tau$

sim $\leftarrow$ simFunc(pred1, pred2)

terms1 $\leftarrow$ applySubs(terms1, substitutions)

terms2 $\leftarrow$ applySubs(terms2, substitutions)

for all term1, term2 $\in$ zip(terms1, terms2) do

if type(term1) $\neq$ type(term2) then return $False$

if isConst(term1) then

sim $\leftarrow$ min(sim, simFunc(term1, term2))

return sim $>$ $\tau$

procedure applySubs($terms,subs$)

newTerms $\leftarrow$ []

for all term $\in$ terms do

newTerm $\leftarrow$ term

if term $\in$ subs then newTerm $\leftarrow$ subs[term]

newTerms $\leftarrow$ newTerms $\cup$ newTerm

return newTerms

We implemented a non-binary unification theorem prover using input resolution
called “Tensor Theorem prover” 444https://github.com/chanind/tensor-theorem-
prover which we use in this paper. A proof in Tensor Theorem Prover consists
of a chain of resolution steps, each with a corresponding similarity score and
substitutions map. The similarity score for the proof as a whole is defined as
the minimum similarity of all steps in the proof, as is the case in Neural
Theorem Provers (?). Tensor Theorem Prover seeks to find the proof with the
highest similarity score for any query, if one exists. We use the unification
algorithm presented in the seminal paper on resolution (?) to calculate
unification in Tensor Theorem Prover, but replace equality checks between
predicates and constants with a non-binary similarity check as shown in
Algorithm 1.

##  4 Social reasoning system

Our social reasoning system takes a natural language SST and ROT as input.
First, we parse the SST and ROT text into token-aligned AMR matched with
contextual word embeddings. Next, we generate alternate AMR trees by merging
leaf nodes together. From here, we convert AMR trees into first-order logic.
Finally, we use a neuro-symbolic theorem prover to query the logical formulae
and determine if the ROT is applicable to the situation. Each of these steps
will be explained in more detail below.

The core insight which allows this to work is that a logical match between a
ROT and a situation can be framed as finding a matching subtree in the AMR
corresponding to the body of the ROT within the AMR for a situation. This is
demonstrated in Figure 5, where the ROT body AMR subtree is found exactly in
the SST AMR.

Figure 5: Full flow going from text to AMR to logical formulae to proof for a
simple case where the SST AMR and ROT AMR align perfectly with identical
predicates and structure. The matching nodes are highlighted in green in the
text, AMR, and logical formulae. In formulae, $n$,$p$,$o$,$n2$, and $g$ refer
to grounded constants and $N$,$P$, and $O$ refer to universally quantified
variables. To address situations where the alignment is not perfect, we
introduce the use of contextual embeddings and AMR node merges, described in
Section 4.2.

However, in practice the AMR for the ROT and the SST rarely align so
perfectly. To help address this, we introduce contextual embeddings aligned to
the nodes in the AMR tree to allow differently worded but semantically similar
nodes to still match using a vector similarity score rather than simply
performing an exact string match on the AMR instance and role names. This is
shown in Figure 6, where the nodes for “kid” and “child” are allowed to match
due to their embedding vector similarity being very high.

Figure 6: Often nodes in AMR are semantically similar, but not identical. We
use contextual embeddings with cosine similarity to allow these nodes to still
match. The AMR for the ROT is shown on the left, and the AMR for the SST is
shown on the right.

Figure 7: Merging leaf nodes and averaging the merged embeddings allows the
ROT (left) and the SST (right) to match, as the resulting tree no longer has a
suprious “w / white-03” node.

###  4.1 Parsing text into AMR and embeddings

The first step when processing ROT and SST text is to generate an AMR tree and
contextual embeddings for each sentence. We use a pretrained ensemble AMR 3.0
model for the IBM Transition AMR parser (?) for parsing the ROT and SST into
AMR. We use a pretrained RoBERTa (?) base model from Huggingface (?) and
average the last 4 layers of the model to generate contextual embeddings (?).

An AMR tree with embeddings
$\mathcal{T}=\langle\mathcal{N},\mathcal{E},meta\rangle$ is a set of nodes
$\mathcal{N}$, edges $\mathcal{E}$, and a metadata function $meta$ which
provides metadata related to each node and edge.

The set $\mathcal{N}=\mathcal{N}_{I}\cup\mathcal{N}_{C}\cup\mathcal{N}_{R}$
where $\mathcal{N}_{I}$, $\mathcal{N}_{C}$, and $\mathcal{N}_{R}$ are
disjoint, and $\mathcal{N}_{I}$ consist of AMR instance nodes (of the form a /
alpha in Figure 2), $\mathcal{N}_{C}$ consist of AMR constants (raw strings
like “Mr Krupp” in Figure 3, numbers, or symbols like “-” in :polarity -), and
$\mathcal{N}_{R}$ consist of coreference nodes which refer to an instance node
(the b coreference in Figure 2). Each edge $\langle
n,n^{\prime}\rangle\in\mathcal{E}$ is such that $n\in\mathcal{N}_{I}$ and
$n^{\prime}\in\mathcal{N}$. The $meta$ function assigns a tuple of meta-data
to each node and edge as follows, where $l$ is a text label, $p$ is a text
predicate, and $v$ is a vector embedding from RoBERTa. $v$ can be null
(represented as $\emptyset$) for cases where an embedding does not exist.

$\begin{array}[]{ll}\mbox{If }e\in\mathcal{E},&meta(e)=\langle l\rangle\\\
\mbox{If }n\in\mathcal{N}_{I},&meta(n)=\langle l,p,v\rangle\\\ \mbox{If
}n\in\mathcal{N}_{C},&meta(n)=\langle l,v\rangle\\\ \mbox{If
}n\in\mathcal{N}_{R},&meta(n)=\langle l\rangle\\\ \end{array}$

For example, the AMR in Figure 3 has 2 instance nodes e / dry with $\langle
l=\textrm{e},p=\textrm{dry},v=\textrm{embedding[dry]}\rangle$ and x / person
with $\langle l=\textrm{x},p=\textrm{person},v=\textrm{embedding[``Mr
Krupp"]}\rangle$, 1 constant node "Mr Krupp" with $\langle l=\textrm{``Mr
Krupp"},v=\textrm{embedding[``Mr Krupp"]}\rangle$, and 1 coreference node x
with $\langle l=\textrm{x}\rangle$. It also contains 3 edges with labels
:ARG0, :named, and :ARG1.

###  4.2 Generating merged AMR trees

While AMR can smooth over some differences between sentences that have similar
semantic meanings, it is still normal for sentences with similar meaning to
have slight differences in the structure of the generated AMR tree. When these
AMR trees are turned into logical formulae, any differences in the structure
of the generated AMR will cause the formula to no longer match even if the
sentences are describing semantically similar concepts. Furthermore, the AMR
parsing may introduce errors, and we want to be as robust as possible against
minor AMR parsing errors.

A common failure mode occurs when AMR has a different structure between the
ROT and the SST despite the ROT and SST having similar semantic meaning. This
manifests itself in extra nodes or missing nodes which break the alignment
between ROT and SST. Figure 8 illustrates this situation, where the text
“white dog” causes an extra node in the AMR tree and breaks alignment with the
ROT, which only refers to “dogs”. Of course, a formula that applies to a “dog”
should also apply to a “white dog”.

Figure 8: Case where the ROT (left) and SST (right) are logically matching,
but the AMR does not align due to an extra node in the SST which is not
present in the ROT.

We address these alignment errors by generating alternative AMR trees by
collapsing AMR leaf nodes into a single node, averaging together the embedding
vectors of the collapsed nodes. The hope is that the merged embedding vector
of the collapsed nodes in the ROT may match the embedding of the corresponding
node in the situation via vector similarity, or vice-versa, where without
collapsing nodes the AMR graph might have a shape which makes a match
impossible. This is illustrated in Figure 7.

Figure 9: All possible collapsed AMR trees with a max of 3 merged alignments
per node for the AMR for “It’s good to keep things clean”

We introduce a new node type $\mathcal{N}_{M}$ to represent a merged node, so
that
$\mathcal{N}=\mathcal{N}_{I}\cup\mathcal{N}_{C}\cup\mathcal{N}_{R}\cup\mathcal{N}_{M}$.
This node contains a label and the average (using the $avg$ function) of
embeddings from the collapsed nodes produced during a merge:

$\begin{array}[]{ll}\mbox{If }n\in\mathcal{N}_{M},&meta(n)=\langle
l,avg(v_{1},\ldots v_{k})\rangle,k>0\end{array}$

where $k$, the number of embeddings being merged together, is referred to as
the merge width. A node $n^{\prime}$ is a _child_ of a node $n$ if there is an
edge from the instance node to the target node. Let
$descendent_{\mathcal{T}}(n,n^{\prime})$ hold if $n$ is a child of
$n^{\prime}$ or if there is an $n^{\prime\prime}$ s.t. $n$ is a child of
$n^{\prime\prime}$ and $descendent_{\mathcal{T}}(n^{\prime\prime},n^{\prime})$
holds. A node $n$ is called the root of tree $\mathcal{T}$,
$n=root_{\mathcal{T}}$, if there is no edge with $n$ as its target. A node $n$
has depth in $\mathcal{T}$ as defined as follows:

$\begin{array}[]{l}depth_{\mathcal{T}}(n)=\left\\{\begin{array}[]{ll}0&\mbox{if
}n=root_{\mathcal{T}}\\\ 1+depth_{\mathcal{T}}(n^{\prime})&\mbox{if $\langle
n^{\prime},n\rangle\in\mathcal{E}$}\end{array}\right.\end{array}$

Only instance nodes $\mathcal{N}_{I}$ have children, and thus only these nodes
may be collapsed in a merge. To perform a collapse, an instance node
$n_{I}\in\mathcal{N}_{I}$ is replaced with a merge node
$n_{M}\in\mathcal{N}_{M}$, such that the label for $n_{M}$ is the word
“MERGE”. The embeddings list for the merge node $n_{M}$ is a list of all
embeddings of $n_{I}$ and all descendents.

For an AMR tree $\mathcal{T}=\langle\mathcal{N},\mathcal{E},meta\rangle$, the
AMR tree obtained by merging instance node $n_{I}$ into new merge node $n_{M}$
is
$\mathcal{T}^{\prime}=\langle\mathcal{N}^{\prime},\mathcal{E}^{\prime},meta^{\prime}\rangle$
is defined as follows where
$J=\\{n^{\prime}|descendent_{\mathcal{T}}(n^{\prime},n_{I})\\}\cup\\{n_{I}\\}$,
$E_{m}=\\{\langle n,n_{M}\rangle|\langle n,n_{I}\rangle\in\mathcal{E}\\}$,
$l^{\prime}=``\mathrm{MERGE}"$, and $v^{\prime}=avg(v\in
meta(n),v\neq\emptyset|n\in J)$

$\begin{array}[]{l}\mathcal{N}^{\prime}=(\mathcal{N}\setminus
J)\cup\\{n_{M}\\}\\\
\mathcal{E}^{\prime}=E_{m}\cup(\mathcal{E}\cap(\mathcal{N}^{\prime}\times\mathcal{N}^{\prime}))\\\
meta^{\prime}(n)=\left\\{\begin{array}[]{ll}meta(n)&\mbox{ if
}n\in\mathcal{N}\\\ \langle l^{\prime},v^{\prime}\rangle&\mbox{ if $n$ is
$n_{M}$}\end{array}\right.\end{array}$

Not all possible merges are considered valid, however. We require that the
number of negations (an edge labeled :polarity pointing to a constant node
with label -) must be the same in $\mathcal{T}^{\prime}$ and $\mathcal{T}$.
This is demonstrated in Figure 11. Furthermore, if instance node
$n_{I}\in\mathcal{T}$ has a corresponding set of $k$ coreference nodes
$\\{n_{1}\ldots n_{k}\\}\subseteq\mathcal{N}_{C},k>0$, then
$\mathcal{T}^{\prime}$ must either contain both $n_{I}$ and all of the
corresponding $\\{n_{1}\ldots n_{k}\\}$ coreferences, or $n_{I}$ and
$\\{n_{1}\ldots n_{k}\\}$ must be removed. This corresponds to not allowing
coreferences to be broken by a merge as this could break the semantic meaning
of the AMR, as illustrated in Figure 10.

Figure 10: Invalid merge, since $b$ is coreferenced outside of the collapsed
region. The AMR is generated from the text “The boy wants the girl to believe
him”.

Figure 11: Invalid merge, since it removes a negation node (:polarity -). The
AMR is generated from the text “He sees a man who is not small”

We introduce additional parameters for controlling allowed merges futher.
$\tau_{M}$ is a threshold which controls the maximum merge width of the merged
embeddings in a single merge node $meta(n_{M})=\langle l,avg(v_{1}\ldots
v_{k})\rangle,k<\tau_{M}$. $\tau_{D}$ is a parameter which sets a minimum
depth at which merges are allowed in a tree $\mathcal{T}$, so
$depth_{\mathcal{T}}(n_{m})>\tau_{D}$.

We recursively generate all possible merged AMR trees such that these above
conditions are satisfied, including the original tree where no merges have
been performed. While this may raise computational complexity concerns if the
trees are large, in practice this was never a problem as AMR trees for a
single sentence take a negligible amount of time relative to theorem proving
and running the AMR parser. Table 2 shows statistics on the number of merge
trees available per sentence in the Social Chemistry dataset.

###  4.3 Converting AMR into logical formulae

To determine if a Social Chemistry ROT matches a SST, we need to turn both the
ROT and SST AMR into first-order logical formulae. Until now, we have treated
both the ROT and the SST identically, but when translating the ROT and SST
into logical formulae we need to treat them differently.

A Social Chemistry ROT is a general logical rule which can apply to any number
of situations. A ROT typically has the form of an action and a verdict on that
action. For instance, the ROT “It’s rude to hang up on someone.” would have
the verdict of “It’s rude”, and action “to hang up on someone”.

We use this knowledge of how ROTs are structured to modify the AMR to formulae
conversion so the result is in the form of an implication. We start with the
AMR to logical formulae algorithm described by (?), but we make modifications
to the quantification and the structure of the formulae. The (?) algorithm
wraps all instances with existential quantifiers, which assumes that each
sentence is a statement which can be true or false. However, natural language
is able to express much more than this, and we use our knowledge of what is
being expressed in the ROT and the SST to justify changing the formulae. For a
ROT, the formulae corresponds to an implication as mentioned above. For a SST,
the formulae describes grounded knowledge which we can query to see if the ROT
matches.

For instance, we can begin with the ROT AMR tree below for “It’s rude to hang
up on someone.”:

    (r / rude
        :ARG1 (h / hang-up
            :ARG2 (s / someone)))

From here, the algorithm from (?) would result in the following formulae:

$\displaystyle\exists R.(\textrm{rude}(R)\land\exists
H.(\textrm{:ARG1}(R,H)\land\textrm{hang-up}(H)$ $\displaystyle\hskip
14.22636pt\land\;\exists S.(\textrm{:ARG2}(H,S)\land\textrm{someone}(S))))$

We know that the ROT is a logical implication, so we remove the existential
quantification. Then, we replace the conjunction between the verdict,
$\textrm{rude}(R)$ with implication, and remove the linking
$\textrm{:ARG1}(R,H)$, since this linking is handled by the implication
itself. In addition, we swap the $R$ variable in $\textrm{rude}(R)$ with $H$,
as this is the target of $\textrm{:ARG1}(R,H)$. This results in the following
implication:

$\displaystyle\textrm{hang-
up}(H)\land\textrm{:ARG2(H,S)}\land\textrm{someone}(S)\to\textrm{rude}(H)$

As a final step, $\textrm{rude}(H)$ is simplified further to just
$\textrm{BAD}(H)$, resulting in the following final logical form:

$\displaystyle\textrm{hang-
up}(H)\land\textrm{:ARG2(H,S)}\land\textrm{someone}(S)\to\textrm{BAD}(H)$

Converting SST AMR to logical formulae is much simpler, as there is no need to
build an implication. Instead, the only changes necessary from the base
algorithm from (?) is to remove existential quantifiers, and to replace
variables with constants. Constants are used to indicate that the formulae
represents facts about items in the domain of discourse, rather than general
theorems. For instance, we can begin with the SST AMR tree below for “Hanging
up on my cousin”:

    (h / hanging
        :ARG2 (p / person
            :ARG0-of (h2 / have-rel-role
                :ARG1 (i / i)
                :ARG2 (c / cousin))))

Following (?), this becomes the following logic:

$\displaystyle\exists H.(\textrm{hanging}(H)\land\exists
P.(\textrm{:ARG2}(H,P)$ $\displaystyle\hskip
14.22636pt\land\;\textrm{person}(P)\land\exists H2.(\textrm{:ARG0}(H2,P)$
$\displaystyle\hskip 14.22636pt\land\;\textrm{have-rel-role}(H2)\land\exists
I.(\textrm{:ARG1}(H2,I)$ $\displaystyle\hskip
14.22636pt\land\;\textrm{i}(I))\land\exists
C.(\textrm{:ARG2}(H2,C)\land\textrm{cousin}(C)))))$

From here, we ground out the existential quantifiers and replace variables
with new constants, resulting in the following final logical form. Note that
in these examples, lowercase letters indicate a constant, and uppercase
letters indicate a variable.

$\displaystyle\mbox{hanging}(h)\land\mbox{:ARG2}(h,p)\land\mbox{person}(p)$
$\displaystyle\hskip 14.22636pt\land\;\mbox{:ARG0}(h2,p)\land\mbox{have-rel-
role}(h2)\land\mbox{:ARG1}(h2,i)$ $\displaystyle\hskip
14.22636pt\land\;\mbox{i}(i)\land\mbox{:ARG2}(h2,i)\land\mbox{cousin}(i)$

While the above process is shown only for the original AMR tree, this process
is also repeated for all merged AMR tree versions of the ROT and SST.

###  4.4 Reasoning

Once logical formulae are generated for a ROT and a SST, we use the Tensor
theorem prover library discussed in Section 3 to check if the SST applies to
the antecedent of the ROT. Since each ROT is transformed into an implication
with a consequent of either $\textrm{GOOD}(X)$, $\neg\textrm{GOOD}(X)$,
$\textrm{BAD}(X)$, or $\neg\textrm{BAD}(X)$, we can verify if the ROT matches
by querying these 4 verdicts one by one in the theorem prover. If a proof can
be found for any of these queries, then the ROT matches the SST.

For unification, we use a hybrid similarity function which can perform exact
string match or cosine similarity between embeddings, depending on the
predicates or constants being compared. This is shown in Algorithm 2, and is
used as the simFunc parameter in unification in Algorithm 1. Here, symbol1 and
symbol2 are either constants or predicates, and embedding1 and embedding2 are
their RoBERTa embeddings, but may be null.

Algorithm 2 Hybrid Similarity Function

Input: symbol1, embedding1
Input: symbol2, embedding2
Output: similarity

bothHaveEmbeds $\leftarrow$ embedding1 $\land$ embedding2

eitherIsMerge $\leftarrow$ “MERGED” $\in$ [symbol1, symbol2]

if eitherIsMerge then

if bothHaveEmbeds then

return $\frac{1}{2}(cos(\textrm{embedding1},\textrm{embedding2})+1)$

else

return 0.0

if symbol1 $=$ symbol2 then

return 1.0

if bothHaveEmbeds then

return $\frac{1}{2}(cos(\textrm{embedding1},\textrm{embedding2})+1)$

return 0.0

We treat the minimum similarity threshold above which a unification is allowed
to succeed as a tunable parameter of the theorem proving process, but find
that using a value near 0.9 performs well.

##  5 Experimental evaluation

To evaluate the performance of this approach, we use the SST and ROT from
Social Chemistry 101 as a test bed. For each sample, we turn the ROT and SST
into logical formulae as described in Section 4, and check if the the theorem
prover is able to prove the verdict of the ROT from the SST. To create
negative examples for evaluation, we randomly select a different ROT from the
dataset and check that the theorem prover should NOT be able to prove the
conclusion of that ROT when applied to the original SST. Based on these, we
calculate precision, recall, and f1 score.

For example, given the SST “being friends with my ex’s sister”, and the
corresponding ROT “You shouldn’t be friend with your ex’s family members”, the
ROT formulae has the following form:

$\displaystyle(\textrm{you}(Y)\land\textrm{:ARG1}(H,Y)\land\textrm{have-rel-
role}(H)\ldots)$ $\displaystyle\hskip 14.22636pt\to\neg\textrm{GOOD}(Y)$

Since the consequent of this ROT is $\neg\textrm{GOOD}(Y)$, if we can prove
$\neg\textrm{GOOD}(Y)$ using Tensor theorem prover after inputting the logical
formulae for the SST and the ROT, then this is considered a true positive.
Likewise, if we cannot find a proof of $\neg\textrm{GOOD}(Y)$ then this a
false negative.

We also pick an unrelated ROT, and correspondingly see if we can prove the
consequent of that ROT given the SST. For instance, if we randomly pick the
unrelated ROT “You should always give gifts to people for their birthday”, and
we are able to find a proof for the consequent of this ROT,
$\textrm{GOOD}(X)$, when applied to the unrelated SST “being friends with my
ex’s sister”, then this is considered a false positive. Likewise, if we are
unable to prove the consequent of the unrelated ROT, then this is considered a
true negative.

We randomly selected 10,000 samples consisting of a ROT and corresponding SST
from the social chemistry 101 dataset as a test set. Statistics related to the
AMR trees for the SSTs and ROTs in this dataset are shown in Table 1.

Dataset statistics
---
ROT AMR |  Mean |  Median |  Stdev
Instance nodes |  7.1 |  7 |  2.5
AMR depth |  5.2 |  5 |  1.3
Logic terms |  13.6 |  13 |  5.9
Merge Trees |  1.6 |  1 |  2.2
SST AMR |  Mean |  Median |  Stdev
Instance nodes |  8.6 |  8 |  3.1
AMR depth |  5.1 |  5 |  1.4
Logic terms |  18.4 |  17 |  9.1
Merge Trees |  2.4 |  1 |  14.1
Table 2: Statistics for the ROT AMR and SST AMR generated from the Social
Chemistry 101 dataset. Instance nodes refers to AMR instances (things like
b/boy in AMR), and logic terms is the number of literal in our generated
logical formulae before merging. Merge trees is the number of alterate merge
trees that can be generated from a SST or ROT AMR tree

Our method includes a hyperparameter for the vector similarity threshold at
which a unification is considered a success in the theorem prover. Raising
this threshold will increase precision but at the cost of recall. The
precision, recall, and f1 score as a function of similarity threshold is shown
for in Figure 12. We set a minimum merge depth of 1, and a maximum leaf merge
width of 6 nodes.

Figure 12: Precision, Recall, and F1 score, varying the similarity threshold

We also investigated the effect of the number of merged nodes on the
performance of this approach. For this experiment, we set the similarity
threshold to 0.925, and kept the remaining parameters identical to the
similarity threshold experiment. The results are shown in Figure 13.

Figure 13: Precision, Recall, and F1 score, varying the maximum merge size

Increasing the allowed number of nodes to be merged does not seem to have a
large effect on precision, but it does have a large impact on recall and f1
score up to a merge size of 6 nodes. Increasing the max node merge size allows
the prover to rely more on the embeddings to unify large chunks of the graph
rather than relying on the AMR structure, so it is encouraging to see these
merges don’t seem to have a negative effect on precision up to 6 nodes.

The merge algorithm we propose has a number of restrictions on when merges are
possible. The most common causes of merges not being allowed come from
crossing negation bounds, and merging across coreferences. However, these
restrictions also limit the performance of our method overall. We introduce a
metric called collapsability, which measures what portion of an AMR tree can
be collapsed following our merge algorithm. For a set of AMR trees
$\\{\mathcal{T}_{1}\dots\mathcal{T}_{n}\\}$,
minNodes($\\{\mathcal{T}_{1}\dots\mathcal{T}_{n}\\}$) and
maxNodes($\\{\mathcal{T}_{1}\dots\mathcal{T}_{n}\\}$) refer to the number of
nodes in the smallest and largest trees in the set, respectively.
Collapsability is thus defined below for an AMR tree $\mathcal{T}$:

$\textrm{collapsability}(\mathcal{T})=1-\frac{\textrm{minNodes}(\textrm{merges}(\mathcal{T}))-1}{\textrm{maxNodes}(\textrm{merges}(\mathcal{T}))-1}$

Collapsability is 0 if no merges are possible, and 1 if the entire AMR tree
can be collapsed into a single merged node. Collapsability is undefined if the
original AMR tree is just 1 node as that would cause $maxNodes$ to be 1, but
this never occurs in our dataset.

Figure 14 shows the results when considering only subsets of the dataset
bucketed by the collapsability percent of the ROT and the SST. The minimum
collapsability between these is used as the collapsability of the sample.

Figure 14: Precision, Recall, and F1 score when bucketing samples based on
what percentage of the AMR is allowed to be collapsed (merged)

In these results, we see the importance of allowing merges on the performance
of our method. When few merges are possible, the recall is very low, since the
majority of the AMR for ROT and SST do not have identical structure. As the
collapsability of samples increases, the recall increases dramatically as
well, even surpassing the precision at around 80%+ collapsability.
Interestingly, at very high collapsability the precision declines slightly,
likely since when so many nodes are merged together it is easier to get false
positives due to excessive averaging of embeddings. Regardless, the
performance increase from higher collapsability should motivate future work to
improve the merge algorithm to allow merging across coreference and negation.

##  6 Related work

The idea of using a semantic parser to turn natural language into a logical
format for reasoning has been used in (?; ?) for solving Winograd schema
challenges (WSCs) (?). Both of these papers use a semantic parser called
K-Parser (?) to parse WSC sentences and turn them into a logical form that can
be reasoned with. (?) then tries to find a theorem which will complete the WSC
and uses Answer set programming (ASP) (?) with the semantic graph output from
K-Parser to complete the WSC task. K-Parser is conceptually similar to AMR,
but is missing key features of AMR like negation and standardized roles, and
does not appear to be actively maintained. This approach does not use vector
embeddings or other neuro-symbolic approaches to improve the robustness of
matches, and thus cannot handle formulae which do not match the exact
structured output of K-Parser exactly.

There has been work done on relaxing the unification condition of theorem
provers to allow for differentiable unification, from which our Tensor Theorem
Prover library takes inspiration. Neural Theorem Provers (NTPs) (?) and Braid
(?) allow unification to return a score between 0 and 1. This is typically a
similarity score between vector embeddings corresponding to each predicate.
However, both of these libraries are back-chaining provers, and can only work
with Horn clauses which is insufficient to represent the full range of
formulae present in Social Chemistry ROTs. Also conceptually similar is (?),
which allows using embedding vector similarity to swap logical predicates
using a SAT solver.

The Social Chemistry 101 dataset includes a number of extra dimensions to the
data relating to ethics, psychology, and morality around samples in the
dataset. Trained models on this dataset tend to focus on generating judgements
conditioned on these attributes (?), or using the data as a component in other
more upstream datasets such as defeasible NLI (?) or focusing on the
intentions of actors in social situations (?). In all these cases, the dataset
is used for different purposes than matching ROTs with SSTs as is done in this
paper.

Perhaps most similar to our paper is (?). Here, a natural-language query is
turned into AMR, and the AMR is turned into logical formulae. However, these
formulae are then used to query a pre-existing knowledge-base using SPARQL
rather than deriving the knowledge-base as well from a natural language
sentence. Furthermore, the focus is on entity recognition and data retrieval
rather than social commonsense reasoning, and does not use contextual word
embedding similarity for neuro-symbolic reasoning.

##  7 Discussion

In this paper, we present a novel system for taking ROTs in natural language
and reasoning with them using a neuro-symbolic theorem prover. Our
contributions in this paper include: (1) an approach for merging and
collapsing AMR nodes for increased flexibility and robustness during
reasoning; (2) a hybrid similarity metric which mixes string matching and
embedding cosine similarity; (3) a modified version of AMR to logic conversion
for working with ROT implications; and (4) our evaluation method using Social
Chemistry 101. Furthermore, this paper introduces Tensor Theorem Prover, our
implementation of a resolution-based neuro-symbolic theorem prover and AMR
Logic Converter, our library for converting AMR to first-order logic.

While this paper deals only with a single SST and ROT at a time, this can be
extended in the future to allow multi-hop reasoning by combining multiple ROTs
and additional background knowledge during the reasoning process.

The theorem proving process in this paper is differentiable. In future work
the embeddings used for semantic similarity can be further trained via
backpropagation.

A particular weakness of the current approach is that it does not have a good
way to deal with antonyms during the solving process. For example, if a ROT
references “gifts that aren’t too big”, it would be reasonable that a SST
describing a ”small gift” to match, since “big” and “small” are opposites. A
formula like $\textrm{big}(X)\to\neg\textrm{small}(X)$ could be injected
dynamically to address this.

Another area in which this approach could be improved is dealing with multiple
sentences. This would require working out coreferences between entities in the
various sentences, but doing so should allows this approach to work with
paragraphs of text instead of single sentences.

A final area of future work is be to develop a version of the merge algorithm
which can allow merging nodes across coreferences and/or negation. Our
evaluation shows better results when more merging is possible, so a clear
direction for improvement is to develop a version of the merge algorithm which
has fewer restrictions on allowed merges.

The method described in this paper is a stepping-stone on the path to building
systems capable of social reasoning using an interpretable, neuro-symbolic
approach. Systems like this will naturally be useful anywhere commonsense
social understanding is necessary, from interacting with users directly, to
summarizing human narrative, to giving recommendations on social situations.
In all these cases, and especially when dealing with sensitive social topics,
it is important for future AI systems to be interpretable and transparent, and
for their reasoning to be debuggable and editable.

## References

  * Abid, Farooqi, and Zou 2021 Abid, A.; Farooqi, M.; and Zou, J.  2021\.  Persistent anti-muslim bias in large language models.  In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, 298–306.
  * Banarescu et al. 2013 Banarescu, L.; Bonial, C.; Cai, S.; Georgescu, M.; Griffitt, K.; Hermjakob, U.; Knight, K.; Koehn, P.; Palmer, M.; and Schneider, N.  2013\.  Abstract meaning representation for sembanking.  In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, 178–186.
  * Bolukbasi et al. 2016 Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and Kalai, A. T.  2016\.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  Advances in neural information processing systems 29\.
  * Bos 2016 Bos, J.  2016\.  Expressive power of abstract meaning representations.  Computational Linguistics 42(3):527–535.
  * Devlin et al. 2018 Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Drozdov et al. 2022 Drozdov, A.; Zhou, J.; Florian, R.; McCallum, A.; Naseem, T.; Kim, Y.; and Astudillo, R. F.  2022\.  Inducing and using alignments for transition-based amr parsing.  arXiv preprint arXiv:2205.01464.
  * Emelin et al. 2020 Emelin, D.; Bras, R. L.; Hwang, J. D.; Forbes, M.; and Choi, Y.  2020\.  Moral stories: Situated reasoning about norms, intents, actions, and their consequences.  arXiv preprint arXiv:2012.15738.
  * Ertel 2018 Ertel, W.  2018\.  Introduction to artificial intelligence.  Springer.
  * Forbes et al. 2020 Forbes, M.; Hwang, J. D.; Shwartz, V.; Sap, M.; and Choi, Y.  2020\.  Social chemistry 101: Learning to reason about social and moral norms.  arXiv preprint arXiv:2011.00620.
  * Gelfond and Lifschitz  Gelfond, M., and Lifschitz, V.  The stable model semantics for logic programming. icslp, 1988.
  * Goodman 2020 Goodman, M. W.  2020\.  Penman: An open-source library and tool for AMR graphs.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 312–319.  Online: Association for Computational Linguistics.
  * Hong et al. 2022 Hong, S. J.; Bennett, B.; Clymo, J.; and Álvarez, L. G.  2022\.  Karaml: Integrating knowledge-based and machine learning approaches to solve the winograd schema challenge.  In AAAI Spring Symposium: MAKE.
  * Hovy et al. 2006 Hovy, E.; Marcus, M.; Palmer, M.; Ramshaw, L.; and Weischedel, R.  2006\.  Ontonotes: the 90% solution.  In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, 57–60.
  * Hunter 2022 Hunter, A.  2022\.  Understanding enthymemes in deductive argumentation using semantic distance measures.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 5729–5736.
  * Kalyanpur, Breloff, and Ferrucci 2022 Kalyanpur, A.; Breloff, T.; and Ferrucci, D. A.  2022\.  Braid: Weaving symbolic and neural knowledge into coherent logical explanations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 10867–10874.
  * Kapanipathi et al. 2020 Kapanipathi, P.; Abdelaziz, I.; Ravishankar, S.; Roukos, S.; Gray, A.; Astudillo, R.; Chang, M.; Cornelio, C.; Dana, S.; Fokoue, A.; et al.  2020\.  Leveraging abstract meaning representation for knowledge base question answering.  arXiv preprint arXiv:2012.01707.
  * Kim et al. 2018 Kim, B.; Wattenberg, M.; Gilmer, J.; Cai, C.; Wexler, J.; Viegas, F.; et al.  2018\.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, 2668–2677.  PMLR.
  * Kingsbury and Palmer 2002 Kingsbury, P. R., and Palmer, M.  2002\.  From treebank to propbank.  In LREC, 1989–1993.
  * Lee et al. 2021 Lee, Y.-S.; Astudillo, R. F.; Hoang, T. L.; Naseem, T.; Florian, R.; and Roukos, S.  2021\.  Maximum bayes smatch ensemble distillation for amr parsing.  arXiv preprint arXiv:2112.07790.
  * Levesque, Davis, and Morgenstern 2012 Levesque, H.; Davis, E.; and Morgenstern, L.  2012\.  The winograd schema challenge.  In Thirteenth international conference on the principles of knowledge representation and reasoning.
  * Liu et al. 2019 Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V.  2019\.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * Lu et al. 2020 Lu, K.; Mardziel, P.; Wu, F.; Amancharla, P.; and Datta, A.  2020\.  Gender bias in neural natural language processing.  Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday 189–202.
  * Lundberg and Lee 2017 Lundberg, S. M., and Lee, S.-I.  2017\.  A unified approach to interpreting model predictions.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Advances in Neural Information Processing Systems 30. Curran Associates, Inc.  4765–4774.
  * Mostafazadeh et al. 2016 Mostafazadeh, N.; Chambers, N.; He, X.; Parikh, D.; Batra, D.; Vanderwende, L.; Kohli, P.; and Allen, J.  2016\.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 839–849.
  * Ribeiro, Singh, and Guestrin 2016 Ribeiro, M. T.; Singh, S.; and Guestrin, C.  2016\.  ”why should I trust you?”: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, 1135–1144.
  * Robinson 1965 Robinson, J. A.  1965\.  A machine-oriented logic based on the resolution principle.  Journal of the ACM (JACM) 12(1):23–41.
  * Rocktäschel and Riedel 2017 Rocktäschel, T., and Riedel, S.  2017\.  End-to-end differentiable proving.  arXiv preprint arXiv:1705.11040.
  * Rudinger et al. 2020 Rudinger, R.; Shwartz, V.; Hwang, J. D.; Bhagavatula, C.; Forbes, M.; Le Bras, R.; Smith, N. A.; and Choi, Y.  2020\.  Thinking like a skeptic: Defeasible inference in natural language.  In Findings of the Association for Computational Linguistics: EMNLP 2020, 4661–4675.
  * Sharma et al. 2015a Sharma, A.; Vo, N.; Aditya, S.; and Baral, C.  2015a.  Identifying various kinds of event mentions in k-parser output.  In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, 82–88.
  * Sharma et al. 2015b Sharma, A.; Vo, N. H.; Aditya, S.; and Baral, C.  2015b.  Towards addressing the winograd schema challenge—building and using a semantic parser and a knowledge hunting module.  In Twenty-Fourth International Joint Conference on Artificial Intelligence.
  * Vaswani et al. 2017 Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I.  2017\.  Attention is all you need.  In Advances in neural information processing systems, 5998–6008.
  * Wolf et al. 2020 Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al.  2020\.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 38–45.