Further author information: (Send correspondence to R. Bowens-Rubin)
R.B.R.: E-mail<EMAIL_ADDRESS>
# Performance of Large-Format Deformable Mirrors Constructed with Hybrid
Variable Reluctance Actuators II: Initial Lab Results from FLASH

Rachel Bowens-Rubina  Daren Dillona  Philip M. Hinza  Stefan Kuiperb aUC Santa
Cruz  1156 High St  Santa Cruz CA  USA;
b TNO Technical Sciences  Delft  The Netherlands

###### Abstract

Advancements in high-efficiency hybrid variable reluctance (HVR) actuators are
an enabling technology for building the next generation of large-format
deformable mirrors, including adaptive secondary mirrors. The Netherlands
Organization for Applied Scientific Research (TNO) has developed a new style
of hybrid variable reluctance actuator that requires approximately seventy-
five times less power to operate as compared to the traditional style of
voice-coil actuators. We present the initial performance results from
laboratory testing of TNO’s latest 19-actuator prototype deformable mirror,
FLASH. We report the actuator cross-coupling, linearity, hysteresis, natural
shape flattening, and drift as measured with a Zygo interferometer and a set
of four capacitive sensors. We also present results of the dynamic performance
of the FLASH on sub-millisecond timescales to estimate the limits of this
technology for high-contrast imaging adaptive optics. We confirm that this
technology has strong potential for use in on-sky adaptive secondary mirrors
without the need for active cooling.

###### keywords:

Adaptive Optics, Adaptive Secondary Mirrors, Deformable Secondary Mirrors,
Large-Format Deformable Mirrors, TNO, Hybrid Variable Reluctance Actuators

##  1 INTRODUCTION

Adaptive secondary mirrors have been used successfully at telescopes like the
Large Binocular Telescope [Esposito2010], Magellan [Close2018], the VLT
Observatory [Biasi2012], and the MMT Observatory [Wildi2003]. However, these
adaptive secondary mirror systems have historically been complex to build and
operate. The listed adaptive secondary mirrors are constructed using power
intensive voice-coil actuators that require a dedicated cooling system to
dissipate heat. They also require closed-loop capacitive sensors to monitor
the mirror position and correct for nonlinearity and hysteresis in the
actuator displacements. [Hinz2016] The mirror facesheet fabrication has
traditionally required the grinding of a large piece of glass into a thin
glass sheet that is flexible enough to be shaped by the actuators.

The Netherlands Organization for Applied Scientific Research (TNO) has made
significant breakthroughs in large-format deformable mirror technology that
could enable adaptive secondary mirrors to become simpler and less costly to
operate. The key advancement is a new style of hybrid variable reluctance
(HVR) actuator that is $\sim$75 times more efficient than traditional voice-
coil actuators (HVR actuator efficiency [Kuiper2018] $=38N/\sqrt{W}$; MMT and
LBT actuator efficiency [Riccardi2003] $=0.5N/\sqrt{W}$). The low required
power offers a pathway to eliminate the actuator cooling system, miniaturize
the system, and pack the actuators at higher density. TNO is also developing
solutions, in conjunction with partners and the Fraunhofer Institute and the
University of California Santa Cruz, that could simplify the facesheet
fabrication process through hot-forming thin, commercially-available glass
sheets.

Adaptive secondary mirrors using the HVR technology are being proposed for use
in three telescopes: the University of Hawaii 2.2-meter (UH2.2m), the
University of California Observatories Automated Planet Finder Telescope
(APF), and the W.M. Keck Observatory (Keck). The UH 2.2m ASM will be the first
on-sky operation of the TNO technology and is expected to be commissioned
during 2022 [Chun2020]. ASM’s for the APF and Keck are in the proposal phase.
More information regarding the science case and initial design concepts for
the APF ASM can be found in Hinz et al 2020 [Hinz2020]. Additional information
regarding the Keck ASM can be found in the Keck White paper, _An adaptive
secondary mirror concept for the W.M. Keck Observatory based on HVR
technology_ by Hinz et al.

While the TNO large-format mirrors show great potential to make advanced
adaptive optics systems accessible to a wide variety of observatories, they
have yet to be proven to the astronomical community. This paper presents the
performance testing of TNO’s fourth deformable-mirror prototype, the First Lab
for adaptive optics Adaptive Secondary Holophote (FLASH). It is the second
SPIE paper in a series, following Bowens-Rubin et al. 2020 which reported the
performance testing results from TNO’s third large-format deformable mirror
(DM3). [BowensRubin2020]

##  2 TECHNOLOGY OVERVIEW

###  2.1 19-Actuator FLASH Prototype

The FLASH is the fourth prototype large-format deformable mirror built by TNO
(Figure 1). The FLASH was commissioned by the University of California Lab for
Adaptive Optics (UCSC-LAO) to verify the performance of the current generation
of TNO technology before the construction of the UH2.2m ASM. It was
constructed in 2020 and delivered to the UCSC-LAO in January of 2021. The
mirror is a companion to the UH 2.2m adaptive secondary mirror [Chun2020] and
shares the same facesheet thickness (3.3mm), actuator spacing (39mm), and
layout geometry (hexapolar). The UH2.2m ASM and FLASH share the same
generation of TNO HVR actuator, which is described in more detail Section 2.2.
Figure 2 shows the internal designs of the FLASH, and Table 1 summarizes the
FLASH design specifications.

FLASH is controlled through a 64 analog output from a Real Time Linux
environment. The user can control the shape of the mirror’s surface by
specifying the value of the current applied to each of the 19 actuators in an
RDA interface in Matlab running on an office PC. The actuators are plugged
into a breadboard. The breadboard and control electronics are connected by
ethernet cables. Each ethernet cable controls the signal to four actuators.
The electronics can produce an applied voltage of up to +10V across each
actuator. 111There is a correction for Bowens-Rubin et al. 2020 related to the
voltage applied across each actuator. It was discovered in 2021 that a driver
setting had previously limited the electronics to producing +5V during the DM3
testing instead of +10V. This caused all currents reported for the DM3 to be
stated as doubled their actual value. If comparing the results from the DM3 in
Bowens-Rubin et al. 2020 to the FLASH, all currents reported for the DM3
should be halved.

Table 1: FLASH Specifications Total Diameter | 160mm
---|---
Facesheet Diameter | 150mm
Number of Actuators | 19
Actuator Spacing | 39mm
Actuator Generation (year) | 2020
Actuator Layout Geometry | Hexapolar
Facesheet Thickness | 3.3mm
Facesheet Material | Borofloat
Control Electronics Used for Testing | Analog
Available Capacitive Sensor Locations | 13

Figure 1:  FLASH 19-Actuator Large-Format Deformable Mirror. (Left) The FLASH
mounted on the testbench at the UC Santa Cruz Lab for Adaptive Optics. It is
pictured here with its protective mirror-facesheet cover on. (Right) Back side
of the FLASH mirror with the actuators and electronics breadboard.
---
Figure 2:  CAD Model of the FLASH Large-Format Deformable Mirror. FLASH is
designed with an aluminium backing structure which supports the mirror
facesheet, the actuators, and the electronics breadboard.

###  2.2 2020 Design of the TNO Hybrid Variable Reluctance Actuator

The FLASH is constructed with 19 hybrid-variable reluctance (HVR) actuators
(Figure 3). The HVR actuators were designed by TNO and constructed by VDL-ETG
in the Netherlands. The 2020 actuator design was based on the actuators
developed for TNO’s 57-actuator DM prototype (DM3) [Kuiper2018]. While
previous generations of the HVR actuators were made as multiple units in a
strip, the 2020 actuators are individual freestanding units that can be bolted
on to a supporting body using two bolts at any spacing greater than the
actuator’s diameter. Dust caps were added to the 2020 generation to prevent
dust from reaching vital parts inside the actuators. Provisions were also made
in the actuator design to allow quick replacement in the event of an actuator
failure. The actuator replacement procedure was tested during the construction
of the FLASH, and it was completed in under four hours.

The force output of the 2020 HVR actuators is $\pm$8N over a linear range
(99.5%) before entering in magnetic saturation and can go up to a maximum
force of $\pm$14N when fully magnetically saturated. Modifications from the
previous generations were made to the internal mechanical hinges and the
internal lever-arm to match the different dynamic characteristics of the
deformable mirror system. This internal lever-arm transmits the force produced
by the actuator to the strut that interfaces with the facesheet with an
amplification factor of 2.6, which results in a force linear range of $\pm$21N
on the facesheet. The modeled free displacement range of the actuator is
35-40µm (PV), and the modeled inter-actuator stroke is $\sim$5µm (PV). The
average resistance measured for each actuator was $R=3.68\pm 0.07\Omega$, and
the average inductance measured for each actuator was $I=14.85\pm 0.07$mH.

Figure 3: TNO’s 2020 version of the HVR Actuator (Left) CAD model of the TNO
HVR Actuator designed for FLASH and the UH2.2m adaptive secondary mirror.
(Right) A picture of the realized FLASH actuator from the batch manufactured
in November 2020.

###  2.3 Facesheet Glass-Slumping Experiments

The facesheet for FLASH was constructed from a 3.3mm-thickness, 160mm-diameter
borofloat wafer supplied by Swift Glass of Elmira, NY. A set of ten wafers
were supplied and tested for flatness using a Mercury vapor illumination
screen and a $\lambda$/10 reference flat. The wafers all showed 5.5-9.5$\mu$m
of wavefront error, which appeared as convex curvature on one side. A similar
amount of concave curvature with edge rolloff was visible on the other side.
The UCSC-LAO attempted to flatten these wafers further via heating in a glass
kiln. This is part of a larger program to eventually produce curved facesheets
for large format deformable mirrors using initially flat borofloat sheets and
a hot forming process, similar to approaches developed for X-ray mirrors
(REF’s).

These initial tests used a fused silica reference flat (the same one used for
measuring flatness) as the bottom mould for the facesheet. Improved figure was
attempted by various tests that used the self weight of the shell to slump, as
well as additional fused silica top weights. While reduction of the curvature
was achieved, the resulting facesheets had several waves of error on spatial
scales of 20-30mm remaining. This was judged to be more problematic for FLASH
operation than the initial larger, but smoother wavefront error on the
unslumped shells. Therefore the final facesheet selected for integration was
one that was not heated and was shown to introduce $2.8\mu$m peak-to-valley
convex curvature of surface error.

The experience with hot forming a facesheet for FLASH lead us to several
conclusions for future slumping work. Primarily, it was found that there is a
very narrow range of temperatures for which the shell becomes soft enough to
take on the shape of the mould but is cool enough to not stick and destroy the
mould and the facesheet. A release agent (such as boron nitride) might help
with this but may lead to high spatial-frequency structure being imprinted on
the glass. Top loading of the shell can help when using cooler temperatures,
but then the accuracy of the surface for the top weight becomes important.
Further, cleanliness of the bottom mould is critical to avoid localized bumps
from any particles caught between the mould and the slumped facesheet. Since
heating is typically carried out in a glass kiln that uses ceramic bricks for
insulation, a clean environment will require a more complicated setup. As we
scale up such an approach, a significant portion of the cost becomes making
the precision mould.

The above process might be simplified by removing the mould and letting the
facesheet slump freely, using a ring slightly smaller than the diameter of the
facesheet for bottom support. Using no top weight results in a curve with
higher order radial terms once the glass is heated. A suitably placed ring top
weight can provide loading on the facesheet, such that the final curve
approximates the desired conic shape to within the precision needed for the
actuators to remove the residual error. The UCSC-LAO is exploring this
approach for future large-format curved facesheets.

The FLASH facesheet was aluminium coated on the front and back surfaces. The
back was coated to minimize the optical warping due to having a mixed material
surface. The back coating also provides a conductive surface for the
capacitive sensors to measure against. The areas on the back of the facesheet
that would be bonded to the actuator struts were not coated. Circles of
$\diameter$9.5mm kapton tape were adhered before coating to prevent these
areas from being aluminized.

###  2.4 Zygo Interferometer Measurement Setup

The large-format deformable mirror testbed at UCSC-LAO utilizes a Zygo
interferometer that is controlled using Metropro Software (Figure 4). It can
measure the shape of the mirror’s surface to a precision of 0.6nm of surface
error within $\diameter$140mm. Measurements can be collected at a maximum
speed of one image per second.

The Zygo interferometer is not capable of measuring the piston changes in an
optical system. To calibrate for piston, we measure the median value of the
image and subtract it. We filter the empty pixels (listed in the Zygo data as
’NaN’ values) by intrapolating the values using nearby pixels. We convert the
measurements reported by the Zygo interferometer in units of ‘Zygos’ to units
of nanometers of surface error by

$PhaseData[nm]=PhaseData[zygos]\frac{\lambda S}{R}$ (1)

where the wavelength of the laser in the Zygo interferometer is
$\lambda=632.8$nm, the geometry factor is $S=0.5$, and the resolution setting
(high) is $R=32768$ (Zygo interferometer manual: zygo-0347M, 12-5 p447). We
analyze the Zygo interferometer data using Matlab and load the images using
the LoadZygoBinary function authored by Massimo Galimberti.

---
Figure 4:  Large-Format Deformable Mirror Testbench at the UCSC Lab for
Adaptive Optics. The testbench utilizes a Zygo Interferometer with MetroPro
software to measure the surface shape. A beam expander is used so that the
majority of the surface of the FLASH can be measured in a single image (up to
$\diameter$140mm).

###  2.5 Capacitive Sensor Measurement Setup

FLASH is the first large-format deformable mirror from TNO to incorporate
space for an internal capacitive sensor system. While the Zygo interferometer
measurement setup can be used to gain an understanding of the full shape of
the mirror’s surface, it cannot collect time series data. The capacitive
sensor system can provide data with a readout speed of 3906Hz over a working
range of 50$\mu$m. This fast frame rate is needed for evaluating if this
technology is suitable for use in fast adaptive optics correction and high-
contrast imaging. However, the capacitive sensor data are limited spatially,
capturing displacement information only in the small coverage area that they
are located. To determine if having limited access to a few points on the
surface is an adequate proxy for the full mirror behavior, our linearity,
hysteresis, and drift testing were performed with both the Zygo interferometer
measurement setup and capacitive sensor systems side-by-side. If successful,
future testing when an interferometer is prohibitively difficult to use may
rely solely on the capacitive sensor system to understand the behavior of the
system.

The FLASH backing structure has thirteen holes available for capacitive
sensors to be mounted. Four Micro Epsilon CS005 capacitive sensors were
internally mounted for testing the FLASH at the UCSC-LAO (Figure 5, left). The
sensors were run using the Micro-epsilon capaNCDT 6200 controller and the
Micro-epsilon web interface software in an open loop system.

The mounts to position the capacitive sensors directly behind the mirror
facesheet were designed by UC Santa Cruz. They were manufactured out of
aluminum and are are 60mm in length. The capacitive sensor is glued in place
on the mounting tube (Figure 5, right). To install the capacitive sensor
mounts in the FLASH backing structure, the actuators were unplugged from the
breadboard and breadboard was removed and reinstalled. The capacitive sensor
mount installation was completed in approximately three hours.

Figure 5:  FLASH Capacitive Sensors. (Left) Four capacitive sensors were added
to the internal structure of FLASH for testing at UCSC-LAO. The sensors were
placed next to the actuators that were used in linearity and hysteresis
testing. (Right) A capacitive sensor inside its mount before it was installed.

##  3 FLASH PERFORMANCE TESTING RESULTS

###  3.1 Influence Function Profile and Actuator Cross-Coupling

_Measurement setup: Zygo Interferometer_

The influence function of each actuator was measured using an individual
actuator poke test. A $+50$mA current was applied in an automated measurement
sequence to displace each of the 19 actuators one-by-one. The displacement of
these pokes corresponded to approximately 2200nm peak-to-valley. Eighteen
actuators responded to the positive current with a positive displacement, and
one actuator responded with a negative displacement. Seven of the actuators
had profiles that were fully visible in the field of view of our Zygo
interferometer setup: the center actuator (Actuator 4) and the six actuators
in the middle ring (Actuators 2, 6, 8, 13, 15, and 17). An example of the
influence function measured for Actuator 4 is shown in Figure 6.

The actuator cross-coupling was measured to be 34.2$\pm 1\%$. This value was
determined using the influence functions from the seven actuators that were
fully visible in the Zygo field of view (Figure 7). The location of the center
of each of the seven actuators in the image was found using a Gaussian fit. A
cross-section of the influence function was then taken in the row and column
directions, emanating from the actuator center location. The fourteen cross
sections were then normalized and stacked. The average distance between the
center actuator to each actuator in the inner ring was then determined using
the location of the centers of the actuators and measured to be 39.4 $\pm$
0.4mm. The value of the fourteen normalized cross-sections at the width
location of average actuator separation were averaged to yield the cross-
coupling measurement. This measurement is consistent with the cross-coupling
value expected from the pre-fabrication modeling done by TNO.

---
Figure 6:  Example Influence Function: An applied voltage of +50mA to Actuator
4 resulted in an actuator displacement of 2210nm. These images are scaled with
the x-y axis in millimeters and the color axis and z-displacement in
nanometers. The natural shape of the FLASH mirror was subtracted, so only the
displacement due to the actuator poke is visible.

Figure 7:  Actuator Cross-Coupling. (Left) Seven actuators were used to
calculate the cross-coupling between actuators. The centers of these actuators
were fully visible within the frame of the Zygo interferometer. (Right) To
measure the cross-coupling the actuator, 14 cross sections from the column and
row direction from each visible actuator were aligned at their centers and
normalized. The average distance between the middle ring actuators to the
center was 39.4 $\pm$ 0.4mm. This corresponds to a cross-coupling measurement
of 34.2 $\pm$ 1%.

###  3.2 Natural-Shape Surface Flattening

_Measurement setup: Zygo Interferometer_

The natural shape of the FLASH mirror facesheet has a peak-to-valley that is
5339.1nm with a 1158.8nm RMS variation (within $\diameter$140mm). To bring the
surface as flat as possible (as measured by surface RMS), a set of flattening
currents was determined using an iterative process. At each step, an image was
taken using the Zygo interferometer. The next desired position for the
actuators to reproduce in the iteration ($i$) was set to the negative of the
last image taken ($S_{i}$). The next set of currents ($F_{i}$) to produce the
desired shape ($S_{i}$) were calculated using the influence functions matrix
($P$) through the Moore-Penrose inverse:

$F_{i}=(P^{T}P)^{-1}P^{T}S_{i}.$ (2)

The currents calculated for each iteration were summed with the previous
currents applied to find the new values to be applied
($F=\sum_{1}^{i=n}F_{i}$).

On April 5th 2021, fourteen iterations of flattening were completed to
calculate the optimal set of flattening currents (Figure 8, top). The majority
of the variation was removed after four iterations. The flattest shape was
found on iteration thirteen which had an RMS $=14.7$nm (within
$\diameter$140mm) with a peak-to-valley of 260.0nm. The surface RMS was
reduced by a factor of 79. Figure 8 (bottom) shows the natural shape of the
mirror alongside the shape after this flattening. The total current needed to
hold this shape across the 19 actuators was 323.9mA, averaging 17.0 $\pm$
4.7mA per actuator. The required power was 7.3mW (0.4mW/actuator). This result
confirms that the use of a passive air cooling will be appropriate for the
UH2.2m adaptive secondary mirror.

–

Figure 8: Natural-Shape Surface Flattening. (Top) _RMS of the Mirror Surface
Shape during Flattening Iterations_. The best set of flattening currents was
determined through an iterative process of 14 steps. (Bottom left) _FLASH
Natural Surface Shape_. Unpowered, the shape of the TNO FLASH mirror surface
has a peak-to-valley that is 5339nm ($RMS=1159$nm within ⌀140mm). (Bottom
right) _Surface Shape after Applied Flattening_. After flattening, the surface
shape was brought down to 15nm RMS with a peak-to-valley of 261nm. The
colorbar is reported in units of nanometers.

###  3.3 Linearity

_Measurement setup: Zygo Interferometer & Capacitive Sensors_

Linearity testing was performed by poking actuators individually (using
Actuators 2, 4, 6, and 13) and as a group simultaneously (all actuators). The
individual actuator test measured the high spacial order performance. The all-
actuator test demonstrated the capabilities of the HVR actuators if minimal
inter-actuator and shell deformation forces exist. The actuators were run
through two current patterns which applied positive and negative currents from
$\pm 150$mA and $\pm 300$mA (Figure 9). The first current pattern probed the
HVR actuators quoted linear range, from 0 to $\pm$150mA. The second pattern
spanned the maximum possible working range of the HVR actuators when
controlled with the analog electronics, from 0 to $\pm$300mA.

Four runs were completed for each configuration. Zygo interferometer data and
capacitive sensor data were collected simultaneously during each run. The Zygo
interferometer data was used to analyze the individual actuator tests because
the capacitive sensors are not able to detect the height of the mirror surface
at the actuator’s center. The capacitive sensors were used to analyze the all-
actuator tests because the Zygo interferometer cannot measure a change in
piston.

Table 2 summarizes the key results from our linearity testing. Our testing
reveals that within the stated linear range of the actuator, the high-order
linearity is $95.2\pm 0.8$% with a max residual of 0.2$\mu$m surface error
(Figure 10). The linearity measured when all-actuators were moved
simultaneously was $96.8\pm 0.5$% with a max residual of 2$\mu$m (Figure 11).
The average displacement measured for the all-actuator movement was roughly
2.5 times larger than the average displacement measured for the individual
actuator movements. All testing showed an asymmetry between positive and
negative applied currents, with larger movements available when positive
currents are applied.

Figure 9:  Linearity Testing (Left) The linearity testing was completed using currents varying across the actuators linear range ($\pm 150$mA) and full working range ($\pm 300$mA). (Right) The four capacitive sensors were used to collect real time displacement data during the linearity testing. This example data set was taken during an individual actuator run of Actuator 4.  Table 2: FLASH Linearity Results Summary | Individual act/
---
All actuators
| Current
---
Range
| Actuators
---
Used
| Average
---
Displacement
| Max
---
Displacement
| Percent
---
Linearity
Individual |  | Linear
---
($\pm$150mA)
2,4,6,13 |  $46.5\pm 1.4$ nm/mA |  |  $-6.8\mu$m @ $\pm$-150mA
---
$6.8\mu$m @ $\pm$+150mA
$95.2\pm 0.8$%
| Working
---
($\pm$300mA)
4 | - |  | -7.2$\mu$m @ -300mA
---
9.4$\mu$m @ +300mA
-
All |  | Linear
---
($\pm$150mA)
all |  $113.9\pm 1.0$ nm/mA |  | -15.1$\mu$m @ -150mA
---
17.0$\mu$m @ +150mA
$96.8\pm 0.5$%
| Working
---
($\pm$300mA)
all | - |  | -15.3$\mu$m @ -300mA
---
20.5$\mu$m @ +300mA
-

Figure 10:  Linearity Measurements from the Zygo Interferometer for the
Actuators Poked Individually. (Left) The linearity measured from the linear-
range test ($\pm 150$mA) for Actuators 2, 4, 6 and 13 was $lin=95.2\pm 0.8$%.
The average displacement was $46.5\pm 1.4$nm/mA. (Right) The displacement
measured by the Zygo interferometer data from five runs of Actuator 4 across
its working range (blue) was averaged and fit with a zero-intercept linear fit
(red). The working range-test confirms that the linear region of the actuators
is approximately $\pm 150$mA (gray), but the displacements are asymmetric
between the positive and negative applied currents.

Figure 11:  Linearity Measurements from the Capacitive Sensors when All
Actuators were Moved Synchronously (Left) The displacement measured by the
capacitive sensor data from three runs (blue) was averaged and fit with a
zero-intercept linear fit (red). The linearity measured from the linear-range
testing ($\pm 150$mA) when all actuators were moved simultaneously was
$lin=96.8\pm 0.5$%. The average displacement was $113.9\pm 1.0$ nm/mA. (Right)
The working-range test ($\pm 300$mA) when all actuator were moved
simultaneously demonstrates that the actuator range cannot extend beyond TNO’s
quoted linear range ($\pm 150mA$), which is shaded in gray.

###  3.4 Hysteresis

_Measurement setup: Zygo Interferometer & Capacitive Sensors_

Actuator displacement varies depending on the previous position of the
actuator. To quantify this effect, hysteresis testing was performed using four
actuators (2, 4, 6, and 13) poke individually, stepping through five
independent hysteresis loops. Each loop was done in eighty steps between
currents of $\pm$5mA, $\pm$10mA, $\pm$25mA, $\pm$50mA, $\pm$100mA, $\pm$150mA.
Three runs were performed for each actuator at $\pm$5mA, $\pm$10mA, $\pm$150mA
and five runs were performed for each actuator at $\pm$25mA, $\pm$50mA,
$\pm$100mA. An example of one of these hysteresis loops is shown in Figure 12.

We quantify the hysteresis using the definition:

$Hyst(\%)=100\left(\frac{|S3-S1|}{|S2-S4|}\right)$ (3)

where $S1$ and $S3$ are the displacements measured at an applied current of
zero and $S2$ and $S4$ are the displacements measured at the maximum and
minimum current applied.

Hysteresis data were collected simultaneously with the Zygo interferometer and
capacitive sensors (Figure 12). The average percent hysteresis was measured
using the data from the $\pm$25mA, $\pm$50mA, $\pm$100mA runs to be $1.80\pm
0.13$% by the Zygo interferometer and $1.93\pm 0.04$% by the capacitive
sensors. This calculation did not include the values measured outside the
20mA-120mA range to assure the hysteresis measurements were taken in the
linear range of the actuators. There was no dependency on the actuator tested
and the measured hysteresis value.

Figure 12: Hysteresis testing results. (Top) _Example Hysteresis Loop_ from
Actuator 4 with max current of 100mA. (Bottom) _Percent Hysteresis_. The data
point at each current loop value are an average of the measurements from each
the four actuators tested. The average percent hysteresis was measured to be
$1.80\pm 0.13$% using the Zygo data and $1.93\pm 0.04$% using the capacitive
sensors within the definite linear range of the actuators (20mA - 120mA).

###  3.5 Drift

_Measurement setup: Zygo Interferometer & Capacitive Sensors_

Nine drift tests were performed for a duration of 8.5 hours with the goal of
determining if the FLASH could hold its surface shape consistently across many
hours of applied current. Table 3 lists the type of current pattern applied
for each drift test. Zygo interferometer and capacitive sensor data were
collected simultaneously. The room temperature was also recorded using an
Elitech RC-5 USB Temperature Data Logger during two of the tests.

No drifting was found on the surface of the FLASH at the end of the test above
the measurement error of the Zygo interferometer (Figure 13, top). The
capacitive sensors measured drifting of up to 90nm of change. However, these
capacitive sensor drifts have a correlation with the recorded temperature
data. These movements are in the range of what could be explained by the
thermal expansion change in the 60mm aluminium capacitive sensor mounts. To
better calibrate out any movement due to temperature changes, the data from
Capacitive Sensors 2-4 were subtracted from the data from Capacitive Sensor 1.
After this correction, the biggest change recorded was 20nm of drift (Figure
13). Future sensor mount designs should consider using a material with a lower
CTE value to increase the measurement precision of the capacitive sensors
without this calibration.

Table 3: Drift Testing Runs Drift Test | Pattern | Magnitude | Note
---|---|---|---
1 | Checkerboard | +50mA |
2 | Checkerboard | +100mA |
3 | Piston | +100mA |
4 | Piston | -100mA |
5 | Piston | +50mA |
6 |  | Flattening Currents
---
found 04-07
– |
7 |  | Flattening Currents
---
found 04-07
– |  | Taken on 04-08-2021,
---
results shown in Figure 13
8 | Checkerboard | -50mA |
9 | Checkerboard | -100mA |

Figure 13: Example Drift Results from Holding the Flattening Currents Pattern.
(Top) The final Zygo interferometer measurement could not measure drifting
outside the measurement error of the Zygo. (Bottom) Four capacitive sensors
provided real time data over the course of the 8.5hr drift tests. The drifting
due to a CTE mismatch of the capacitive sensor mounts was calibrated by
subtracting the Capacitive Sensor 1 data from sensors 2-4. In this run, the
largest drift measured was $12$nm by Capacitive Sensor 3.

###  3.6 Zernike Mode Testing

_Measurement setup: Zygo Interferometer & Capacitive Sensors_

To determine how closely the FLASH can reproduce an applied Zernike shape, the
first 19 Zernike Modes were applied to FLASH. The currents needed to reproduce
the Zernike modes were calculated using Equation 2, where $S_{i}$ was the
shape of each Zernike pattern. No iterations were used in trying to reproduce
the Zernike pattern.

An example of the side-by-side comparison between DM3 and an ideal Zernike
pattern can be viewed in Figure 14 for mode 6. A movie of the simultaneous
capacitive sensor and Zygo interferometer data collection can be watched at
the following youtube link: https://youtu.be/scUhmqljJVc.

---
Figure 14:  Example Zernike Mode: Zernike Mode 6 applied on FLASH. (Top) The
Zernike pattern fed to the FLASH. (Bottom) The resulting facesheet pattern
produced by FLASH.

###  3.7 Settling Time

_Measurement setup: Capacitive Sensors_

To quantify the response time for an actuator to reach its final position
after a current step is applied, we used the capacitive sensor system to
measure the actuator settling time. A current step of $+$10mA was applied to
Actuator 4 (the center actuator). Our definition of settling time was adopted
from Rochette et al. 2018 [Rochette2018], which began/ended the settling time
measurement when the system rose/reached 5% of its final displacement. Ten
runs were averaged to measure each settling time value.

We find a settling time of 14.3 $\pm$ 0.1 millisec when the current is applied
as a step response (from 0 to 10mA in one step). To overcome the actuators
internal eddy-current behavior and speed up the settling time, a lead filter
can be implemented when applying the current step (discussed here in Section
3.7) or the PI control settings can be tuned (discussed in Section 3.8).

We implemented a lead filter using the discrete-space state Matlab Simulink
model with zero on the pole location of the measured plant and a pole at
500Hz. This filter applied the current using a spike and then a ramp-down from
the peak. It was designed to reach a peak current of 9.058 times the final
current; for a step of 10mA, this resulted in a peak current of 90.58mA.

Four tuning values for the lead filter were tried with different decay times
(Figure 15). The lead filter with the optimal tuning (Lead Filter 2) was set
with the parameters of $A=0.6$, $B=2.18$, $C=-1.478$, $D=9.058$. The settling
time measured with this lead filter was $t_{set}=1.08\pm 0.08$ms,
approximately an order of magnitude faster than the settling time with no lead
filter applied.

---
Figure 15:  Settling Time By applying a lead filter to the actuators, the
response time of the actuator was improved by an order of magnitude from
$t_{set}=14.3\pm 0.1$ms to $t_{set}=1.08\pm 0.08$ms.

###  3.8 Dynamic Testing

To verify the dynamic behavior of the FLASH DM, a non-parametric
identification was performed by applying a random input signal to the center
actuator and capturing the facesheet displacements. This testing was conducted
at TNO using a capacitive sensor system external to the FLASH (separate from
the capacitive sensor system described here in Section 2.5).

The frequency response of the transfer function measurement reveals a first
order drop with a roll-off frequency of $\sim$40Hz (Figure 16). The first
second-order mechanical resonance is $\sim$1.2kHz. The observed first order
lowpass response can be explained by the eddy-current behavior within the
magnetic circuitry from the fast-changing currents in the coil. The open-loop
transfer function shows the dynamical response of a first order parametric
model that matches the first order low-passing behavior. A single pole at
42.6Hz and a two-sample delay at 5kHz are present, which is expected from the
data acquisition system that was used. The close match between the response of
this parametric model and the measurement data in phase and amplitude verify
that the low-passing behavior can be characterized as first-order low-pass.

Although the first order low-pass behavior starts at 40Hz, the closed-loop
control bandwidth can be pushed to a faster frequency by properly tuning the
PI controller. To demonstrate this, a loop-shaping exercise was performed
based on the measured frequency responses using a PI controller. Figure 17
shows the resulting Loop-gain ($L(j\omega)=H(j\omega)C(j\omega)$) and
Sensitivity ($S(j\omega)=1/(1+L(j\omega))$) responses when tuning the PI
controller to match the pole of the first order low-passing behavior within
the actuator. With this PI controller, an open-loop bandwidth of 225Hz and a
closed-loop sensitivity bandwidth of around 150Hz (-3dB) is achieved. Higher
closed-loop performances are deemed possible when high-order controller design
methods are employed.

---
Figure 16:  Dynamical Test Bode Plot (Top) The non-parametric identification
of the transfer function (Bottom) The first order parametric model fit to
estimate pole location and IO delay.

Figure 17:  Loop-shaping Exercise (Left) Open-Loop Transfer function
($L(j\omega)$) (Right) Sensitivity Transfer function ($S(j\omega)$).

##  4 CONCLUSIONS

The key results from the UCSC-LAO initial performance testing of the FLASH are
summarized in Table 4. Our linearity, hysteresis, drift, and Zernike mode
testing verify that the performance of the 2020 generation of TNO HVR
actuators performed as expected and suggest that the performance of the UH2.2m
adaptive secondary mirror will behave as TNO’s modeling predicts.

The dynamic and settling time measurements provide the first measured results
of how the TNO systems perform at kilohertz speeds. Without tuning the profile
of the current applied, the TNO deformable mirrors have a settling time of
$t_{set}=14.3\pm 0.1$ms. Our testing demonstrates that the correction speed
can be improved to $t_{set}=1.08\pm 0.08$ms using a lead filter. This speed is
consistent with the operating speed of the current on-sky adaptive secondary
mirrors constructed with voice-coil style actuators. [Biasi2010]

The Zygo interferometer and capacitive sensor data were consistent when
simultaneous data collection was performed. The two data collection methods
complemented each other; the Zygo interferometer provided a holistic view of
the mirror shape and the capacitive sensors provided dynamic time response
information. The capacitive sensors are capable of measuring displacements at
a higher precision than the Zygo interferometer, as long as having limited
spatial information is acceptable. In future situations when an interferometer
is not possible to use in the testing setup (i.e. in an environmental chamber
or on-sky optical system), our testing indicates that the capacitive sensors
can accurately provide feedback about the real time performance of the
deformable mirror.

Table 4: Summary of FLASH Testing Results Section | Test | Takeaway
---|---|---
3.1 |  | Influence Functions &
---
Actuator Cross-Coupling
|  Measured actuator spacing $=39.4\pm 0.4$mm,
---
Actuator cross-coupling $=34.2\pm 1.0$%
3.2 |  | Natural Shape
---
Surface Flattening
| The surface was brought from RMS = 1158nm to RMS = 15nm
---
with an average current per actuator = $17.0\pm 4.7$mA and
total power $=7.3$mW ($0.4$mW/actuator)
3.3 | Linearity |  |  Individual act: avg disp. $=46.5\pm 1.4$nm/mA , lin $=95.2\pm 0.8$%
---
All actuators: avg disp. $=113.9\pm 1.0$ nm/mA, lin $=96.8\pm 0.5$%
There are asymmetries between negative and positive applied currents
3.4 | Hysteresis |  | Average percent hysteresis from loops with max current 20mA-120mA:
---
$1.80\pm 0.13$% (zygo), $1.93\pm 0.04$% (capacitive sensors)
3.5 | Drift |  | Max drift seen from cap sensors in 9 room-temperature tests:
---
Drift $<20$nm
3.6 | Zernike Mode | The applied zernike modes are consistent with the measured modes
3.7 | Settling Time |  |  $t_{set}=14.3\pm 0.1$ms;
---
A lead filter can be implemented to lower the actuator response time;
a well-tuned lead filter achieved $t_{set}=1.08\pm 0.08$ms for a
step of 10mA ($\sim 3\mu$m).
3.8 | Dynamic Testing |  |  First-order roll-off frequency $=40$Hz;
---
Second-order mechanical resonance = $1.2$kHz

###### Acknowledgements.

The authors would like to thank Max Baeten, Arjo Bos, Mark Chun, Wouter
Jonker, Matthew Maniscalco, Joost Peters and the memebers of the Lab for
Adaptive Optics at UC Santa Cruz for their guidance and assistance in the
completion of this work.


# Integrating Summarization and Retrieval for Enhanced Personalization via
Large Language Models

Christopher Richardson  Georgia Institute of TechnologyAtlantaGAUSA
<EMAIL_ADDRESS>1234-5678-9012 ,  Yao Zhang  Amazon Alexa
AIAustinTXUSA<EMAIL_ADDRESS>,  Kellen Gillespie  Amazon Alexa
AISeattleWAUSA<EMAIL_ADDRESS>,  Sudipta Kar  Amazon Alexa AISeattleWAUSA
<EMAIL_ADDRESS>,  Arshdeep Singh  Amazon Alexa AISeattleWAUSA
<EMAIL_ADDRESS>,  Zeynab Raeesy  Amazon Alexa AISeattleWAUSA
<EMAIL_ADDRESS>,  Omar Zia Khan  Amazon Alexa AISeattleWAUSA
<EMAIL_ADDRESS>and  Abhinav Sethy  Amazon Alexa AISeattleWAUSA
<EMAIL_ADDRESS>
###### Abstract.

Personalization, the ability to tailor a system to individual users, is an
essential factor in user experience with natural language processing (NLP)
systems. With the emergence of Large Language Models (LLMs), a key question is
how to leverage these models to better personalize user experiences. To
personalize a language model’s output, a straightforward approach is to
incorporate past user data into the language model prompt, but this approach
can result in lengthy inputs exceeding limitations on input length and
incurring latency and cost issues. Existing approaches tackle such challenges
by selectively extracting relevant user data (i.e. selective retrieval) to
construct a prompt for downstream tasks. However, retrieval-based methods are
limited by potential information loss, lack of more profound user
understanding, and cold-start challenges. To overcome these limitations, we
propose a novel summary-augmented approach by extending retrieval-augmented
personalization with task-aware user summaries generated by LLMs. The
summaries can be generated and stored offline, enabling real-world systems
with runtime constraints like voice assistants to leverage the power of LLMs.
Experiments show our method with 75% less of retrieved user data is on-par or
outperforms retrieval augmentation on most tasks in the LaMP personalization
benchmark. We demonstrate that offline summarization via LLMs and runtime
retrieval enables better performance for personalization on a range of tasks
under practical constraints.

LLM, Personalization, Summarization, NLP, Chatbot, Voice Assistant,
Conversational AI

††ccs: Computing methodologies Natural language generation††ccs: Information
systems Personalization††ccs: Computing methodologies Information extraction

##  1\. Introduction

As virtual assistants and other natural language processing (NLP) systems
become increasingly integrated into our daily lives, personalization has
become an essential factor in user experience. Tailoring virtual assistant
interactions and NLP model outputs to individual users’ preferences, styles,
needs, and contexts is essential in improving the performance of these systems
to make them more natural and conversational.

Traditional personalization methods, such as collaborative filtering (Schafer
et al., 2007), deep neural networks (Covington et al., 2016), deep interest
network (Zhou et al., 2018) and their variations (Qi et al., 2020; Chen et
al., 2021), have enhanced user experiences in recommendation systems. These
methods leverage historical user behavior data to make personalized
recommendations, offering a practical and effective solution for various
domains. Despite their success, these methods still struggle with the cold-
start problem, where new users lack sufficient behavior history, leading to
sub-optimal recommendations. The cold-start problem highlights the need for
alternative approaches.

Large Language Models (LLMs) represent a promising avenue for advancing
personalization techniques. LLMs have demonstrated remarkable capabilities in
understanding context and generating coherent text (Brown et al., 2020). By
incorporating knowledge about users, LLMs can potentially enhance
personalization by capturing subtle user preferences, but how to capture the
full spectrum of user preferences in a personalized manner remains a
challenge. To personalize a language model output, a straightforward approach
is to incorporate user data into the language model prompt. However,
incorporating a comprehensive view of customer preferences with long-term
historical user data into the prompt may exceed the input length limitations
of language models and result in considerable increases in inference cost.
Further, language models tend to degrade with lengthy contexts (Chen et al.,
2023). To address these concerns, a personalized retrieval augmentation
framework was proposed (Salemi et al., 2023). This framework selectively
extracts relevant user data to construct prompts for downstream language
models. Recent work has also shown promise in combining retrieval approaches
with LLMs to improve performance in recommender systems (Chen, 2023; Xu et
al., 2022; Li et al., 2023), as well as general NLP tasks (Dudy, 2022; Flek,
2020; Qian et al., 2021; Rafieian and Yoganarasimhan, 2022; Wu et al., 2021).
However, retrieval-based methods have constraints in potential information
loss, lack the ability to comprehend user data on a more profound level, and
may suffer from the cold-start problem.

Our research aims to address the aforementioned limitations of both
traditional personalization methods and retrieval-based methods with LLMs by
proposing the hybrid approach shown in Figure 1. By integrating retrieval
techniques with LLM-generated summaries of user data, we intend to create a
more robust personalized system. To prevent information loss, the user summary
offers contextual information at a higher level of abstraction for the
downstream task. To understand user data on a more profound level, the summary
generation is aware of the task and incorporates this information in the
prompt for summary generation. For example, for a personalized paraphrase text
generation task, the summary model is instructed by a prompt to pay attention
to the user writing style in addition to the semantic content. Also, this
hybrid model could overcome the cold-start problem and provide personalized
outputs even in data-sparse scenarios by providing user summaries for new
users based on available user data from other applications or user’s self
description. The summaries in our approach can be generated offline and
stored, ensuring negligible increased runtime latency and enabling systems
with runtime constraints to leverage the power of LLMs into real-work online
applications, such as voice assistant scenarios.

We demonstrate our method of integrating summarization and retrieval on a
publicly available Language Model Personalization (LaMP) benchmark (Salemi et
al., 2023), including both text classification and generation tasks across a
variety of domains. Experiments show our method achieves comparable or better
performance compared to retrieval augmentation on most tasks. With our method,
the retrieval component can use 75% less of retrieved user data without
sacrificing performance on five out of six tasks, and achieves superior
performance on two tasks.

In summary, our main contributions are as follows. First, we propose
augmenting traditional retrieval-based personalization methods with LLMs’
summarization of user data to address the limitations of existing methods:
potential information loss, the inability to understand user data at a high
level, and the cold-start challenge. Our method enables powerful LLMs to
provide comprehensive information about users with no additional runtime
latency. Further, we implemented our proposed approach and conducted
experiments on a language model personalization benchmark dataset LaMP with 6
public tasks. With promise shown in our experiment results, we envision a
personalized system that better caters to individual user preferences
especially for new users by integrating summarization via LLMs and retrieval.

Figure 1. Personalization is achieved by combining runtime-retrieved samples
with an offline-generated user summary. Given a textual input $x$ that
describes a task in natural language, the goal is to generate a personalized
output $y$ for users. The retrieval model identifies the most relevant items
from user data, and the retrieved items along with the offline user summary
and $x$ form the basis for creating a prompt. This prompt is constructed using
a prompt construction function $\phi_{p}$.

##  2\. Methodology

###  2.1. Problem Formulation

The problem formulation follows LaMP (Salemi et al., 2023): given a textual
input $x$ that describes a task in natural language, we want to generate a
personalized output $y$ for user $u$. The goal is thus to learn the
distribution $p(y|x,u)$.

####  2.1.1. Baseline

Our baseline is a retrieval-augmented method that follows a retrieve-then-
model paradigm (Salemi et al., 2023). For retrieval, a manually defined query
generation function $\phi_{q}(x)$ is first used to help extract salient
information from $x$ as inputs to the retrieval model
$\mathcal{R}(\phi_{q}(x),P_{u},k)$. The retrieval model returns the top-$k$
relevant items from the user profile $P_{u}$ using the retrieval query
$\phi_{q}(x_{i})$, and the returned items are used to construct a prompt for a
downstream model using a prompt construction model $\phi_{p}$. The input to
the downstream language model is as follows.

(1)
$\displaystyle\overline{x}_{i}=\phi_{p}(x_{i},\mathcal{R}(\phi_{q}(x_{i}),P_{u},k))$

The downstream language model is fine-tuned on the dataset
$\\{\overline{x}_{i},y_{i}\\}$.

###  2.2. Integrating Summarization and Retrieval

Runtime constraints can limit the number of user data retrieved by
$\mathcal{R}$ that can be utilized. In this work, we consider scenarios where
there are both latency considerations as well as input length limits for the
downstream model. To improve performance without adding runtime latency, we
introduce a summary of the user data, $s_{u}$, to augment the retrieved data:

(2)
$\displaystyle\overline{x}_{i}=\phi_{p}(x_{i},\mathcal{R}(\phi_{q}(x_{i}),P_{u},k),s_{u})$

Our approach involves using LLMs to summarize salient information from $u$ as
it relates to optimizing $p(y|x,u)$. We use instruction-tuned models to
generate an abstractive summary of user data:

(3) $\displaystyle s_{u}=\text{LLM}(P_{u})$

An overview of our method is shown in Figure 1. The summaries can be generated
offline and stored along with the user data itself. At runtime, the retrieval
algorithm retrieves the top-$k$ profile entries and concatenates them with the
task input and the summary to create the full context for the downstream
language model, which is fine-tuned using the standard language modeling loss
against the output $y$.

##  3\. Experiments

###  3.1. Datasets and Evaluation Metrics

LaMP is a public benchmark dataset for training and evaluating methods for
personalization with language models (Salemi et al., 2023). It consists of
seven personalization NLP tasks, including three classification tasks and four
text generation tasks. Data for each task include input text, reference
completion text as ground truth output, and a user profile consisting of an
array of items with textual data. A brief description of each task and their
evaluation metrics are shown in Table 1. We have excluded task LaMP-6 from our
study as it relies on private data to which we do not have access. The LaMP
benchmark organizes the data in both user-based separation and time-based
separation. In our study, we utilize the user-based separation approach to
address cold-start issues for new users.

Table 1. Description of LaMP tasks and data. LaMP-1 |  Task: Citation Identification (binary choice)
---|---
User Profile: Scholarly article titles and abstracts
Metric(s): Accuracy
LaMP-2 |  Task: News Categorization (classification)
User Profile: Categorized articles published
Metric(s): Accuracy and F1
LaMP-3 |  Task: Product Rating (classification)
User Profile: Product reviews and scores
Metric(s): Mean Absolute Error (MAE) and Root Mean Square Error (RMSE)
LaMP-4 |  Task: News Headline Generation (text generation)
Profile: News articles and their headlines
Metric(s): ROUGE-1 and ROUGE-L
LaMP-5 |  Task: Scholarly Title Generation (text generation)
Profile: Scholarly article titles and abstracts
Metric(s): ROUGE-1 and ROUGE-L
LaMP-7 |  Task: Tweet Paraphrasing (text generation)
Profile: Tweets
Metric(s): ROUGE-1 and ROUGE-L

###  3.2. Experimental Setup

Following LaMP, we used FlanT5-base (Chung et al., 2022) as our downstream
model. This model demonstrated satisfactory runtime performance in our
experiments (approximately 125 milliseconds per sample when we included as
many user data as possible within the 512-token limit of the input length)
and, as shown in the LaMP experiments, it achieved superior performance to
that achieved in zero-shot experiments with FlanT5-XXL and ChatGPT (Salemi et
al., 2023). For all experiments, we used the same settings reported in LaMP: a
learning rate of $5\times 10^{-5}$, weight decay of $10^{-4}$, warmup ratio of
0.05, and a beam size of 4, and we trained for 10 epochs for text
classification (tasks 1-3) and 20 epochs for text generation (tasks 4, 5, and
7).

In our experiments, we utilized the BM25 retrieval algorithm (Robertson et
al., 1995) due to its speed and performance. We found neural methods like
Contriever (Asai et al., 2022) to be too slow for voice assistant scenarios,
which induced approximately 10-30 seconds of latency per sample while not
significantly outperforming BM25 on many LaMP tasks.

We experimented with two instruction-tuned models for generating summaries.
The first is Vicuna (Zheng et al., 2023), a 13-billion parameter model
distilled from LLaMA (Touvron et al., [n. d.]). The second model is ChatGPT
using OpenAI’s API with the gpt-3.5-turbo-16k model. Vicuna has a context
length of 2048 tokens, while ChatGPT’s limit is 16,384. Prompts used for
generating summaries are shown in Table 2. For tasks 2 and 3, to achieve good
performance given the simplicity of the tasks, we constrained the
summarization model to output according to a strict template shown in the
Table 2, therefore ChatGPT summaries were not included for these tasks.

We compare our methods to the retrieval-only baselines, using $k$ values of 0,
1, and 4 for the baselines. We observed $k=4$ to be the limit for some tasks
given the context length of FlanT5 (512 tokens). Thus, we had to reduce the
number of retrieved samples to fit the summaries into the input of FlanT5
without truncating and chose $k=1$ (for direct comparison with the baseline),
as well as $k=0$ to investigate the impact of summaries alone (no retrieval).
We report means of three repeated runs of each experiment for comparison with
statistical significance.

Table 2. Prompts used for summarization. Additional tokens were used for Vicuna summaries to match the expected prompt format for that model but the content was the same for Vicuna and ChatGPT. Task |  Prompt
---|---
LaMP-1 |  Write a summary, in English, of the research interests and topics of a researcher who has published the following papers. Only generate the summary, no other text.
LaMP-2 |  Look at the following past articles this journalist has written and determine the most popular category they write in. Answer in the following form: most popular category: ¡category¿
LaMP-3 |  Based on this user’s past reviews, what are the most common scores they give for positive and negative reviews? Answer in the following form: most common positive score: ¡most common positive score¿, most common negative score: ¡most common negative score¿
LaMP-4 |  Given this author’s previous articles, try to describe a template for their headlines. I want to be able to accurately predict the headline gives one of their articles. Be specific about their style and wording, don’t tell me anything generic.
LaMP-5 |  Given this author’s previous publications, try to describe a template for their titles. I want to be able to accurately predict the title of one of the papers from the abstract. Only generate the template description, nothing else.
LaMP-7 |  Given this person’s previous tweets, try to describe a template for their tweets. I want to take a generic sentence and rephrase it to sound like one of their tweets, with the same style/punctuation/capitalization/wording/tone/etc. as them. Only give me the template description, nothing else.

##  4\. Results and Discussion

Table 3 shows the results of our methods for both summary models alongside the
baselines across various NLP tasks. Our experiments prove our summary-
augmented method with $k=1$ is on-par or outperforms the retrieval-only
baseline with $k=4$ on most tasks (reducing the amount of retrieved user data
by 75%). As the bold results indicate the best results among compared
experiments for each task, our methods (Vicuna Summ. and GPT-3.5 Summ.)
outperform baselines consistently on tasks 1 and 2 at a statistical
significance level with $p-value<0.05$, and we achieve comparable performance
on tasks 3, 5, and 7 with no statistically significant difference.

On LaMP-1 task, our method GPT-3.5 Summ. outperforms the baseline with $k=4$
using only offline generated summaries ($k=0$). Furthermore, Our method
GPT-3.5 Summ. yields better results compared to $k=1$ baseline on all tasks.
Worth noting that ChatGPT summaries mostly outperform those provided by
Vicuna, likely due to the disparity in model size.

Table 3. Results for FlanT5-base model fine-tuned on LaMP benchmark tasks. Baseline: retrieval of k user data entries; Vicuna Summ.: Baseline + summary of user data generated by vicuna; GPT-3.5 Summ.: Baseline + summary of user data generated by GPT-3.5. Underline means summary improved the corresponding baselines with the same k, and bold means the best results among compared experiments for each task. For all metrics, higher is better except in the case of MAE and RMSE used for LaMP-3. |  | Baseline | Vicuna Summ. | GPT-3.5 Summ.
---|---|---|---|---
Task | Metric | $k=0$ | $k=1$ | $k=4$ | $k=0$ | $k=1$ | $k=0$ | $k=1$
LaMP-1: Personalized | Accuracy | 0.516 | 0.650 | 0.709 | 0.704 | 0.728 | 0.738 | 0.743
Citation Identification
LaMP-2: Personalized | Accuracy | 0.731 | 0.782 | 0.807 | 0.801 | 0.814 | N/A
News Categorization | F1 | 0.511 | 0.573 | 0.574 | 0.550 | 0.601
LaMP-3: Personalized | MAE | 0.311 | 0.284 | 0.280 | 0.305 | 0.277 | N/A
Product Rating | RMSE | 0.626 | 0.595 | 0.593 | 0.632 | 0.594
LaMP-4: Personalized | ROUGE-1 | 0.152 | 0.177 | 0.188 | 0.157 | 0.173 | 0.170 | 0.181
News Headline Generation | ROUGE-L | 0.137 | 0.162 | 0.173 | 0.142 | 0.159 | 0.155 | 0.166
LaMP-5: Personalized | ROUGE-1 | 0.424 | 0.447 | 0.448 | 0.426 | 0.447 | 0.424 | 0.448
Scholarly Title Generation | ROUGE-L | 0.382 | 0.408 | 0.409 | 0.386 | 0.408 | 0.383 | 0.409
LaMP-7: Personalized | ROUGE-1 | 0.510 | 0.502 | 0.513 | 0.510 | 0.512 | 0.510 | 0.512
Tweet Paraphrasing | ROUGE-L | 0.455 | 0.448 | 0.459 | 0.455 | 0.459 | 0.456 | 0.460

We observed a gap between Vicuna Summ. and GPT-3.5 Summ. attributed to the
differing quality of the offline summaries.. Despite studies showing Vicuna
achieving up to 90% the performance of ChatGPT (Chiang et al., 2023), our
results suggest that Vicuna did not perform as well as GPT-3.5 on task LaMP-1
and 4. To assess the summary quality, we provide examples in the Appendix. A.

While we have shown promise in combining offline summaries with runtime
retrieval for personalization, there are a few limitations to this work. For
one, the data and tasks provided in the LaMP benchmark are simplistic and
narrow in scope. More work is needed to assess the potential of our method on
more realistic user data. Also, the benefits of summarization can be improved
by fine-tuning a larger language model and end-to-end training for the tasks.

##  5\. Conclusion

This paper introduces a novel method for augmenting retrieval with offline
summarization for improving personalization in various NLP tasks. We
implemented our method and achieved comparable or better performance on most
NLP tasks in the LaMP personalization benchmark while reducing the amount of
retrieved user data by 75%. In some cases, we even achieved superior
performance after removing retrieval entirely, showing an advantage for sparse
data scenarios, such as the cold-start problem. Our method efficiently
leverages offline summary generation and is suitable for runtime constrained
applications such as voice assistants.

###### Acknowledgements.

Thanks to Camden Parsons, Farideh Tavazoee, Xingzhi (Jacky) Guo for reviewing
and providing feedbacks on our initial paper draft.

## References

  * (1)
  * Asai et al. (2022) Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022.  Task-aware retrieval with instructions.  _arXiv preprint arXiv:2211.09260_ (2022).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Chen et al. (2023) Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023.  How Many Demonstrations Do You Need for In-context Learning?  arXiv:2303.08119 [cs.AI]
  * Chen et al. (2021) Qiwei Chen, Changhua Pei, Shanshan Lv, Chao Li, Junfeng Ge, and Wenwu Ou. 2021.  End-to-End User Behavior Retrieval in Click-Through RatePrediction Model.  http://arxiv.org/abs/2108.04468 arXiv:2108.04468 [cs].
  * Chen (2023) Zheng Chen. 2023.  PALR: Personalization Aware LLMs for Recommendation.  _arXiv preprint arXiv:2305.07622_ (2023).
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.  https://lmsys.org/blog/2023-03-30-vicuna/
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022\.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ (2022).
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016.  Deep Neural Networks for YouTube Recommendations. In _Proceedings of the 10th ACM Conference on Recommender Systems_. New York, NY, USA.
  * Dudy (2022) Shiran Dudy. 2022.  Personalization and Relevance in NLG. In _Companion Proceedings of the Web Conference 2022_. 1178–1178.
  * Flek (2020) Lucie Flek. 2020.  Returning the N to NLP: Towards contextually personalized classification models. In _Proceedings of the 58th annual meeting of the association for computational linguistics_. 7828–7838.
  * Li et al. (2023) Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023.  GPT4Rec: A generative framework for personalized recommendation and user interests interpretation.  _arXiv preprint arXiv:2304.03879_ (2023).
  * Qi et al. (2020) Pi Qi, Xiaoqiang Zhu, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, and Kun Gai. 2020.  Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction.  https://doi.org/10.48550/arXiv.2006.05639 arXiv:2006.05639 [cs, stat].
  * Qian et al. (2021) Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021.  Pchatbot: a large-scale dataset for personalized chatbot. In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_. 2470–2477.
  * Rafieian and Yoganarasimhan (2022) Omid Rafieian and Hema Yoganarasimhan. 2022.  AI and Personalization.  _Available at SSRN 4123356_ (2022).
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995\.  Okapi at TREC-3.  _Nist Special Publication Sp_ 109 (1995), 109.
  * Salemi et al. (2023) Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023.  LaMP: When Large Language Models Meet Personalization.  _arXiv preprint arXiv:2304.11406_ (2023).
  * Schafer et al. (2007) J. Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007.  Collaborative Filtering Recommender Systems.  In _The Adaptive Web: Methods and Strategies of Web Personalization_ , Peter Brusilovsky, Alfred Kobsa, and Wolfgang Nejdl (Eds.). Springer, Berlin, Heidelberg, 291–324.  https://doi.org/10.1007/978-3-540-72079-9_9
  * Touvron et al. ([n. d.]) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [n. d.].  LLaMA: open and efficient foundation language models, 2023.  _URL https://arxiv. org/abs/2302.13971_ ([n. d.]).
  * Wu et al. (2021) Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021.  Personalized response generation via generative split memory network. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 1956–1970.
  * Xu et al. (2022) Jiajing Xu, Andrew Zhai, and Charles Rosenberg. 2022.  Rethinking personalized ranking at Pinterest: An end-to-end approach. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 502–505.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023\.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  _arXiv preprint arXiv:2306.05685_ (2023).
  * Zhou et al. (2018) Guorui Zhou, Chengru Song, Xiaoqiang Zhu, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018.  Deep Interest Network for Click-Through Rate Prediction.  https://doi.org/10.48550/arXiv.1706.06978 arXiv:1706.06978 [cs, stat].

##  Appendix A Summary Examples

We are presenting examples in which the LaMP baseline model exhibited errors,
while our methods demonstrated superior performance in Example 1, and in which
the LaMP baseline model and Vicuna Summ. exhibited errors while GPT-3.5 Summ.
demonstrated superior performance in Example 2.

###  A.1. Example 1

Task: LaMP-1

Input: For an author who has written the paper with the title ”QuickSense:
Fast and energy-efficient channel sensing for dynamic spectrum access
networks”, which reference is related? Just answer with [1] or [2] without
explanation. [1]: ”A first look at cellular network performance during crowded
events” [2]: ”On the complexity of scheduling in wireless networks”

Output: [1]

Retrieved user data: On the Stability of Rational, Heterogeneous Interdomain
Route Selection

A decision-theoretic approach to resource allocation in wireless multimedia
networks

A general algorithm for interference alignment and cancellation in wireless
networks

Enterprise social network analysis and modeling: A tale of two graphs

Baseline Prediction: [2]

User Summary by Vicuna: (The original output is in Chinese and here is the
translation:) Network Traffic Analysis

Wireless Communication

Software-Defined Networking

Traffic Routing

Low-Latency Communication

Multimedia Networks

Network Security

Error Handling in Networks

Sensor Networks

Multi-Path Routing

IP Routing

Traffic Analysis Tools

Vicuna Summ. Prediction: [1]

User Summary by GPT-3.5: The research interests of this researcher span
several areas in computer networking and wireless communications. They have
published papers on topics such as unified cellular and ad-hoc network
architecture, optimization of wireless mesh networks, networking analytics,
incentive-compatible routing protocols, software-defined networking, and
network security. They have expertise in optimizing network throughput,
designing scalable network architectures, developing game-theoretic and
cryptographic techniques for wireless networks, and leveraging machine
learning and data analytics for network monitoring and diagnosis. Their recent
works also focus on addressing the latency and energy efficiency challenges in
wireless multimedia networks and leveraging software-defined networking for
next-generation cellular networks. In summary, this researcher’s work revolves
around developing novel algorithms, protocols, and systems that improve the
performance, resilience, and security of networked systems.

GPT-3.5 Summ. Prediction: [1]

###  A.2. Example 2

Task: LaMP-1

Input: For an author who has written the paper with the title ”Visual-audio
integration for user authentication system of partner robots”, which reference
is related? Just answer with [1] or [2] without explanation. [1]:
”Pronunciation Modeling for Improved Spelling Correction” [2]: ”Path following
algorithm for highly redundant manipulators”

Output: [1]

Retrieved user data: Advancing Matrix Completion by Modeling Extra Structures
beyond Low-Rankness

Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner
Product Search (MIPS)

Real-Time Implementation of Improved State-Space MPC for Air Supply in a Coke
Furnace

On Practical Algorithms for Entropy Estimation and the Improved Sample
Complexity of Compressed Counting

Baseline Prediction: [2]

User Summary by Vicuna: 1\. Nystrom Method for Approximating the GMM Kernel.

2\. Very sparse random projections.

3\. Very sparse stable random projections for dimension reduction in lalpha (0
<alpha<=2) norm.

4\. A Comparison Study of Nonlinear Kernels.

5\. One sketch for all: Theory and Application of Conditional Random Sampling.

6\. Collaborative Multi-objective Ranking.

7\. Accurate Estimators for Improving Minwise Hashing and b-Bit Minwise
Hashing.

8\. A new space for comparing graphs.

9\. Stability and Risk Bounds of Iterative Hard Thresholding.

10\. Hashing Algorithms for Large-Scale Learning.

11\. Variational Flow Graphical Model.

Vicuna Summ. Prediction: [2]

User Summary by GPT-3.5: The researcher’s primary research interests revolve
around the development of practical algorithms for entropy estimation,
compressed counting, and matrix completion. They explore various techniques
for improving the efficiency and accuracy of these methods. Additionally, the
researcher focuses on advancing hashing algorithms, particularly in the
context of maximum inner product search (MIPS) and locality-sensitive hashing
(LSH). They also investigate sketch-based sampling techniques, especially for
sparse data, and propose novel approaches for estimating entropy of data
streams. Furthermore, the researcher has contributions in optimization
methods, classification, and dimensionality reduction using stable random
projections. Their work demonstrates a strong emphasis on efficient and
scalable solutions for a wide range of machine learning and data analysis
tasks.

GPT-3.5 Summ. Prediction: [1]