# Circumventing size-bandwidth limits in imaging with flat lenses

Apratim Majumder,1 Monjurul Meem,1 Nicole Brimhall,2 Rajesh Menon,1,2∗

1Department of Electrical and Computer Engineering, University of Utah,
50 Central Campus Dr, Salt Lake City, UT 84112, USA
2Oblate Optics, Inc., San Diego, CA 92130, USA

∗To whom correspondence should be addressed; E-mail<EMAIL_ADDRESS>
> Recent theoretical work suggested upper bounds on the operating bandwidths
> of flat lenses. Here, we show how these bounds can be circumvented via a
> multi-level diffractive lens (MDL) of diameter = 100 mm, focal length = 200
> mm, device thickness = 2.4$\mu$ m and operating bandwidth from $\lambda$ =
> 400 nm to 800 nm. We further combine the MDL with a refractive lens to
> demonstrate a hybrid telescope. By appealing to coherence theory, we show
> that the upper bound on relative bandwidth is surprisingly independent of
> lens diameter or numerical aperture, but is only limited by the bandwidth of
> the image sensor. Since large-area achromatic flat lenses produce
> significant reductions in weight over their refractive counterparts, these
> calculations and experiments open up opportunities for very large scale
> diffractive and diffractive-refractive telescopes.

Lightweight, large-area optics could be extremely useful for airborne and
space-based imaging and sensing systems (1, 2). Although there has been
tremendous progress in the field of ultra-lightweight flat lenses,
computational and fabrication constraints have generally prevented
demonstrations of large apertures. Previous efforts included the design and
preliminary demonstrations of diffractive flat lenses, but these suffered from
narrow operating bandwidths (3). Furthermore, full aperture experimental
demonstrations were not completed. Due to their low weight, diffractive flat
lenses hold great promise for large-aperture space telescopes with potential
applications in imaging exoplanets (4, 5). Recent work has suggested the use
of holographic elements for large-aperture spectral imaging (6). Advances in
subwavelength diffractive optics such as metalenses have resulted in 20 mm
diameter flat lenses, but again only for a narrow bandwidth (4).

Recent theoretical work has noted a fundamental limit on the size-bandwidth
product of a flat lens as function of numerical aperture (NA) (4).
Intuitively, this limit arises from the time delay experienced by the photons
traversing the edges of the lens when they reach the focal point compared to
that experienced by those photons traversing the optical axis (see inset Fig.
1a). It was pointed out that this time delay imposes a fundamental limit on
the bandwidth of the illumination that can be focused by the lens. This time
delay is given by $\Delta T=\frac{\sqrt{R^{2}+f^{2}}-f}{c}$, where $R$ and $f$
are the radius and focal length of the lens, and $c$ is the speed of light. In
order to ensure coherent interactions between these photons, this time delay
then imparts an upper bound on the maximum bandwidth of the illumination as:
$\Delta T\times\Delta\omega\leq\kappa$, where $\kappa$ is a device-dependent
constant, typically $\sim$ 10\. Assuming R = 50 mm, f = 200 mm, we get $\Delta
T=0.21$ ns. And the upper bound on bandwidth becomes (for $\kappa\sim$ 10)
$\Delta\omega\ <\ 48\times{10}^{9}$ rad/s.

In this work, we exploit a loophole specific to imaging to overcome this
limit, where we first recognize that coherent interactions between photons
transiting the edges of the lens and those along the optical axis are not
required. As a result, the bandwidth of an imaging system is not limited by
the bandwidth of the lens, but by the much smaller bandwidth of the sensor
(and associated electronics) used to record the image. Specifically, the
integration time of the image sensor is the key limiting factor, since the
goal in imaging is to record the photons transiting the edges and the center
of the lens within the same frame. To illustrate this limit, we plot the time
delay between photons arriving from the edges of the lens compared to the ones
arriving along the optical axis for different lens diameters and numerical
apertures (NAs) in Fig. 1a. As expected, this time delay increases with NA and
with diameter. However, even for extremely fast detectors, say 1 MHz, lens
diameter of 1000 m with NA as high as 0.85 is possible. For ultrafast
detectors (1 GHz), the lens diameter is limited to approximately 1 m at NA
$\sim$ 0.9. Note that these limits are independent of the bandwidth of the
lens itself. The pixels of the sensor accumulate the photons during the
exposure time period. Imaging is possible as long as the photons passing the
edges of the lens arrive within the same interval of integration as those
along the optical axis. Here, we exploit this concept to demonstrate a
visible-band flat lens (Figs. 1b and 1c) with diameter = 100 mm and focal
length = 200 mm, which corresponds to NA = 0.24 (f/2). The focal length of
such a lens remains fixed (200 mm) over an operating bandwidth of 400 nm to
800 nm, which corresponds to a radial-frequency bandwidth, $\Delta\omega\ =\
2.4\times{10}^{15}$ rad/s, many orders of magnitude larger than the strict
limit dictated above. We note that bandwidth of this lens is far below the
limit illustrated in Fig. 1a, suggesting significant room for improvement from
our current demonstration.

Fig. 1:  Size-bandwidth limits of flat lenses. (a) Time delay (ns) experienced
by marginal rays relative to on-axis rays when a flat lens is illuminated by a
collimated plane wave at normal incidence as function of numerical aperture
(NA). Inset shows the relevant geometry. Horizontal lines labelled 1 MHz and 1
GHz denote the limits on NA dictated by the bandwidth of the image sensor for
different lens radii. (b) Photograph of the MDL with diameter = 100 mm and
focal length = 200 mm (NA = 0.24). Bottom insets show the design (left) and
optical micrograph (right) of the central rings of the MDL. (c) Design of a
multilevel diffractive lens (MDL) with diameter = 100 mm and focal length of
200 mm. (d) Bottom inset shows magnified view of the central 100 rings.

Increasing the bandwidth of diffractive lenses have been subject of research
for many years using approaches including diffractive-refractive hybrid
systems (9). multi-order lenses (10), transformation-optics lenses (11), and
more recently, multi-level diffractive lenses (12). Recent design of a 240 mm-
diameter harmonic diffractive lens for the astronomical R band (589-727nm) has
revived interest in these approaches for weight reduction in space-based
telescopes (13). Although dispersion engineering via subwavelength features is
promising for broadband flat lenses, the fabrication complexity and the
requirement of high-refractive index materials has limited demonstrations to
very small diameters (14 – 16).

## Design

The MDL is comprised of concentric rings of fixed width and varying heights.
We exploit the cylindrical symmetry of this geometry to implement the scalar-
diffraction model using the Hankel transform, which dramatically reduces
computational complexity (17). A gradient-assisted direct-binary-search
algorithm is used to vary the heights of the constituent rings with the goal
of minimizing the difference between the simulated point-spread function (PSF)
and the diffraction-limited Airy function for each wavelength sample, averaged
over the bandwidth of interest (18). In order to simplify fabrication, we
limited the ring width to 5 $\mu$m. This resulted in a total of 10,000 rings.
A second MDL with ring width of 2.5 $\mu$m (20,000 rings) was also fabricated
with slightly smaller bandwidth. Although the designed bandwidth of this
second lens was 450 nm to 650 nm, we noted that the lens was achromatic from
450 nm to 800 nm (see section 1 of supplement). In both cases, the ring
heights varied between 0 and 2.4 $\mu$m, which was dictated by our grayscale
lithography process. The measured dispersion curve of a positive-tone
photoresist (Shipley S1813, Microchem) was also used in the design. We further
note that iterative optimization-based inverse design is now commonly used in
all flat-lens implementations including metalenses (19).

## Fabrication

The MDL geometry was patterned in the photoresist layer (maximum thickness =
2.4 $\mu$m) on top of a double-side polished sodalime glass wafer (diameter =
150 mm, thickness = 550 $\mu$m). Patterning was performed using optical
grayscale lithography (see section 2 of supplement). The fabricated device is
shown in Fig. 1c. An optical micrograph of the central rings is compared to
the design in the inset panels. A photograph of the MDL along with a plano-
convex refractive singlet (diameter = 100 mm, focal length = 200 mm at
$\lambda=587.6$ nm, Edmund Optics, Model 27-501) is shown in Fig. S3.1
(supplement). The total weight of the MDL is 25 g, which is dominated by the
weight of the glass wafer. Note that the wafer has no optical function, merely
serving as a mechanical support. In comparison, a 100 mm-diameter commercial
plano-convex refractive singlet weighs $\sim 211$ g. But we emphasize that the
refractive singlet is, in fact not achromatic. A fully achromatic refractive
imaging system will require multiple aspheric lenses and will be considerably
heavier, and more expensive. Careful measurements of the height distribution
of the fabricated rings confirm that the standard deviation of the height
errors (compared to the designed values) is $\sim 120$ nm (see section 2.1 in
the supplement).

## Experiments

We first measured the point-spread functions (PSFs) of the lenses as function
of incident wavelength. Details of the experiment are provided in section 4
the supplement. Briefly, the setup consisted of a wavelength-tunable
collimated beam (supercontinuum source SuperK FIU-15, NKT Photonics, connected
to a SuperK VARIA filter, NKT Photonics), which was expanded by a series of
negative lenses over $\sim 1.5$ m to produce a large diameter (approx. 230 mm)
beam. The small divergence in the resulting beam caused a slight shift in the
focal length of the MDL from 200 mm to $\sim 230$ mm. This was confirmed using
a Shack-Hartmann sensor by measuring the focal length of the commercial plano-
convex lens. The PSFs were recorded directly on a monochrome CMOS sensor (The
Imaging Source, DMM 27UP031-ML, pixel pitch = 2.2 $\mu$m) and the results are
summarized in Fig. 2a. The distance between the lens and the sensor was fixed
at 230 mm for all measurements. The MDL clearly produces focused spots at all
wavelengths from 400 nm to 800 nm. Corresponding measurements for the
refractive lens are included in the supplement confirming that the refractive
lens is not achromatic as expected (see section 3 in the supplement). Fig. 2b
shows the full-width-at-half-maximum (FWHM) of the focal spot produced by the
MDL as shown in Fig. 2a, with respect to wavelength. Fig. 2c shows a plot of
the Strehl ratio versus wavelength (average value = 0.38). The corresponding
plots for the refractive lens are in section 3 (supplement). We note that the
average Strehl ratio of the MDL is higher than that of the refractive singlet
(at its design wavelength). The FHWM of the MDL PSFs are similar to those of
the refractive lens. The background of the MDL reaches the average dark-frame
value within a radius of $\sim 1.5$ mm from the peak. In comparison, the
refractive singlet is clearly not achromatic. In section 8 of the supplement,
we analyzed the PSFs of a commercial achromatic reractive lens (Thorlabs Model
AC127-025-A). Both simulations (Fig. S8.1) and experiments (Fig. S8.2)
confirmed that the MDL outperforms the achromat. As discussed later. we
hypothesize that the differences in the background for the simulated and
measured PSFs of the MDL are due to the 15 nm bandwidth of the experiment
(smallest achievable value in our measurement setup). The off-axis PSFs of the
MDL were also characterized, and we confirmed a field of view of approximately
4∘ (see Fig. 3a and details in section 4 of supplement).

Fig. 2:  Characterization of focusing. (a) Measured point-spread functions
(PSFs) for various central wavelengths (bandwidth = 15 nm) and broadband
illumination (400 - 800 nm). (b) Plot showing full-width-at-half-maximum
(FWHM) of the measured PSFs in (a) with respect to wavelength as well as the
corresponding diffraction limited spot size. (c) Strehl Ratio as function of
wavelength. (d) Simulated and (e) measured radial cross-sections of a subset
of the PSFs to emphasize the background level. The measured sensor dark level
is shown as black line. The distance between the MDL and the sensor is fixed
at 230 mm for all experiments. This is slightly longer than the designed focal
length of 200 mm due to the small curvature of the incident beam.

Images of a USAF resolution chart were recorded using the MDL with the same
sensor (Figs. 3b and 3c). The object and image distances were 400 mm and 400
mm in Fig. 3b, and 216 mm and 2735 mm in Fig. 3c, respectively. The smallest
resolved lines are of Group 7, Element 4 corresponding to 118 line-pairs/mm.
All images were processed using standard deblurring techniques using ImageJ
(see section 5 in supplement). In each case, the exposure time was adjusted to
ensure that the frames were not saturated.

Next, we took the camera (MDL + image sensor) outside to photograph the moon
(details in section 5 of the supplement). The captured image of the moon
(November 30, 2020 in Salt Lake City, UT) in shown in bottom panel of Fig. 3d.

Fig. 3:  Imaging performance of the MDL. (a) Recorded off-axis PSFs under
various narrow-band illuminations (450 nm to 750 nm, bandwidth = 15 nm). (b)
Image of USAF resolution chart with object distance = 400 mm and image
distance = 400 mm. (c) Magnified image of the resolution chart with object
distance = 216 mm and image distance = 2735 mm. The smallest resolved lines
correspond to 181 lp/mm (Group 7, Element 4). (d) Photograph of the moon
captured with MDL-sensor distance = 200 mm.

Finally, we built a hybrid Keplerian telescope comprised of the MDL as the
primary lens and an achromatic refractive lens (diameter = 12.5 mm, focal
length = 25 mm, Model Thorlabs AC127-025A) as the secondary element (Fig. 4a).
The distance between the two lenses was 235 mm, while that from the secondary
lens to the sensor was 87 mm (see details in section 6 of supplement). The
recorded images of the resolution chart and that of the moon are in Figs. 4b
and 4c, respectively. These images clearly show the potential of the MDL to
replace components within conventional imaging systems to reduce overall
weight.

Fig. 4:  Hybrid diffractive-refractive telescope. (a) Combining MDL with a
refractive lens to create a hybrid Keplerian telescope. Magnified images
captured of (b) a resolution chart and (c) the Moon.

## Discussion

The MDL design was performed with discrete wavelength spacing of 25 nm from
400 nm to 800 nm. At these sampled wavelengths, the focusing performance is
good (see Fig. 2 and Fig. S7.1). However, we observed that for wavelengths
that were not sampled during design, the point-spread functions exhibit higher
background (section 7 of supplement). Since the minimum bandwidth of our
illumination system is 15 nm, this degrades the overall PSF of, even the
narrowband measurements. We confirmed this by simulating the PSF for a
continuous 15 nm spectrum with 1 nm spectral samples, and showed that it
agrees well with the experimental data (Fig. S7.10). This accounts for
somewhat increased background in the measured PSFs. It is also noted that such
spectral dependencies could potentially be used for spectral imaging (6, 20).
The ideal ring-width should be less than the diffraction-limited FWHM of the
smallest wavelength, which would be 0.5 x 400 nm / 0.24 = 833 nm. Since we
used a much larger ring-width (5 $\mu$m or 2.5 $\mu$m), higher diffraction
orders are required for focusing. This gives rise to not only increased
background (and consequent reduction in Strehl ratio), but also an extended
depth of focus (see Figs. S3.4 and section 7 in supplement). We speculate that
by using ring-widths smaller than the diffraction limit as well with over-
sampling in spectrum, it may be possible to overcome this spectral dependence,
and elucidating these limits will be subject of future research.

One can derive a simpler, but weaker bound on the lens bandwidth by appealing
to the fact that for high-contrast interference, the smallest optical-path-
length difference must be larger than the coherence length of the illumination
(21). The coherence length, $l_{c}$ is approximately related to the bandwidth,
$\Delta\lambda$ and the center wavelength, $\lambda_{0}$ as
$l_{c}\sim\frac{{\lambda_{0}}^{2}}{\Delta\lambda}$. The smallest path-length
difference between two points (spaced by the ring-width, $w$, on the MDL with
radius, $R$ and focal length, $f$) and the focus assuming plane-wave
illumination is given by:
$\sqrt{\left(R+w\right)^{2}+f^{2}}-\sqrt{R^{2}+f^{2}}$. In order to ensure
interference between light traversing these two points and the focus, the
coherence length of illumination must be greater than this path-length
difference, i.e.,
$l_{c}>\sqrt{\left(R+w\right)^{2}+f^{2}}-\sqrt{R^{2}+f^{2}}$. Putting these
together, we can derive a simple upper bound on relative bandwidth:
$\frac{\Delta\lambda}{\lambda_{0}}\leq\frac{\lambda_{0}}{\sqrt{\left(R+w\right)^{2}+f^{2}}-\sqrt{R^{2}+f^{2}}}$.
Since $w\ll\left\\{R,f\right\\}$, we can simplify the bound as
$\frac{\Delta\lambda}{\lambda_{0}}\leq\frac{\lambda_{0}\sqrt{R^{2}+f^{2}}}{Rw}$.
If we further note that
$w>\frac{min\left\\{\lambda\right\\}}{2NA}\geq\frac{\lambda_{0}}{2NA}$ to
ensure diffraction into non-zero orders, we obtain the bound as
$\frac{\Delta\lambda}{\lambda_{0}}\leq 2$, which is independent of NA or the
diameter of the lens. Since this is the same as the theoretical upper limit of
relative bandwidth,
$\frac{\Delta\lambda}{\lambda_{0}}=2\times\frac{\lambda_{max}-\lambda_{min}}{\lambda_{max}+\lambda_{min}}\leq
2$, we conclude that coherence imparts no fundamental limit on the operating
bandwidth of a lens. Rather, it is the recording device (image sensor) that
imparts an upper bound as explained in the introduction. In our experiments,
we have center wavelength, $\lambda_{0}=0.6\thinspace\mu$m, and
$\Delta\lambda=0.4\thinspace\mu$m, which gives
$\frac{\Delta\lambda}{\lambda_{0}}=0.67$, obeying this bound. In prior work,
(24), relative bandwidth close to this bound was achieved at the expense of
increased background (and consequent reduction in Strehl ratio).

In conclusion, we showed that it is indeed possible to achieve large aperture
(diameter = 100 mm), relatively low f# (f/2), flat (thickness =
$2.4\thinspace\mu$m) lenses that exhibit a large operating bandwidth (400 nm
to 800 nm). We experimentally confirmed achromatic focusing with Strehl ratio
higher than that of a refractive singlet at its design wavelength. We also
demonstrated standalone and hybrid refractive-diffractive telescopes. Due to
the extremely low weight, such lenses, especially when scaled to even larger
diameters, and when combined with advanced image-processing methods, can
enable new space-based telescopes for direct imaging of exoplanets, one of the
key challenges emphasized by the recent decadal survey in astronomy and
astrophysics (25).

## Acknowledgements

We thank Drs. Stacie Williams, Rohith Chandrasekhar, Augustine Urbas, Philip
Hon, Peter Crabtree, and David Monet for fruitful discussions. We also
gratefully acknowledge assistance from Heidelberg Instruments (Christian Bach
and Dominique Colle), the University of Utah Nanofab (Joseph Spencer and Brian
Baker) for fabrication, and from University of Utah Surface Science Lab (Brian
van Devener) for metrology assistance. We are grateful to Fernando Vasquez
Guevara and Hank Smith for feedback on the manuscript.

## Funding

DARPA FA8650-20-C-7020

ONR N00014-19-1-2458

NSF 936729

References and Notes

  1. 1.

D. Apai, T. D. Milster, D. W. Kim, A. Bixel, G. Schneider, B. V. Rackham, R.
Liang, and J. Arenberg, Nautilus observatory: a space telescope array based on
very large aperture ultralight diffractive optical elements, Proc. SPIE 11116,
1111608 (2019).

  2. 2.

A. V. Nikonorov, M. V. Petrov, S. A. Bibikov, P. Y. Yakimov, V. V. Kutikova,
Y. V. Yuzifovich, A. A. Morozov, R. V. Skidanov, and N. L. Kazanskiy, Toward
ultralightweight remote sensing with harmonic lenses and convolutional neural
networks, IEEE J. Sel. Top. Appl. Earth Observ. Remote Sensing 11, 3338–3348
(2018).

  3. 3.

J. A. Britten, et al, Large-aperture fast multilevel Fresnel zone lenses in
glass and ultrathin polymer films for visible and near-infrared applicatios,
Appl. Opt. 53(11) 2312-2316 (2014).

  4. 4.

H. A. MacEwen and J. B. Breckinridge, Large diffractive/refractive apertures
for space and airborne telescopes,” Proc. SPIE Sens. Syst. Space Appl. 8739,
873904 (2013).

  5. 5.

S. V. Beryugina, J. R. Kuhn, M. Langlois, G. Moretto, J. Krissansen-Totton, D.
Catling, J. L. Grenfell, T. Santi-Temkiv, K. Finster, J. Tarter, F. Marchis,
H. Hargitai and D. Apai, The Exo-Life Finder (ELF) telescope: New strategies
for direct detection of exoplanet biosignatures and technosignatures, Proc.
SPIE v. 10700, 1070041 (2018).

  6. 6.

M-L. Hsieh, T. D. Ditto, Y-W. Lee, S-H. Lin, H. J. Newberg and S-Y. Lin,
Experimental realization of a Fresnel hologram as a super spectral resolution
optical element, Sci. Rep. 11, 20764 (2021).

  7. 7.

A. She, S. Zhang. S. Shian, D. R. Clarke and F. Capasso, Large area
metalenses: design, characterization and mass manufacturing, Opt. Exp. 26(2)
1573-1585 (2018).

  8. 8.

F. Presutti and F. Monticone, Focusing on bandwidth: achromatic metalens
limits, Optica 7(6) 624-631 (2020).

  9. 9.

T. Stone and N. George, Hybrid diffractive-refractive lenses and achromats,”
Appl. Opt. 27(14) 2960-2971 (1988).

  10. 10.

D. W. Sweeney and G. E. Sommargren, Harmonic diffractive lenses, Appl. Opt.
34(14) 2469-2475 (1995).

  11. 11.

R. Yang, W. Tang and Y. Hao, A broadband zone plate lens from transformation
optics, Opt. Exp 19(13) 12348-12355 (2011).

  12. 12.

M. Meem, S. Banerji, A, Majumder, J. C. Garcia, O. Kigner, P. Hon, B. Sensale-
Rodriguez and R. Menon, Imaging from the visible to the longwave infrared via
an inverse-designed flat lens,” Opt. Exp. 29(13) 20715-20723 (2021).

  13. 13.

Z. Wang, Y. Kim, and T. Milster, High-harmonic diffractive lens color
compensation, Appl. Opt. 60(19) DD73-D82 (2021).

  14. 14.

A. Ndao, L. Hsu, J. Ha, J-H. Park, C. Chang-Hasnain and B. Kante, Octave
bandwidth photonic fishnet-achromatic-metalens, Nat. Comm.  11, 3205 (2020).

  15. 15.

S. Shreshta, A. C. Overvig, M. Lu, A. Stein and N. Yu, Broadband achromatic
dielectric metalenses, Light: Sci & Appl. 7, 85 (2018).

  16. 16.

W. T. Chen, A. Y. Zhu, J. Sisler, Z. Bharwani andd F. Capasso, A broadband
achromatic polarization-insensitive metalens consisting of anisotropic
nanostructures, Nat. Comm. 10, 355 (2019).

  17. 17.

J. W. Goodman, Introduction to Fourier Optics. New York: McGraw-Hill, (1996).

  18. 18.

S. Banerji and B. Sensale-Rodriguez, A computational design framework for
efficient, fabrication error-tolerant planar THz diffractive optical elements,
Sci. Rep. 9, 5801 (2019).

  19. 19.

R. Pestourie, C. Pérez-Arancibia, Z. Lin, W. Shin, F. Capasso, and Steven G.
Johnson, Inverse design of large-area metasurfaces, Opt. Exp. 26(26)
33732-33747 (2018).

  20. 20.

P. Wang and R. Menon, Computational multi-spectral video imaging, J. Opt. Soc.
Am. A 35(1), 189-199 (2018).

  21. 21.

R. Pestourie, C. Pérez-Arancibia, Z. Lin, W. Shin, F. Capasso, and Steven G.
Johnson, Inverse design of large-area metasurfaces, Opt. Exp. 26(26)
33732-33747 (2018).

  22. 22.

J. W. Goodman, Statistical Optics. John Wiley & Sons, 2000.

  23. 23.

C. Akcay, P. Parrein, and J. P. Rolland, Estimation of longitudinal resolution
in optical coherence imaging, Appl. Opt. 41(25) 5256-5262 (2002).

  24. 24.

M. Meem, S. Banerji, A, Majumder, J. C. Garcia, O. Kigner, P. Hon, B. Sensale-
Rodriguez and R. Menon, Imaging from the visible to the longwave infrared via
an inverse-designed flat lens, Opt. Exp. 29(13) 20715-20723 (2021).

  25. 25.

National Academies of Sciences, Engineering, and Medicine. 2021. Pathways to
Discovery in Astronomy and Astrophysics for the 2020s. Washington, DC: The
National Academies Press.


# Atomic indirect measurement and robust binary quantum communication under
phase-diffusion noise

Min Namkung  Jeong San Kim<EMAIL_ADDRESS>Department of Applied
Mathematics and Institute of Natural Sciences, Kyung Hee University, Yongin
17104, Republic of Korea

###### Abstract

It was known that a novel quantum communication protocol surpassing the shot
noise limit can be proposed by an atomic indirect measurement based on the
Jaynes-Cummings model. Moreover, the quantum communication with the atomic
indirect measurement can nearly achieve the Helstrom bound as well as the
accessible information when message is transmitted by an ideal coherent state.
Here, we show that the atomic indirect measurement is robust against the
phase-diffusion noise. By considering the error probability of discriminating
received signal, we show that the atomic indirect measurement can also nearly
achieve the Helstrom bound as well as the accessible information even the
channel is exposed to the phase-diffused noise. Moreover, we further show that
atomic indirect measurement outperforms the feedback-excluded receiver
composed of a photon number resolving detector and maximum-a-posteriori
decision rule when the standard deviation of the phase-diffusion channel is
not too large.

††preprint: APS/123-QED

##  I Introduction

Over several decades, communication protocols with conventional measurements
such as a homodyne measurement m.ban0  and a direct measurement k.tsujino
have been widely used for sharing information between distant parties
j.g.proakis ; i.a.burenkov . Although the conventional measurements can be
easily understood and implented simply by means of the electromagnetic theory
m.fox , conventional measurements can never surpass the shot noise limit
(standard quantum limit) due to the statistical and physical properties of the
optical signal and the measurement device i.a.burenkov .

In quantum theory, the shot noise limit can be surpassed by novel and
unconventional measurements using a displacement operation, a photon detector,
and real-time electric feedback g.cariolaro ; r.s.kennedy . It is also shown
that a certain type of unconventional measurement with real-time electric
feedback s.j.dolinar  can reach the Helstrom bound when the message is encoded
in one of two ideal coherent states r.j.glauber . Unfortunately, the real-time
electric feedback is generally hard to be implemented in practical quantum
communication g.cariolaro .

For this reason, several unconventional measurements have been proposed
without real-time electric feedback m.sasaki ; k.tsujino ; s.izumi ; r.han ;
m.namkung ; m.namkung2 ; m.t.dimario . Recently, it has been shown that the
atomic indirect measurement r.han ; m.namkung ; m.namkung2  can nearly achieve
the Helstrom bound when a message is encoded in one of several standard
r.j.glauber  or non-standard coherent states e.m.f.curado . However, these
works only consider the case of ideal quantum channels between a sender and a
receiver, which is hard to be assumed in practice. Thus, it is natural to
consider and analyze the quantum communication protocol performed under the
existence of noise. Recently, phase-diffusion noise has been considered in
several quantum communication protocols s.olivares ; m.t.dimario2 .

Here, we show that the atomic indirect measurement is robust against the
phase-diffusion noise. By considering the error probability of discriminating
received signal, we show that the atomic indirect measurement can also nearly
achieve the Helstrom bound as well as the accessible information even the
channel is exposed to the phase-diffused noise. Moreover, we further show that
atomic indirect measurement outperforms the feedback-excluded receiver
m.t.dimario ; m.t.dimario2  composed of a photon number resolving detector and
maximum-a-posteriori decision rule when the standard deviation of the phase-
diffusion channel is not too large. As the situation here deals with noisy
quantum communication channels, our result is more relevant to the general
quantum communication protocols than the previous results r.han ; m.namkung ;
m.namkung2 .

The article is organized as follows. In Section II, we briefly introduce the
preliminaries regarding the binary quantum communication scheme. In Section
III, we consider the minimum value of the error probability and the maximum
value of the Shannon mutual information as figures of merit for a quantum
communication protocol under phase-diffused noise. In Section IV, we show that
the atomic indirect measurement is still robust against the phase-diffusion
noise in terms of the figures of merit obtained in Secton III. Moreover, we
also show that the atomic indirect measurement outperforms the feedback-
excluded receiver m.t.dimario ; m.t.dimario2 . Finally, we conclude our
results in Section V.

##  II Quantum communication protocol under phase-diffusion noise

###  II.1 Binary quantum communication protocol

Suppose that Alice (sender) prepares one bit message $x\in\\{1,2\\}$ with a
prior probability $q_{x}\in\\{q_{1},q_{2}\\}$ and encodes it in a quantum
state $\rho_{x}\in\\{\rho_{1},\rho_{2}\\}$. In practical quantum
communication, a quantum channel $\mathcal{N}$ transforms the initial quantum
state $\rho_{x}$ to a noisy quantum state $\mathcal{N}(\rho_{x})$, then Bob
(receiver) performs a quantum measurement on the quantum state
$\mathcal{N}(\rho_{x})$ to recover the message $x$. If Bob’s measurement is
described by a positive-operator-valued-measurement (POVM)
$\mathcal{M}=\\{M_{1},M_{2}\\}$, the conditional probability that Bob obtains
a measurement outcome $y\in\\{1,2\\}$ is described by

$\mathrm{Pr}(y|x)=\mathrm{Tr}\left\\{\mathcal{N}(\rho_{x})M_{y}\right\\}.$ (1)

Using Eq. (1), the error probability that Bob’s measurement incorrectly
decides the Alice’s message can be given by

$P_{e}(\mathcal{E},\mathcal{M})=\sum_{x\not=y}q_{x}\mathrm{Tr}\left\\{\mathcal{N}(\rho_{x})M_{y}\right\\}=1-\sum_{x=1}^{2}\mathrm{Tr}\left\\{\mathcal{N}(\rho_{x})M_{x}\right\\},$
(2)

where $\mathcal{E}=\\{q_{x},\mathcal{N}(\rho_{x})\\}_{x=1}^{2}$ is the
ensemble composed of the transformed quantum states $\mathcal{N}(\rho_{x})$
with the corresponding prior probability $q_{x}$.

The minimum value of $P_{e}(\mathcal{E},\mathcal{M})$ in Eq. (2) over all
possible POVM $\mathcal{M}$ of Bob:

$P_{hel}(\mathcal{E})=\min_{\mathcal{M}}P_{e}(\mathcal{E},\mathcal{M}),$ (3)

is called the Helstrom bound c.w.helstrom . Moreover, the amount of
information shared between Alice and Bob is given by t.cover

$I(\mathcal{E},\mathcal{M})=\sum_{x,y=1}^{2}\mathrm{Pr}(x,y)\log_{2}\frac{\mathrm{Pr}(x,y)}{q_{x}\mathrm{Pr}(y)},$
(4)

where

$\mathrm{Pr}(x,y)=q_{x}\mathrm{Tr}\\{\mathcal{N}(\rho_{x})M_{y}\\},$ (5)

is a joint probability that Alice prepares a binary message $x\in\\{1,2\\}$
and Bob obtains a measurement outcome $y$ from the POVM
$\mathcal{M}=\\{M_{1},M_{2}\\}$.

$\mathrm{Pr}(y)=\sum_{x}\mathrm{Pr}(x,y),$ (6)

is a marginal probability that Bob obtains a measurement outcome $y$. The
quantity in Eq. (4) is also referred to as the Shannon mutual information. The
maximum value of Eq.(4) over all possible POVM $\mathcal{M}$ of Bob:

$I_{acc}(\mathcal{E})=\max_{\mathcal{M}}I(\mathcal{E},\mathcal{M}),$ (7)

is called the accessible information. Thus, the efficiency of a quantum
channel can be maximized by constructing POVMs realizing the Helstrom bound or
the accessible information.

Due to the optimization problem over all possible POVMs, the analytic
evaluation of the accessible information in Eq. (7) is generally hard except
for two qubit pure states r.han2  and some simple case of three pure states
d.kaszlikowski . Moreover, it is also known that the POVM realizing the
Helstrom bound is not always equal to that of the accessible information m.ban
; d.kaszlikowski .

###  II.2 Phase-diffusion noise

In quantum communication, the phase-diffusion noise is described as the
following quantum channel s.olivares

$\mathcal{N}_{\sigma}(\cdot)=\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}d\phi
e^{-\frac{\phi^{2}}{2\sigma^{2}}}U(\phi)(\cdot)U^{\dagger}(\phi),$ (8)

where $\sigma$ is the standard deviation of the phase $\phi\in\mathbb{R}$,
$U(\phi)$ is a unitary operator

$U(\phi):=e^{i\phi a^{\dagger}a}.$ (9)

Here, $a$ and $a^{\dagger}$ are annihilation and creation operators,
respectively, such that

$\displaystyle a|n\rangle$ $\displaystyle=\sqrt{n}|n-1\rangle,\ \forall
n\in\mathbb{Z}^{+},$ $\displaystyle a^{\dagger}|n\rangle$
$\displaystyle=\sqrt{n+1}|n+1\rangle,\ \forall n\in\mathbb{Z}^{+}\cup\\{0\\},$
(10)

with the Fock basis $\\{|n\rangle:n\in\mathbb{Z}^{+}\cup\\{0\\}\\}$.

If Alice encodes her message $x\in\\{1,2\\}$ in an ideal coherent state
r.j.glauber

$|\alpha_{x}\rangle=e^{-\frac{1}{2}|\alpha_{x}|^{2}}\sum_{n=0}^{\infty}\frac{\alpha_{x}^{n}}{\sqrt{n!}}|n\rangle,\
\ \alpha_{x}\in\mathbb{R},$ (11)

then the phase-diffusion noise in Eq. (8) transforms the ideal coherent state
in Eq. (11) to

$\tau_{x}:=\mathcal{N}_{\sigma}(|\alpha_{x}\rangle\langle\alpha_{x}|)=\frac{1}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}d\phi
e^{-\frac{\phi^{2}}{2\sigma^{2}}}|\alpha_{x}e^{i\phi}\rangle\langle\alpha_{x}e^{i\phi}|=e^{-\alpha_{x}^{2}}\sum_{n,m=0}^{\infty}\frac{e^{-\frac{1}{2}\sigma^{2}(n-m)^{2}}}{\sqrt{n!m!}}\alpha_{x}^{n+m}|n\rangle\langle
m|.$ (12)

The state Eq. (12) is called the phase-diffused coherent state. The rest of
this section is to evaluate the Helstrom bound and the accessible information
of the phase-diffusion noise described as a quantum channel in Eq. (8).

####  II.2.1 Helstom bound

Suppose that Alice encodes a binary message $x\in\\{1,2\\}$ to a coherent
state $|\alpha_{x}\rangle$ with a prior probabilities $q_{x}$, and sends it to
Bob through a phase-diffusion channel $\mathcal{N}_{\sigma}$. To decode the
message transmitted as the phase-diffused coherent state
$\tau_{x}=\mathcal{N}_{\sigma}(|\alpha_{x}\rangle\langle\alpha_{x}|)$ in Eq.
(12), Bob needs to perform the measurement on $\tau_{x}$. The Helstrom bound
for these binary phase-diffused coherent states
$\mathcal{E}_{\sigma}=\\{q_{x},\tau_{x}\\}_{x=1}^{2}$ is given by s.olivares

$P_{hel}(\mathcal{E}_{\sigma})=\frac{1}{2}-\frac{1}{2}\left\|\Lambda_{\sigma}\right\|_{1},$
(13)

where $\Lambda_{\sigma}$ is a Hermitian operator

$\Lambda_{\sigma}=\sum_{n,m=0}^{\infty}\frac{e^{-\frac{1}{2}\sigma^{2}(n-m)^{2}}}{\sqrt{n!m!}}\left(q_{1}e^{-\alpha_{1}^{2}}\alpha_{1}^{n}\alpha_{1}^{m}-q_{2}e^{-\alpha_{2}^{2}}\alpha_{2}^{n}\alpha_{2}^{m}\right)|n\rangle\langle
m|.$ (14)

####  II.2.2 Accessible information

Although the the analytical form of fhe accessible information is not known
except for the case that binary pure states are discriminated by 2 element
POVM r.han2 , we can efficiently approximate the accessible information by
using the steepest ascent method j.rehacek ; let us consider the Hermitian
operator defined as

$R_{y}=\sum_{x=1}^{2}q_{x}\tau_{x}\log_{2}\left(\frac{\mathrm{Pr}(x,y)}{q_{x}\mathrm{Pr}(y)}\right),\
\ y=1,2$ (15)

where $\mathrm{Pr}(x,y)$ is the joint probability in Eq. (5) and
$\mathrm{Pr}(y)$ is the marginal probability in Eq. (6) when Bob performs the
POVM $\mathcal{M}=\\{M_{1},M_{2}\\}$. Now, we perform the iterations composed
of the following two steps,

  * •

Define $G_{y}=\mathbb{I}+\lambda(R_{y}-\sum_{z=1}^{2}R_{z}M_{z})$ and compute
$\widetilde{M}_{y}=G_{y}^{\dagger}M_{y}G_{y}$, where $\mathbb{I}$ is an
identity operator and $\lambda$ is a small positive number.

  * •

Update new POVM elements as $S^{-1/2}\widetilde{M}_{y}S^{-1/2}\rightarrow
M_{y}$, where $S=\sum_{z=1}^{2}\widetilde{M}_{z}$.

It is noted that newly updated POVM converges to a POVM satisfying the
stationary conditions m.ban ; j.rehacek

$M_{y}\Gamma=M_{y}R_{y},\ \ R_{y}M_{y}=\Gamma M_{y},$ (16)

where

$\Gamma=\sum_{y}R_{y}M_{y}=\sum_{y}M_{y}R_{y}.$ (17)

Eq. (16) is a necessary condition that the POVM
$\mathcal{M}=\\{M_{1},M_{2}\\}$ maximizes the Shannon mutual information of
Eq.(4).

Because $\tau_{x}$ is an infinite-dimensional quantum state, it also needs to
be approximated onto some finite-dimensional quantum system as s.guerrini

$\tau_{x}\approx
e^{-\alpha_{x}^{2}}\sum_{n=0}^{N}\sum_{m=0}^{M}\frac{e^{-\frac{1}{2}\sigma^{2}(n-m)^{2}}}{\sqrt{n!m!}}\alpha_{x}^{n+m}|n\rangle\langle
m|,$ (18)

for sufficiently large $N$ and $M$.

Figure 1: Schematic of the quantum communication protocol with an atomic
indirect measurement under a phase-diffusion channel $\mathcal{N}$. Here,
$\mathcal{E}_{\sigma}$ is an ensemble of Alice,
$\tau_{x}=\mathcal{N}_{\sigma}(|\alpha_{x}\rangle\langle\alpha_{x}|)$ is a
phase-diffused coherent state, and $\mathcal{M}^{(at)}$ is an atomic indirect
measurement. In $\mathcal{M}^{(at)}$, $\varsigma=|g\rangle\langle g|$ is an
initial (ground) atomic state on a Hilbert space $\widetilde{\mathcal{H}}$,
$\Pi$ is an atomic projective measurement, and $U$ is a unitary operator.

##  III Binary quantum Communication protocol
with atomic indirect measurement

The atomic indirect measurement is described by the four elements m.ozawa

$\\{\widetilde{\mathcal{H}},\varsigma,\Pi,U\\},$ (19)

where $\widetilde{\mathcal{H}}$ is a Hilbert space describing a two-level
atom, $\varsigma$ is the ground state of the two-level atom,
$\Pi=\\{\Pi_{1},\Pi_{2}\\}$ is an projective measurement on the two-level
atom, and $U$ is the unitary operator describing the interaction between the
light and the two-level atom. The atomic indirect measurement is illustrated
in Fig. 1.

By denoting $|g\rangle$ and $|e\rangle$ as the ground and excited states of
the two-level atom, respectively, we have

$\widetilde{\mathcal{H}}=\mathrm{span}\\{|g\rangle,|e\rangle\\},\ \
\varsigma=|g\rangle\langle g|.$ (20)

Moreover, the elements of $\Pi$ can also be written as
$\Pi_{y}=|\pi_{y}\rangle\langle\pi_{y}|$ where

$|\pi_{1}\rangle=\cos\theta|g\rangle+e^{i\xi}\sin\theta|e\rangle,\ \
|\pi_{2}\rangle=\sin\theta|g\rangle-e^{i\xi}\cos\theta|e\rangle.$ (21)

We consider the atomic indirect measurement based on the Jaynes-Cummings model
e.t.jaynes  where the interaction between the light and the two-level atom is
described by the unitary operator m.namkung3

$U(t)=\exp\left\\{-i\Phi(t)(a\otimes\sigma_{+}+a^{\dagger}\otimes\sigma_{-})\right\\},$
(22)

with non-Hermitian operators

$\sigma_{+}=|e\rangle\langle g|,\ \ \sigma_{-}=|g\rangle\langle e|,$ (23)

and the time-dependent real parameter $\Phi=\Phi(t)$. If the atomic projective
measurement is performed at time $T$, the unitary operator $U$ in Eq. (19) is
given by $U=U(T)$.

After the phase-diffused coherent state $\tau_{x}$ in Eq. (12) interacts with
the two-level atom by the unitary operator $U(t)$, the state of the composite
system becomes

$U(t)\left(\tau_{x}\otimes\varsigma\right)U^{\dagger}(t).$ (24)

If Bob obtains an outcome $y\in\\{1,2\\}$ by measuring the two-level atom,
then the phase-diffused coherent state is transformed to the (unnormalized)
quantum state

$\mathrm{Tr}_{A}\left\\{\left(\mathbb{I}_{L}\otimes\Pi_{y}\right)U(t)\left(\tau_{x}\otimes\varsigma\right)U^{\dagger}(t)\left(\mathbb{I}_{L}\otimes\Pi_{y}\right)^{\dagger}\right\\}=K_{y}\tau_{x}K_{y}^{\dagger},$
(25)

where $\mathrm{Tr}_{A}$ denotes the partial trace over the atom system,
$\mathbb{I}_{L}$ denotes the identity operator of the light system, and
$\\{K_{1},K_{2}\\}$ is the set of the Kraus operators on the light system
m.namkung3

$\displaystyle
K_{1}=\cos\theta\cos\\{\Phi\sqrt{a^{\dagger}a}\\}-ie^{-i\xi}\sin\theta\sum_{n=0}^{\infty}\frac{(-1)^{n}}{(2n+1)!}\Phi^{2n+1}a(a^{\dagger}a)^{n},$
$\displaystyle
K_{2}=\sin\theta\cos\\{\Phi\sqrt{a^{\dagger}a}\\}+ie^{-i\xi}\cos\theta\sum_{n=0}^{\infty}\frac{(-1)^{n}}{(2n+1)!}\Phi^{2n+1}a(a^{\dagger}a)^{n}.$
(26)

In other words, the atomic indirect measurement can be described by a POVM
$\mathcal{M}^{(at)}=\\{K_{1}^{\dagger}K_{1},K_{2}^{\dagger}K_{2}\\}$ of the
light system. These Kraus operators are used for deriving both the error
probability and the Shannon mutual information.

We first evaluate the error probability of discriminating the phase-diffused
coherent states given by $\mathcal{E}_{\sigma}=\\{q_{x},\tau_{x}\\}_{x=1}^{2}$
From the definition of the error probability in Eq. (2), it is straightforward
to show that the error probability for the Kraus operators given by Eq. (26)
is

$\displaystyle P_{e}(\mathcal{E}_{\sigma},\mathcal{M}^{(at)})$
$\displaystyle=1-\frac{q_{1}}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}d\phi
e^{-\frac{\phi^{2}}{2\sigma^{2}}}\left\|K_{1}|\alpha_{1}e^{i\phi}\rangle\right\|^{2}-\frac{q_{2}}{\sqrt{2\pi}\sigma}\int_{-\infty}^{+\infty}d\phi
e^{-\frac{\phi^{2}}{2\sigma^{2}}}\left\|K_{2}|\alpha_{2}e^{i\phi}\rangle\right\|^{2},$
$\displaystyle=1-q_{1}e^{-\alpha_{1}^{2}}f_{1}^{(\sigma)}(\xi,\theta,\Phi)-q_{2}e^{-\alpha_{2}^{2}}f_{2}^{(\sigma)}(\xi,\theta,\Phi),$
(27)

where $f_{x}^{(\sigma)}(\xi,\theta,\Phi)$ are defined as

$\displaystyle f_{1}^{(\sigma)}$ $\displaystyle(\xi,\theta,\Phi)$
$\displaystyle=\sum_{n=0}^{\infty}\left\\{\frac{\alpha_{1}^{2n}}{n!}\cos^{2}\theta\cos^{2}(\Phi\sqrt{n})+\frac{\alpha_{1}^{2(n+1)}}{(n+1)!}\sin^{2}\theta\sin^{2}(\Phi\sqrt{n+1})+\frac{\alpha_{1}^{2n+1}e^{-\frac{1}{2}\sigma^{2}}}{\sqrt{n!(n+1)!}}\sin\xi\sin
2\theta\cos(\Phi\sqrt{n})\sin(\Phi\sqrt{n+1})\right\\},$ $\displaystyle
f_{2}^{(\sigma)}$ $\displaystyle(\xi,\theta,\Phi)$
$\displaystyle=\sum_{n=0}^{\infty}\left\\{\frac{\alpha_{2}^{2n}}{n!}\sin^{2}\theta\cos^{2}(\Phi\sqrt{n})+\frac{\alpha_{2}^{2(n+1)}}{(n+1)!}\cos^{2}\theta\sin^{2}(\Phi\sqrt{n+1})-\frac{\alpha_{2}^{2n+1}e^{-\frac{1}{2}\sigma^{2}}}{\sqrt{n!(n+1)!}}\sin\xi\sin
2\theta\cos(\Phi\sqrt{n})\sin(\Phi\sqrt{n+1})\right\\}.$ (28)

Here, the following integration is used to derive Eqs. (27) and (28):

$\int_{-\infty}^{+\infty}d\phi
e^{-\frac{\phi^{2}}{2\sigma^{2}}}\sin(\phi-\xi)=\sqrt{2\pi}\sigma
e^{-\frac{1}{2}\sigma^{2}}\sin\xi.$ (29)

We note that Eq. (27) encapsulates the result of Ref. r.han  as a special case
that $\sigma$ is equal to zero.

Now, we evaluate the Shannon mutual information shared between Alice and Bob
by discriminating the binary phase-diffused coherent states by the atomic
indirect measurement. From the phase-diffused coherent states in Eq. (12) and
the Kraus operators in Eq. (26), the joint probability $\mathrm{Pr}(x,y)$ in
Eq. (4) is straightforwardly evaluated as

$\displaystyle\mathrm{Pr}(1,1)=q_{1}e^{-\alpha_{1}^{2}}f_{1}^{(\sigma)}(\xi,\theta,\Phi),\
\
\mathrm{Pr}(1,2)=q_{1}e^{-\alpha_{1}^{2}}\bar{f}_{1}^{(\sigma)}(\xi,\theta,\Phi),$
$\displaystyle\mathrm{Pr}(2,1)=q_{2}e^{-\alpha_{2}^{2}}\bar{f}_{2}^{(\sigma)}(\xi,\theta,\Phi),\
\ \mathrm{Pr}(2,2)=q_{2}e^{-\alpha_{2}^{2}}f_{2}^{(\sigma)}(\xi,\theta,\Phi),$
(30)

where $f_{x}^{(\sigma)}(\xi,\theta,\Phi)$ are defined in Eq. (28) and
$\bar{f}_{x}^{(\sigma)}(\xi,\theta,\Phi)$ are defined as

$\displaystyle\bar{f}_{1}^{(\sigma)}$ $\displaystyle(\xi,\theta,\Phi)$
$\displaystyle=\sum_{n=0}^{\infty}\left\\{\frac{\alpha_{1}^{2n}}{n!}\sin^{2}\theta\cos^{2}(\Phi\sqrt{n})+\frac{\alpha_{1}^{2(n+1)}}{(n+1)!}\cos^{2}\theta\sin^{2}(\Phi\sqrt{n+1})-\frac{\alpha_{1}^{2n+1}e^{-\frac{1}{2}\sigma^{2}}}{\sqrt{n!(n+1)!}}\sin\xi\sin
2\theta\cos(\Phi\sqrt{n})\sin(\Phi\sqrt{n+1})\right\\},$
$\displaystyle\bar{f}_{2}^{(\sigma)}$ $\displaystyle(\xi,\theta,\Phi)$
$\displaystyle=\sum_{n=0}^{\infty}\left\\{\frac{\alpha_{2}^{2n}}{n!}\cos^{2}\theta\cos^{2}(\Phi\sqrt{n})+\frac{\alpha_{2}^{2(n+1)}}{(n+1)!}\sin^{2}\theta\sin^{2}(\Phi\sqrt{n+1})+\frac{\alpha_{2}^{2n+1}e^{-\frac{1}{2}\sigma^{2}}}{\sqrt{n!(n+1)!}}\sin\xi\sin
2\theta\cos(\Phi\sqrt{n})\sin(\Phi\sqrt{n+1})\right\\}.$ (31)

As shown in Eqs. (27)-(28) and Eqs. (30)-(31), the error probability and the
Shannon mutual information are the nonlinear functions of the three parameters
$(\xi,\theta,\Phi)$. To use the conventional method of optimization, we need
to find the vanishing gradients of the error probability and the Shannon
mutual information. As this imposes us to deal with a system of nonlinear
equations, it is hard to analytically find the vanishing gradients. In next
section, we try the numerical method to optimize the error probability and the
Shannon mutual information over all possible atomic indirect measurements

##  IV numerical optimization

We numerically obtain the minimum value of the error probability and the
maximum value of the Shannon mutual information over all possible atomic
indirect measurements. Here, the average of the mean photon number of the
ensemble $\mathcal{E}_{\sigma}=\left\\{q_{x},\tau_{x}\right\\}_{x=1}^{2}$ of
Alice is given by m.t.dimario2

$\bar{n}=q_{1}\mathrm{Tr}\left\\{\tau_{1}a^{\dagger}a\right\\}+q_{2}\mathrm{Tr}\left\\{\tau_{2}a^{\dagger}a\right\\}=q_{1}\alpha_{1}^{2}+q_{2}\alpha_{2}^{2},$
(32)

where $a$ and $a^{\dagger}$ are annihilation and creation operators,
respectively, and $\alpha_{x}$ is the amplitude of the phase-diffused coherent
state $\tau_{x}$ in Eq. (12).

For a fixed $\bar{n}$, we consider the following two cases: The first case is
when $\alpha_{1}=0$ or $\alpha_{2}=0$. In this case, $\mathcal{E}_{\sigma}$
represents on-off keying (OOK) signal. The second case is when
$\alpha_{1}=\alpha_{2}$. In this case, $\mathcal{E}_{\sigma}$ represents
binary phase shift keying (BPSK) signal. We compare the minimum value of the
error probability and the maximum value of the Shannon mutual information over
all possible atomic indirect measurements with these of the feedback-excluded
receivers composed of a photon number-resolving detector and maximum-a-
posteriori decision rule m.t.dimario ; m.t.dimario2 . Throughout this section,
we consider the case of $q_{1}=q_{2}$, that means equiprobable input signals.

###  IV.1 Numerical minimum value of error probability

Figure 2: The minimum error probability of the atomic indirect measurement
when $\bar{n}=0.5$ in the region of $0\leq\sigma\leq 1.2$. In this figure,
BPSK signal is considered in Fig. 2(a) whereas OOK signal is in Fig. 2(b). The
thick solid red lines show the minimum error probability of the atomic
indirect measurement. The dotted black lines show the minimum error
probability of the feedback-excluded receiver that the photon number
resolution is equal to one and the invisibility is given by 0.998. The solid
blue lines show the Helstrom bound.

Fig. 2 depicts the minimum error probability of the atomic indirect
measurement when $\bar{n}=0.5$ in the region of $0\leq\sigma\leq 1.2$. In this
figure, BPSK signal is considered in Fig. 2(a) whereas OOK signal is in Fig.
2(b). The thick solid red lines show the minimum error probability of the
atomic indirect measurement. The dotted black lines show the minimum error
probability of the feedback-excluded receiver that the photon number
resolution is equal to one and the invisibility is given by 0.998 m.t.dimario
; m.t.dimario2 . The solid blue lines show the Helstrom bound.

According to Fig. 2(a), the minimum error probability of the atomic indirect
measurement is closer to the Helstrom bound than that of the feedback-excluded
receiver. In Fig. 2(b), the minimum error probability of the feedback-excluded
receiver is closer to the Helstrom bound in most range of $\sigma$. This
result means that the atomic indirect measurement outperforms the feedback-
excluded receiver when BPSK signal was sent through the phase-diffused quantum
channel instead of OOK signal. For this reason, we mainly focus on the quantum
communication protocol with BPSK signal.

Figure 3: The minimum error probability of the atomic indirect measurement
when the message is encoded in BPSK signal. Here, we consider the case that
$\bar{n}=0.75$ in the region of $0\leq\sigma\leq 1.2$ (Fig.3(a) and (b)) and
$\bar{n}=1.0$ in the region of $0\leq\sigma\leq 1.0$ (Fig.3(c) and (d)). In
this figure, we consider the case that $\bar{n}=0.75$ in the region of
$0\leq\sigma\leq 1.2$ (Fig.3(a) and (b)) and $\bar{n}=1.0$ in the region of
$0\leq\sigma\leq 1.0$ (Fig.3(c) and (d)). The thick solid red lines show the
minimum error probability of the atomic indirect measurement. The dotted,
dashed-dotted, and dashed black lines show the minimum error probability of
the feedback-excluded receiver with the photon number resolution one, two, and
three, respectively. The invisibility of the feedback-excluded receiver is set
to be 0.998. The solid blue lines show the Helstrom bound.

Fig. 3 depicts the minimum error probability of the atomic indirect
measurement when the message is encoded in BPSK signal. In this figure, we
consider the case that $\bar{n}=0.75$ in the region of $0\leq\sigma\leq 1.2$
(Fig.3(a) and (b)) and $\bar{n}=1.0$ in the region of $0\leq\sigma\leq 1.0$
(Fig.3(c) and (d)). The thick solid red lines show the minimum error
probability of the atomic indirect measurement. The dotted, dashed-dotted, and
dashed black lines show the minimum error probability of the feedback-excluded
receiver with the photon number resolution one, two, and three, respectively
m.t.dimario ; m.t.dimario2 . The invisibility of the feedback-excluded
receiver is set to be 0.998 m.t.dimario2 . The solid blue lines show the
Helstrom bound.

According to Fig. 3(a) and Fig. 3(b), the minimum error probability of the
atomic indirect measurement is closer to the Helstrom bound than that of the
feedback-excluded receiver with the photon number resolution one and two. When
the photon number resolution is three, the minimum error probability of the
atomic indirect measurement is closer to the Helstrom bound on the range of
$\sigma<0.61707$. In Fig. 3(c) and Fig. 3(d), the minimum error probability of
the atomic indirect measurement is closer to the Helstrom bound than that of
the feedback-excluded receiver with the photon number resolution one. When the
photon number resolution is two (three), the minimum error probability of the
atomic indirect measurement closer to the Helstrom bound on the range of
$\sigma<0.29838$ ($\sigma<0.33528$). From these results, we note that the
atomic indirect measurement outperforms the feedback-excluded receiver in the
aspect of the error probability when BPSK signal was sent through the phase-
diffused quantum channel with not too large $\sigma$.

Figure 4: maximal Shannon mutual information of the atomic indirect
measurement when the message is encoded in BPSK signal. In this figure, we
consider the case that $\bar{n}=0.5$ (Fig.4(a) and (b)) and $\bar{n}=0.75$
(Fig.4(c) and (d)) in the region of $0\leq\sigma\leq 1.2$. The thick solid
black lines show the maximal Shannon mutual information of the atomic indirect
measurement. The dotted, dashed-dotted, and dashed black lines show the
maximal Shannon mutual information of the feedback-excluded receiver with the
photon number resolution one, two, and three, respectively . The invisibility
of the feedback-excluded receiver is set to be 0.998. The solid blue lines
show the accessible information.

###  IV.2 Shannon mutual information

Fig. 4 depicts the maximal Shannon mutual information of the atomic indirect
measurement when the message is encoded in BPSK signal. In this figure, we
consider the case that $\bar{n}=0.5$ (Fig.4(a) and (b)) and $\bar{n}=0.75$
(Fig.4(c) and (d)) in the region of $0\leq\sigma\leq 1.2$. The thick solid
black lines show the maximal Shannon mutual information of the atomic indirect
measurement. The dotted, dashed-dotted, and dashed black lines show the
maximal Shannon mutual information of the feedback-excluded receiver with the
photon number resolution one, two, and three, respectively m.t.dimario ;
m.t.dimario2 . The invisibility of the feedback-excluded receiver is set to be
0.998 m.t.dimario2 . The solid blue lines show the accessible information.

According to Fig. 4(a) and Fig. 4(b), the maximal Shannon mutual information
of the atomic indirect measurement is closer to the accessible information
than that of the feedback-excluded receiver when the photon number resolution
is one. When the photon number resolution is two (three), the maximal Shannon
mutual information of the atomic indirect measurement is closer to the
accessible information on the range of $\sigma<0.68653$ ($\sigma<0.4062$). In
Fig. 4(c) and Fig. 4(d), the maximal Shannon mutual information of the atomic
indirect measurement is closer to the accessible information than that of the
feedback-excluded receiver when the photon number resolution is one. When the
feedback-excluded receiver that the photon number resolution is equal to two
(three), the maximal Shannon mutual information of the atomic indirect
measurement is closer to the accessible information on the range of
$\sigma<0.3453$ ($\sigma<0.2515$). From these results, we note that the atomic
indirect measurement outperforms the feedback-excluded receiver in the aspect
of the Shannon mutual information when BPSK signal was sent through the phase-
diffused quantum channel with not too large $\sigma$.

##  V Conclusion

We have shown that the atomic indirect measurement is robust against the
phase-diffusion noise. By considering the error probability of discriminating
received signal, we have shown that the atomic indirect measurement can also
nearly achieve the Helstrom bound as well as the accessible information even
the channel is exposed to the phase-diffused noise. Moreover, we further have
shown that atomic indirect measurement outperforms the feedback-excluded
receiver composed of a photon number resolving detector and maximum-a-
posteriori decision rule m.t.dimario ; m.t.dimario2  when the standard
deviation of the phase-diffusion channel is not too large.

Our result is more relevant to the general quantum communication protocols
than the previous results that are only about the ideal quantum channels r.han
; m.namkung ; m.namkung2 . It is recently shown that squeezing operation can
be used to reduce the phase-diffusion noise occured in the coherent state
s.cialdi ; g.carrara ; m.n.notarnicola . Thus, it is an interesting future
work to consider the hybrid protocol composed of the atomic indirect
measurement together with the squeezing operation for robust quantum
communication against the phase-diffusion noise.

###### Acknowledgements.

We thank Dr. Donghoon Ha for his insightful discussions. This work was
supported by Quantum Computing Technology Development Program
(NRF2020M3E4A1080088) through the National Research Foundation of Korea (NRF)
grant funded by the Korea government (Ministry of Science and ICT).

## References

  * (1) M. Ban, “Quantum theory of the homodyne-direct receiver”, J. Mod. Opt. 44, 1175 (1997).
  * (2) K. Tsujino et al., “Sub-shot-noise-limit discrimination of on-off keyed coherent signals via a quantum receiver with a superconducting transition edge sensor”, Opt. Express 18, 8107 (2010).
  * (3) J. G. Proakis and M. Salehi, Digital Communications, 5th ed. (McGraw-Hill, 2008).
  * (4) I. A. Burenkov, M. V. Jabir, and S. V. Polyakov, “Practical quantum-enhanced receivers for classical communication”, AVS Quantum Sci. 3, 025301 (2021).
  * (5) M. Fox, Quantum Optics: An Introduction (Oxford University Press, 2006).
  * (6) G. Cariolaro, Quantum Communications (Springer, 2015).
  * (7) R. S. Kennedy, “A Near-Optimum Receiver for the Binary Coherent State Quantum Channel”, MIT Res. Lab. Elect. Q. Prog. Rep. 108, 219-225 (1973).
  * (8) S. Dolinar, “An optimum receiver for the binary coherent state quantum channel”, MIT Res. Lab. Elect. Q. Prog. Rep. 111, 115-120 (1973).
  * (9) R. J. Glauber, “Coherent and Incoherent States of the Radiation Field”, Phys. Rev. 131, 2766 (1963).
  * (10) M. Sasaki and O. Hirota, “Optimal decision scheme with a unitary control process for binary quantum-state signals”, Phys. Rev. A 54, 2728 (1996).
  * (11) S. Izumi et al., “Projective measurement onto arbitrary superposition of weak coherent state bases”, Sci. Rep. 8, 2999 (2018).
  * (12) R. Han, J. A. Bergou, and G. Leuchs, “Near optimal discrimination of binary coherent signals via atom-light interaction”, New J. Phys. 20, 043005 (2018).
  * (13) M. Namkung and Y. Kwon, “Almost minimum error discrimination of N-ary weak coherent states by Jaynes-Cummings Hamiltonian dynamics”, Sci. Rep. 9, 19664 (2019).
  * (14) M. Namkung and J. S. Kim, “Indirect Measurement for Optimal Quantum Communication Enhanced by Binary Non-standard Coherent States”, J. Opt. Soc. Am. B 39, 1247 (2022).
  * (15) M. T. DiMario and F. E. Becerra, “Robust Measurement for the Discrimination of Binary Coherent States”, Phys. Rev. Lett. 121, 023603 (2018).
  * (16) E. M. F. Curado, S. Faci, J.-P. Gazeau, and D. Noguera, “Lowering the Helstrom bound with non-standard coherent states”, J. Opt. Soc. Am. B 38, 3556 (2021).
  * (17) S. Olivares, S. Cialdi, F. Castelli, and M. G. A. Paris, “Homodyne detection as a near-optimum receiver for phase-shift keyed binary communication in the presence of phase diffusion”, Phys. Rev. A 87, 050303(R) (2013).
  * (18) M. T. DiMario, L. Kunz, K. Banaszek, and F. E. Becerra, “Optimized communication strategies with binary coherent states over phase noise channels”, NPJ Quant. Inf. 5, 65 (2019).
  * (19) M. Ban, K. Kurokawa, R. Momose, and O. Hirota, “Optimal Measurements for Discrimination Among Symmetric Quantum States and Parameter Estimation”, Int. J. Theor. Phys. 36, 1269 (1997).
  * (20) J. Rehacek, B.-G. Englert, and D. Kaszlikowski, “Iterative procedure for computing accessible information in quantum communication”, Phys. Rev. A 71, 054303 (2005).
  * (21) T. Cover and J. A. Thomas, Elements of Information Theory (2nd Edition) (Wiley, 2006).
  * (22) C. W. Helstrom, Detection and Estimation Theory (Academic Press, 1976).
  * (23) R. Han, G. Leuchs, and M. Grassl, “Residual and Destroyed Accessible Information and Measurements”, Phys. Rev. Lett. 120, 160501 (2018).
  * (24) D. Kaszlikowski, A. Gopinathan, Y. C. Liang, L. C. Kwek, and B.-G. Englert, “Quantum cryptography: Security criteria reexamined”, Phys. Rev. A 70, 032306 (2004).
  * (25) M. Ozawa, “On information gain by quantum measurements of continuous observables”, J. Math. Phys. 25, 79 (1984).
  * (26) E. T. Jaynes and F. W. Cummings, “Comparison of quantum and semiclassical radiation theories with application to the beam maser”, Prof. IEEE 51, 89.
  * (27) M. Namkung and E. R. Loubenets, “Two-sequential conclusive discrimination between binary coherent states via indirect measurements”, Phys. Scr. 96, 105103 (2021).
  * (28) S. Guerrini, M. Z. Win, M. Chiani, and A. Conti, “Quantum Discrimination of Noisy Photon-Added Coherent States”, IEEE Journal on Selected Areas in Information Theory, 1, 469 (2020).
  * (29) S. Cialdi, E. Suerra, S. Olivares, S. Capra, and M. G. A. Paris, “Squeezing Phase Diffusion”, Phys. Rev. Lett. 124, 163601 (2020).
  * (30) G. Carrara, M. G. Genoni, S. Cialdi, M. G. A. Paris, and S. Olivares, “Squeezing as a resource to counteract phase diffusion in optical phase estimation”, Phys. Rev. A 102, 062610 (2020).
  * (31) M. N. Notarnicola, M. G. Genoni, S. Cialdi, M. G. A. Paris, and S. Olivares, “Phase noise mitigation by a realistic optical parametric oscillator”, J. Opt. Soc. Am. B 39, 1059 (2022).


# Towards Ethical Multimodal Systems

Alexis Roger
Department of Computer Science and Operations Research
University of Montreal
Mila - Quebec AI Institute
<EMAIL_ADDRESS>
Esma Aïmeur
Department of Computer Science and Operations Research
University of Montreal
Irina Rish
Department of Computer Science and Operations Research
University of Montreal
Mila - Quebec AI Institute

###### Abstract

Generative AI systems (ChatGPT, DALL-E, etc) are expanding into multiple areas
of our lives, from art Rombach et al. (2021) to mental health Rob Morris and
Kareem Kouddous (2022); their rapidly growing societal impact opens new
opportunities, but also raises ethical concerns. The emerging field of AI
alignment aims to make AI systems reflect human values. This paper focuses on
evaluating the ethics of multimodal AI systems involving both text and images
- a relatively under-explored area, as most alignment work is currently
focused on language models. We first create a multimodal ethical database from
human feedback on ethicality. Then, using this database, we develop
algorithms, including a RoBERTa-large classifier and a multilayer perceptron,
to automatically assess the ethicality of system responses.

##  1 Introduction

AI ethics encompasses safety, security, human concerns, and environmental
issues, gaining significance in high-school education and AI literacy, as
highlighted, for example, in Talagala (2022). Addressing AI ethics is
critical, underscored in papers like Concrete Problems in AI Safety (Amodei et
al. (2016)) and Unsolved Problems in ML Safety (Hendrycks et al. (2021)).

Efforts to regulate AI ethics include the Montreal Declaration for a
Responsible Development of Artificial Intelligence (University of Montreal
(2018)). The United States is developing an AI Bill of Rights (House (2022)),
and Europe is working on The Artificial Intelligence Act (Commision (2021)),
all aimed at ensuring AI’s safe development. Projects like the Moral Machine
by Awad et al. (2020) address AI ethical concerns. IBM’s AI Fairness 360
toolkit (Bellamy et al. (2018)) offers tools to detect bias and promote
fairness in algorithms, addressing issues like age or gender discrimination.
However, none of these initiatives have focused on large multimodal models.

Our Contributions. Similar to the Moral Machine project (Awad et al. (2020)),
our paper aims to create a dataset of ethical and unethical samples. However,
our focus is on multimodal models, which take both an image and a question as
input and provide an answer. In this paper, we present our results for the
MAGMA model by Eichenberg et al. (2022), while evaluation of other similar
multimodal models remains the topic of our ongoing work. Note that, as shown
in Figure 1, MAGMA can sometimes generate responses not well-aligned with
common ethical standard, thus serving as a motivation for our line of work.

Our first contribution is a dataset consisting of 789 question and image pairs
covering various ethical fields, including economics, medicine, society and
discrimination. Each pair includes an answer generated by the MAGMA model by
Eichenberg et al. (2022), along with user votes indicating whether it’s
ethical, unethical, or unclear. Unique user IDs are included to track
responses and identify abnormal behavior. This dataset, detailed in section 3,
maintains a balanced proportion of ethical and unethical responses.

Our second contribution is the development of an ethical evaluation pipeline
via crowd-sourcing. This lightweight, user-friendly pipeline gamifies the
evaluation process and is designed around Discord, a popular messaging
service. Although initially used for multimodal prompt evaluations, it can be
applied to other evaluation processes requiring more than a simple "like" from
social media, as explained in section 3.

Our third contribution involves a preliminary study of classification
algorithms to automatically assess multimodal systems. We evaluate existing
models and build our own, based on a multilayer perceptron using text and
image embeddings as input. We delve into this comparison in section 4.

##  2 Related Work

State of the art. Ethical considerations regarding Natural Language Processing
(NLP) models have been explored in prior research, notably in Hendrycks et al.
(2020). This study assessed various NLP models, including GPT-3 (few-shot
learner) (Brown et al. (2020)), BERT (Devlin et al. (2018)), RoBERTa-large
(Liu et al. (2019)), word averaging (Wieting et al. (2015)) and ALBERT-xxlarge
(Lan et al. (2019)), across five key values: justice, deontology, virtue,
utilitarianism, and commonsense. Larger fine-tuned models trained on more data
generally exhibited better performance. However, these models primarily
operated with textual input, while our focus is on multimodal input, combining
text and images.

A study on the ethics of multimodal systems was conducted by Layoun et al.
(2022), which involved evaluating the ethics of a chosen multimodal algorithm
and exploring avenues for model improvement. Notably, the authors faced
limitations due to a small hand-crafted dataset and manual evaluation, which
introduced bias and hindered scalability. Our aim is to address these
challenges.

Background. In this study we focus on a text and image combination as input
with a text output. Herein, we used the MAGMA model by Eichenberg et al.
(2022), as the authors made both the code (Constantin (CoEich) (2021)) and a
checkpoint of the trained model publicly available; evaluating more recent and
more advanced models of this type is a topic of our ongoing work. MAGMA is
based on the CLIP visual encoder (Radford et al. (2021)) and the GPT-J
language model (Ben Wang and Aran Komatsuzaki (2021)). MAGMA is also the model
on which the previous paper from Layoun et al. (2022) was based on. In that
paper the authors showed MAGMA had some critical ethical deficiencies (e.g.,
see Figure 1). We therefore decided to expand on this analysis.

In order to perform such an ethical evaluation, we needed a reliable dataset
of ethical and non-ethical examples on which we could rely. Due to the
multimodal aspect of our study, we were unable to find any available datasets.
In the preliminary study, Layoun et al. (2022) had used a small hand-crafted
dataset as a proof of concept. This database contained a few tens of examples
crafted and evaluated by the authors. This would not be scalable as we were
hoping to build a dataset of about a thousand examples. To this end we looked
at the paper by Bai et al. (2022) where they present a technique they used to
gather human feedback. After reviewing their method, we realized that we
needed to have a clear and straight-forward interface. Along with this we also
needed a way to evaluate the participating people, to ensure they had proper
ethics. These became our design goals for our own feedback gathering pipeline.

##  3 Crafting an Ethical Dataset

In order to build an ethical database we considered different alternatives.
Many constraints had to be taken into consideration when building this
database. For example, we had to avoid our personal bias but also wanted to
rapidly craft a massive database. We will go over the different steps used to
build our dataset one at the time, explaining our choices as we go.

Creating a framework. Our first step was to create a framework in which to
build the dataset. The first dataset of this kind, from the paper Layoun et
al. (2022), consisted of 100 handcrafted examples. This was a very labor
intensive process and was prone to the introduction of personal bias within
the dataset. Reproducing this method to build a neutral dataset an order of
magnitude bigger would not be feasible. The Amazon Mechanical Turk (MTurk)
service was not use as we feared that the monetary transaction involved could
influence the ethics.

We decided to use crowd sourcing to build our dataset. This would allow us to
gain more input on our ethical questions and mitigate individual biases. With
crowd sourcing, we could query different demographics, all the while
controlling the deployment and monitoring the differences between these
demographics. We could go as far as to confront the ethics of the different
populations and see the results. The most optimal way to do crowd sourcing is
to go through social media. The biggest platform with no ban on
algorithmically generated content is Discord. Hence, this is the platform that
was chosen.

Creating the original prompts. Our study involved using prompts to evaluate
the MAGMA model’s answers to image and question pairs. We employed the MAGMA
model (Eichenberg et al. (2022)) to generate responses, simplifying content
generation and enabling a comprehensive analysis of the model’s ethical
aspects and answer quality. To cover a wide range of ethical topics, we used
examples from the "Banque de cas éthiques" (Enseigner l’éthique (2021)) and
compiled 218 diverse images. We crafted 14 general prompts designed to elicit
ethically interesting responses and accounted for the model’s tone imitation
by using positive and negative words. After processing all prompts through the
MAGMA model, we observed that some responses lacked Latin alphabet characters,
resulting in 2,844 retained prompts after filtering out such responses. This
is the first indication that the MAGMA model we are using has some
shortcomings.

The inspiration of the "Banque de cas éthiques" (Enseigner l’éthique (2021))
allowed us to cover a wide array of ethics scenarios, including human values,
greed, exploitation, child labor, provocative advertisements, alcohol and
tobacco products, discrimination (sexism, racism, and disability
discrimination), family settings (abuse, drugs, and malnutrition), abortion,
euthanasia, doping, medical experiments, animal experiments, and "extreme"
situations (hostage taking and shootings).

Initial user feedback. With a substantial dataset of image, question, and
answer combinations, we initiated the evaluation process on our Discord bot.
Our system aims to simplify and gamify the process, making it user-friendly.
Upon issuing the command "$eval_batch" to our bot in a channel or via private
message, the bot presents 50 combinations one at a time, with a 5-second gap
between prompts for users to assess. Each prompt offers three reactions:
thumbs up for "ethical," thumbs down for "unethical," and a shrug for
"unclear." A full evaluation session takes approximately 5 minutes.

We enlisted 50 university students to test our bot and gather initial results.
These 50 evaluations covered 1108 prompts, which we classified as ethical or
not based on majority opinion. Surprisingly, we achieved an almost equal
distribution between ethical, unethical, and unclear prompts. However,
approximately two-thirds of the prompts received only a single evaluation,
raising concerns about potential user bias. In the Extended testing section,
we will address and mitigate this issue.

User trustworthiness. The main problem with our method is how we can trust the
users. We must ensure our users possess a generally approved ethical vision,
hence requiring safeguards to evaluate our users and ensure that they are not
actively attempting to pollute the results with false information.

Demographics control. Our first safeguard involves controlling the user
demographic. We began with a small group within our department for preliminary
tests and gradually expanded to include a broader demographic. While this
approach might limit the experiment’s scope, it allowed better control and
deliberate widening.

We then tested with 50 university students, primarily computer science
undergraduates. This choice raised concerns about the potential confinement of
our ethical discussions to local perspectives. However, our university’s
diverse demographics, including a substantial number of international
students, provided a range of viewpoints while still maintaining control to
prevent sabotage and ensure active participation. This is further expanded in
the Extended testing section.

Pre and post-tests. Our second safeguard involved user testing through pre-
test and post-test evaluations, each comprising of 5 hand-picked prompts. The
pre-test contained ethical, unethical, and unclear propositions, providing a
baseline for user evaluation. When users requested a batch, they received
these 5 pre-test prompts, 40 random ones, and 5 post-test prompts. Assessing
user responses to these standardized prompts helped gauge trustworthiness, and
these prompts had the most reliable data as they were evaluated by all users.

The pre-test established an initial baseline for user evaluation, while the
post-test ensured users’ attentiveness throughout the process. Comparing pre-
test and post-test responses also helped identify changes in user behavior.
It’s worth noting that if a user did not complete all 50 prompts, including
the post-test, the completed prompts were still considered.

Figure 2 illustrates the percentage of responses for each test prompt, with
the first five being pre-test prompts (shown in order in figure 1) and the
following five being post-test prompts. Notably, fewer "unclear" responses
correlated with higher user agreement on ethical classification. If less than
10% of users found a prompt unclear, over 60% agreed on its ethical status.
Conversely, if over 25% of users marked a prompt as unclear, disagreements on
its ethical status were more common, resulting in closer-to-parity splits.
This indicates a cutoff between 10% and 25% unclear responses, above which the
answer should be considered unclear, regardless of the dominant response. An
exact threshold will be determined as more users evaluate prompts.

User monitoring. Our third safeguard involves user monitoring, where we
silently track user responses to identify patterns like consistently marking
prompts as ethical or engaging in malicious behaviors, such as consistently
providing opposite responses to disrupt the dataset. We leverage Discord’s
API, which provides unique user IDs for each reaction, helping us prevent
duplicate responses from the same user and detect if a user stops evaluating
mid-process. For example, if a user answered "unclear" to the last 10 prompts,
we removed those responses to maintain data reliability, assuming the user
lost interest. It is important to note that users are not made aware of this
to not influence their responses.

The user IDs provided by Discord are non-invasive, offering no personal
information, only usernames. To further protect users, we will hash these
numbers before publicly sharing the dataset, preserving their anonymity.

Extended testing. In the initial 50 rounds of testing, 1108 out of 2844
prompts were evaluated. Around one-third of these prompts were deemed
"unclear" by a majority of votes. To enhance evaluation efficiency and
reliability, we temporarily set aside these unclear prompts and those without
any evaluations. This allowed us to focus on prompts worth evaluating,
resulting in 789 remaining prompts. The ones set aside could be reinstated at
a later date.

We assigned these 789 prompts to a new group of evaluators, which
significantly increased the number of responses per prompt. As a result, the
number of prompts evaluated only once decreased to less than half, as shown in
Table 2. We also maintained label balance, as demonstrated in Table 2. While a
few more responses were classified as "unclear," they now represented a clear
minority, reducing distractions. Additionally, all new users completed the
test, likely because there were fewer "unclear" prompts to confuse or
frustrate them.

Figure 1: Example of MAGMA’s responses (in red) to a question (in grey) on the
image above.

Figure 2: (a) Percentage of reactions to the pre-test and post-test prompts;
(b) Histogram of the evaluation results of the dataset by the RoBERTa-large
classifier.

Table 1: Table showing the amount of prompts in each category by the $65^{th}$
volunteer.

Classification | Amount of prompts | Percentage
---|---|---
Ethical | 369 | 46%
Unethical | 386 | 49%
Unclear | 34 | 5%

Table 2: Table showing the amount of reactions per prompt by the $65^{th}$
volunteer.

Amount of reactions | Amount of prompts | Percentage
---|---|---
1 | 322 | 41%
2 | 278 | 35%
$>=$3 | 189 | 24%

##  4 A Multimodal Ethics Classifier

Now that we have collected a dataset, we can run different classification
algorithms on it. This will allow us to see if current multimodal
classification methods provide adequate results or if more powerful models
need to be built. We will evaluate two different methods, the first being the
RoBERTa-large classifier published in the paper by Hendrycks et al. (2020) and
the second will be a self-build multilayer perceptron.

A RoBERTa-large classifier. To start, we utilized the RoBERTa-large common-
sense classifier introduced in Hendrycks et al. (2020) as it exhibited
promising performance in their research, achieving a 63.4% accuracy on the
hard test set. Additionally, this model has a publicly available checkpoint,
eliminating the need to train a new model from scratch.

Given that this classifier exclusively processes text, we concatenated the
question and answer from our database into a prompt. The classifier assigns
each prompt a score between 0 and 1, reflecting its confidence in the
statement’s morality. A score of 0 indicates an ethical response, while a
score of 1 denotes an unethical response.

Figure 2b reveals that the classifier tends to be uncertain, with most of the
789 prompts receiving scores around 0.5. Notably, 35% of the results exhibited
confidence levels between 0.45 and 0.5. Beyond this uncertainty, there’s a
discernible trend: the classifier more readily assigns a score of 0 than 1.
This outcome is unexpected. We were expecting a mixed normal distribution
shape, with spikes for ethical (0), uncertain (0.5), and unethical (1)
prompts, with small variations in-between.

Importantly, the prompts did not contain image information, explaining many
"unclear" scores between 0.4 and 0.6. These can be attributed to a lack of
context for the evaluation. Thus, we categorize results into three groups:
scores below 0.4 as ethical, scores above 0.6 as unethical, and scores between
0.4 and 0.6 as unclear. Using this, our classification accuracy, when compared
to the users’ most voted response, was 52%. This indicates that most question-
answer combinations do not provide sufficient context for ethical assessment.
This shows a well-built multimodal prompt, as the image is indeed required.

A multilayer perceptron classifier. To improve our results, we explored an
alternative approach inspired by Elourajini and Aïmeur (2022), involving a
multilayer perceptron. We based our architecture on the MAGMA algorithm
(Eichenberg et al. (2022)). Hence, we used a GPT2 tokenizer for the text (Ben
Wang and Aran Komatsuzaki (2021)) and a CLIP (Resnet large) embedder for the
image (Radford et al. (2021)). The results of these two operations was
concatenated together and used as input.

We then built a multilayer perceptron with 3 hidden layers and 3 possible
outputs: "ethical", "unethical" and "unclear". The layers that we used were
linear layers with a ReLU activation function, based on the results in the
paper by Elourajini and Aïmeur (2022).

After training this model on the previously collected data, our model achieved
55% accuracy on the test set. Interestingly enough, the model rarely predicts
"unclear". However, these preliminary result does not rule out the proposed
approach; rather, we must focus on improving the classifiers, including
training larger-scale deep networks.

##  5 Conclusions

Given the rapid growth of AI social impact, AI ethics becomes and increasingly
important research area. We have shown a method for developing a multimodal
dataset in a cost-effective manner. This is the first dataset of its kind and
can greatly help research in the field. To this end, the dataset will be made
available, and will receive regular updates as more users complete the
evaluation. This dataset contains a diverse set of examples on all fields of
ethics, including all forms of discrimination. The multiple safeguards within
the dataset helps to protect against a manipulation of the data by a malicious
user, and these should scale well with the amount of users. We tried showing
preliminary uses for the dataset we build, however our techniques require
further work in order to achieve a better classification rate.

## References

  * Amodei et al. [2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané.  Concrete Problems in AI Safety.  arXiv 1606.06565, June 2016.
  * Awad et al. [2020] Edmond Awad, Sohan Dsouza, Jean-François Bonnefon, Azim Shariff, and Iyad Rahwan.  Crowdsourcing Moral Machines.  Communications of the ACM, March 2020.
  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.  Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.  arXiv 2204.05862, April 2022.
  * Bellamy et al. [2018] Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang.  Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, 2018.
  * Ben Wang and Aran Komatsuzaki [2021] Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, 2021.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Commision [2021] European Commision.  The Artificial Intelligence Act.  https://eur-lex.europa.eu/resource.html?uri=cellar:e0649735-a372-11eb-9585-01aa75ed71a1.0001.02/DOC_1&format=PDF, 2021.
  * Constantin (CoEich) [2021] Sid (sdtblck) Constantin (CoEich), Mayukh (Mayukhdeb).  Magma’s git repository.  https://github.com/Aleph-Alpha/magma, 2021.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * Eichenberg et al. [2022] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.  "magma – multimodal augmentation of generative models through adapter-based finetuning".  arXiv 2112.05253, Decembre 2022.
  * Elourajini and Aïmeur [2022] Fahed Elourajini and Esma Aïmeur.  AWS-EP: A Multi-Task Prediction Approach for MBTI/Big5 Personality Tests.  Data Mining in Biomedical Informatics and Healthcare, November 2022.
  * Enseigner l’éthique [2021] Enseigner l’éthique.  Banque de cas éthiques.  https://www.enseignerlethique.be/content/banque-de-cas-%C3%A9thiques, 2021.
  * Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.  Aligning AI With Shared Human Values.  arXiv 2008.02275, August 2020.
  * Hendrycks et al. [2021] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt.  Unsolved Problems in ML Safety.  arXiv 2109.13916, September 2021.
  * House [2022] The White House.  Blueprint for an AI Bill of Rights.  https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf, October 2022.
  * Lan et al. [2019] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations, 2019.
  * Layoun et al. [2022] Jean-Charles Layoun, Alexis Roger, and Irina Rish.  Aligning MAGMA by Few-Shot Learning and Finetuning.  arXiv 2210.14161, Octobre 2022.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  arXiv 2103.00020, February 2021.
  * Rob Morris and Kareem Kouddous [2022] Rob Morris and Kareem Kouddous.  koko.  https://www.kokocares.org/, 2022.
  * Rombach et al. [2021] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models.  arXiv 2112.10752, December 2021.
  * Talagala [2022] Nisha Talagala.  AI Ethics: What It Is And Why It Matters.  Forbes, May 2022.
  * University of Montreal [2018] University of Montreal.  Montreal Declaration for a Responsible Development of Artificial Intelligence.  https://www.montrealdeclaration-responsibleai.com/, 2018.
  * Wieting et al. [2015] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu.  Towards universal paraphrastic sentence embeddings, 2015.