# Magnetic order in the two-dimensional metal-organic framework manganese
pyrazinecarboxylate with Mn-Mn dimers

S. Calder<EMAIL_ADDRESS>Neutron Scattering Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831, USA.  R. Baral  Neutron Scattering
Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, USA.  N.
Narayanan  University of Missouri Research Reactor (MURR), University of
Missouri, Columbia, MO 65211, USA  Department of Chemistry, University of
Missouri, Columbia, Missouri 65211, USA  L. D. Sanjeewa  University of
Missouri Research Reactor (MURR), University of Missouri, Columbia, MO 65211,
USA  Department of Chemistry, University of Missouri, Columbia, Missouri
65211, USA

###### Abstract

The magnetic properties of [Mn(pyrazinecarboxylate)2]n (Mn-pyrazine),
empirical formula C10H6MnN4O4, are investigated through susceptibility, heat
capacity and neutron scattering measurements. The structure of Mn-pyrazine
consists of Mn-Mn dimers linked on a distorted 2D hexagonal structure. The
weak out of plane interactions create a quasi-2D magnetic material within the
larger three dimensional metal organic framework (MOF) structure. We show that
this material undergoes a two stage magnetic transition, related to the low
dimensionality of the Mn lattice. First at 5 K, which is assigned to the
initial development of short range order in the 2D layers. This is followed by
long range order at 3.3 K. Applied field measurements reveal the potential to
induce magnetic transitions in moderately small fields of $\sim$2 T. Neutron
powder diffraction enabled the determination of a unique magnetic space group
$P2_{1}^{\prime}/c$ (#14.77) at 1.5 K. This magnetic structure consists of
antiferromagnetically coupled Mn-Mn dimers with spins principally along the
out of plane $a$-axis.

##  I Introduction

Investigations of two-dimensional (2D) layered magnetic materials has revealed
properties of interest for both fundamental and applied research. This is
exemplified by graphene and beyond-graphene materials where the bulk compound
has 2D layers weakly coupled through van der Waals bonding that can often be
exfoliated or otherwise isolated down to a few or single layers [1, 2, 3, 4,
5, 6]. In these inorganic compounds numerous non-trivial topological and
quantum behaviors have been observed and predicted due to the low
dimensionality enhancing this behavior. Investigations on inorganic quasi-2D
materials have uncovered topologically protected Skyrmions [7, 8], quantum
spin liquids with emergent Majorana fermions [9] and spontaneous topological
Hall effect [10]. Conversely, the tunability of coordination polymers, or
equivalently magnetic metal-organic frameworks (MOFs), offers a powerful but
less explored material space to achieve analogous physics when magnetic metal
ions are added to well isolated 2D layered coordination structures [11, 12,
13, 14, 15, 16]. An extremely large variety of structures are available
through often highly predictable organic chemistry routes. The ability to
control the in-plane 2D motif, the spacing of layers, and potential introduce
hybrid functionality on the organic linkers affords multiple intriguing
research avenues for magnetic coordination polymers in the realm of quantum
materials.

The material [Mn(pzc)2]n (pzc = pyrazinecarboxylate), henceforth referred to
as Mn-pyrazine, contains well isolated 2D layers of Mn2+ ions. Ref. 17 is the
only previous literature report on this material, with x-ray diffraction and
magnetic susceptibility measurements. The structure contains only one Mn site,
however the Mn ions within the layer have two distinct bonding environments
with spacings of $\sim$3.5 $\rm\AA$ and $\sim$5.6 $\rm\AA$. This results in
Mn-Mn dimers that are linked to form a distorted 2D hexagonal network. These
2D layers are well isolated by an interlayer Mn-Mn distance of $\sim$9.4
$\rm\AA$, with the bonding containing weak hydrogen bonds between ligands. Mn-
pyrazine is therefore expected to be a good realization of a quasi-2D material
in a bulk compound. The previous powder magnetic susceptibility measurements
were fit to a Curie-Weiss law down to 5 K and showed antiferromagnetic
interactions with the Mn2+ ion in the S=5/2 spin state. Inspecting the
magnetic susceptibility in Ref. 17 reveals an apparent low temperature
anomaly, however there was no discussion of any potential magnetic order
transition.

Here, we undertake magnetic susceptibility, heat capacity and neutron powder
diffraction measurements that reveal long range magnetic order in Mn-pyrazine.
The directional dependence of the field behavior is investigated through
single crystal magnetic susceptibility measurements that show a two stage
magnetic transition. In addition there is a potential in-field magnetic
transition, which indicates routes to tune the magnetic properties in moderate
fields. Neutron powder diffraction is utilized to investigate the crystalline
and magnetic structure through temperature dependent measurements. Despite the
presence of increased incoherent scattering from hydrogen in Mn-pyrazine,
empirical formula C10H6MnN4O4, good quality neutron diffraction data are
obtained. This highlights the strength of monochromatic high flux reactor
based neutron instruments coupled with the choice of large moment metal ions
when investigating magnetic MOFs. At the lowest temperature of 2 K long range
magnetic order is observed with several magnetic reflections identified.
Symmetry analysis of this magnetic structure shows that the Mn-Mn dimers form
antiferromagnetic pairs, with spins preferentially aligned in the out of plane
$a$-direction in the $P2_{1}^{\prime}/c$ $(\\#14.77)$ magnetic space group.

##  II Methods

###  II.1 Synthesis

Single crystals of Mn-pyrazine were grown using the low-temperature
hydrothermal method. First, a total of 0.37 grams of pyrazinecarboxylate acid
(C5H4N2O2) and anhydrous manganese(II) acetate (Mn(CH3COO)2) were mixed in a
stoichiometric ratio of 2:1 with 5 mL of water and 5 mL of ethanol in a small
beaker. The mixture was then mixed using a magnetic stirrer until everything
was fully dissolved. Then the final mixture was loaded into a Teflon-lined
stainless-steel autoclave, sealed well and heated at 140∘C for 12 hrs. After
cooling to room-temperature a dark yellow solution was recovered and left to
evaporate at room temperature. The Mn-pyrazine crystals were formed during the
solvent evaporation.

###  II.2 Magnetic property characterization

Temperature-dependent and field dependent magnetic measurements were performed
using a Quantum Design Magnetic Property Measurement System (MPMS). Magnetic
properties were determined using one single crystal with the weight of 3.8 mg.
The single crystal specimen was affixed to a quartz rod using GE varnish and
temperature dependent magnetization measurements were carried out along three
crystal directions. The temperature dependent data were collected in the range
of 2 - 350 K in an applied magnetic field up to 50 kOe. The anisotropic
isothermal magnetization measurements were performed between 2 - 100 K in
magnetic fields up to 60 kOe. The heat capacity ($\rm C_{p}$) of the sample
was measured using a Physical Property Measurement System (PPMS) between 2 -
200 K under magnetic field in the range 0-60 kOe.

###  II.3 Neutron powder diffraction

Neutron powder diffraction measurements on 5 grams of undeuterated Mn-pyrazine
were carried out on the HB-2A powder diffractometer at the High Flux Isotope
Reactor (HFIR), Oak Ridge National Laboratory (ORNL) [18, 19]. Hydrogen based
materials present an extra challenge for neutron scattering by both adding to
the neutron absorption and creating an increased background from incoherent
scattering. These effects can be easier to account for with constant
wavelength instruments due to the simpler data correction. Constant wavelength
measurements were performed at 2.41 $\rm\AA$ from the Ge(113) monochromator
reflection and 1.54 $\rm\AA$ from the Ge(115) reflection. The pre-mono, pre-
sample and pre-detector collimation was open-open-12’. A pyrolytic graphite
(PG) filter was placed before the sample to remove higher order reflections
for the 2.41 $\rm\AA$ wavelength. The sample was contained in a 6 mm diameter
vanadium can and cooled in a liquid 4He cryostat in the temperature range 1.5
K - 300 K. The diffraction pattern was collected by scanning a 120∘ bank of 44
3He detectors in 0.05∘ steps to give 2$\theta$ coverage from 5∘ to 130∘.
Counting times were 8 hours for the 2.41 $\rm\AA$ measurements and 2 hours for
the 1.54 $\rm\AA$ wavelength. Rietveld refinements were performed with
Fullprof [20]. Symmetry allowed magnetic structures were considered using both
representational analysis with SARAh [21] and magnetic space groups with the
Bilbao Crystallographic Server [22]. Plots of the crystal and magnetic
structure were prepared using VESTA [23].

Figure 1:  (a) Rietveld refinement of neutron powder diffraction data for Mn-
pyrazine collected on the HB-2A instrument with a wavelength of 1.54 $\rm\AA$
at 20 K. (b) Crystal structure of Mn-pyrazine. The box represents the unit
cell in the $P2_{1}/c$ space group. (c) Nearest and next nearest neighbor Mn
ions are bonded through distinct pathways of Mn-O-Mn and Mn-O-C-O-Mn. Figure
2:  (a) View of the 2D layers in Mn-pyrazine. (b) The nearest neighbor dimers
(thick red line) and next nearest neighbor (thin blue line) interactions
within the 2D layers are shown between the Mn ions (purple sphere). The Mn
ions form a distorted 2D honeycomb structure. The crystallographic unit cell
is indicated by the grey box. Table 1: Refined crystal structure parameters
for Mn-pyrazine, empirical formula C10H6MnN4O4, at 20 K for space group
$P2_{1}/c$ with lattice constants a=10.2078(4) $\rm\AA$, b=10.8444(4)
$\rm\AA$, c=10.1095(4) $\rm\AA$, $\alpha$=90∘, $\beta$=108.429(3)∘,
$\gamma$=90∘.
$\chi^{2}=1.72$, $\rm R_{wp}$=8.80. Atom | $x$ | $y$ | $z$ | site
---|---|---|---|---
Mn | 0.498(3) | 0.128(2) | 1.101(3) | $4e$
C | 0.835(2) | 0.119(1) | 1.146(1) | $4e$
C | 0.968(2) | 0.147(1) | 1.176(2) | $4e$
H | 1.038(3) | 0.104(2) | 1.130(2) | $4e$
C | 0.937(1) | 0.289(1) | 1.326(1) | $4e$
H | 0.974(3) | 0.368(2) | 1.394(2) | $4e$
C | 0.798(1) | 0.263(1) | 1.292(1) | $4e$
H | 0.727(2) | 0.305(2) | 1.351(2) | $4e$
C | 0.373(2) | 0.374(1) | 0.966(2) | $4e$
C | 0.271(1) | 0.468(1) | 0.918(1) | $4e$
H | 0.287(3) | 0.523(2) | 0.840(2) | $4e$
C | 0.159(1) | 0.417(1) | 1.079(2) | $4e$
H | 0.103(2) | 0.436(3) | 1.121(2) | $4e$
C | 0.258(2) | 0.322(1) | 1.125(1) | $4e$
H | 0.250(2) | 0.272(2) | 1.194(2) | $4e$
C | 0.756(1) | 0.018(1) | 1.035(1) | $4e$
C | 0.479(1) | 0.337(1) | 0.891(1) | $4e$
N | 0.740(1) | 0.176(1) | 1.205(1) | $4e$
N | 1.026(1) | 0.232(1) | 1.273(1) | $4e$
N | 0.360(1) | 0.297(1) | 1.066(1) | $4e$
N | 0.182(1) | 0.497(1) | 0.976(1) | $4e$
O | 0.829(2) | -0.034(2) | 0.973(2) | $4e$
O | 0.627(2) | -0.004(2) | 1.039(1) | $4e$
O | 0.551(2) | 0.247(1) | 0.939(1) | $4e$
O | 0.476(2) | 0.416(1) | 0.784(2) | $4e$

##  III Results and Discussion

###  III.1 Crystal structure of Mn-pyrazine

We begin by considering the crystal structure of Mn-pyrazine, which was
previously reported as being in the $P2_{1}/c$ space group [17]. To confirm
this we carried out neutron powder diffraction measurements in the
paramagnetic regime of 20 K. The shorter wavelength of 1.54 $\rm\AA$ was used
to give the widest Q coverage to increase the number of measured reflections.
The data has the expected elevated background and Q dependence from the
incoherent hydrogen scattering. There are, however, well resolved and strong
nuclear Bragg peaks. This data was refined with the reported $P2_{1}/c$ space
group, with the background readily accounted for by a simple 6 coefficient
polynomial function used for most measurements on this neutron powder
diffractometer. The data and refinement show good agreement, see Fig. 1(a).
The refined lattice constants and atomic positions are given in Table 1. Due
to the large number of parameters the thermal parameters were fixed during the
analysis.

The structural unit cell is shown in Fig. 1(b). Considering the local Mn
environment in Fig. 1(c) indicates two distinct bonding pathways. Nearest
neighbor bonds are Mn-O-Mn, which would suggest standard superexchange
magnetic pathways. Whereas the next nearest neighbor bonds are Mn-O-C-O-Mn,
requiring extended superexchange magnetic interactions. The nearest neighbor
distance is 3.46(5) $\rm\AA$ and next nearest neighbor is 5.71(3) $\rm\AA$.
The Mn-Mn distance between the layers is 10.2078(5) $\rm\AA$ and is mediated
by a complex exchange pathway which includes weak hydrogen bonds. This creates
the 2D layered structure of interest. Considering multiple unit cells, as
shown in Fig. 2(a), highlights the well isolated 2D network of Mn-Mn ions. The
2D network in the $bc$-plane is shown in Fig. 2(b). The nearest neighbor Mn
bonds can be viewed as Mn-Mn dimers which interact with the next nearest
neighbor Mn ions to form a distorted 2D hexagonal layer.

###  III.2 Magnetic susceptibility and heat capacity results

Figure 3:  (a) Magnetic susceptibility measurements from 2 K to 300 K. The
solid black line is a fit using the Curie-Weiss law. (b) Directional dependent
measurements showing a broad anomaly centered around 5 K. Measurements were in
a 10 kOe field for all three directions. (Inset) out of plane magnetic field
dependence. (c) Isothermal magnetization measurements for a field applied out
of the plane. (Inset) Directional dependent measurements at 2 K. (d)
Derivative of the susceptibility reveals two anomalies at 5 K and 3.2 K in a
field of 10 kOe out of the plane Figure 4:  (a) Zero field heat capacity
measurements on Mn-pyrazine from 2 to 200 K. (b) Low temperature and field
dependent heat capacity measurements.

The initial report of Mn-pyrazine in Ref. 17 measured magnetic susceptibility
of a powder sample. Despite the apparent low temperature anomaly recorded, no
detailed discussion of potential magnetic ordering was reported. In the
synthesis reported here, small Mn-pyrazine crystals grew as rectangular rods.
Therefore, we performed our magnetic measurements on single crystals in three
orientations as displayed in Fig. 3. The magnetic susceptibility produces a
broad peak below 10 K with a maximum around 5 K. After that, it continuously
decreases down to the lowest temperature measured of 2 K, indicating an
antiferromagnetic transition. A broad transition is often a signature of
short-range ordering associate with the layered nature of the structure. The
inverse susceptibility was fitted using the Curie-Weiss law in the range of
temperature 100 $<$ T $<$ 300 K, see Fig. 3(a). This gives $\theta_{\rm CW}$ =
-7 K, indicating the antiferromagnetic nature of this compound. The effective
magnetic moment from the fit was $\mu_{\rm eff}$ = 5.7 $\mu_{B}$/Mn, which is
comparable to the expected moment of a Mn2+ ion in the high-spin $d^{5}$ state
of 5.9$\mu_{B}$.

The magnetic susceptibility measured in different crystal orientations are
displayed in Fig. 3(b)-(c). All field directions of in-plane and out-of-plane
show a broad peak around the same temperature of 5 K at low fields $\leq 10$
kOe. Increasing the field shifts the peak to lower temperatures. In general,
the behavior under applied field is non-trivial with potentially a spin-flop
transition which is further confirmed from our isothermal magnetization
measurements in Fig. 3(c). This shows an anomaly at 2 K and 15 kOe. The inset
of Fig. 3(c) showing directional dependent measurements indicates this anomaly
occurs for fields applied out of the plane only. Considering the derivative of
susceptibility (d$M$/dT) indicates a two-stage transition at T2= 5 K and T1=3
K.

To gain further insights into the low temperature behavior in Mn-pyrazine we
performed heat capacity measurements. Figure 4 shows the results. The peak in
the heat capacity for 0 T is observed at 3.3 K. This is lower then the broad
peak from magnetic susceptibility which occurs around 5 K, but consistent with
the T2 transition in the d$M$/dT analysis. Applying a field leads to a
lowering of the transition temperature in the heat capacity measurements and
the observation of a further transition in the form of a shoulder in the heat
capacity anomaly. This can be seen in the inset of Fig. 4 for a field of 3 T
where a shoulder is measured at 3.2 K, followed by a further anomaly at 2.8 K.

Collectively the susceptibility and heat capacity results reveal multiple
transitions that can be rationalized by considering the low dimensional nature
of Mn-Pyrazine. We postulate that short range order occurs around 5 K in the
2D layers. This gives the broad peak in the susceptibility, rather than a
sharp transition. Then at 3.3 K the transition to three dimensional long range
order occurs. This is observed in the heat capacity and also the magnetic
susceptibility through d$M$/dT as a two stage transition.

###  III.3 Magnetic structure of Mn-pyrazine

Figure 5:  (a) Neutron powder diffraction data collected at 1.5 K and 20 K. (Inset) Intensity at Q=0.65$\rm\AA$ as a function of temperature through the magnetic transition. (b) Difference of intensity at 1.5 K and 20 K in the powder diffraction data. Figure 6:  (a) Refinement of neutron powder diffraction data collected at 1.5 K using the $P2_{1}^{\prime}/c$ with moments along all ($\bf{m}_{a}$,$\bf{m}_{b}$,$\bf{m}_{c}$) directions. (b) Refinement of neutron powder diffraction data collected at 1.5 K using the $P2_{1}^{\prime}/c$ with moments only along $\bf{m}_{a}$. Magnetic structures for (c) ($\bf{m}_{a}$,$\bf{m}_{b}$,$\bf{m}_{c}$) and (d) ($\bf{m}_{a}$,0, 0) models. The moments are shown as red arrows on the Mn ions, the nearest neighbor dimer bonds are blue and the next nearest neighbor bond are the dashed lines. The grey box represents the magnetic unit cell. (e) View of the 2D layers along the $a$-axis. The red/black circles correspond to up/down spin directions on the Mn ion. IR | BV | Point Group | Magnetic Space Group
---|---|---|---
$\Gamma_{1}$ | $\psi_{1}$ | $2/m$ | $P2_{1}/c$ | $(\\#14.75)$
| $\psi_{2}$ | $2/m$ | $P2_{1}/c$ | $(\\#14.75)$
| $\psi_{3}$ | $2/m$ | $P2_{1}/c$ | $(\\#14.75)$
$\Gamma_{2}$ | $\psi_{4}$ | $2/m$ | $P2_{1}/c^{\prime}$ | $(\\#14.78)$
| $\psi_{5}$ | $2/m$ | $P2_{1}/c^{\prime}$ | $(\\#14.78)$
| $\psi_{6}$ | $2/m$ | $P2_{1}/c^{\prime}$ | $(\\#14.78)$
$\Gamma_{3}$ | $\psi_{7}$ | $2/m$ | $P2_{1}^{\prime}/c^{\prime}$ | $(\\#14.79)$
| $\psi_{8}$ | $2/m$ | $P2_{1}^{\prime}/c^{\prime}$ | $(\\#14.79)$
| $\psi_{9}$ | $2/m$ | $P2_{1}^{\prime}/c^{\prime}$ | $(\\#14.79)$
$\Gamma_{4}$ | $\psi_{10}$ | $2/m$ | $P2_{1}^{\prime}/c$ | $(\\#14.77)$
| $\psi_{11}$ | $2/m$ | $P2_{1}^{\prime}/c$ | $(\\#14.77)$
| $\psi_{12}$ | $2/m$ | $P2_{1}^{\prime}/c$ | $(\\#14.77)$
Table 2: The point symmetry and magnetic space group for the space group $P2_{1}/c$ with ${\bf k}=(0,0,0)$. The decomposition of the magnetic representation for the Mn site $(0.510,0.134,0.598)$ is $\Gamma_{Mag}=3\Gamma_{1}^{1}+3\Gamma_{2}^{1}+3\Gamma_{3}^{1}+3\Gamma_{4}^{1}$. IR | BV | Atom | BV components
---|---|---|---
|  |  | $m_{\|a}$ | $m_{\|b}$ | $m_{\|c}$
$\Gamma_{4}$ | $\psi_{10}$ | 1 | 1 | 0 | 0
|  | 2 | 1 | 0 | 0
|  | 3 | -1 | 0 | 0
|  | 4 | -1 | 0 | 0
| $\psi_{11}$ | 1 | 0 | 1 | 0
|  | 2 | 0 | -1 | 0
|  | 3 | 0 | -1 | 0
|  | 4 | 0 | 1 | 0
| $\psi_{12}$ | 1 | 0 | 0 | 1
|  | 2 | 0 | 0 | 1
|  | 3 | 0 | 0 | -1
|  | 4 | 0 | 0 | -1
Table 3: Basis vectors for the space group $P2_{1}/c$ with ${\bf k}=(0,0,0)$.
The decomposition of the magnetic representation for the Mn site
$(0.51009,~{}0.13426,~{}0.598)$ is
$\Gamma_{Mag}=3\Gamma_{1}^{1}+3\Gamma_{2}^{1}+3\Gamma_{3}^{1}+3\Gamma_{4}^{1}$.
The atoms of the nonprimitive basis are defined according to 1:
$(0.510,0.134,0.598)$, 2: $(0.490,0.634,0.902)$, 3: $(0.490,0.866,.402)$, 4:
$(0.510,0.366,0.098)$.

We now turn to neutron powder diffraction measurements to investigate the
microscopic magnetic spin structure. Diffraction patterns were collected at
1.5 K and 20 K using the 2.41 $\rm\AA$ wavelength, see Fig. 5(a). The high Q
scattering is unchanged, indicating no structural symmetry change. The low Q
behavior below 2 $\rm\AA^{-1}$, however, reveals additional intensity in the
form of Bragg peaks. These can be assigned to magnetic ordering, with the
width of the new magnetic peaks the same as the nuclear peaks, indicating long
range magnetic order at 1.5 K. The change in scattering is emphasized in the
temperature difference plot in Fig. 5(b).

The intensity of the peak at Q=0.65 $\rm\AA^{-1}$, which has no observable
nuclear contribution at 20 K, was measured as a function of temperature to
follow the onset of magnetic ordering. Increased scattering is observed at 5
K, which corresponds to the broad peak in magnetic susceptibility. This
increases, until it becomes saturated below 3 K. This measurement was
exclusively of the intensity at Q=0.65 $\rm\AA^{-1}$. It did not allow for a
measurement the peak width, therefore it is not possible to distinguish
between short or long range order scattering. Since any diffuse scattering
intensity will be small, it would be of interest for future studies to measure
deuterated powder or larger single crystals to further investigate the
potential short range order in Mn-pyrazine through this transition.

At 1.5 K Mn-pyrazine is in the long ranged magnetically ordered state. The
measured magnetic Bragg peaks can all be indexed with a ${\bf k}$=(0,0,0)
propagation vector. Starting from the paramagnetic space group of $P2_{1}/c$
and using the determined propagation vector gives equivalently four
irreducible representations (IRs) in a representational analysis approach or
four maximal magnetic space groups. The magnetic space groups are $P2_{1}/c$
(#14.75), $P2_{1}^{\prime}/c$ (#14.77), $P2_{1}/c^{\prime}$ (#14.78) and
$P2_{1}^{\prime}/c^{\prime}$ (#14.79). The corresponding IRs $\Gamma_{1}$,
$\Gamma_{2}$, $\Gamma_{3}$ and $\Gamma_{4}$, in Kovalevs representation, are
shown in Table. 2.

For all candidate models there is symmetry allowed spin components along all
crystallographic directions ($\bf{m}_{a}$,$\bf{m}_{b}$,$\bf{m}_{c}$). The data
was refined against all 4 candidate magnetic models. Only magnetic space group
$P2_{1}^{\prime}/c$ $(\\#14.77)$ ($\Gamma_{4}$) was able to reproduce the
intensity of all observed magnetic reflections. Allowing the moments to refine
freely gave ($\bf{m}_{a}$,$\bf{m}_{b}$,$\bf{m}_{c}$)= (4.363(101), 1.261(562),
1.532(237) ) and total moment 4.3(2)$\mu_{B}$/Mn2+. Which is reduced but
nevertheless close to the full spin for a S=5/2 ion of 5$\mu_{B}$. This
magnetic structure is shown in Fig.6(c). Confining the spins to only have a
component along the $a$-axis gives an equivalently good refinement, as can be
seen in Fig.6(b). The corresponding moment is slightly increased and closer to
the full S=5/2 values with ($m_{a}$,$m_{b}$,$m_{c}$)= (4.47(9),0,0) and total
moment 4.47(9) $\mu_{B}$/Mn2+. This magnetic structure is shown in Fig.6(d).
Allowing the spins to only be constrained to either the $b$ or $c$ axis could
not fully account for the data. These results confirm the S=5/2 nature of the
Mn ion, however the reduction in moment may be a consequence of the extended
nature of the Mn-Mn exchange pathways leading to a degree of moment
delocalization onto the surrounding ligands.

The spin behavior in the 2D layer can be visualized in Fig.6(e). The black/red
correspond to up/down Mn moments. The magnetic structure of Mn-pyrazine has
antiferromagnetic dimers of Mn-Mn ions. Each next nearest neighbor Mn-Mn
interaction in the layer is also antiferromagnetic. Despite the large Mn-Mn
distance of $>$ 10 $\rm\AA$ three dimensional long range order occurs, with
the nearest neighbor Mn-Mn interlayer correlation being ferromagnetic. The
large Mn moment of S=5/2 is likely a driving factor in realizing long range
order.

To induce further interesting behavior in Mn-pyrazine and related materials it
will be of interest to reduce the moment size down to S=1/2 to enhance quantum
phenomena. This will be of particular interest with the dimer and 2D hexagonal
layers that are hosts to exotic physics. As one example the Mn-pyrazine
structure can be considered a distorted Shastry-Sutherland lattice [24, 25].
Removing or controlling this distortion may provide routes to investigate this
physics in MOFs and allow for a wider phase space of materials than the
currently limited candidates.

##  IV Conclusions

Mn-pyrazine (C10H6MnN4O4) has been investigated with magnetic susceptibility,
heat capacity and neutron powder diffraction. The magnetic susceptibility and
heat capacity collectively indicate development of short range order at 5 K
that proceeds the long range magnetic phase transition at 3.2 K in zero field.
The applied field measurements show anisotropic behavior with a field driven
anomaly above 2 T. Moreover heat capacity measurements in fields above 2 T
reveal two observable anomalies with a small temperature window of $\sim$0.5
K. Neutron powder diffraction was able to determine the magnetic structure in
this undeuterated material. Following a symmetry analysis only a single
magnetic space group, $P2_{1}^{\prime}/c$ (#14.77), was found to be consistent
with the data. The analysis revealed the moments primarily aligned along the
$a$-axis, which is out of the 2D layers. These moments form antiferromagnetic
dimers that are linked within a wider distorted hexagonal network. In general
the results show that coordination polymers, or equivalently magnetic metal-
organic frameworks (MOFs), with both organic and inorganic building blocks
offer unique material avenues to explore tailored structural motifs due to the
versatility and predictability of organic chemistry.

##  V acknowledgments

This research used resources at the High Flux Isotope Reactor, a DOE Office of
Science User Facility operated by the Oak Ridge National Laboratory. This
research used resources at the Missouri University Research Reactor (MURR).
This work was supported in part by a University of Missouri Research Council
Grant (Grant Number: URC-22-021). This manuscript has been authored by UT-
Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of
Energy. The United States Government retains and the publisher, by accepting
the article for publication, acknowledges that the United States Government
retains a non-exclusive, paidup, irrevocable, world-wide license to publish or
reproduce the published form of this manuscript, or allow others to do so, for
United States Government purposes. The Department of Energy will provide
public access to these results of federally sponsored research in accordance
with the DOE Public Access Plan(http://energy.gov/downloads/doepublic-access-
plan).

## References

  * Novoselov  _et al._ [2005] K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, M. I. Katsnelson, I. V. Grigorieva, S. V. Dubonos, and A. A. Firsov, Nature 438, 197 (2005).
  * Novoselov  _et al._ [2016] K. S. Novoselov, A. Mishchenko, A. Carvalho, and A. H. Castro Neto, Science 353 (2016).
  * Zeng  _et al._ [2018] M. Zeng, Y. Xiao, J. Liu, K. Yang, and L. Fu, Chemical Reviews 118, 6236 (2018).
  * Burch  _et al._ [2018] K. S. Burch, D. Mandrus, and J.-G. Park, Nature 563, 47 (2018).
  * Huang  _et al._ [2020] B. Huang, M. A. McGuire, A. F. May, D. Xiao, P. Jarillo-Herrero, and X. Xu, Nature Materials 19, 1276–1289 (2020).
  * Baltz  _et al._ [2018] V. Baltz, A. Manchon, M. Tsoi, T. Moriyama, T. Ono, and Y. Tserkovnyak, Rev. Mod. Phys. 90, 015005 (2018).
  * Ding  _et al._ [2020] B. Ding, Z. Li, G. Xu, H. Li, Z. Hou, E. Liu, X. Xi, F. Xu, Y. Yao, and W. Wang, Nano Letters 20, 868 (2020).
  * Park  _et al._ [2021] T.-E. Park, L. Peng, J. Liang, A. Hallal, F. S. Yasin, X. Zhang, K. M. Song, S. J. Kim, K. Kim, M. Weigand, G. Schütz, S. Finizio, J. Raabe, K. Garcia, J. Xia, Y. Zhou, M. Ezawa, X. Liu, J. Chang, H. C. Koo, Y. D. Kim, M. Chshiev, A. Fert, H. Yang, X. Yu, and S. Woo, Phys. Rev. B 103, 104410 (2021).
  * Banerjee  _et al._ [2017] A. Banerjee, J. Yan, J. Knolle, C. A. Bridges, M. B. Stone, M. D. Lumsden, D. G. Mandrus, D. A. Tennant, R. Moessner, and S. E. Nagler, Science 356, 1055 (2017).
  * Takagi  _et al._ [2023] H. Takagi, R. Takagi, S. Minami, T. Nomoto, K. Ohishi, M. t. Suzuki, Y. Yanagi, M. Hirayama, N. D. Khanh, K. Karube, H. Saito, D. Hashizume, R. Kiyanagi, Y. Tokura, R. Arita, T. Nakajima, and S. Seki, NATURE PHYSICS 10.1038/s41567-023-02017-3 (2023).
  * Zheng  _et al._ [2014] Y.-Z. Zheng, Z. Zheng, and X.-M. Chen, Coordination Chemistry Reviews 258-259, 1 (2014).
  * Goddard  _et al._ [2012] P. A. Goddard, J. L. Manson, J. Singleton, I. Franke, T. Lancaster, A. J. Steele, S. J. Blundell, C. Baines, F. L. Pratt, R. D. McDonald, O. E. Ayala-Valenzuela, J. F. Corbey, H. I. Southerland, P. Sengupta, and J. A. Schlueter, Phys. Rev. Lett. 108, 077208 (2012).
  * Mínguez Espallargas and Coronado [2018] G. Mínguez Espallargas and E. Coronado, Chem. Soc. Rev. 47, 533 (2018).
  * Stock and Biswas [2012] N. Stock and S. Biswas, Chemical Reviews 112, 933 (2012).
  * Saines and Bristowe [2018] P. J. Saines and N. C. Bristowe, Dalton Trans. 47, 13257 (2018).
  * Kurmoo [2009] M. Kurmoo, Chem. Soc. Rev. 38, 1353 (2009).
  * Cai  _et al._ [2002] C. Cai, Y. Tian, X. Ren, Y. Li, X. You, and S. Li, Transition Metal Chemistry 27, 924 (2002).
  * Garlea  _et al._ [2010] V. O. Garlea, B. C. Chakoumakos, S. A. Moore, G. B. Taylor, T. Chae, R. G. Maples, R. A. Riedel, G. W. Lynn, and D. L. Selby, Applied Physics A 99, 531 (2010).
  * Calder  _et al._ [2018] S. Calder, K. An, R. Boehler, C. R. Dela Cruz, M. D. Frontzek, M. Guthrie, B. Haberl, A. Huq, S. A. J. Kimber, J. Liu, J. J. Molaison, J. Neuefeind, K. Page, A. M. dos Santos, K. M. Taddei, C. Tulk, and M. G. Tucker, Review of Scientific Instruments 89, 092701 (2018).
  * Rodríguez-Carvajal [1993] J. Rodríguez-Carvajal, Physica B: Condensed Matter 192, 55 (1993).
  * Wills [2000] A. Wills, Physica B 276, 680 (2000).
  * Perez-Mato  _et al._ [2015] J. Perez-Mato, S. Gallego, E. Tasci, L. Elcoro, G. de la Flor, and M. Aroyo, Annual Review of Materials Research 45, 217 (2015).
  * Momma and Izumi [2011] K. Momma and F. Izumi, J. Appl. Crystallogr. 44, 1272 (2011).
  * Miyahara and Ueda [1999] S. Miyahara and K. Ueda, Phys. Rev. Lett. 82, 3701 (1999).
  * Haravifard  _et al._ [2014] S. Haravifard, A. Banerjee, J. van Wezel, D. M. Silevitch, A. M. dos Santos, J. C. Lang, E. Kermarrec, G. Srajer, B. D. Gaulin, J. J. Molaison, H. A. Dabkowska, and T. F. Rosenbaum, Proceedings of the National Academy of Sciences 111, 14372 (2014).


††thanks: Equal contribution††thanks: Equal contribution††thanks: Equal
contribution††thanks: Equal contribution

# Binary classifiers for noisy datasets: a comparative study of existing
quantum machine learning frameworks and some new approaches

N. Schetakis  School of Electronic and Computer Engineering, Technical
University of Crete, Chania, Greece 73100  ALMA Sistemi Srl, Guidonia (Rome),
00012, Italy<EMAIL_ADDRESS>D. Aghamalyan  School of Information
Systems, Singapore Management University, 81 Victoria Street, Singapore 188065
Centre for Quantum Technologies, National University of Singapore, Singapore
117543  M. Boguslavsky  Tradeteq Ltd, London, UK  P. Griffin  School of
Information Systems, Singapore Management University, 81 Victoria Street,
Singapore 188065

###### Abstract

One of the most promising areas of research to obtain practical advantage is
Quantum Machine Learning which was born as a result of cross-fertilisation of
ideas between Quantum Computing and Classical Machine Learning. In this paper,
we apply Quantum Machine Learning (QML) frameworks to improve binary
classification models for noisy datasets which are prevalent in financial
datasets. The metric we use for assessing the performance of our quantum
classifiers is the area under the receiver operating characteristic curve
(ROC/AUC). By combining such approaches as hybrid-neural networks, parametric
circuits, and data re-uploading we create QML inspired architectures and
utilise them for the classification of non-convex 2 and 3-dimensional figures.
An extensive benchmarking of our new FULL HYBRID classifiers against existing
quantum and classical classifier models, reveals that our novel models exhibit
better learning characteristics to asymmetrical Gaussian noise in the dataset
compared to known quantum classifiers and performs equally well for existing
classical classifiers, with a slight improvement over classical results in the
region of the high noise.

Introduction.— Noisy Intermediate-Scale Quantum (NISQ) Preskill (2018); Bharti
_et al._ (2021); Deutsch (2020) devices hold a promise to deliver a practical
quantum advantage by harnessing the complexity of quantum systems. Despite
being several years away from having fault-tolerant quantum computingPreskill
(1998a); Gottesman (1998); Shor (1996), researchers have been hopeful to
achieve this task. Perhaps one of the most exciting breakthroughs in this
direction was a demonstration of ”quantum supremacy” by Google researchers
Arute  _et al._ (2019), using their programmable superconducting Sycamore chip
with 53 qubits, in which single-qubit gate fidelities of 99.85% and two-qubit
gate fidelities of 99.64% were obtained on average. Here the task of sampling
the output of a pseudo-random quantum circuit was successfully achieved.
Quantum Supremacy would imply that a universal quantum computer has the
ability to perform certain tasks exponentially faster than a classical
computerHarrow and Montanaro (2017). However, it has been argued later that
Google’s achievement amounted to a demonstration of a quantum advantage but
not a practical advantage, in other words, the performed task was not useful
for any real-life applications. Another quantum advantage breakthrough
experiment has been implemented Zhong  _et al._ (2020) utilising a Jiuzhang
photonic quantum computer and performing Gaussian boson sampling (GBS) with 50
indistinguishable single-mode squeezed states. Here, quantum advantage has
been elucidated in the sampling time complexity of a Torontonian matrix, which
has exponential scaling with output photon clicks. However, this experiment
demontsrates quantum advantage but fails to demonstrate quantum supremacy as
this photonic quantum computer is not programmable.

One of the most promising areas of research to obtain practical advantage is
Quantum Machine LearningBiamonte  _et al._ (2017); Wittek (2014); Schuld
(2018) which was born as a result of cross-fertilisation of ideas between
Quantum Computing Nielsen and Chuang (2002); Preskill (1998b) and Classical
Machine Learning Goodfellow  _et al._ (2016); Jordan and Mitchell (2015). QML
in its spirit is similar to classical machine learning but with the main
difference being that instead of classical neurons in the layers of a deep
neural network, now we have qubits and quantum gates acting on qubits combined
with quantum measurements playing the role of the activation function. The
elegant field of QML has been providing a new platform for devising algorithms
that exhibit quantum speedups. For instance, it has been demonstrated that
such basic linear algebra subroutines as solving certain types of linear
equations (the quantum version is known in the community as HHL), finding
eigenvectors and eigenvalues, principal component analysis (PCA) exhibit
exponential speedups compared to their classical counterpartsHarrow  _et al._
(2009); Huang  _et al._ (2019); Rebentrost  _et al._ (2018); Lloyd  _et al._
(2014); Wiebe  _et al._ (2012). Since we are dealing with a quantum system,
one can utilise such quantum resources as coherence, entanglement, negativity,
contextuality to leverage towards achieving practical advantage. However, it
is still not completely understood what the role of different types of
resources is in harnessing practical advantage from available 50-100 qubit
noisy devices Deutsch (2020). The three main building blocks of any QML
algorithm are data encoding, unitary evolution of the system followed by the
state readout performed through the measurement Schuld (2018). Uploading
classical data in the quantum computer is not a trivial task and can account
for most of the complexity of the algorithm, determining what kind of speed-
ups are feasible. This procedure is called quantum embedding which can be
achieved, for instance, with help of ”quantum feature maps”Schuld and Killoran
(2019); Pérez-Salinas  _et al._ (2020, 2021); Lloyd  _et al._ (2020a); Mitarai
_et al._ (2018) which take classical data and map it to the high-dimensional
Hilbert space, where one hopes to achieve higher separation between the data
classes compared to the original coordinate system. Moreover, one can train
the quantum embedding to achieve maximal separation between the data clusters
in the Hilbert space (this approach has been coined as ”quantum metric
learning”)Lloyd  _et al._ (2020a); Mitarai  _et al._ (2018), paving the way
towards constructing faithful quantum classifiers.

Binary classification is a ubiquitous task in machine learning. Perhaps the
most prominent example is the cat recognition algorithm, which gives a flavour
of the power brought by utilising such basic tools as logistic regression
combined with deep neural network architectures Goodfellow  _et al._ (2016).
Quantum classifiers hold a promise to bring feasible speedups compared to
their classical counterparts. Several theoretical proposals combined with
actual experimental runs on commercially available backends have been put
forward for realising faithful quantum classifiersSchuld  _et al._ (2020);
Schuld and Killoran (2019); Farhi and Neven (2018); Pérez-Salinas  _et al._
(2020); Tacchino  _et al._ (2019a); Cappelletti  _et al._ (2020); Wiebe  _et
al._ (2016); Liao  _et al._ (2018); Schuld  _et al._ (2017); Tiwari and
Melucci (2019); Blank  _et al._ (2020); Park  _et al._ (2020). For instance,
approaches in Refs. Blank  _et al._ (2020); Park  _et al._ (2020) are inspired
by kernel methods used in classical machine learning. Refs. Schuld  _et al._
(2020); Schuld and Killoran (2019); Farhi and Neven (2018) are combining
certain types of quantum embeddings to achieve quantum hybrid neural networks,
which are promising candidates for building a faithful classifier. Ref.
Tacchino  _et al._ (2019b) suggests using hypergraph-statesRossi  _et al._
(2013), where the assumption is that such states can lower the circuit depth
of the classifier. Refs.Wiebe  _et al._ (2016); Liao  _et al._ (2018) are
based on quantum Grover’s search algorithm.

In this manuscript, we take a rather pragmatic approach and try to benefit
from a plethora of available QML software packagesBergholm  _et al._ (2018);
Killoran  _et al._ (2019); Broughton  _et al._ (2020); Efthymiou  _et al._
(2020); Kottmann  _et al._ (2021), which grant access to run the quantum
circuit in the quantum simulator or an actual hardware (such as IBM Quantum
Experience, Amazon Braket, Rigetti Computing, Strawberry Fields). By utilising
these tools we provide new software that is particularly well suited for
targeting classification problems in the unbalanced and noisy datasets which
are prevalent in the financial industryOrus  _et al._ (2019).

In this paper at first we briefly outline and review three different necessary
building block QML architectures for our software package : hybrid-neural
networks Schuld  _et al._ (2020); Schuld and Killoran (2019); Farhi and Neven
(2018), parametric quantum circuits Benedetti  _et al._ (2019); Cerezo  _et
al._ (2020); Bharti  _et al._ (2021); Funcke  _et al._ (2021) and data-
reuploading Pérez-Salinas  _et al._ (2020, 2021).

The metric we use for assessing the performance of our quantum classifiers is
the area under the receiver operating characteristic curve (ROC/AUC). ROC is a
probability curve and AUC represents the degree of separability. In general a
good model has AUC close to 1. We test our FULL HYBRID models and benchmark
them against existing QML classifiers and also to the best known classical
machine learning counterparts by running simulations on quantum simulators for
three different 2-dimensional non-convex surfaces. It is believed that non
convex boundaries represent more difficult classification problems as linear
regression is bound to fail in this tasks. Then by introducing asymmetrical
Gaussian noise we study the resilience of our different approaches to the
noise. This kind of study sheds light on learning properties for the amount of
noise in the dataset. We also perform systematic hyperparameter tuning by
studying how ROC/AUC changes with the number of repeating units in the data-
re-uploading approach, number of qubits, batch size, number of epochs and
number of strongly entangling units. We remark, that our binary classifiers
can be extended to multi-class classification problems using a one-versus-all
approach.

The manuscript is organised as follows. In section I we explain what kind of
classification problems for 2 and 3-dimensional non-convex surfaces we tackle
in the current study. In Section II, we briefly review the three building
blocks in the QML frameworks which we utilise in the next section III for our
novel classification circuit which combines the best features of the building
blocks. In Section IV we benchmark several known QML approaches(including our
QML classifier) along with the best known classical counterparts for binary
classification problems. Here we focus, in particular, on prediction grids and
ROC/AUC characteristics for assessing the performance of the classifier.
Section V is devoted to the conclusions and future directions.Finally, in
Appendix A material we show some results for 3-dimensional non-convex boundary
classification problems and demonstrate the performance of our FLL HYBRID
classifiers.

##  I Problem setting

We consider a non-trivial classification problem and will train single and
multi-qubit variational quantum circuits to achieve this goal. The data is
generated as a set of random points in a plane $x_{1},x_{2}$ and labelled as 1
(blue) or 0 (red) depending on whether they lie inside or outside of a given
2-dimensional non-convex figure. The goal is to train a quantum circuit to
predict the label (red or blue) given an input point’s coordinate.

##  II Review of existing QML frameworks

In this section we briefly review three different necessary building block QML
architectures for our software package : hybrid-neural networks Schuld  _et
al._ (2020); Schuld and Killoran (2019); Farhi and Neven (2018), variational
circuits Cerezo  _et al._ (2020); Bharti  _et al._ (2021) and data-reuplodaing
Pérez-Salinas  _et al._ (2020, 2021).

###  II.1 Hybrid classical-quantum classifier (Hybrid)

Hybrid neural networks are formed by concatenating classical and quantum
neural networks and can bring a great advantage by having a number of features
in the initial classical layers that exceeds the number of qubits in the
quantum layer. Normally we assume that in each layer we have one qubit for
each feature and a sequence of one and two-qubit gates acting on it.

Figure 1: Quantum circuit implementing hybrid classical-quantum classifier ,
each block corresponds to the layer of classical neural network.Taken from the
Ref. hyb (2020).

To create a quantum-classical neural network a hidden layer is normally
implemented utilising a parameterized quantum circuit. By ”parameterized
quantum circuit”, we mean a quantum circuit where, for instance, the rotation
angles for each gate are trainable parameters, specified by the components of
a classical input vector. The outputs from our neural network’s previous layer
will be collected and used as the inputs for our parameterized circuit.
Normally the measurement statistics of the quantum circuit can then be fed as
inputs for the following layer. Notice that this kind of approach establishes
a link between the classical and quantum neural networks. An important point
to note is that a single qubit classifier generates no entanglement, and can
therefore be simulated classically. If one hopes to achieve a quantum
advantage using hybrid neural networks, one needs to introduce several qubits
and consequently entangle them, harnessing that quantum resource.

###  II.2 Variational Quantum Algorithms(VQA)

Variational circuits are quantum circuits that have learning parameters that
are optimised through classical learning subroutines, in spirit, this kind of
approach is reminiscent of a Variational Quantum Eigensolver (VQE) Cerezo  _et
al._ (2020); Bharti  _et al._ (2021).

Figure 2: Schematic diagram of a Variational Quantum Algorithm (VQA).Figure is
borrowed from the Ref. Cerezo  _et al._ (2020).

As schematically shown in Fig.2, the first step towards developing a VQA is to
define a cost or loss function C which encompasses the solution to the
problem. After that, an ansatz is introduced through the quantum operation
depending on a set of continuous or discrete parameters that can be optimized.
This ansatz is then trained in a hybrid quantum-classical loop to solve the
optimization task at hand

$\theta^{*}=\arg\min_{\theta}C(\theta).$ (1)

The trademark of VQAs is that a quantum computer is utilised to estimate the
cost function $C(\theta)$ while harnessing the power of classical optimizers
for training the quantum parameters. A rather crucial assumption here is that
one cannot efficiently compute the cost function on the classical computer, as
this would imply an absence of quantumm advantage in the VQA framework.

###  II.3 DRC: Data-reuploading classifier

Data re-uploading is a subclass of quantum embedding which is realised by
catenating repeating units in a row. Single-qubit rotations applied several
times along the circuit generate the necessary non-linearity for engineering a
functional neural network. Moreover, it has been demonstrated that a single
qubit can realise both being a universal quantum classifier Pérez-Salinas  _et
al._ (2020) and being a universal approximant Pérez-Salinas  _et al._ (2021)

Figure 3: Quantum circuit implementing data re-uploading, each block
corresponds to the layer of classical neural network. Image is take from the
Ref. Ahmed (2021).

To load $[x_{1},x_{2}]$ into the qubit, we just start from some initial state
vector, $|0\rangle$, apply the unitary operation $U(x_{1},x_{2},0)$ and end up
at a new point on the Bloch sphere. Here we have padded 0 since our data is
only 2-dimensional. Authors of Ref. Pérez-Salinas  _et al._ (2020) discuss how
to load a higher dimensional data point
$[x_{1},x_{2},x_{3},x_{4},x_{5},x_{6}]$ by breaking it down into sets of three
parameters $(U(x_{1},x_{2},x_{3},U(x_{4},x_{5},x_{6})$.

After the data loading stage, we want to have some trainable non-linear model
analogous to a deep neural network with a non-linear activation function where
one can learn the weights of the model. Fig. 3 are showing how data
reuploading is implemented by the sequence of $B$ repeating units which
correspond to the layers of classical neural networks, consequently one
expects that with increasing $B$ one gets a deeper neural network and
consequently better learning can be obtained. Each unit is realised as a
product of two unitaries $U(x_{1},x_{2},0)$ and
$U(\theta_{1},\theta_{2},\theta_{3})$, where the second unitary contains the
trainable parameters. This approach can be boosted by introducing strongly
entangling layers through utilisation of CNOT gates as it is shown on Fig. 4.

As it has been mentioned in the previous section, one can also speculate that
multiple qubits with an entanglement between them could provide some quantum
advantage over classical neural networks.

Figure 4: Quantum circuit implementing data-reuploading with strongly
entangling layers, where entanglement between the blocks is introduced with
controlled two qubit gates. Figure is take from the Ref. Ahmed (2021).

.

Figure 5: VC-DRC example for two Blocks and one entangling layer per block
(B=2 , L=1).

##  III FH:NN/VC-DRC and FH:VC-DRC/NN Full hybrid neural networks enriched
with Variational and data-reuploading technics

Figure 6: (First row) Block Diagram of the specific QNode proposed in the
current study. (Second row) Block Diagram of the Full Hybrid (NN/VC-DRC)
classifier where a VC-DRC circuit is placed after a classical neural network,
(Third row) Block Diagram of the Full Hybrid classifier where a fully quantum
VC-DRC circuit is placed before classical neural network (VC-DRC/NN).

###  III.1 VC-DRC

A Variational classifier circuit (VC) consists of a data embedding layer which
in turn loads the classical data into the qubits followed by the entangling
layers (CNOT gates that entangle each qubit with its neighbour) and the
measurement outcome is the expectation value of a Pauli observable for each
qubit pen (2021). In our case we use an angle embedding $R_{x}$. In order to
combine a VC circuit with DRC technique we define as one block (B) a sequence
of data embedding and entangling layers (L). By adding many blocks we re-
introduce the input data into the model. In Fig. 5 we illustrate such VC-DRC
circuit for B=2 , L=1.

###  III.2 QNode

Pennylane is an open-source software framework for differentiable programming
of quantum computers. All our models are builded using this framework. In
Pennylane an object QNode represents a quantum node in the hybrid
computational graph. Here a quantum function is used to create a quantum node,
or QNode object, encapsulating the quantum function (corresponding to a
variational circuit) and the device used to execute the function.

Here we would like to clarify what we call a QNode in the scope of the current
manuscript. As depicted in the first row of the Fig. 6, a QNode is a specific
circuit where input data are passed to the quantum Node which consists of a
VC-DRC and a final classical decision layer.

The input classical data is passed into the quantum circuit as rotation angles
$R_{x}$ (”angle embedding”) on the Bloch sphere. After the computation on the
quantum node is completed, measurement is performed and the outcome is passed
to the classical decision layer which decides the final prediction label of
the binary classifier.

###  III.3 FH:NN/VC-DRC and FH:VC-DRC/NN

In this section we propose two varieties of a new binary classifier
architectures which which are named under the common name Full Hybrid (FH). On
a basic level, FH consists of a VC-DRC combined with classical layers. We came
up with two novel architectures for the FH circuits depending on whether the
VC-DRC circuit is at the end or at the beggining of the model(named FH: NN/VC-
DRC and FH: VC-DRC/NN respectively). These architectures are extensively
studied in the current manuscript as novel candidates for performing binary
classification on noisy datasets (See second and third row of Fig. 6).
Moreover, we demonstrate in great detail, in the next section, that FH
architectures outperforms several previously known quantum classifiers and
performs equally well compared to classical counterparts. We comment that, in
the FH:VC-DRC/NN case the power of the approach is given by the fact that the
VC-DRC part can be acting as a quantum embedding as evidenced by Refs. Lloyd
_et al._ (2020b); Mitarai  _et al._ (2018). In this case the goal is to derive
the angle embedding for which the separation of the data labels is maximized
in the Hilbert space. In what follows, we provide more technical details on
the Full Hybrid architectures with an emphasis on explaining and giving more
details on the Master, Feeding and Decision classical layers which are
depicted in the second row of the Fig. 6.

For FH: NN/VD-DRC the first part is a classical neural network(NN), followed
by the VC-DRC circuit and a final decision layer, which is just a single
neuron layer with a sigmoid activation function. We use a classical NN that is
not fine-tuned for this specific classification task. Moreover, as it can be
seen on Fig. 6, the classical NN can contain an arbitrary number of layers,
and each layer can contain an arbitrary number of neurons but the last layer
(Feeding classical layer) should always have the same number of neurons as
number of qubits. In our 2D case the classical NN, consists of a 2-neuron
layer with ReLU as the activation function (Master classical layer), followed
by a 2-neuron layer with a Leaky ReLU activation function (Feeding classical
layer).

For FH: VC-DRC/NN the first part is a VC-DRC circuit, followed by the
previously described NN network and the same final decision layer. We remark
that we also tried Sigmoid , tanh and general geometric functions, and the
best performing activation functions were selected.

##  IV Comparative study of different quantum and classical classifiers

Here we test several models (including our proposed models) and benchmark them
against each other as well as to the best-known classical machine learning
counterparts by running on the simulator backends(such as Aer in qiskit) for
2-dimensional and 3-dimensional non-convex datasets. Then we will study the
resilience of our different approaches to the noise by introducing
asymmetrical Gaussian noise by studying the prediction grids and ROC/AUC
characteristics.

This kind of study sheds light on the learning properties as a function of the
amount of existing noise in the dataset. These results have been obtained by
systematic hyperparameter tuning, by observing how the ROC/AUC changes with:
the number of repeating units in the data-re- uploading approach, batch size,
number of epochs and the number of strongly entangling units.

Figure 7: (First row) Pattern of Dataset. (Second row) Pattern of dataset for
different noise levels.

To produce datasets with noise, we introduce asymmetrical (here noise is only
applied to one class) Gaussian noise (N). In bottom Figure 7 we plot the case
of N=0.0 , N=0.6 and N=1.2. Each dataset has 6000 data points and is further
equally splitted into training and testing datasets.

To demonstrate the power of the data-reuploading technique combined with the
variational classifier in the VC-DRC model, we plot the ROC/AUC versus noise
for different number of blocks. The results are shown in Fig.8. It is apparent
from Fig. 8 (first row) that with an increasing number of repeating blocks, we
get better ROC/AUC for every noise level for the DRC classifier. On the bottom
row of figure Fig. 8 we show results for the VC-DRC where compared to DRC we
get even higher ROC/AUC. We remark that no major improvements are seen for a
Block number greater than six. From now on, in all codes of this section, we
will set the number of blocks equal to six (B=6). In what follows we specify
number of blocks and layers for each classifier: 1\. The single qubit DRC
(B=6) 2\. 2 qubit VC (with 6 layers, L=6) 3\. VC-DRC (B=6,L=1) 4\. QNode
(B=6,L=1) 5\. FH: VC-DRC/NN (B=6,L=1) 6\. FH: NN/VC-DRC (B=6,L=1). All models
have been trained for maximum 35 epochs, using the same optimizer and learning
rate. The best result during the training process is shown. On the left Fig.9
we compare all the previously mentioned classifiers. As we can see from on the
left Fig. 9 VC-DRC outperforms both VC and DRC.VC-DRC and Qnode have almost
identical performance. The FH:NN/VC-DRC outperforms all classifiers whilst
FH:VC-DRC/NN has slightly worse behavior. In the right column of Fig.9 we can
see the prediction grids for all classifiers at different noise levels. For
low noise levels (Noise/10=0), DRC and VC struggle to capture the prediction
grid pattern while VC-DRC and FH almost capture it. For medium noise levels
(Noise/10=6), DRC tends to capture the noise (overfitting) while VC looks more
stable. VC-DRC still captures the main pattern but also shows signs of
overfitting. FH performs very well thanks to the classical preprocessing and
utilising the power brought by VC-DRC. For high noise levels (Noise/10=12) FH
captures the pattern and shows robustness to the noise while the rest of the
classifiers are capturing the noise. In order to demonstrate that FULL HYBRID
does not perform well only because of the strong classical NN attached to the
quantum circuit, we benchmark FH versus just the Classical part (NN) and
versus just the Quantum part (QNode). On from Fig. 10 we show results for two
NN’s one with 35 epochs training (same training epochs as in the FH) and 3000
epochs just to see what is the best outcome this NN can produce. We conclude
that the FH outperforms both it’s components (NN and QNode) which shows that
FH is more powerful classifier than it’s isolated parts.

To test even further the FH classifier, we benchmark its performance against a
great number of classical counterparts, which are specified in the inset of
the Fig.11. Interestingly, this figure shows that in the high noise region,
the quantum classifier even outperforms some classical ones or performing
equally well in all noise regions. We also see that compared to the other
classical approaches (QDA, Decision tree, KNN and Random forest) that are well
suited for non-convex classification problems and showing good performance in
all noise regimes. In Fig. 12 we are showing results for a more complicated
non-convex classification problem versus noise. In the table on the right we
summarize the highest ROC/AUC scores for the respective classifiers. In the
left figure we show prediction grids for the respective quantum classifiers.
As in previous case, VC is more stable to noise and DRC tends to overfit and
explores richer prediction grids. That is why VC-DRC, which combines both
features, and the more complex approach like FH, is giving great results as
apparent from row number 6. Surprisingly, for this particular dataset FH:
NN/VC-DRC fails to capture the pattern of the dataset while FH: VC-DRC/NN
captures the pattern and has the highest ROC/AUC score. It should be noted
that the FH models outperforms again both it’s components (NN and QNode).

Figure 8: ROC/AUC as a function of number of repeating blocks of data re-
uploading for the DRC classifier (Top row) and VC-DRC classifier (Bottom row)

.

Figure 9: (Left) ROC/AUC for DRC, VC and VC-DRC classifiers for different
noise levels. (Right) Prediction grids for respective classifiers for
different noise levels.

.

Figure 10: ROC/AUC for increasing level of noise for the classification of 2d
dataset. Here we benchmark FH versus just the classical part (NN) and versus
just the quantum part (QNode). Figure 11: ROC/AUC for increasing level of
noise for the classification of 2d dataset. Here we benchmark a great number
of classical classifiers against our proposed FULL HYBRID classifier. Figure
12: (Right) ROC/AUC for classifiers for different noise levels. (Left)
Prediction grids for respective classifiers for different noise levels.

##  V Conclusions and Future Directions

In this paper, we applied Quantum Machine Learning frameworks to improve
binary classification models for noisy datasets which are prevalent in
financial markets. The metric used for assessing the performance of our
quantum classifiers is the area under the receiver operating characteristic
curve (ROC/AUC). By combining such approaches as hybrid-neural networks,
parametric circuits, and data re-uploading we created a new approach called
Full Hybrid (FH). We tested our models for the classification of 2 and
3-dimensional non-convex datasets and benchmarked them against each other as
well as to the best known classical machine learning counterpart by running
simulations on quantum simulators. Then, by introducing asymmetrical Gaussian
noise in the input datasets, we studied the resilience of our different
approaches to noise. This kind of study sheds light on the learning efficacy
to the amount of noise in the dataset. In the scope of the manuscript we also
performed systematic hyperparameter tuning by studying how ROC/AUC changes
with the number of repeating units in the data-re-uploading approach, number
of qubits, batch size, number of epochs and number of strongly entangling
units. An extensive benchmarking of our new QML approach against existing
quantum and classical classifier models reveals that our novel (FH) models
exhibits better learning properties to asymmetric Gaussian noise in the
dataset compared to known quantum classifiers, and performs equally well for
existing classical counterparts. Yet more understanding of the merits of the
(FH) classifier has been gained by a detailed analysis and comparison of the
prediction grids for the VC, DRC, VC-DRC, QNode binary classifiers. We
observed that for low noise levels , DRC and VC struggle to capture the
prediction grid pattern while VC-DRC and FH almost fully capture it. For
medium noise levels, DRC tends to capture the noise (overfitting) while VC
looks more stable. VC-DRC still captures the main pattern but also shows signs
of overfitting. FH performs very well thanks to the classical preprocessing
and utilising the power brought by VC-DRC. For high noise levels, (FH)
captures the pattern and shows robustness in noise while the rest of the
classifiers are capturing the noise in the dataset. It is a well conceived
fact that one of the bottlenecks for VQAs is the phenomenon called ”barren
plateau” McClean  _et al._ (2018). As it has been demonstrated in Ref. McClean
_et al._ (2018), a given spin-spin interacting Hamiltonians cost function may
exhibit a barren plateau, associated with exponentially vanishing variance in
its first derivative, when one increases the number of qubits. Moreover, the
VQE based algorithms perform a classical-quantum feedback loop to update the
parameters of the parametric quantum circuits. For future studies, it would be
interesting to implement non-VQA algorithms for building more efficient
quantum classifiers. By the time a classical computer calculates its output,
the classical-quantum feedback loop limits the efficiency of the quantum
device, slowing the algorithm execution on current cloud computing frameworks.
Most of the obstacles faced by VQE, such as the barren plateau issue McClean
_et al._ (2018) as well as lacking a systematic method to select the ansatz
and the innate necessity of having controlled unitaries, have been recently
tackled by suggesting a quantum assisted simulator (QAS) Haug and Bharti
(2020); Bharti (2020). Remarkably, The QAS algorithm does not require any
classical-quantum feedback loop, can be parallelized, alleviates the barren
plateau problem by prescribing a systematic approach to constructing the
ansatz, and is not based on the usage of complicated unitaries. Of course, for
the future studies, one has to keep in mind that sensitivity to errors and
noise in qubits and quantum gates are the two most prominent obstacles towards
scalable universal quantum computers. Given that, it would be nice to study
how our results are affected if one implements noise models for realistic
quantum backends. In general, a noisy quantum system is described by the open
system model and systems dynamics within the Born-Markov approximation is
governed by the Lindblad master equation for the system’s density matrix
Carmichael (1993). Another approach to describe the different noise channels
is based on Kraus operators which are the most general physical operations
acting on density matricesNielsen and Chuang (2002). Since most of our codes
were based on PennyLane, it is instructive to mention that Pennylane has 3
different ways for implementing noise in quantum circuits: classical
parametric randomness, PennyLane’s built-in default.mixed device, and plugins
for other platforms. Of course, Quantum circuits may be run on a variety of
backends, some of which have their own associated programming languages and
simulators. PennyLane interfaces to these other languages via plugins such as
for Cirq and Qiskit. Finally, it is also worth mentioning that we plan to test
our classifiers on real world financial data. Here we hope to demonstrate that
our proposed classifiers have the potential to improve credit scoring
accuracy. Credit scoring provides lenders and counterparties better
transparency of the credit risk they are taking when dealing with a
counterparty. For large companies, this transparency is provided by public
credit ratings. Small and medium enterprise companies(SMEs) are not covered by
rating agencies and are suffering from reduced availability of credit. These
datasets, along with the best classical neural networks, will by provided by
the company called Tradeteq (Tradeteq is a value-added service provider to the
Networked Trading Platform (NTP) of Singapore).

In summary, we have demonstrated that the FH architecture outperforms several
previously known quantum classifiers along with some of the best known
classical counterparts. Interestingly, in the FH: VC-DRC/NN case, the power of
the approach is given by the fact that, the VC-DRC part is acting as quantum
embedding.

##  Appendix A Benchmarking binary classiffier for 3 dimensional non-convex
datasets

In order to gain insights about the performance of our FH classifier
architecture in higher dimensions we study classification of 3-dimensional
non-convex figure, dataset of which for the different levels of noise is
presented in the first rows of Fig 13 and Fig 14. Then we proceed by
benchmarking FH classifiers against the QNode and VC. As we can see for this
particular dataset, FH:NN/VC-DRC shows the best performance.

We also found it interesting to show (See Fig 14) a 2D projection of the
prediction grid for the dataset which has 3 red regions and thus is the
hardest non-convex example considered so far. In rows 2-6 we show prediction
grids for the VC-DRC, QNode, FH:VC-DRC and FH:NN/VC-DRC, respectively. We see
that in the low noise regime FH:VC-DRC/NN and QNode perform the best at the
same time, VC-DRC fails to capture upper right corner of that red region.
Similar findings hold for the intermediate noise regime, However in the high
noise regime FH:VC-DRC/NN and QNode obtain only two regions in red due to too
much noise in the dataset, yet VC-DRC is still largely overfitting showing 3
disconnected regions, but guessing their locations wrongly. To summarise, FH
classifiers show the maximal resilience to the noise in the dataset and are
well suited for non-convex boundary classification problems.

Figure 13: (First Row) Dataset for the 3 dimensional non-convex problem, with
increasing noise levels from left to the right (Second row) ROC/AUC for the
QNode (red dots) , Full-Hybrid (grey dots) and MLPC (orange dots) classifiers
for different noise levels. Figure 14: (First row) ROC/AUC for classifiers for
different noise levels for 3D dataset. (Second row) 2D projection prediction
grids for respective classifiers for different noise levels.

Acknowledgements— All the codes used in the manuscript will be provided under
the reasonable request. D.A. would like to thank Kishor Bharti for useful
discussions on Quantum Machine Learning (QML) and on newly emerging non-VQA
algorithms. P.Griffin would like to acknowledge to the Monetary Authority of
Singapore (MAS) and to Tradeteq for their support in this work.

## References

  * Preskill (2018) J. Preskill, Quantum 2, 79 (2018).
  * Bharti  _et al._ (2021) K. Bharti, A. Cervera-Lierta, T. H. Kyaw, T. Haug, S. Alperin-Lea, A. Anand, M. Degroote, H. Heimonen, J. S. Kottmann, T. Menke, _et al._ , arXiv preprint arXiv:2101.08448 (2021).
  * Deutsch (2020) I. H. Deutsch, PRX Quantum 1, 020101 (2020).
  * Preskill (1998a) J. Preskill, in  _Introduction to quantum computation and information_ (World Scientific, 1998) pp. 213–269.
  * Gottesman (1998) D. Gottesman, Physical Review A 57, 127 (1998).
  * Shor (1996) P. W. Shor, in  _Proceedings of 37th Conference on Foundations of Computer Science_ (IEEE, 1996) pp. 56–65.
  * Arute  _et al._ (2019) F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell, _et al._ , Nature 574, 505 (2019).
  * Harrow and Montanaro (2017) A. W. Harrow and A. Montanaro, Nature 549, 203 (2017).
  * Zhong  _et al._ (2020) H.-S. Zhong, H. Wang, Y.-H. Deng, M.-C. Chen, L.-C. Peng, Y.-H. Luo, J. Qin, D. Wu, X. Ding, Y. Hu, _et al._ , Science 370, 1460 (2020).
  * Biamonte  _et al._ (2017) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Nature 549, 195 (2017).
  * Wittek (2014) P. Wittek,  _Quantum machine learning: what quantum computing means to data mining_ (Academic Press, 2014).
  * Schuld (2018) M. Schuld,  _Supervised learning with quantum computers_ (Springer, 2018).
  * Nielsen and Chuang (2002) M. A. Nielsen and I. Chuang, “Quantum computation and quantum information,” (2002).
  * Preskill (1998b) J. Preskill, California Institute of Technology 16, 10 (1998b).
  * Goodfellow  _et al._ (2016) I. Goodfellow, Y. Bengio, and A. Courville, Deep learning 1, 98 (2016).
  * Jordan and Mitchell (2015) M. I. Jordan and T. M. Mitchell, Science 349, 255 (2015).
  * Harrow  _et al._ (2009) A. W. Harrow, A. Hassidim, and S. Lloyd, Physical review letters 103, 150502 (2009).
  * Huang  _et al._ (2019) H.-Y. Huang, K. Bharti, and P. Rebentrost, arXiv preprint arXiv:1909.07344 (2019).
  * Rebentrost  _et al._ (2018) P. Rebentrost, A. Steffens, I. Marvian, and S. Lloyd, Physical review A 97, 012327 (2018).
  * Lloyd  _et al._ (2014) S. Lloyd, M. Mohseni, and P. Rebentrost, Nature Physics 10, 631 (2014).
  * Wiebe  _et al._ (2012) N. Wiebe, D. Braun, and S. Lloyd, Physical review letters 109, 050505 (2012).
  * Schuld and Killoran (2019) M. Schuld and N. Killoran, Physical review letters 122, 040504 (2019).
  * Pérez-Salinas  _et al._ (2020) A. Pérez-Salinas, A. Cervera-Lierta, E. Gil-Fuster, and J. I. Latorre, Quantum 4, 226 (2020).
  * Pérez-Salinas  _et al._ (2021) A. Pérez-Salinas, D. López-Núñez, A. García-Sáez, P. Forn-Díaz, and J. I. Latorre, arXiv preprint arXiv:2102.04032 (2021).
  * Lloyd  _et al._ (2020a) S. Lloyd, M. Schuld, A. Ijaz, J. Izaac, and N. Killoran, arXiv preprint arXiv:2001.03622 (2020a).
  * Mitarai  _et al._ (2018) K. Mitarai, M. Negoro, M. Kitagawa, and K. Fujii, Physical Review A 98, 032309 (2018).
  * Schuld  _et al._ (2020) M. Schuld, A. Bocharov, K. M. Svore, and N. Wiebe, Physical Review A 101, 032308 (2020).
  * Farhi and Neven (2018) E. Farhi and H. Neven, arXiv preprint arXiv:1802.06002 (2018).
  * Tacchino  _et al._ (2019a) F. Tacchino, C. Macchiavello, D. Gerace, and D. Bajoni, npj Quantum Information 5, 1 (2019a).
  * Cappelletti  _et al._ (2020) W. Cappelletti, R. Erbanni, and J. Keller, in  _2020 IEEE International Conference on Quantum Computing and Engineering (QCE)_ (IEEE, 2020) pp. 22–29.
  * Wiebe  _et al._ (2016) N. Wiebe, A. Kapoor, and K. M. Svore, arXiv preprint arXiv:1602.04799 (2016).
  * Liao  _et al._ (2018) Y. Liao, D. Ebler, F. Liu, and O. Dahlsten, arXiv preprint arXiv:1810.12948 (2018).
  * Schuld  _et al._ (2017) M. Schuld, M. Fingerhuth, and F. Petruccione, EPL (Europhysics Letters) 119, 60002 (2017).
  * Tiwari and Melucci (2019) P. Tiwari and M. Melucci, IEEE Access 7, 42354 (2019).
  * Blank  _et al._ (2020) C. Blank, D. K. Park, J.-K. K. Rhee, and F. Petruccione, npj Quantum Information 6, 1 (2020).
  * Park  _et al._ (2020) D. K. Park, C. Blank, and F. Petruccione, Physics Letters A 384, 126422 (2020).
  * Tacchino  _et al._ (2019b) F. Tacchino, C. Macchiavello, D. Gerace, and D. Bajoni, npj Quantum Information 5, 1 (2019b).
  * Rossi  _et al._ (2013) M. Rossi, M. Huber, D. Bruß, and C. Macchiavello, New Journal of Physics 15, 113022 (2013).
  * Bergholm  _et al._ (2018) V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, M. S. Alam, S. Ahmed, J. M. Arrazola, C. Blank, A. Delgado, S. Jahangiri, _et al._ , arXiv preprint arXiv:1811.04968 (2018).
  * Killoran  _et al._ (2019) N. Killoran, J. Izaac, N. Quesada, V. Bergholm, M. Amy, and C. Weedbrook, Quantum 3, 129 (2019).
  * Broughton  _et al._ (2020) M. Broughton, G. Verdon, T. McCourt, A. J. Martinez, J. H. Yoo, S. V. Isakov, P. Massey, M. Y. Niu, R. Halavati, E. Peters, _et al._ , arXiv preprint arXiv:2003.02989 (2020).
  * Efthymiou  _et al._ (2020) S. Efthymiou, S. Ramos-Calderer, C. Bravo-Prieto, A. Pérez-Salinas, D. García-Martín, A. Garcia-Saez, J. I. Latorre, and S. Carrazza, arXiv preprint arXiv:2009.01845 (2020).
  * Kottmann  _et al._ (2021) J. Kottmann, S. Alperin-Lea, T. Tamayo-Mendoza, A. Cervera-Lierta, C. Lavigne, T.-C. Yen, V. Verteletskyi, P. Schleich, A. Anand, M. Degroote, _et al._ , Quantum Science and Technology (2021).
  * Orus  _et al._ (2019) R. Orus, S. Mugel, and E. Lizaso, Reviews in Physics 4, 100028 (2019).
  * Benedetti  _et al._ (2019) M. Benedetti, E. Lloyd, S. Sack, and M. Fiorentini, Quantum Science and Technology 4, 043001 (2019).
  * Cerezo  _et al._ (2020) M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, _et al._ , arXiv preprint arXiv:2012.09265 (2020).
  * Funcke  _et al._ (2021) L. Funcke, T. Hartung, K. Jansen, S. Kühn, and P. Stornati, Quantum 5, 422 (2021).
  * hyb (2020) https://qiskit.org/textbook/ch-machine-learning/machine-learning-qiskit-pytorch.html (2020).
  * Ahmed (2021) S. Ahmed, https://pennylane.ai/qml/demos/tutorial-data-reuploading-classifier.html (2021).
  * pen (2021) https://pennylane.ai/qml/demos/tutorial/variational/classifier.html (2021).
  * Lloyd  _et al._ (2020b) S. Lloyd, M. Schuld, A. Ijaz, J. Izaac, and N. Killoran, arXiv preprint arXiv:2001.03622 (2020b).
  * McClean  _et al._ (2018) J. R. McClean, S. Boixo, V. N. Smelyanskiy, R. Babbush, and H. Neven, Nature communications 9, 1 (2018).
  * Haug and Bharti (2020) T. Haug and K. Bharti, arXiv preprint arXiv:2011.14737 (2020).
  * Bharti (2020) K. Bharti, arXiv preprint arXiv:2009.11001 (2020).
  * Carmichael (1993) H. Carmichael, An Open Systems Approach to Quantum Optics: Lectures Presented at the Université Libre de Bruxelles October 28 to November 4, 1991 , 5 (1993).