#  Minimum Detection Efficiencies for Loophole-free Genuine Nonlocality Tests

Subhendu B. Ghosh  Physics and Applied Mathematics Unit, 203 B.T. Road Indian
Statistical Institute Kolkata, 700108  Snehasish Roy Chowdhury  Physics and
Applied Mathematics Unit, 203 B.T. Road Indian Statistical Institute Kolkata,
700108  Ranendu Adhikary  Electronics and Communication Sciences Unit, 203
B.T. Road Indian Statistical Institute Kolkata, 700108  Arup Roy  Department
of Physics, A B N Seal College Cooch Behar, West Bengal 736101, India  Tamal
Guha  QICI Quantum Information and Computation Initiative, Department of
Computer Science, The University of Hong Kong, Pokfulam Road, Hong Kong

###### Abstract

The certification of quantum nonlocality, which has immense significance in
architecting device-independent technologies, confronts severe experimental
challenges. Detection loophole, originating from the unavailability of perfect
detectors, is one of the major issues amongst them. In the present study we
focus on the minimum detection efficiency (MDE) required to detect various
forms of genuine nonlocality, originating from the type of causal constraints
imposed on the involved parties. In this context, we demonstrate that the MDE
needed to manifest the recently suggested $T_{2}$-type nonlocality deviates
significantly from perfection. Additionally, we have computed the MDE
necessary to manifest Svetlichny’s nonlocality, with state-independent
approach markedly reducing the previously established bound. Finally,
considering the inevitable existence of noise we demonstrate the robustness of
the imperfect detectors to certify $T_{2}$-type nonlocality.

Introduction.– Over the past thirty years, a distinctive category of
information theory has emerged popularly known as Device-Independent (DI)
technology, which surpasses several limitations of classical physics [2, 1, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. Admittedly, the foundational constituent
behind such an ever expanding architecture was implanted in the seminal
results of John S. Bell [14, 15], which derives a testable criteria for
certifying the nonlocal aspect of any operational theory. In a nutshell, the
nonlocality is a signature of multipartite correlations, incompatible with the
classical outlook to the causal constraints imposed on the involved parties.
For example, the pioneering nonlocality test as proposed by Clauser-Horne-
Shimony-Holt (CHSH) considers a bipartite scenario, where spatially separated
agents, supplemented with some noncommunicating correlation, are asked to
produce a dichotomic random variable depending upon their individual classical
inputs [16]. Finally, an inequality based on their local outputs is so
designed that the violation of the same led to an apparent contradiction with
their imposed causal structure, which can further be removed by relaxing the
local-realistic description of their shared correlation. This in turn
certifies the nonlocal signature of that correlation. However in the
multipartite scenario, there are various causal constraints that can be
imposed on the spatially separated parties and accordingly with an increasing
number of parties involved, the complexity associated with their local input-
output statistics also grows. In the most basic multipartite scenario, three
spatially separated parties are asked to generate local random variables
depending upon the input given to them locally. On the other hand, when there
is no causal constraint imposed on two of the parties (personified as Bob and
Charlie), then their local input-output statistics should be factorized
between Alice vs. Bob-Charlie when their measurements are space like separated
from that of Alice. While any of their obtained statistics compatible with
such a scenario is termed as bi-local (BL) correlations, an apparent
contradiction of the same can be characterized by an inequality proposed by
Svetlichny [17] and coined as genuinely nonlocal. In a more complicated
scenario, one may further consider a temporal ordering in the input-output
generation of the parties Bob and Charlie. Intuitively, this allows Charlie to
make a redundant signalling to Bob, when he is in causal future of Bob and
vice-versa. It is instructive that such a causal constraint is stronger than
that of the earlier. Accordingly the set of correlations compatible with these
constraints, namely the time-ordered bi-local (TOBL) correlations is strictly
included inside the set of BL correlations [18]. In isolation, the
characterization of BL correlations has emerged over the past decade when an
operational inconsistency of the Svetlichny’s definition of bi-locality is
reported and it has further fuelled the development of various refined causal
structures on genuine nonlocality [18, 19, 20].

Interestingly, quantum theory exhibits the signature of nonlocality in its
simplest possible bipartite scenario. However, in the recent era of many body
physics the importance of multipartite quantum correlations such as
entanglement and nonlocality is needless to over-emphasize. Notably, the
genuine nonlocality has garnered significant attention, primarily owing to its
implications in Device-Independent Random Number Generation (DIRNG), Device-
Independent Quantum Key Distribution (DIQKD), and Device-Independent
certification of genuine entanglement [23, 21, 26, 22, 24, 25, 27]. However,
the experimental certification of the potential nonlocal signature of quantum
correlations encounter serious challenges by the possible loopholes in Bell-
test. One such major operational loophole is the detection loophole, which
emerges due to an inefficient photon detector, used to record the local input-
output statistics obtained by each of the spatially separated agents. In
particular, there are possible instances where the imperfect detector may
yield no click, i.e., an inclusive outcome. While one might consider
overcoming this issue by simply discarding the no-click outcomes, such post-
selection requires an additional assumption, specifically the fair-sampling
assumption. Without this assumption, even a local hidden variable model can
violate Bell inequalities [28]. A more comprehensive and appropriate approach
to addressing this challenge is to regard the no-click event as a potential
valid outcome achievable during the Bell test. The second approach needs a
cut-off on the detection efficiency to observe nonlocality. However, given an
entangled state, accompanied with a set of observable may require further
bound on the minimum detection efficiency. For instance, Bell test involving
two qubit maximally entangled state demands 83${\%}$ detection efficiency,
while non maximally entangled state needs 67${\%}$ [29, 30, 31]. Moreover,
motivated by experimental set-up, a minimum detector efficiency for the
asymmetric bipartite Bell-test has also been reported [32, 33]. The study of
such bipartite nonlocality tests under inefficient detectors has gained
further importance by developing a series of sophisticated experimental set-up
[34, 35, 36, 37, 38]. In spite of having several foundational and practical
importance, in the multipartite scenario the nonlocality test with inefficient
detectors is mostly restricted to its simplest form [39, 40], with a recent
addition for Svetlichny-type genuine nonlocality test [41, 42].

On the other hand, the resource theoretic construction of quantum nonlocality
contradicts the existence of an unique nonlocal measure, even in its simplest
scenario [43]. This further motivates to investigate possible detection
efficiencies for other possible tests for genuine nonlocality by imposing
different causal-constraints on the involved parties [18, 19, 20]. In this
work, we consider various possibilities of nonlocality tests and derive the
minimal possible sophistication one may require for the detectors to certify
genuine nonlocality in the proposed scenario. With the help of a state-
independent generalized study, we have been able to significantly lower the
already existing bound on detection efficiency, for the Svetlichny-type
nonlocality test. Moreover, our results encapsulates an experimentally
motivated scenario for hybrid entanglement, where different quantum particles
are allowed to be entangled via inter-particle interactions [44, 45, 46, 47].
Finally, a brief analysis of noise robustness for the inefficient detectors
have been reported. This, in turn, hints towards a complimentary relation
between the detection efficiency and the possible range of one-parameter
quantum settings (i.e., the state along with the triplets of measurement
pairs) to exhibit genuine nonlocality, in the presence of an unavoidable
noise.

Genuine nonlocality under various causal constraints.– Consider the simplest
multipartite scenario, where three spatially separated agents perform local
operations on their individual subsystems. Further, their space-time relation
is restricted in such a way that no more than two of them can reside in a same
light cone. As a consequence, any local-realistic hidden variable description
simulating their local input-output statistics must be factorized in either of
the three possible bi-partitions. This in essence reads,

$\displaystyle
P(abc|XYZ)=\sum_{\lambda}q_{\lambda}\,P_{\lambda}(ab|XY)\,P_{\lambda}(c|Z)+$
$\displaystyle\sum_{\mu}q_{\mu}\,P_{\mu}(ac|XZ)\,P_{\mu}(b|Y)+\sum_{\nu}q_{\nu}\,P_{\nu}(bc|YZ)\,P_{\nu}(a|X),$
(1)

where, $0\leq q_{\lambda},q_{\mu},q_{\nu}\leq 1$ are the respective
probabilities compatible with the causal constraint and
$\sum_{\lambda}q_{\lambda}+\sum_{\mu}q_{\mu}+\sum_{\nu}q_{\nu}=1$. In 1987,
Svetlichny pointed out that all possible correlations that can be decomposed
in the form of Eq.( Minimum Detection Efficiencies for Loophole-free Genuine
Nonlocality Tests), respects the following inequality over their operational
statistics [17]:

$\displaystyle-\langle A_{0}B_{0}C_{0}\rangle-\langle
A_{1}B_{0}C_{0}\rangle+\langle A_{0}B_{1}C_{0}\rangle-\langle
A_{1}B_{1}C_{0}\rangle$ $\displaystyle-\langle A_{0}B_{0}C_{1}\rangle+\langle
A_{1}B_{0}C_{1}\rangle-\langle A_{0}B_{1}C_{1}\rangle-\langle
A_{1}B_{1}C_{1}\rangle\leq 4,$ (2)

where $\langle A_{x}B_{y}C_{z}\rangle$ for every possible $x,y,z\in\\{0,1\\}$
denotes the expectation value of the two-outcome $\\{\pm 1\\}$ measurements
performed by the respective parties on their local constituents. In contrast,
any correlation violating the above inequality ( Minimum Detection
Efficiencies for Loophole-free Genuine Nonlocality Tests) can be characterized
as Svetlichny-type genuine nonlocal.

It is instructive that for the above scenario the parties, sharing a common
light cone (say Bob and Charlie), are free to communicate with each other.
However, such a relaxed causal constraint allows them to establish a bipartite
nonlocality with Alice. With this operational inconsistency, one may further
impose a definite causal order between input-output generation of Bob and
Charlie, which, in effect, allows only a single direction of relevant
communication between them [18, 19]. Any correlation compatible with such a
constraint is referred to be $T_{2}$ local and admits the following
decomposition:

$\displaystyle
P(abc|xyz)=\sum_{\lambda}q_{\lambda}P_{\lambda}^{T_{AB}}(ab|xy)P_{\lambda}(c|z)+$
$\displaystyle\sum_{\mu}q_{\mu}P_{\mu}^{T_{AC}}(ac|xz)P_{\mu}(b|y)+\sum_{\nu}q_{\nu}P_{\nu}^{T_{BC}}(bc|yz)P_{\nu}(a|x)$
(3)

where, the joint term $p_{\lambda}^{T_{AB}}(ab|xy)=p_{\lambda}^{A<B}(ab|xy)$
if the local statistics obtained by Alice is in causal past of Bob and
$p_{\lambda}^{T_{AB}}(ab|xy)=p_{\lambda}^{B<A}(ab|xy)$ when the order is
reversed. All the other bipartite marginals can also be explained in similar
fashion. Notably, in both the cases there are (at most) 1-way signaling. This
refinement seem to offer a potential solution to sidestep the earlier
mentioned issues that might arise within the Svetlichny scenario. Now if a
probability distribution $p(abc|xyz)$ can be written in the form ( Minimum
Detection Efficiencies for Loophole-free Genuine Nonlocality Tests) then it is
said to be $T_{2}$ local otherwise it is called genuinely three-way nonlocal.
In [19], the authors proposed an experimentally testable criteria to verify
the compatibility of a tripartite correlation with such a time-ordered space-
time structure:

$\displaystyle\mathcal{B}_{T_{2}}:=$
$\displaystyle-2\\{P(00|A_{1}B_{1})+P(00|B_{1}C_{1})+P(00|A_{1}C_{1})\\}$
$\displaystyle-P(000|A_{0}B_{0}C_{1})-P(000|A_{0}B_{1}C_{0})$
$\displaystyle-P(000|A_{1}B_{0}C_{0})+2P(000|A_{1}B_{1}C_{0})$
$\displaystyle+2P(000|A_{1}B_{0}C_{1})+2P(000|A_{0}B_{1}C_{1})$
$\displaystyle+2P(000|A_{1}B_{1}C_{1})\leq 0,$ (4)

where, $A_{x},B_{y},C_{z}$ are the dichotomic local observable with possible
outcomes $\\{0,1\\}$. A violation of the inequality (4) certifies the genuine
nonlocality possessed by the correlation.

Minimum detection efficiency (MDE)– The requirement of a loophole-free
experimental test for nonlocality can be classified into two different
aspects: Firstly, from the information theoretic perspective, it is required
to obtain a nonlocal statistics out of a black box promising to generate a
particular form of correlation or state. On the other hand, from the
foundational perspective, one may eager to certify the nonlocal signature of
an operational theory in a device-independent manner. Notably, while the first
one conceives a state-dependent outlook, the latter one asks about the nature
of the theory regardless of any possible preparations. These two different
perspectives demand different forms of experimental sophistication, one of
which is the efficiency of the detectors. However, in the following, we will
see how they are connected via an optimization.

Let us, consider one such Bell-inequality $\mathcal{B}$ for nonlocality test,
which is supposed to be verified on an $N$-partite entangled state $\rho$ with
$i^{\text{th}}$ party performing $m_{i}$ numbers of incompatible measurements
$\mathcal{M}_{i}:=\\{M_{1},\cdots,M_{m_{i}}\\}$. Then the detection efficiency
required for such a test is said to be the cut-off detection efficiency (CDE)
and is denoted as

$\displaystyle\eta^{\mathcal{B}}:=\eta^{\mathcal{B}}(\rho,\\{\mathcal{M}_{k}\\}_{k=1}^{N}).$
(5)

However, the minimum detection efficiency (MDE) for the same Bell-test
$\mathcal{B}$ can be identified as,

$\displaystyle\eta^{\mathcal{B}}_{\min}=\underset{\rho,\\{\mathcal{M}_{k}\\}_{k=1}^{N}}{\text{inf}}\eta^{\mathcal{B}}(\rho,\\{\mathcal{M}_{k}\\}_{k=1}^{N}),$
(6)

where the infimum is over the set of all possible $N$-partite entangled
density matrices acting on the joint Hilbert space $\mathcal{H}$ and
$\\{\mathcal{M}_{k}\\}_{k=1}^{N}$ are the set of all possible $m_{k}$
incompatible measurements for the $k^{\text{th}}$-cite. Notably, CDE is a
quantity of interest for the self-testing scenario, while MDE can be assigned
with the device-independent certification of nonlocality in quantum theory.

Generally, the CDE is a very complicated nonlinear function of both the states
and measurements. Hence, the optimization involved in (6) is highly nontrivial
and the characterization of MDE for an arbitrary Bell-test becomes very
challenging in general. However, motivated by the device-independent
architectures and their implications, in the present work we derive the MDE
for detection of various tripartite genuine nonlocality. In particular, for
the $T_{2}$-type nonlocality test we estimate $\eta_{\min}$ bypassing the
optimization complexities related to CDE, while for the Svetlichny-type test,
we adopt a rigorous numerical optimization.

Furthermore, in a real-life experimental setup, things become more intricate
due to the inevitable existence of noise. If, by $\\{\mathbf{p}\\}$ one
denotes the set of parameters defining the noise, eq.(6) is modified as:

$\eta^{\mathcal{B}}_{\min}(\\{\mathbf{p}\\})=\underset{\rho,\\{\mathcal{M}_{k}\\}_{k=1}^{N}}{\text{inf}}\eta^{\mathcal{B}}(\rho,\\{\mathcal{M}_{k}\\}_{k=1}^{N},\\{\mathbf{p}\\}),$
(7)

Clearly, in the laboratory the efficiencies of the detectors must follow the
relation: $\eta>\eta^{\mathcal{B}}_{\min}(\\{\mathbf{p}\\})$.

MDE for genuine nonlocality.– Let us begin with a stronger causal constraint
where the parties residing in the same light cone are bounded with a definite
causal order to generate their local input-output statistics. As mentioned
earlier, the incompatibility of a correlation in such a scenario can be
certified by the violation of the inequality (4). To establish such a genuine
three-way nonlocal signature of shared quantum correlation, we consider a
situation where the associated parties will perform the local measurements
with inefficient detectors with corresponding efficiencies
$\eta_{A},~{}\eta_{B}\text{ and }\eta_{C}$. The following lemma derives the
minimum requirement for detection efficiencies to violate (4).

######  Lemma 1.

The spatially separated parties would be able to certify genuine three-way
nonlocality in terms of the inequality $\mathcal{B}_{T_{2}}$, when the
following condition holds

$\displaystyle(4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{A}\eta_{C}-\eta_{B}\eta_{C})>0.$

###### Proof.

From now onward, we will use $P^{\prime}(\mathcal{O}|\mathcal{X})$ and
$P\left(\mathcal{O}|\mathcal{X}\right)$ for theoretical and observed
probability respectively, when input variable $\mathcal{X}$ produces output
variable $\mathcal{O}$. Assuming the independence of each of the individual
local detectors the observed probability for each individual can only depend
upon the corresponding theoretical probability and the detection efficiency of
the local detector. Now, if all the parties agrees to assume
$\mathcal{O}\equiv 1$ for the no-click instances, then
$P(\mathcal{O}|{\mathcal{X}})=\eta
P^{\prime}\left(\mathcal{O}|\mathcal{X}\right)$, only when $\mathcal{O}\equiv
0$. This readily implies,

$\displaystyle P(000|A_{1}B_{1}C_{1})$
$\displaystyle=\eta_{A}\eta_{B}\eta_{C}P^{\prime}(000|A_{1}B_{1}C_{1})$
$\displaystyle\leq\eta_{A}\eta_{B}\eta_{C}~{}~{}\underset{X}{\min}~{}P^{\prime}(00|X)$

where, $X\in\\{A_{1}B_{1},B_{1}C_{1},A_{1}C_{1}\\}$ .

Again, $\displaystyle P(000|A_{1}B_{1}C_{0})-P(00|A_{1}B_{1})$
$\displaystyle=\eta_{A}\eta_{B}\eta_{C}P^{\prime}(000|A_{1}B_{1}C_{0})-\eta_{A}\eta_{B}P^{\prime}(00|A_{1}B_{1})$
$\displaystyle\leq\eta_{A}\eta_{B}\eta_{C}\underset{X}{\min}P^{\prime}(00|X)-\eta_{A}\eta_{B}\underset{X}{\min}P^{\prime}(00|X)$

where the last inequality holds since
$P(000|A_{1}B_{1}C_{0})-P(00|A_{1}B_{1})\leq 0$ and
$\eta_{C},\eta_{A},\eta_{B}\leq 1$. Deriving the similar inequalities for
other pair of terms $P(000|A_{1}B_{0}C_{1})-P(00|A_{1}C_{1})$ and
$P(000|A_{0}B_{1}C_{1})-P(00|B_{1}C_{1})$, we can rewrite the violation of
$\mathcal{B}_{T_{2}}$ (4) as

$\displaystyle(4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{A}\eta_{C}-\eta_{B}\eta_{C})~{}~{}\underset{X}{\min}~{}P^{\prime}(00|X)>0$
$\displaystyle\Rightarrow(4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{A}\eta_{C}-\eta_{B}\eta_{C})>0$
(8)

Hence, the above condition among the detector efficiencies, becomes necessary
to exhibit genuine three-way nonlocality. ∎

At this point, a pertinent question naturally arises regarding the sufficiency
of the inequality (8). In other words, whether every triplet of
$\\{\eta_{A},\eta_{B},\eta_{C}\\}$, satisfying
$4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{A}\eta_{C}-\eta_{B}\eta_{C}>0$,
corresponds to CDE for $\mathcal{B}_{T_{2}}$. In the following, we will answer
this question affirmatively.

######  Lemma 2.

There exists a quantum settings with a single parameter $\theta$, which
violates the inequality (4), namely $\mathcal{B}_{T_{2}}$, whenever
$(4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{A}\eta_{C}-\eta_{B}\eta_{C})>0$.

By the single parameter quantum settings, we would like to mean a class of
genuinely entangled states along with two incompatible dichotomic measurements
for each individual party, both parameterized by a single parameter $\theta$.

###### Proof.

Consider the three-qubit state

$\displaystyle\ket{\Psi_{ABC}(\theta)}=k_{\theta}[(\ket{011}+\ket{101}+\ket{110})+\frac{(1-3\cos\theta)}{\sin\theta}\ket{111}]$
(9)

where $k_{\theta}$ is a function of $\theta$, denoting the normalization of
the state. Each party performs an identical set of incompatible measurements
in the following bases,

$\displaystyle X_{0}\equiv\\{\ket{0},\ket{1}\\}$ $\displaystyle
X_{1}\equiv\\{\cos\theta\ket{0}+\sin\theta\ket{1},\sin\theta\ket{0}-\cos\theta\ket{1}\\}$

where $X\in\\{A,B,C\\}$. Such a measurement setting readily implies

$P(000|A_{i}B_{j}C_{k})=0$

where, at least two of $\\{i,j,k\\}$ equals to $0$. On the other hand when two
of $\\{i,j,k\\}$ equals to $1$, probabilities reduce to,

$\displaystyle
P(000|A_{i}B_{j}C_{k})=\eta_{A}\eta_{B}\eta_{C}k_{\theta}^{2}\sin^{4}{\theta},$

and the following marginals become

$\displaystyle
P(00|X_{1}Y_{1})=\eta_{X}\eta_{Y}k_{\theta}^{2}\sin^{4}{\theta}(1+\tan^{2}{\frac{\theta}{2}}),$

where, $X\neq Y\in\\{A,B,C\\}$.

Replacing the above expressions for the imperfect detectors, the inequality
(4) reduces to the following

$k_{\theta}^{2}\sin^{4}{\theta}[4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{B}\eta_{C}-\eta_{A}\eta_{C}\\\
-\tan^{2}{\frac{\theta}{2}}(\eta_{A}\eta_{B}+\eta_{B}\eta_{C}+\eta_{A}\eta_{C})]\leq
0$ (10)

Therefore, to violate the above inequality, we must have,

$\tan^{2}{\frac{\theta}{2}}<\frac{4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{B}\eta_{C}-\eta_{A}\eta_{C}}{\eta_{A}\eta_{B}+\eta_{B}\eta_{C}+\eta_{A}\eta_{C}}$

Therefore, for every possible value of $\\{\eta_{A},\eta_{B},\eta_{C}\\}$,
satisfying the condition
$4\eta_{A}\eta_{B}\eta_{C}-\eta_{A}\eta_{B}-\eta_{B}\eta_{C}-\eta_{A}\eta_{C}>0$
it is possible to find a range of $\theta$ for which a three-way genuine
nonlocality can be certified experimentally. Notably, the inequality on the
detection efficiencies is strict, otherwise one may consider
$\theta=n\pi,~{}n\in\mathbb{Z}$ and the measurements $X_{0}$ and $X_{1}$ will
become compatible.∎

From Lemma (1) and Lemma (2), we can immediately infer the subsequent theorem.

######  Theorem 1.

The MDE’s for $\mathcal{B}_{T_{2}}$ for each of the party satisfy the
following relation

$\displaystyle
4\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{A}\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{B}\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{C}-\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{A}\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{B}$
$\displaystyle-\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{A}\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{C}-\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{B}\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{C}=0.$

As a consequence of Theorem 1 the MDE for two different scenarios can be
concluded. For the case of symmetric detection efficiencies, i.e.
$\eta_{A}=\eta_{B}=\eta_{C}=\eta$, the inequality (4) can be violated if and
only if $\eta>\eta^{\mathcal{B}_{T_{2}}}_{\min}=75\%$. On the other hand, when
a hybrid entanglement between different quantum particles are observed, then
the MDE for each of the particles may demand different experimental
sophistication. In such a scenario, if two of the detectors work perfectly,
i.e., say $\eta_{A}=\eta_{B}=100\%$, then the inequality (4) can be violated
if and only if
$\eta_{C}>\Big{(}\eta^{\mathcal{B}_{T_{2}}}_{\min}\Big{)}_{C}=50\%$.
Therefore, perfect detectors are not imperative in this scenario and the
genuine non-locality can be established even when detectors deviate
significantly from being perfect.

Background Noise.– While the above analysis is restricted to pure entangled
states, in the practical experimental set-up the existence of background noise
becomes unavoidable. The presence of noise in a preparation device is
generally regarded as a junk state, prepared with a significantly low
frequency. Without having any prior knowledge about the noise, a randomization
over all possible junk states can be assumed as noise. In presence of such a
white noise the tripartite state takes the form

$\displaystyle{\rho}_{ABC}=(1-p)|{\Psi}(\theta)\rangle\langle{\Psi}(\theta)|_{ABC}+\frac{p}{8}\mathbb{I},$

where, the state $\ket{\psi(\theta)}$ is same as in Eq. (9). Naturally, the
MDE $\eta_{\min}^{\mathcal{B}_{T2}}(\theta,p)$ (as introduced in Eq. (7))
becomes highly nonlinear function. However, we show that with a reasonable
amount of detection efficiency $0.92\leq\eta^{\mathcal{B}_{T2}}\leq 0.96$, one
can expect tolerance even up to $1.6\%$ background noise (see Fig. LABEL:fig1
for details).

MDE for Svetlichny Nonlocality.– We will now consider a genuine nonlocality
experiment with relaxed causal constraints on involved parties, i.e., by
invoking no-time orderings among their local input-output generation. The
violation of Svetlichny inequality ( Minimum Detection Efficiencies for
Loophole-free Genuine Nonlocality Tests) serves as a prominent benchmark for
illustrating any incompatibility with such a causal-structure and is
identified as a Svetlichny-type genuinely nonlocal correlation. However, the
experimental sophistication required to certify such a violation is pretty
high – almost 97% of CDE for three-particle GHZ state [41]. In the following,
we will first derive an expression for CDE, depending on the statistics
obtained by the individual parties. Thereafter, we numerically estimate the
MDE considering any possible tripartite entangled preparation, which is
significantly lower than the bound obtained in [41].

Let us first express the inequality ( Minimum Detection Efficiencies for
Loophole-free Genuine Nonlocality Tests) in terms of the outcomes
probabilities. Relabeling the outcomes of each measurement $\\{\pm
1\\}\mapsto\\{0,1\\}$, we can rewrite the inequality as,

$\displaystyle 4+8[m_{010}+m_{101}-(m_{000}+m_{001}+m_{011}+m_{100}+m_{110}$
$\displaystyle+m_{111})]+8[q_{00}+q_{11}+r_{01}+r_{10}+s_{00}+s_{11}]$
$\displaystyle-4[a_{0}+a_{1}+b_{0}+b_{1}+c_{0}+c_{1}]\leq 4$ (11)
$\displaystyle\text{where,
}m_{ijk}=P(000|A_{i}B_{j}C_{k}),q_{ij}=P(00|A_{i}B_{j}),$ $\displaystyle
r_{jk}=P(00|B_{j}C_{k}),s_{ik}=P(00|A_{i}C_{k}),a_{i}=P(0|A_{i}),$
$\displaystyle b_{j}=P(0|B_{j}),c_{k}=P(0|C_{k}),~{}\forall
i,j,k\in\\{0,1\\}.$

######  Theorem 2.

For the Svetlichny-type nonlocality test, with symmetric detectors, the CDE
becomes
$\eta^{\text{S}}=\frac{\sqrt{\beta^{2}+4|\alpha|\gamma}-\mathbf{sgn}(\alpha)\times\beta}{2|\alpha|}$,
where

$\displaystyle\beta:=2\sum_{i,j,k}[\overline{(i\oplus j)}q_{ij}+(j\oplus
k)r_{ij}+\overline{(i\oplus k)}s_{ik}],$
$\displaystyle\alpha:=2\sum_{i,j,k}(-1)^{(\bar{i}.\bar{j}+j.k+i.\bar{k})}m_{ijk},~{}\gamma:=\sum_{i,j,k}(a_{i}+b_{j}+c_{k})$
$\displaystyle\text{ and }\mathbf{sgn}(\alpha)=\pm 1,\text{ for
}\alpha>0\text{ and }\alpha<0\text{ respectively}.$

###### Proof.

The Svetlichny’s inequality, in the form ( Minimum Detection Efficiencies for
Loophole-free Genuine Nonlocality Tests), can be violated with a properly
chosen quantum setting, whenever

$\displaystyle 8[m_{010}+m_{101}-(m_{000}+m_{001}+m_{011}+m_{100}+m_{110}$
$\displaystyle+m_{111})]+8[q_{00}+q_{11}+r_{01}+r_{10}+s_{00}+s_{11}]$
$\displaystyle-4[a_{0}+a_{1}+b_{0}+b_{1}+c_{0}+c_{1}]>0$

Now, consider the situation where all three imperfect detectors are of the
same detection efficiency $\eta$. Then replacing all the theoretical
probabilities with the observed probabilities, we can rewrite the above
inequality as,

$\displaystyle\sum_{i,j,k~{}\in\\{0,1\\}}\bigg{[}(-1)^{(\bar{i}.\bar{j}+j.k+i.\bar{k})}2\eta^{3}m_{ijk}+2\eta^{2}\Big{\\{}\overline{(i\oplus
j)}q_{ij}$ $\displaystyle({j\oplus k})r_{jk}+\overline{(i\oplus
k)}s_{ik}\Big{\\}}-\eta\Big{(}a_{i}+b_{j}+c_{k}\Big{)}\bigg{]}>0$

The detection efficiency being always positive, above inequality simplifies
to,

$\alpha\eta^{2}+\beta\eta-\gamma>0$

where, $\alpha,~{}\beta\text{ and }\gamma$ are same as defined earlier.

Note that, for any quantum settings that exhibit Svetlichny-type nonlocality,
must satisfy the above inequality at least for $\eta=1$, i.e., for the perfect
detector. Hence $\alpha+\beta+\gamma>0$. This, together with the fact that
$\beta\geq 0$ and $\gamma\geq 0$, provides
$\eta=\frac{\sqrt{\beta^{2}+4|\alpha|\gamma}-\mathbf{sgn}(\alpha)\times\beta}{2|\alpha|}$
saturating the above inequality. Note that, the $\mathbf{sgn}(\alpha)$
function incorporates that $\alpha$ can be both positive or, negative.
Finally, noting that all the parameters $\alpha,\beta,\gamma$ are the
parameters generated from the quantum settings, i.e., state and measurements,
we can identify $\eta$ as $\eta^{\text{S}}$ – the CDE for Svetlichny-type
nonlocality test. ∎

Optimizing over all possible quantum settings numerically, it can be shown
that the $\eta_{\text{min}}^{S}\simeq 88.1\%$ is sufficient for demonstrating
Svetlichny-type nonlocality.

Discussion– In summary, we have estimated the minimum detection efficiency
required for loophole free test of various forms of genuine nonlocality
exhibited by multipartite quantum systems. In particular, taking the
operational framework into account we have investigated both the time-ordered
($T_{2}$) genuine nonlocality, as well as the traditional Svetlichny-type
genuine nonlocality tests. To this goal, we have primarily introduced the
notion of cut-off detection efficiency which explicitly depends upon the
quantum settings used to demonstrate a particular Bell nonlocality. Then
systematically we have reached to the minimum detection efficiency for the
concerned nonlocality experiment, optimizing over all possible quantum states
and measurement settings. Within this framework, our study reveals that, to
demonstrate loophole-free $T_{2}$ nonlocality in symmetric cases, a detection
efficiency of $75\%$ at each site is both necessary and sufficient. This
requirement can be relaxed to $50\%$ when any two out of the three parties
possess perfect detection efficiency. Interestingly, our results suggest that
to achieve the MDE for violating $T_{2}$ inequality is more feasible in the
scenarios where the correlations nearly breach the inequality, instead of
demonstrating the maximal violations. Further, we have shown that in presence
of a significantly low white-noise, the range of quantum settings showing
$T_{2}$ nonlocality sharply decreases. Nevertheless, for the given inefficient
detectors and a permissible noise limit, our analysis characterizes the range
of quantum settings viable to exhibit genuine nonlocality in experiments. On
the other hand, we have also established that a detection efficiency of
$88.1\%$ is deemed sufficient for demonstrating Svetlichny nonlocality, which
is well below the previously estimated one.

Though Svetlichny inequality and $T_{2}$ inequality, both are distinct facets
of the set of the $T_{2}$ correlations, the set of all $T_{2}$ local
correlations is a strict subset of that of the Svetlichny-local correlations.
This, in turn, admits that the correlations violating Svetlichny inequality
are stronger than those violating the $T_{2}$ inequality and hence demand more
sophistication in the involved experimental set-up. Thus, as a consequence of
our results, the investigation and comparison of MDE across distinct nonlocal
classes has emerged as a compelling avenue for future research.

Acknowledgements.–SRC acknowledges support from University Grants Commission,
India (reference no. 211610113404). RA acknowledges funding and support from
ISI DCSW Project No. PU/506/PL-MISC/521. We acknowledge fruitful discussions
with Guruprasad Kar, Manik Banik, Asutosh Rai and Amit Mukherjee.

## References

  * [1] V. Scarani; The device-independent outlook on quantum physics (lecture notes on the power of Bell’s theorem), Acta Physica Slovaca 62, 347 (2012).
  * [2] A. K. Ekert; Quantum cryptography based on Bell’s theorem, Phys. Rev. Lett. 67, 661 (1991).
  * [3] J. Barrett, L. Hardy, and A. Kent; No Signaling and Quantum Key Distribution, Phys. Rev. Lett. 95, 010503 (2005).
  * [4] A. Acín, N. Gisin, and L. Masanes; From Bell’s Theorem to Secure Quantum Key Distribution, Phys. Rev. Lett. 97, 120405 (2006).
  * [5] S. Pironio, A. Acín, S. Massar, A. Boyer de la Giroday, D. N. Matsukevich, P. Maunz, S. Olmschenk, D. Hayes, L. Luo, T. A. Manning, and C. Monroe; Random numbers certified by Bell’s theorem, Nature 464, 1021 (2010).
  * [6] R. Colbeck and R. Renner; Free randomness can be amplified, Nature Phys 8, 450 (2012).
  * [7] A. Chaturvedi and M. Banik; Measurement-device-independent randomness from local entangled states, Europhys. Lett. 112, 30003 (2015).
  * [8] A. Mukherjee, A. Roy, S. S. Bhattacharya, S. Das, Md. R. Gazi, and M. Banik; Hardy’s test as a device-independent dimension witness, Phys. Rev. A 92, 022302 (2015).
  * [9] N. Brunner and N. Linden; Connection between Bell nonlocality and Bayesian game theory, Nat. Commun. 4, 2057 (2013).
  * [10] A. Pappa, N. Kumar, T. Lawson, M. Santha, S. Zhang, E. Diamanti, and I. Kerenidis; Nonlocality and Conflicting Interest Games, Phys. Rev. Lett. 114, 020401 (2015).
  * [11] H. Buhrman, R. Cleve, S. Massar, and R. de Wolf; Nonlocality and communication complexity; Rev. Mod. Phys. 82, 665 (2010)
  * [12] A. Roy, A. Mukherjee, T. Guha, S. Ghosh, S. S. Bhattacharya, and M. Banik; Nonlocal correlations: Fair and unfair strategies in Bayesian games, Phys. Rev. A 94, 032120 (2016).
  * [13] M. Banik, S. S. Bhattacharya, N. Ganguly, T. Guha, A. Mukherjee, A. Rai, and A. Roy; Two-Qubit Pure Entanglement as Optimal Social Welfare Resource in Bayesian Game, Quantum 3, 185 (2019).
  * [14] J. S. Bell; On the Einstein Podolsky Rosen paradox, Physics Physique Fizika 1, 195 (1964).
  * [15] J. S. Bell; On the Problem of Hidden Variables in Quantum Mechanics, Rev. Mod. Phys. 38, 447 (1966).
  * [16] J. F. Clauser, M. A. Horne, A. Shimony, and R. A. Holt, Phys. Rev. Lett. 23, 880 (1969).
  * [17] G. Svetlichny; Distinguishing three-body from two-body nonseparability by a Bell-type inequality, Phys. Rev. D 35 3066 (1987)
  * [18] R. Gallego, L. E. Würflinger, A. Acín, and M. Navascués; Operational Framework for Nonlocality, Phys. Rev. Lett. 109, 070401 (2012)
  * [19] J-D Bancal, J. Barrett, N. Gisin, and S. Pironio; Definitions of multipartite nonlocality, Phys. Rev. A 88, 014102 (2013)
  * [20] Sagnik Dutta, Amit Mukherjee and Manik Banik; Operational characterization of multipartite nonlocal correlations, Phys. Rev. A 102, 052218(2020)
  * [21] Qing Chen, Sixia Yu, Chengjie Zhang, C.H. Lai, and C.H. Oh; Test of Genuine Multipartite Nonlocality without Inequalities; Phys. Rev. Lett. 112, 140404
  * [22] Timo Holz, Hermann Kampermann, and Dagmar Bruß; Genuine multipartite Bell inequality for device-independent conference key agreement; Phys. Rev. Research 2, 023251(2020)
  * [23] Jean-Daniel Bancal, Nicolas Gisin, Yeong-Cherng Liang, and Stefano Pironio; Device-Independent Witnesses of Genuine Multipartite Entanglement; Phys. Rev. Lett. 106, 250404 (2011)
  * [24] Jean-Daniel Bancal; Device-Independent Witnesses of Genuine Multipartite Entanglement; 2014
  * [25] Y-C Liang, D. Rosset, J-D Bancal, G. Pütz, TJ Barnea, and N. Gisin; Family of Bell-like Inequalities as Device-Independent Witnesses for Entanglement Depth; Phys. Rev. Lett. 114, 190401(2014)
  * [26] Randomness versus nonlocality in the Mermin-Bell experiment with three parties; Erik Woodhead, Boris Bourdoncle, Antonio Acín;  Quantum 2, 82 (2018)
  * [27] L. Aolita, R. Gallego, A. Cabello, and A. Acín; Fully Nonlocal, Monogamous, and Random Genuinely Multipartite Quantum Correlations; Phys. Rev. Lett. 108, 100401 ,(2012).
  * [28] P.M.Pearle, Hidden-Variable Example Based upon Data Rejection; Phys. Rev. D 2, 1418 (1970)
  * [29] A. Garg and N. D. Mermin; Detector inefficiencies in the Einstein-Podolsky-Rosen experiment; Phys. Rev. D 35, 3831 (1987)
  * [30] Jan-Åke Larsson; Bell’s inequality and detector inefficiency; Phys. Rev. A 57, 3304 (1998)
  * [31] P.H.Ebarhard; Background level and counter efficiencies required for a loophole-free Einstein-Podolsky-Rosen experiment; Phys. Rev. A 47, R747(R) (1993)
  * [32] A. Cabello and J-Å Larsson; Minimum Detection Efficiency for a Loophole-Free Atom-Photon Bell Experiment; Phys. Rev. Lett. 98, 220402 (2007)
  * [33] N. Brunner, N. Gisin, V. Scarani, and C. Simon; Detection Loophole in Asymmetric Bell Experiments; Phys. Rev. Lett. 98, 220403(2007)
  * [34] B. B. Blinov, D. L. Moehring, L.- M. Duan and C. Monroe; Observation of entanglement between a single trapped atom and a single photon,Nature. 428, 153–157 (2004)
  * [35] Hanson et. al; Loophole-free Bell inequality violation using electron spins separated by 1.3 kilometres, Nature, 526, 682(2015)
  * [36] Zeilinger et. al; Significant-Loophole-Free Test of Bell’s Theorem with Entangled Photons, Phys. Rev. Lett. 115, 250401(2015)
  * [37] Lynden K. Shalm et al; Strong Loophole-Free Test of Local Realism, Phys. Rev. Lett. 115, 250402(2015)
  * [38] Wallraff. et. al; Loophole-free Bell inequality violation with superconducting circuits, Nature, 617, 265(2023)
  * [39] J.-Å. Larsson and J. Semitecolos; Strict detector-efficiency bounds for n-site Clauser-Horne inequalities; Phys. Rev. A 63, 022117 (2001)
  * [40] A. Cabello, D. Rodríguez, and I. Villanueva; Necessary and Sufficient Detection Efficiency for the Mermin Inequalities, Phys. Rev. Lett. 101, 120402 (2008)
  * [41] Y. Xiang, H-X Wang, and F-Y. Hong; Detection efficiency in the loophole-free violation of Svetlichny’s inequality, Phys. Rev. A 86, 034102 (2012)
  * [42] Valentin Gebhart and Augusto Smerzi; Coincidence postselection for genuine multipartite nonlocality: Causal diagrams and threshold efficiencies, Phys. Rev. A 106, 062202(2022)
  * [43] S. B. Ghosh, S. Roy Chowdhury, G. Kar, A. Roy, T. Guha, M. Banik; Quantum Nonlocality: Multi-copy Resource Inter-convertibility & Their Asymptotic Inequivalence, arXiv:2310.16386v1.
  * [44] Hyunseok Jeong, Alessandro Zavatta, Minsu Kang, Seung-Woo Lee, Luca S. Costanzo, Samuele Grandi, Timothy C. Ralph, Marco Bellini; Generation of hybrid entanglement of light, nphoton., 8, 564–569 (2014)
  * [45] Jianming Wen, Irina Novikova, Chen Qian, Chuanwei Zhang, hengwang Du; Hybrid Entanglement between Optical Discrete Polarizations and Continuous Quadrature Variables, Photonics 2021, 8(12), 552
  * [46] Armin Feist, Guanhao Huang, Germaine Arend, Yujia Yang, Jan-Wilke Henke, Arslan Sajid Raja, F. Jasmin Kappert, Rui Ning Wang, Hugo Lourenço-Martins, Zheru Qiu, Junqiu Liu, Ofer Kfir, Tobias J. Kippenberg, Claus Ropers; Cavity-mediated electron-photon pairs, Science377,777-780(2022)
  * [47] Mingjian He, Robert Malaney; R. Teleportation of hybrid entangled states with continuous-variable entanglement, Sci Rep 12, 17169 (2022)


# Learning To Traverse Latent Spaces for
Musical Score Inpainting

###### Abstract

Music Inpainting is the task of filling in missing or lost information in a
piece of music. We investigate this task from an interactive music creation
perspective. To this end, a novel deep learning-based approach for musical
score inpainting is proposed. The designed model takes both past and future
musical context into account and is capable of suggesting ways to connect them
in a musically meaningful manner. To achieve this, we leverage the
representational power of the latent space of a Variational Auto-Encoder and
train a Recurrent Neural Network which learns to traverse this latent space
conditioned on the past and future musical contexts. Consequently, the
designed model is capable of generating several measures of music to connect
two musical excerpts. The capabilities and performance of the model are
showcased by comparison with competitive baselines using several objective and
subjective evaluation methods. The results show that the model generates
meaningful inpaintings and can be used in interactive music creation
applications. Overall, the method demonstrates the merit of learning complex
trajectories in the latent spaces of deep generative models.

##  1 Introduction

Over the last decade, machine learning techniques have emerged as the tool of
choice for the design of symbolic music generation models [1] with deep
learning being the most widely used [2]. Deep generative models have been
successfully applied to several different music generation tasks, e.g.,
monophonic music generation [3, 4, 5], polyphonic music generation [6, 7] and
creating musical renditions with expressive timing and dynamics [8, 9].
However, most of these models assume sequential generation of music, i.e, the
generated music depends only on the music that has preceded it. In other
words, the models rely only on the past musical context. This approach does
not align with typical human compositional practices which are often iterative
and non-sequential in nature. In addition, the sequential generation paradigm
places severe limitations on the degree of interactivity allowed by these
models [10, 11]. Once generated, there is no way to tweak specific parts of
the generation so as to conform to users’ aesthetic sensibilities or
compositional requirements.

Figure 1: Musical Score Inpainting task schematic. A generative model needs to
take past and future musical contexts into account to generate a sequence that
can connect them in a musically meaningful manner.

In this paper, we seek to address these problems by incorporating future
musical context into the generation process. Specifically, the task is to
train models to fill in missing information in musical scores, duly taking
into account the complete musical context — both past and future. In essence,
this is similar to inpainting where the objective is to reconstruct missing or
degraded parts of any kind of media [12]. For music, inpainting has been
traditionally used for restoration purposes [13] or to remove unwanted
artifacts such as clipping [14, 15] and packet loss [16]. However, we
investigate models for Musical Score Inpainting (see Figure 1 and Section 3.1)
as tools for music creation which can aid people in  (i) getting new musical
ideas based on specific styles,  (ii) joining different musical sections
together, and  (iii) modifying or extending solos.  In addition, such models
can allow interactive music generation by enabling users to change the musical
context and get new suggestions based on the updated context.

Our main technical contribution is a novel approach for musical score
inpainting which relies on latent representation-based deep generative models.
These models are trained to compress information from high-dimensional spaces,
e.g., the space of all $1$-bar melodies, to low-dimensional latent spaces.
While these latent spaces have been shown to be able to encode hidden
attributes of musical data (see Section 2.3), the primary form of interaction
with latent spaces has been using simple operations such as attribute vectors
[17, 18] or linear interpolations [19, 20]. Using the proposed method (see
Section 3), we demonstrate that Recurrent Neural Networks (RNNs) can be
trained using latent embeddings to learn complex trajectories in the latent
space. This, in turn, is used to predict how to fill in missing measures in a
piece of symbolic music. Our secondary contributions are:  (i) a stochastic
training scheme which helps model training and generalization (see Section
3.4), and  (ii) a novel data encoding scheme using uneven tick durations that
allows encoding triplets without substantial increase in sequence length (see
Section 3.5).  The effectiveness of the proposed method is demonstrated using
several objective and subjective evaluation methods in Section 4.

##  2 Related Work

###  2.1 Audio & Music Inpainting

The first applications of audio inpainting methods were restoration-oriented
[13, 14, 21, 16, 22] using different methods such as matrix factorization
[14], non-local similarity measures [22] and audio similarity graphs [16].
While these techniques have been useful for audio-based tasks, they are not
easily extendable to symbolic music.

For inpainting in the symbolic domain, the early attempts were based on Markov
Chain Monte Carlo (MCMC) methods which allowed users to specify certain
constraints, e.g., which notes to generate and which to retain [23, 24].
Another approach, proposed by Lattner et al., used iterative gradient descent
to force the output of a deep generative model to conform to a specified
structural plan [25]. However, methods based on MCMC (which rely on repeated
sampling), and those using iterative gradient descent are slow during
inference time and hence unsuitable for interactive applications. More
recently, Hadjeres et al. proposed the AnticipationRNN framework [10] which
used a pair of stacked RNNs to enforce user-defined constraints during
inference. This allowed selective regeneration of specific parts of the music
(generated or otherwise) using only two forward passes through the RNN-pair
and enabled real-time generations.

###  2.2 Variational Auto-Encoders

The Variational Auto-Encoder (VAE) [26] is a type of generative model which
uses an auto-encoding [27] framework; during training, the model is forced to
reconstruct its input. The architecture comprises an encoder and a decoder.
The encoder learns to map real data-points $\mathbf{x}$ from a high-
dimensional data-space $X$ to points in a low-dimensional space $Z$ which is
referred to as the latent space. The decoder learns to map the latent vectors
back to the data-space. VAEs treat the latent vector as a random variable and
model the generative process as a sequence of sampling operations:
$\mathbf{z}\sim p(\mathbf{z})$, and $\mathbf{x}\sim p(\mathbf{x}|\mathbf{z})$,
where $p(\mathbf{z})$ is a prior distribution over the latent space, and
$p(\mathbf{x}|\mathbf{z})$ is the conditional pdf. Variational inference [28]
is used to approximate the posterior by minimizing the KL-divergence [29]
between the approximate posterior $q(\mathbf{z}|\mathbf{x})$ and the true
posterior $p(\mathbf{z}|\mathbf{x})$ by maximizing the evidence lower bound
(ELBO) [26]. The training ensures that the reconstruction accuracy is
maximized and realistic samples are generated when latent vectors are sampled
using the prior $p(\mathbf{z})$.

###  2.3 Leveraging Latent Spaces for Music Generation

Latent representation-based models such as VAEs have been found to be quite
useful for several music generation tasks. Bretan et al. used the latent
representation of an auto-encoder-based model to generate musical phrases
[30]. Lattner et al. forced the latent space of a gated auto-encoder to learn
pitch interval-based representations which improved the performance of
predictive models of music [31, 32]. Latent spaces of music generation models
have also been used to explicitly encode and control musical attributes [20,
33, 3, 34], inter-track dependencies [35] and musical genre [36]. These
studies show that trained latent spaces are able to encode hidden attributes
of musical data which can be leveraged for different music generation tasks.
However, latent space traversals have been relying on simpler methods such as
attribute vectors [17, 18] or linear interpolations [19, 20].

##  3 Method

###  3.1 Problem Statement

We define the score inpainting problem as follows: given a past musical
context $\mathcal{C}_{\mathrm{p}}$ and a future musical context
$\mathcal{C}_{\mathrm{f}}$, the modeling task is to generate an inpainted
sequence $\mathcal{C}_{\mathrm{i}}$ which can connect
$\mathcal{C}_{\mathrm{p}}$ and $\mathcal{C}_{\mathrm{f}}$ in a musically
meaningful manner. In other words, the model should be trained to maximize the
likelihood
$p(\mathcal{C}_{\mathrm{i}}~{}|~{}\mathcal{C}_{\mathrm{p}},\mathcal{C}_{\mathrm{f}})$.
Without much loss of generality, we assume that $\mathcal{C}_{\mathrm{p}}$,
$\mathcal{C}_{\mathrm{f}}$, and $\mathcal{C}_{\mathrm{i}}$ comprise of
$n_{\mathrm{p}}$, $n_{\mathrm{f}}$, and $n_{\mathrm{i}}$ measures of music,
respectively.

Figure 2: Schematic of the proposed approach. The pre-trained MeasureVAE
encoder is used to convert the past and future context sequences
($\mathcal{C}_{\mathrm{p}}$ and $\mathcal{C}_{\mathrm{f}}$) into their
respective latent vector sequences ($\mathcal{Z}_{\mathrm{p}}$ and
$\mathcal{Z}_{\mathrm{f}}$). The LatentRNN learns to traverse the latent space
of MeasureVAE to output a latent vector sequence $\mathcal{Z}_{\mathrm{i}}$
which is passed through the pre-trained decoder to output the inpainted
musical sequence $\mathcal{C}_{\mathrm{i}}$.

###  3.2 Approach

The key motivation behind the proposed method is that the latent embeddings of
deep generative models of music encode hidden attributes of music which can be
leveraged to perform inpainting. Firstly, we train a VAE-model, referred to as
MeasureVAE, to reconstruct single measures of music, i.e., the latent vectors
of this model $\mathbf{z}\in Z$ map to individual measures of music. Once
trained, the encoder of this model can be used to process sequences
$\mathcal{C}_{\mathrm{p}}$ and $\mathcal{C}_{\mathrm{f}}$ and output
corresponding latent vector sequences $\mathcal{Z}_{\mathrm{p}}$ and
$\mathcal{Z}_{\mathrm{f}}$. Secondly, we train an RNN-based model, referred to
as LatentRNN, to take as input the past and future latent vector sequences
($\mathcal{Z}_{p}$ and $\mathcal{Z}_{\mathrm{f}}$) and output a third latent
vector sequence $\mathcal{Z}_{\mathrm{i}}$ which can be passed through the
decoder of MeasureVAE to obtain $\mathcal{C}_{\mathrm{i}}$.

Effectively, the LatentRNN model learns to traverse the latent space of the
MeasureVAE model so as to connect the provided contexts in a musically
meaningful manner. The inference is fast since it only requires forward passes
through the two models. This overall approach is shown in Figure 2. We call
this joint architecture InpaintNet. While we restrict ourselves to $4/4$
monophonic melodic sequences in this paper, the approach can be extended to
other time signatures and polyphonic sequences as well. The individual model
architectures are discussed next.

Figure 3: MeasureVAE schematic. Individual components of the encoder and
decoder are shown below the main blocks (dotted arrows indicate data flow
within the individual components). $\mathbf{z}$ denotes the latent vector and
$\hat{\mathbf{x}}$ denotes the reconstructed measure.

###  3.3 Model Architectures

####  3.3.1 MeasureVAE

The MeasureVAE architecture (see Figure 3) is loosely based on the
hierarchical recurrent MusicVAE architecture [3] which proved successful in
modeling individual measures of music.

The encoder consists of a learnable embedding layer (operating on tick-level)
followed by a bi-directional RNN [37]. The concatenated hidden state from both
directions of the RNN is then passed through two identical parallel linear
stacks to obtain the mean $\mu$ and variance $\sigma$ which are used to sample
the latent vector $\mathbf{z}$ via
$\mathbf{z}\sim\mathcal{N}(\mu,\sigma^{2})$.

The decoder follows a hierarchical structure where the sampled latent vector
$\mathbf{z}$ is used to initialize the hidden state of a beat-RNN which is
unrolled $b$ times (where $b$ is the number of beats in a measure). The output
at each step of the beat-RNN is passed through a linear stack before being
used to initialize the hidden state of a tick-RNN which is unrolled $t$ times
(where $t$ is the number of events/ticks in a beat). The outputs of the tick-
RNN are individually passed through a second linear stack which maps them back
to the data-space. The hierarchical architecture mitigates the auto-regressive
nature of the RNN and forces the decoder to use the latent vector more
efficiently (as advocated in [3]).

Figure 4: LatentRNN schematic. The Past-Context and Future-Context-RNNs encode
$\mathcal{Z}_{\mathrm{p}}$ and $\mathcal{Z}_{\mathrm{f}}$, respectively. The
Generation-RNN initialized using a concatenation of context-RNNs embeddings is
unrolled $n_{\mathrm{i}}$ times to get $\mathcal{Z}_{\mathrm{i}}$.

####  3.3.2 LatentRNN

The LatentRNN model (see Figure 4) consists of $3$ sub-components. There are
$2$ identical bi-directional RNNs, referred to as Past-Context-RNN and Future-
Context-RNN, which process the latent vector sequences for the past and future
contexts ($\mathcal{Z}_{\mathrm{p}}$ and $\mathcal{Z}_{\mathrm{f}}$),
respectively. These are unrolled for $n_{\mathrm{p}}$ and $n_{\mathrm{f}}$
times in order to encode the context sequences, respectively. The final hidden
states of the two context-RNNs are concatenated and then used to initialize
the hidden state of a third RNN, referred to as the Generation-RNN, which is
unrolled $n_{\mathrm{i}}$ times. The outputs of the Generation-RNN are passed
through a linear stack to obtain $n_{\mathrm{i}}$ latent vectors corresponding
to the inpainted measures.

The hyper-parameters for the model configurations are chosen based on initial
experiments and are provided in Table 1. For the RNN layers in both models,
Gated Recurrent Units (GRU) [38] are used.

Measure VAE
---
Embedding Layer | i=dict size, o=10
EncoderRNN | n=2, i=10, h=512, d=0.5
| Linear Stack 1
---
Linear Stack 2
i=1024, o=256, n=2, non-linearity=SELU
BeatRNN | n=2, i=1, h=512, d=0.5
TickRNN | n=2, i=522, h=512, d=0.5
Linear Stack 3 | i=512, o=1024, n=1, non-linearity=ReLU
Linear Stack 4 | i=512, o=dict size, n=1, non-linearity=ReLU
Latent RNN
| Past-Context-RNN
---
Future-Context-RNN
n=2, i=256, h=512, d=0.5
Generation RNN | n=2, i=1, h=1024, d=0.5
Linear Stack | i=2048, o=256, n=1, non-linearity=None
Table 1: Table showing configurations of both models. n: Number of Layers, i:
Input Size, o: Output Size, h: Hidden Size, d: Dropout Probability, SELU:
Scaled Exponential Linear Unit [39], ReLU: Rectifier Linear Unit

###  3.4 Stochastic Training Scheme

We propose a novel stochastic training scheme for training the model. For each
training batch, the number of measures to be inpainted $n_{\mathrm{i}}$ and
the number of measures in the past context $n_{\mathrm{p}}$ are randomly
sampled from a uniform distribution. Thus, the number of measures in the
future context becomes $n_{\mathrm{f}}=N-n_{\mathrm{i}}-n_{\mathrm{p}}$, where
$N$ is the total number of measures in each sequence of the training batch.
Using these, the input sequences are split into past, future and target
sequences and the model is trained to predict the target sequence given the
past and future context sequences. This stochastic training scheme ensures
that the model learns to deal with variable length contexts and can perform
inpaintings at arbitrary locations.

###  3.5 Data Encoding Scheme

We use a variant of the encoding scheme proposed by Hadjeres et al. [24] for
our data representation. The original encoding scheme quantizes time uniformly
using the sixteenth note as the smallest sub-division. For each sub-division
or tick, the note which starts on that tick is represented by a token
corresponding to the note name. If no note starts on a tick, a special
continuation symbol ‘__’ is used to denote that the previous note is held.
Rest is considered as a note and has a special token. The main advantages of
this encoding scheme are  (i) it uses only a single sequence of tokens, and
(ii) uses real note names (e.g., separate tokens for A# and Bb) which allows
generation of readable sheet music.

However, a limitation of using the sixteenth note as the smallest sub-division
is that it cannot encode triplets. The naive approach of evenly subdividing
the sixteenth note divisions to encode triplets increases the sequence length
a factor of $3$ which can make the sequence modeling task harder. To mitigate
this limitation, we propose a novel uneven subdivision scheme. Each beat is
divided into $6$ uneven ticks (shown in Figure 5). This allows encoding
triplets while only increasing the sequence length by a factor of $1.5$.
Consequently, each $4/4$ time signature measure is a sequence of $24$ tokens.

##  4 Experiments

The proposed method is compared with two baseline methods (see Section 4.1)
using a dataset of monophonic folk melodies in the Scottish and Irish style
taken from the Session website [5]. For the purposes of this work, only
melodies with $4/4$ time signature in which the shortest note is greater than
or equal to the sixteenth note are considered resulting in approx. $21000$
melodies. Implementation details and source code are available
online.111https://github.com/ashispati/InpaintNet

Figure 5: Figure showing the data representation. The token string on the
bottom demonstrates the encoding scheme for the measure displayed on the top-
left. Top-right shows the proposed uneven tick-duration scheme for each beat.

###  4.1 Baseline

The performance of the proposed method is compared with the AnticipationRNN
model proposed by Hadjeres et al. [10]. This model, referred to as Base-ARNN,
uses a stack of $2$ LSTM-based [40] RNN layers. Each of the $2$ RNNs comprises
of $2$ layers with a hidden size of $256$. In addition to the note-sequence
tokens, this model also uses additional metadata information, i.e., tokens to
indicate beat and down-beat locations as part of the user-defined constraints.
For more details, the readers are directed to [10].

The original model operates on tick-level sequences and inpainting locations
are specified in terms of individual tick locations. Hence, the inpainting
locations may or may not be contiguous. In order to make a fair comparison, a
second variant of the AnticipationRNN model is considered, referred to as Reg-
ARNN, where the stochastic training scheme from Section 3.4 is used instead.

###  4.2 Training Configuration

The MeasureVAE model was pre-trained using single measures following the
standard VAE optimization equation [26] with the $\beta$-weighting scheme [41,
42]. In order to prioritize high reconstruction accuracy, a low value of
$\beta=$ $1$e$-3$ was used. Pre-training was done for $30$ epochs resulting in
a reconstruction accuracy of approx. $99\%$. While this seems to be better
than results in [3], we attribute this to the shorter duration of generation
(single measures) and the differences in datasets and data encoding.
MeasureVAE parameters were frozen after pre-training and no gradient-based
updates were performed on these parameters during the InpaintNet model
training.

The Adam algorithm [43] was used for model training, with a learning rate of
$1$e$-3$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=1$e$-8$. To ensure
consistency, all models were trained for $100$ epochs (with early-stopping)
with the same batch-size using a sub-sequence length of $16$ measures ($384$
ticks). For the InpaintNet and Reg-ARNN models, the number of measures to be
inpainted and the number of past measures were randomly selected:
$n_{\mathrm{i}}\in[2,6]$, $n_{\mathrm{p}}\in[1,16-1-n_{\mathrm{i}}]$. This
ensured that past and future contexts each contain at least $1$ measure. For
the baseline models, teacher-forcing was used with a probability of $0.5$.

###  4.3 Predictions on Test Data

Two experiments were conducted to evaluate the predictive power of the models.

The first experiment considered the average token-wise negative log-likelihood
(NLL) on a held-out test set. The results (see first 3 rows of Table 2)
indicate that our proposed model outperforms both baselines, showing an
improvement of approx. $25\%$ in the NLL over the Reg-ARNN model and approx.
$55\%$ over the Base-ARNN model.

Model Variant | Test NLL
---|---
Base-ARNN | 0.662
Reg-ARNN | 0.402
InpaintNet (Our Method) | 0.300
PastInpaintNet | 0.643
FutureInpaintNet | 0.481
Table 2: Table showing the average token-wise NLL (nats/token) on the held-out
test set (lower is better). InpaintNet outperforms both baselines. The last
two rows show the results for the ablation models described in Section 4.4.

The next experiment compared the models by varying the number of measures to
be inpainted. Figure 6 shows the average token-wise NLL when $n_{\mathrm{i}}$
was increased from $2$ to $8$. Again, our proposed model outperforms both
baselines. It should be noted that since the sub-sequence length is constant
at $16$ measures, increasing $n_{\mathrm{i}}$ means that the available context
is reduced. Thus, there is an expected drop in the performance with increasing
$n_{\mathrm{i}}$ as the models are forced to make longer predictions with less
contextual information. However, the InpaintNet model performs better even
when forced to predict beyond the training limit of $6$ measures.

###  4.4 Ablations Studies

In order to further ascertain the efficiency of the proposed approach,
ablation studies were conducted to evaluate the benefit of adding past and
future context information. Specifically, we trained two variants of the
InpaintNet model which relied on only one type of contextual information. The
first model, referred to as PastInpaintNet only considered the past context
$\mathcal{C}_{\mathrm{p}}$ as input whereas the second model, referred to as
FutureInpaintNet considered only the future context
$\mathcal{C}_{\mathrm{f}}$. The last two rows of Table 2 summarize the
performance of these ablation models. It is clear that both past and future
contexts are important for the modeling process. In addition, we also tried
training a variant of the InpaintNet model with an untrained (randomly
initialized) MeasureVAE model. This model failed to train properly achieving
an NLL of approx. $1.33$. This indicates that a structured latent space where
latent vectors are trained to encode hidden data attributes is important for
training the LatentRNN model.

Figure 6: Figure showing token-wise NLL (nats/token) for different number of
inpainted measures on the held-out test set (lower is better). InpaintNet
outperforms both baselines. Models were trained to predict only $2$ to $6$
measures.

###  4.5 Qualitative Analysis

Figure 7: Figure showing the inpaintings generated by different models for the
same context. From top to bottom — a.: Base-ARNN, b.: Reg-ARNN, c.:
InpaintNet, d.: Original Melody.

Considering that we are primarily interested in the aesthetic quality of the
inpaintings, we encourage the readers to browse through the inpainting
examples provided in the supplementary
material.222https://ashispati.github.io/inpaintnet/ We consider some of those
examples in the analysis below.

Figure 7 shows sample inpaintings by the models for one of the melodies in the
test set. While the Base-ARNN model collapses to produce long half notes which
do not effectively reflect the surrounding context, the other two models do
better. Both the Reg-ARNN and InpaintNet model generate rhythmically
consistent inpaintings. The InpaintNet, in particular, mimics the rhythmic
properties of the context better. For instance, measures $7$ and $10$ of the
inpainted measures match the rhythm of measures $6$, $14$, and $15$. Also,
measure $8$ matches measure $16$. However, the use of G (subdominant scale
degree in D-major) in the half-note to end measure $8$ is unusual. We observed
that in other examples also, the InpaintNet model occasionally produces
pitches which are anomalous — either out-of-key or not fitting in the context.
The Reg-ARNN model, on the other hand, tends to stay in key. Additional
examples are provided in the supplementary material.

Figure 8: Figure showing different inpaintings (using the InpaintNet model)
for the same context as Figure 7.

One advantage of working with the latent space is that the sampling operation,
inherent in the VAE inference process, ensures that for the same context we
can get different inpainting results. Figure 8 shows three such generations
for the context of Figure 7. It is interesting to note that the base rhythm is
retained across all three inpaintings. This feature is particularly
interesting from an interactive music generation perspective, as this model
can be used to quickly provide users with multiple ideas and will be
investigated further in future work.

Figure 9: Figure based on the subjective listening study showing the
probability that the InpaintNet model is rated higher. Analysis is based on
the Bradley-Terry model [44, 45]. The proposed model loses against the real
data but performs at par with the baseline models.

###  4.6 Subjective Listening Study

To evaluate the perceived quality of the inpainted measures, a listening test
was conducted to compare our proposed model against the two baselines. A set
of $30$ melodies from the held-out test set were randomly selected and their
first $16$ measures were extracted. The models were then used to inpaint $4$
measures (measure number $7$ to $10$) in these melodic excerpts. Participants
were presented with pairs of melodic excerpts and asked to select the one in
which they thought the inpainted measures fit better within the surrounding
context. In some of the pairs, one melodic excerpt was the real data (without
any inpainting). Each participant was presented with $10$ such pairs. A total
of $72$ individuals participated in the study ($720$ comparisons). The
location of the inpainted measures was kept consistent across all examples so
as to prevent confusion among participants and allow them to focus better on
the inpainted measures.

The Bradley-Terry model [44, 45] for paired comparisons was used to get an
estimate of how the proposed model performs against the baselines and the real
data (see Figure 9). While the proposed model expectedly has a very low
probability of winning against the real data (wins approx. $1$ out of $5$
times), it performs only at par with the baseline models (with probability
approx. $0.5$). Significance tests using the Wilcoxon signed rank test were
further conducted which validated that differences between the proposed model
and the baselines were not statistically significant ($p$-value $>0.01$). This
was unexpected since the proposed model showed significant improvement over
the baselines in the NLL metric. Further dividing the study population into
two groups differing in musical proficiency (based on the Ollen index [46])
showed that, comparatively, the group with greater musical proficiency favored
the generations from the InpaintNet model more than the group with less
musical proficiency.

Additional analysis revealed that cases where the InpaintNet model performed
the worst (maximum losses against the baselines), had anomalies in the
predicted pitch similar to those discussed in Section 4.5. Specifically, they
either had a single out-of-key note (e.g., F note in G-Major scale) or used a
pitch or interval not used in the provided contexts. We conjecture that it is
these anomalous pitch predictions which lead to poor perceptual ratings in
spite of the model performing better in terms of modeling rhythmic features.
This will be analyzed further in future studies.

##  5 Conclusion

This paper investigates the problem of musical score inpainting and proposes a
novel approach to generate multiple measures of music to connect two musical
excerpts by using a conditional RNN which learns to traverse the latent space
of a VAE. We also improve upon the data encoding and introduce a stochastic
training process which facilitate model training and improve generalization.
The proposed model shows good performance across different objective and
subjective evaluation experiments. The architecture also enables multiple
generations with the same contexts, thereby, making it suitable for
interactive applications [47]. We think the idea of learning to traverse
latent spaces could be useful for other music generation tasks also. For
instance, the architecture of the LatentRNN model can be changed to add
contextual information from other voices/instruments to perform multi-
instrument music generation. Future work will include a more thorough
investigation of the anomalies in pitch prediction. A possible way to address
that would be to add the context embedding as input at each step of unrolling
the LatentRNN or use additional regularizers. Another promising avenue for
future work is substituting RNNs with attention-based models [48] which have
had success in sequential music generation tasks [9].

## References

  * [1] Rebecca Fiebrink, Baptiste Caramiaux, R Dean, and A McLean.  The machine learning algorithm as creative musical tool.  Oxford University Press, 2016.
  * [2] Jean-Pierre Briot and François Pachet.  Deep learning for music generation: Challenges and directions.  Neural Computing and Applications, Oct 2018.
  * [3] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck.  A hierarchical hatent vector model for learning long-term structure in music.  In Proc. of the 35th International Conference on Machine Learning (ICML), pages 4364–4373, Stockholmsmässan, Stockholm Sweden, 2018\.
  * [4] Florian Colombo, Samuel P. Muscinelli, Alexander Seeholzer, Johanni Brea, and Wulfram Gerstner.  Algorithmic composition of melodies with deep recurrent neural networks.  In Proc. of the 1st Conference on Computer Simulation of Musical Creativity (CSMC), 2016.
  * [5] Bob L Sturm, Joao Felipe Santos, Oded Ben-Tal, and Iryna Korshunova.  Music transcription modelling and composition using deep learning.  In Proc. of the 1st Conference on Computer Simulation of Musical Creativity (CSMC), Huddersfield, UK, 2016.
  * [6] Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang.  MidiNet: A convolutional generative adversarial network for symbolic-domain music generation.  In Proc. of International Society of Music Information Retrieval Conference (ISMIR), pages 324–331, Suzhou, China, 2017.
  * [7] Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent.  Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription.  In Proc. of 29th International Conference on Machine Learning (ICML), Edinburgh, Scotland, 2012.
  * [8] Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan.  This time with feeling: Learning expressive musical performance.  Neural Computing and Applications, pages 1–13, 2018.
  * [9] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck.  Music transformer.  In Proc. of International Conference of Learning Representations (ICLR), New Orleans, USA, 2019.
  * [10] Gaëtan Hadjeres and Frank Nielsen.  Anticipation-RNN: Enforcing unary constraints in sequence generation, with application to interactive music generation.  Neural Computing and Applications, Nov 2018.
  * [11] Jean-Pierre Briot, Gaëtan Hadjeres, and François Pachet.  Deep learning techniques for music generation-A survey.  arXiv preprint arXiv:1709.01620, 2017.
  * [12] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In Proc. of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pages 417–424. ACM Press/Addison-Wesley Publishing Co., 2000.
  * [13] Amir Adler, Valentin Emiya, Maria G Jafari, Michael Elad, Rémi Gribonval, and Mark D. Plumbley.  Audio inpainting.  IEEE Transactions on Audio, Speech, and Language Processing, 20(3):922–932, 2012.
  * [14] Çağdaş Bilen, Alexey Ozerov, and Patrick Pérez.  Audio declipping via nonnegative matrix factorization.  In Proc. of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1–5. IEEE, 2015.
  * [15] Christopher Laguna and Alexander Lerch.  An efficient algorithm for clipping detection and declipping audio.  In Proc. of the 141st AES Convention, Los Angeles, USA, 2016.
  * [16] Nathanael Perraudin, Nicki Holighaus, Piotr Majdak, and Peter Balazs.  Inpainting of long audio segments with similarity graphs.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(6):1083–1094, 2018.
  * [17] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in Neural Information Processing Systems (NeurIPS), pages 3111–3119, 2013.
  * [18] Shan Carter and Michael Nielsen.  Using artificial intelligence to augment human intelligence.  Distill, 2017.  https://distill.pub/2017/aia.
  * [19] Adam Roberts, Jesse Engel, Sageev Oore, and Douglas Eck.  Learning latent representations of music to generate interactive musical palettes.  In Proc. of IUI Workshops, 2018.
  * [20] Gaëtan Hadjeres, Frank Nielsen, and François Pachet.  GLSR-VAE: Geodesic latent space regularization for variational autoencoder architectures.  In Proc. of IEEE Symp. Series on Computational Intelligence (SSCI), pages 1–7. IEEE, 2017.
  * [21] Çağdaş Bilen, Alexey Ozerov, and Patrick Pérez.  Joint audio inpainting and source separation.  In Proc. of International Conference on Latent Variable Analysis and Signal Separation, pages 251–258. Springer, 2015.
  * [22] Ichrak Toumi and Valentin Emiya.  Sparse non-local similarity modeling for audio inpainting.  In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 576–580. IEEE, 2018.
  * [23] Jason Sakellariou, Francesca Tria, Vittorio Loreto, and François Pachet.  Maximum entropy model for melodic patterns.  In Proc. of the ICML Workshop on Constructive Machine Learning, Lille, France, 2015.
  * [24] Gaëtan Hadjeres, François Pachet, and Frank Nielsen.  DeepBach: A steerable model for Bach chorales generation.  In Proc. of the 34th International Conference on Machine Learning (ICML), volume 70, pages 1362–1371, Sydney, Australia, 2017.
  * [25] Stefan Lattner, Maarten Grachten, and Gerhard Widmer.  Imposing higher-level structure in polyphonic music generation using convolutional restricted boltzmann machines and constraints.  Journal of Creative Music Systems, 2, March 2018.
  * [26] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In Proc. of International Conference of Learning Representations (ICLR), Banff, Canada, 2014.
  * [27] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proc. of the 25th International Conference on Machine learning (ICML, pages 1096–1103, Helsinki, Finland, 2008.
  * [28] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational Inference: A review for statisticians.  Journal of the American Statistical Association, 112(518):859–877, 2017.
  * [29] Solomon Kullback and Richard A Leibler.  On information and sufficiency.  The annals of mathematical statistics, 22(1):79–86, 1951.
  * [30] Mason Bretan, Gil Weinberg, and Larry Heck.  A unit selection methodology for music generation using deep neural networks.  In Proc. of the 8th International Conference on Computational Creativity (ICCC), Atlanta, USA, 2016.
  * [31] Stefan Lattner, Maarten Grachten, and Gerhard Widmer.  A predictive model for music based on learned interval representations.  In Proc. of International Society of Music Information Retrieval Conference (ISMIR), pages 26–33, Paris, France, 2018.
  * [32] Andreas Arzt and Stefan Lattner.  Audio-to-score alignment using transposition-invariant features.  In Proc. of International Society of Music Information Retrieval Conference (ISMIR), pages 592–599, Paris, France, 2018.
  * [33] Jesse Engel, Matthew Hoffman, and Adam Roberts.  Latent constraints: Learning to generate conditionally from unconditional generative models.  In Proc. of International Conference on Learning Representations (ICLR), Toulon, France, 2017.
  * [34] Ashis Pati and Alexander Lerch.  Latent space regularization for explicit control of musical attributes.  In ICML Machine Learning for Music Discovery Workshop (ML4MD), Extended Abstract, Long Beach, CA, USA, 2019.
  * [35] Ian Simon, Adam Roberts, Colin Raffel, Jesse Engel, Curtis Hawthorne, and Douglas Eck.  Learning a latent space of multitrack measures.  In Proc. of the 2nd Workshop on Machine Learning for Creativity and Design, Montréal, Québec, 2018.
  * [36] Gino Brunner, Andres Konrad, Yuyi Wang, and Roger Wattenhofer.  MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer.  In Proc. of International Society of Music Information Retrieval Conference (ISMIR), pages 747–754, Paris, France, 2018.
  * [37] Mike Schuster and Kuldip K Paliwal.  Bidirectional recurrent neural networks.  IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [38] Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever.  An empirical exploration of recurrent network architectures.  In Proc. of 32nd International Conference on Machine Learning (ICML), pages 2342–2350, Lille, France, 2015.
  * [39] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.  Self-normalizing neural networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 971–980, 2017.
  * [40] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [41] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  Beta-Vae: Learning basic visual concepts with a constrained variational framework.  In Proc. of International Conference on Learning Representations (ICLR), Toulon, France, 2017.
  * [42] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio.  Generating sentences from a continuous space.  In Proc. of the 20th Conference on Computational Language Processing, 2015.
  * [43] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Proc. of International Conference on Learning Representations (ICLR), San Diego, USA, 2015.
  * [44] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. The method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [45] David R Hunter et al.  MM algorithms for generalized Bradley-Terry models.  The annals of statistics, 32(1):384–406, 2004.
  * [46] Joy E Ollen.  A criterion-related validity test of selected indicators of musical sophistication using expert ratings.  PhD thesis, The Ohio State University, 2006.
  * [47] Théis Bazin, Ashis Pati, and Gaëtan Hadjeres.  A model-agnostic web interface for interactive music composition by inpainting.  Neural Information Processing Systems (NeurIPS), 2018.  Demonstration Track.
  * [48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017.