#  Global quantum phase diagram and non-Abelian chiral spin liquid in a
spin-3/2 square lattice antiferromagnet

Wei-Wei Luo1,2, Yixuan Huang3,4, D. N. Sheng5, W. Zhu1,2 1Institute of Natural
Sciences, Westlake Institute for Advanced Study, Hangzhou 310024, China
2School of Science, Westlake University, Hangzhou 310024, China  3Theoretical
Division, Los Alamos National Laboratory, Los Alamos, New Mexico 87545, USA
4Center for Integrated Nanotechnologies, Los Alamos National Laboratory, Los
Alamos, New Mexico 87545, USA  5Department of Physics and Astronomy,
California State University, Northridge, California 91330, USA

###### Abstract

Since strong quantum fluctuations are essential for the emergence of quantum
spin liquids, there have been extensive exploration and identification of spin
liquid candidates in spin-$1/2$ systems, while such activities are rare in
higher spin systems. Here we report an example of non-Abelian chiral spin
liquid emerging in spin-$3/2$ Heisenberg model on a square lattice. By tuning
Heisenberg exchange interaction and scalar chirality interaction, we map out a
quantum phase diagram enclosing three conventional magnetic orders and a
chiral spin liquid based on density matrix renormalization group studies. The
nature of the spin liquid is identified as a long-sought bosonic version of
Read-Rezayi state that supports non-Abelian Fibonacci anyonic statistics,
identified by the ground state entanglement spectrum. Significantly, we
establish that the non-Abelian CSL emerges through the enlarged local degrees
of freedom and enhanced quantum fluctuations near the classical phase
boundaries of competing magnetic orders. Our numerical discovery of an exotic
quantum spin liquid in spin-$3/2$ system suggests a new route for discovering
fractionalized quantum phases in frustrated higher spin magnetic compounds.

Introduction.— One main theme in condensed matter physics is to search and
classify various quantum states of matter. While most quantum phases can be
heuristically understood in terms of the symmetry breaking paradigm, some
strongly correlated states go beyond the conventional classification by local
order parameter and interesting phenomena may emerge. Quantum spin liquid
(QSL) [1, 2, 3, 4], which does not form any conventional magnetic order even
down to zero temperature, is such an example of exotic state that internally
possesses fractionalized quasi-particles and long range quantum entanglement.
Anderson [5] initially envisioned such a quantum disordered state to be
realized in the frustrated triangular Heisenberg model. Such a possible QSL in
the triangular Heisenberg model was predicted to be a gapped chiral spin
liquid (CSL) [6], which breaks time reversal symmetry as a spin analogy of the
fractional quantum Hall liquid [7, 8, 9] exhibiting a topological order [10].
Recently, using large-scale numerical simulations, CSLs have been
unambiguously identified in local spin-$1/2$ models on kagome lattice [11, 12,
13, 14], triangular lattice [15, 16], honeycomb lattice [17, 18, 19], and
square lattice [20, 18]. The mechanism of the formation of the CSLs is
attributed to the strong interplay of geometric frustration and quantum
fluctuation. The examples of CSLs available so far all share some common
features, i.e. they are produced in $S=1/2$ models where quantum fluctuations
are strong and they are identified as bosonic Laughlin state processing
Abelian topological order.

While the Abelian CSL appears to be common in these frustrated spin-1/2
systems, much less is understood regarding the emergence of CSL that possesses
non-Abelian fractional statistics in realistic local spin model. Kitaev model
is one remarkable example demonstrating the existence of such a non-Abelian
topological order, relevant to Kitaev materials [21]. Besides the Kitaev
materials, one natural place to search for such a state is in the systems with
larger spin ($S>1/2$) [22]. So far, only a few studies on CSLs have been
reported in spin $S=1$ system [23, 24, 25, 26, 27, 28, 29], which have
identified non-Abelian Moore-Read state [30] with quasi-particles obeying the
Ising anyonic statistics. While the Moore-Read CSL has potential to realize
topological quantum computation [31], from a practical point of view the Read-
Rezayi state [32] that hosts the non-Abelian Fibonacci anyon has better
performance in topological quantum computation under a noisy environment due
to its universality in quantum computing algorithm [31, 33, 34, 35]. Although
there are rare cases of Fibonacci anyon that are proposed in exotic fractional
quantum Hall state [36, 37, 38, 39, 40] and Kondo anyons [41], it is highly
desired to search for the emergent non-Abelian CSL supporting Fibonacci
anyonic statistics in general and realistic large spin $S>1$ systems. Such
systems may be realized in quasi-two dimensional antiferromagets with 3d
transition metal including $\text{Ba}_{2}\text{Co}\text{Ge}_{2}\text{O}_{7}$
[42, 43].

In this paper, we address the central issue whether the CSL can arise by
suppressing magnetically ordered states in a higher spin system. Specifically
we consider an antiferromagnetic Heisenberg model with quantum $S=3/2$ spins
on the square lattice. Using large-scale density matrix renormalization group
(DMRG) calculations [44, 45], we establish a global phase diagram including
three conventional magnetic orders which survive even in the classical limit,
and importantly, among the phase boundaries of magnetically ordered states
there exists a CSL state. This CSL is characterized by exponentially decaying
spin correlation and characteristic level countings in entanglement spectrum
[46] as a finger-print of the non-Abelian Read-Rezayi state [32].

Model and method.— We study the spin-$3/2$ $J_{1}$-$J_{2}$-$J_{\chi}$
Heisenberg model on a square lattice,

$\displaystyle H=$ $\displaystyle
J_{1}\sum_{\left<ij\right>}\mathbf{S}_{i}\cdot\mathbf{S}_{j}+J_{2}\sum_{\left<\left<ij\right>\right>}\mathbf{S}_{i}\cdot\mathbf{S}_{j}$
$\displaystyle+J_{\chi}\sum_{ijk\in\bigtriangleup}\mathbf{S}_{i}\cdot(\mathbf{S}_{j}\times\mathbf{S}_{k}),$

where $\mathbf{S}_{i}$ denotes $SU(2)$ symmetric spin-$3/2$ operator on site
$i$. The exchange interactions $J_{1}$ and $J_{2}$ run over all nearest-
neighbor bonds $\left<ij\right>$ and next-nearest-neighbor bonds
$\left<\left<ij\right>\right>$, respectively. The three-spin scalar chiral
interaction $J_{\chi}$ runs over all four triangles within each primitive
square plaquette, and the vertices $ijk$ of each triangle are ordered in
clockwise manner (see Fig. 1(a)). This term explicitly breaks time reversal
symmetry and thus favors long range chiral orders. Physically this chiral spin
interaction can be deduced from the extended Hubbard model in a magnetic
field, whose large repulsion $U$ limit at half filling gives rise to
$J_{\chi}\propto\frac{t_{1}^{2}t_{2}}{U^{2}}\sin\Phi$ for a primitive triangle
enclosed by magnetic flux $\Phi$ [47] ($t_{1}$ and $t_{2}$ are the nearest and
next nearest neighbor electron hoppings, respectively). In the following we
fix $J_{1}=1$ as the unit of energy scale.

To determine possible quantum phases and quantum phase diagram, we
systematically utilize both finite and infinite DMRG calculations with $U(1)$
symmetry on cylinder geometry [44, 48]. Due to the much larger dimension of
Hilbert space compared to spin $S=1/2$ case, we mainly focus our study on
finite and infinite cylinder with circumferences $L_{y}=4$ and $6$. We keep
the bond dimension of matrix product state up to $\chi=4000$, which allows to
obtain the ground state on a $L_{y}=4$ ($L_{y}=6$) cylinder with a typical
truncation error of about $10^{-8}$ ($10^{-5}$).

Figure 1: (a) Spin-$3/2$ $J_{1}$-$J_{2}$-$J_{\chi}$ Heisenberg model on the
square lattice. (b) Quantum phase diagram by tuning $J_{2}$ and $J_{\chi}$ (by
setting $J_{1}=1$). CSL, CSS, and QSL* respectively denotes chiral spin
liquid, chiral spin state, and possible quantum spin liquid. We also show
classical phase boundaries as dashed lines, which separate three magnetic
ordered phases (marked by green color).

Phase diagram.— Analyzing the classical spin system will give us important
clues on possible quantum phases. The classical Heisenberg model on the square
lattice harbors three magnetically ordered phases in $J_{2}$-$J_{\chi}$ phase
diagram [49] including a Neel state at small $J_{2}$ regime, a stripe state at
large $J_{2}$ regime and a chiral spin state (CSS) at larger $J_{\chi}$ regime
(see Fig. 1(b)). These three states meet at classical transition point
$(J_{2},J_{\chi})=(0.5,0.25)$, near which quantum fluctuation is expected to
be strong and may promote spin disordered state in the quantum spin systems.

We present a global quantum phase diagram of spin-$3/2$
$J_{1}$-$J_{2}$-$J_{\chi}$ model in Fig. 1(b) using infinite DMRG
calculations. When tuning off the chiral term $J_{\chi}=0$, we find the
conventional Neel and stripe order at small and large $J_{2}$, respectively.
Between them we find a quantum disordered regime near classical transition
point $J_{2}=0.5$, where the spin-spin and dimer-dimer correlation decays
exponentially. This state does not break lattice translational symmetry nor
time reversal symmetry and we label it by QSL* in the phase diagram (see
Supple. Mat. Sec. V [50]). By gradually increasing $J_{\chi}$, both of two
magnetic phases extend to a finite regime in the phase diagram. When the
chiral term dominates ($J_{\chi}\geq 0.25$ ), a non-coplanar CSS is observed,
which is also a magnetic state that survives even in the classical limit. In
vicinity of classical transition boundaries, we discover a finite regime for
CSL which hosts extremely short-ranged spin correlations. The nature of this
CSL state is identified as bosonic version of Read-Rezayi state via the
characteristic entanglement spectrum as we demonstrate below.

Figure 2: Upper panel: Spin structure factor $S(\mathbf{k})$ in the magnetic
(a) Neel, (b) stripe, and (c) CSS phases at representative points. Lower
panel: profile of magnetic order parameters (d) $S(\pi,\pi)$, (e) $S(\pi,0)$
and (f) $S(0,\pi)$ in the $J_{2}$-$J_{\chi}$ parameter space. The white dashed
line marks the phase boundary.

Magnetic orders.— In order to determine magnetic orders, we compute spin
structure factor
$S(\mathbf{k})=\frac{1}{N}\sum_{i,j}\braket{\mathbf{S}_{i}\cdot\mathbf{S}_{j}}e^{i\mathbf{k}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}$,
where both $i$ and $j$ sum over $N$ lattice sites. The structure factors in
different phases are shown in the upper panel of Fig. 2, where different peak
locations specifically correspond to different magnetic orders. When
$J_{\chi}$ is small, we find Bragg peaks at $(\pi,\pi)$ for small $J_{2}$ and
at $(0,\pi)$ for large $J_{2}$, which are consistent with Neel and stripe
magnetic orders, respectively. When $J_{\chi}$ is large, we find multi-Q
feature in spin structure factor, with two peaks at $(0,\pi)$ and $(\pi,0)$,
as well as two satellite peaks at $(\pm\pi/2,\pm\pi/2)$, which are consistent
with the spin arrangement of non-coplanar chiral spin state [49]. As discussed
above, these three magnetic orders are also found in the corresponding
classical Heisenberg model. In the regime in between these three magnetic
ordered phases, no sharp Bragg peak is present, which indicates a nonmagnetic
regime where quantum fluctuation is substantially strong to destroy long range
orders. In the lower panel of Fig. 2, we show the profile of spin structure
factors at specific momentum locations $\mathbf{S}(\pi,\pi)$,
$\mathbf{S}(\pi,0)$ and $\mathbf{S}(0,\pi)$ in the whole $J_{2}$-$J_{\chi}$
phase diagram, where large values are observed in the Neel order phase, stripe
order phase, and CSS phase, respectively. They are consistent with the phase
boundaries in Fig. 1(b), which are also labeled as dashed white lines in Fig.
2 for comparison.

Figure 3: Momentum resolved entanglement spectrum of CSL in (a) $L_{y}=4$
cylinder and (b) $L_{y}=6$ cylinder. The low-lying entanglement spectrum
clearly shows level counting $\\{1,1,3,6,\dots\\}$ in $S^{z}=0$ sector and
$\\{1,2,5,\dots\\}$ in $S^{z}=\pm 1$ sectors in both system sizes.

Figure 4: Spin correlation function $|\langle S_{0}S_{x}\rangle|$ versus
distance for different phases at (a) $J_{\chi}=0.15$ and (c) $J_{\chi}=0.05$.
Long range spin correlation $\sqrt{|\langle S_{0}S_{d}\rangle|}$ versus
$J_{2}$ for (b) $J_{\chi}=0.15$ and (d) $J_{\chi}=0.05$. We set $L_{y}=4$
here.

Read-Rezayi non-Abelian CSL.— In the vicinity of the boundaries of magnetic
phases, the spin correlation
$|\langle\mathbf{S}_{0}\cdot\mathbf{S}_{x}\rangle|$ shows an exponential decay
along the cylinder, much faster than those in magnetic ordered phases. This is
shown in Fig. 4 for $L_{y}=4$ cylinder (see results for $L_{y}=6$ cylinder in
Supp. Mat. Sec. I [50]), where distinct behaviors of spin correlations are
clearly observed in ordered and disordered phases. We have also checked
nearest neighbor bond energy in the nonmagnetic regime and find no lattice
translational symmetry breaking consistent with a uniform state without
valence bond order.

The explicit nature of this underlying spin liquid can be unambiguously
revealed by characteristic level countings of edge excitations. Here we
extract momentum resolved entanglement spectrum [46, 51] from the matrix
product state representation of the ground state, which is found to have one-
to-one correspondence with the edge spectrum. In Fig. 3(a), we show the low-
lying entanglement spectrum of the ground state at $J_{2}=0.5$ and
$J_{\chi}=0.15$ in $L_{y}=4$ cylinder using finite DMRG calculations. We find
that the quasi-degenerate level counting exactly matches the tower of states
of $SU(2)_{3}$ Wess-Zumino-Witten theory [52] in vacuum sector, which
describes edge excitations of the Read-Rezayi fractional quantum Hall state.
As anticipated (see Supp. Mat. [50]), the low-lying entanglement spectrum
reveals characteristic level counting of $\\{1,1,3,6,\dots\\}$ in $S^{z}=0$
sector and $\\{1,2,5,\dots\\}$ in $S^{z}=\pm 1$ sectors. We also find the same
edge countings in $L_{y}=6$ cylinder, which is shown in Fig. 3(b). Intuitively
these characteristic countings can also be understood in terms of generalized
Pauli principle [53, 38] in the thin-torus limit, which states no more than
$3$ particles in $2$ consecutive orbitals for bosonic Read-Rezayi state
indicating the importance of larger local Hilbert space.

Phase transition.— The phase boundaries in the phase diagram can be determined
by long-distance spin correlation defined as $\sqrt{|\langle
S_{0}S_{x}\rangle|}$, where $x$ denotes the distance between two spins. In
Fig. 4(a), for two magnetic phases, the spin correlation decays relatively
slow with distances. They correspond to the Neel order and stripe order, as
indicated by the Bragg peaks in spin structure factor in Fig. 2. In Fig. 4(b),
the spin correlation at long distance $d=19$ is vanishing small for
$0.48<J_{2}<0.52,J_{\chi}=0.15$, leaving a window for the CSL phase. A similar
behavior is also observed for small values of $J_{\chi}$. When we vary $J_{2}$
at fixed $J_{\chi}=0.05$, we find a similar nonmagnetic phase for $0.5\leq
J_{2}\leq 0.54$ as shown in Figs. 4(c) and (d). In this case, however, the
influence of spin chirality term is relatively small and we do not observe
characteristic edge countings of the CSL in entanglement spectrum. For this
reason, we label this quantum disordered regime as QSL*, whose exact nature is
beyond current investigation and will be left for future studies. For larger
$J_{\chi}>0.25$ the influence of spin chirality term dominates and stabilizes
CSS, which is consistent with classical analysis. The quantum phase diagram
shown in Fig. 1 is obtained in $L_{y}=4$ system. We have also checked the
phase diagram for $L_{y}=6$ and observe a similar transition behavior (see
Supp. Mat. [50]) We find all the quantum phases found in $L_{y}=4$ cylinder
persists for the $L_{y}=6$ system.

Summary and discussion.— We have numerically studied the spin-$3/2$
$J_{1}$-$J_{2}$-$J_{\chi}$ Heisenberg model on the square lattice using
unbiased density-matrix renormalization group calculation. We map out the
global quantum phase diagram that contains three conventional magnetic orders,
including a Neel state, a stripe state, and a chiral spin state. Crucially,
among these magnetic phases an interesting chiral spin liquid (CSL) is
uncovered, where the quantum fluctuations are strongly enhanced near the phase
boundaries of different magnetic ordered phases. The quantum fluctuations
destroy conventional long-ranged magnetic orders and induce a CSL. The nature
of this CSL is identified as a non-Abelian Read-Rezayi state via
characteristic momentum resolved entanglement spectrum. Additionally, we also
find another possible quantum spin liquid near $J_{\chi}\sim 0$, whose exact
nature deserves future study. Our findings demonstrate the existence of non-
Abelian CSL in higher spin quantum antiferromagnets which supports the non-
Abelian Fibonacci anyonic statistics and also paves the way to searching other
intriguing QSLs in these higher spin systems via the mechanism of enhancing
quantum fluctuations through tuning competing interactions. Our model
Hamiltonian may be realized in the 3d transition metal compound
$\text{Ba}_{2}\text{Co}\text{Ge}_{2}\text{O}_{7}$ [42, 43] with effective
spin-3/2 antiferromagnetic Heisenberg exchange, where the effective chiral
spin interactions can be induced by an applied out-of-plane magnetic field.

At last, several remarks are given in order. First, in comparison with the
non-Abelian spin liquid with Ising-type anyons in $S=1$ model [28], the spin
liquid in spin-$3/2$ model is not only a simple extension, but a new class of
exotic topological orders which will inspire and call for new theoretical
proposals. Another difference is that Fibonacci anyon does not have a free-
Fermion description, in sharp contrast to the Ising anyon [54]. Second, how to
understand the current findings in spin-$3/2$ model and connect the current
model with previous studies in soft-boson model [55] are highly nontrival. One
possible way to think about it is from coupled-wire construction [56, 57] or
the projective construction [58, 59, 60, 37], i.e. if treating a spin-$3/2$
system as three coupled spin-$1/2$ layers, one could image the non-Abelian
$Z_{3}$ order is manifested by three Abelian spin liquid states. However, the
effective interactions which allow such a state to emerge from three coupled
spin-1/2 systems can become another challenge issue demanding future studies.

W.Z. thanks Z. X. Liu for simulating discussions. This work was supported by
National Science Foundation of China under project number 92165102, 11974288
(W.W.L., W.Z.). Y.H. was supported by the U.S. DOE NNSA under Contract No.
89233218CNA000001 and by the Center for Integrated Nanotechnologies, a DOE BES
user facility, in partnership with the LANL Institutional Computing Program
for computational resources. D.N.S. was mainly supported by National Science
Foundation (NSF) Partnership in Research and Education in Materials
DMR-1828019 and partially supported by NSF Princeton Center for Complex
Materials, a Materials Research Science and Engineering Center DMR-2011750.

## References

  * Balents [2010] L. Balents, Spin liquids in frustrated magnets, Nature 464, 199 (2010).
  * Zhou  _et al._ [2017] Y. Zhou, K. Kanoda, and T.-K. Ng, Quantum spin liquid states, Rev. Mod. Phys. 89, 025003 (2017).
  * Savary and Balents [2017] L. Savary and L. Balents, Quantum spin liquids: A review, Rep. Prog. Phys. 80, 016502 (2017).
  * Broholm  _et al._ [2020] C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, Quantum spin liquids, Science 367, eaay0668 (2020).
  * Anderson [1973] P. W. Anderson, Resonating valence bonds: A new kind of insulator?, Materials Research Bulletin 8, 153 (1973).
  * Kalmeyer and Laughlin [1987] V. Kalmeyer and R. B. Laughlin, Equivalence of the resonating-valence-bond and fractional quantum Hall states, Phys. Rev. Lett. 59, 2095 (1987).
  * v. Klitzing  _et al._ [1980] K. v. Klitzing, G. Dorda, and M. Pepper, New Method for High-Accuracy Determination of the Fine-Structure Constant Based on Quantized Hall Resistance, Phys. Rev. Lett. 45, 494 (1980).
  * Tsui  _et al._ [1982] D. C. Tsui, H. L. Stormer, and A. C. Gossard, Two-Dimensional Magnetotransport in the Extreme Quantum Limit, Phys. Rev. Lett. 48, 1559 (1982).
  * Laughlin [1983] R. B. Laughlin, Anomalous Quantum Hall Effect: An Incompressible Quantum Fluid with Fractionally Charged Excitations, Phys. Rev. Lett. 50, 1395 (1983).
  * Wen [1990] X. G. Wen, Topological orders in rigid states, Int. J. Mod. Phys. B 04, 239 (1990).
  * Gong  _et al._ [2014] S.-S. Gong, W. Zhu, and D. N. Sheng, Emergent Chiral Spin Liquid: Fractional Quantum Hall Effect in a Kagome Heisenberg Model, Sci. Rep. 4, 6317 (2014).
  * He  _et al._ [2014] Y.-C. He, D. N. Sheng, and Y. Chen, Chiral Spin Liquid in a Frustrated Anisotropic Kagome Heisenberg Model, Phys. Rev. Lett. 112, 137202 (2014).
  * Bauer  _et al._ [2014] B. Bauer, L. Cincio, B. P. Keller, M. Dolfi, G. Vidal, S. Trebst, and A. W. W. Ludwig, Chiral spin liquid and emergent anyons in a Kagome lattice Mott insulator, Nat. Commun. 5, 1 (2014).
  * He and Chen [2015] Y.-C. He and Y. Chen, Distinct spin liquids and their transitions in spin-$1/2$ $xxz$ kagome antiferromagnets, Phys. Rev. Lett. 114, 037201 (2015).
  * Gong  _et al._ [2017] S.-S. Gong, W. Zhu, J.-X. Zhu, D. N. Sheng, and K. Yang, Global phase diagram and quantum spin liquids in spin-1/2 triangular antiferromagnet, Phys. Rev. B 96, 075116 (2017), arXiv:1705.00510 .
  * Cookmeyer  _et al._ [2021] T. Cookmeyer, J. Motruk, and J. E. Moore, Four-Spin Terms and the Origin of the Chiral Spin Liquid in Mott Insulators on the Triangular Lattice, Phys. Rev. Lett. 127, 087201 (2021).
  * Hickey  _et al._ [2016] C. Hickey, L. Cincio, Z. Papić, and A. Paramekanti, Haldane-Hubbard Mott Insulator: From Tetrahedral Spin Crystal to Chiral Spin Liquid, Phys. Rev. Lett. 116, 137202 (2016).
  * Hickey  _et al._ [2017] C. Hickey, L. Cincio, Z. Papić, and A. Paramekanti, Emergence of chiral spin liquids via quantum melting of noncoplanar magnetic orders, Phys. Rev. B 96, 115115 (2017).
  * Huang  _et al._ [2021] Y. Huang, X.-Y. Dong, D. N. Sheng, and C. S. Ting, Quantum phase diagram and chiral spin liquid in the extended spin-$\frac{1}{2}$ honeycomb xy model, Phys. Rev. B 103, L041108 (2021).
  * Nielsen  _et al._ [2013] A. E. B. Nielsen, G. Sierra, and J. I. Cirac, Local models of fractional quantum Hall states in lattices and physical implementation, Nat. Commun. 4, 1 (2013).
  * Hermanns  _et al._ [2018] M. Hermanns, I. Kimchi, and J. Knolle, Physics of the Kitaev Model: Fractionalization, Dynamic Correlations, and Material Connections, Annu. Rev. Condens. Matter Phys. 9, 17 (2018).
  * Greiter  _et al._ [2014] M. Greiter, D. F. Schroeter, and R. Thomale, Parent Hamiltonian for the non-Abelian chiral spin liquid, Phys. Rev. B 89, 165125 (2014).
  * Greiter and Thomale [2009] M. Greiter and R. Thomale, Non-Abelian Statistics in a Quantum Antiferromagnet, Phys. Rev. Lett. 102, 207203 (2009).
  * Liu  _et al._ [2015] T. Liu, W. Li, A. Weichselbaum, J. von Delft, and G. Su, Simplex valence-bond crystal in the spin-1 kagome Heisenberg antiferromagnet, Phys. Rev. B 91, 060403 (2015).
  * Changlani and Läuchli [2015] H. J. Changlani and A. M. Läuchli, Trimerized ground state of the spin-1 Heisenberg antiferromagnet on the kagome lattice, Phys. Rev. B 91, 100407 (2015).
  * Liu  _et al._ [2018] Z.-X. Liu, H.-H. Tu, Y.-H. Wu, R.-Q. He, X.-J. Liu, Y. Zhou, and T.-K. Ng, Non-abelian $s=1$ chiral spin liquid on the kagome lattice, Phys. Rev. B 97, 195158 (2018).
  * Chen  _et al._ [2018] J.-Y. Chen, L. Vanderstraeten, S. Capponi, and D. Poilblanc, Non-Abelian chiral spin liquid in a quantum antiferromagnet revealed by an iPEPS study, Phys. Rev. B 98, 184409 (2018).
  * Huang  _et al._ [2022] Y. Huang, W. Zhu, S.-S. Gong, H.-C. Jiang, and D. N. Sheng, Coexistence of non-Abelian chiral spin liquid and magnetic order in a spin-1 antiferromagnet, Phys. Rev. B 105, 155104 (2022).
  * Jaworowski and Nielsen [2022] B. Jaworowski and A. E. B. Nielsen, Non-Abelian chiral spin liquid on a spin-1 kagome lattice: Truncation of an exact Hamiltonian and numerical optimization, Phys. Rev. B 106, 115131 (2022).
  * Moore and Read [1991] G. Moore and N. Read, Nonabelions in the fractional quantum hall effect, Nucl. Phys. B 360, 362 (1991).
  * Nayak  _et al._ [2008] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. Das Sarma, Non-Abelian anyons and topological quantum computation, Rev. Mod. Phys. 80, 1083 (2008).
  * Read and Rezayi [1999] N. Read and E. Rezayi, Beyond paired quantum Hall states: Parafermions and incompressible states in the first excited Landau level, Phys. Rev. B 59, 8084 (1999).
  * Lahtinen and Pachos [2017] V. Lahtinen and J. Pachos, A Short Introduction to Topological Quantum Computation, SciPost Phys. 3, 021 (2017).
  * Field and Simula [2018] B. Field and T. Simula, Introduction to topological quantum computation with non-Abelian anyons, Quantum Sci. Technol. 3, 045004 (2018).
  * Génetay Johansen and Simula [2021] E. Génetay Johansen and T. Simula, Fibonacci anyons versus majorana fermions: A monte carlo approach to the compilation of braid circuits in $\mathrm{SU}(2{)}_{k}$ anyon models, PRX Quantum 2, 010334 (2021).
  * Xia  _et al._ [2004] J. S. Xia, W. Pan, C. L. Vicente, E. D. Adams, N. S. Sullivan, H. L. Stormer, D. C. Tsui, L. N. Pfeiffer, K. W. Baldwin, and K. W. West, Electron Correlation in the Second Landau Level: A Competition Between Many Nearly Degenerate Quantum Phases, Phys. Rev. Lett. 93, 176809 (2004).
  * Vaezi and Barkeshli [2014] A. Vaezi and M. Barkeshli, Fibonacci Anyons From Abelian Bilayer Quantum Hall States, Phys. Rev. Lett. 113, 236804 (2014).
  * Zhu  _et al._ [2015] W. Zhu, S. Gong, F. Haldane, and D. Sheng, Fractional quantum hall states at $\nu=13/5$ and $12/5$ and their non-abelian nature, Phys. Rev. Lett. 115, 126805 (2015).
  * Ghazaryan  _et al._ [2017] A. Ghazaryan, T. Graß, M. J. Gullans, P. Ghaemi, and M. Hafezi, Light-Induced Fractional Quantum Hall Phases in Graphene, Phys. Rev. Lett. 119, 247403 (2017).
  * Mong  _et al._ [2017] R. S. K. Mong, M. P. Zaletel, F. Pollmann, and Z. Papić, Fibonacci anyons and charge density order in the 12/5 and 13/5 quantum Hall plateaus, Phys. Rev. B 95, 115136 (2017).
  * Komijani [2020] Y. Komijani, Isolating Kondo anyons for topological quantum computation, Phys. Rev. B 101, 235131 (2020).
  * Miyahara and Furukawa [2011] S. Miyahara and N. Furukawa, Theory of Magnetoelectric Resonance in Two-Dimensional S = 3/2 Antiferromagnet Ba2CoGe2O7 via Spin-Dependent Metal-Ligand Hybridization Mechanism, J. Phys. Soc. Jpn. 80, 073708 (2011).
  * Romhányi and Penc [2012] J. Romhányi and K. Penc, Multiboson spin-wave theory for ba2coge2o7: A spin-3/2 easy-plane néel antiferromagnet with strong single-ion anisotropy, Phys. Rev. B 86, 174428 (2012).
  * White [1992] S. R. White, Density matrix formulation for quantum renormalization groups, Phys. Rev. Lett. 69, 2863 (1992).
  * Schollwöck [2011] U. Schollwöck, The density-matrix renormalization group in the age of matrix product states, Annals of Physics January 2011 Special Issue, 326, 96 (2011).
  * Li and Haldane [2008] H. Li and F. D. M. Haldane, Entanglement Spectrum as a Generalization of Entanglement Entropy: Identification of Topological Order in Non-Abelian Fractional Quantum Hall Effect States, Phys. Rev. Lett. 101, 010504 (2008).
  * Sen and Chitra [1995] D. Sen and R. Chitra, Large-U limit of a Hubbard model in a magnetic field: Chiral spin interactions and paramagnetism, Phys. Rev. B 51, 1922 (1995).
  * Hauschild and Pollmann [2018] J. Hauschild and F. Pollmann, Efficient numerical simulations with Tensor Networks: Tensor Network Python (TeNPy), SciPost Phys. Lect. Notes , 005 (2018).
  * Rabson and Trugman [1995] D. A. Rabson and S. A. Trugman, A spin model for investigating chirality, J. Phys.: Condens. Matter 7, 9005 (1995).
  * [50] See Supplemental Materials at [URL will be inserted by publisher] for detailed numerical results and discussions.
  * Cincio and Vidal [2013] L. Cincio and G. Vidal, Characterizing Topological Order by Studying the Ground States on an Infinite Cylinder, Phys. Rev. Lett. 110, 067208 (2013).
  * Francesco  _et al._ [1997] P. D. Francesco, P. Mathieu, and D. Sénéchal,  _Conformal Field Theory_ (Springer, New York, 1997).
  * Bernevig and Haldane [2008] B. A. Bernevig and F. D. M. Haldane, Model Fractional Quantum Hall States and Jack Polynomials, Phys. Rev. Lett. 100, 246802 (2008).
  * Kitaev [2001] A. Y. Kitaev, Unpaired Majorana fermions in quantum wires, Phys.-Usp. 44, 131 (2001).
  * Cooper  _et al._ [2001] N. R. Cooper, N. K. Wilkin, and J. M. F. Gunn, Quantum Phases of Vortices in Rotating Bose-Einstein Condensates, Phys. Rev. Lett. 87, 120405 (2001).
  * Kane  _et al._ [2002] C. L. Kane, R. Mukhopadhyay, and T. C. Lubensky, Fractional Quantum Hall Effect in an Array of Quantum Wires, Phys. Rev. Lett. 88, 036401 (2002).
  * Teo and Kane [2014] J. C. Y. Teo and C. L. Kane, From Luttinger liquid to non-Abelian quantum Hall states, Phys. Rev. B 89, 085101 (2014).
  * Cappelli  _et al._ [2001] A. Cappelli, L. S. Georgiev, and I. T. Todorov, Parafermion Hall states from coset projections of abelian conformal theories, Nuclear Physics B 599, 499 (2001).
  * Regnault  _et al._ [2008] N. Regnault, M. O. Goerbig, and T. Jolicoeur, Bridge Between Abelian and Non-Abelian Fractional Quantum Hall States, Phys. Rev. Lett. 101, 066803 (2008).
  * Barkeshli and Wen [2010] M. Barkeshli and X.-G. Wen, Anyon Condensation and Continuous Topological Phase Transitions in Non-Abelian Fractional Quantum Hall States, Phys. Rev. Lett. 105, 216804 (2010).
  * Ardonne  _et al._ [2001] E. Ardonne, P. Bouwknegt, and K. Schoutens, Non-Abelian Quantum Hall States—Exclusion Statistics, K-Matrices, and Duality, Journal of Statistical Physics 102, 421 (2001).
  * Ardonne  _et al._ [2004] E. Ardonne, R. Kedem, and M. Stone, Filling the Bose sea: Symmetric quantum Hall edge states and affine characters, J. Phys. A: Math. Gen. 38, 617 (2004).
  * Haldane [2007] F. D. M. Haldane, Generalized Pauli principle for Read- Rezayi non-Abelian Quantum Hall States, YKIS 2007 Workshop Interact. Nanostructural Eff. Low-Dimens. Syst. (2007).

Supplementary Materials for “ Global quantum phase diagram and non-Abelian
chiral spin liquid in a spin-3/2 square lattice antiferromagnet ”

In this supplementary materials, we present more details to support the
conclusion in the main text. Sec. I shows the spin correlations as a function
of bond dimensions, as a consistent check of the robustness of numerical
results. We also show the spin correlations on $L_{y}=6$ cylinder. Sec. II
presents the dimer correlations and discusses the possibility of valence bond
solids. In Sec. III, we show the evidence of phase transitions on a wider
cylinder $L_{y}=6$. In Sec. IV, we list the countings of the effect edge
theory of the bosonic Read-Rezayi state. In Sec. V, we examine the possible
nematicity in the QSL*.

##  I Spin correlations

In Fig. S1 we show the behavior of spin correlation in different phases on
$L_{y}=4$ cylinder. In this case spin correlations has ignorable dependence on
the maximum bond dimension used in DMRG calculations, indicating well
converged behavior. For the QSL* state at $(0.52,0.05)$ and CSL state at
$(0.49,0.15)$, we use semi-log plot and clearly find exponentially decaying
behavior in all bond dimensions. On the other hand, the spin correlations in
magnetic phases decay much slower than those in spin liquid phases, and tends
to a power-law behavior in large bond dimension limit.

Figure S1: Spin correlation function
$|\langle\mathbf{S}_{0}\cdot\mathbf{S}_{x}\rangle|$ versus distance for
various bond dimensions. Here we set $L=4$, $J_{\chi}=0.05$ (upper panel) and
$J_{\chi}=0.15$ (lowere panel). Please note that the scale in (b) and (e) are
different from the others.

In Fig. S2 we show the behavior of spin correlation in different phases on
$L_{y}=6$ cylinder. As in the case of $L_{y}=4$ cylinder, we also find
exponential spin decay in QSL* and CSL phases for all studied bond dimensions.
On the other magnetic phases, the spin correlations decays much slower as
expected. On increasing the bond dimension, we find the tendency of power-law
decay in these magnetic phases.

Figure S2: Spin correlation function
$|\langle\mathbf{S}_{0}\cdot\mathbf{S}_{x}\rangle|$ versus distance for
various bond dimensions. Here we set $L=6$, $J_{\chi}=0.05$ (upper panel) and
$J_{\chi}=0.15$ (lowere panel). Please note that the scale in (b) and (e) are
different from the others.

##  II Dimer correlations

We examine the dimer-dimer correlation function $\langle
D^{x}_{0}D^{x}_{x}\rangle$ in QSL* phase to see whether valence bond solid may
exists. The dimer operator is defined by

$D^{\alpha}_{x}=\mathbf{S}_{x}\cdot\mathbf{S}_{x+\alpha}$ (S1)

where $\alpha=\hat{x}(\hat{y})$ labels the nearest neighbor site along $x(y)$
direction.

As shown in Fig. S3, we find a uniform dimer-dimer correlation in both
$L_{y}=4$ cylinder and $L_{y}=6$ cylinder. No signal of “strong-weak” pattern
is found. Moreover, we also show the fluctuation term $\langle
D^{x}_{0}D^{x}_{x}\rangle-\langle D_{0}\rangle\langle D_{x}\rangle$, we
observe an exponentially decaying behavior. And the correlation length does
not increase if going from $L_{y}=4$ to $L_{y}=6$. These observations rule out
the possibility of valence bond order.

Figure S3: Dimer correlation function $\langle D_{0}D_{x}\rangle$ versus
distance for different phases at (0.52,0.05)

##  III Phase transition on $L_{y}=6$ cylinder

We show phase transitions in $L_{y}=6$ cylinder at fixed $J_{\chi}=0.15$ and
$J_{\chi}=0.05$ in Fig. S4. (In Fig. 4 in main text, we show the results on
$L_{y}=4$.) In both cases, we find Neel order at small $J_{2}$ and stripe
order at large $J_{2}$, both of which possess long range magnetic order. At
intermediate $J_{2}$ a disordered regime exists, which correspond to CSL at
$J_{\chi}=0.15$ and QSL* phase at $J_{\chi}=0.05$. Based on the calculations
on $L_{y}=4,6$, we believe the finding of CSL and QSL* in this
$J_{1}$-$J_{2}$-$J_{\chi}$ model is robust. Nevertheless, we cannot totally
exclude the finite-size effect beyond $L_{y}>6$, which is out of reach of our
computational capbility.

Figure S4: Spin correlation function
$|\langle\mathbf{S}_{0}\cdot\mathbf{S}_{x}\rangle|$ versus distance for
different phases at (a) $J_{\chi}=0.15$ and (c) $J_{\chi}=0.05$. Long range
spin correlation $\sqrt{|\langle S_{0}S_{d}\rangle|}$ versus $J_{2}$ for (b)
$J_{\chi}=0.15$ and (d) $J_{\chi}=0.05$. We set $L_{y}=6$ here.

##  IV Edge counting of Read-Rezayi state

Here we list degeneracy sequences of edge modes in all topological sectors of
$\nu=3/2$ bosonic Read-Rezayi (RR) state. For the details one may consult the
references like [61, 62, 63, 53]. We didnot find these information and results
in the literature, so we list them here.

Starting from a highest density root configuration with momentum $\Delta L=0$,
we can enumerate all admissible configurations constrained by generalized
Pauli principle, which states a fractional exclusion statistics that no more
than 3 bosons are allowed in 2 consecutive orbitals in this RR state. These
admissible configurations are in one to one correspondence with RR edge modes
and thus yields the degeneracy in each momentum sector. The following tables
list all possible countings for $\Delta L\leq 4$. They correspond to two root
configurations …303030303… and …2121212121… with different number of bosons.

Table S1: bosonic RR state root configuration …303030303 (3n) with edge counting 1,1,3,6,12 $\Delta L=0$ | $\Delta L=1$ | $\Delta L=2$ | $\Delta L=3$ | $\Delta L=4$
---|---|---|---|---
303030303 | 3030303021 | 30303030201 | 303030302001 | 3030303020001
|  | 30303021210 | 303030212010 | 3030302120010
|  | 30303030120 | 303021212100 | 3030212120100
|  |  | 303030211200 | 3021212121000
|  |  | 303030301110 | 3030212112000
|  |  | 303030300300 | 3030302030100
|  |  |  | 3030302111100
|  |  |  | 3030301212000
|  |  |  | 3030302103000
|  |  |  | 3030303011010
|  |  |  | 3030303002100
|  |  |  | 3030303010200
Table S2: bosonic RR state root configuration …303030302 (3n+2) or …303030301 (3n+1) with edge counting 1,2,5,9,18 $\Delta L=0$ | $\Delta L=1$ | $\Delta L=2$ | $\Delta L=3$ | $\Delta L=4$
---|---|---|---|---
303030302 | 3030303011 | 30303030101 | 303030301001 | 3030303010001
| 3030302120 | 30303021110 | 303030211010 | 3030302110010
|  | 30302121200 | 303021211100 | 3030212110100
|  | 30303020300 | 302121212000 | 3021212111000
|  | 30303030020 | 303021203000 | 2121212120000
|  |  | 303030121100 | 3021212030000
|  |  | 303030202100 | 3030211211000
|  |  | 303030210200 | 3030212021000
|  |  | 303030300110 | 3030203030000
|  |  |  | 3030212102000
|  |  |  | 3030301210100
|  |  |  | 3030301121000
|  |  |  | 3030301202000
|  |  |  | 3030302020100
|  |  |  | 3030302012000
|  |  |  | 3030302101100
|  |  |  | 3030303001010
|  |  |  | 3030303000200
Table S3: bosonic RR state root configuration …2121212121 (3n) or …212121212 (3n+2) with edge counting 1,2,5,10,20 $\Delta L=0$ | $\Delta L=1$ | $\Delta L=2$ | $\Delta L=3$ | $\Delta L=4$
---|---|---|---|---
2121212121 | 21212121201 | 212121212001 | 2121212120001 | 21212121200001
| 21212121120 | 212121203010 | 2121212030010 | 21212120300010
|  | 212121211110 | 2121203030100 | 21212030300100
|  | 212121121200 | 2121212021100 | 21203030301000
|  | 212121210300 | 2121211211100 | 21212030211000
|  |  | 2121121212000 | 21212120210100
|  |  | 2121211203000 | 21212112110100
|  |  | 2121212111010 | 21211212111000
|  |  | 2121212102100 | 21121212120000
|  |  | 2121212110200 | 21211212030000
|  |  |  | 21212111211000
|  |  |  | 21212112021000
|  |  |  | 21212103030000
|  |  |  | 21212112102000
|  |  |  | 21212120121000
|  |  |  | 21212120202000
|  |  |  | 21212121110010
|  |  |  | 21212121020100
|  |  |  | 21212121012000
|  |  |  | 21212121101100
Table S4: bosonic RR state root configuration …212121211 (3n+1) with edge counting 1,3,6,13,24 $\Delta L=0$ | $\Delta L=1$ | $\Delta L=2$ | $\Delta L=3$ | $\Delta L=4$
---|---|---|---|---
212121211 | 2121212101 | 21212121001 | 212121210001 | 2121212100001
| 2121211210 | 21212112010 | 212121120010 | 2121211200010
| 2121212020 | 21211212100 | 212112120100 | 2121121200100
|  | 21212111200 | 211212121000 | 2112121201000
|  | 21212030200 | 212112112000 | 1212121210000
|  | 21212120110 | 212121030100 | 2112121120000
|  |  | 212121111100 | 2121120301000
|  |  | 212120301100 | 2121121111000
|  |  | 212030302000 | 2121112120000
|  |  | 212120212000 | 2120302120000
|  |  | 212121103000 | 2030303020000
|  |  | 212121201010 | 2121121030000
|  |  | 212121200200 | 2121210300100
|  |  |  | 2121210211000
|  |  |  | 2121202111000
|  |  |  | 2120303011000
|  |  |  | 2121202030000
|  |  |  | 2121211110100
|  |  |  | 2121203010100
|  |  |  | 2121203002000
|  |  |  | 2121211021000
|  |  |  | 2121211102000
|  |  |  | 2121212010010
|  |  |  | 2121212001100

##  V Possible nematicity in spin liquid phases

In our performed DMRG calculations, we notice possisble nematicity in QSL*
phases. The nematic order can be measured by the difference of bond energies
along $x$ and $y$ directions as $\langle S_{i}S_{i+x}\rangle-\langle
S_{i}S_{i+y}\rangle$, where $i$ labels lattice sites in the bulk. Fig. S5
shows the tendency of nematic order on increasing bond dimension $\chi$ or
cylinder circumference $L_{y}$. Although the nematicity is quite strong in
each case, it quickly reduces by increasing cylinder width from $L_{y}=4$ to
$L_{y}=6$. Due to the limited system sizes accessible, we cannot determine
whether such nematic order persists in the thermodynamic limit. If this
nematic order persists in larger system size, this indicates the QSL* phase
has a rotational symmetry breaking nature. Since available system sizes is
limited in DMRG calculation, we propose to study this possible QSL* phase
using other complementary methods such as variational Monte Carlo.

Figure S5: Nematicity in QSL* on increasing bond dimension $\chi$ for
different cylinder circumferences $L_{y}$. We also show the real-space
distribution of NN bond energy on the right side, which corresponds to QSL*
phase at (0.52,0.05) on a $12\times 6$ finite cylinder. The line width and
color (red/blue for positive/negative) denote the amplitude and sign of
corresponding bond energy.

##  VI Entanglement spectrum in QSL*

As is shown in the main text, the entanglement spectrum of CSL state exhibits
characteristic quasi-degenerate pattern that manifests its underlying
topological order. On the contrary, the QSL* found at small $J_{\chi}$ does
not possess such clear signature. We show in Fig. S6 the entanglement spectrum
of disordered QSL* at $(0.52,0)$. The entanglement spectrum in this extreme
case has only two possible momentum quantum numbers due to the real-valued
wave function. We can see clear distinctions between the disordered CSL and
QSL* phases.

Figure S6: Entanglement spectrum in the QSL* phase at $(0.52,0)$ on a
$L_{y}=6$ cylinder.


# DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models

Chao Huang
University of Rochester
Susan Liang
University of Rochester
Yapeng Tian
University of Texas at Dallas
Anurag Kumar
Meta Reality Labs Research
Chenliang Xu
University of Rochester

###### Abstract

We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework
that solves the audio-visual sound source separation task through a generative
manner. While existing discriminative methods that perform mask regression
have made remarkable progress in this field, they face limitations in
capturing the complex data distribution required for high-quality separation
of sounds from diverse categories. In contrast, DAVIS leverages a generative
diffusion model and a Separation U-Net to synthesize separated magnitudes
starting from Gaussian noises, conditioned on both the audio mixture and the
visual footage. With its generative objective, DAVIS is better suited to
achieving the goal of high-quality sound separation across diverse categories.
We compare DAVIS to existing state-of-the-art discriminative audio-visual
separation methods on the domain-specific MUSIC dataset and the open-domain
AVE dataset, and results show that DAVIS outperforms other methods in
separation quality, demonstrating the advantages of our framework for tackling
the audio-visual source separation task.

##  1 Introduction

Visually-guided sound source separation, also referred to as audio-visual
separation, is a pivotal task for assessing a machine perception system’s
ability to integrate multisensory signals. The primary goal is to isolate
individual sounds from a complex audio mixture by utilizing visual cues about
the objects that are producing the sounds, e.g., separate the “barking” sound
from the mixture by querying the “dog” object. To achieve human-like
intelligence, an effective separation model should be capable of handling a
diverse range of sounds and produce high-quality separations that can deliver
a realistic auditory experience.

The community has dedicated significant efforts to this task, and existing
methods [1, 2, 3, 4, 5] have made extensive attempts to tackle this problem,
such as developing more powerful separation frameworks [1, 2, 4], proposing
more effective training pipeline [5], and incorporating additional visual cues
[3] to enhance the separation performance. For optimization, these approaches
usually take mask regression [1] or spectrogram reconstruction [6] as training
objectives.

While these methods have shown promising separation performance in specific
domains, such as musical instrument sounds, they are not yet satisfactory in
dealing with open-domain sounds where background noise and off-screen sounds
are prevalent. These sounds produce complicated mosaic of time and frequency
patterns, posing a significant challenge in achieving high-quality separation.
Thus, a natural question arises: is there an effective approach to model these
complex audio data distribution and produce high-quality separated sounds?

We answer the question by introducing a generative framework for the audio-
visual separation. A new class of generative models called denoising diffusion
probabilistic models (DDPMs) [7, 8, 9], also known as diffusion models, has
emerged recently and demonstrated remarkable abilities in generating diverse
and high-quality images [10] and audio [11]. The impressive capabilities of
generative diffusion models in capturing complex data distributions inspire us
to explore their potential for enhancing audio-visual separation. Unlike
discriminative modeling, we believe that generative diffusion models can
effectively approximate more intricate data distributions, allowing us to
handle open-domain time and frequency patterns and lead to superior separation
results.

To this end, we present DAVIS, a novel framework for audio-visual separation
that is built upon a generative diffusion model. Unlike typical discriminative
methods that predict a mask representing the separated sound from the input
mixture, DAVIS approaches the separation task as a conditional generation
process. Specifically, our method incorporates a T-step diffusion and reverse
process [7, 10, 8]: during the training stage, Gaussian noise controlled by a
variance schedule [12] is added to the unmixed sound at each diffusion step.
In the reverse process, our method initiates from a standard Gaussian
distribution, and an effective Separation U-Net is proposed to estimate the
noise added at each diffusion step, iteratively generating the separated
magnitude with guidance from the mixture and visual footage. The Separation
U-Net comprises an encoder-decoder structure with enabled skip connections. To
capture both local time-frequency patterns and long-range time dependencies,
we introduce a CA block consisting of two ResNet blocks for capturing local
patterns and a Time-Attention block for learning long-range time correlation.
Furthermore, to enhance audio-visual association learning, we devise a Feature
Interaction module to facilitate interactions between audio and visual
features and inject visual cues into the separation.

Experiments on the MUSIC [1] and AVE [13] datasets demonstrate that DAVIS
outperforms the state-of-the-art methods in terms of separation quality. Our
contributions are summarized as follows:

  * •

We are the first study, to the best of our knowledge, to approach the audio-
visual separation task as a conditional generation process and solve it using
a diffusion model.

  * •

We design a Separation U-Net, which incorporates CA blocks and a Feature
Interaction module to capture the audio-visual association effectively.

  * •

Our framework surpasses previous methods on both specific and open-domain
sound datasets, highlighting the benefits of solving audio-visual separation
through a generative approach.

##  2 Related Work

Audio-Visual Sound Source Separation. In this section, our focus is on modern
audio-visual sound source separation approaches while acknowledging the
prolonged research efforts dedicated to sound source separation in signal
processing. Recent deep learning-based audio-visual sound source separation
methods have been applied to different categories of audio, such as speech
signals [14, 6, 15, 16], musical instrument sounds [1, 3, 5, 2, 17, 4], and
universal sound sources [18, 19, 20, 21, 22, 23]. These methods typically
employ a learning regime that involves mixing two audio streams from different
videos to provide supervised training signals. A sound separation network,
often implemented as a U-Net, is then used for mask regression [1, 2, 5, 24]
conditioned on associated visual representations. In recent years, research in
this area has shifted from solving domain-specific sound source separation to
addressing the challenge of open-domain sounds [20, 21, 19]. However, such
methods require text-query [20] or estimating motion information [19]. In this
paper, we propose a novel audio-visual separation approach that can handle
both specific and open-domain sound sources.

Diffusion Models. Diffusion models [7, 25, 26] fall under the category of deep
generative models that start with a sample in a random distribution and
gradually restore the data sample through a denoising process. Recently,
diffusion models have exhibited remarkable performance across various domains,
including computer vision [10, 27, 28, 29, 30, 31, 32, 33, 34], natural
language processing [35, 36, 37, 38], and audio applications [11, 39, 40, 38,
41, 42, 43]. While diffusion models have been successfully employed for
single-modality generation, their potential for audio-visual tasks remains
largely unexplored. For instance, only recently has MM-diffusion [44] proposed
simultaneous generation of videos and audio. Furthermore, there has been a
growing interest in employing diffusion models for discriminative tasks. Some
pioneer works have explored the application of diffusion models to image
segmentation [45, 46, 47] and object detection [48]. However, despite
significant interest in this direction, there have been no prior successful
attempts to apply generative diffusion models to audio-visual scene
understanding, which has notably lagged behind the progress in visual
perception tasks. To the best of our knowledge, this paper presents the first
work that adopts a diffusion model to learn audio-visual associations for
audio-visual sound source separation.

##  3 Method

In this section, we introduce DAVIS, our novel diffusion model-based audio-
visual separation framework designed for achieving high-quality separation
results. We begin by providing a brief recap of the preliminary knowledge of
diffusion models in Sec. 3.1. Next, we present our proposed Separation U-Net
architecture, which effectively captures the audio-visual association through
the generation process in Sec. 3.3. Finally, we discuss the training and
inference designs in Sec. 3.4 and Sec. 3.5, respectively.

###  3.1 Preliminaries

We introduce the concept of diffusion models, which serves to illustrate the
pipeline of our framework. A diffusion model consists of a forward and a
reverse process. The forward process is defined as a Markovian chain that
gradually adds noise to the data sample $x_{0}$ according to a variance
schedule $\beta_{1},...,\beta_{T}$:

$q(x_{1:T}|x_{0})=\prod_{i=1}^{T}q(x_{t}|x_{t-1}),$ (1)
$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t}|\sqrt{\bar{\alpha}_{t}}x_{t-1},\beta_{t}\mathbf{I}),$
(2)

where $\alpha_{t}=1-\beta_{t}$ and
$\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$. Note that the variance schedule
is also fixed during the reverse process. If the total number of $T$ goes to
infinity, the diffusion process will finally lead to pure noise, i.e., the
distribution of $p(x_{T})$ is $\mathcal{N}(x_{t};\mathbf{0},\mathbf{I})$ with
only Gaussian noise.

The reverse process aims to recover samples from Gaussian distribution by
removing the noise gradually, which is a Markovian chain parameterized by
$\theta$:

$p_{\theta}(x_{0:T})=p_{\theta}(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t}),$
(3)

where at each iteration, the noise $\epsilon$ added in the forward process is
estimated as:

$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\boldsymbol{\mu_{\theta}}(x_{t},t),\boldsymbol{\Sigma_{\theta}}(x_{t},t)).$
(4)

Note that we set the variances
$\boldsymbol{\Sigma_{\theta}}(x_{t},t)=\tilde{\beta_{t}}\mathbf{I}$ to
untrained constants and $\boldsymbol{\mu_{\theta}}(x_{t},t)$ is usually
implemented as neural networks. To adapt the diffusion process into a
conditional one, we include the conditional context $\boldsymbol{c}$ as
additional network inputs, which modifies Eq. 4 as follows:

$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\boldsymbol{\mu_{\theta}}(x_{t},t,\boldsymbol{c}),\tilde{\beta_{t}}\mathbf{I}),\quad\text{where}\;\tilde{\beta_{t}}:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}.$
(5)

We follow [7] to use a simplified training objective:

$L_{simple}(\theta)=\mathbb{E}_{t,x_{0},\epsilon}[||\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\boldsymbol{c},t)||],$
(6)

where $\epsilon_{\theta}$ represents a function approximator used to predict
$\epsilon$, while $t$ denotes a uniformly sampled value ranging from 1 to $T$.
The loss function computes a weighted variational bound that highlights
different aspects of the reconstruction.

Figure 1: Overview of the DAVIS framework. Our objective is to synthesize the
separated sound $x_{0}$ by leveraging an audio mixture $x^{mix}$ and an image
$v$, while taking into account the diffusion timestep $t$. Firstly, we sample
a latent variable $x_{T}$ from a standard distribution. Next, we encode the
image $v$ and the timestep $t$ into the embedding space, resulting in feature
vectors $\boldsymbol{v}$ and $\boldsymbol{t}$. These features serve as
conditions in the Separation U-Net $\epsilon_{\theta}$, which performs
iterative denoising on $x_{T}$ to obtain the separated sound $x_{0}$.
Specifically, $\boldsymbol{t}$ is passed to all the modules within
$\epsilon_{\theta}$, while $\boldsymbol{v}$ is only utilized in the Feature
Interaction Module (Sec. 3.3) to enhance audio-visual association learning.

###  3.2 Task Setup and Method Overview

Given an unlabeled video clip $V$, we can extract an audio-visual pair
$(a,v)$, where $a$ and $v$ denote the audio and visual stream, respectively.
In real-world scenarios, the audio stream can be a mixture of $N$ individual
sound sources, denoted as $a=\sum_{i=1}^{N}s_{i}$, where each source $s_{i}$
can be of various categories. The primary goal of the visually-guided sound
source separation task is to utilize visual cues from $v$ to effectively
separate $a$ into its constituent sources $s_{i}$, for $i\in\\{1,2,...,N\\}$.
Since no labels are provided to distinguish the sound sources $s_{i}$, prior
works [1, 5, 24] have commonly used a “mix and separate” strategy, which
involves mixing audio streams from two different videos and manually create
the mixture: $a^{mix}=a^{(1)}+a^{(2)}$. Furthermore, the time series $a$ is
usually transformed into magnitude spectrogram by short-time Fourier transform
(STFT): $x=\mathbf{STFT}(a)\in\mathbb{R}^{T\times F}$, allowing for
manipulations in the 2D-like Time-Frequency domain, where $F$ and $T$ are the
numbers of frequency bins and time frames, respectively. Consequently, the
goal of training is to learn a separation network capable of mapping
$(x^{mix},v^{(1)})\rightarrow x^{(1)}$. For simplicity, we will omit the video
index notation in the subsequent sections 111In this paper, superscripts
denote video indices, while subscripts refer to diffusion timesteps..

In contrast to discriminative approaches that perform the mapping through
regression, our proposed DAVIS framework is built on a diffusion model with a
T-step diffusion and reverse process. The diffusion process is determined by a
fixed variance schedule as described in Eq. 1 and Eq. 2, which gradually adds
noises to the magnitude spectrogram $x_{0}$ and converts it to latent $x_{T}$.
As depicted in Fig. 1, the reverse process (according to Eq. 3 and Eq. 5) of
DAVIS is specified by our proposed separation network $\epsilon_{\theta}$.
This reverse process iteratively denoises a latent variable $x_{T}$, which is
sampled from a uniform distribution, to obtain a separated magnitude
conditioned on the magnitude of the input sound mixture $x^{mix}$ and the
visual footage $v$. Consequently, the objective of the separation network
$\epsilon_{\theta}$ is to predict the noise $\epsilon$ added at each diffusion
timestep during the forward process.

###  3.3 Separation U-Net

Previous works [2, 4] often utilize a U-Net [49]-like architecture for
separation network designs. This choice is attributed to the U-Net’s
effectiveness in capturing multi-level feature representations and producing
separated magnitudes with the same shape as inputs. Exploiting the grid-like
nature of magnitude spectrograms, existing methods employ convolution-based
U-Nets for separation and concatenate audio and visual features directly at
the bottleneck to incorporate visual cues. While these approaches achieve good
separation performance, we argue that they may be inadequate for real-world
sound mixtures due to two key reasons. Firstly, similar frequency patterns can
occur in temporally distant frames, and distinct frequency patterns can mix
within a single time frame. Such occurrences necessitate the network to
capture both local patterns and long-range dependencies across time and
frequency dimensions, where pure convolution may fall short. Motivated by
this, we propose a novel Separation U-Net (depicted in Fig. 1) that
incorporates Convolution-Attention (CA) blocks (elaborated in Fig. 2). These
blocks combine sequential convolution and time-attention mechanisms to learn
both local and global time-frequency associations. Secondly, learning
representative audio-visual associations is crucial for the task. While
existing methods directly concatenate visual features and audio embeddings at
the bottleneck and feed them to the decoder network for mask regression, such
approaches lack the ability to foster further interactions between audio and
visual features. To address this limitation, we introduce an audio-visual
feature interaction module, which enhances association learning by enabling
interactions between audio and visual modalities.

Figure 2: Illustrations on CA Block and Feature Interaction Module. (a) Our CA
block operates by taking audio feature maps and a time embedding
$\boldsymbol{t}$ as inputs. Each sub-block, except the up/down sampling layer,
is conditioned on $\boldsymbol{t}$. It consists of two groups of convolutions
within each ResNet block to capture local time-frequency patterns, while the
Time-Attention block captures long-range dependencies along the time
dimension. (b) The Feature Interaction Module functions by replicating and
concatenating $\boldsymbol{v}$ with $\boldsymbol{f_{a}}$. Two identical ResNet
blocks, as described in (a), are used to process the concatenated features.
However, in contrast to (a), the attention block here operates on both the
frequency and time dimensions instead of solely aggregating temporal contexts.

Timestep Embedding. In a diffusion model, the timestep embedding serves to
inform the model about the current position of the input within the Markovian
chain. As shown in Fig. 1, diffusion time $t$ is specified by the Transformer
sinusoidal position embedding [50] and further transformed by an MLP, which
will be passed to each CA block as a condition.

Encoder/Decoder Designs. Our proposed Separation U-Net architecture comprises
an encoder and a decoder, with an audio-visual feature interaction module in
between. Both the encoder and decoder consist of four CA Blocks, and skip
connections are utilized to facilitate information flow. Initially, we
concatenate the latent variable $x_{T}$ with the mixture $x^{mix}$ along the
channel dimension and employ a 1x1 convolution to project it to the feature
space. As depicted in Fig. 2, each CA block in the encoder consists of two
ResNet blocks for extracting local time-frequency patterns and one Time-
Attention block for capturing long-range time dependencies. Following this, a
downsample layer (or upsample layer for the decoder) with a scale factor of 2
is used. Specifically, we construct the ResNet block using WeightStandardized
2D convolution [51] along with GroupNormalization [52] and SiLU activation
[53]. To incorporate the time embedding $\boldsymbol{t}$ as a conditioning
factor, we employ an MLP to generate $\boldsymbol{t}$-dependent scaling and
shifting vectors for feature-wise affine transformation [54] before the
activation layer. A Time-Attention block is appended after the ResNet blocks
to capture long-range time dependencies. For implementation, we adopt the
design proposed by Wang et al.[55], which includes Pre-Layer Normalization and
Multi-Head Attention along the time dimension within the residual connection.
The downscale and upscale layers are implemented using 2D convolutions with a
stride of 2. As a result, we can obtain audio feature maps
$\boldsymbol{f_{a}}\in\mathbb{R}^{C\times\frac{T}{16}\times\frac{F}{16}}$ at
the bottleneck, where $C$ represents the number of channels. Additionally, we
include a 1x1 convolution to convert the decoder output into magnitude.

Audio-Visual Feature Interaction Module. The key to achieving successful
audio-visual separation lies in effectively utilizing visual information to
separate visually-indicated sound sources. Therefore, the interaction between
audio and visual modalities at the feature level becomes crucial. Existing
approaches typically concatenate audio and visual features at the bottleneck
[2, 4] and pass them to the decoder for further fusion. In this paper, we
propose a novel audio-visual feature interaction module to enhance this
capability. Concretely, given the visual footage $v$, we first use a pre-
trained ResNet-18 [56] visual backbone $\mathbf{Enc_{v}}$ to extract the
global embedding $\boldsymbol{v}:=\mathbf{Enc_{v}}(v)\in\mathbb{R}^{C}$. Since
the number of channels in the visual embedding $\boldsymbol{v}$ matches that
of $\boldsymbol{f_{a}}$, we can omit the modality notation. Next, we tile
$\boldsymbol{v}$ to match the shape of $\boldsymbol{f_{a}}$, resulting in
visual feature maps $\boldsymbol{f_{v}}$. Subsequently, the audio and visual
feature maps are concatenated and fed into the feature interaction module
(FIM):
$\hat{\boldsymbol{f_{a}}}:=\mathbf{FIM}([\boldsymbol{f_{a}},\boldsymbol{f_{v}}])$,
where
$\hat{\boldsymbol{f_{a}}}\in\mathbb{R}^{C\times\frac{T}{16}\times\frac{F}{16}}$.
The details of the $\mathbf{FIM}$ module are illustrated in Fig. 2(b),
encompassing ResNet blocks and a time-frequency attention block that
facilitate the establishment of audio-visual associations in both local and
global regions.

###  3.4 Training

Algorithm 1 Training

1:Input: A dataset $D$ that contains audio-visual pairs
$\\{(a^{(k)},v^{(k)})\\}_{k=1}^{K}$, total diffusion step $T$

2:Initialize: randomly initialize Separation U-Net $\epsilon_{\theta}$ and
pre-trained visual encoder $\mathbf{Enc_{v}}$

3:repeat

4: Sample $(a^{(1)},v^{(1)})$ and $(a^{(2)},v^{(2)})\sim D$

5: Mix and compute $x^{mix},x^{(1)}$

6: Scale $x=\log_{e}(1+x)\cdot\sigma$ and clip $x^{mix},x^{(1)}$ to [0,1]

7: Encode visual frame $v^{(1)}$ as
$\boldsymbol{v^{(1)}}:=\mathbf{Enc_{v}}(v^{(1)})$

8: Sample $\epsilon\sim\mathcal{N}(\textbf{0, I})$, and $t\sim$
Uniform${(1,...,T)}$

9: Take gradient step on

10:
$\nabla_{\theta}||\epsilon-\epsilon_{\theta}(x^{(1)}_{t},x^{mix},\boldsymbol{v}^{(1)},t)||,x^{(1)}_{t}=\sqrt{\bar{\alpha}_{t}}x^{(1)}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$

11:until converged

Algorithm 1 depicts the complete training procedure of our DAVIS framework.
Given the sampled audio-visual pairs from the dataset, we first adopt the “mix
and separate” strategy and compute the magnitudes $x^{(1)},x^{(2)},x^{mix}$
with STFT.

Data Scaling: To align with the frequency decomposition of the human auditory
system, we apply a logarithmic transformation to the magnitude spectrogram,
converting it to a log-frequency scale. Additionally, we ensure consistent
scaling of the log-frequency magnitudes by multiplying them with a scale
factor $\sigma$ and clipping the values to fall within the range [0, 1].

The visual frames are encoded to embeddings
$\boldsymbol{v^{(1)}}$,$\boldsymbol{v^{(2)}}$. Taking video (1) as an example,
we sample $\epsilon$ from a standard Gaussian distribution and $t$ from the
set $\\{1,...,T\\}$. Then, we input
$x^{(1)}_{t},x^{mix},\boldsymbol{v}^{(1)},t$ to the Separation U-Net
$\epsilon_{\theta}$ and optimize the network by taking a gradient step on Eq.
6. In practice, we use both video (1) and (2) for optimization, therefore the
final loss term is formulated as
$\mathcal{L}=\mathcal{L}^{(1)}_{simple}(\theta)+\mathcal{L}^{(2)}_{simple}(\theta)$.

###  3.5 Inference

Algorithm 2 Inference

1:Input: Audio mixture $a^{mix}$ and the query visual frame $v$, total
diffusion step $T$

2:Sample $x_{T}\sim\mathcal{N}(\textbf{0, I})$

3:Compute $x^{mix}:=\mathbf{STFT}(a^{mix})$

4:Encode visual frame $\boldsymbol{v}:=\mathbf{Enc_{v}}(v)$

5:for $t=T,...,1$ do

6: Sample $z\sim\mathcal{N}(\textbf{0, I})$ if $t>1$, else $z=0$

7: Compute $x_{t-1}$:
$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},x^{mix},\boldsymbol{v},t))+\sqrt{\tilde{\beta_{t}}}z$

8:end for

9:return $e^{x_{0}/\sigma}-1$

As illustrated in Algorithm 2, our inference process starts from a sampled
latent variable $x_{T}$, and takes the mixture $x^{mix}$ and visual frame
embedding $\boldsymbol{v}$ as conditions to produce the separated magnitude
$x_{0}$ through T iterations. Finally, the output is rescaled to the original
value range.

Methods | Output |  MUSIC [1] |  |  AVE [13]
---|---|---|---|---
SDR | SIR | SAR |  | SDR | SIR | SAR
SoP [1] | Mask | 3.42 | 4.98 | - |  | 0.46 | 4.17 | 12.08
CoSep [2] | Mask | 2.04 | 6.21 | - |  | -1.33 | 2.54 | 5.77
CCoL [5] | Mask | 7.18 | 12.55 | 11.09 |  | 1.77 | 3.25 | 22.52
DAVIS | Mag. | 9.29 | 14.19 | 15.76 |  | 1.78 | 6.61 | 7.76
|  |  |  |  |  |  |  |
Table 1: Comparisons with different discriminative audio-visual separation
approaches on MUSIC and AVE test sets. The “Output” category highlights the
distinction between our method (magnitude synthesis) and others (mask
regression). We report SDR, SIR, and SAR metrics and mark our results in gray.
Note that numbers marked in red actually indicate poor performance (Sec. 4.2).

##  4 Experiments

###  4.1 Experimental Setup

Datasets.222All collection, processing, and use of data obtained from datasets
was conducted by the University of Rochester. Our model demonstrates the
ability to handle both specific and open-domain sound separation. To evaluate
our approach, we use MUSIC [1] and AVE [13] datasets, which cover musical
instruments and open-domain sounds. The evaluation settings are described in
detail below:

  * •

MUSIC: We evaluate our proposed method on the widely-used MUSIC [1] dataset,
which includes 11 musical instrument categories: accordion, acoustic guitar,
cello, clarinet, erhu, flute, saxophone, trumpet, tuba, violin, and xylophone.
All the videos are clean solo and the sounding instruments are usually
visible. We follow CCoL [5] and use the same train/val/test splits, resulting
in a total of 468/26/26 videos across various instrument categories.

  * •

AVE: In addition to the MUSIC dataset, we also evaluate our method on the
Audio-Visual Event (AVE) dataset [13]. This dataset contains 4143 10-second
videos, including 28 diverse sound categories, such as Church Bell, Barking,
and Frying (Food), among others. The AVE dataset presents greater challenges
as the audio in these videos may not span the entire duration and can be
noisy, including off-screen sounds (e.g., human speech) and background noise.
We conduct training and evaluation on this demanding dataset using the
original train/val/test splits, consisting of 3339/402/402 videos,
respectively.

Baselines. To the best of our knowledge, we are the first to adopt a
generative model for the audio-visual source separation task. Thus, we compare
DAVIS against the following state-of-the-art discriminative methods: (i) Sound
of Pixels (SoP) [1] that learns ratio mask predictions with a 1-frame-based
model, (ii) Co-Separation (CoSep) [2] that takes a single visual object as the
condition to perform mask regression, (iii) Cyclic Co-Learn (CCoL) [5] which
jointly trains the model with sounding object visual grounding and visually-
guided sound source separation tasks. Since the AVE dataset lacks bounding box
annotation for detected objects, we use the entire image for CoSep and CCoL.
For all the comparative methods, we use the authors’ publicly available code.
Several other recent works [22, 57] have achieved impressive separation
results. However, due to the unavailability of their source code, we will not
include them in our comparative analysis.

Evaluation Metrics. To quantitatively evaluate the audio-visual sound source
separation performances, we use the standard metrics [1, 5, 2], namely:
Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and
Signal-to-Artifact Ratio (SAR). We adopt the widely-used mir eval library [58]
to report the standard metrics. Note that SDR and SIR evaluate the accuracy of
source separation, whereas SAR specifically measures the absence of artifacts
[2]. Consequently, SAR can be high even if the separation performance is poor
in terms of accurately separating the sources.

Implementation Details. In our experimental setup, we down-sample audio
signals at 11kHz, and the video frame rate is set to 1 fps. For the MUSIC
dataset, each video is approximately 6 seconds and we randomly select 3 frames
per video. As for the AVE dataset, we use the entire 10-second audio as input
and only select 1 frame that falls into the audio-visual event boundary [13]
for training the model. This selection ensures that the auditory object occurs
within the visual frame, facilitating audio-visual association learning.
During training, the frames are first resized to 256$\times$256 and then
randomly cropped to $224\times 224$. We set the total time step $T=1000$ to
train our DAVIS model. During inference, all the frames are directly resized
to the desired size without cropping. To accelerate the separation process, we
use DDIM [9] with a sampling step of 25. The audio waveform is transformed
into a spectrogram with a Hann window of size 1022 and a hop length of 256.
The obtained magnitude spectrogram is subsequently resampled to $256\times
256$ to feed into the separation network. We set the number of audio and
visual feature channels $C$ as 512 and empirically choose the scale factor
$\sigma=0.15$. Our model is trained with the Adam optimizer, with a learning
rate of $10^{-4}$. The training is conducted on a single A6000 GPU for 200
epochs with a batch size of 10.

Figure 3: Visualizations of audio-visual separation results on the MUSIC (top)
and AVE (bottom) datasets. Two sounds are mixed (mixture), and referenced
frames are provided to guide the separation. We show the comparison between
ground truth spectrograms and DAVIS/CCoL’s predictions.

###  4.2 Comparisons with State-of-the-art

To evaluate the effectiveness of our method, we present separation results by
comparing DAVIS with state-of-the-art approaches on the MUSIC and AVE
datasets, as depicted in Tab. 1. Indeed, SDR is a reliable metric for
measuring source separation accuracy in clean datasets like MUSIC, as it does
not consider interference. On the other hand, SIR is useful for evaluating
source separation in multi-source scenarios, which is suitable for open-domain
datasets like AVE. Our results highlight the advantages of utilizing
generative modeling for audio-visual separation. DAVIS consistently
outperforms previous approaches across various evaluation categories,
achieving up to a 2.1 dB improvement on the SDR scale for the MUSIC dataset
and a 3.4 dB improvement on the SIR scale for the AVE dataset, surpassing the
performance of the next best approach CCoL. These results clearly demonstrate
the versatility of our method across diverse datasets with varying visual and
audio contexts. Among the competing techniques, we observe that CCoL and SoP
yield higher SAR results than our method on the AVE dataset. However, we argue
that high SAR values do not necessarily imply effectiveness, as they can arise
from poor separation. It is worth noting that a comparison between the mixture
spectrogram and the ground truth unmixed spectrogram can surprisingly yield
high SAR values. In this context, we believe that our method significantly
improves separation performance compared to others. In Fig. 3, we visually
compare our separation results to the CCoL baseline. Our visualizations
demonstrate that DAVIS achieves higher separation quality, as evidenced by the
closer resemblance of our separated magnitude spectrograms to the ground
truth. Moreover, the successful handling of diverse time patterns in the
provided examples highlights the importance of incorporating time-attention
mechanisms in DAVIS.

###  4.3 Experimental Analysis

Ablation Study. We conduct ablation experiments on the MUSIC validation set to
examine the different components of DAVIS. Firstly, we validate the
effectiveness of our proposed CA block by designing two baselines: (a)
replacing the Time-Attention mechanism within the CA block with a ResNet block
(shown in Fig. 2 (a)), which only captures local time-frequency patterns, and
(b) employing a Time-Frequency Attention mechanism that calculates attention
weights along both the time and frequency dimensions. To ensure a fair
comparison, we adopt an efficient attention mechanism [59] to mitigate
computational overhead, resulting in a similar computation cost as our Time-
Attention block. The results presented in Fig. 6 demonstrate the significance
of capturing long-range time dependencies, while our Time-Attention mechanism
achieves a favorable trade-off between computation and performance compared to
Time-Frequency attention. Second, in Fig. 6, we investigate the impact of
varying the number of sampling steps. We observe that setting the step value
to 25 yields the best results in our setting. Meanwhile, even with step = 10,
satisfactory results are obtained, indicating the potential for further
acceleration if faster inference speed is prioritized.

Methods | SDR | SIR | SAR
---|---|---|---
ResNet Block | 4.90 | 10.21 | 9.65
Time Attn. | 7.57 | 12.55 | 15.50
Time-Freq Attn. | 7.39 | 14.03 | 12.63
|  |  |

Figure 4: Ablation on CA block design.

Sampling step | SDR | SIR | SAR
---|---|---|---
Step=10 | 7.03 | 11.48 | 15.45
Step=15 | 7.29 | 11.95 | 15.56
Step=25 | 7.57 | 12.55 | 15.50
Step=50 | 7.40 | 12.61 | 15.31
|  |  |

Figure 5: Number of sampling steps.

Figure 6: An visualization example showing that our DAVIS model can capture
accurate audio-visual association to perform visually-guided separation.

Learned Audio-Visual Association. The learned associations between audio and
visual features play a crucial role in the success of separation. To
demonstrate our learned associations, we present an example from the AVE
dataset in Fig. 6. In this example, a video clip labeled “Rats” is mixed with
another video clip labeled "Motorcycle." However, human perception reveals the
presence of an off-screen sound “Speech.” occurring in the “Rats” clip, while
only the “rat” object is visible in the reference frame. In this scenario, our
method successfully separates the “Rats” sound from the complicated mixture
while disregarding the “Speech,” thus affirming the accuracy of our learned
audio-visual associations and our method’s capability to capture complex data
distribution.

##  5 Limitation and Discussion

Our proposed DAVIS framework incorporates the extraction of global visual
embedding as a condition for visually-guided source separation. This
technique, which utilizes global visual features, has been widely adopted in
audio-visual learning [1, 24]. Unlike methods that rely on pre-trained object
detectors for extracting visual features, our framework does not have such a
dependency. However, it may encounter limitations when trained on
unconstrained video datasets. Intuitively, successful results can be achieved
when the video contains a distinct sounding object, such as solo videos in the
MUSIC dataset or videos capturing a sounding object performing a specific
event in the AVE dataset. Nonetheless, this training assumption may not hold
in more challenging scenarios, where multiple objects are likely producing
sounds, rendering the global visual embedding inadequate for accurately
describing the content of sounding objects. To address this issue, one
possible approach is to adapt our framework to leverage more fine-grained
visual features and jointly learn sounding object localization and visually-
guided sound separation. This adaptation enables the model to utilize
localized sounding object information to enhance the audio-visual association.

##  6 Conclusion and Future Work

In this paper, we propose DAVIS, a diffusion model-based audio-visual
separation framework designed to address the problem in a generative manner.
Unlike approaches relying on discriminative training objectives for
regression, our separation framework is built upon a T-step diffusion model,
allowing for iterative synthesis of the separated magnitude spectrogram while
conditioning on the visual footage. Leveraging the power of generative
modeling, our method effectively handles complex data distributions and
achieves high-quality sound separation. Extensive experiments on the MUSIC and
AVE datasets validate the efficacy of our framework, demonstrating its
effectiveness in separating sounds within specific domains (e.g., music
instrument sounds) as well as its ability to generalize to open-domain sound
categories.

Our approach initiates the utilization of generative models for audio-visual
scene understanding, paving the way for potential extensions to other multi-
modal perception tasks like audio-visual object localization. Humans
demonstrate the ability to imagine a "dog" upon hearing a "barking" sound,
highlighting the potential of cross-modal generation in advancing audio-visual
association learning. This implies that localization and separation tasks can
be integrated into a single generative framework. In the future, we plan to
explore the application of generative models to jointly address audio-visual
localization and separation tasks.

## References

  * Zhao et al. [2018] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 570–586, 2018.
  * Gao and Grauman [2019] Ruohan Gao and Kristen Grauman.  Co-separating sounds of visual objects.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3879–3888, 2019.
  * Gan et al. [2020] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba.  Music gesture for visual sound separation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10478–10487, 2020.
  * Chatterjee et al. [2021] Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian.  Visual scene graphs for audio source separation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1204–1213, 2021.
  * Tian et al. [2021] Yapeng Tian, Di Hu, and Chenliang Xu.  Cyclic co-learning of sounding object visual grounding and sound separation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2745–2754, 2021.
  * Owens and Efros [2018] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 631–648, 2018.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Kong et al. [2020] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Jabri et al. [2022] Allan Jabri, David Fleet, and Ting Chen.  Scalable adaptive computation for iterative generation.  _arXiv preprint arXiv:2212.11972_ , 2022.
  * Tian et al. [2018] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu.  Audio-visual event localization in unconstrained videos.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 247–263, 2018.
  * Ephrat et al. [2018] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  _arXiv preprint arXiv:1804.03619_ , 2018.
  * Afouras et al. [2020] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In _European Conference on Computer Vision_ , pages 208–224. Springer, 2020.
  * Michelsanti et al. [2021] Daniel Michelsanti, Zheng-Hua Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen.  An overview of deep-learning-based audio-visual speech enhancement and separation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:1368–1396, 2021.
  * Zhao et al. [2019] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba.  The sound of motions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1735–1744, 2019.
  * Gao et al. [2018] Ruohan Gao, Rogerio Feris, and Kristen Grauman.  Learning to separate object sounds by watching unlabeled video.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 35–53, 2018.
  * Mittal et al. [2022] Himangi Mittal, Pedro Morgado, Unnat Jain, and Abhinav Gupta.  Learning state-aware visual representations from audible interactions.  In _Proceedings of the European conference on computer vision (ECCV)_ , 2022.
  * Dong et al. [2022] Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick.  Clipsep: Learning text-queried sound separation with noisy unlabeled videos.  _arXiv preprint arXiv:2212.07065_ , 2022.
  * Tzinis et al. [2020] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey.  Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds.  _arXiv preprint arXiv:2011.01143_ , 2020.
  * Tzinis et al. [2022] Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey.  Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII_ , pages 368–385. Springer, 2022.
  * Chatterjee et al. [2022] Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian.  Learning audio-visual dynamics using scene graphs for audio source separation.  In _NeurIPS_ , 2022.
  * Huang et al. [2023] Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu.  Egocentric audio-visual object localization.  _arXiv preprint arXiv:2303.13471_ , 2023.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18208–18218, 2022.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Gu et al. [2022] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10696–10706, 2022.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Ho et al. [2022] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv preprint arXiv:2204.03458_ , 2022.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Ruiz et al. [2022] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Austin et al. [2021] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.  Structured denoising diffusion models in discrete state-spaces.  _Advances in Neural Information Processing Systems_ , 34:17981–17993, 2021.
  * Gong et al. [2022] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong.  Diffuseq: Sequence to sequence text generation with diffusion models.  _arXiv preprint arXiv:2210.08933_ , 2022.
  * Li et al. [2022] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  _Advances in Neural Information Processing Systems_ , 35:4328–4343, 2022.
  * Chen et al. [2022a] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton.  Analog bits: Generating discrete data using diffusion models with self-conditioning.  _arXiv preprint arXiv:2208.04202_ , 2022a.
  * Popov et al. [2021] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In _International Conference on Machine Learning_ , pages 8599–8608. PMLR, 2021.
  * Lee and Han [2021] Junhyeok Lee and Seungu Han.  Nu-wave: A diffusion probabilistic model for neural audio upsampling.  _arXiv preprint arXiv:2104.02321_ , 2021.
  * Chen et al. [2020] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020.
  * Huang et al. [2022] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 2595–2605, 2022.
  * Scheibler et al. [2023] Robin Scheibler, Youna Ji, Soo-Whan Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi.  Diffusion-based generative speech source separation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE, 2023.
  * Ruan et al. [2022] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo.  Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation.  _arXiv preprint arXiv:2212.09478_ , 2022.
  * Amit et al. [2021] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf.  Segdiff: Image segmentation with diffusion probabilistic models.  _arXiv preprint arXiv:2112.00390_ , 2021.
  * Baranchuk et al. [2021] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  _arXiv preprint arXiv:2112.03126_ , 2021.
  * Brempong et al. [2022] Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, and Mohammad Norouzi.  Denoising pretraining for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4175–4186, 2022.
  * Chen et al. [2022b] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo.  Diffusiondet: Diffusion model for object detection.  _arXiv preprint arXiv:2211.09788_ , 2022b.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Qian et al. [2020] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin.  Multiple sound sources localization from coarse to fine.  In _European Conference on Computer Vision_ , pages 292–308. Springer, 2020.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Elfwing et al. [2018] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural Networks_ , 107:3–11, 2018.
  * Dumoulin et al. [2018] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio.  Feature-wise transformations.  _Distill_ , 3(7):e11, 2018.
  * Wang et al. [2023] Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeol Kim, and Shinji Watanabe.  Tf-gridnet: Making time-frequency domain models great again for monaural speaker separation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Tan et al. [2023] Reuben Tan, Arijit Ray, Andrea Burns, Bryan A. Plummer, Justin Salamon, Oriol Nieto, Bryan Russell, and Kate Saenko.  Language-guided audio-visual source separation via trimodal consistency.  2023\.
  * Raffel et al. [2014] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel.  Mir_eval: A transparent implementation of common mir metrics.  In _ISMIR_ , pages 367–372, 2014.
  * Shen et al. [2021] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Efficient attention: Attention with linear complexities.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 3531–3539, 2021.