# Multiplicative complements II.

Anett Kocsis , Dávid Matolcsi , Csaba Sándor and György Tőtős Eötvös Loránd
University, Budapest, Hungary. Email<EMAIL_ADDRESS>Supported by the
ÚNKP-21-1 New National Excellence Program of the Ministry for Innovation and
Technology from the source of the National Research, Development and
Innovation Fund. Eötvös Loránd University, Budapest, Hungary. Email:
<EMAIL_ADDRESS>Supported by the ÚNKP-21-1 New National Excellence
Program of the Ministry for Innovation and Technology from the source of the
National Research, Development and Innovation Fund.Department of Stochastics,
Institute of Mathematics, Budapest University of Technology and Economics,
Műegyetem rkp. 3., H-1111, Budapest, Hungary. Department of Computer Science
and Information Theory, Budapest University of Technology and Economics,
Műegyetem rkp. 3., H-1111 Budapest, Hungary, MTA-BME Lendület Arithmetic
Combinatorics Research Group, ELKH, Műegyetem rkp. 3., H-1111 Budapest,
Hungary . Email<EMAIL_ADDRESS>This author was supported by the NKFIH
Grants No. K129335. Faculty of Mathematics and Computer Science, Babeş-Bolyai
University.

###### Abstract

In this paper we prove that if $A$ and $B$ are infinite subsets of positive
integers such that every positive integer $n$ can be written as $n=ab$, $a\in
A$, $b\in B$, then $\displaystyle\lim_{x\to\infty}\frac{A(x)B(x)}{x}=\infty$.
We also prove many other results about sets like this.

2010 Mathematics Subject Classification: 11B34, 11N25

Keywords and phrases: Additive complements; counting function;

##  1 Introduction

The set of nonnegative integers is denoted by $\mathbb{N}$. The counting
function of a set $A\subseteq\mathbb{N}$ is defined as
$A(x)=|A\cap\\{0,1,\dots,x\\}|$ for every $x\in\mathbb{N}$. Let
$A,B\subseteq\mathbb{N}$. The sets $A$ and $B$ are said to be additive
complements if every nonnegative integers $n$ can be written as $n=a+b$, $a\in
A$, $b\in B$. Clearly, if $A,B\subseteq\mathbb{N}$ are additive complements,
then $A(x)B(x)\geq x+1$ for every $x\in\mathbb{N}$, therefore
$\displaystyle\liminf_{x\to\infty}\frac{A(x)B(x)}{x}\geq 1$. In 1964,
answering a question of Hanani, Danzer [1] proved that this bound is sharp.

######  Theorem 1 (Danzer, 1964).

There exist infinite additive complements $A,B\subseteq\mathbb{N}$ such that

$\lim_{x\to\infty}\frac{A(x)B(x)}{x}=1.$

In [2] we introduced the concept of multiplicative complements. Let us denote
by $\mathbb{Z}^{+}$ the set of positive integers and let
$A_{i}\subseteq\mathbb{Z}^{+}$ for every $1\leq i\leq h$. The $h$-tuple
$(A_{1},\dots,A_{h})$ form multiplicative complements of order $h$ if every
positive integers $n$ can be written as $n=a_{1}\dots a_{h}$, $a_{i}\in
A_{i}$. For brevity, we will use the notation $MC_{h}$ for the set of
multiplicative complements of order $h$. Similar to the additive complements,
if $(A,B)\in MC_{2}$, then we have $A(x)B(x)\geq x$, therefore

$\liminf_{x\to\infty}\frac{A(x)B(x)}{x}\geq 1.$

We show that, in contrast to the additive complements,

$\lim_{x\to\infty}\frac{A(x)B(x)}{x}=\infty$

for every infinite $(A,B)\in MC_{2}$.

We proved in [2] the following statement:

######  Theorem 2.

For every $\varepsilon>0$, there exists infinite $(A,B)\in MC_{2}$ such that

$\liminf_{x\to\infty}\frac{\max\\{A(x)B(x)\\}}{\frac{x}{\log x}}\leq
0.5+\varepsilon.$

Clearly, for every $(A,B)\in MC_{2}$ we have $\log min\\{A(x),B(x)\\}\leq\log
x$. It follows that

######  Corollary 3.

For every $\varepsilon>0$, there exists infinite $(A,B)\in MC_{2}$ such that

$\liminf_{x\to\infty}\frac{\max\\{A(x)B(x)\\}\log\min\\{A(x),B(x)\\}}{x}\leq
0.5+\varepsilon.$

We show that this bound can not be improved.

######  Theorem 4.

For every infinite $(A,B)\in MC_{2}$,

$\liminf_{x\to\infty}\frac{\max\\{A(x)B(x)\\}\log\min\\{A(x),B(x)\\}}{x}>0.5.$

If $|A|=\infty$ and $|B|=\infty$, then
$\displaystyle\lim_{x\to\infty}\frac{\min\\{A(x),B(x)\\}}{\log\min\\{A(x),B(x)\\}}=\infty$.
It follows that

######  Corollary 5.

For every infinite $(A,B)\in MC_{2}$,

$\lim_{x\to\infty}\frac{A(x)B(x)}{x}=\infty.$

In Theorem 4 we show that for infinite $(A,B)\in MC_{2}$, the fraction
$\frac{\max\\{A(x)B(x)\\}\log\min\\{A(x),B(x)\\}}{x}>0.5$ if $x$ is large
enough. Next theorem shows that this fraction can be arbitrary large.

######  Theorem 6.

For every infinite $(A,B)\in MC_{2}$,

$\limsup_{x\to\infty}\frac{\max\\{A(x)B(x)\\}\log\min\\{A(x),B(x)\\}}{x}=\infty.$

We proved in [2] that

######  Theorem 7.

  1. 1.

For every $(A,B)\in MC_{2}$,

$\limsup_{x\to\infty}\frac{max\\{A(x),B(x)\\}}{\frac{x}{\sqrt{\log
x}}}\geq\frac{1}{\sqrt{\pi}}.$

  2. 2.

There exists an $(A,B)\in MC_{2}$ such that

$A(x)=\left(\frac{1}{\sqrt{\pi}}+o(1)\right)\frac{x}{\sqrt{\log x}}\quad\hbox{
and }\quad B(x)=\left(\frac{1}{\sqrt{\pi}}+o(1)\right)\frac{x}{\sqrt{\log
x}}.$

In the next three theorems we consider the function $\min\\{A(x),B(x)\\}$. It
is easy to see that for every $\varepsilon>0$, there exists an $(A,B)\in
MC_{2}$ such that $|B|<\infty$ and
$\displaystyle\limsup_{x\to\infty}\frac{A(x)}{x}<\varepsilon$. On the other
hand, if $|B|<\infty$ and $(A,B)\in MC_{2}$, then
$\displaystyle\liminf_{x\to\infty}\frac{A(x)}{x}>0$. Therefore, the natural
requirement is that $A(x)=o(x)$ and $B(x)=o(x)$.

######  Theorem 8.

Let $f(x)$ be a function such that $f(x)\to\infty$ as $x\to\infty$. Then there
exists infinite $(A,B)\in MC_{2}$ such that $A(x)=o(x)$, $B(x)=o(x)$ and

$\liminf_{x\to\infty}\frac{\min\\{A(x),B(x)\\}}{f(x)}=0.$

As a corollary, we get that in Corollary 5 the function $x$ can not be
replaced by any other function $g(x)$, where $\frac{g(x)}{x}\to\infty$ as
$x\to\infty$.

######  Corollary 9.

Let $g(x)$ be a function such that
$\displaystyle\lim_{x\to\infty}\frac{g(x)}{x}=\infty$. Then there exist an
$(A,B)\in MC_{2}$ such that

$\limsup_{x\to\infty}\frac{A(x)B(x)}{g(x)}=0.$

The following theorem shows that for $(A,B)\in MC_{2}$ the function
$\min\\{A(x),B(x)\\}$ can be arbitrary large.

######  Theorem 10.

Let us suppose that for some function $f(x)>0$, $x\geq 1$, the series
$\displaystyle\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}}$ converges. Then there is
no $(A,B)\in MC_{2}$ such that $A(x)=O(f(x))$ and $B(x)=o(x)$.

As a corollary, we get

######  Corollary 11.

Let $\varepsilon>0$. Then there is no infinite $(A,B)\in MC_{2}$, $B(x)=o(x)$
such that $A(x)=O\left(\frac{x}{(\log x)(\log\log x)^{1+\varepsilon}}\right)$.

If the function $f(x)$ satisfies some smoothness conditions and
$\displaystyle\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}}=\infty$, then we can find
$(A,B)\in\mathbb{Z}^{+}$ such that $A(x)=O(f(x))$ and $B(x)=o(x)$ as
$x\to\infty$.

######  Theorem 12.

Let us suppose that the function $f(x)>0$ satisfies the following conditions

  1. 1.

$f(x)$ is monotonically increasing for $x\geq 1$,

  2. 2.

$f(x)=o(\frac{x}{\log x})$ as $x\to\infty$,

  3. 3.

there exist $c_{1},c_{2}>0$ such that
$c_{1}\leq\frac{\frac{f(x_{2})}{f(x_{1})}}{\frac{x_{2}}{x_{1}}}\leq c_{2}$ for
$1\leq x_{1}\leq x_{2}\leq x_{1}^{2}$,

  4. 4.

$\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}}=\infty$.

Then there exists $(A,B)\in MC_{2}$ such that $A(x)=O(f(x))$ and $B(x)=o(x)$.

As a corollary, we get

######  Corollary 13.

There exists $(A,B)\in MC_{2}$ such that $A(x)=O(\frac{x}{(\log x)(\log\log
x)})$ and $B(x)=o(x)$.

In the last theorem, we consider $(A,B)\in MC_{2}$ such that $|B|<\infty$.

######  Theorem 14.

  1. 1.

There exists a $c_{1}>0$ such that for every $(A,B)\in MC_{2}$,
$2\leq|B|<\infty$, we have

$\frac{A(x)\log|B|}{x}\geq c_{1}$

for every $x\in\mathbb{Z}^{+}$.

  2. 2.

There exists a constant $c_{2}$ such that for every $N\geq 3$, there exists
$(A,B)\in MC_{2}$, $|B|=N$ such that

$\frac{A(x)\log\log|B|}{x}\leq c_{2}$

for every $x\geq 10$.

We finally pose some open problems for further research. As a corollary of
Theorem 7, we get the following statement:

######  Corollary 15.

There exists $(A,B)\in MC_{2}$ such that

$\lim_{x\to\infty}\frac{max\\{A(x),B(x)\\}\sqrt{\log\min\\{A(x),B(x)\\}}}{x}=\frac{1}{\sqrt{\pi}}.$

Inspired by this, Theorem 6 may be sharpened as follows:

######  Problem 1.

Is it true that for any infinite $(A,B)\in MC_{2}$

$\limsup_{x\to\infty}\frac{max\\{A(x),B(x)\\}\sqrt{\log\min\\{A(x),B(x)\\}}}{x}>0?$

It follows from Theorem 14 that

$\frac{c_{1}}{\log k}\leq\inf_{B:|B|=k}\\{\inf_{A:(A,B)\in
MC_{2}}\\{\limsup_{x\to\infty}\frac{A(x)}{x}\\}\\}\leq\frac{c_{2}}{\log\log
k}.$

It would be interesting to determine the right magnitude of the above infimum.

######  Problem 2.

Find the magnitude of the function

$f(k)=\inf_{B:|B|=k}\\{\inf_{A:(A,B)\in
MC_{2}}\\{\limsup_{x\to\infty}\frac{A(x)}{x}\\}\\}.$

In Theorem 12 we have some smoothness assumptions. Is it true that we can omit
these conditions?

######  Problem 3.

Is it true that if $f(x)>0$, $x\geq 1$ is a monotonically increasing function
such that $\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}}=\infty$, then there exists an
$(A,B)\in MC_{2}$ such that $A(x)=O(f(x))$ and $B(x)=o(x)$?

##  2 Proofs

Throughout the paper the set of primes is denoted by $P$ and $p$ will always
denote a prime number.

###### Proof of Theorem 4.

The set $S\subseteq\mathbb{Z}^{+}$ is called multiplicative basis of order 2
if every positive integer $n$ can be written as $n=ss^{\prime}$, where
$s,s^{\prime}\in S$. The set of multiplicative bases of order 2 is denoted by
$MB_{2}$. Pach and Sándor [3] proved that if $S\in MB_{2}$, then
$\displaystyle\liminf_{x\to\infty}\frac{S(x)}{\frac{x}{\log x}}>1$. For
$(A,B)\in MC_{2}$, we have $A\cup B\in MB_{2}$ and $(A\cup B)(x)\leq
A(x)+B(x)$. It follows that

$\liminf\limits_{x\to\infty}\frac{A(x)+B(x)}{\frac{x}{\log x}}>1,$

that is, there exists a $\delta>0$ such that

$\displaystyle\liminf\limits_{x\to\infty}\frac{A(x)+B(x)}{\frac{x}{\log
x}}\geq 1+\delta.$ (2.1)

Let $\beta>0$ be a constant such that

$\frac{1}{2+\delta}<\frac{1}{2}-\beta<\frac{1}{2+\frac{\delta}{2}}.$

In order to prove the theorem, we define a partition of $\mathbb{Z}^{+}$. Let

$H_{1}=\\{x\in\mathbb{Z}^{+}:\frac{\delta}{2}\frac{x}{\log
x}\leq\min\\{A(x),B(x)\\}\\}$
$H_{2}=\\{x\in\mathbb{Z}^{+}:\frac{x^{\frac{1}{2}-\beta}}{\log^{2}x}\leq\min\\{A(x),B(x)\\}<\frac{\delta}{2}\frac{x}{\log
x}\\}$
$H_{3}=\\{x\in\mathbb{Z}^{+}:\min\\{A(x),B(x)\\}<\frac{x^{\frac{1}{2}-\beta}}{\log^{2}x}\\}.$

Because of (2.1),

$\liminf_{x\in H_{1},x\to\infty}\frac{\max\\{A(x),B(x)\\}}{\frac{x}{\log
x}}\geq\frac{1}{2}+\frac{\delta}{2}$

It is easy to see from the definition of $H_{1}$ that

$\liminf_{x\in H_{1},x\to\infty}\frac{\log\min\\{A(x),B(x)\\}}{\log x}=1.$

Hence

$\liminf_{x\in
H_{1},x\to\infty}\frac{\max\\{A(x),B(x)\\}\log\min\\{A(x),B(x)\\}}{x}\geq\frac{1}{2}+\frac{\delta}{2}.$

In the second case, it follows from (2.1) that

$\liminf_{x\in H_{2},x\to\infty}\frac{\max\\{A(x),B(x)\\}}{\frac{x}{\log
x}}\geq 1+\frac{\delta}{2}$

and it is easy to see from the definition of $H_{2}$ that

$\liminf_{x\in H_{2},x\to\infty}\frac{\log\min\\{A(x),B(x)\\}}{\log
x}\geq\frac{1}{2}-\beta.$

Hence

$\liminf_{x\in
H_{2},x\to\infty}\frac{\max\\{A(x),B(x)\\}\log\min\\{A(x),B(x)\\}}{x}\geq\left(1+\frac{\delta}{2}\right)\left(\frac{1}{2}-\beta\right)>\frac{1}{2}.$

It remains to consider the case $x\in H_{3}$ as $x\to\infty$. In this case

$\min\\{A(x),B(x)\\}\log^{2}\min\\{A(x),B(x)\\}<x^{\frac{1}{2}-\beta}.$

Let us suppose that $\min\\{A(x),B(x)\\}=B(x)$ for a given $x\in H_{3}$. Then

$\frac{\log x}{\log(B(x)\log^{2}B(x))}\geq 2+\frac{\delta}{2}.$ (2.2)

Clearly,

$\begin{split}x&\leq\phantom{=}|\\{n\leq x:\exists(a,b)\in A\times B\colon
n=ab,b\geq B(x)\log^{2}B(x)\\}|+\\\ &\phantom{=}+|\\{n\leq x:\exists(a,b)\in
A\times B\colon n=ab,1<b<B(x)\log^{2}B(x)\\}|+\\\ &\phantom{=}+|\\{n\leq
x:\exists(a,b)\in A\times B\colon n=ab,b=1\\}|.\end{split}$ (2.3)

The first term in (2.3) can be estimated by

$\displaystyle|\\{n\leq x:\exists(a,b)\in A\times B\colon n=ab,b\geq
B(x)\log^{2}B(x)\\}|$ $\displaystyle\leq\sum_{B(x)\log^{2}B(x)\leq b\leq
x,b\in B}\frac{x}{b}$
$\displaystyle\leq\frac{x}{B(x)\log^{2}B(x)}B(x)\leq\frac{x}{\log^{2}B(x)}.$

The second term in (2.3) can be estimated by

$\displaystyle|\\{n\leq x:$ $\displaystyle\exists(a,b)\in A\times B\colon
n=ab,1<b<B(x)\log^{2}B(x)\\}|\leq$ $\displaystyle\leq|\\{n\leq x:\exists p\in
P\colon p\mid n,1<p<B(x)\log^{2}B(x)\\}|.$

The third term in (2.3) is

$|\\{n\leq x\mid\exists(a,b)\in A\times B\colon n=ab,b=1\\}|=A(x),$

and so

$\displaystyle x\leq\frac{x}{\log^{2}B(x)}+|\\{n\leq x:\exists p\in P\colon
p\mid n,1<p<B(x)\log^{2}B(x)\\}|+A(x)$ (2.4)

On the other hand,

$\begin{split}x=&1+|\\{n\leq x:\exists p\in P\colon p\mid
n,1<p<B(x)\log^{2}B(x)\\}|\\\ &+|\\{n\leq x\mid\forall p\in P\colon p\mid
n\implies p\geq B(x)\log^{2}B(x)\\}|.\end{split}$ (2.5)

It follows from (2.4) and (2.5) that

$|\\{n\leq x:\forall p\in P\colon p\mid n\implies p\geq
B(x)\log^{2}B(x)\\}|+1-\frac{x}{\log^{2}B(x)}\leq A(x).$ (2.6)

To estimate the first term we need the Buchstab function. The Buchstab
function is the unique continuous function
$\omega:[1,\infty[\to\mathbb{R}^{+}$ defined by the delay differential
equation

$\omega(u)=\frac{1}{u}\quad\hbox{for $1\leq u\leq 2$}$
$\frac{d}{du}(u\omega(u))=\omega(u-1)\quad\hbox{for $u\geq 2$.}$

In the second equation, the derivative at $u=2$ should be taken as $u$
approaches $2$ from the right. Denote by $\varphi(x,y)$ the number of positive
integers less than or equal to $x$ with no prime factor less than $y$. In
1970, Warlimont [4] proved the following remarkable result.

######  Theorem 16 (Warlimont, 1970).

Let $u_{0}>1$. There exists a $C_{0}=C_{0}(u_{0})$ such that if $y\geq 2$,
$\frac{\log x}{\log y}\geq u_{0}$ then

$|\varphi(x,y)-e^{\gamma}\omega(u)x\prod_{p<y}(1-\frac{1}{p})|\leq\frac{C_{0}x\prod_{p<y}\left(1-\frac{1}{p}\right)}{\log
x}.$

It is well known that the minimum of $\omega(u)$ is taken at $u=2$,
$\omega(2)=0.5$ and for every $\mu>0$, there exists an $\varepsilon>0$ such
that for $u\geq 2+\mu$, we have $\omega(u)\geq 0.5+\varepsilon$. Let us
suppose that $\omega(u)\geq 0.5+\varepsilon_{0}$ if $u\geq
2+\frac{\delta}{2}$. According to the Warlimont’s theorem and (2.2) we get
that

$|\\{n\leq x\mid\forall p\in P\colon p\mid n\implies p\geq
B(x)\log^{2}B(x)\\}|=\varphi(x,B(x)\log^{2}B(x))\geq$
$e^{\gamma}\omega\left(\frac{\log
x}{\log(B(x)\log^{2}B(x))}\right)x\prod_{p<B(x)\log^{2}B(x)}(1-\frac{1}{p})-\frac{C_{0}x\prod_{p<B(x)\log^{2}B(x)}\left(1-\frac{1}{p}\right)}{\log
x}\geq$
$(0.5+\varepsilon_{0}+o(1))e^{\gamma}x\prod_{p<B(x)\log^{2}B(x)}(1-\frac{1}{p}),$

where $\gamma$ is the Euler-Mascheroni constant.

It is well-known that

$\prod_{p<y,\>p\in P}(1-\frac{1}{p})=(1+o(1))\frac{e^{-\gamma}}{\log y}$

as $y\to\infty$, so

$|\\{n\leq x\mid\forall p\in P\colon p\mid n\implies p\geq
B(x)\log^{2}B(x)\\}|\geq(0.5+\varepsilon_{0}+o(1))\frac{x}{\log\left(B(x)\log^{2}B(x)\right)}=$
$(0.5+\varepsilon_{0}+o(1))\frac{x}{\log B(x)}$

as $x\to\infty$. By (2.6),

$A(x)\geq 1-\frac{x}{\log^{2}B(x)}+(0.5+\varepsilon_{0}+o(1))\frac{x}{\log
B(x)}=(0.5+\varepsilon_{0}+o(1))\frac{x}{\log B(x)}.$

Hence we have

$(0.5+\varepsilon_{0}+o(1))x\leq A(x)\log B(x).$

It follows that

$\liminf_{x\in
H_{3},x\to\infty}\frac{\max\\{A(x),B(x)\\}\log\min\\{A(x),B(x)\\}}{x}\geq
0.5+\varepsilon_{0}.$

This means that

$\liminf\limits_{x\to\infty}\frac{\max\\{A(x),B(x)\\}\log\min\\{A(x),B(x)\\}}{x}\geq\min\\{\frac{1}{2}+\frac{\delta}{2},(1+\frac{\delta}{2})(\frac{1}{2}-\beta),\frac{1}{2}+\varepsilon_{0}\\}>\frac{1}{2},$

which completes the proof.

∎

###### Proof of Theorem 6.

If $A(x)\neq o(x)$ or $B(x)\neq o(x)$, then $\max\\{A(x),B(x)\\}\neq o(x)$ and
$\min\\{A(x),B(x)\\}\to\infty$, the conclusion directly follows. So we may
suppose that $A(x)=o(x)$ and $B(x)=o(x)$. It follows from Theorem 7 that

$\limsup_{x\to\infty}\frac{\max\\{A(x),B(x)\\}}{\frac{x}{\sqrt{\log
x}}}\geq\frac{1}{\sqrt{\pi}}.$ (2.7)

If $\displaystyle\liminf_{x\to\infty}\frac{\log\min\\{A(x),B(x)\\}}{\sqrt{\log
x}}=\infty$, then we are done. So we can suppose that there exists a constant
$D$ such that for infinitely many $x\in\mathbb{Z}^{+}$,

$\frac{\textrm{min}\\{A(x),B(x)\\}}{e^{D\sqrt{\log x}}}<1.$

Without loss of generality, we can assume that

$\frac{B(x)}{e^{D\sqrt{\log x}}}<1$

for infinitely many $x\in\mathbb{Z}^{+}$.

If $A(x)=o(x)$, then there exist infinitely many $x\in\mathbb{Z}^{+}$ such
that $B(x)>\sqrt{x}$, otherwise for $B=\\{b_{1},b_{2},\dots\\}$,
$b_{1}<b_{2}<\dots$ there exists an $n_{0}$ such that for $n\geq n_{0}$, we
have $n=B(b_{n})\leq\sqrt{b_{n}}$, that is $b_{n}\geq n^{2}$. It follows that

$\sum_{b\in
B}\frac{1}{b}=\sum_{n=1}^{n_{0}-1}\frac{1}{b_{n}}+\sum_{n=n_{0}}^{\infty}\frac{1}{b_{n}}\leq\sum_{n=1}^{n_{0}-1}\frac{1}{b_{n}}+\sum_{n=n_{0}}^{\infty}\frac{1}{n^{2}}<\infty,$

therefore $\sum_{b\in B}\frac{1}{b}$ convergent. Then there exists an $N_{1}$
such that $\sum_{b\geq N_{1},b\in B}\frac{1}{b}<\frac{1}{2}$. Clearly

$x\leq|\\{n:n\leq x,\exists b<N_{1},b\in B,a\in A,n=ab\\}|+|\\{n:n\leq
x,\exists b\geq N_{1},b\in B,a\in A,n=ab|\leq$ $\sum_{b<N_{1},b\in
B}A(\frac{x}{b})+\sum_{b\geq N_{1},b\in B}A(\frac{x}{b})\leq o(x)+\sum_{b\geq
N_{1},b\in B}\frac{x}{b}\leq(0.5+o(1))x,$

a contradiction.

So we may assume that there are infinitely many positive integers
$y_{1}<y_{2}<y_{3}<\ldots$ such that $0.5e^{D\sqrt{\log
y_{n}}}<B(y_{n})<e^{D\sqrt{\log y_{n}}}$ and $x_{1}<x_{2}<x_{3}<\dots$ such
that $0.5\sqrt{x_{n}}<B(x_{n})<\sqrt{x_{n}}$ with $e^{\sqrt{\log
x_{n}}}>y_{n}$ for every $n$.

We are going to estimate $A(x_{n})$. We need the following sets:

$C_{n}=\\{p:p>y_{n}\textrm{ and }\exists m\in B\cap\\{1,\dots,x_{n}\\}\textrm{
such that }p\textrm{ is the largest prime factor of }m\\},$

$D_{n}=\\{p:p<y_{n}\textrm{ and }p\in B\\},$

$E_{n}=\\{p:p<y_{n}\\}.$

It is easy to check that if $k$ is a positive integer such that $k<x_{n}$ and
$k=pq$, where $p$ and $q$ are primes, $p\in E_{n}\setminus D_{n}$, $q\notin
C_{n}$ and $q>y_{n}$, then $k$ must belong to the set $A$. Now we want to
estimate the number of such $k$’s. Clearly,

$A(x_{n})\geq\sum\limits_{p\in E_{n}\setminus
D_{n}}\left(\pi\left(\frac{x_{n}}{p}\right)-|C_{n}|-\pi(y_{n})\right)$

If $x_{n}$ is large enough, then for any $p\in E_{n}\setminus D_{n}$,

$\pi\left(\frac{x_{n}}{p}\right)=(1+o(1))\frac{x_{n}/p}{\log(x_{n}/p)}=(1+o(1))\frac{1}{p}\frac{x_{n}}{\log
x_{n}-\log p}>(1+o(1))\frac{1}{p}\frac{x_{n}}{\log
x_{n}}\geq\frac{1}{p}\frac{x_{n}}{2\log x_{n}}.$

Clearly,

$|C_{n}|\leq B(x_{n})<\sqrt{x_{n}},$

and

$\pi(y_{n})\leq y_{n}<e^{\sqrt{\log x_{n}}}.$

It follows that

$A(x_{n})>\sum\limits_{p\in E_{n}\setminus
D_{n}}\left(\frac{1}{p}\frac{x_{n}}{2\log x_{n}}-\sqrt{x_{n}}-e^{\sqrt{\log
x_{n}}}\right)>\frac{x_{n}}{3\log x_{n}}\sum\limits_{p\in E_{n}\setminus
D_{n}}\frac{1}{p}$

if $n$ is large enough. It is well known that
$\sum\limits_{p<N}\frac{1}{p}=\log\log N+O(1)$ and
$\sum\limits_{i=1}^{N}\frac{1}{p_{i}}=\log\log N+O(1)$ as well. Since
$|D_{n}|\leq B(y_{n})<e^{D\sqrt{\log y_{n}}}$, there are at most
$e^{\sqrt{\log y_{n}}}$ primes in the set $D_{n}$, so $\sum_{p\in
D_{n}}\frac{1}{p}\leq\sum\limits_{i=1}^{e^{\sqrt{\log
y_{n}}}}\frac{1}{p_{i}}$. Therefore

$\sum\limits_{p\in E_{n}\setminus D_{n}}\frac{1}{p}\geq\sum\limits_{p\in
E_{n}}\frac{1}{p}-\sum\limits_{i=1}^{e^{\sqrt{\log
y_{n}}}}\frac{1}{p_{i}}=\log\log y_{n}-\log\log e^{\sqrt{\log
y_{n}}}+O(1)=\frac{\log\log y_{n}}{2}+O(1).$

It follows that

$A(x_{n})\geq\frac{x_{n}}{6\log x_{n}}(\log\log y_{n}+O(1)).$

We have supposed that $B(x_{n})\geq 0.5\sqrt{x_{n}}$. Hence

$\frac{A(x_{n})\log B(x_{n})}{x_{n}}\geq\frac{\log\log y_{n}}{12}+O(1).$

This completes the proof, because $y_{n}\to\infty$ as $n\to\infty$. ∎

It order to prove Theorems 8 and 12 we need the following lemma.

######  Lemma 1.

Let $Q$ be a subset of prime numbers such that $\sum_{q\in
Q}\frac{1}{q}=\infty$. Let

$A=\\{n:\hbox{ $n$ is squarefree },n\in\mathbb{Z}^{+},p|n\Rightarrow p\in
Q\\}.$

Then there exists a set $B\subseteq\mathbb{Z}^{+}$ such that $(A,B)\in MC_{2}$
and $B(x)=o(x)$.

###### Proof.

Let $Q=\\{q_{1},q_{2},\dots\\}$, $q_{1}<q_{2}<\dots$. Define
$Q_{k}=\\{q_{1},\dots,q_{k}\\}$ for every $k\in\mathbb{Z}^{+}$. Let

$A_{k}=\\{n:\hbox{ $n$ is squarefree },n\in\mathbb{Z}^{+},p|n\Rightarrow p\in
Q_{k}\\}.$

Let

$B_{k}=\\{n:n=ml^{2},\hbox{$m$ is squarefree and }p|m\Rightarrow p\notin
Q_{k}\\}.$

Then $(A_{k},B_{k})\in MC_{2}$, $A_{1}\subseteq A_{2}\subseteq\dots$,
$\displaystyle A=\cup_{k=1}^{\infty}A_{k}$ and $B_{1}\supseteq
B_{2}\supseteq\dots$. Let $\displaystyle B=\cap_{k=1}^{\infty}B_{k}.$ Then
$(A,B)\in MC_{2}$. We are going to prove that for any $\varepsilon>0$, there
exists an $x_{0}=x_{0}(\varepsilon)$ such that

$B(x)\leq\varepsilon x$ (2.8)

for $x\geq x_{0}$.

Let $k$ be an integer such that
$4e^{-\sum_{i=1}^{k}\frac{1}{2q_{i}}}<\varepsilon$. Let
$1=b_{1}^{(k)}<b_{2}^{(k)}<\dots$ be the squarefree integers in the set
$B_{k}$. Then

$B(x)\leq B_{k}(x)\leq\sum_{b_{l}^{(k)}\leq x}\sqrt{\frac{x}{b_{l}^{(k)}}}.$

It follows from $\gcd(b_{l}^{(k)},q_{1}q_{2}\dots q_{k})=1$

$|\\{b_{1}^{(k)},b_{2}^{(k)},\dots\\}\cap\\{Nq_{1}\dots
q_{k}+1,\dots,(N+1)q_{1}\dots q_{k}\\}|\leq(q_{1}-1)\dots(q_{k}-1)$

for any nonnegative integer $N$. If $M(q_{1}-1)\dots(q_{k}-1)\leq
l<(M+1)(q_{1}-1)\dots(q_{k}-1)$, then

$b_{l}^{(k)}\geq(M+1)q_{1}\dots q_{k}-q_{1}\dots q_{k}\geq
l\prod_{i=1}^{k}\frac{q_{i}}{q_{i}-1}-q_{1}\dots q_{k}\geq
le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}-q_{1}\dots q_{k}.$

If $l>2q_{1}\dots q_{k}$, then

$\frac{1}{\sqrt{b_{l}^{(k)}}}\leq\frac{1}{\sqrt{le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}-q_{1}\dots
q_{k}}}=\frac{1}{\sqrt{le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}(1-\frac{q_{1}\dots
q_{k}}{le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}})}}\leq$
$\frac{1}{\sqrt{l}e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}\left(1-\frac{q_{1}\dots
q_{k}}{le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}}\right)}\leq\frac{1}{\sqrt{l}e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}}+\frac{1}{\sqrt{l}e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}}\frac{2q_{1}\dots
q_{k}}{le^{\sum_{i=1}^{k}\frac{1}{q_{i}}}}.$

Hence

$B(x)\leq\sum_{l\leq 2q_{1}\dots
q_{k}}\sqrt{\frac{x}{b_{l}^{(k)}}}+\sum_{2q_{1}\dots q_{k}<l\leq
x}\sqrt{\frac{x}{b_{l}^{(k)}}}\leq O(\sqrt{x})+\sum_{2q_{1}\dots q_{k}<l\leq
x}\left(\frac{\sqrt{x}}{\sqrt{l}e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}}+\frac{\sqrt{x}}{l^{3/2}e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}}\right)\leq$
$O(\sqrt{x})+\frac{\sqrt{x}}{e^{\sum_{i=1}^{k}\frac{1}{2q_{i}}}}\int_{0}^{x}\frac{1}{\sqrt{t}}dt+O(1)=O(\sqrt{x})+2xe^{-\sum_{i=1}^{k}\frac{1}{2q_{i}}}\leq\frac{\varepsilon}{2}x+O(\sqrt{x}),$

therefore there exists an $x_{0}=x_{0}(\varepsilon)$ such that (2.8) holds. ∎

###### Proof of Theorem 8.

Let us choose sequences $M_{1}<M_{2}<\dots$ and $N_{1}<N_{2}<\dots$ such that
$M_{k}<N_{k}$, $\displaystyle\sum_{M_{k}<p\leq N_{k}}\frac{1}{p}\geq 1$,
$\displaystyle\prod_{p\leq N_{k-1}}p<M_{k}$ and $k2^{N_{k-1}}<f(M_{k})$ for
every $k\in\mathbb{Z}^{+}$. Let

$Q=\\{p:\hbox{ there exists a $k\in\mathbb{Z}^{+}$ such that }M_{k}<p\leq
N_{k}\\}.$

Let

$A=\\{n:n\in\mathbb{Z}^{+},\hbox{ $n$ is squarefree }p|n\Rightarrow p\in
Q\\}.$

By construction, we have $\displaystyle\sum_{q\in
Q}\frac{1}{q}=\sum_{k=1}^{\infty}\sum_{M_{k}<p\leq
N_{k}}\frac{1}{p}\geq\sum_{k=1}^{\infty}1=\infty$. By Lemma 1, we know that
there exists a set $B\in\mathbb{Z}^{+}$ such that $(A,B)\in MC_{2}$ and
$B(x)=o(x)$ as $x\to\infty$. Moreover,

$\frac{A(M_{k})}{f(M_{k})}\leq\frac{2^{\pi(N_{k-1})}}{f(M_{k})}<\frac{2^{N_{k-1}}}{k2^{N_{k-1}}}=\frac{1}{k}$

for every $k\in\mathbb{Z}^{+}$, therefore

$\liminf_{x\to\infty}\frac{A(x)}{f(x)}=0,$

which completes the proof. ∎

###### Proof of Theorem 10.

Let us suppose that for the function $f(x)>0$, $x\geq 1$ the series
$\displaystyle\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}}$ is finite. Let us suppose
that for some $(A,B)\in MC_{2}$, we have $A(x)\leq Cf(x)$ for every
$x\in\mathbb{Z}^{+}$ and $B(x)=o(x)$. Then

$\sum_{a\leq N,a\in
A}\frac{1}{a}=\sum_{n=1}^{N}\frac{A(n)-A(n-1)}{n}=\sum_{n=1}^{N-1}\frac{A(n)}{n(n+1)}+\frac{A(N)}{N}\leq
1+C\sum_{n=1}^{\infty}\frac{f(n)}{n^{2}},$

therefore the series $\sum_{a\in A}\frac{1}{a}$ converges. It follows that
there exists an $n_{1}$ such that $\sum_{a\geq n_{1},a\in
A}\frac{1}{a}<\frac{1}{2}$. Clearly

$x\leq|\\{n:n\leq x,\exists a<n_{1},a\in A,b\in B,n=ab\\}|+|\\{n:n\leq
x,\exists a\geq n_{1},a\in A,b\in B,n=ab|\leq$ $\sum_{a<n_{1},a\in
A}B(\frac{x}{a})+\sum_{a\geq n_{1},a\in A}B(\frac{x}{a})\leq o(x)+\sum_{a\geq
n_{1},a\in A}\frac{x}{a}\leq(0.5+o(1))x,$

a contradiction. ∎

###### Proof of Theorem 12.

We are going to find a set $Q\subseteq P$ such that

$\sum_{q\in Q}\frac{1}{q}=\infty$ (2.9)

and the set

$A=\\{n:n\hbox{ is squarefree }p|n\Rightarrow p\in Q\\}$

satisfies

$A(x)=O(f(x)).$ (2.10)

Then by Lemma 1, there exists a $B\subseteq\mathbb{Z}^{+}$ such that $(A,B)\in
MC_{2}$ and $B(x)=o(x)$.

Let $\chi_{Q}(n)=\left\\{\begin{array}[]{ll}1,&\mbox{if $n\in Q$}\\\ 0,&\mbox{
if $n\notin Q$}\end{array}\right.$. Let

$A_{k}=\\{n:n\in A,n\hbox{ has $k$ different prime factors}\\}.$

To prove (2.10) it is enough to verify

$A_{k}(x)\leq\frac{e^{\frac{1}{c_{1}}}f(x)\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)^{k-1}}{(k-1)!exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}$
(2.11)

holds for $x\geq 1$ and $k\in\mathbb{Z}^{+}$, because

$A(x)\leq 1+\sum_{k=1}^{\infty}A_{k}(x)\leq
1+\sum_{k=1}^{\infty}\frac{e^{\frac{1}{c_{1}}}f(x)\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)^{k-1}}{(k-1)!exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}=1+e^{\frac{1}{c_{1}}}f(x).$

First, we define a set $R\subseteq P$, then by truncating it we get the new
set $Q$. The set $R$ is defined recursively. We run over the primes $p$ and
the prime number $p$ is included in the set $R$ if and only if

$R(p-1)+1\leq\frac{f(p)}{exp\left(\frac{2}{c_{1}}\sum_{n<p}\frac{\chi_{R}(n)}{n}\right)}.$

First, we show that

$\sum_{r\in R}\frac{1}{r}=\infty.$

By contradiction, let us suppose that $\sum_{r\in R}\frac{1}{r}=K<\infty$. Let
us suppose that $p\in R$ and the next prime in the set $R$ is $p^{\prime}$.
Let $p\leq x<p^{\prime}$. Then

$R(x)=R(p)=R(p-1)+1\leq\frac{f(p)}{exp\left(\frac{2}{c_{1}}\sum_{n<p}\frac{\chi_{R}(n)}{n}\right)}\leq\frac{f(x)}{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{R}(n)}{n}-\frac{2}{c_{1}}\frac{1}{p}\right)}=$
$\frac{e^{\frac{2}{c_{1}}\frac{1}{p}}f(x)}{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{R}(n)}{n}\right)}\leq\frac{e^{\frac{1}{c_{1}}}f(x)}{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{R}(n)}{n}\right)}\leq
e^{\frac{1}{c_{1}}}f(x).$

It follows that for any $x\geq 1$,

$R(x)\leq e^{\frac{1}{c_{1}}}f(x)=o(\frac{x}{\log x}).$

It is well known that $\pi(n)-\pi(\frac{n}{2})=(0.5+o(1))\frac{n}{\log n}$ as
$n\to\infty$, and so there exists an $n_{1}$ such that for any $n\geq n_{1}$
there exists a prime number $p$ in the interval $[0.5n,n]$ such that $p\notin
R$. Then by the definition of set $R$ and condition 3,

$R(p-1)+1>\frac{f(p)}{exp\left(\frac{2}{c_{1}}\sum_{n<p}\frac{\chi_{R}(n)}{n}\right)}>\frac{f(p)}{e^{\frac{2K}{c_{1}}}}=\frac{\frac{f(p)}{f(n)}}{\frac{p}{n}}\frac{p}{n}\frac{1}{e^{\frac{2K}{c_{1}}}}f(n)\geq\frac{c_{1}f(n)}{2e^{\frac{2K}{c_{1}}}}.$

If $f(n)>\frac{6e^{\frac{2K}{c_{1}}}}{c_{1}}$, then

$R(n)\geq
R(p-1)\geq\frac{c_{1}f(n)}{2e^{\frac{2K}{c_{1}}}}-1\geq\frac{c_{1}f(n)}{2e^{\frac{2K}{c_{1}}}}-\frac{c_{1}f(n)}{6e^{\frac{2K}{c_{1}}}}\geq\frac{f(n)}{3c_{2}e^{\frac{2K}{c_{1}}}}.$
(2.12)

So inequality (2.12) holds if $n\geq n_{2}$. By condition 4, we get that
$\sum_{n=1}^{\infty}\frac{R(n)}{n^{2}}=\infty$, but

$\sum_{r\leq N,r\in
R}\frac{1}{r}=\sum_{n=1}^{N}\frac{R(n)-R(n-1)}{n}=\sum_{n=1}^{N-1}\frac{R(n)}{n(n+1)}+\frac{R(N)}{N}\geq\frac{1}{2}\sum_{n=1}^{N-1}\frac{R(n)}{n^{2}}\to\infty$

as $N\to\infty$, a contradiction.

Let $R=\\{r_{1},r_{2},\dots\\}$, $r_{1}<r_{2}<\dots$. It is well known that
$\displaystyle\lim_{x\to\infty}\sum_{\sqrt{x}\leq p\leq x}\frac{1}{p}=\log 2$.
We can find an $N_{0}\in\mathbb{Z}^{+}$ such that

$\sum_{\sqrt{x}\leq r_{i}\leq x,N_{0}|i}\frac{1}{r_{i}}<\frac{c_{1}\log 2}{2}$
(2.13)

for every $x\geq 1$. Let

$Q=\\{r_{N_{0}},r_{2N_{0}},\dots\\}.$

Clearly

$\sum_{q<x,q\in Q}\frac{1}{q}\geq\frac{\sum_{r<x,r\in
R}\frac{1}{r}-\sum_{i=1}^{N_{0}-1}\frac{1}{r_{i}}}{N_{0}}\to\infty$

as $N\to\infty$, therefore (2.9) holds.

It remains to prove (2.11). We use by induction on $k$.

For $k=1$,

$A_{1}(x)=Q(x)\leq
R(x)\leq\frac{e^{\frac{1}{c_{1}}}f(x)}{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{R}(n)}{n}\right)}\leq\frac{e^{\frac{1}{c_{1}}}f(x)}{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}.$

Let us suppose that the statement holds for $k$ and we prove it for $k+1$. If
$n\leq x$, $n\in A_{k+1}$, that is $n=p_{1}\dots p_{k+1}$,
$p_{1}<\dots<p_{k+1}$, $p_{i}\in Q$, then $p_{i}<\sqrt{x}$ for $1\leq i\leq
k$, therefore $kA_{k+1}(x)\leq\sum_{q\in Q,q<\sqrt{x}}A_{k}(\frac{x}{q})$.
Hence

$kA_{k+1}(x)\leq\sum_{q\in Q,q<\sqrt{x}}A_{k}(\frac{x}{q})\leq\sum_{q\in
Q,q<\sqrt{x}}\frac{e^{\frac{1}{c_{1}}}f(\frac{x}{q})\left(\frac{2}{c_{1}}\sum_{n<\frac{x}{q}}\frac{\chi_{Q}(n)}{n}\right)^{k-1}}{(k-1)!exp\left(\frac{2}{c_{1}}\sum_{n<\frac{x}{q}}\frac{\chi_{Q}(n)}{n}\right)}.$

By condition 3, we have

$f(\frac{x}{q})\leq\frac{1}{c_{1}}\frac{1}{q}f(x)$

and by (2.13),

$exp\left(\frac{2}{c_{1}}\sum_{n<\frac{x}{q}}\frac{\chi_{Q}(n)}{n}\right)=\frac{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}{exp\left(\frac{2}{c_{1}}\sum_{\frac{x}{q}\leq
n<x}\frac{\chi_{Q}(n)}{n}\right)}\geq\frac{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}{exp\left(\frac{2}{c_{1}}\sum_{\sqrt{x}\leq
n<x}\frac{\chi_{Q}(n)}{n}\right)}\geq$
$\frac{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}{exp\left(\frac{2}{c_{1}}\frac{c_{1}\log
2}{2}\right)}=\frac{exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}{2}.$

Hence

$kQ_{k+1}(x)\leq\sum_{q\in
Q,q<\sqrt{x}}\frac{e^{\frac{1}{c_{1}}}\frac{1}{c_{1}}\frac{1}{q}f(x)\left(\frac{2}{c_{1}}\sum_{n<\frac{x}{q}}\frac{\chi_{Q}(n)}{n}\right)^{k-1}}{(k-1)!\frac{1}{2}exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}\leq$
$\frac{e^{\frac{1}{c_{1}}}f(x)\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)^{k-1}}{(k-1)!exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)}\sum_{q\in
Q,q<\sqrt{x}}\frac{2}{c_{1}}\frac{1}{q}<\frac{e^{\frac{1}{c_{1}}}f(x)\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)^{k}}{(k-1)!exp\left(\frac{2}{c_{1}}\sum_{n<x}\frac{\chi_{Q}(n)}{n}\right)},$

which completes the proof. ∎

###### Proof of Theorem 14.

First we prove the first part of Theorem 14. Let

$V=\\{p\>\text{prime }:\exists n\in B,p\hbox{ is the largest prime factors of
}n\\}.$

If no prime in $V$ divides a number $n$, then equality $n=ab,a\in A,B\in B$
implies $b=1,a=n$, thus $n\in A$. Hence

$\liminf_{x\to\infty}\frac{A(x)}{x}\geq\prod_{p\in
V}\left(1-\frac{1}{p}\right)\geq\prod_{i=1}^{|B|}\left(1-\frac{1}{p_{i}}\right),$

where $p_{i}$ is the $i$th smallest prime number.

It is well known that

$\lim_{k\to\infty}\log
k\prod_{i=1}^{k}\left(1-\frac{1}{p_{i}}\right)=e^{-\gamma},$ (2.14)

where $\gamma$ is the Euler-Mascheroni constant. It follows that there exists
a constant $c_{1}>0$ such that

$\liminf_{x\to\infty}\frac{A(x)\log|B|}{x}\geq c_{1}.$

Now we prove the second part of Theorem 14. Let $2^{k}$ be the largest power
of $2$ that is less than or equal to $N$. Let $B\subseteq\mathbb{Z}^{+}$ be a
set that contains the set of all square-free numbers composed of the $k$
smallest primes. Clearly $2^{k-1}<|B|=2^{k}\leq N$. Let

$A=\\{mr^{2}:p_{i}\nmid m\>\forall\>1\leq i\leq k,r\in\mathbb{Z}^{+}\\}.$

Denote by $A_{sf}$ the set of squarefree integers in the set $A$. If $n\in
A_{sf}$, then $\gcd(n,p_{1}\dots p_{k})=1$, therefore $A_{sf}(x)\leq
O(1)+x\prod_{i=1}^{k}\left(1-\frac{1}{p_{i}}\right)$. Hence

$A(x)\leq\sum_{r\leq\sqrt{x}}A_{sf}\left(\frac{x}{r^{2}}\right)\leq\sum_{r\leq\sqrt{x}}\left(\frac{x}{r^{2}}\prod_{i=1}^{k}\left(1-\frac{1}{p_{i}}\right)+O(1)\right)=O(\sqrt{x})+\frac{\pi^{2}x}{6}\prod_{i=1}^{k}\left(1-\frac{1}{p_{i}}\right).$

Then $A$ and $B$ are multiplicative complements and by (2.14),

$\limsup_{x\to\infty}\frac{A(x)}{x}=\frac{\pi^{2}}{6}\prod_{i=1}^{k}\left(1-\frac{1}{p_{i}}\right)\leq\frac{c^{\prime}}{\log
k}\leq\frac{c_{2}}{\log\log N},$

which completes the proof.

∎

## Bibliography

  * [1] L. Danzer, Über eine Frage von G. Hanani aus der additiven Zahlentheorie, J. Reine Angew. Math 214/215 (1964), 392–394.
  * [2] A. Kocsis, D. Matolcsi, C. Sándor and G. Tőtős, Multiplicative complements I., Acta Arith. 207 (2023), no. 2, 101–120.
  * [3] P. P. Pach; and C. Sándor, Multiplicative bases and an Erdős problem, Combinatorica 38 (2018), no. 5, 1175–1203.
  * [4] R. Warlimont, Eine Bemerkung zu einem Ergebnis von N. G. de Bruijn, Monatsh. Math. 74 (1970), 273–276.


# Improving On-Screen Sound Separation for Open-Domain Videos with Audio-
Visual Self-Attention

Efthymios Tzinis
UIUC
<EMAIL_ADDRESS>
&Scott Wisdom
Google Research
<EMAIL_ADDRESS>
Tal Remez
Google Research
<EMAIL_ADDRESS>
&John R. Hershey
Google Research
<EMAIL_ADDRESS>
Work done during an internship at Google.

(Last compiled )

###### Abstract

We introduce a state-of-the-art audio-visual on-screen sound separation system
which is capable of learning to separate sounds and associate them with on-
screen objects by looking at in-the-wild videos. We identify limitations of
previous work on audio-visual on-screen sound separation, including the
simplicity and coarse resolution of spatio-temporal attention, and poor
convergence of the audio separation model. Our proposed model addresses these
issues using cross-modal and self-attention modules that capture audio-visual
dependencies at a finer resolution over time, and by unsupervised pre-training
of audio separation model. These improvements allow the model to generalize to
a much wider set of unseen videos. We also propose a robust way to further
improve the generalization capability of our models by calibrating the
probabilities of our audio-visual on-screen classifier, using only a small
amount of in-domain videos labeled for their on-screen presence. For
evaluation and semi-supervised training, we collected human annotations of on-
screen audio from a large database of in-the-wild videos (YFCC100m). Our
results show marked improvements in on-screen separation performance, in more
general conditions than previous methods.

##  1 Introduction

Humans are able to effortlessly perceive sounds in a noisy scene, and
associate them with any corresponding visible objects. In audio processing, a
corresponding challenge is to isolate sound sources from a mixture waveform
and identify the associated visual appearance of each sound source. Sound
sources in audio are analogous to objects in computer vision. An advantage in
vision is that objects tend to occupy distinct regions of pixels, so that
features from different regions of the image correspond to different objects.
In contrast, sound sources are superimposed in the audio, making it even more
difficult to disentangle the features of different sounds.

This creates a challenge for unsupervised learning of audio-visual separation,
because unlike their visual counterparts, the audio sources in a scene cannot
be easily selected for alignment with the video objects. The audio needs to be
separated into sources at some point in the process, either by conditioning
separation on selected video objects, or by separating the audio _a priori_ ,
before associating the sounds with video objects, or both. The _a priori_
separation of the sounds, which we pursue here, has a few advantages: thanks
to recent work, it can be learned in an unsupervised way, and it can handle an
unknown number of sounds, including those that do not appear on-screen.

Despite the recent progress in audio-visual machine perception using neural
networks, trained models still suffer from a variety of problems. Many of the
previous works rely heavily on labeled training data, or data with special
properties such as having one high quality sound that is on screen in each
example. They may also rely on pre-trained supervised object detectors that
require labeled data. This can work well in a restricted domain, where such
labeled data are available; however in an open-domain setting the reliance on
human labels effectively precludes scaling to large open-domain data. Thus a
typical problem is difficulty in generalizing from the domain of labeled
training data to the domain of realistic data that would be seen in a
practical application.

Although the co-perception and alignment of audio and visual modalities is not
trivial, a variety of works have shown promising results by proposing neural
network architectures apt for the task [8, 45, 6, 2, 13, 40, 7]. Audio-visual
sound separation [17, 10, 1] and more specifically separation of on-screen
versus off-screen sounds [29] have also seen remarkable performance
improvements since the initial works. Important innovations have included
using localization of objects [45, 48, 26], forcing consistency between audio
and visual modalities representation [24, 14, 3, 13], weakly [32] and self-
supervised approaches [33, 22, 2, 40]. Despite the remarkable progress in the
field of on-screen sound source separation, most of these works are
constrained to isolating a specific set of sounds which can appear on-screen
such as speech [10, 14, 1] or music [11, 12]. Recent work has started to
expand beyond music and speech to a wider variety of classes, using visual
scene graphs to model relationships between detected objects and use this
information to condition audio separation [7], but this approach is still
inherently limited by requiring labeled data to train supervised object
detection. Although the seminal works in separating on-screen sounds proposed
models which were somewhat invariant to the types of sources [29, 12], those
systems were unable to be trained with real world videos mainly because they
needed labeled videos assuming that sound sources always appeared on-screen
during training. A recently proposed model, known as AudioScope [40],
addressed this problem by enabling unsupervised training of audio-visual sound
separation models from in-the-wild videos, without requiring object detection
modules nor assuming that all sources have to be on-screen. The model is
trained to separate individual sound sources from a mixture of two videos
using mixture invariant training (MixIT) [44], which works by assigning
estimated sources to the best matching video mixture. The AudioScope model
then uses these source assignments as pseudo-labels to help train the audio-
visual correspondence classifier.

However, the problem of generalizing to a wide variety of videos still remains
largely unsolved. AudioScope [40] was able to obtain adequate performance in
terms of on-screen sound detection and reconstruction when training was
injected with a few thousands labeled examples, but performance suffered in
the purely unsupervised setting. In this work, we try to identify some of the
possible limitations of this approach and propose ways to overcome the
associated challenges. We hypothesize that AudioScope’s performance is limited
by the simplicity of its visually guided spatio-temporal attention layer [5],
and the low temporal resolution (one frame per second) of its visual model.
These factors may prevent AudioScope from capturing synchronization features
which might be crucial for detecting the inter-play between audio and video
[16, 22, 2]. Another limitation of the AudioScope architecture stems from the
outputs of the audio-visual classifier which are produced independent of the
possible domain mismatch between the train/test distributions. Consequently,
the probabilities which are later used as soft weights in front of the
separated sources might be heavily biased because of slight variations in the
recording setup (e.g. audio-energy levels, camera position, etc.), thus,
leading to poor generalization to previously unseen audio and video classes.

We propose to address these problems by introducing richer transformer-style
audio-visual attention models, operating at higher frame rates, in order to
better detect audio-visual synchrony. Furthermore, we propose to refine the
estimates of the on-screen audio-visual classifier per source by utilizing the
powerful representations obtained from unsupervised pre-training of the audio
source separation module with MixIT. We also propose a new way to perform
weakly-supervised domain adaptation by calibrating the output probabilities
produced by the classifier using a very small amount of videos which were
annotated for their on-screen presence. Finally, we also propose even more
challenging evaluation datasets containing unfiltered videos from the YFCC100m
[36] data collection, and show that our new proposed models both generalize
and perform better on these evaluations.

##  2 Relation to Prior Work

Audio-visual separation of in-the-wild on-screen sounds relies on the
capability of separating the individual sources that are superimposed in an
input audio recording. Recent work has shown that it is possible to train an
open-domain model to isolate individual sound sources using a mask-based
convolutional architecture regardless of the category of sound [20, 38]. A
related promising direction is to extract sources of interest by conditioning
the separation networks using identity cues. This has yielded performance
improvements for speech [42] as well as universal source separation [39, 28].
However, these experiments relied on having sufficient supervised training
data and were evaluated only on test sets with similar environmental
conditions and sound distributions. In order to extend the reach of this
approach, methods have been proposed to train separation models with no access
to ground truth clean sources by utilizing weak class labels [31], the spatial
separability of the sources [37, 34, 9] and self-supervision in the form of
MixIT [44]. This makes it possible to learn separation of signals well outside
the domains for which isolated source databases exist.

AudioScope [40] used MixIT in order to avoid the dependency on clean in-domain
training data, which may be unavailable for many types of sounds. However, it
also relied on filtering of the dataset in order to increase the proportion of
videos containing on-screen sounds. This was done using scores produced by an
audio-visual coincidence model [19] pre-trained on AudioSet [15]. A
disadvantage of this approach is that the coincidence model may be susceptible
to a looser semantic association, where sounds are thematically related to a
visual scene, as opposed to true audio-visual correspondence, where the sounds
actually appear onscreen. This, and any other biases in that model, may have
limited the ability of the resulting AudioScope model to generalize to the
distribution of sounds in unfiltered data. Our approach extends the
utilization of MixIT by pre-training on audio from a broader in-the-wild video
data collection. This may enable less reliance on pre-filtering, perhaps by
producing more robust pseudo-labels for training the audio-visual classifier.
Building upon that, we propose a novel way to refine the on-screen probability
estimates and perform domain adaptation on an on-screen separation model using
isotonic regression [47] with a small subset of labeled videos for their on-
screen presence.

An open question in audio-visual correspondence models concerns the level of
processing at which audio and video objects can be aligned. Typically audio-
visual models have used high-level features at the output of deep neural
networks to estimate correspondence between audio and video signals [24, 40,
19, 22, 7]. Such high-level representations may tend to focus on semantic
information about the class of objects and sounds, especially when the
features are computed from single frames at a low frame rate. Such methods may
work well for single instances of a class of object or sound, but may struggle
with identification for multiple instances of a class, or for classes not seen
during training. In contrast, there may be significant information in the
correspondence between lower-level features. Mutual information between low-
level features was used for audio-visual localization [16], and several more
recent works have shown promising results for self-supervised audio-visual
learning using low-level, motion [48] and optical flow [2] features. Such
features may help with generalization and instance-level correspondence by
detecting synchronous dynamics of the audio and video, regardless of their
semantic class.

Attention is a framework that may be useful at multiple levels of processing.
Attention-based models have been extensively utilized across audio-visual
learning tasks. An attention-based framework was recently used to modulate
audio representations using motion-based visual features [48] for separation
and localization. Conversely, modulating video features based on audio
embeddings has also been used for speech separation [24] as well as in the
spatio-temporal attention module of the initial AudioScope [40]. Other works
combined self-attention layers [41] for modeling inter-modality temporal
patterns as well as cross-modal attention modules for intra-modality
associations [46, 8, 45] for sound source localization. Our proposed model in
this paper applies attention at the level of audio sources, which is similar
to the operation of _slot attention_ models which apply attention across video
objects [27].

Inspired by the flexibility of self-attention models to represent spatio-
temporal associations, we propose to extend the applicability of self-
attention and cross-modal attention modules to unsupervised audio-visual on-
screen sound separation tasks. However, the applicability of these attention
layers is not trivial when we also want to scale to higher resolution audio-
visual representations in order to capture synchrony. The attention tensor
grows quadratically in complexity with the dimensionality of the space. For
this reason, we propose a variety of self-attention architectures that
factorize the attention operation across different dimensions and modalities.
The separable attention layers allow us to achieve similar performance to full
self-attention with a much lower computational footprint. Our proposed layers
differ from recently proposed separable attention layers [6, 25] in the sense
that we also accommodate modeling intra-modality patterns from both audio and
video features.

##  3 Model architecture

In Figure 1, the overall architecture of the proposed model is displayed
alongside the replaced modules from the previous version. The most noticeable
changes are with respect to the way we align the audio-visual features
extracted from the embedding networks. In the proposed version, we introduce a
family of transformer-based architectures (explained in Section 3.3) which are
able to obtain semantically rich representations between the audio and the
video modalities. In addition, we omit the temporal pooling operation for the
inputs of the proposed attention blocks since we want to let the model build
implicit latent representations capable of capturing low-level intra-modal
synchrony-based dependencies.

Figure 1: System diagram comparing our proposed approach with AudioScope [40],
with two input mixtures and four output sources. The modules common between
AudioScope and our approach are represented with green dark background,
AudioScope modules that are replaced use lighter font and dashed border, and
new proposed modules have orange background with solid border.

###  3.1 Separation Module

The separation module $\mathcal{M}_{s}$ has the same dilated convolutional
architecture as the one used in AudioScope [40] with learnable encoder and
decoder. This module takes as input a mixture waveform
$x\in\mathbb{R}^{T^{\prime}}$, estimates $M$ masks in the encoded latent space
and as a result outputs $M$ estimated sources $\hat{s}\in\mathbb{R}^{M\times
T^{\prime}}$ which are forced to add up to the input mixture through a
consistency layer [43]. Based on the ablation studies of [40] and our
experiments, we completely remove the conditional separation [39], as it did
not provide any significant gains over not using the visual representations as
inputs to the separation framework. More importantly, by removing the
dependence on the visual conditional embeddings, we are able to pre-train the
separation module on all YFCC100m [36] audio tracks in order to provide a
better initialization for training the audio-visual model.

###  3.2 Audio and video embedding networks

We extract the audio features for the $M$ estimated sources $\hat{s}$ and the
corresponding $T$ input video frames, with $128\times 128$ spatial resolution
each, using a MobileNetV1 architecture [18] following a similar setup used for
AudioScope [40]. Specifically, the audio embedding network takes as input log
mel-scale spectrogram representations of the time-domain separated sources
$\hat{s}$ while the visual embedding network is applied to each of the $T$
input video frames independently. We use local features extracted from
intermediate levels of the embedding networks as audio and video features. For
the audio part, we use the output of the intermediate activation at the $23$rd
layer, while for the video part, we use the one that has $8\times 8$ spatial
locations. These activations provide audio-visual representations which enjoy
both local and global feature descriptors. The final audio and video feature
tensors are also fed through dense layers in order to force them to have the
same depth $D$.

(a) Self-Attention (SA).

(b) Cross-modal attention (CMA).

(c) Separable attention.

Figure 2: Attention architectures for audio-visual alignment and feature
extraction. All features are depicted without including the batch and depth
dimensions for simplicity.

###  3.3 Audio-visual spatio-temporal attention

We seek to propose mechanisms which are able to effectively capture
correlation across source, space, and time for the audio features
$Z_{\mathrm{A}}\in\mathbb{R}^{M\times T\times D}$ and the corresponding video
features $Z_{\mathrm{V}}\in\mathbb{R}^{G\times T\times D}$, where $G=8\cdot 8$
is the total number of spatial locations, and the time dimension $T$ is shared
across both tensors. We provide a slightly more general version of an
attention layer [5] which computes similarities between a packed tensor of
queries $Q\in\mathbb{R}^{X_{Q}\times T_{Q}\times D}$ with respect to some
packed keys $K\in\mathbb{R}^{X_{K}\times T_{K}\times D}$, where $D$ denotes
the depth dimensionality of the tensors. The similarities are computed using a
generalized version of the typical inner product for tensors $\langle
Z_{1},Z_{2}\rangle_{\mathcal{A}}$, which reduces across the specified
dimensions $\mathcal{A}$ of the second tensor $Z_{2}$. Note that we assume
that the dimensions of $Z_{2}$ are a subset of the dimensions of $Z_{1}$. By
using a scaled tensor inner-product [41] and a
$\operatorname{softmax}_{\mathcal{A}}$ activation that averages over the
dimensions specified by $\mathcal{A}$ at the output of the tensor product
$\langle K,Q\rangle_{\\{\mathrm{D}\\}}$ we produce the resulting similarity
tensor which modulates the values $V\in\mathbb{R}^{X_{V}\times T_{V}\times
D}$:

$\displaystyle\operatorname{attention}_{\mathcal{A}}(Q,K,V)=\langle\alpha,f_{\mathrm{V}}(V)\rangle_{\mathcal{A}},\enskip\alpha=\operatorname{softmax}_{\mathcal{A}}\left(\frac{1}{\sqrt{D}}\langle
f_{\mathrm{K}}(K),f_{\mathrm{Q}}(Q)\rangle_{\\{\mathrm{D}\\}}\right),$ (1)

where $Q$, $K$, $V$, and $\alpha$ are the query tensor, the key tensor, the
value tensor, and the attention weight distribution tensor across the set of
specified axes $\mathcal{A}$ of the value/key tensors over which attention is
applied. For example, for input query $Q$ of shape $X_{Q}\times T_{Q}\times D$
and value $V$ of shape $X_{V}\times T_{V}\times D$,
$\operatorname{attention}_{\\{\mathrm{X_{V}},\mathrm{T_{V}}\\}}(Q,V,V)$
performs attention over the first and second axes of $V$, yielding an output
tensor of shape $X_{Q}\times T_{Q}\times D$. The dense layers
$f_{\mathrm{Q}}$, $f_{\mathrm{V}}$, $f_{\mathrm{K}}$ are all trainable and
applied to the depth dimension $D$.

We utilize as a main block of our network the multi-head attention (MHA) layer
proposed in [41]. Each one of the $H$ heads performs attention to some low-
dimensional embeddings derived from the tensors $Q$ and $V$, with the output
embedding depth reduced to $D/H$. These independent attention heads have the
capability to focus on different semantics of the input tensors. The final
output, after performing attention across the specified axes $\mathcal{A}$, is
given by aggregating all the intermediate heads’ outputs as shown next:

$\displaystyle
o^{(h)}=\operatorname{attention}_{\mathcal{A}}(f_{Q}^{(h)}(Q),f_{V}^{(h)}(V),f_{V}^{(h)}(V)),\enskip
h\in{1,\dots,H}$ (2)
$\displaystyle\operatorname{MHA}_{\mathcal{A}}(Q,V)=f(\operatorname{Concat}([o_{1},\dots,o_{H}]))\in\mathbb{R}^{X_{Q}\times
T_{Q}\times D}$

where $f$ denotes a dense layer $\mathbb{R}^{X_{Q}\times T_{Q}\times
D}\rightarrow\mathbb{R}^{X_{Q}\times T_{Q}\times D}$ and the dense layers
$f_{Q}^{(h)}$ and $f_{V}^{(h)}$ are defined as linear maps:
$\mathbb{R}^{X_{Q}\times T_{Q}\times D}\rightarrow\mathbb{R}^{X_{Q}\times
T_{Q}\times D/H}$ and $\mathbb{R}^{X_{V}\times T_{V}\times
D}\rightarrow\mathbb{R}^{X_{V}\times T_{V}\times D/H}$, respectively. In this
version we have simplified the usage of the attention layer in that we always
assume that the keys and values tensors are the same. Now, we describe in
detail the proposed versions of the transformer-based [41] audio-visual
attention architectures.

####  3.3.1 Self-attention (SA)

First, we concatenate the audio and the video tensors across the sources and
spatial dimensions:

$\displaystyle
Z_{\mathrm{AV}}^{(0)}=\operatorname{Concat}([Z_{\mathrm{A}},Z_{\mathrm{V}}])\in\mathbb{R}^{(M+G)\times
T\times D}$ (3)

which is used as the input to the first self-attention layer.

Joint: In the joint version of the self-attention module we seek to model
correlations across space, time, and sources by attending across all those
dimensions. Formally, we express the operation performed at the $l$-th layer
of a joint self-attention module as follows, also illustrated in Figure 2(a):

$\displaystyle r^{(l)}=$
$\displaystyle\operatorname{LayerNorm}\left(\operatorname{MHA}_{\\{\mathrm{M+G},\mathrm{T}\\}}\left(Z_{\mathrm{AV}}^{(l-1)},Z_{\mathrm{AV}}^{(l-1)}\right)+Z_{\mathrm{AV}}^{(l-1)}\right)$
(4) $\displaystyle Z_{\mathrm{AV}}^{(l)}=$
$\displaystyle\operatorname{LayerNorm}\left(f^{(l)}\left(\operatorname{Dropout}(r^{(l)})\right)\right)+r^{(l)}$

where $f^{(l)}$ is an output dense layer, $\operatorname{LayerNorm}$ is layer
normalization [4] and $\operatorname{Dropout}$ denotes a dropout layer [35].
We define the sequence of operations in Equation (4) as
$Z_{\mathrm{AV}}^{(l)}=\operatorname{SA}_{\\{\mathrm{M+G},\mathrm{T}\\}}(Z_{\mathrm{AV}}^{(l-1)})$
where the self-attention is performed across the joint sources-and-spatial
dimension $\mathrm{M+G}$ and the time axis $\mathrm{T}$. The final
representation $z$, after the repetition of $L$ self-attention blocks, for all
$M$ sources, is obtained through the appropriate slicing and by performing
attentional pooling [40] across the time axis.

$\displaystyle z=$
$\displaystyle\operatorname{MHA}_{\\{\mathrm{T}\\}}\left(\sum_{t}^{T}\widehat{z}_{t},\widehat{z}\right)\in\mathbb{R}^{M\times
D},\enskip\widehat{z}=$ $\displaystyle
Z_{\mathrm{AV}}^{(L)}[:M]\in\mathbb{R}^{M\times T\times D}.$ (5)

Separable: An issue with the joint attention defined above is that as the
number of audio sources $M$, spatial locations $G$, and/or time resolution $T$
of the input tensor $Z_{\mathrm{AV}}^{(0)}$ is increased, the computational
and space complexity of the conventional multi-head attention layer grows
quadratically. For this reason, we propose a separable version of attention:
first attend across time only, $\\{\mathrm{T}\\}$, and then subsequently
attend across the sources and spatial axis, $\\{\mathrm{M+G}\\}$, similar to a
previous work [6]. In this way, we are able to separate the computation and
not construct the full attention tensor with space complexity
$\mathcal{O}(T^{2}\cdot(M+G)^{2}\cdot H)$, where $H$ is the number of
attention heads. Formally, we write the following sequential operations for
the $l$-th layer of the separable self-attention block using the self-
attention module defined in Equation (4):

$\displaystyle a^{(l)}=$
$\displaystyle\operatorname{SA}_{\\{\mathrm{T}\\}}(Z_{\mathrm{AV}}^{(l-1)}[1:M])$
(6) $\displaystyle v^{(l)}=$
$\displaystyle\operatorname{SA}_{\\{\mathrm{T}\\}}(Z_{\mathrm{AV}}^{(l-1)}[M:M+G])$
$\displaystyle Z_{\mathrm{AV}}^{(l)}=$
$\displaystyle\operatorname{Concat}([a^{(l)},v^{(l)}])$ $\displaystyle
Z_{\mathrm{AV}}^{(l)}=$
$\displaystyle\operatorname{SA}_{\\{\mathrm{M+G}\\}}\left(Z_{\mathrm{AV}}^{(l)},Z_{\mathrm{AV}}^{(l)}\right).$

The final audio-visual representation is obtained through attentional pooling
and slicing as before. This is illustrated in Figure 2(c).

###  3.4 Cross-modal attention (CMA)

In this variation of the audio-visual attention module we keep the audio and
the video modality tensors separate, and we perform queries from one modality
to another. Formally the input to the stacked CMA blocks are always a pair of
an audio feature tensor $A^{(0)}_{\mathrm{CMA}}\in\mathbb{R}^{M\times T\times
D}$ and a video feature tensor $V^{(0)}_{\mathrm{CMA}}\in\mathbb{R}^{G\times
T\times D}$.

Joint: We perform a directional attention from the audio (video) modality
tensor to the video (audio) tensor, attending across both sources and time
(space and time) axes. Formally, at the $l$-th layer we have the following
sequence of operations, also illustrated in Figure 2(b):

$\displaystyle a_{1}^{(l)}=$
$\displaystyle\operatorname{MHA}_{\\{\mathrm{G},\mathrm{T}\\}}\left(A_{\mathrm{CMA}}^{(l-1)},V_{\mathrm{CMA}}^{(l-1)}\right)$
$\displaystyle v_{1}^{(l)}=$
$\displaystyle\operatorname{MHA}_{\\{\mathrm{M},\mathrm{T}\\}}\left(V_{\mathrm{CMA}}^{(l-1)},A_{\mathrm{CMA}}^{(l-1)}\right)$
(7) $\displaystyle a_{2}^{(l)}=$
$\displaystyle\operatorname{LayerNorm}(a_{1}^{(l)}+A_{\mathrm{CMA}}^{(l-1)})$
$\displaystyle v_{2}^{(l)}=$
$\displaystyle\operatorname{LayerNorm}\left(v_{1}^{(l)}+V_{\mathrm{CMA}}^{(l-1)}\right)$
$\displaystyle a_{3}^{(l)}=$ $\displaystyle
f\left(\operatorname{Dropout}(a_{2}^{(l)})\right)$ $\displaystyle
v_{3}^{(l)}=$ $\displaystyle
f\left(\operatorname{Dropout}(v_{2}^{(l)})\right)$ $\displaystyle
A_{\mathrm{CMA}}^{(l)}=$
$\displaystyle\operatorname{LayerNorm}\left(a_{3}^{(l)}+A_{\mathrm{CMA}}^{(l-1)}\right)$
$\displaystyle V_{\mathrm{CMA}}^{(l)}=$
$\displaystyle\operatorname{LayerNorm}\left(v_{3}^{(l)}+V_{\mathrm{CMA}}^{(l-1)}\right).$

The left side of the set of the above equations describes video-guided
attention where we use the video features $v^{(l)}$ to modulate the audio
features $a^{(l)}$, and vice versa on the right side. We define the sequence
of operations in Equation (7) as
$A_{\mathrm{CMA}}^{(l)},V_{\mathrm{CMA}}^{(l)}=\operatorname{CMA}_{\\{\mathrm{M|G},\mathrm{T}\\}}(A_{\mathrm{CMA}}^{(l-1)},V_{\mathrm{CMA}}^{(l-1)})$
where each cross-modal attention is performed across the dimension of audio
sources $M$ or spatial locations $G$ (denoted in our notation as
"$\mathrm{M|G}$") and the time axis. The output audio-visual embedding $z$
contains information for all $M$ sources and is obtained after the repetition
of $L$ cross-modal attention blocks by performing attentional pooling across
the time axis on the output audio tensor, as follows:

$\displaystyle z=$
$\displaystyle\operatorname{MHA}_{\\{\mathrm{T}\\}}\left(\sum_{t}^{T}\widehat{z}_{t},\widehat{z}\right)\in\mathbb{R}^{M\times
D},\enskip\widehat{z}=$ $\displaystyle
A_{\mathrm{CMA}}^{(l)}\in\mathbb{R}^{M\times T\times D}.$ (8)

Separable: Similar as discussed in Section 3.3.1, we can reduce the space
complexity of the intermediate representations of the proposed cross-modal
attention layer by performing a separate self-attention across the time-axis
for each modality individually and then perform CMA across the remaining axis
(i.e. sources or spatial locations) as shown next, also illustrated in Figure
2(c):

$\displaystyle a^{(l)}=$
$\displaystyle\operatorname{SA}_{\\{\mathrm{T}\\}}(A_{\mathrm{CMA}}^{(l-1)})$
(9) $\displaystyle v^{(l)}=$
$\displaystyle\operatorname{SA}_{\\{\mathrm{T}\\}}(V_{\mathrm{CMA}}^{(l-1)})$
$\displaystyle A_{\mathrm{CMA}}^{(l)},V_{\mathrm{CMA}}^{(l)}=$
$\displaystyle\operatorname{CMA}_{\\{\mathrm{M|G}\\}}(a^{(l)},v^{(l)}).$

###  3.5 Audio-visual on-screen sound classifier

The audio-visual correspondence between each estimated source $\hat{s}_{m}$
and the video is computed using the extracted audio-visual representation from
the output of our transformer based models defined in Equations (5) and (8),
for the self-attention and cross-modal attention encoders, respectively.
Specifically, we feed this audio-visual embedding $z\in\mathbb{R}^{M\times D}$
through a dense layer $f_{z}$ and we apply a sigmoid activation element-wise
to compute the audio-visual coincidence probabilities $\hat{y}_{1:M}$ for all
sources. Therefore, the final on-screen waveform estimate
$\hat{x}^{\mathrm{on}}$ is produced using these probabilities as soft weights
and multiplied with the corresponding estimated sources:

$\hat{y}_{m}=\sigma\left(f_{z}(z)\right)_{m}\in[0,1],\forall\enskip
m={1,\dots,M},\enskip\hat{x}^{\mathrm{on}}=\sum_{m=1}^{M}\hat{y}_{m}\hat{s}_{m}.$
(10)

##  4 Experimental Framework

###  4.1 Data preparation

We use the same training data as AudioScope [40], except instead of only using
a subset of the data filtered by an audio-visual coincidence prediction model
[19], we use all data, respecting the published
splits111https://github.com/google-research/sound-
separation/tree/master/datasets/yfcc100m for the Yahoo Flickr Creative Commons
100 Million Dataset (YFCC100m) [36]. The unfiltered training data consists of
about 1600 hours. We gathered human annotations for 5 second clips from
unfiltered videos from the validation and test splits. Out of 6500 validation
clips labeled, 109 were unanimously rated as on-screen-only, and 1421
unanimous off-screen-only. For 3500 test clips, there were 44 unanimous on-
screen-only, and 788 unanimous off-screen-only. We also experiment with using
a faster video frame rate of 16 frames per second (FPS), instead of 1 FPS as
used by AudioScope [40].

###  4.2 Training

We use the same training procedure as AudioScope [40]. We focus on the
unsupervised scenario, which means all batches are composed of “noisy-labeled
on-screen (NOn)” examples (in the terminology of the original paper [40]).
Each NOn example consists of the video frames and audio for a 5 second video
clip, where additional audio from another random 5 second video clip is used
as synthetic off-screen audio, and is added to the audio of the first clip.
These examples provide noisy labels, because we make the assumption that all
sources that map to the first clip’s audio are on-screen. However, not all
sources that map to the clip’s audio will necessarily be on-screen (e.g. there
could be off-screen background noise in the first clip that is separated as a
source).

Despite the training examples having these noisy labels, we found that the
exact cross-entropy loss works well, which is computed between the pseudo-
label assignments $y$ provided by MixIT and the audio-visual on-screen
classifier predictions $\hat{y}$ provided by the model. Both audio and visual
embedding networks were pre-trained on AudioSet [15] for unsupervised
coincidence prediction [19]. But unlike AudioScope, we freeze these networks
during training, which we found to yield better results. Also, instead of
initializing the separation model from scratch, we initialize the separation
model with weights pre-trained on unfiltered audio-only MoMs drawn from
YFCC100m, trained for 3.6M steps, which also significantly boosted the
performance of our models. All models are trained on 32 Google Cloud TPU v3
cores with the Adam optimizer [21], a batch size of $256$, and learning rate
of $10^{-4}$.

####  4.2.1 Calibration

In contrast to AudioScope, we perform an additional post-training calibration
step. Using validation data that is human-labeled as unanimously on-screen-
only or unanimously off-screen-only, we calibrate the on-screen classifier
using the scikit-learn [30] implementation of isotonic regression [47]. The
dataset for this calibration consists of both single-mixture and MoM examples.
For single-mixture examples, the labels are all 1 for on-screen-only videos,
and all 0 for off-screen only videos. MoM examples are constructed from an on-
screen-only or off-screen-only video and the audio from either off-screen-only
videos, or random videos chosen from the entire validation set. For on-screen
MoM examples, the labels are the MixIT assignments for the on-screen mixture,
so they are 1 for sources that map to the on-screen-only video, and 0
otherwise. For off-screen MoM examples, the labels are all 0.

###  4.3 Evaluation

For AudioScope [40], the videos in the validation and test sets were drawn
from a subset of YFCC100m filtered by an unsupervised coincidence model [19].
MoM examples in this dataset use unanimously-rated off-screen-only videos as
the background regardless of whether the foreground video is on-screen-only or
off-screen-only. We refer to this as “filtered off-screen background.” In
addition to this filtered dataset, we constructed new test sets drawn from the
unfiltered validation and test splits of YFCC100m. We created two versions of
these unfiltered datasets: one version, “unfiltered off-screen background,”
uses audio from unanimously-rated off-screen-only unfiltered videos as the
background in MoMs, and the other version, “unfiltered random background,”
uses audio sampled from all videos in the unfiltered validation and test sets.

Evaluation metrics are the same as used for AudioScope [40]: we measure power-
weighted area under the curve of the receiver operating characteristic (AUC-
ROC), scale-invariant signal-to-noise ratio (SI-SNR) [23] of the on-screen
estimate $\hat{x}^{\mathrm{on}}$ for on-screen examples (a measure of the
reconstruction fidelity of on-screen audio), and off-screen suppression ratio
(OSR) for off-screen examples, which measures the power reduction of the on-
screen estimate $\hat{x}^{\mathrm{on}}$ relative to the input audio mixture
power. These metrics are measured on both single mixtures and MoMs. For MoMs,
we additionally report an oracle metric, MixIT∗, which is the SI-SNR of the
estimated on-screen audio using the MixIT assignments from separated sources
to the reference on-screen audio.

##  5 Results

Results are shown in Tables 1, 2, and 3 for filtered off-screen background,
unfiltered off-screen background, and unfitered random background,
respectively. The first row in each table lists the performance of the
equivalent original AudioScope [40] model. Note that this model performs
relatively well on the filtered off-screen background dataset (Table 1), but
its performance drops precipitously on the unfiltered datasets (Tables 2 and
3). In particular, AUC-ROC drops from from 0.81 to 0.63 and 0.65, and on-
screen MoM SI-SNR of $\hat{x}^{\mathrm{on}}$ drops from 8.0 dB to 0.7 dB and
-0.7 dB. Notice that the OSRs actually increase from filtered to unfiltered
evaluation; this is due to the model tending to predict probability 0 for all
sources regardless of whether the video is on-screen or not. Thus, AudioScope
exhibits significant mismatch to the unfiltered evaluation datasets.

Using a pre-trained separation model and fine-tuning on unfiltered boosts the
performance of AudioScope by about 2 dB in terms of both oracle MixIT* on-
screen SI-SNR and $\hat{x}^{\mathrm{on}}$ on-screen SI-SNR, and also
generalizes better to unfiltered datasets. There is a slight performance
improvement using 16 FPS; in particular, OSR increases a bit. Notice that
using calibration shifts the operating point of models: it tends to reduce on-
screen SI-SNR by up to 1 dB, while substantially boosting OSR. This is likely
because the uncalibrated on-screen probablities tend towards predicting 1, and
this bias degrades rejection of off-screen sounds when the on-screen estimate
is created.

Our proposed attention-based models further improve performance over pre-
trained models that use the original spatio-temporal attention used by
AudioScope [40]. For calibrated models, on filtered off-screen background
MoMs, our attention-based models achieve a slight gain of 0.4 dB on-screen SI-
SNR and 0.6 dB OSR. On unfiltered MoMs with off-screen background, attention-
based models boost on-screen SI-SNR by 1.2 dB with comparable OSR, and on
unfiltered random background MoMs gain almost 3 dB on-screen SI-SNR, with only
slightly lower OSR. It is remarkable that the performance improvements for
these attention based models seem to increase as the difficulty of the
evaluation data increases.

Table 1: Evaluation results for “filtered off-screen background” test set with
calibration. On-screen MoMs have a median input SI-SNR of 4.4 dB. “PT”
indicates separation model pre-training, and “Cal.” indicates calibration on
labeled validation data.

|  |  |  | Single mixture | Mixture of mixtures
---|---|---|---|---|---
|  |  |  |  | SI-SNR (dB) | OSR (dB) |  | SI-SNR (dB) | OSR (dB)
AV alignment | PT | Cal. | Train data | AUC | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$ | AUC | MixIT* | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$
Spatio-temporal [40] |  |  | Filt. 1 FPS | 0.62 | 36.6 | 0.5 | 0.81 | 10.6 | 8.0 | 5.3
Spatio-temporal | ✓ |  | Unfil. 1 FPS | 0.58 | 29.2 | 1.7 | 0.79 | 12.5 | 9.7 | 4.0
Spatio-temporal | ✓ |  | Unfil. 16 FPS | 0.65 | 29.0 | 1.6 | 0.82 | 12.5 | 10.0 | 3.7
Spatio-temporal | ✓ | ✓ | Unfil. 1 FPS | 0.58 | 33.0 | 9.5 | 0.79 | 12.5 | 9.4 | 10.2
Spatio-temporal | ✓ | ✓ | Unfil. 16 FPS | 0.65 | 30.6 | 9.2 | 0.82 | 12.5 | 8.4 | 10.2
Joint SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.67 | 29.4 | 1.5 | 0.84 | 12.5 | 11.0 | 4.6
Separable SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.71 | 28.8 | 2.2 | 0.85 | 12.5 | 10.9 | 4.9
Joint CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.71 | 30.8 | 1.8 | 0.86 | 12.5 | 11.0 | 4.7
Separable CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.69 | 33.2 | 1.3 | 0.85 | 12.5 | 10.9 | 4.3
Joint SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.67 | 31.3 | 8.6 | 0.84 | 12.5 | 9.5 | 10.6
Separable SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.71 | 30.4 | 8.9 | 0.85 | 12.5 | 9.1 | 10.7
Joint CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.71 | 30.5 | 8.9 | 0.86 | 12.5 | 9.8 | 10.8
Separable CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.69 | 32.5 | 8.1 | 0.85 | 12.5 | 9.6 | 10.6

Table 2: Evaluation results for “unfiltered off-screen background” test set
with calibration. On-screen MoMs have a median input SI-SNR of 4.0 dB.

|  |  |  | Single mixture | Mixture of mixtures
---|---|---|---|---|---
|  |  |  |  | SI-SNR (dB) | OSR (dB) |  | SI-SNR (dB) | OSR (dB)
AV alignment | PT | Cal. | Train data | AUC | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$ | AUC | MixIT* | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$
Spatio-temporal [40] |  |  | Filt. 1 FPS | 0.54 | 19.5 | 9.1 | 0.63 | 10.2 | 0.7 | 16.6
Spatio-temporal | ✓ |  | Unfil. 1 FPS | 0.59 | 29.1 | 5.2 | 0.72 | 12.1 | 8.1 | 7.0
Spatio-temporal | ✓ |  | Unfil. 16 FPS | 0.70 | 30.2 | 4.9 | 0.78 | 12.1 | 8.4 | 6.3
Spatio-temporal | ✓ | ✓ | Unfil. 1 FPS | 0.59 | 30.2 | 22.6 | 0.72 | 12.1 | 8.6 | 24.9
Spatio-temporal | ✓ | ✓ | Unfil. 16 FPS | 0.70 | 29.2 | 24.8 | 0.78 | 12.1 | 8.4 | 26.0
Joint SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.68 | 30.5 | 5.0 | 0.80 | 12.0 | 10.2 | 6.3
Separable SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.77 | 30.4 | 5.2 | 0.83 | 12.0 | 10.4 | 6.8
Joint CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.70 | 31.1 | 4.7 | 0.80 | 12.0 | 9.8 | 6.6
Separable CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.69 | 32.1 | 4.1 | 0.81 | 12.1 | 9.8 | 6.4
Joint SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.68 | 29.9 | 24.6 | 0.80 | 12.0 | 9.1 | 25.1
Separable SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.77 | 29.8 | 24.8 | 0.83 | 12.0 | 9.8 | 26.2
Joint CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.70 | 32.1 | 24.5 | 0.80 | 12.0 | 9.5 | 25.7
Separable CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.69 | 32.0 | 23.1 | 0.81 | 12.1 | 9.0 | 25.2

Table 3: Evaluation results for “unfiltered random background” test set with
calibration. On-screen MoMs have a median input SI-SNR of 2.5 dB.

|  |  |  | Single mixture | Mixture of mixtures
---|---|---|---|---|---
|  |  |  |  | SI-SNR (dB) | OSR (dB) |  | SI-SNR (dB) | OSR (dB)
AV alignment | PT | Cal. | Train data | AUC | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$ | AUC | MixIT* | On: $\hat{x}^{\mathrm{on}}$ | Off: $\hat{x}^{\mathrm{on}}$
Spatio-temporal [40] |  |  | Filt. 1 FPS | 0.54 | 19.5 | 9.1 | 0.65 | 8.0 | -0.7 | 17.4
Spatio-temporal | ✓ |  | Unfil. 1 FPS | 0.59 | 29.1 | 5.2 | 0.72 | 10.1 | 5.9 | 5.6
Spatio-temporal | ✓ |  | Unfil. 16 FPS | 0.70 | 30.2 | 4.9 | 0.73 | 10.1 | 5.8 | 4.2
Spatio-temporal | ✓ | ✓ | Unfil. 1 FPS | 0.59 | 30.2 | 22.6 | 0.72 | 10.1 | 5.8 | 23.4
Spatio-temporal | ✓ | ✓ | Unfil. 16 FPS | 0.70 | 29.2 | 24.8 | 0.73 | 10.1 | 5.2 | 22.5
Joint SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.68 | 30.5 | 5.0 | 0.76 | 10.1 | 8.9 | 3.9
Separable SA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.77 | 30.4 | 5.2 | 0.78 | 10.1 | 8.9 | 4.1
Joint CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.70 | 31.1 | 4.7 | 0.75 | 10.2 | 8.0 | 4.4
Separable CMA $\times 4$ | ✓ |  | Unfil. 16 FPS | 0.69 | 32.1 | 4.1 | 0.77 | 10.1 | 8.9 | 3.9
Joint SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.68 | 29.9 | 24.6 | 0.76 | 10.1 | 8.1 | 21.2
Separable SA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.77 | 29.8 | 24.8 | 0.78 | 10.1 | 8.4 | 20.0
Joint CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.70 | 32.1 | 24.5 | 0.75 | 10.2 | 7.1 | 20.6
Separable CMA $\times 4$ | ✓ | ✓ | Unfil. 16 FPS | 0.69 | 32.0 | 23.1 | 0.77 | 10.1 | 8.5 | 20.6

##  6 Conclusion

In this paper we have presented extensions and refinements of the AudioScope
model that further improve upon previous audio-visual self-supervised methods
for separating on-screen sounds. Our proposed model is able to operate on
higher frame-rates and build rich semantic associations between audio and
video modalities using the proposed self-attention as well as cross-modal
attention mechanisms. Our experimental results show that our model is able to
generalize to a much wider set of in-the-wild videos than existing approaches
while being trained solely with in-the-wild videos. The approach we followed
here is to first separate the audio before aligning each on-screen source with
its corresponding video object. Other works have followed the converse process
of first identifying visual objects and using them to condition sound
separation for each object. Future works may explore unifying these two
directions into a single model that works in both ways.

## References

  * [1] T. Afouras, J. S. Chung, and A. Zisserman.  The conversation: Deep audio-visual speech enhancement.  Proc. Interspeech 2018, pages 3244–3248, 2018.
  * [2] T. Afouras, A. Owens, J. S. Chung, and A. Zisserman.  Self-supervised learning of audio-visual objects from video.  arXiv preprint arXiv:2008.04237, 2020.
  * [3] R. Arandjelovic and A. Zisserman.  Objects that sound.  In Proceedings of the European conference on computer vision (ECCV), pages 435–451, 2018.
  * [4] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [5] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [6] G. Bertasius, H. Wang, and L. Torresani.  Is space-time attention all you need for video understanding?  arXiv preprint arXiv:2102.05095, 2021.
  * [7] M. Chatterjee, J. Le Roux, N. Ahuja, and A. Cherian.  Visual scene graphs for audio source separation.  In Proc. IEEE International Conference on Computer Vision (ICCV), pages 1204–1213, 2021.
  * [8] Y. Cheng, R. Wang, Z. Pan, R. Feng, and Y. Zhang.  Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 3884–3892, 2020.
  * [9] L. Drude, D. Hasenklever, and R. Haeb-Umbach.  Unsupervised training of a deep clustering model for multichannel blind source separation.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 695–699, 2019.
  * [10] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein.  Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation.  ACM Transactions on Graphics (TOG), 37(4):1–11, 2018.
  * [11] C. Gan, D. Huang, H. Zhao, J. B. Tenenbaum, and A. Torralba.  Music gesture for visual sound separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10478–10487, 2020.
  * [12] R. Gao, R. Feris, and K. Grauman.  Learning to separate object sounds by watching unlabeled video.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 35–53, 2018.
  * [13] R. Gao and K. Grauman.  Co-separating sounds of visual objects.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3879–3888, 2019.
  * [14] R. Gao and K. Grauman.  Visualvoice: Audio-visual speech separation with cross-modal consistency.  arXiv preprint arXiv:2101.03149, 2021.
  * [15] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 776–780, 2017.
  * [16] J. Hershey and J. Movellan.  Audio vision: Using audio-visual synchrony to locate sounds.  Advances in neural information processing systems, 12:813–819, 1999\.
  * [17] J. R. Hershey and M. Casey.  Audio-visual sound separation via hidden markov models.  In Advances in Neural Information Processing Systems, pages 1173–1180, 2002.
  * [18] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [19] A. Jansen, D. P. Ellis, S. Hershey, R. C. Moore, M. Plakal, A. C. Popat, and R. A. Saurous.  Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 121–125, 2020.
  * [20] I. Kavalerov, S. Wisdom, H. Erdogan, B. Patton, K. Wilson, J. Le Roux, and J. R. Hershey.  Universal sound separation.  In Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2019.
  * [21] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Proc. International Conference on Learning Representations (ICLR), 2015.
  * [22] B. Korbar, D. Tran, and L. Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  In Advances in Neural Information Processing Systems, pages 7763–7774, 2018.
  * [23] J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey.  SDR–half-baked or well done?  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 626–630, 2019.
  * [24] J. Lee, S.-W. Chung, S. Kim, H.-G. Kang, and K. Sohn.  Looking into your speech: Learning cross-modal affinity for audio-visual speech separation.  arXiv preprint arXiv:2104.02775, 2021.
  * [25] X. Li, Y. Zhang, C. Liu, B. Shuai, Y. Zhu, B. Brattoli, H. Chen, I. Marsic, and J. Tighe.  Vidtr: Video transformer without convolutions.  arXiv preprint arXiv:2104.11746, 2021.
  * [26] Y.-B. Lin, H.-Y. Tseng, H.-Y. Lee, Y.-Y. Lin, and M.-H. Yang.  Unsupervised sound localization via iterative contrastive learning.  arXiv preprint arXiv:2104.00315, 2021.
  * [27] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf.  Object-centric learning with slot attention.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11525–11538. Curran Associates, Inc., 2020.
  * [28] T. Ochiai, M. Delcroix, Y. Koizumi, H. Ito, K. Kinoshita, and S. Araki.  Listen to what you want: Neural network-based universal sound selector.  arXiv preprint arXiv:2006.05712, 2020.
  * [29] A. Owens and A. A. Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018.
  * [30] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [31] F. Pishdadian, G. Wichern, and J. Le Roux.  Finding strength in weakness: Learning to separate sounds with weak supervision.  arXiv preprint arXiv:1911.02182, 2019.
  * [32] T. Rahman and L. Sigal.  Weakly-supervised audio-visual sound source detection and separation.  arXiv preprint arXiv:2104.02606, 2021.
  * [33] A. Rouditchenko, H. Zhao, C. Gan, J. McDermott, and A. Torralba.  Self-supervised audio-visual co-segmentation.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2357–2361. IEEE, 2019.
  * [34] P. Seetharaman, G. Wichern, J. Le Roux, and B. Pardo.  Bootstrapping single-channel source separation via unsupervised spatial clustering on stereo mixtures.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 356–360, 2019.
  * [35] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [36] B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [37] E. Tzinis, S. Venkataramani, and P. Smaragdis.  Unsupervised deep clustering for source separation: Direct learning from mixtures using spatial information.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 81–85, 2019.
  * [38] E. Tzinis, S. Venkataramani, Z. Wang, C. Subakan, and P. Smaragdis.  Two-step sound source separation: Training on learned latent targets.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 31–35. IEEE, 2020.
  * [39] E. Tzinis, S. Wisdom, J. R. Hershey, A. Jansen, and D. P. W. Ellis.  Improving universal sound separation using sound classification.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 96–100, 2020.
  * [40] E. Tzinis, S. Wisdom, A. Jansen, S. Hershey, T. Remez, D. P. Ellis, and J. R. Hershey.  Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds.  In Proc. International Conference on Learning Representations (ICLR), 2021.
  * [41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30:5998–6008, 2017.
  * [42] Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. Hershey, R. A. Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno.  Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking.  In Proc. Interspeech, 2019.
  * [43] S. Wisdom, J. R. Hershey, K. Wilson, J. Thorpe, M. Chinen, B. Patton, and R. A. Saurous.  Differentiable consistency constraints for improved deep speech enhancement.  In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 900–904, 2019.
  * [44] S. Wisdom, E. Tzinis, H. Erdogan, R. J. Weiss, K. Wilson, and J. R. Hershey.  Unsupervised sound separation using mixtures of mixtures.  In Advances in Neural Information Processing Systems, 2020.
  * [45] Y. Wu, L. Zhu, Y. Yan, and Y. Yang.  Dual attention matching for audio-visual event localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6292–6300, 2019.
  * [46] J. Yu, Y. Cheng, and R. Feng.  Mpn: Multimodal parallel network for audio-visual event localization.  arXiv preprint arXiv:2104.02971, 2021.
  * [47] B. Zadrozny and C. Elkan.  Transforming classifier scores into accurate multiclass probability estimates.  In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002.
  * [48] L. Zhu and E. Rahtu.  Visually guided sound source separation and localization using self-supervised motion representations.  arXiv preprint arXiv:2104.08506, 2021.