$\displaystyle+\ln\left(1+\exp\left(y\right)\right)-\ln\left(1+\exp\left(e\left(\boldsymbol{\theta}_{2}\right)\right)\right),$

and

$h\left(\tau,\boldsymbol{\theta}\right):=\tau\left[FZ_{\tau}^{sp}\left(q\left(\boldsymbol{\theta}_{1}\right),e\left(\boldsymbol{\theta}_{2}\right),Y\right)-\ln\left(1+\exp\left(Y\right)\right)\right].$

Let $\mathcal{T}:=\left(0,1\right)$ and

$\displaystyle B_{1}\left(V\right)$ $\displaystyle:=$
$\displaystyle\sup_{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{1}}\left|q\left(\boldsymbol{\theta}_{1}\right)\right|,$
$\displaystyle B_{2}\left(V\right)$ $\displaystyle:=$
$\displaystyle\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\left\|q\left(\boldsymbol{\theta}_{1}\right)\nabla_{\boldsymbol{\theta}_{2}}e\left(\boldsymbol{\theta}_{2}\right)\right\|,$
$\displaystyle B_{3}\left(V\right)$ $\displaystyle:=$
$\displaystyle\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{1}}\left\|\nabla_{\boldsymbol{\theta}_{1}}q\left(\boldsymbol{\theta}_{1}\right)\right\|,$
$\displaystyle B_{4}\left(V\right)$ $\displaystyle:=$
$\displaystyle\sup_{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{2}}\left\|Y\nabla_{\boldsymbol{\theta}_{2}}e\left(\boldsymbol{\theta}_{2}\right)\right\|,$
$\displaystyle B_{5}\left(V\right)$ $\displaystyle:=$
$\displaystyle\sup_{\boldsymbol{\theta}\in\boldsymbol{\Theta}}\left\|\left(e\left(\boldsymbol{\theta}_{2}\right)-q\left(\boldsymbol{\theta}_{1}\right)\right)\nabla_{\boldsymbol{\theta}_{2}}e\left(\boldsymbol{\theta}_{2}\right)\right\|.$

Let
$\boldsymbol{\theta}^{1}:=\left(\boldsymbol{\theta}_{1}^{1},\boldsymbol{\theta}_{2}^{1}\right)$
and
$\boldsymbol{\theta}^{2}:=\left(\boldsymbol{\theta}_{1}^{2},\boldsymbol{\theta}_{2}^{2}\right)$
be two vectors of parameter values.

######  Lemma 3 (Global Lipschitz continuity)

If Assumption 2.4 holds, for all
$\left(\tau_{1},\tau_{2}\right)\in\mathcal{T}$,
$\left(\boldsymbol{\theta}_{1}^{1},\boldsymbol{\theta}_{1}^{2}\right)\in\boldsymbol{\Theta}_{1}$
and
$\left(\boldsymbol{\theta}_{2}^{1},\boldsymbol{\theta}_{2}^{2}\right)\in\boldsymbol{\Theta}_{2}$,
given $V=\left(Y,W\right)$, there exists a constant
$B_{\tau}^{*}\left(V\right)$ such that

$\left|h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)\right|\leq
B_{\tau}^{*}\left(V\right)\left(\left\|\tau_{1}-\tau_{2}\right\|+\left\|\boldsymbol{\theta}^{1}-\boldsymbol{\theta}^{2}\right\|\right).$

Proof. We use similar notations and strategy as in proving Lemma 2. At first
note that $h\left(\tau,\boldsymbol{\theta}\right)$ can be rewritten as

$\displaystyle h\left(\tau,\boldsymbol{\theta}\right)$ $\displaystyle=$
$\displaystyle\frac{\exp\left(e\left(\boldsymbol{\theta}_{2}\right)\right)}{1+\exp\left(e\left(\boldsymbol{\theta}_{2}\right)\right)}\left\\{0.5\left[\left|q\left(\boldsymbol{\theta}_{1}\right)-Y\right|+q\left(\boldsymbol{\theta}_{1}\right)-Y\right]+\tau\left(e\left(\boldsymbol{\theta}_{2}\right)-q\left(\boldsymbol{\theta}_{1}\right)\right)\right\\}$
$\displaystyle-\tau\ln\left(1+\exp\left(e\left(\boldsymbol{\theta}_{2}\right)\right)\right).$

Then

$\displaystyle
h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)$
$\displaystyle=$
$\displaystyle\frac{0.5\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left[\left|q_{1}-Y\right|+q_{1}-Y\right]-\frac{0.5\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left[\left|q_{2}-Y\right|+q_{2}-Y\right]$
$\displaystyle+\left.\left[\frac{\tau_{1}\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left(e_{1}-q_{1}\right)-\frac{\tau_{2}\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left(e_{2}-q_{2}\right)\right.\right.$
$\displaystyle\left.-\left(\tau_{1}\ln\left(1+\exp\left(e_{1}\right)\right)-\tau_{2}\ln\left(1+\exp\left(e_{2}\right)\right)\right)\right]$

We assume $q_{1}\geq q_{2}$ and $\tau_{1}\geq\tau_{2}$. The results for
$q_{1}<q_{2}$ and $\tau_{1}<\tau_{2}$ can be proved in a similar manner. We
separate the proof into three cases.
Cases 1. Suppose $q_{1}\geq q_{2}>Y$. It can be shown that

$\displaystyle\left|h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)\right|$
$\displaystyle\leq$
$\displaystyle\left|\frac{\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}q_{1}-\frac{\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}q_{2}\right|$
$\displaystyle+\left|Y\frac{\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}-Y\frac{\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\right|$
$\displaystyle+\left|\frac{\tau_{1}\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left(e_{1}-q_{1}\right)-\frac{\tau_{2}\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left(e_{2}-q_{2}\right)\right.$
$\displaystyle\left.-\left(\tau_{1}\ln\left(1+\exp\left(e_{1}\right)\right)-\tau_{2}\ln\left(1+\exp\left(e_{2}\right)\right)\right)\right|$

Case 2. Suppose $q_{1}\geq Y\geq q_{2}$. It can be shown that

$\displaystyle\left|h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)\right|$
$\displaystyle\leq$
$\displaystyle\left|\frac{\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}q_{1}-\frac{\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}q_{2}\right|$
$\displaystyle+\left|\frac{\tau_{1}\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left(e_{1}-q_{1}\right)-\frac{\tau_{2}\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left(e_{2}-q_{2}\right)\right.$
$\displaystyle\left.-\left(\tau_{1}\ln\left(1+\exp\left(e_{1}\right)\right)-\tau_{2}\ln\left(1+\exp\left(e_{2}\right)\right)\right)\right|$

Case 3. Suppose $Y>q_{1}\geq q_{2}$. It can be shown that

$\displaystyle\left|h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)\right|$
$\displaystyle\leq$
$\displaystyle+\left|\frac{\tau_{1}\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left(e_{1}-q_{1}\right)-\frac{\tau_{2}\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left(e_{2}-q_{2}\right)\right.$
$\displaystyle\left.-\left(\tau_{1}\ln\left(1+\exp\left(e_{1}\right)\right)-\tau_{2}\ln\left(1+\exp\left(e_{2}\right)\right)\right)\right|$

Let $\left(\bar{\tau},\bar{q},\bar{e}\right)$ be some middle point between
$\left(\tau_{1},q_{1},e_{1}\right)$ and $\left(\tau_{2},q_{2},e_{2}\right)$.
Using the mean value theorem, it is straightforward to show that

$\displaystyle\left|\frac{\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}q_{1}-\frac{\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}q_{2}\right|$
$\displaystyle\leq$
$\displaystyle\left|\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\right|\left|q_{1}-q_{2}\right|$
(50)
$\displaystyle+\left|\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\left(1-\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\right)\right|\left|\bar{q}\left(e_{1}-e_{2}\right)\right|$
$\displaystyle\leq$
$\displaystyle\left|q_{1}-q_{2}\right|+\left|\bar{q}\left(e_{1}-e_{2}\right)\right|,$
$\displaystyle\left|Y\frac{\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}-Y\frac{\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\right|$
$\displaystyle\leq$
$\displaystyle\left|\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\left(1-\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\right)\right|\left|Y\left(e_{1}-e_{2}\right)\right|$
(51) $\displaystyle\leq$
$\displaystyle\left|Y\left(e_{1}-e_{2}\right)\right|,$

and

$\displaystyle\left|\frac{\tau_{1}\exp\left(e_{1}\right)}{1+\exp\left(e_{1}\right)}\left(e_{1}-q_{1}\right)-\frac{\tau_{2}\exp\left(e_{2}\right)}{1+\exp\left(e_{2}\right)}\left(e_{2}-q_{2}\right)\right.$
$\displaystyle\left.-\left(\tau_{1}\ln\left(1+\exp\left(e_{1}\right)\right)-\tau_{2}\ln\left(1+\exp\left(e_{2}\right)\right)\right)\right|$
$\displaystyle\leq$
$\displaystyle\left|\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\bar{e}-\ln\left(1+\exp\left(\bar{e}\right)\right)\right|\left|\tau_{1}-\tau_{2}\right|$
$\displaystyle+\left|\frac{\exp\left(\bar{e}\right)\bar{q}}{1+\exp\left(\bar{e}\right)}\right|\left|\tau_{1}-\tau_{2}\right|+\left|\frac{\bar{\tau}\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\right|\left|q_{1}-q_{2}\right|$
$\displaystyle+\left|\frac{\bar{\tau}\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\left(1-\frac{\exp\left(\bar{e}\right)}{1+\exp\left(\bar{e}\right)}\right)\right|$
$\displaystyle\times\left|\left(\bar{e}-\bar{q}\right)\left(e_{1}-e_{2}\right)\right|$
$\displaystyle\leq$ $\displaystyle\left(\ln
2+\left|\bar{q}\right|\right)\left|\tau_{1}-\tau_{2}\right|+\left|q_{1}-q_{2}\right|$
$\displaystyle+\left|\left(\bar{e}-\bar{q}\right)\left(e_{1}-e_{2}\right)\right|$

To see why (A.4) holds, let
$\varpi\left(x\right)=\exp\left(x\right)\left(1+\exp\left(x\right)\right)^{-1}x-\ln\left(1+\exp\left(x\right)\right)$.
It can be shown that $\varpi\left(x\right)$ is monotonically decreasing for
$x<0$ and monotonically increasing for $x\geq 0$ and
$\varpi\left(0\right)=-\ln 2$,
$\lim_{x\rightarrow\infty}\varpi\left(x\right)=0$, and
$\lim_{x\rightarrow-\infty}\varpi\left(x\right)=0$. Therefore we can conclude
that

$\left|\varpi\left(x\right)\right|\leq\ln 2$

for all $x\in\mathbb{R}$. If Assumption 2.4 holds, it can be shown for (50),

$\displaystyle\left|q_{1}-q_{2}\right|+\left|\bar{q}\left(e_{1}-e_{2}\right)\right|$
$\displaystyle\leq$
$\displaystyle\left|\nabla_{\boldsymbol{\theta}_{1}}q\left(\bar{\boldsymbol{\theta}}_{1}\right)\right|\left\|\boldsymbol{\theta}_{1}^{1}-\boldsymbol{\theta}_{1}^{2}\right\|+\left|q\left(\boldsymbol{\theta}_{1}\right)\nabla_{\boldsymbol{\theta}_{2}}e\left(\boldsymbol{\theta}_{2}\right)\right|\left\|\boldsymbol{\theta}_{2}^{1}-\boldsymbol{\theta}_{2}^{2}\right\|$
$\displaystyle\leq$ $\displaystyle
B_{3}\left(V\right)\left\|\boldsymbol{\theta}_{1}^{1}-\boldsymbol{\theta}_{1}^{2}\right\|+B_{2}\left(V\right)\left\|\boldsymbol{\theta}_{2}^{1}-\boldsymbol{\theta}_{2}^{2}\right\|,$

for (51),

$\displaystyle\left|Y\left(e_{1}-e_{2}\right)\right|$ $\displaystyle\leq$
$\displaystyle
B_{4}\left(V\right)\left\|\boldsymbol{\theta}_{2}^{1}-\boldsymbol{\theta}_{2}^{2}\right\|,$

and for (A.4),

$\displaystyle\left(\ln
2+\left|\bar{q}\right|\right)\left|\tau_{1}-\tau_{2}\right|+\left|q_{1}-q_{2}\right|+\left|\left(\bar{e}-\bar{q}\right)\left(e_{1}-e_{2}\right)\right|$
$\displaystyle\leq$ $\displaystyle\left(\ln
2+B_{1}\left(V\right)\right)\left|\tau_{1}-\tau_{2}\right|+B_{3}\left(V\right)\left\|\boldsymbol{\theta}_{1}^{1}-\boldsymbol{\theta}_{1}^{2}\right\|$
$\displaystyle+B_{5}\left(V\right)\left\|\boldsymbol{\theta}_{2}^{1}-\boldsymbol{\theta}_{2}^{2}\right\|.$

Summing the above inequalities together, given $V=\left(Y,W\right)$, we can
conclude that

$\left|h\left(\tau_{1},\boldsymbol{\theta}^{1}\right)-h\left(\tau_{2},\boldsymbol{\theta}^{2}\right)\right|\leq
B^{*}\left(V\right)\left(\left\|\tau_{1}-\tau_{2}\right\|+\left\|\boldsymbol{\theta}^{1}-\boldsymbol{\theta}^{2}\right\|\right),$

where $B^{*}\left(V\right)$ is a constant and is determined by a sum of $\ln
2$ and constants $B_{i}\left(V\right)$, $i=1,\ldots,5$.

###  A.5 Constructing the simultaneous confidence bands (scb) with the
bootstrap

We use the models of linear in parameters of (6) and (7) as an example to
illustrate the bootstrap procedures for constructing the simultaneous
confidence bands. Let
$\boldsymbol{\theta}_{\tau}=\left(\alpha_{1,\tau},\boldsymbol{\beta}_{1,\tau}^{\top},\alpha_{2,\tau},\boldsymbol{\beta}_{2,\tau}^{\top}\right)^{\top}$
be a vector for parameters at the $\tau$-quantile. The bootstrap procedures
are summarized as follows.

  1. 1.

Draw a bootstrap sample of size $n$: $W_{1}^{*},\ldots,W_{n}^{*}$ with
replacement, where

$W_{i}^{*}=\left(Y_{i}^{*},D_{i}^{*},X_{i}^{*\top},Z_{i}^{*}\right)^{\top},$

$i=1,\ldots,n$, is a vector for the $i$th random draw sample.

  2. 2.

Re-estimate the weight $\bar{K}$ with the bootstrap sample. Let
$\hat{\tilde{K}}_{i}^{*}$, $i=1,\ldots,n$ denote the bootstrap estimated
truncated weight evaluated with $\left(Y_{i}^{*},D_{i}^{*},X_{i}^{*}\right)$.

  3. 3.

With the bootstrap estimated truncated weight $\hat{\bar{K}}_{i}^{*}$, obtain
the bootstrap estimated parameters

$\hat{\boldsymbol{\theta}}_{\tau,b}^{*}=\left(\hat{\alpha}_{1,\tau,b}^{*},\hat{\boldsymbol{\beta}}_{1,\tau,b}^{*\top},\hat{\alpha}_{2,\tau,b}^{*},\hat{\boldsymbol{\beta}}_{2,\tau,b}^{*\top}\right)^{\top}$

for $\tau\in\left(0,1\right)$.

  4. 4.

Repeat procedures 1 to 3 $B$ times to obtain $B$ bootstrap estimated
parameters
$\hat{\boldsymbol{\theta}}_{\tau,1}^{*},\ldots,\hat{\boldsymbol{\theta}}_{\tau,B}^{*}$.

With the bootstrap estimated parameters, we use $\hat{\alpha}_{2,\tau}$ as an
example to illustrate the procedures to construct the scb as follows.

  1. 1.

Calculate the bootstrap standard deviation of $\hat{\alpha}_{2,\tau}$,
$\hat{\sigma}_{\hat{\alpha}_{2,\tau}}^{*}$ with the $B$ bootstrap estimates
$\left(\hat{\alpha}_{2,\tau,1}^{*},\ldots,\hat{\alpha}_{2,\tau,B}^{*}\right)$.

  2. 2.

Calculate the bootstrap $t$ statistic of $\hat{\alpha}_{2,\tau}$:

$\hat{t}_{\hat{\alpha}_{2,\tau},b}^{*}=\frac{\sqrt{n}\left(\hat{\alpha}_{2,\tau,b}^{*}-\hat{\alpha}_{2,\tau}\right)}{\hat{\sigma}^{*}_{\hat{\alpha}_{2,\tau}}},$

$b=1,\ldots,B$.

  3. 3.

For each $b=1,\ldots,B$, calculate the maximal absolute bootstrap $t$
statistic for $\tau\in\left(0,1\right)$:

$\hat{t}_{\hat{\alpha}_{2},b}^{*}=\sup_{\tau\in\left(0,1\right)}\left|\hat{t}_{\hat{\alpha}_{2,\tau},b}\right|.$

  4. 4.

Let $\hat{t}_{\hat{\alpha}_{2}}^{*(1-g)}$ be the $\left(1-g\right)$th sample
quantile of
$\hat{t}_{\hat{\alpha}_{2},1}^{*},\ldots,\hat{t}_{\hat{\alpha}_{2},B}^{*}$.
The $1-g$ scb of $\hat{\alpha}_{2,\tau}$ is

$\left[\hat{\alpha}_{2,\tau}-\hat{t}_{\hat{\alpha}_{2}}^{*(1-g)}\frac{\hat{\sigma}_{\hat{\alpha}_{2,\tau}}^{*}}{\sqrt{n}},\hat{\alpha}_{2,\tau}+\hat{t}_{\hat{\alpha}_{2}}^{*(1-g)}\frac{\hat{\sigma}_{\hat{\alpha}_{2,\tau}}^{*}}{\sqrt{n}}\right].$
(53)

## References

  * Abadie (2003) Abadie, A. (2003): “Semiparametric instrumental variable estimation of treatment response models,” _Journal of Econometrics_ , 113, 231–263.
  * Abadie et al. (2002) Abadie, A., J. Angrist, and G. Imbens (2002): “Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings,” _Econometrica_ , 70, 91–117.
  * Angrist et al. (2006) Angrist, J., V. Chernozhukov, and I. Fernández-Val (2006): “Quantile Regression under Misspecification, with an Application to the U.S. Wage Structure,” _Econometrica_ , 74, 539–563.
  * Angrist et al. (1996) Angrist, J. D., G. W. Imbens, and D. B. Rubin (1996): “Identification of Causal Effects Using Instrumental Variables,” _Journal of the American Statistical Association_ , 91, 444–455.
  * Belloni et al. (2017) Belloni, A., V. Chernozhukov, I. Fernández-Val, and C. Hansen (2017): “Program Evaluation and Causal Inference With High-Dimensional Data,” _Econometrica_ , 85, 233–298.
  * Chen et al. (2020) Chen, Y.-T., Y.-C. Hsu, and H.-J. Wang (2020): “A Stochastic Frontier Model with Endogenous Treatment Status and Mediator,” _Journal of Business & Economic Statistics_, 38, 243–256.
  * Chernozhukov et al. (2013) Chernozhukov, V., I. Fernández-Val, and B. Melly (2013): “Inference on Counterfactual Distributions,” _Econometrica_ , 81, 2205–2268.
  * Chernozhukov and Hansen (2005) Chernozhukov, V. and C. Hansen (2005): “An IV Model of Quantile Treatment Effects,” _Econometrica_ , 73, 245–261.
  * Chernozhukov and Hansen (2006) ——— (2006): “Instrumental quantile regression inference for structural and treatment effect models,” _Journal of Econometrics_ , 132, 491–525.
  * Chernozhukov and Hansen (2008) ——— (2008): “Instrumental variable quantile regression: A robust inference approach,” _Journal of Econometrics_ , 142, 379–398.
  * Chernozhukov et al. (2007) Chernozhukov, V., C. Hansen, and M. Jansson (2007): “Inference approaches for instrumental variable quantile regression,” _Economics Letters_ , 95, 272–277.
  * Chernozhukov et al. (2009) ——— (2009): “Finite sample inference for quantile regression models,” _Journal of Econometrics_ , 152, 93–103.
  * Chou et al. (2020) Chou, R. Y., T.-J. Yen, and Y.-M. Yen (2020): “Forecasting Expected Shortfall and Value-at-Risk with the FZ Loss and Realized Variance Measures,” Available at SSRN: https://ssrn.com/abstract=3448882.
  * Dimitriadis and Bayer (2019) Dimitriadis, T. and S. Bayer (2019): “A joint quantile and expected shortfall regression framework,” _Electron. J. Statist._ , 13, 1823–1871.
  * Engle and Manganelli (2004) Engle, R. F. and S. Manganelli (2004): “CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles,” _Journal of Business & Economic Statistics_, 22, 367–381.
  * Fissler and Ziegel (2016) Fissler, T. and J. F. Ziegel (2016): “Higher order elicitability and Osband’s principle,” _The Annals of Statistics_ , 44, 1680–1707.
  * Fricke et al. (2020) Fricke, H., M. Frölich, M. Huber, and M. Lechner (2020): “Endogeneity and non-response bias in treatment evaluation – nonparametric identification of causal effects by instruments,” _Journal of Applied Econometrics_ , 35, 481–504.
  * Frölich and Huber (2017) Frölich, M. and M. Huber (2017): “Direct and indirect treatment effects–causal chains and mediation analysis with instrumental variables,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 79, 1645–1666.
  * Frölich and Melly (2013) Frölich, M. and B. Melly (2013): “Unconditional Quantile Treatment Effects Under Endogeneity,” _Journal of Business and Economic Statistics_ , 31, 346–357.
  * Heckman et al. (1997) Heckman, J. J., J. Smith, and N. Clements (1997): “Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts,” _The Review of Economic Studies_ , 64, 487–535.
  * Hill (2015) Hill, J. B. (2015): “Expected Shortfall Estimation and Gaussian Inference for Infinite Variance Time Series,” _Journal of Financial Econometrics_ , 13, 1–44.
  * Imbens and Angrist (1994) Imbens, G. and J. Angrist (1994): “Identification and Estimation of Local Average Treatment Effects,” _Econometrica_ , 62, 467–75.
  * Imbens and Rubin (2015) Imbens, G. W. and D. B. Rubin (2015): _Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction_ , Cambridge University Press.
  * Kianian et al. (2021) Kianian, B., J. I. Kim, J. P. Fine, and L. Peng (2021): “Causal Proportional Hazards Estimation with a Binary Instrumental Variable,” _Statistica Sinica_ , 31, 673–699.
  * Koenker (2005) Koenker, R. (2005): _Quantile Regression_ , Econometric Society Monographs, Cambridge University Press.
  * Koenker and Bassett (1978) Koenker, R. and G. Bassett (1978): “Regression Quantiles,” _Econometrica_ , 46, 33–50.
  * Levy (2016) Levy, H. (2016): _Stochastic Dominance-Investment Decision Making under Uncertainty_ , Springer, 3 ed.
  * Linton and Xiao (2013) Linton, O. and Z. Xiao (2013): “Estimation of and Inference about the expected shortfall for time series with infinite variance,” _Econometric Theory_ , 29, 771–807.
  * Meng and Taylor (2020) Meng, X. and J. W. Taylor (2020): “Estimating Value-at-Risk and Expected Shortfall using the intraday low and range data,” _European Journal of Operational Research_ , 280, 191 – 202.
  * Newey (1991) Newey, W. K. (1991): “Uniform Convergence in Probability and Stochastic Equicontinuity,” _Econometrica_ , 59, 1161–1167.
  * Newey (1997) ——— (1997): “Convergence rates and asymptotic normality for series estimators,” _Journal of Econometrics_ , 79, 147 – 168.
  * Newey and McFadden (1994) Newey, W. K. and D. McFadden (1994): “Chapter 36: Large sample estimation and hypothesis testing,” in _Handbook of Econometrics_ , Elsevier, vol. 4, 2111–2245.
  * Patton et al. (2019) Patton, A. J., J. F. Ziegel, and R. Chen (2019): “Dynamic semiparametric models for expected shortfall (and Value-at-Risk),” _Journal of Econometrics_ , 211, 388 – 413.
  * Powell (1984) Powell, J. L. (1984): “Least absolute deviations estimation for the censored regression model,” _Journal of Econometrics_ , 25, 303–325.
  * Taylor (2019) Taylor, J. W. (2019): “Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution,” _Journal of Business & Economic Statistics_, 37, 121–133.
  * van der Vaart (1998) van der Vaart, A. W. (1998): _Asymptotic Statistics_ , Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press.
  * Wei et al. (2021) Wei, B., L. Peng, M.-J. Zhang, and J. P. Fine (2021): “Estimation of causal quantile effects with a binary instrumental variable and censored data,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , forthcoming.
  * Wüthrich (2020) Wüthrich, K. (2020): “A Comparison of Two Quantile Models With Endogeneity,” _Journal of Business & Economic Statistics_, 38, 443–456.

Figure 1: Bias, variance and MSE of the estimated conditional CTATE for the
compliers under different quantile levels when $\rho=0$.

Figure 2: Bias, variance and MSE of the estimated conditional QTE for the
compliers under different quantile levels when $\rho=0$.

Figure 3: Bias, variance and MSE of the estimated conditional CTATE for the
compliers under different quantile levels when $\rho=0.5$.

Figure 4: Bias, variance and MSE of the estimated conditional QTE for the
compliers under different quantile levels when $\rho=0.5$.

Figure 5: CTATE, QTE and the corresponding 95% pointwise and simultaneous
confidence bands: Adult men’s earnings.

Figure 6: CTATE, QTE and the corresponding 95% pointwise and simultaneous
confidence bands: Adult women’s earnings.

Figure 7: IQATE and 95% pointwise confidence bands. Upper panel: Adult men’s
earnings. Lower panel: Adult women’s earnings.


Universal Source Separation with Weakly Labelled Data

M. Shell was with the Department
of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta,
GA, 30332.

E-mail: see http://www.michaelshell.org/contact.html
J. Doe and J. Doe are with Anonymous University.
Manuscript received April 19, 2005; revised August 26, 2015.

Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals

Universal source separation (USS) is a fundamental research task for computational auditory scene analysis (CASA), which aims to separate mono recordings into individual source tracks. From the perspective of system designs, there are two potential challenges awaiting the solution to the audio source separation task. First, previous audio source separation systems mainly focus on separating one or a limited number of specific sources, such as speech, singing, or musical instruments. However, there is a lack of research on building a unified system that can separate arbitrary sources via a single model. Second, most previous systems require strongly labeled data, i.e., clean source and mixture tracks to train a separator. Nonetheless, strongly labeled data is scarce. There should be a potential usage of large-scale weakly labeled/unlabeled audio data for audio source separation. To combat those problems, we propose a universal audio source separation framework containing: 1) an audio tagging model trained on weakly labeled data only as a query net for source separation; and 2) a conditional source separation model that can separate arbitrary sound sources according to the mixture and the query. We investigate various query nets, source separation models, and different training strategies to verify how they affect the performance in the universal source separation framework. We propose a hierarchy USS strategy to automatically detect and separate sound classes from the AudioSet ontology. Our separation system is recognized with a strong generalization ability. By solely leveraging AudioSet with 2 million weakly labeled audio event samples for the model training, our USS system could achieve compatible performance on various source separation benchmarks, such as sound event separation, music source separation, and speech enhancement. Namely, an average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over AudioSet's 527 sound classes; an SDRi of 10.57 dB on the DCASE 2018 Task 2 dataset; an SDRi of 8.12 dB on the MUDDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of 9.00 on the voicebank-demand dataset. We release the source code at <https://github.com/bytedance/audioset_source_separation>.

Universal source separation, weakly labelled data, AudioSet, computational auditory scene analysis.

§ INTRODUCTION

Source separation is a task to separate mono audio recordings into individual source tracks. An audio recording may consist of several sound events and acoustic scenes.

Source separation has been researched for several years and has a wide range of applications, including speech enhancement <cit.>, music source separation <cit.>, and sound events separation<cit.>.

Universal source separation (USS) is closely related to the well-known cocktail party problem <cit.>, where sounds from different sources in the world mix in the air before arriving at the ear, requiring the brain to estimate individual sources from the received mixture. Humans can focus on a particular sound source and discriminate it from others, alternatively described as selective hearing. As a study of auditory scene analysis by computational means, computational auditory scene analysis (CASA) <cit.> systems are machine listening systems that aim to separate mixtures of sound sources in the same way that human listeners do. Different from specified source separation tasks such as speech enhancement or music source separation, a universal source separation system aims to separate the tracks of arbitrary sound sources from a mixture, regarded as a unified system to separate all different sounds.

However, many previous works mainly focus on specific source separation that only separate one or a few sources, such as the speech enhancement <cit.> and music source separation <cit.> tasks. USS remains a challenging problem. One difficulty is that there are hundreds of different sounds in the world, and it is difficult to separate all sounds using a unified model. Recently, the USS problem has attracted several interests. A system <cit.> was proposed to separate arbitrary sounds by predicting the masks of sounds. Unsupervised USS systems <cit.> were proposed to separate sounds by mixture invariant training, where training examples are constructed by mixing together existing mixtures, and the model separates them into a variable number of latent sources. A Free Universal Sound Separation (FUSS) dataset and a time-domain convolutional network (TDCN++) were proposed to separate a mixture into up to 4 separate sources. Some other methods use sound embeddings as conditions to separate sources from a mixture <cit.>. A sound selector applies one-hot encodings of sound classes as conditions to separate corresponding sources <cit.>. Weak supervision methods were proposed in <cit.>. SuDoRM-RM systems <cit.> were proposed to separate sound signals. Language-based source separation was proposed in <cit.>. A class conditions system <cit.> was proposed for music source separation.
Those systems are mainly trained on small datasets and do not scale to hundreds of sound classes.

The standard architecture of deep-learning-based audio source separation model. Left: two types of time-domain separation model. Right: the general type of frequency-domain separation model.

Another challenge of universal source separation is that there is a lack of clean source data for a wide range of sound events.

For the source separation problem, we define strongly labelled data as audio segments that only contain target sources without other sources. We can mix those clean sources as audio mixtures and train the separators. However, the collection of clean data is time-consuming. For sound classes such as speech, clean data can be recorded in the laboratory. But for many other sounds, the collection of clean data is difficult. Recently, the concept of weakly labelled data <cit.> was used in audio signal processing. In contrast to strongly labelled data, weakly labelled data are only labelled with one or multiple tags, while the time information of tags is unknown. For example, a 10-second audio track is labelled as  “thunder” and “rain”, but when these two events exactly appear within this 10-second is not provided. Sometimes, different audio events may interfere with each other. Weakly labelled data has been widely used in audio tagging <cit.> and sound event detection <cit.>. There is a strong potential usage of it in the audio source separation task.

To address the challenge of lacking strongly labelled data to separate hundreds of sound classes. In this work, we propose a USS framework that can be trained with weakly labelled data only. This work is an extension of our previously proposed source separation systems <cit.>. The whole system contains: 1) pretrained audio tagging models to provide both the anchor segments of events in weakly labelled data and latent embeddings for source separation controls; 2) a query-based separation model for conditional audio source separation via the given source queries; and 3) a pipeline to train USS systems with weakly labelled data. The contribution of this work includes:

* We are the first to propose USS systems trained only on weakly labelled data only, without using any strongly labelled data. We train the USS system on AudioSet <cit.>.

* We propose an anchor segment mining strategy to detect short segments from long weakly labelled audio recordings. We show that anchor segments are the core parts to constitute mixtures to build the USS system.

* We propose several latent condition strategies and show that pretrained audio tagging system is essential to build USS systems.

* Our proposed USS systems are the first to separate 527 sound classes. We propose a hierarchy USS strategy to separate sources of different AudioSet ontology levels. Our proposed USS system is able to automatically detect and separate active sound classes of an audio recording without the need to specify the sound classes to separate.

* We show that a single USS system is able to achieve compatible performance to various source separation benchmarks, including sound events separation, music source separation, and speech enhancement tasks. We conduct comprehensive ablation studies to investigate how different factors in our system affect the separation performance.

This article is organized as follows. Section 2 introduces neural network-based source separation systems. Section 3 introduces our proposed weakly labelled source separation framework. Section 4 shows experiments. Section 5 concludes this work.

§ SOURCE SEPARATION VIA NEURAL NETWORKS

Deep learning methods for audio source separation has already outperformed traditional methods such as Non-negative Matrix Factorization <cit.>. Fig. <ref> shows the source separation methods in the time-domain (left) and in the frequency-domain (right). Here, we introduce the basic methodology of those separation models.

§.§ Time-domain Separation Models

A time-domain separation model $f$ via neural networks is typically constructed as an encoder-decoder architecture, as shown in the left of Fig. <ref>. Formally, given a mono audio clip $x \in \mathbb{R}^{L}$ and a separation target $s \in \mathbb{R}^{L}$, where $L$ is sample length, the separator $f$ performs two methods: synthesis separation and mask separation.

Synthesis separation systems such as Demucs V2 <cit.> and Wave-U-Net <cit.> $f$ directly estimates the final target samples of separation: $\hat{s} = f(x)$. The encoder is designed to extract the key feature for the separated source, and the decoder is designed to reconstruct the feature back to the separation samples together with the input features. Mask separation systems such as TasNet <cit.> and ConvTasNet <cit.> predict masks in the latent space. Then, decoders are designed to reconstruct the separated waveform from the masked latent feature.

§.§ Frequency-domain Separation Models

Different from time-domain models, frequency-domain models leverages the spectrogram such as short-time Fourier transform (STFT) to facilitate the separation process. The harmonic features are more discriminative in the frequency domain than those in the time domain, which might help improve the separation performance in specific source separation tasks, such as music source separation and environmental sound separation.

Formally, given a mono audio clip $x$. We denote the STFT of $x$ as $X \in \mathbb{C}^{T \times F}$, where $T$ is the number of time frames and $F$ is the number of frequency bins. The STFT spectrogram $X$ is in the complex domain. We denote the magnitude and the phase of $ X $ as $|X|$ and $\angle S_X$, respectively. The right part of Fig. <ref> shows the frequency-domain separation system $f$ predict the mask $ M $, where $ M $ can be an ideal ratio mask (IRM) $M \in \mathbb{R}^{T \times F}$ or a complex mask $M \in \mathbb{C}^{T \times F}$. When using the complex mask $ M $, the complex STFT of the separated source $ \hat{S} \in \mathbb{C}^{T \times F} $ can be calculated by:

\begin{align}
\hat{S}=M \odot X
\end{align}

Then, the separated source $ \hat{s} \in \mathbb{R}^{L} $ can be obtained by applying an inverse STFT on $ \hat{S} $.

Frequency domain models include fully connected neural networks <cit.>, recurrent neural networks (RNNs) <cit.>, and convolutional neural networks (CNNs) <cit.>. UNets <cit.> are variants of CNN that contain encoder and decoder layers for source separation. Band-split RNN (BSRNN) <cit.> proposes to apply RNNs along both time and frequency axes to capture time and frequency domain dependencies. There are also researches such as hybrid Demucs <cit.> combines time and frequency domain systems to build source separation systems.

§.§ Challenges of Source Separation Models

As mentioned above, many previous source separation systems are trained with strongly labelled data, which requires clean source tracks. However, the collection of strongly labelled data is difficult and time-consuming. Table <ref> summarizes the datasets that can be used for source separation. On one hand, previous strongly labelled datasets have durations of around tens of hours. On the other hand, weakly labelled datasets are usually larger than strongly labelled datasets and clean datasets. AudioSet <cit.> is a representative of weakly labelled datasets containing over 5,800 hours of 10-second audio clips and is larger in both size and sound class numbers than strongly labelled datasets. AudioSet has an ontology of 527 sound classes in its released version. The ontology of AudioSet has a tree structure. Each audio clip may contain multiple tags.

In this work, we use the weakly labelled AudioSet dataset containing 5,800 hours to train the USS system that can separate hundreds of sound classes as described in the next section.

Source separation datasets

Dataset     Dur. (h)     Classes     Labels

Voicebank-Demand      18.9     1     Strong

MUSDB18 <cit.>       6.0     4     Strong

UrbanSound8K       9.7     10     Strong

FSDKaggle 2018          41     Strong

FUSS         357     Strong

AudioCaps         —     Language

AudioSet     5800     527     Weak

Left: Strongly labelled data of sound class “Flute”. Right: Weakly labelled data of sound class “Air horn, truck horn” which only occur between 2.5s - 4.0s.

The architecture of our proposed query-based audio source separation pipeline trained from weakly-labeld data, including datasets, sampling strategies, audio tagging model, and conditional audio source separation models.

Two audio tagging models for audio classification, sound event detection, and latent feature production. Left: Pretrained Audio Neural Networks (PANN) in CNN-14 architecture. Right: Hierarchical Token-Semantic Transformer (HTS-AT) in 4-block architecture.

§ USS WITH WEAKLY LABELLED DATA

§.§ Weakly Labelled Data

Different from strongly labelled data containing clean sources of sound classes, weakly labelled data only contains the labels of what sound classes are present in an audio recording. Weakly labelled data may contain interference sounds. There are no time stamps information of sound classes nor clean sources. We denote a weakly labelled audio clip as $ x $ and its tags as $ y \in \{0, 1\}^{K} $, where $ K $ is the number of sound classes. The value $ 1 $ indicates the presence of sound class and the value $ 0 $ indicates the absence of the sound class in the audio clip. We denote a weakly labelled dataset as $ D = \{x_{n}, y_{n}\}_{n=1}^{N} $, where $ N $ is the number of training samples in the dataset. The left part of Fig. <ref> shows a strongly labelled audio clip containing the clean waveform of “Flute”. The right part of Fig. <ref> shows a weakly labelled audio clip containing a target sound class “Air horn, truck horn” which only occur between 2.5 s and 4.0 s. The weakly labelled audio recording also contain unknown interference sounds.

The goal of a weakly labelled USS system is to separate arbitrary sounds via a unified model.

Fig. <ref> depicts the architecture of our proposed system, containing four steps:

* Apply a sampling strategy to sample audio clips from a weakly labelled audio dataset, such as AudioSet.

* Apply an anchor segment mining algorithm to localize the occurrence of events/tags in the weakly labelled audio tracks.

* Calculate the segment predictions or latent embeddings of anchor segments using pretrained audio tagging models. Mix anchor segments as input mixtures.

* Train a query-based separation network to separate the mixture into target sources queried by the source embedding.

§.§ Audio Clips Sampling Strategies

For a large-scale unbalanced dataset, we apply two sampling strategies: 1) random sampling: randomly sample audio clips from the dataset to constitute a mini-batch; and 2) balanced sampling: sample audio clips from different sound classes to constitute a mini-batch to ensure the mixture of them contain as many as independent sounds. AudioSet is highly unbalanced. Sound classes such as “Speech” and “Music” have almost 1 million audio clips, while sound classes such as “tooth breath” has only tens of training samples. Without balanced sampling, “tooth breath” are less likely to be selected for training. Following the training scheme of the audio classification systems <cit.>, we apply the balanced sampling strategy to retrieve audio data from AudioSet. We denote a mini-batch of sampled audio clips as $ \{ x_{1}, ..., x_{B} \} $, where $ B $ is the mini-batch size.

The anchor segment mining algorithm involves an audio tagging model to localize the sound events in the given audio track. Since the whole pipeline requires only weakly labelled data, it is better that we could leverage audio tagging models that require only the weakly-labelled data for training, but is able to localize the occurrence (i.e., time stamps) of the sound classes. Recently, audio tagging systems trained with the weakly labelled AudioSet <cit.> have outperformed systems trained with strongly labelled data. We apply Pretrained Audio Neural Networks (PANN) <cit.> and Hierarchical Token-semantic Audio Transformer (HTS-AT) <cit.> as our audio tagging models to perform the anchor segment mining procedure. Such models are able to extract audio clips, with relatively clean sound sources, from weakly labelled audio samples. In section <ref>, we introduce the detail of audio tagging models and the mining algorithm.

To extract the embedding from audio clips, we have different formulations. The most intuitive embedding can be the one-hot class vector, and the class probability vector or the latent embedding from audio tagging models could be a potential choice. Together with the query-based source separation model, we demonstrate how it works through this separation pipeline in section
<ref> and <ref>.

§.§ Anchor Segments Mining

Anchor segment mining is the core part of USS systems with weakly labelled data. Since the weakly labeled audio track does not always contain the labeled sound class throughout its timeline, we need to extract a short audio segment inside this track to create a relative cleaner source data for training the separation model. Formally, given an audio clip $x \in \mathbb{R}^L$, an anchor segment mining algorithm extracts an anchor segment $s \in \mathbb{R}^{L'}$ from $x$, where $L' < L$ is the samples number of the anchor segment. The center of $s$ is the time stamp where the sound class label is most likely to occur. As shown in Fig <ref> and illustrated in Algorithm <ref>, an anchor mining algorithm is performed as three steps:

* By the sampling strategy, we sample $n$ audio tracks $\{x_1, x_2, ..., x_n\}$ with the corresponding tags $\{e_1,e_2, ..., e_n\}$ as one input for our system.

* Two anchor mining algorithms can be performed to extract the audio clip $s_{i}$ from $x_{i}$: i) random method: randomly select an audio clip from the audio track; or ii) sound event detection (SED) method: apply an audio tagging model to localize the labeled sound class as an anchor position $t$ in the audio track, then extract an audio clip whose center is the anchor position.

* To further verify the localization, an optional step can be performed to filter the audio clips from duplicate sound classes, ensuring that the mixture does not come from the same sound classes.

The detail demonstration of each step is illustrated in Algorithm <ref>.

For the SED method of anchor mining, in this paper, we leverage two audio tagging models, Pretrain Audio Neural Networks (PANN) <cit.> and Hierarchical Token-Semantic Audio Transformer (HTS-AT) <cit.>, to perform the sound event detection of the audio track. These models are also applied into the construction of query embeddings during the separation process.

Formally, given an audio track $x$ as input, the audio tagging model produces two main outputs: 1) event classification vector $v_e \in \mathbb{R}^K$; and 2) framewise event presence map $m_e \in \mathbb{R}^{T \times K}$, where $T$ the size of time frames and $K$ the number of sound classes. The framewise event presence map is usually connected to the last neural network layer by taking the average value along the time axis to produce the event classification vector. As mentioned in <ref>, both PANN and HTS-AT are trained with the weakly labeled data by minimizing the binary cross entropy loss between the label vector $y \in [0,1]^K$ and $v_e$:

\begin{equation} \label{eq:aggregate_max}
l = - \sum_{k=1}^{K} y(k) \ln v_e(k) + (1 - y(k)) \ln (1 - v_e(k)).
\end{equation}

where $y(k)$ and $v_e(k)$ denote the $k$-th component of $y$ and $v_e$.

After training the audio tagging model, we can not only use it to predict the event of the given audio track, but also weakly predict the event of each time frame via framewise event presence map $m_e$ (i.e., sound event detection). We denote $m_e(t,k)$ as the probability value of the $k$-th sound class at the $t$-th time frame. Then the anchor segment $t_{anchor}$ for the labeled sound class $k$ is obtain by:

\begin{align} \label{eq:sed_area}
q_{k}(t) = \sum_{t - \tau / 2}^{t + \tau / 2} m_e(t, k) \\
t_{\text{anchor}} = \underset{t}{\text{argmax}} \ q_{k}(t).
\end{align}

where $ \tau $ is the duration of anchor segments. We apply the sound event detection algorithm to the $n$-track audio input and get $n$ audio clips $\{s_{1}, ..., s_{n}\}$ as anchor segments (if Step-3 in Algorithm <ref> is applied, the number of anchor segments will be smaller than the number of input audio tracks). Finally, we mix these segments as the input mixture of the separation model.

Fig. <ref> shows the model architectures of both PANN and HTS-AT as two audio tagging models we employed. PANN contains VGG-like CNNs to convert an audio mel-spectrogram into a $(T, K)$ featuremap. The model averages the featuremap over the time axis to obtain a final event classification vector $(1, K)$ and computes the binary cross-entropy loss between it and the groudtruth label. Since CNNs can capture the information in each time window, the featuremap $(T, K)$ is empirically regarded as a framewise presence probability map of each sound event at each time frame. Additionally, the penultimate layer's output $(T, H)$ can be used to obtain its averaged vector $(1, H)$ as a latent source embedding for the query-based source separation in our system.

HTS-AT is a hierarchical token-semantic transformer for audio classification. It applies swin-transformer <cit.> into the audio classification task. In the right of Figure <ref>, a mel-spectrogram is cut into different patch tokens with a patch-embed CNN and sent into the transformer in order. The time and frequency lengths of the patch is equal as $P \times P$. To better capture the relationship between frequency bins of the same time frame, HTS-AT first splits the mel-spectrogram into windows $w_1, w_2, ..., w_n$ and then splits the patches in each window. The order of tokens $Q$ follows time$\to$frequency$\to$window.

The patch tokens pass through several network groups, each of which contains several transformer-encoder blocks. Between every two groups, a patch-merge layer is applied to reduce the number of tokens to construct a hierarchical representation. Each transformer-encoder block is a swin-transformer block with the shifted window attention module <cit.>, a modified self-attention module to improve the training efficiency.

Then, HTS-AT applies a token-semantic 2D-CNN to further process the reshaped output $(\frac{T}{8P}, \frac{F}{8P}, H)$ into the framewise event presence map $(T,K)$, then average it as the event classification vector $(1,K)$. The latent embedding, at the same time, is produced by averaging the reshaped output into a $H$-dimension vector with an average-pooling layer.

In the following sections, we will evaluate how these two models perform in our universal source separation system.

Anchor segment mining.
[1]
Input: dataset $ D $.
Step 1 - Balanced Sampling: Sample $ n $ sound classes from $ \{1, ..., K\} $ without replacement.
For each selected sound class, sample one audio track $x_{i} \in \mathbb{R}^L$ to constitute $ \{x_{1}, ..., x_{n}\} $.
Step 2 - Random Method: for each audio track $x_{i}$, randomly select an audio clip $s_{i} \in \mathbb{R}^{L'} (L' < L)$ from the audio track.
Step 2 - Sound Event Detection:
Apply an audio tagging model to detect an anchor position $t_{i}$ throughout the audio track $x_{i}$, which is the most probable occurrence of the labeled sound class. Then extract the audio clip $s_{i} \in \mathbb{R}^{L'} (L' < L)$ whose center is $t_{i}$.
Step 3 (optional): Use the same audio tagging model to obtain the event presence probability of each extracted audio clip $\{s_1, s_2, ..., s_n\}$ and apply thresholds $ \theta \in [0, 1]^{K} $ to get binary results $ r \in \{0, 1\}^{K} $. Then, given audio clips $S=\{s_1, s_2, ..., s_n\}$:
each $ r_{i} $
$ r_{i} \cap tr = \o $
$tr = tr \lor r_{i} $
Remove $s_i$ from $S$
Output: A set $S$ of anchor segments .

§.§ Query-based Source Separation

A typical source separator is a single-input-single-output (SISO) model that deals with one specific source, such as vocal, drum, bass, and others. To enable the model to separator arbitrary sound sources, we apply a query-based source separator by adding conditional embeddings into the source separation backbone – ResUNet <cit.>.

As mentioned in <ref>, the input to the ResUNet separator is a mixture audio segment. First, we apply short-time Fourier transform (STFT) on the waveform to extract the complex spectrum $ X = \mathbb{C}^{T \times F} $. Then, we follow the same setting of <cit.> to construct an encoder-decoder network to process the magnitude spectrogram $ |X| $.  The ResUNet encoder-decoder consists of 6 encoder blocks, 4 bottleneck blocks, and 6 decoder blocks. Each encoder block consists of 4 residual convolutional blocks to downsample the spectrogram into a bottleneck feature, and each decoder block consists of 4 residual deconvolutional blocks to upsample the feature back to separation components. The skip-connection is applied from each encoder block to the corresponding decoder block of the same downsampling/upsampling rate. For the residual block, it contains 2 CNN/DCNN layers, 2 batch normalization <ref> layers and 2 Leaky-ReLU activation layers. An additional residual shortcut is added between the input and the output of each residual block. The detail of the model architecture can be found at <cit.>.

The ResUNet separator outputs the magnitude and the phases of the complex mask $ M \in \mathbb{C}^{T \times F} $. The separated complex spectrum can be obtained by:

\begin{equation} \label{eq:mask_mul}
\begin{split}
\hat{S} &= M \odot X. \\
& = |M| \odot |X|e^{j (\angle M + \angle X)}.
\end{split}
\end{equation}

where both $|M|$ and $\angle M$ are the output of the separator. The separated source can be obtained by multiplying the STFT of mixture and the complex ideal ratio mask (cIRM). The multiplication can be also decoupled into a magnitude and a phase part. The magnitude $ |M| $ controls how much the magnitude of $ |X| $ should be scaled, and the angle $ \angle M $ controls how much the angle of $ X $ should be rotated.

Based on the ResUNet separator, we adopt a feature-wise linear modulation (FiLM) <cit.> method to construct convolutional blocks within the separator. Similar to voice filtering work <cit.> and multi-task source separation work <cit.>, for each encoder and decoder block, we incorporate the conditional embedding as:

\begin{equation} \label{eq:FiLM}
h = W * (\sigma(\text{bn}(x) + Vc))
\end{equation}

where $x$ is the output feature of the encoder and decoder block, $ V $ is a embedding layer to map the conditional vector $ c $ into an embedding space. Then, this embedding is regarded as a bias added to the the linear convolutional operation after batch normalization (BN).

A ReLU activation layer is applied to increase the non-linearity of the neural network. The final output feature $h$ will be sent to the next encoder and decoder block.

Let $f(s,c)$ denote the query-based source separator, the complete algorithm of our weakly labeled universal source separation system is illustrated in Algorithm <ref>.

§.§ Source Query Embeddings

The query-based source separator extracts pre-defined representations <cit.> or learnable representations <cit.> used to control what sources to separate from a mixture. To determine the choice of the source embedding $c$ for query-based source separation in our system, we investigate three types as follows.

§.§.§ Hard One-Hot Condition

The first type is the one-hot vector of sound classes. Each anchor segment $ s $ has a corresponding tag $ y \in [0,1]^K$. Then, the source query embedding is $ c = y $ in both training and inference stages.

§.§.§ Soft Probability Condition

The second type of source query embeddings is to use pretrained audio tagging models, such as PANN and HTS-AT, to obtain its final output, the event classification probability vector $v_e$, as the query embedding. Since the audio tagging model $ f_{\text{AT}} $ is trained on weakly labelled data with the binary cross entropy loss, which indicates that each audio clip $s$ can be categorized into each sound class with at some probabilities. Compared with the one-hot vector  The event classification probability vector $v_e$ is a good description of the audio clip $ s $.

§.§.§ Latent Embedding Condition

As mentioned in <ref>, the latent embedding $(1,H)$ from pretrained audio tagging models can be applied as the source query embedding. The advantage of using the latent embedding is that the separation are not limited to the given $ K $ sound classes. In the experiment section, we investigate a variety of PANNs, including CNN5, CNN10, CNN14, and a HTS-AT model to extract the latent embeddings and we evaluate their efficiency on the separation performance. Apart from AudioSet tagging embeddings, our conditional source separation system also supports to use automatic speech recognition (ASR) embeddings (e.g., wav2vec <cit.>) and speaker embeddings <cit.>. We also compare them with our PANN and HST-AT latent embeddings.

Furthermore, the latent embedding can be learnable during the training of our universal source separation system. One method is to train both audio tagging model and source separator from scratch. Another method is to use the pretrained audio tagging model but add a learnable shortcut to build a joint representation: $c = $ . Or, the pretrained audio tagging model can connect to a fully connected layer as an adaptor by $c=f_{\text{ada}}(f_{\text{AT}}(s))$. We will investigate how these variations of latent embeddings contribute to the source separation performance.

Source separation
[1]
Dataset $D$: AudioSet.
loss function $l$ does not converge
Sample $n$ audio tracks $\{x_1,x_2,...,x_n\}$ from the balanced sampling strategy.
Obtain $n$ audio clips $\{\s_1, s_2, ..., s_n\\}$ using the anchor segment mining algorithm <ref>.
Create the mixture $x=\sum_i s_i$ .
Obtain the source query embedding $ \{c_1,c_2,...,c_n\} $.
each $s_i$
Obtain the separation $\hat{s_{i}}=f(x,c_i)$
Calculate loss by $ l(\hat{s_{i}}, s_{i}) $

§.§ Data augmentation

AudioSet is highly unbalanced. Some sound classes only have tens number of training samples. Furthermore, when constituting the mixture with $ s_{1} $ and $ s_{2} $, the loudness of $ s_{1} $ and $ s_{2} $ can be different. To address those problems, we propose a loudness augmentation method. That is, we first calculate the energy of a signal $ s $ by $ E = || s ||_{2}^{2} $. We denote the energy of $ s_{1} $ and $ s_{2} $ as $ E_{1} $ and $ E_{2} $. We apply a scaling factor $ \alpha = \sqrt{E_{1} / E_{2}} $ to $ s_{2} $ so that both anchor segments have the same energy. That is:

\begin{equation} \label{eq:scale_augmentation}
x = s_{1} + \alpha s_{2}.
\end{equation}

On one hand, we match the energy of anchor segments to let the neural network to learn to separate the sound classes. On the other hand, the loudness diversity of sound classes are increased. We will show this loudness augmentation is beneficial in our experiments.

§.§ Loss functions

We propose to use loss functions to train the end-to-end universal source separation system. Several loss functions have been discussed in <cit.> for source separation. The most commonly used loss function is the spectrogram domain L2 loss. However, the spectrogram based loss function does not consider the phase information of the separated source. In this work, we apply a L1 loss in the time-domain waveform between ground truth source and separated source:

\begin{equation} \label{eq:loss}
l = ||s - \hat{s}||_{1},
\end{equation}

where $ l $ is the loss function to train the neural network. Lower loss in (<ref>) indicates that the separated signal $ \hat{s} $ is close to the ground truth signal $ s $. In training, the gradients of parameters are calculated by $ \partial l / \partial \theta $, where $ \theta $ are the parameters of a neural network.

§.§ Inference

In inference, the oracle embedding is unavailable due to the clean sources are unknown. Instead, we calculate $ c $ from the training dataset by:

\begin{equation} \label{eq:avg_emb}
c = \frac{1}{N}\sum_{n=1}^{N}f_{\text{emb}}(s_{n}).
\end{equation}

where $ \{s_{n}\}_{n=1}^{N} $ are query samples of one sound class and $ N $ is the number of query samples. To explain, we average all conditional embeddings of query samples from a same sound class to constitute $ c $. The $ f_{\text{emb}} $ can be either segment prediction or the latent embedding prediction.

Automatic Filtering and Separation
[1]
Inputs: audio $ x $, source separation system $ f_{\text{ss}} $, audio tagging system $ f_{\text{at}} $.
Outputs: Separated sources $ D = \{\hat{s}_{1}, ..., \hat{s}_{I}\} $.
Split $ x $ into 2-second segments. Apply $ f_{\text{at}} $ on all segments to obtain $ P(t, k) $.
# Calculate ontology predictions.
hierarchy separation
$ Q(t, m) = \text{HierarchyOntologyGrouping}(P(t, k), l) $
$ Q(t, m) = P(t, k) $
# Detect sound event candidates.
$ C = \{\} $
$m=1, ..., M$
$ \text{max}_{t}Q(t, m) > \delta$:
$ C = C \cup \{m\} $
# Do separation.
$ D = \{\} $
$ m \in C $
$t=1, ..., T$
$ Q(t, m) > \delta$:
Get condition $ c $ by (<ref>).
$ \hat{s}_{t} = f_{\text{ss}}(x, c) $
$ \hat{s}_{t} = \textbf{0} $
$\hat{s_{k}} = \{\hat{s}_{1}, ..., \hat{s}_{T}\}$
$ D = D \cup \{\hat{s}_{k}\} $

Hierarchy Ontology Grouping.
[1]
Inputs: segment-wise prediction $ P(t, k) $, hierarchy level $ l $.
Outputs: $ Q(t, m) $
Denote sound classes from AudioSet ontology with level $ l $ as a set $ C = \{ c_{1}, ..., c_{M} \} $.
$ m = 1, ..., M $
$ Q(t, m) = \text{max}\{P(t, c')\}_{c' \in \text{child}(c_{m})} $
# $\text{child}(c_{m})$ is all children sound classes of $ c_{m} $ including $ c_{m} $.

§.§ Inference with Hierarchy AudioSet Ontology

For the CASA problem, usually it is unknown how many and what sound classes are present and to separate in an audio clip. To address this problem, we propose a hierarchy sound class detection strategy to detect the present sound classes. Then, we separate the present sound classes by using the trained USS system.

We first split long audio recordings into short segments. Algorithm <ref> shows the automatic sound classes detection and separation steps. The input to the USS system is an audio clip $ x $. We first split the audio clip into 2-second segments and apply an audio tagging system $ f_{\text{at}} $ to calculate the segment-wise prediction $ P(t, k) $ with a shape of $ T \times K $, where $ T $ is the number of segments. The AudioSet ontology has a tree structure. The first level of the tree structure contains seven sound classes, including “Human sounds”, “Animal”, “Music”, “Source-ambiguous sounds”, “Sounds of things”, “Nature sounds”, and “Channel, environment and background”. Each root category contains several sub-level sound classes. The second level contains 41 sound classes. The third level contains 251 sound classes. The tree structure have a maximum depth of six levels.

In inference, the USS system supports hierarchy source separation with different levels. To separate sources of level $ l $, we denote the sound classes of level $ l $ as $ C = \{c_{1}, ..., c_{M}\} $, where $ M $ is the number of sound classes in the level $ l $. For a sound class $ m $, we denote the set of its all children sound classes including itself as $ \text{child}(m) $. For example, for the human sounds class 0, $ \text{children}(0) = \{0, 1, ..., 72\} $. For each sound class $ c_{m} $ we set its hierarchy score $ Q(t, m) $ as the maximum probability of its all children sound classes including itself: $\text{max}\{P(t, c')\}_{c' \in \text{child}(c_{m})}$. The hierarchy ontology grouping is described in Algorithm <ref>.

\begin{equation} \label{eq:inference_hierarchy}
\left\{\begin{matrix}
c_{j}=f_{\text{AT}}(x),j \in \text{child}(c)\\
c_{j}=0,j \notin \text{child}(c).
\end{matrix}\right.
\end{equation}

Then, we detect active sound classes $ C = \{m\} $ if $ \text{max}_{t}Q(t, m) $ is larger than a threshold $\delta$. Then, for each acitve sound class $m$, we detect it active segments if $ \text{max}_{t}Q(t, m) $ is larger than a threshold $ \delta $. We set separated segments to silence if $ Q(t, m) $ is smaller than $ \theta $. Then, we apply the conditional source separation system by using (<ref>) as the condition.

§ EXPERIMENTS

In this section, we investigate our proposed universal source separation system on several tasks, including AudioSet separation <cit.>, sound events separation <cit.>, music source separation <cit.>, and speech enhancement <cit.>. Our USS system is only trained on the large-scale weakly labelled AudioSet <cit.> without using any clean training data, which is a major difference from the previous source separation systems that are trained on specific datasets with clean sources <cit.>. The trained USS system can address a wide range of source separation tasks without being finetuned.

§.§ Training Dataset

AudioSet is a large-scale weakly labelled audio dataset containing 2 million 10-second audio clips sourced from the YouTube website[<https://www.youtube.com/>]. Audio clips are only labelled with the presence or absence of sound classes, without knowing when sound events occur. There are 527 sound classes in its released version, covering a wide range of sound classes in the world, such as “Human sounds”, “Animal”, etc. The training set consists of 2,063,839 audio clips, including a balanced subset of 22,160 audio clips. There are at least 50 audio clips for each sound class in the balanced training set.

Due to some audio links are no longer available, we successfully downloaded 1,934,187 (94%) audio clips from the full training set.

All audio clips are padded with silence or truncated into 10 seconds. Due to the fact that a large amount of audio recordings from YouTube have sampling rates lower than 32 kHz, we resample all audio recordings into mono and 32 kHz which will remain the information of most audio clips.

§.§ Training Details

We select anchor segments as described in <ref> and mix two anchor segments to constitute a mixture $ x $. The duration of each anchor segment is 2 seconds. We apply matching energy data augmentation to scale two anchor segments to have the same energy. and We extract the short-time Fourier transform (STFT) feature $ X $ from $ x $ with a Hann window size 1024 and a hop size 320. This hop size leads to 100 frames in a second consistent to the audio tagging systems in PANNs <cit.> and HTS-AT <cit.>.

The query net is a Cnn14 of PANNs or HTS-AT. The query net is pretrained on the AudioSet tagging task <cit.> and the parameters are frozen during the training of the USS system. The prediction and the embedding layers of the query net have dimensions of 527 and 2048, respectively. Either the prediction or the embedding layer is connected to fully connected layers and input to all layers of the source separation branch as FiLMs. We adopt ResUNet <cit.> as the source separation branch. The 30-layer ResUNet consists of 6 encoder and 6 decoder blocks. Each encoder block consists of two convolutional layers with kernel sizes of $ 3 \times 3 $. Following the pre-activation strategy <cit.>, we apply batch normalization <cit.> and leaky ReLU <cit.> before each convolutional layer. The FiLM is added to each convolutional layer as described in (<ref>). The number of output feature maps of the encoder blocks are 32, 64, 128, 256, 512, and 1024, respectively. The decoder blocks are symmetric to the encoder blocks. We apply an Adam optimizer <cit.> with a learning rate of $ 10^{-3} $ to train the system. A batch size of 16 is used to train the USS system. The total training steps is 600 k trained for 3 days on a single Tesla V100 GPU card.

§.§ Conditional Embedding Calculation

For AudioSet source separation, the oracle embedding is calculated by:

\begin{equation} \label{eq:ora_emb}
c = f_{\text{emb}}(s),
\end{equation}

where $ s $ is the clean source. Using oracle embedding as condition indicates the upper bound of the universal source separation system. We calculate the conditional embeddings by (<ref>) from the training set of the AudioSet, FSD50Kaggle2018, FSD50k, Musdb18, Slakkh2100, and VoicebankDemand datasets to evaluate on those datasets, respectively.

Confusion matrix of event based accuracy in Task 1.

Learning curve.

USS results with different conditional embedding types

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

wav2vec (46c)     8.87     4.30     8.95     8.91     4.52     4.70     1.90     7.03     2.96     8.37     -1.08     6.66     2.11     6.02

speaker (46d)     8.87     2.82     6.69     6.65     3.00     3.03     1.52     6.85     2.48     7.94     0.18     7.92     2.13     4.72

Cnn6 (45a2)     8.68     5.30     10.36     10.31     5.25     5.50     3.05     8.43     3.94     9.42     -0.37     7.37     2.27     9.39

Cnn10 (45a3)     8.35     5.36     9.95     9.90     5.19     5.43     2.87     8.10     4.11     9.34     -0.27     7.47     2.27     8.68

+Cnn14 (44a)     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

HTSAT (45c)     9.38     3.78     7.95     7.91     3.38     3.51     2.83     8.48     3.77     9.36     0.81     8.55     2.23     8.78

Segment prediction condition and embedding condition

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

Segment prediction (dim=527) (46b)     7.80     6.42     11.22     11.18     6.60     6.92     2.48     7.26     3.58     8.80     -1.69     6.05     2.20     8.30

+Embedding (dim=2048)     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

§.§ Evaluation Datasets

§.§.§ AudioSet
The evaluation set of AudioSet <cit.> contains 20,317 audio clips with 527 sound classes. We successfully downloaded 18,887 audio clips from the evaluation set (93%) out of 20,317 audio clips. The AudioSet source separation is a challenging problem due to the USS need to separate 527 sound classes using a single model. We are the first to propose using AudioSet <cit.> to evaluate USS. To create evaluation data, similar to Section <ref>, we first apply a sound event detection system to each 10-second audio clip to detect anchor segments. Then, select two anchor segments from different sound classes and sum them as a mixture. We constitute 100 mixtures for each sound class, leading to 52,700 mixtures for all sound classes in total.

§.§.§ FSDKaggle2018

The FSDKaggle2018 <cit.> is a general-purpose audio tagging dataset containing 41 sound classes ranging from musical instruments, human sounds, domestic sounds, and animals, etc. The duration of audio clips ranges from 300 ms to 30 s. Each audio clips contain a unique audio tag. The test set is composed of 1,600 audio clips with manually-verified annotations. We pad or truncate each audio clip into 2-second segments from the start due to sound events usually occur in the start of audio clips. We mix two segments from different sound classes to consist a pair. We constitute 100 mixtures for each sound class. This leads to in total 4,100 evaluation pairs.

§.§.§ FSD50K dataset

The Freesound Dataset 50k (FSD50K) dataset <cit.> contains 51,197 training clips distributed in 200 sound classes from the AudioSet ontology. Different from the FSDKaggle2018 dataset, each audio clip may contain multiple tags with a hierarchy architecture. There are in average 2.79 tags in each audio clip. All audio clips are sourced from Freesound[<https://freesound.org/>]. There are 10,231 audio clips distributed in 195 sound classes in the test set. Audio clips have variable durations between 0.3s to 30s. the average duration of audio clips are 7.1 seconds. We mix two segments from different sound classes to consist a pair. We constitute 100 mixtures for each sound class. This leads to in total 19,500 evaluation pairs.

§.§.§ MUSDB18

The MUSDB18 dataset <cit.> is designed for the music source separation task. The test set of the MUSDB18 dataset contains 50 songs with four types of stems, including vocals, bass, drums, and other. We linearly sum all stems to constitute mixtures as input to the USS system. We use the museval toolkit[https://github.com/sigsep/sigsep-mus-eval] to evaluate the SDR metrics.

§.§.§ Slakh2100

The Slakh2100 dataset <cit.> is a multiple-instrument dataset for music source separation and transcription. The test of the Slakh2100 dataset contains 225 songs. The sound of different instruments are rendered by 167 different types of plugins. We filtered 151 non-silent plugin types for evaluation. Different from the MUSDB18 dataset, there can be over 10 instruments in a song, leading to the Slakh2100 instrument separation a challenging problem.

§.§.§ Voicebank-Demand

The Voicebank-Demand <cit.> dataset is designed for the enhancement task. The Voicebank dataset <cit.> contains clean speech. The Demand dataset <cit.> contains multiple different background sounds that are used to create mixtures. The noisy utterances are created by mixing the VoiceBank dataset and the Demand dataset under under signal-to-noise ratios of 15, 10, 5, and 0 dB. The test set of the Voicebank-Demand dataset contains in total 824 utterances.

§.§ Evaluation Metrics

We use the signal-to-distortion ratio (SDR) <cit.> and SDR improvement (SDRi) <cit.> to evaluate the source separation performance. The SDR is defined as:

\begin{equation} \label{eq:sdr}
\text{SDR}(s, \hat{s}) = 10 \text{log}_{10} \left ( \frac{||s||^2}{||s - \hat{s}||^2} \right ),
\end{equation}

where $ s $ and $ \hat{s} $ are the target source and estimated source, respectively. Larger SDR indicates better separation performance. The SDRi is proposed to evaluate how much SDR a USS system improves compared to without separation:

\begin{equation} \label{eq:sdr}
\text{SDRi} = \text{SDR}(s, \hat{s}) - \text{SDR}(s, x),
\end{equation}

where $ x $ is the mixture signal. For the speech enhancement task, we apply the Perceptual evaluation of speech quality (PESQ) <cit.> and segmental signal-to-ratio noise (SSNR) <cit.> for evaluation.

Freeze, finetune, and adapt conditional embeddings

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

Cnn14 (random weights) (45b)     8.51     2.82     5.96     5.91     2.82     2.82     -0.48     4.59     1.97     7.08     -1.34     6.40     2.28     6.96

Cnn14 (scratch) (47b)     2.38      2.38     2.46     2.41     2.22     2.15     0.71     5.78     1.16     6.30     -1.20     6.54     1.62     -0.28

Cnn14 (finetune) (47b2)     9.83     1.96     3.42     3.38     1.50     1.40     2.10     7.77     3.10     8.52     1.39     9.12     1.77     3.52

+Cnn14 (freeze) (44a)     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

Cnn14 + shortcut. (50b)     6.95     4.57     9.29     9.25     4.74     4.94     1.84     7.05     3.40     8.78     -1.44     6.30     2.06     8.91

Cnn14 + adaptor (49a2)     8.01     5.81     11.00     10.96     5.79     6.07     2.95     7.96     3.90     9.24     -0.87     6.87     2.30     9.60

Universal source separation models

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

open-unmix (55c)     3.96     3.39     3.90     3.86     2.96     2.92     0.40     5.50     2.13     7.42     -1.09     6.65     2.40     2.26

ConvTasNet (56a)     6.96     5.00     9.49     9.45     5.31     5.54     0.61     5.61     2.64     8.10     -2.96     4.77     1.87     6.46

UNet (55a)     8.14     5.50     10.83     10.79     5.49     5.75     2.49     7.78     3.70     9.17     -0.45     7.29     2.12     8.47

ResUNet30     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

ResUNet60 (55b)     7.97     5.70     11.34     11.30     6.04     6.32     1.71     6.81     3.64     9.01     -2.77     4.97     2.40     9.35

§.§ Results Analysis

§.§.§ Conditional Embedding Types

The default configuration of our USS system is a 30-layer ResUNet30 trained on the balanced set of AudioSet. Table <ref> shows the USS system results trained with different conditional embedding types including wav2vec <cit.>, speaker embeddings[https://github.com/RF5/simple-speaker-embedding], Cnn6, Cnn10, Cnn14 from PANNs <cit.>, and HTS-AT <cit.>. The wav2vec embedding is trained using unsupervised contrastive learning on 960 hours of speech data. The wav2vec embeddings are averaged along the time axis to a single embedding with a dimension of 512. The speaker embedding is a gated recurrent unit (GRU) with three recurrent layers operates on log mel-spectrogram and has output has a shape of 256. The Cnn6 and the Cnn10 have dimensions of 512. The Cnn14 and the HTS-AT have dimensions of 2048. The oracle embedding (ora emb) shows the results using (<ref>) as condition. The average embedding (avg emb) shows the results using (<ref>) as condition.

Table <ref> shows that the Cnn6, Cnn10, Cnn14 embeddings achieve AudioSet SDR between 5.30 dB and 5.57 dB using the average embedding, outperforming the wav2vec of 4.30 dB and the speaker embedding of 2.82 dB. One explanation is that both wav2vec and the speaker embeddings are trained on speech data only, so that they are not comparable to PANNs and HTS-AT trained for general audio tagging. The wav2vec embedding slightly outperforms the speaker embedding on FSDKaggle2018, FSD50k, and Musdb18 separation, indicating that the unsupervised learned ASR embeddings are more suitable for universal source separation. The HTS-AT achieves the highest oracle embedding SDR among all systems.  All of Cnn6, Cnn10, Cnn14, and HTS-AT outperform the wav2vec embedding and the speaker embedding in AudioSet, FSDKaggle2018, FSD50k, Musdb18, Slakh2100, and Voicebank-Demand datasets by a large margin. The Cnn14 slightly outperforms Cnn6 and Cnn10. In the following experiments, we use Cnn14 as the default conditional embedding.

Table <ref> shows the comparison between using the Cnn14 segment prediction with a dimension of 527 and the Cnn14 embedding condition with a dimension of 2048 to build the USS system. On one hand, the segment prediction embedding achieves an SDR of 7.80 dB on AudioSet, outperforming the embedding condition of 5.57 dB. The segment prediction also achieves higher SDRis than the embedding condition on the FSDKaggle2018, and the FSD50k dataset datasets. An explaination is that the sound classes of all of the AudioSet, FSDKaggle2018, and the FSD50k datasets are sub-classes of the AudioSet. The segment prediction performs better than embedding condition in in-vocabulary sound classes separation. On the other hand, the embedding condition achieves higher SDRs than the segment prediction on the Musdb18 and the Slakh2100 dataset. This result indicates that the embedding condition perform better than segment prediction in new-vocabulary sound classes separation.

Fig. <ref> in the end of this paper shows the classwise SDRi results of AudioSet separation including 527 sound classes. The dashed lines show the SDRi with oracle segment prediction or embedding as conditions. The solid lines show the SDRi with averaged segment prediction or embedding calculated from the anchor segments mined from the balanced training subset. Fig. <ref> shows that sound classes such as busy signal, sine wave, bicycle bell achieve the highest SDRi over 15 dB. We discovered that clear defined sound classes such as instruments can achieve high SDRi scores. Most of sound classes achieve positive SDRi scores. The tendency of using segment prediction and embedding as conditions are the same, although the segment prediction outperform the embedding and vice versa in some sound classes.

§.§.§ Freeze, Finetune, and Adapt Conditional Embeddings

Table <ref> shows the comparison of using random, frozen, finetuned, and adapted conditional embeddings to build the USS system. All the variations of the conditional embeddings extractors are based on the Cnn14 architecture. Using random weights to extract conditional embeddings achieves an SDRi of 2.82 dB on AudioSet, compared to use pretrained Cnn14 to extract conditional embeddings achieves an SDR 5.57 dB. We show that using random weights to extract conditional embeddings work for all USS tasks, such as achieves an SDRi of 5.91 dB on the FSDKaggle2018 dataset compared to the pretrained Cnn14 embedding extractor of 10.57 dB.

Next, we experiment with learning the parameters of the conditional embedding extractor from scratch or finetune the weights from pretrained models. Table <ref> shows that neither the learning from scratch nor the finetuning approach improves the USS system performance. The learning from scratch approach and the finetuning approaches achieve SDRi of 2.41 dB and 3.38 dB on the FSDKaggle2018 dataset, even underperform the random weights of 5.91 dB. One explanation is that the parameters of the conditional embedding branch and the source separation branch are difficult to be jointly optimized when both of them are deep. The training falls to a collapse mode. Using the pretrained frozen Cnn14 system as conditional embedding extractor significantly improves the SDRi to 10.57 dB on the FSDKaggle2018 dataset.

Based on the pretrained frozen Cnn14 conditional embedding extractor, we propose to add a learnable shortcut or add an learnable adaptor on top of the Cnn14 system. The learnable short cut has a Cnn14 architecture with learnable parameters. Table <ref> shows that the learnable shortcut conditional embedding extractor achieves an SDR of 9.29 dB on FSDKaggle2018, less than using the pretrained frozen Cnn14 conditional embedding extractor of 10.57 dB. One explanation is that the learnable shortcut destorys the embedding information for source separation. The adaptor is a 2-layer fully connected neural network on top of the pretrained frozen Cnn14 conditional embedding extractor. With the adaptor, we achieve an SDR of 11.10 dB and outperforms the Cnn14 system. This result indicates that the adaptor is beneficial for the USS task.

USS results trained with different anchor segment durations.

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

0.5 s (52a)     4.07     2.86     4.51     4.47     2.55     2.51     0.97     0.78     2.61     2.43     -0.79     6.95     1.57     5.96

1s (52a2)     7.50     4.99     9.45     9.41     4.81     5.00     0.18     -0.02     2.54     2.50     -1.66     6.08     2.17     8.55

+2s     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

4s (52a3)     7.39     5.21     10.22     10.17     5.38     5.60     1.83     6.79     3.38     8.68     -2.62     5.12     -2.62     5.12

6s (52a4)     6.39     4.68     9.20     9.16     5.05     5.24     0.00     4.98     2.70     7.97     -4.26     3.48     2.21     2.56

8s (52a5)     6.26     4.48     8.85     8.80     4.77     4.94     -3.67     -4.00     1.60     1.50     -5.68     2.06     2.24     2.35

10s (52a6)     6.29     4.47     9.11     9.07     4.80     4.98     -2.68     -2.79     1.56     1.53     -5.07     2.67     2.13     2.14

USS results with different anchor mining strategies

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

mining     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

in-clip random (53b)     4.89     3.94     5.53     5.49     3.63     3.66     1.10     6.05     2.36     7.79     -1.72     6.01     2.21     5.69

out-clip random (53a)     8.19     5.90     11.06     11.01     6.04     6.34     2.57     7.68     3.81     9.17     -1.08     6.66     2.39     9.48

§.§.§ Separation Architectures

Table <ref> shows the results of building USS systems with different source separation backbones. The open-unmix system <cit.> is a 3-layer bidirectional long short term memory (BLSTM) system. The BLSTM is applied on the mixture spectrogram to output the estimated clean spectrogram. The open-unmix system achieves an SDR of 3.39 dB on AudioSet separation and achieves a PESQ of 2.40 on the Voicebank-Demand speech enhancement task, indicating that the BLSTM backbone performs well for speech enhancement. The open-unmix system underperforms other backbone source separation systems in FSDKaggle2018, FSD50k, Musdb18, and Slakh2100 separation, indicating that the capacity of the open-unmix system is not large enough to separate a wide range of sound classes. The ConvTasNet <cit.> is a time-domain source separation system consists of one-dimensional convolutional encoder and decoder layers. The ConvTasNet achieves an SDRi of 5.00 dB on AudioSet separation and outperforms the open-unmix system.

Our proposed UNet30 <cit.> is an encoder-decoder convolutional architecture consists of 30 convolutional layers. The ResUNet30 <cit.> adds residual shortcuts in the encoder and decoder blocks in UNet30. The UNet30 and the ResUNet30 systesm achieve SDRis of 5.50 dB and 5.57 dB on AudioSet, outperforming the ConvTasNet by around 1 dB in all source separation tasks. We extend ResUNet30 to a deeper system ResUNet60 with 60 convolutiona layers. Table <ref> shows that ResUNet60 outperforms ResUNet30 by around 0.5 dB in all USS tasks. This result indicates that deeper architectures are beneficial for USS.

§.§.§ Different Anchor Segment Durations

Table <ref> shows the results of USS systems trained with different anchor segment durations ranging from 0.5 s to 10 s. The anchor segments are mined by a pretrained SED system as described in Section <ref>. On one hand, Table <ref> shows that the separation scores increase with anchor segment durations increase from 0.5 s to 2 s and achieves the best SDRi of 5.57 dB at anchor segments of 2 s on AudioSet separation. This result shows that the anchor segment should be long enough to contain sufficient context information to build the USS system. On the other hand, the separation scores decrease with anchor segment durations decrease from 2 s to 10 s on all tasks. One explanation is that long anchor segments contain undesired interfering sounds that will impair the training of the USS system. Therefore, we use 2-second anchor segments in all other experiments.

§.§.§ Different Anchor Segment Mining Strategies

Table <ref> shows the results of different anchor mining strategies. The in-clip random strategy randomly select two anchor segments from a same 10-second audio clip which significantly underperform the SED mining strategy in all of the source separation tasks. The out-clip random strategy randomly select two anchor segments from two different 10-second audio clips. The out-clip random strategy achieves an SDRi of 5.90 dB on AudioSet, outperforms the SED mining of 5.57 dB. On one hand, the out-clip random strategy also outperforms the SED mining strategy in FSDKaggle2018 and the FSD50k dataset. On the other hand, the SED mining strategy outperforms the out-clip random strategy in Musdb18 and Slakh2100 source separation. Both the out-clip and the SED mining strategies outperform the in-clip random strategy.

USS results with different sources number

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

2 srcs to 1 src     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

3 srcs to 1-2 srcs (54a)     7.37     4.71     8.30     8.26     4.36     4.52     2.43     8.08     3.56     8.69     -0.48     7.26     2.37     8.34

4 srcs to 1-3 srcs (54a2)     7.03     4.38     7.49     7.45     3.99     4.10     2.43     7.99     3.51     8.98     0.70     8.44     2.38     7.78

USS results with different data augmentation

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

no aug (51a)     7.11     3.81     7.19     7.14     3.27     3.35     1.78     7.22     3.09     8.74     0.69     8.43     2.39     6.36

+- 20 dB (51a2)     5.51     3.62     5.77     5.73     2.93     2.94     1.69     7.02     2.51     8.03     -0.34     7.40     2.22     5.34

+match energy     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

Training with balanced and full AudioSet

2cAudioSet (SDRi)     2cFSDK2018     2cFSD50k     4cMusdb18     2cSlakh2100     2cVoicebankDemand

(lr)2-3 (lr)4-5 (lr)6-7(lr)8-11(lr)12-13(lr)14-15
ora. emb     avg. emb     SDR     SDRi     SDR     SDRi     SDR     SDRi     wSDR     wSDRi     SDR     SDRi     PESQ     SSNR

+Balanced set, 44a     8.26     5.57     10.61     10.57     5.54     5.79     3.08     8.12     4.02     9.22     -0.46     7.28     2.18     9.00

Full set, 44b     8.21     6.14     10.34     10.30     5.45     5.71     2.31     7.20     3.60     8.92     -1.29     6.45     2.40     9.45

Full set (long train), 44b2     9.60     6.76     13.07     13.03     6.52     6.85     1.77     7.00     3.51     9.00     -2.62     5.12     2.45     10.00

§.§.§ Sources number to mix during training

Table <ref> shows the USS results trained with different number of sources $ J $ to constitute a mixture. Table <ref> shows that $ J=2 $ performs the best on AudioSet with an SDRi of 5.57 dB and also performs the best on the FSDKaggle2018, FSD50k, and on Musdb18 datasets. This result shows that mixing two sources is sufficient for those source separation tasks. By using $ J=4 $ the USS system perform the beston the Slakh2100 dataset. An explanation is that the Slakh2100 contains audio clips contain multiple instruments being played simultaneously. Using more sources to constitute a mixture perform better than using fewer sources.

§.§.§ Data augmentation

Table <ref> shows the USS results with different augmentation strategies applied to sources to create a mixture. First, we do not apply any data augmentation to create a mixture. Second, we randomly scale the volume of each source by $ \pm 20 $ dB. Third, we propose a matching energy data augmentation to scale the volume of sources to create a mixture to ensure the sources have the same energy. <ref> shows that the matching energy data augmentation significantly outperform the systems trained without data augmentation and random volume scale augmentation, with an SDRi of 5.57 dB compared to 3.81 dB and 3.63 dB on AudioSet separation. The matching energy data augmentation also outperform no data augmentation and random volume augmentation on all the other tasks.

§.§.§ USS results Trained with balanced and full AudioSet

Table <ref> shows the results of training the USS systems with the balanced and the full AudioSet, respectively. The full training data is 100 times larger than the balanced data. We also experiment with training the USS system with 4 GPUs and a larger batch size of 64. Fig. <ref> shows the oracle embedding SDRi on the test set of AudioSet. The USS system trained on the full AudioSet outperforms the USS system trained on the balanced set after trained 1 million steps. Fig. <ref> also shows that a large batch sizes of 64 is beneficial to train the USS system. Table <ref> shows that training on the full AudioSet with a batch size of 64 achieves an SDRi of 6.76 dB, outperforming training on the balanced set of 5.57 dB.

§.§.§ Visualization of Hierarchy Separation

One application of the hierarchy separation is to separate arbitrary audio recordings into individual sources with AudioSet ontology. For example, the USS system can separate the sound in a movie into different tracks. One challenge of the hierarchy separation is the number of present sources are unknown. We use the methods in Section <ref> to detect and separate the present sound events.

Fig. <ref> shows the automatically detected and separated waveforms of a movie clip Harry Potter and the Sorcerer's Stone from ontology levels 1 to 3 by using Algorithm <ref>. Level 1 indicates coarse sound classes and level 3 indicates fine sound classes. In level 1, the USS system successfully separate human sounds, music and sounds of things. In level 2, the USS system further separate human group actions, vehicle, and animals. In level 3, the USS system separate fine-grained sound classes such as bell, bird, crowd, and scary music.

Confusion matrix of event based accuracy in Task 1.

§ CONCLUSION

In this paper, we propose universal source separation (USS) systems trained on the large-scale weakly labelled AudioSet. The USS systems can separate hundreds of sound classes using a single model. The separation system can achieve zero-shot separation by using the embedding calculated from query examples as condition. In training, we first apply a sound event detection (SED) system to detect the anchor segments that are most likely to contain sound events. We constitute a mixture by mixing several anchor segments. Then, we use a pretrained audio tagging system to calculate the segment prediction probability or the embedding vector as the condition of the target anchor segment. The USS system takes the mixture and the condition as input to output the desired anchor segment waveform. In inference, we propose both zero-shot separation and hierarchy separation with an AudioSet ontology. We evaluated our proposed USS systems on a wide range of separation tasks, including AudioSet separation, FSDKaggle2018 and FSD50k general sound separation, Musdb18 and Slakh2100 music instruments separation, and Voicebank-Demand speech enhancement without training on those datasets. We show the USS system is an approach to address the computation auditory scene analysis (CASA) problem. In future, we will improve the quality of the separated waveforms of the weakly labelled USS systems.

AudioSet source separation result.

Confusion matrix of event based accuracy in Task 1.

Confusion matrix of event based accuracy in Task 1.