# Enforcing Conditional Independence for
Fair Representation Learning and Causal Image Generation

Jensen Hwa1, Qingyu Zhao2, Aditya Lahiri3, Adnan Masood4, Babak Salimi3, Ehsan
Adeli1
1Stanford University 2Weill Cornell Medicine 3University of California San
Diego 4UST
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Conditional independence (CI) constraints are critical for defining and
evaluating fairness in machine learning, as well as for learning unconfounded
or causal representations. Traditional methods for ensuring fairness either
blindly learn invariant features with respect to a protected variable (_e.g_.,
race when classifying sex from face images) or enforce CI relative to the
protected attribute only on the model output (_e.g_., the sex label). Neither
of these methods are effective in enforcing CI in high-dimensional feature
spaces. In this paper, we focus on a nascent approach characterizing the CI
constraint in terms of two Jensen-Shannon divergence terms, and we extend it
to high-dimensional feature spaces using a novel dynamic sampling strategy. In
doing so, we introduce a new training paradigm that can be applied to any
encoder architecture. We are able to enforce conditional independence of the
diffusion autoencoder latent representation with respect to any protected
attribute under the equalized odds constraint and show that this approach
enables causal image generation with controllable latent spaces. Our
experimental results demonstrate that our approach can achieve high accuracy
on downstream tasks while upholding equality of odds.

##  1 Introduction

Figure 1: We propose a new way to ensure fairness in downstream tasks by
enforcing conditional independence constraints on the latent representation.
This is achieved by minimizing the Jensen-Shannon divergence (JS) between
distributions obtained using a novel dynamic sampling technique. In the
setting shown here, we apply our technique to the diffusion autoencoder’s
semantic representation to disentangle the sensitive attribute of skin type (a
proxy variable for race) and perform causal image generation.

Fairness in machine learning and computer vision is an increasingly important
topic with growing applications to everyday life. The literature includes
several methods for learning fair and unconfounded models based on domain
adversarial invariant learning [13, 37, 33, 18, 4], statistical methods [27,
16], and information theory [21].

In the context of predictive modeling, algorithmic fairness aims to learn
models that are unbiased towards protected subgroups by ensuring that the
output label $\hat{y}$ is invariant to a sensitive attribute $s$. The existing
plethora of fairness definitions typically captures this invariance in terms
of a conditional independence (CI) (see Mehrabi et al. for a survey). For
instance, the equalized odds criterion [10] requires prediction output
$\hat{y}$ to be independent of the sensitive attribute $s$ conditioned on the
true class label $y$. We note that the conditional independence formulation
$s\perp\\!\\!\\!\\!\perp\hat{y}\mid y$ is widely recognized as a more precise
indicator of fairness compared with the marginal independence formulation
$s\perp\\!\\!\\!\\!\perp\hat{y}$, due to the fact that some correlations
between sensitive attributes and the target variable may be benign. CI plays a
crucial role in capturing the true causal relationships between the sensitive
attributes and the target variable, ensuring that fairness is based on genuine
causes of disparities rather than arbitrary associations [14, 24].

However, most existing work in fair machine learning typically focuses on
enforcing independence on low-dimensional features in cases where the
conditioning set is either empty or merely a low-dimensional categorical
variable [24, 25, 3, 37, 2]. This restricts the learning of fair
representations to limited settings involving only binary attributes in
tabular datasets [35]. These methods cannot directly translate to high-
dimensional spaces, such as the latent representations of large generative
models.

Recently, Ahuja et al. developed a differentiable framework for enforcing CI
in high-dimensional and continuous feature spaces by using a GAN-based
approach in the context of data generation. To enforce
$s\perp\\!\\!\\!\\!\perp\hat{y}\mid y$, they minimize the Jensen-Shannon (JS)
divergence between the joint distribution $p(s,\hat{y},y)$ and an auxiliary
distribution $q(s,y^{\prime},y)$ while the joint marginal $q(s,y)$ is
distributed identically to $p(s,y)$ and $q(y^{\prime}\mid y)$ is independent
of $s$. Although this formulation enforces CI, it can only sample the joint
probability distributions in the label space $y$ and does not necessarily
guarantee the learning of fair representations, _i.e_., only the last network
layer before predicting $y$ removes the dependence while the rest of the
network (including latent representations) remain biased or confounded.

Our approach introduces the concept of CI into the latent space by using a
dynamic sampler to form the joint distributions with respect to the learned
representations. This yields a computationally rigorous approach to fair and
causal representation learning that exhibits a level of versatility not found
in existing methods: We are able to control not just which features are
encoded, but also where within the representation they appear. With this, we
gain improved classification performance and, in the case of generative models
such as the diffusion autoencoder (DiffAE), the ability to maintain fine-
grained control over generated content. As a result, we can not only protect
downstream tasks from bias against specific attributes, but also generate
feature-invariant images corresponding to the enforced schema, allowing for a
richer understanding of a model’s representations.

##  2 Related Work

Figure 2: High-level view of our architecture. We introduce two variants of a
conditional independence enforcer that can be added to any off-the-shelf
encoder.

##### Invariant Representation Learning

In the wake of increasingly biased large-scale models [10, 1, 18, 31], the
learning of fair and unconfounded representations has taken on outsized
importance. Methods based on domain adversarial learning are increasingly
popular tools of choice in reducing bias and confounding effects [33, 1, 37,
15, 34]. Among these, Johndrow et al. and Tan et al. proposed methods for
invariant feature learning and Zhao et al. presented models that minimize
statistical mean dependence using a correlation-based adversarial loss
function. Recently, Pirhadi et al. introduced a data cleaning method that
enforces the conditional independence constraint for tabular data using
optimal transport.

Other types of methods incorporating statistical operations have also been
explored in the literature, such as using multivariate regression analysis
[19] with general linear models [36]. Other approaches enforce fairness
through post-processing steps on unfair trained classification models [6, 10,
36]. But since the training and fairness enforcement steps are conducted
separately, these algorithms often lead to suboptimal fairness and accuracy
trade-offs [32]. Recently, using traditional statistical methods, Lu et al.
proposed a normalization layer that corrects the feature distributions with
respect to labeled sensitive attributes. Their approach, known as metadata
normalization, entailed a simple layer that could plug into any end-to-end
model to protect the models against bias. Vento et al. then extended this work
by turning the closed-form normalization operation into a network optimizable
step. This is an active research field and many other approaches are being
studied.

##### Conditional Independence for Algorithmic Fairness

Consider a classifier or regression function $f(x)$ with output $\hat{y}$ and
a protected attribute $s$, _e.g_., sex or race. Algorithmic fairness aims to
learn prediction functions or transformations that make the final outcome
invariant or insensitive to protected attributes. Several widely used
associational and causal notions of fairness, including the equalized odds
criterion, can be attained by means of independence [25].

Rather than attempting to transform already trained unfair models to fair ones
in a post-hoc manner, it is critical to enforce fairness constraints from the
beginning of the model training pipeline [11]. Apart from preventing the
propagation of bias to downstream modeling tasks, this approach of fairness
from a data management standpoint also leads to more robust and significant
fairness measures by ensuring that data sources, transformations, and other
training assumptions are sound [24].

##### Representation Learning with Diffusion Models

Diffusion-based models learn to generate images by a denoising process: Random
Gaussian noise is added to input images, and the model learns how to reverse
this process to hallucinate new images from random noise. This family of
models has proven to offer remarkable quality in image generation, promising
to replace generative adversarial networks (GANs) [8] as the dominant
architectural paradigm. Chief among them is the diffusion autoencoder [23],
which encodes an image into a two-part latent subcode, capturing a stochastic
representation via the aforementioned denoising approach and conditioning this
process upon a semantic representation learned by a CNN. Interpolation of the
resulting subcode results in smooth and meaningful changes in the decoded
image. The semantic separation capability of DiffAEs has enabled various
applications, such as attribute manipulation and various low-shot or zero-shot
downstream applications [26, 7, 9]. In this work, we further enable DiffAEs to
learn unbiased, causal, and conditionally independent latent subcode
representations.

##  3 Method

Let $\\{x,y,s\\}$ be a training sample in the dataset, where $x$ is an input
image, $y$ is the target prediction label, and $s$ is a protected sensitive
attribute. We aim to learn an encoder network $g_{\theta}(x)=v$ resulting in a
latent representation $v$, from which a prediction network
$f_{\phi}(v)=\hat{y}$ produces the final predicted label $\hat{y}$. With a
slight abuse of notation, we use lower-case letters to also denote random
variables. Ultimately, we aim to train a model that achieves high accuracy in
predicting label $y$ while ensuring equalized odds:

$\begin{split}\operatorname*{argmin}_{\theta,\phi}~{}-\frac{1}{n}\sum_{i=1}^{n}y_{i}\log(\hat{y}_{i})&+(1-y_{i})\log(1-\hat{y}_{i})\\\
\text{s.t.}\quad s\perp\\!\\!\\!\\!\perp\hat{y}~{}|~{}y.\end{split}$ (1)

We discuss two possible ways of enforcing the equalized odds fairness
constraint: First, we enforce conditional independence with respect to the
label $y$, _i.e_., $s\perp\\!\\!\\!\\!\perp\hat{y}~{}|~{}y$. As discussed in
Ahuja et al., this can be seen as a means to enforce equalized odds through
CI. However, we claim that enforcing the independence constraint with respect
to only the label is limited in its ability to enforce the equalized odds
fairness constraint effectively. We therefore additionally propose to enforce
the conditional independence with respect to the more information-rich latent
representation $v$, or $s\perp\\!\\!\\!\\!\perp v~{}|~{}y$. We can then use
this learned fair latent representation in a downstream prediction or
generation task.

###  3.1 Enforcing CI with respect to Label

When the label $y$ is highly correlated with the protected attribute $s$ in
the training data, a basic aim for a fair machine learning model is to ensure
conditional independence of the predicted $\hat{y}$ from $s$, _i.e_.,
$s\perp\\!\\!\\!\\!\perp\hat{y}~{}|~{}y$.

Assuming that the joint distribution $p(s,\hat{y},y)$ is well-defined with
respect to Lebesgue measure $\mu$, previous work [3] demonstrates that the
above conditional independence can be defined with respect to the Jensen-
Shannon divergence (JS) between $p(s,\hat{y},y)$ and an auxiliary distribution
$q(s,y^{\prime},y)$, where the joint marginal $q(s,y)$ is distributed
identically to $p(s,y)$, and $q(y^{\prime}|y)$ is independent of $s$. In other
words, when $q(s,y^{\prime},y)=p(s,y)q(y^{\prime}|y)$, the conditional
independence constraint $s\perp\\!\\!\\!\\!\perp\hat{y}~{}|~{}y$ is equivalent
to enforcing

$\text{JS}(p(\hat{y},y),q(y^{\prime},y))=\text{JS}(p(s,\hat{y},y),q(s,y^{\prime},y)).$
(2)

Accordingly, we can rewrite (1) in terms of JS as

$\begin{split}\operatorname*{argmin}_{\theta,\phi}~{}&-\frac{1}{n}\sum_{i=1}^{n}y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})\\\
\text{s.t.}\quad&\big{(}\text{JS}(p(\hat{y},y),q(y^{\prime},y))-\\\
&\text{JS}(p(s,\hat{y},y),q(s,y^{\prime},y))\leq\delta\big{)}\end{split}$ (3)

for some $\delta>0$ sufficiently small.

With the help of a conditional sampler $q(y^{\prime}~{}|~{}y)$, the JS
constraint of (3) can be satisfied by a general GAN architecture with two
discriminators (Fig. 2) [3], _i.e_., by minimizing

$L_{\text{CI}}=(L_{\text{D1}}-L_{\text{D2}})^{2}$ (4)

where

$\displaystyle L_{\text{D1}}$
$\displaystyle=\mathbb{E}[\text{log}(1-D_{1}(y^{\prime},y))+\mathbb{E}_{y,s}[\log
D_{1}(\hat{y},y)]\quad\text{and}$ $\displaystyle L_{\text{D2}}$
$\displaystyle=\mathbb{E}[\text{log}(1-D_{2}(y^{\prime},s,y)]+\mathbb{E}_{y,s,y^{\prime}}[\log
D_{2}(\hat{y},s,y)].$

The key advantage of using (4) to derive conditional independence is that
$q(y^{\prime}|~{}y)$ does not have to be a perfect sampler that exactly
matches $p(\hat{y}~{}|~{}y)$. The only sufficient condition is that
$q(y^{\prime}|~{}y)$ shares overlapping support with the original distribution
$p(\hat{y}~{}|~{}y)$. This gives us many flexible ways to construct $q$, such
as using a uniform distribution.

Specifically, in a supervised learning setting, where $y$ is known at training
time, we can set $q$ using a uniform distribution over all raw model outputs
that correspond to $y$. The encoder and prediction networks can then be
learned by optimizing the loss function

$L_{\text{Enc}}=\lambda L_{\text{CI}}+L_{\text{task}},$ (5)

where $\lambda$ is a hyperparameter and $L_{\text{task}}$ is the loss function
for the prediction task. The discriminators can be trained adversarially to
minimize the composite loss function:

$L_{\text{D}}=L_{\text{D1}}+L_{\text{D2}}.$ (6)

###  3.2 Enforcing CI with respect to Latent Representation

Enforcing conditional independence only with respect to $\hat{y}$ does not
necessarily prevent the model from learning biased information from $s$ in its
representations. In this paper, we propose to enforce conditional independence
with respect to the latent representation $v$ in a way that most directly
impacts the learnable parameters of the encoder. We achieve this by replacing
the predicted label $\hat{y}$ with the latent vector $v$ within our
conditional independence loss described above, _i.e_.,
$s\perp\\!\\!\\!\\!\perp v~{}|~{}y$. Similar to (2), our Jensen-Shannon
divergence constraint becomes

$\text{JS}(p(v,y),q(v^{\prime},y))=\text{JS}(p(s,v,y),q(s,v^{\prime},y)).$ (7)

As before, this requires a conditional sampler $q(v^{\prime}|y$). However, one
problem that arises with this approach is that the conditional sampler is no
longer well-defined at training time. Since the distribution and support of
$p(v|y)$ is learned by the model on the fly, we cannot construct
$q(v^{\prime}|y)$ a priori.

To resolve this challenge, we implement the imperfect conditional sampler $q$
via a novel dynamic sampling procedure. Specifically, to sample
$v^{\prime}\sim q(v^{\prime}|y)$, we sample with replacement from the latent
vectors associated with the given $y$ within the same training batch. Similar
to the bootstrapping procedure, our dynamic sampling can approximate the
empirical distribution function $p(v|y)$, allowing the discriminators to keep
up with and robustly train the encoder as the latent space is learned. Note
that, other than the difference in the sampler, the objective function of (4)
and the GAN architecture in Fig. 2 can be translated here by replacing
$\\{\hat{y},y^{\prime}\\}$ with $\\{v,v^{\prime}\\}$.

###  3.3 Disentangling DiffAE Latent Representation

The above construction of $q$ provides us with a powerful technique that can
be applied to any encoder architecture. We demonstrate this by using the
state-of-the-art diffusion autoencoder model [23], which encodes images using
a denoising diffusion process [26] conditioned upon a semantically meaningful
representation inferred by a CNN encoder. An image, therefore, is encoded by
two representations: a semantic subcode learned by the CNN, along with a
stochastic subcode that, given the semantic subcode, denoises into the
original image. The model learns to compress the most common high-level
features into the semantic subcode and delegate the remaining, highly variable
features to the stochastic representation. When applied to human face images,
this yields a clean separation between anatomical features such as sex, facial
structure, and skin type in the semantic subcode, and more transient qualities
like pose, hairstyle, and expression in the stochastic subcode.

By enforcing conditional independence with respect to the semantic subcode of
the diffusion autoencoder, we produce a latent representation that is
invariant to a protected attribute of choice. The model adapts by encoding any
facial features associated with the protected attribute in the stochastic
subcode instead. When such facial features include the high-level attributes
described earlier, this can unduly affect the natural dichotomy between the
semantic and stochastic subcodes, restricting the expressive power of the
semantic subcode and reducing the efficiency of the denoising process used for
image generation. Empirically, this results in blurred and unrealistic
generated images.

To overcome this, we enforce conditional independence with respect to only a
portion of the semantic subcode. To further encourage the model to use the
remaining portion to encode the facial features related to the sensitive
attribute, we add a prediction head on the remaining portion to estimate the
sensitive attribute $s$. We optimize this new prediction head alongside the
encoder; instead of (5), we have

$L_{\text{Enc}}=\lambda_{1}L_{\text{CI}}+\lambda_{2}~{}\text{BCE}(\hat{s},s)+L_{\text{task}}.$
(8)

In this way, the semantic subcode is apportioned into two parts: one variant
to the sensitive attribute, and the other invariant. Although we only
demonstrate this capability on tasks involving a single sensitive attribute,
this approach can be easily extended to disentangle multiple attributes within
the semantic subcode. With this formulation for causal disentanglement of the
latent space, we show that the specific subcode in which a given facial
attribute is represented can be methodically adjusted (_e.g_., to change the
race of the generated image), all while preserving the model’s ability to
accurately reconstruct other characteristics of the input image.

##  4 Experiments and Results

We apply our architecture to two different settings, each with a unique
dataset, model, and confounding variable. In each case, we analyze the
effectiveness of applying CI in the latent space as opposed to the label space
and also compare with several baselines. Experiments were performed using a
P100 GPU, with the exception of the diffusion autoencoder experiments which
required four A100 GPUs.

##### Metrics

We report prediction accuracies for each value of the protected attribute when
$s$ is discrete, along with the balanced accuracy (bAcc) to account for class
imbalances. To evaluate invariance to the sensitive attribute, we use the
squared distance correlation (dcor2) [28] to quantitatively measure the
correlation between the protected variable and the learned features. When the
protected variable is discrete, we also use the equality of opportunity (EO)
independence metric, which measures the average gap in true positive rates for
different possible values of the protected variable.

Figure 3: Synthetic data format and sample. The diagonal kernels are
controlled by $\sigma_{A}$, while the off-diagonals are controlled by
$\sigma_{B}$.

###  4.1 Synthetic Experiments

As an initial proof of concept, we train a convolutional neural network on
synthetically generated image data. Each image in this dataset is of size
$32\times 32$ and composed of four Gaussian kernels, as shown in Fig. 3. The
diagonal kernels are of equal intensity $\sigma_{A}$, and the off-diagonal
kernels are of equal intensity $\sigma_{B}$. For half of the images, we sample
$\sigma_{A},\sigma_{B}\sim\mathcal{U}(1,4)$ and assign the label $Y=0$. For
the other half, we sample $\sigma_{A},\sigma_{B}\sim\mathcal{U}(3,7)$ and
assign $Y=1$. We then denote $\sigma_{B}$ as the protected attribute and
analyze the model’s ability to predict an image’s label based on the diagonal
kernels controlled by $\sigma_{A}$ while ignoring the off-diagonal kernels
controlled by $\sigma_{B}$. We do this by enforcing the conditional
independence condition separately with respect to the label space
($\sigma_{B}\perp\\!\\!\\!\\!\perp\hat{Y}~{}|~{}Y$) and the latent space
($\sigma_{B}\perp\\!\\!\\!\\!\perp V~{}|~{}Y$).

Whereas an unconstrained model may achieve maximum theoretical accuracy of
$1-\frac{1}{2}\left(\frac{1}{3}\right)^{2}=0.94$, correctly predicting all
cases except half of those where $\sigma_{A},\sigma_{B}\in(3,4)$, a fair
classifier using only the information represented by $\sigma_{A}$ can attain a
theoretical accuracy of $1-\frac{1}{2}\left(\frac{1}{3}\right)=0.83$.

Our encoder is a simple CNN comprised of a convolutional layer, ReLU
activation, max pooling, convolution, ReLU, and two linear layers, applied in
that order. The first linear layer has an output of length 10, which we define
as the latent space $v$, while the second linear layer converts this latent
space into the single output logit corresponding to $y$.

In Table 1, we report metrics after training each model using a batch size of
512 and a learning rate of $1\mathrm{e}{-4}$. We select the $\lambda$
hyperparameter by 5-fold cross-validation and report metrics based on a
separate test set.

Fig. 5 demonstrates the tradeoff of fairness versus accuracy and dcor2 for
both model variants. The latent space variant introduces some instability for
larger $\lambda$, but achieves increased accuracy alongside decreased dcor2.
While some instability is present for large $\lambda$, this largely does not
appear until after dcor2 has converged.

In Fig. 6, we show each model’s unnormalized output for each integer
combination of $\sigma_{A}$ and $\sigma_{B}$. Enforcing CI in the latent space
produces outputs that remain nearly unaffected by changes in the protected
attribute $\sigma_{B}$, as evidenced by near-constant values within each row.

Figure 4: Synthetic data causal diagram. We apply the $L_{CI}$ component to either the latent vector space (V) or the label space (Y). Table 1: Synthetic data experiment results. Balanced accuracy (bAcc) closer to $0.83$ is better, as is lower dcor2.  Model | bAcc | dcor2
---|---|---
Vanilla CNN | 0.94 | 0.432
Regularized $v$-space CNN | 0.80 | 0.382
$y$-space CI-CNN | 0.87 | 0.298
$v$-space CI-CNN | 0.84 | 0.055
Figure 5: Fairness and accuracy metrics versus conditional independence
strength $\lambda$. Orange lines correspond to the $y$-space CI-CNN, and blue
lines to the $v$-space CI-CNN. Figure 6: Unnormalized model predictions on
synthetic data for given $\sigma_{A},\sigma_{B}$. Negative values imply a
prediction of $\hat{y}=0$, while positive values correspond to $\hat{y}=1$.

##### Model B1: Vanilla CNN

As a baseline, we train the standalone CNN without any conditional
independence enforcing mechanism, using only a binary cross entropy loss
function. This model relies heavily on the protected attribute $\sigma_{B}$,
as shown by a high bAcc and $dcor^{2}$.

##### Model B2: Regularized $v$-space CNN

To evaluate the efficacy of our conditional independence enforcement, we
establish a baseline by embedding the equalized odds condition directly into
the loss function. Defining $R_{0}$ and $R_{1}$ to be the absolute difference
between the true positive rate and false positive rate for $Y=0$ and $Y=1$
respectively, the loss function becomes

$L=\lambda(R_{0}+R_{1})+\text{BCE}(\hat{y},y).$ (9)

We tune the $\lambda$ parameter over the validation set to target the
theoretical maximum accuracy under conditional independence. This regularizer
appears to have an overbearing effect on the model, and despite the reduced
dcor2, it struggles to learn a meaningful representation.

##### Model 1: $y$-space CI-CNN

We use the output of the final linear layer as input to our conditional
independence enforcing loss component and apply binary cross entropy loss:

$L_{\text{Enc}}=\lambda L_{\text{CI}}+\text{BCE}(\hat{y},y),$ (10)

where, as in (5), $\lambda$ controls the strength of conditional independence
enforcement and $L_{CI}$ is the loss from the dual discriminators. We find
that the model learns a representation somewhat uncorrelated to $\sigma_{B}$,
but only in an unstable manner without much improvement upon the regularized
baseline.

##### Model 2: $v$-space CI-CNN

In our second iteration, we insert the conditional independence component
closer to the latent space of the model, before the final linear layer, and
employ the dynamic sampling procedure to simulate the conditional latent space
distribution. The resulting model exhibits a much smaller confounding effect,
achieving an accuracy closest to the theoretical maximum under CI and the
lowest correlation between the protected and target variables. This
demonstrates the effectiveness of our dynamic resampling procedure, which
makes practical the enforcement of conditional independence with respect to
the latent space and is one of the key contributions of this paper.

###  4.2 Face Image Experiments

Table 2: Classification of sex from Gender Shades dataset facial images, with mean and standard deviation across five runs.  Model | bAcc (%) | I | II | III | IV | V | VI | EO (%) | dcor2
---|---|---|---|---|---|---|---|---|---
Kim _et al_. [13] | 88.8 $\pm$ 1.3 | 87.8 | 93.0 | 93.6 | 91.7 | 87.4 | 86.2 | 8.4 $\pm$ 3.6 |  0.022 $\pm$ 0.012
Multi-task [17] | 93.9 $\pm$ 0.4 | 92.7 | 95.6 | 93.6 | 100 | 95.8 | 87.3 | 7.3 $\pm$ 1.6 | 0.190 $\pm$ 0.012
BR-Net [1] | 94.8 $\pm$ 0.4 | 92.7 | 97.4 | 93.6 | 100 | 95.8 | 88.5 |  5.3 $\pm$ 1.2 | 0.170 $\pm$ 0.007
Vanilla Diffusion Autoencoder | 93.0 $\pm$ 0.6 | 85.4 | 95.6 | 100 | 91.7 | 87.3 | 90.1 | 11.5 $\pm$ 1.1 | 0.326 $\pm$ 0.000
$y$-space CI-DiffAE |  94.8 $\pm$ 2.1 | 80.5 | 93.7 | 93.6 | 87.5 | 81.1 | 86.2 | 9.9 $\pm$ 6.0 | 0.120 $\pm$ 0.034
$v$-space CI-DiffAE |  96.6 $\pm$ 1.8 | 97.5 | 100 | 97.9 | 91.7 | 93.7 | 87.3 |  5.0 $\pm$ 2.5 |  0.076 $\pm$ 0.025

In selecting an experimental setting in which to apply our architecture to the
diffusion autoencoder, we searched for datasets (1) labeled with a relevant
secondary attribute that we could treat as a confounding variable, and (2)
compatible with other public datasets that could be used to pretrain the
DiffAE before finetuning. One of the few datasets meeting these criteria was a
set of 1,270 face images provided by the Gender Shades project [5],
representing subjects from 3 African countries and 3 European countries. Each
image is labeled with both skin type, using the Fitzpatrick classification
system, and sex, inheriting a binary male/female grouping as a simplified
proxy for gender. We aligned and cropped the images to the format expected by
the diffusion autoencoder model, and then employed our architecture to train
the diffusion autoencoder to predict sex while being conditionally independent
to skin type.

We initialized a pretrained diffusion model based on the $128\times 128$
Flickr-Faces-HQ Dataset, which consists of 70,000 highly varied face images.
We treat the model’s semantic subcode as the latent space and add a single
linear layer prediction head to convert from this 512-length vector space to a
single logit representing the sex label space.

##### Models B1-3

We implement three baseline models. BR-Net [1] uses adversarial training to
learn image features that have statistical “mean independence” with the
protected attribute. The adversarial objective of BR-Net is based on Pearson’s
correlation between the true and the predicted value of the protected
attribute. Second, Kim et al. uses mutual information minimization to ensure
that the learned image features cannot predict the protected attribute.
Finally, a multi-task network [17] is implemented to use the same encoder to
predict both the target value and the protected attribute simultaneously.

##### Model B4: Vanilla Diffusion Autoencoder

For a closer comparison to our architecture, we train the diffusion
autoencoder using its default loss function, denoted in [23] as
$L_{\text{simple}}$. In a separate optimization step, we train the prediction
head using binary cross entropy loss (BCE).

##### Model 1: $y$-space CI-DiffAE

We unify the training of the diffusion autoencoder and the prediction head
under a single loss function and enforce conditional independence with respect
to the output of the prediction head. In the context of (5), the encoder
objective function becomes

$L_{\text{Enc}}=\lambda
L_{\text{CI}}+\left(L_{\text{DiffAE}}+\rho~{}\text{BCE}(y,\hat{y})\right),$
(11)

for hyperparameter $\rho$.

Figure 7: Causal graph corresponding to face dataset. By way of the latent
space $V$, we train a model to predict sex ($Y$) from face image ($X$) while
being invariant to skin color ($S$). Here, countless other factors ($I$) such
as genetics and lighting also influence each image, and hidden confounders
(dashed lines) cause correlation among inputs. Figure 8: Selected image
reconstructions. For each variant of our CI-DiffAE, we reconstruct two images
from the dataset while adjusting the race-variant portion of the semantic
subcode: the leftmost image in each group corresponds to skin type 1, and the
rightmost to skin type 6. The latent ($v$-space) CI constraint can effectively
disentangle skin shade from other facial features.  Figure 9: Race-invariant
image generation using our $v$-space partial CI-DiffAE architecture. Each
column contains hallucinated images of the same skin type.

##### Model 2: $v$-space CI-DiffAE

In an analogous manner to the $v$-space CI-CNN, we enforce conditional
independence directly on the semantic subcode, keeping other portions of the
model the same.

#####

Results in Table 2 indicate that the $v$-space constraint yields an all-around
improvement in accuracy and fairness compared to the vanilla model, as opposed
to the $y$-space constraint which achieves lower dcor2 with considerable
impact on accuracy. This improvement is statistically significant as measured
by a McNemar test ($\chi^{2}=8.257$, $p\text{-value}=0.0040$). Overall, these
results demonstrate that the $v$-space CI constraint produces results well on-
par with existing work. Moreover, this technique has the added capability of
causal image generation, described below.

####  4.2.1 Visual Results

Due to the strong association between skin type and race, skin type impacts a
large number of facial features. Our architecture removes much of this
association, which creates a race-invariant representation, but as a result,
also constricts the model’s ability to encode anatomical facial features in
the semantic subcode. Therefore, nearly all information becomes encoded by the
stochastic subcode, and generated images are less realistic.

As discussed previously, we remedy this effect by enforcing conditional
independence on only a portion of the semantic subcode and simultaneously
training a sensitive attribute prediction head on the remaining portion. We
find that this partial CI-DiffAE architecture allows the semantic subcode as a
whole to continue encoding the facial features necessary to produce a clean
image. To interpret the effects on the semantic subcode, we generate images
after adjusting the race-variant portion of the semantic subcode to values
corresponding to each skin type (Fig. 8). We observe that the $v$-space model
is effectively (and causally) able to disentangle race-related features within
the semantic subcode, resulting in a smooth and meaningful transition with
changes limited to those relating to race.

Furthermore, by sampling the race-invariant portion of the semantic subcode,
we retain the diffusion autoencoder’s ability to generate novel images, but
with added control over skin type (Fig. 9). These images demonstrate how our
technique creates the race-invariant portion of the latent representation by
removing the direct causal effect of skin shade, analogous to deleting the
solid arrow between $S$ and $X$ in Fig. 7. At the same time, due to the fine-
grained nature of the equalized odds criterion we enforce, the representation
preserves the indirect effect of skin shade that is mediated by sex, shown in
Fig. 7 as the dashed arrow between $Y$ and $S$. Therefore, the model continues
to capture differences in gender expression across race, most notably in the
headwraps often worn by women in Sub-Saharan African cultures. By
disentangling race from the latent representation, we gain a richer insight
into the causal processes that underlie facial images, crucial for effective
image generation.

##  5 Conclusion

We introduced a framework to ensure fair and unconfounded representation
learning during training and demonstrated both its versatility when applied to
complex models and its effectiveness when compared to alternative methods. We
iterated upon the theoretical idea of expressing the conditional independence
constraint as an equality of two Jensen-Shannon divergences and extended this
to high dimensional latent space via a dynamic sampling technique that can be
easily implemented for any encoder. Our work exposes a new approach to
generally enforce a conditional independence constraint on a model, which can
then be used in downstream tasks such as causal image generation and fair
predictive models. To our knowledge, this is the only model-agnostic training
approach to be shown effective on enforcing specific features to be encoded in
given dimensions of the latent space. We are optimistic that further
experimentation will reveal applications to other tasks concerning fairness
and disentanglement. Moreover, as a direction for future work, the use of
conditionally invariant embeddings may prove useful in extending traditional
causal methods like mediational analysis to complex, high-dimensional
settings.

## Acknowledgement

This research was partially supported by UST, Stanford Institute for Human-
Centered AI (HAI) GCP cloud credits, and National Institutes of Health (NIH)
grants U54HG012510 and AG084471. J.H. was also supported by a research fund
from Panasonic and E.A. by the Stanford School of Medicine Jaswa Innovator
Award.

## References

  * Adeli et al. [2021] Ehsan Adeli, Qingyu Zhao, Adolf Pfefferbaum, Edith V. Sullivan, Li Fei-Fei, Juan Carlos Niebles, and Kilian M. Pohl.  Representation learning with statistical independence to mitigate bias.  In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 2512–2522, 2021.
  * Agrawal et al. [2021] Raj Agrawal, Chandler Squires, Neha Prasad, and Caroline Uhler.  The decamfounder: Non-linear causal discovery in the presence of hidden variables, 2021.
  * Ahuja et al. [2021] Kartik Ahuja, Prasanna Sattigeri, Karthikeyan Shanmugam, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Murat Kocaoglu.  Conditionally independent data generation.  In _Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence_ , pages 2050–2060. PMLR, 2021.
  * Akuzawa et al. [2019] Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo.  Adversarial invariant feature learning with accuracy constraint for domain generalization.  _arXiv preprint arXiv:1904.12543_ , 2019.
  * Buolamwini and Gebru [2018] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pages 77–91, 2018.
  * Feldman [2015] Michael Feldman.  _Computational fairness: Preventing machine-learned discrimination_.  PhD thesis, Haverford College, 2015.
  * Giannone et al. [2022] Giorgio Giannone, Didrik Nielsen, and Ole Winther.  Few-shot diffusion models.  _arXiv preprint arXiv:2205.15463_ , 2022.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Guan et al. [2020] Jiechao Guan, Manli Zhang, and Zhiwu Lu.  Large-scale cross-domain few-shot learning.  In _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * Hardt et al. [2016] Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In _Advances in neural information processing systems_ , pages 3315–3323, 2016.
  * Islam et al. [2022] Maliha Tashfia Islam, Anna Fariha, Alexandra Meliou, and Babak Salimi.  Through the data management lens: Experimental analysis and evaluation of fair classification.  In _Proceedings of the 2022 International Conference on Management of Data_ , pages 232–246, 2022.
  * Johndrow et al. [2019] James E Johndrow, Kristian Lum, et al.  An algorithm for removing sensitive information: application to race-independent recidivism prediction.  _The Annals of Applied Statistics_ , 13(1):189–220, 2019.
  * Kim et al. [2019] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim.  Learning not to learn: Training deep neural networks with biased data.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9012–9020, 2019.
  * Kusner et al. [2017] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  _Advances in neural information processing systems_ , 30, 2017.
  * Li et al. [2018] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.  Domain generalization with adversarial feature learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5400–5409, 2018.
  * Lu et al. [2021] Mandy Lu, Qingyu Zhao, Jiequan Zhang, Kilian M Pohl, Li Fei-Fei, Juan Carlos Niebles, and Ehsan Adeli.  Metadata normalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10917–10927, 2021.
  * Lu et al. [2017] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris.  Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1131–1140, 2017.
  * Madras et al. [2018] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel.  Learning adversarially fair and transferable representations.  _arXiv preprint arXiv:1802.06309_ , 2018.
  * McNamee [2005] Roseanne McNamee.  Regression modelling and other methods to control confounding.  _Occupational and environmental medicine_ , 62(7):500–506, 2005.
  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021.
  * Moyer et al. [2018] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg.  Invariant representations without adversarial training.  In _Advances in Neural Information Processing Systems 31_ , pages 9084–9093. Curran Associates, Inc., 2018.
  * Pirhadi et al. [2024] Alireza Pirhadi, Mohammad Hossein Moslemi, Alexander Cloninger, Mostafa Milani, and Babak Salimi.  Otclean: Data cleaning for conditional independence violations using optimal transport.  _SIGMOD_ , 2024.
  * Preechakul et al. [2022] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10619–10629, 2022.
  * Salimi et al. [2019] Babak Salimi, Luke Rodriguez, Bill Howe, and Dan Suciu.  Interventional fairness: Causal database repair for algorithmic fairness.  In _Proceedings of the 2019 International Conference on Management of Data_ , pages 793–810, 2019.
  * Salimi et al. [2020] Babak Salimi, Bill Howe, and Dan Suciu.  Database repair meets algorithmic fairness.  _ACM SIGMOD Record_ , 2020.
  * Sinha et al. [2021] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon.  D2c: Diffusion-decoding models for few-shot conditional generation.  _Advances in Neural Information Processing Systems_ , 34:12533–12548, 2021.
  * Squires et al. [2020] Chandler Squires, Dennis Shen, Anish Agarwal, Devavrat Shah, and Caroline Uhler.  Causal imputation via synthetic interventions, 2020.
  * Székely et al. [2007] Gábor J Székely, Maria L Rizzo, Nail K Bakirov, et al.  Measuring and testing dependence by correlation of distances.  _The annals of statistics_ , 35(6):2769–2794, 2007.
  * Tan et al. [2020] Zilong Tan, Samuel Yeom, Matt Fredrikson, and Ameet Talwalkar.  Learning fair representations for kernel models.  In _International Conference on Artificial Intelligence and Statistics_ , pages 155–166. PMLR, 2020.
  * Vento et al. [2022] Anthony Vento, Qingyu Zhao, Robert Paul, Kilian M Pohl, and Ehsan Adeli.  A penalty approach for normalizing feature distributions to build confounder-free models.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 387–397. Springer, 2022.
  * Wang et al. [2019] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez.  Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5310–5319, 2019.
  * Woodworth et al. [2017] Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro.  Learning non-discriminatory predictors.  _arXiv preprint arXiv:1702.06081_ , 2017.
  * Zhang et al. [2018] Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell.  Mitigating unwanted biases with adversarial learning.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 335–340, 2018.
  * Zhang et al. [2022] Zhongwei Zhang, Mingyu Shao, Chicheng Ma, Zhe Lv, and Jilei Zhou.  An enhanced domain-adversarial neural networks for intelligent cross-domain fault diagnosis of rotating machinery.  _Nonlinear Dynamics_ , 108(3):2385–2404, 2022.
  * Zhao et al. [2019a] Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon.  Conditional learning of fair representations.  _arXiv preprint arXiv:1910.07162_ , 2019a.
  * Zhao et al. [2019b] Qingyu Zhao, Ehsan Adeli, Adolf Pfefferbaum, Edith V Sullivan, and Kilian M Pohl.  Confounder-aware visualization of convnets.  _arXiv preprint arXiv:1907.12727_ , 2019b.
  * Zhao et al. [2020] Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl.  Training confounder-free deep learning models for medical applications.  _Nature communications_ , 11(1):1–9, 2020.


# Multichannel Audio Source Separation with Independent Deeply Learned Matrix
Analysis Using Product of Source Models

Takuya Hasumi†, Tomohiko Nakamura†, Norihiro Takamune†
Hiroshi Saruwatari†, Daichi Kitamura⋆, Yu Takahashi‡, Kazunobu Kondo‡ †The
University of Tokyo, Tokyo, Japan
⋆National Institute of Technology, Kagawa College, Kagawa, Japan
‡Yamaha Corporation, Shizuoka, Japan

###### Abstract

Independent deeply learned matrix analysis (IDLMA) is one of the state-of-the-
art multichannel audio source separation methods using the source power
estimation based on deep neural networks (DNNs). The DNN-based power
estimation works well for sounds having timbres similar to the DNN training
data. However, the sounds to which IDLMA is applied do not always have such
timbres, and the timbral mismatch causes the performance degradation of IDLMA.
To tackle this problem, we focus on a blind source separation counterpart of
IDLMA, independent low-rank matrix analysis. It uses nonnegative matrix
factorization (NMF) as the source model, which can capture source spectral
components that only appear in the target mixture, using the low-rank
structure of the source spectrogram as a clue. We thus extend the DNN-based
source model to encompass the NMF-based source model on the basis of the
product-of-expert concept, which we call the product of source models (PoSM).
For the proposed PoSM-based IDLMA, we derive a computationally efficient
parameter estimation algorithm based on an optimization principle called the
majorization-minimization algorithm. Experimental evaluations show the
effectiveness of the proposed method.

##  1 Introduction

Multichannel audio source separation is a technique to separate concurrent
sources out of mixture signals observed by a microphone array [1]. In the
overdetermined case, many blind source separation (BSS) methods using the
statistical independence between sources have thus far been proposed for
decades, for example, frequency-domain independent component analysis [2, 3]
and independent vector analysis [4, 5]. One of the state-of-the-art BSS
methods is independent low-rank matrix analysis (ILRMA) [6], which estimates
demixing filters, using the nonnegative matrix factorization (NMF) [7] as a
source model. When a sufficient amount of training data of the sources is
available, we previously showed that the separation performance can be further
improved by replacing the NMF-based source model of ILRMA with the source
model based on a deep neural network (DNN) [8]. We call this DNN-based method
independent deeply learned matrix analysis (IDLMA), which is one of the state-
of-the-art supervised but spatially blind multichannel audio source separation
methods.

IDLMA uses the sourcewise DNN-based source models that are trained in advance
to extract power spectrograms of target sources from noisy mixtures. Thus, its
separation performance strongly depends on the DNN-based power estimation
performance. The DNNs work well for sounds that include sources having timbres
similar to those of the training data. However, owing to the difference in
musical genre and mixing, the sounds to which IDLMA is applied sometimes
differ in timbre from those of the training data, which leads to performance
degradation of IDLMA. For example, in the DSD100 dataset [9], which we will
use in the experiments described in Section 4, most of the sounds labeled as
bass are played on an electric bass guitar. The DNN trained with these sounds
should work well for the electric bass guitar sounds. However, some of the
test data labeled as bass are played by bass instruments other than the
electric bass guitar, e.g., synth bass. This difference leads to the failure
of the DNN-based source power estimation, as we will later show in Section
4.3.

On the other hand, since ILRMA is a fully blind method, it is free from such
performance degradation caused by the timbral discrepancy. However, the
separation performance of ILRMA is often lower than that of IDLMA because its
NMF-based source model assumes the low-rank structure of the source
spectrograms, which does not always hold. For the sources that do not have the
low-rank structure, the DNN-based source model is effective as long as the
timbral discrepancy is less significant. From this viewpoint, the DNN- and
NMF-based source models are complementary.

Motivated by this observation, we extend the DNN-based source model to
encompass the NMF-based source model on the basis of the product-of-expert
concept [10]. Since this extension combines the source models of IDLMA and
ILRMA, we call it the product of source models (PoSM). For the data to which
the proposed method is applied, the DNN part represents the source components
similar to the training data, whereas the NMF part represents those that
appear only in the target mixture. We build an IDLMA extension that instead
uses the proposed PoSM as the source model, and we derive a computationally
efficient parameter estimation algorithm using an optimization principle named
the majorization-minimization (MM) algorithm [11]. Experimental results show
that even for sound having a timbral gap with the DNN training data, the
separation performance is further improved in the proposed method.

##  2 Conventional methods

(a) Separation process of ILRMA |  (b) Separation process of IDLMA
---|---
Figure 1: Separation processes of (a) ILRMA and (b) IDLMA.

###  2.1 Formulation

Let us denote the number of sources and channels by $N$ and $M$, respectively.
The short-time Fourier transform (STFT) of the source, observed, and separated
signals are defined as

$\displaystyle\bm{s}_{ij}=$
$\displaystyle(s_{ij1},\ldots,s_{ijN})^{\mathsf{T}}\in\mathbb{C}^{N},$ (1)
$\displaystyle\bm{x}_{ij}=$
$\displaystyle(x_{ij1},\ldots,x_{ijM})^{\mathsf{T}}\in\mathbb{C}^{M},$ (2)
$\displaystyle\bm{y}_{ij}=$
$\displaystyle(y_{ij1},\ldots,y_{ijN})^{\mathsf{T}}\in\mathbb{C}^{N},$ (3)

where $i=1,\ldots,I$, $j=1,\ldots,J$, $n=1,\ldots,N$, and $m=1,\ldots,M$ are
the frequency, time frame, source, and channel indices, respectively, and T
denotes the transpose operator. We define the matrices
$\bm{X}_{m}\in\mathbb{C}^{I\times J}$ and $\bm{Y}_{n}\in\mathbb{C}^{I\times
J}$ whose $(i,j)$th entries are $x_{ijm}$ and $y_{ijn}$, respectively. When
the mixing system is time-invariant and the window length of the STFT is
sufficiently longer than the reverberation time, the observed signal
$\bm{x}_{ij}$ is represented as

$\displaystyle\bm{x}_{ij}=$ $\displaystyle\bm{A}_{i}\bm{s}_{ij},$ (4)

where $\bm{A}_{i}\in\mathbb{C}^{M\times N}$ is a mixing matrix. If the number
of channels is equal to that of sources (i.e., $M=N$) and the mixing matrix
$\bm{A}_{i}$ is nonsingular, the separated signal $\bm{y}_{ij}$ is represented
as

$\displaystyle\bm{y}_{ij}=$ $\displaystyle\bm{W}_{i}\bm{x}_{ij},$ (5)

where
$\bm{W}_{i}=(\bm{w}_{i1},\ldots,\bm{w}_{iN})^{\mathsf{H}}\in\mathbb{C}^{N\times
M}$ is a demixing matrix and H denotes the Hermite transpose operator.

As in [6] and [8], we assume that $y_{ijn}$ follows an isotropic complex
Gaussian distribution with zero mean and variance $r_{ijn}$:

$\displaystyle p(y_{ijn};r_{ijn})$
$\displaystyle=\mathcal{N}_{\mathbb{C}}(y_{ijn};0,r_{ijn})$
$\displaystyle=\frac{1}{\pi
r_{ijn}}\exp\left(-\frac{|y_{ijn}|^{2}}{r_{ijn}}\right).$ (6)

Owing to the generative model given by (6), the problem of separating the
source signals $y_{ijn}$ out of the given observed signals $x_{ijm}$ can be
formulated as a maximum likelihood estimation problem with respect to
$r_{ijn}$ and $\bm{W}_{i}$. The cost function is given as the negative log-
likelihood of the observed signals [1]:

$\displaystyle\mathcal{L}=$ $\displaystyle-\log p(\mathcal{X})$
$\displaystyle=$ $\displaystyle-\log
p(\mathcal{Y})-2J\sum_{i}\log|\det\bm{W}_{i}|$
$\displaystyle\mathop{=}\limits^{c}$ $\displaystyle\sum_{i,j,n}\left(\log
r_{ijn}+\frac{|\bm{w}_{in}^{\mathsf{H}}\bm{x}_{ij}|^{2}}{r_{ijn}}\right)-2J\sum_{i}\log|\det\bm{W}_{i}|,$
(7)

where $\mathcal{X}=\\{\bm{X}_{1},\ldots,\bm{X}_{M}\\}$ and
$\mathcal{Y}=\\{\bm{Y}_{1},\ldots,\bm{Y}_{N}\\}$ denote the sets of observed
and separated signals, respectively. Here, $\mathop{=}\limits^{c}$ denotes
equality up to constants. Let us denote a matrix whose $(i,j)$th entry is
$r_{ijn}$ by $\bm{R}_{n}\in\mathbb{R}_{\geq 0}^{I\times J}$. ILRMA and IDLMA
are derived on the basis of the above formulation, and their difference lies
in the representation of $\bm{R}_{n}$. To distinguish the NMF- and DNN-based
source models, we hereafter add superscripts (NMF) and (DNN) to $\bm{R}_{n}$,
respectively.

###  2.2 ILRMA [6]

####  2.2.1 Representation of $\bm{R}_{n}^{(\mathrm{NMF})}$

ILRMA uses NMF as the source model. The NMF represents
$\bm{R}_{n}^{(\mathrm{NMF})}$ as a product of two nonnegative matrices, one of
which is a basis matrix $\bm{T}_{n}\in\mathbb{R}_{\geq 0}^{I\times K}$
consisting of $K$ spectral templates and the other of which is an activation
matrix $\bm{V}_{n}\in\mathbb{R}_{\geq 0}^{K\times J}$ representing the
temporal energies of the corresponding spectral templates.

$\displaystyle r_{ijn}^{(\mathrm{NMF})}=$
$\displaystyle\sum_{k}t_{ik,n}v_{kj,n}.$ (8)

Here, $t_{ik,n}$ and $v_{kj,n}$ are the $(i,j)$th entries of $\bm{T}_{n}$ and
$\bm{V}_{n}$, respectively. By substituting (8) into (7), we can write the
cost function of ILRMA $\mathcal{L}_{\mathrm{ILRMA}}$ as

$\displaystyle\mathcal{L}_{\mathrm{ILRMA}}\mathop{=}\limits^{c}$
$\displaystyle\sum_{i,j,n}\left(\log\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}+\frac{|\bm{w}_{in}^{\mathsf{H}}\bm{x}_{ij}|^{2}}{\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}}\right)$
$\displaystyle-2J\sum_{i}\log|\det\bm{W}_{i}|.$ (9)

####  2.2.2 Parameter Estimation Algorithm

The parameter estimation algorithm of ILRMA iteratively updates $t_{ik,n}$,
$v_{kj,n}$, and $\bm{W}_{i}$ [6]. The first two terms of the cost function (9)
are the same form as in the cost function of NMF with the Itakura–Saito
divergence criterion [12] up to constants. For updating $t_{ik,n}$ and
$v_{kj,n}$, we can use the convergence-guaranteed iterative algorithm derived
in [12], which is based on the MM algorithm [11].

The MM algorithm consists of two steps. For a to-be-minimized cost function of
$\bm{\theta}$, $f(\bm{\theta})$, by introducing an auxiliary variable
$\bar{\bm{\theta}}$, we construct its upper bound
$f^{+}(\bm{\theta},\bar{\bm{\theta}})$ to satisfy the condition that there
exists $\bar{\bm{\theta}}$ such that $f^{+}(\bm{\theta},\bar{\bm{\theta}})$ is
tangent to $f(\bm{\theta})$ for any $\bm{\theta}$ as follows:

$\displaystyle\min_{\bar{\bm{\theta}}}f^{+}(\bm{\theta},\bar{\bm{\theta}})$
$\displaystyle=f(\bm{\theta}).$ (10)

If $f^{+}$ can be minimized with $\bm{\theta}$ and $\bar{\bm{\theta}}$ in
closed form, we iteratively update $\bm{\theta}$ and $\bar{\bm{\theta}}$:

$\displaystyle\bar{\bm{\theta}}$ $\displaystyle\leftarrow\mathop{\rm
arg~{}min}\limits_{\bar{\bm{\theta}}}f^{+}(\bm{\theta},\bar{\bm{\theta}}),$
(11) $\displaystyle\bm{\theta}$ $\displaystyle\leftarrow\mathop{\rm
arg~{}min}\limits_{\bm{\theta}}f^{+}(\bm{\theta},\bar{\bm{\theta}}).$ (12)

Since $f^{+}$ always satisfies (10), $f$ is guaranteed to be nonincreasing at
each iteration.

The application of the MM algorithm to the minimization of (9) with respect to
$t_{ik,n}$ and $v_{kj,n}$ yields the following update rules [6]:

$\displaystyle t_{ik,n}$ $\displaystyle\leftarrow
t_{ik,n}\left[{\frac{\displaystyle\sum_{j}\frac{v_{kj,n}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}|y_{ij}|^{2}}{\displaystyle\sum_{j}\frac{v_{kj,n}}{\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}}}}\right]^{\frac{1}{2}},$
(13) $\displaystyle v_{kj,n}$ $\displaystyle\leftarrow
v_{kj,n}\left[{\frac{\displaystyle\sum_{i}\frac{t_{ik,n}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}|y_{ij}|^{2}}{\displaystyle\sum_{i}\frac{t_{ik,n}}{\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}}}}\right]^{\frac{1}{2}}.$
(14)

By using $t_{ik,n}$ and $v_{kj,n}$ obtained from (13) and (14), we update
$r_{ijn}^{(\mathrm{NMF})}$ in accordance with (8).

Since (9) consists only of the quadratic and log-determinant terms in
$\bm{w}_{in}$, the iterative projection (IP) algorithm [13] can be applied,
which guarantees the nonincrease in the cost function. This method updates
$\bm{w}_{in}$ sequentially with respect to $n$:

$\displaystyle\bm{w}_{in}$
$\displaystyle\leftarrow(\bm{W}_{i}\bm{U}_{in})^{-1}\bm{e}_{n},$ (15)
$\displaystyle\bm{w}_{in}$
$\displaystyle\leftarrow\frac{\bm{w}_{in}}{\sqrt{\bm{w}_{in}^{\mathsf{H}}\bm{U}_{in}\bm{w}_{in}}},$
(16) $\displaystyle\bm{U}_{in}$
$\displaystyle:=\frac{1}{J}\sum_{j}\frac{1}{r_{ijn}^{(\mathrm{NMF})}}\bm{x}_{ij}\bm{x}_{ij}^{\mathsf{H}},$
(17)

where $\bm{e}_{n}\in\mathbb{R}^{N}$ is a unit vector whose $n$th element is
one.

To compensate for the scale uncertainty between $\bm{w}_{in}$ and
$r_{ijn}^{(\mathrm{NMF})}$, the projection back (PB) technique [14] is applied
to $\bm{y}_{ij}$. This technique determines the scale so that the sum of the
separated signals matches the observation of the reference microphone whose
index is denoted by $m_{\mathrm{ref}}$:

$\displaystyle
x_{ijm_{\mathrm{ref}}}=\sum_{n}y_{ijn}=\sum_{n}\bm{w}_{in}^{\mathsf{H}}\bm{x}_{ij}.$
(18)

The PB technique scales $\bm{y}_{ij}$ as

$\displaystyle\bm{y}_{ij}\leftarrow\mathrm{diag}(\bm{d}_{i})\bm{y}_{ij},$ (19)

where $\mathrm{diag}(\bm{d}_{i})\in\mathbb{C}^{N\times N}$ has the elements of
$\bm{d}_{i}\in\mathbb{C}^{N}$ on the main diagonal and $0$ elsewhere, and
$\bm{d}_{i}$ is computed by

$\displaystyle\bm{d}_{i}=(\bm{W}_{i}^{\mathsf{T}})^{-1}\bm{e}_{m_{\mathrm{ref}}}.$
(20)

The outline of the ILRMA separation process is shown in Fig. 1(a), where
$|\cdot|^{\cdot q}$ returns the absolute value of each entry of a matrix to
the $q$th power.

###  2.3 IDLMA [8]

The cost function of IDLMA is given as (7) by replacing $r_{ijn}$ with
$r_{ijn}^{(\mathrm{DNN})}$. In contrast to ILRMA,
$\bm{R}^{(\mathrm{DNN})}_{n}$ is updated using the pretrained DNNs. The DNN
for $n$th source, $\mathrm{DNN}_{n}$, takes the magnitude spectrogram of the
current separated signal $\bm{Y}_{n}$ and outputs the estimate of the source
standard deviation $\bm{\Sigma}_{n}\in\mathbb{R}_{\geq 0}^{I\times J}$:

$\displaystyle\bm{\Sigma}_{n}=\mathrm{DNN}_{n}(|\bm{Y}_{n}|^{\cdot 1}).$ (21)

$\bm{R}_{n}^{(\mathrm{DNN})}$ is updated as

$\displaystyle
r_{ijn}^{(\mathrm{DNN})}\leftarrow\max(\sigma_{ijn}^{2},~{}\varepsilon),$ (22)

where $\sigma_{ijn}$ denotes the $(i,j)$th entry of $\bm{\Sigma}_{n}$, and
$\varepsilon$ is a small value to avoid numerical instability. The demixing
matrix $\bm{W}_{i}$ can be updated with the IP algorithm, as in ILRMA. To
reduce the linear distortion, the PB technique is also applied to
$\bm{y}_{ij}$. The outline of the IDLMA separation process is shown in Fig.
1(b).

The DNNs are trained in advance to extract the target source spectrogram
$\tilde{\bm{S}}_{n}\in\mathbb{C}^{I\times J}$ from the single-channel
instantaneous noisy mixture $\tilde{\bm{Y}}_{n}\in\mathbb{C}^{I\times J}$. The
loss function of training $\mathrm{DNN}_{n}$ is defined as

$\displaystyle\mathcal{L}_{\mathrm{IDLMA}}^{(\mathrm{DNN}_{n})}$
$\displaystyle=\sum_{i,j}\left(\frac{|\tilde{s}_{ijn}|^{2}+\delta}{\hat{\sigma}_{ijn}^{2}+\delta}-\log\frac{|\tilde{s}_{ijn}|^{2}+\delta}{\hat{\sigma}_{ijn}^{2}+\delta}-1\right),$
(23)

where $\hat{\sigma}_{ijn}\in\mathbb{R}_{\geq 0}$ and
$\tilde{s}_{ijn}\in\mathbb{C}$ denote the $(i,j)$th entry of
$\mathrm{DNN}_{n}(|\tilde{\bm{Y}}_{n}|^{\cdot 1})$ and $\tilde{\bm{S}}_{n}$,
respectively, and $\delta$ is a small value to prevent division by zero. Since
(23) is given by replacing $|\bm{w}_{in}^{\mathsf{H}}\bm{x}_{ij}|$ with
$|\tilde{s}_{ijn}|$ in (7) up to constants, the minimization of (23) with
respect to $\hat{\sigma}_{ijn}$ can be seen as a simulation of the maximum
likelihood estimation with respect to $r_{ijn}^{(\mathrm{DNN})}$ based on the
IDLMA cost function.

##  3 Proposed method

###  3.1 Motivation

Since IDLMA uses trained DNNs, its separation performance is affected by the
gap in timbre between the DNN training data and the sounds to which IDLMA is
applied. Such a timbral gap frequently appears in music audio signals owing to
differences in mixing, musical styles, and genres. For example, as a bass
instrument, the electric bass guitar is typically used in rock and pop music,
whereas the synth bass is frequently used in electronic music. Although the
sounds played by the electric and synth basses are both labeled as bass in the
DSD100 dataset [9], their spectral characteristics differ significantly, which
can lead to the performance degradation of IDLMA, as we will show later in
Section 4. Although one method to address this problem would be to collect
various instrument sounds so that the trained DNN can deal with any possible
timbral variations, this can be costly and impractical.

Figure 2: Separation process of proposed method.

To overcome this problem, we take an approach that combines the supervised and
unsupervised source models. The former represents the source components that
can be learned from the training data, and the latter represents those unique
in the data to which the method is applied. As the supervised and unsupervised
source models, we can use the DNN- and NMF-based source models of IDLMA and
ILRMA, respectively. By combining them in an appropriate manner, we extend
IDLMA to work robustly against the discrepancy in timbre with the training
data.

###  3.2 Formulation of Product of Source Models

Since both the DNN- and NMF-based source models are given as the generative
models of the observed signals, we can combine them, following the product-of-
expert concept [10]. This concept represents a probability distribution as a
product of multiple probability distributions called experts. Each expert
corresponds to the desired constraint, and the resulting probability
distribution becomes high at the events where all the constraints tend to be
satisfied.

Following the product of experts, we can define a novel source model, which we
call the PoSM, on the basis of $r_{ijn}^{(\mathrm{NMF})}$ and
$r_{ijn}^{(\mathrm{DNN})}$ as

$\displaystyle p(y_{ijn};r_{ijn}^{(\mathrm{NMF})},r_{ijn}^{(\mathrm{DNN})})$
$\displaystyle\propto\left[\mathcal{N}_{\mathbb{C}}(y_{ijn};0,r_{ijn}^{(\mathrm{NMF})})\right]^{\alpha}\left[\mathcal{N}_{\mathbb{C}}(y_{ijn};0,r_{ijn}^{(\mathrm{DNN})})\right]^{\beta},$
(24)

where $\alpha>0$ and $\beta>0$ are the weights for the NMF- and DNN-based
probability distributions. Since a product of two complex Gaussian
distributions with zero means is also a complex Gaussian distribution with
zero mean, the PoSM can be explicitly described as

$\displaystyle p(y_{ijn};r_{ijn}^{(\mathrm{NMF})},r_{ijn}^{(\mathrm{DNN})})$
$\displaystyle=\mathcal{N}_{\mathbb{C}}(y_{ijn};0,\tilde{r}_{ijn})$
$\displaystyle=\frac{1}{\pi\tilde{r}_{ijn}}\exp\left(-\frac{|y_{ijn}|^{2}}{\tilde{r}_{ijn}}\right),$
(25)

where

$\displaystyle\frac{1}{\tilde{r}_{ijn}}=\frac{\alpha}{r_{ijn}^{(\mathrm{NMF})}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}.$
(26)

Interestingly, when we set $\alpha+\beta=1$, $\tilde{r}_{ijn}$ is a weighted
harmonic mean of $r_{ijn}^{(\mathrm{NMF})}$ and $r_{ijn}^{(\mathrm{DNN})}$,
which means $\alpha$ and $\beta$ balance the importance of the source
estimates obtained with the DNN and NMF.

We define a matrix $\tilde{\bm{R}}_{n}\in\mathbb{R}_{\geq 0}^{I\times J}$
whose $(i,j)$th entry is $\tilde{r}_{ijn}$. As in Section 2.1, the separation
problem based on the PoSM can be formulated as a maximum likelihood estimation
problem with respect to $t_{ik,n}$, $v_{kj,n}$, $r_{ijn}^{(\mathrm{DNN})}$,
and $\bm{W}_{i}$, and the cost function of the proposed method is given as

$\displaystyle\mathcal{L}_{\mathrm{prop}}\mathop{=}\limits^{c}$
$\displaystyle-\sum_{i,j,n}\log\left(\frac{\alpha}{\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right)$
$\displaystyle+\sum_{i,j,n}\left(\frac{\alpha}{\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right)|\bm{w}_{in}^{\mathsf{H}}\bm{x}_{ij}|^{2}$
$\displaystyle-2J\sum_{i}\log|\det\bm{W}_{i}|.$ (27)

If $\alpha=1$ and $\beta=0$, the cost function of the proposed method (27)
reduces to that of ILRMA given by (9). If $\alpha=0$ and $\beta=1$, the cost
function of the proposed method (27) reduces to that of IDLMA.

###  3.3 DNN Training and Update Rules

####  3.3.1 Outline of Separation Process

Fig. 2 shows the outline of the separation process of the proposed PoSM-based
IDLMA. As in ILRMA and IDLMA, the separation is achieved by iteratively
updating the parameters of the PoSM and the demixing matrix. For the PoSM
parameters, $r_{ijn}^{(\mathrm{DNN})}$ is updated in the same manner as IDLMA,
and $t_{ik,n}$ and $v_{kj,n}$ are updated by a convergence-guaranteed
algorithm, as we will show in Section 3.3.3. The variance of the PoSM
$\tilde{r}_{ijn}$ is then updated using the current estimates of $t_{ik,n}$,
$v_{kj,n}$, and $r_{ijn}^{(\mathrm{DNN})}$. The demixing matrix is updated by
the IP algorithm followed by the PB technique.

####  3.3.2 DNN Training and Update Rule of $\bm{R}_{n}^{(\mathrm{DNN})}$

Since $r_{ijn}^{(\mathrm{NMF})}$ represents the source components independent
of the training data, we can set $\alpha=0$ and $\beta=1$ during the DNN
training, which reduces the cost function (27) to that of IDLMA, as described
in Section 3.2. This justifies training the DNNs in the same manner as in
IDLMA, using the cost function (23). In the separation process, we update
$\bm{\Sigma}_{n}$ and $\bm{R}_{n}^{(\mathrm{DNN})}$ in accordance with (21)
and (22), respectively.

####  3.3.3 Update Rules of $\bm{R}_{n}^{(\mathrm{NMF})}$

The first and second terms of (27) respectively include the sums over $k$ in
the logarithmic function and the denominator of the fractional function. These
terms make it difficult to analytically solve the minimization of (27) with
respect to $t_{ik,n}$ and $v_{kj,n}$. However, we can instead derive a
computationally efficient algorithm that iteratively updates $t_{ik,n}$ and
$v_{kj,n}$, using the MM algorithm [11].

Focusing on the first term of (27), when $\alpha$, $\beta$, $r$, $z>0$, its
second-order derivate with respect to $z$ is always negative:

$\displaystyle\frac{\partial^{2}}{\partial
z^{2}}\left[-\log\left(\frac{\alpha}{z}+\frac{\beta}{r}\right)\right]=-\frac{\alpha
r(\alpha r+2\beta z)}{z^{2}(\alpha r+\beta z)^{2}}<0.$ (28)

Since $-\log\left(\alpha/z+\beta/r\right)$ is concave, by applying the tangent
inequality to the first term of (27), we obtain its upper bound:

$\displaystyle-\log\left(\dfrac{\alpha}{\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}}+\dfrac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right)$
$\displaystyle\leq\dfrac{\alpha r_{ijn}^{(\mathrm{DNN})}}{\alpha
r_{ijn}^{(\mathrm{DNN})}+\beta
c_{ijn}}\dfrac{1}{c_{ijn}}\left(\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}-c_{ijn}\right)$
$\displaystyle\hskip
12.80373pt-\log\left(\frac{\alpha}{c_{ijn}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right),$
(29)

where $c_{ijn}>0$ is an auxiliary variable. The equality of (29) holds if and
only if

$\displaystyle c_{ijn}=\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}.$ (30)

Since $1/z$ is convex for $z>0$, we can apply Jensen’s inequality to the
second term of (27):

$\displaystyle\frac{1}{\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}}\leq\sum_{k}\frac{\lambda_{ijk,n}^{2}}{t_{ik,n}v_{kj,n}},$
(31)

where $\lambda_{ijk,n}$ is an auxiliary variable that satisfies
$\lambda_{ijk,n}\geq 0$ and $\sum_{k}\lambda_{ijk,n}=1$ for all $i,j$, and
$n$. The equality of (31) holds if and only if

$\displaystyle\lambda_{ijk,n}=\frac{t_{ik,n}v_{kj,n}}{\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}}.$
(32)

In summary, the auxiliary function of $\mathcal{L}_{\mathrm{prop}}$ is given
as

$\displaystyle\mathcal{L}^{+}_{\mathrm{prop}}\mathop{=}\limits^{c}$
$\displaystyle\sum_{i,j,n}\frac{\alpha r_{ijn}^{(\mathrm{DNN})}}{\alpha
r_{ijn}^{(\mathrm{DNN})}+\beta
c_{ijn}}\frac{1}{c_{ijn}}\sum_{{k}}t_{{i}{k},{n}}v_{{k}{j},{n}}$
$\displaystyle+\sum_{i,j,n}\left(\alpha\sum_{k}\frac{\lambda_{ijk,n}^{2}}{t_{ik,n}v_{kj,n}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right)|y_{ijn}|^{2}$
$\displaystyle-\sum_{i,j,n}\left[\frac{\alpha r_{ijn}^{(\mathrm{DNN})}}{\alpha
r_{ijn}^{(\mathrm{DNN})}+\beta
c_{ijn}}+\log\left(\frac{\alpha}{c_{ijn}}+\frac{\beta}{r_{ijn}^{(\mathrm{DNN})}}\right)\right].$

Setting the partial derivatives of $\mathcal{L}^{+}_{\mathrm{prop}}$ with
respect to $t_{ik,n}$ and $v_{kj,n}$ equal to zero yields

$\displaystyle
t_{ik,n}=\left[{\frac{\displaystyle\sum_{j}\frac{\lambda_{ijk,n}^{2}}{v_{kj,n}}|y_{ijn}|^{2}}{\displaystyle\sum_{j}\frac{r_{ijn}^{(\mathrm{DNN})}}{\alpha
r_{ijn}^{(\mathrm{DNN})}+\beta
c_{ijn}}\frac{v_{kj,n}}{c_{ijn}}}}\right]^{\frac{1}{2}},$ (34) $\displaystyle
v_{kj,n}=\left[{\frac{\displaystyle\sum_{i}\frac{\lambda_{ijk,n}^{2}}{t_{ik,n}}|y_{ijn}|^{2}}{\displaystyle\sum_{i}\frac{r_{ijn}^{(\mathrm{DNN})}}{\alpha
r_{ijn}^{(\mathrm{DNN})}+\beta
c_{ijn}}\frac{t_{ik,n}}{c_{ijn}}}}\right]^{\frac{1}{2}}.$ (35)

By substituting the equality conditions (30) and (32) into (34) and (35), we
obtain the following update rules:

$\displaystyle t_{ik,n}$ $\displaystyle\leftarrow
t_{ik,n}\left[{\frac{\displaystyle\sum_{j}\frac{v_{kj,n}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}|y_{ijn}|^{2}}{\displaystyle\sum_{j}\frac{v_{kj}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}\tilde{r}_{ijn}}}\right]^{\frac{1}{2}},$
(36) $\displaystyle v_{kj,n}$ $\displaystyle\leftarrow
v_{kj,n}\left[{\frac{\displaystyle\sum_{i}\frac{t_{ik,n}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}|y_{ijn}|^{2}}{\displaystyle\sum_{i}\frac{t_{ik,n}}{\left(\sum_{{k^{\prime}}}t_{{i}{k^{\prime}},{n}}v_{{k^{\prime}}{j},{n}}\right)^{2}}\tilde{r}_{ijn}}}\right]^{\frac{1}{2}}.$
(37)

####  3.3.4 Update Rule of $\bm{W}_{i}$

Since the cost function (27) is an IP-applicable form, we can reuse the update
rule (17) merely by replacing $r_{ijn}^{(\mathrm{NMF})}$ with
$\tilde{r}_{ijn}$. The PB technique is used as in ILRMA and IDLMA.

####  3.3.5 Summary of Update Rules

The entire separation process of the proposed method is shown in Algorithm 1,
where $L$ denotes the number of DNN-based source model updates and
$L^{\prime}$ denotes that of NMF-based source model and demixing matrix
updates per DNN-based source model update.

Algorithm 1 Iterative algorithm of proposed method

0: $\bm{X}_{1},\ldots,\bm{X}_{M}$, $\mathrm{DNN}_{1},\ldots,\mathrm{DNN}_{N}$

0: $\bm{Y}_{1},\ldots,\bm{Y}_{N}$

1: for $l=1,\ldots,L$ do

2: for all source index $n$ do

3: Update DNN source model $\bm{R}_{n}^{(\mathrm{DNN})}$ by (21) and (22)

4: Update source model $\tilde{\bm{R}}_{n}$ by (26)

5: end for

6: for $l^{\prime}=1,\ldots,L^{\prime}$ do

7: for all source index $n$ do

8: Update NMF source model $\bm{R}_{n}^{(\mathrm{NMF})}$ by (36) and (37)

9: Update source model $\tilde{\bm{R}}_{n}$ by (26)

10: end for

11: for all frequency bin $i$ and source index $n$ do

12: Update $\bm{w}_{in}$ by (15) and (16) with (17)

13: end for

14: for all frequency bin $i$ and time frame $j$ do

15: Update $\bm{y}_{ij}$ by (5)

16: end for

17: for all frequency bin $i$ and time frame $j$ do

18: Apply PB technique to $\bm{y}_{ij}$ by (19) with (20)

19: end for

20: end for

21: end for

##  4 Experimental Evaluation

Figure 3: Recording conditions for impulse response.

(a) Vo./Ba.

(b) Ba./Dr.

(c) Vo./Dr.

Figure 4: SDR improvements of proposed method with varying $\alpha$ and of
conventional methods.

(a) Girls Under Glass - We Feel Alright.

(b) James May - Don’t Let Go.

Figure 5: Examples of SDR improvements of IDLMA and of proposed method for
Ba./Dr.

###  4.1 Experimental Setting

To evaluate the effectiveness of the proposed method, we conducted a
multichannel music source separation experiment using the DSD100 dataset [9].
We downsampled the audio signals to $8$ kHz and used the Hamming window of
$512$ ms with a shift length of $256$ ms for STFT.

As the test data, we created multichannel mixtures by convolving dry sources
and the E2A impulse response (reverberation time is $300$ ms) in the RWCP
database [15]. The dry sources were the $30$\- to $60$-s segments of the top
$25$ songs in the test set in alphabetical order. We prepared three pairs of
instruments: vocal and bass (Vo./Ba.), bass and drums (Ba./Dr.), and vocal and
drums (Vo./Dr.). Fig. 3 shows the two recording conditions, and the number of
the multichannel mixtures for each instrument pair was $50$.

We compared the proposed method with ILRMA [6] and IDLMA [8]. In accordance
with the experimental conditions used in [6] and [8], we set $L=10$ and
$L^{\prime}=10$ for both IDLMA and the proposed method. Since ILRMA does not
use the DNNs, its NMF parameters and demixing matrix were updated for $100$
iterations, i.e., we set the number of NMF parameter updates to be the same as
in the proposed method. In accordance with [8], the number of bases for NMF
was $K=20$ for both ILRMA and the proposed method. The source-specific DNN
consists of five fully connected (FC) blocks, each of which consists of an FC
layer with $2048$ hidden units and a rectified linear unit (ReLU)
nonlinearity. A dropout layer with a drop rate of $0.3$ was placed after each
ReLU in the FC blocks except for the last one. As described in Section 3.3.2,
the proposed method can use the DNNs trained in the same manner as in IDLMA,
and we used the same trained DNNs for IDLMA and the proposed method. In the
proposed method, we varied $\alpha$ from $5\times 10^{-1}$ to $1\times
10^{-5}$ with the constraint of $\alpha+\beta=1$.

###  4.2 DNN Training

We used all $50$ songs in the dev set of the DSD100 dataset for DNN training,
and the bottom $25$ songs in alphabetical order in the test set for the
validation. During the DNN training, as described in Section 2.3, single-
channel noisy mixtures were created. The mixtures were created as described in
[16]. The DNNs were trained for $2000$ epochs on an Adadelta [17] optimizer
with a batch size of $128$. The learning rate was set to $1.0$ with a weight
of $l^{2}$ regularization of $10^{-5}$. We clipped the norm of the gradients
of the DNN parameters before the parameter updates so that their $l^{2}$ norms
were less than or equal to $10$. The other hyperparameters were set as
$\delta=10^{-5}$ and $\varepsilon=10^{-1}$, following [8].

###  4.3 Results

Fig. 4 shows the average source-to-distortion ratio (SDR) improvements over
$50$ mixtures of test data for each instrument pair, which were computed with
the BSSEval toolbox [18]. Compared with ILRMA, IDLMA provided SDR improvements
of more than $3$ dB higher in Vo./Ba. and Vo./Dr. separations. However, for
Ba./Dr., IDLMA provides an SDR improvement of only $0.4$ dB higher than ILRMA,
which is lower than for the other instrument pairs. This may be because the
spectrograms of bass and drums are likely to be of low-rank, which fits the
assumption of NMF. On the other hand, the proposed method outperformed IDLMA
with the appropriate choice of $\alpha$ for any instrument pair, showing the
effectiveness of the proposed method.

Fig. 5 shows the average SDR improvement at each iteration for IDLMA and the
proposed method with $\alpha=5\times 10^{-1}$. The results were for the
Ba./Dr. mixture and averaged over the two instruments. In IDLMA, the SDR
improvements at the DNN updates increased until the $40$th iteration, but they
decreased at subsequent iterations, as shown in Fig. 5(a). Although most of
the DNN training data labeled as bass were performed by the electric bass
guitar, the bass sound of this musical piece was performed by the synth bass.
This result shows the performance degradation caused by the timbral
discrepancy between the DNN training data and the data to which IDLMA was
applied. On the other hand, the SDR improvements of the proposed method did
not decrease even after approximately the $40$th iteration. This phenomenon
can also be observed in Fig. 5(b). The drums sound of this musical piece was
performed by the conga, which was not included in the training data. Although
the average SDR improvements of IDLMA seemed to reach the upper performance
limit, the proposed method gave higher average SDR improvements in the later
iterations. These results show that the introduction of the source model
independent of the DNN training data reduces the performance degradation
caused by the timbral gap.

##  5 Conclusion

We proposed the PoSM by combining the DNN- and NMF-based source models, which
are respectively used in IDLMA and ILRMA. The DNN-based part represents the
components similar to the training data, and the NMF-based part represents the
components independent of the training data. We introduced the PoSM into IDLMA
to develop the PoSM-based IDLMA and derived the computationally efficient
separation algorithm of its parameters. For the NMF parameters, we derived the
convergence-guaranteed iterative algorithm based on the MM algorithm. Through
the multichannel music source separation experiments, we showed the
effectiveness of the proposed PoSM-based IDLMA. Furthermore, we showed that
the use of the PoSM reduces the performance degradation caused by the timbral
gap between the DNN training data and the sounds to which the proposed method
is applied.

## Acknowledgment

This work was supported by JSPS-CAS Joint Research Program, Grant number
JPJSBP120197203, and JSPS KAKENHI Grant numbers 19K20306, 19H01116, and
17H06101.

## References

  * [1] H. Sawada, N. Ono, H. Kameoka, D. Kitamura, and H. Saruwatari, “A review of blind source separation methods: two converging routes to ILRMA originating from ICA and NMF,” _APSIPA Transactions on Signal and Information Processing_ , vol. 8, no. e12, 14 pages, 2019.
  * [2] S. Ikeda and N. Murata, “A method of ICA in time-frequency domain,” in _Proceedings of International Conference on Independent Component Analysis and Signal Separation_ , 1999, pp. 365–371.
  * [3] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano, “Blind source separation based on a fast-convergence algorithm combining ICA and beamforming,” _IEEE Transactions on Audio, speech, and language processing_ , vol. 14, no. 2, pp. 666–678, 2006.
  * [4] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation exploiting higher-order frequency dependencies,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 1, pp. 70–79, 2006.
  * [5] A. Hiroe, “Solution of permutation problem in frequency domain ICA, using multivariate probability density functions,” in _Proceedings of International Conference on Independent Component Analysis and Signal Separation_ , 2006, pp. 601–608.
  * [6] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 9, pp. 1626–1641, 2016.
  * [7] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” _Nature_ , vol. 401, no. 6755, pp. 788–791, 1999\.
  * [8] N. Makishima, S. Mogami, N. Takamune, D. Kitamura, H. Sumino, S. Takamichi, H. Saruwatari, and N. Ono, “Independent deeply learned matrix analysis for determined audio source separation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 10, pp. 1601–1615, 2019.
  * [9] A. Liutkus, F.-R. Stöter, Z. Rafii, D. Kitamura, B. Rivet, N. Ito, N. Ono, and J. Fontecave, “The 2016 signal separation evaluation campaign,” in _Proceedings of International Conference on Latent Variable Analysis and Signal Separation_ , 2017, pp. 323–332.
  * [10] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural Computation_ , vol. 14, no. 8, pp. 1771–1800, 2002\.
  * [11] D. R. Hunter and K. Lange, “Quantile regression via an MM algorithm,” _Journal of Computational and Graphical Statistics_ , vol. 9, no. 1, pp. 60–77, 2000.
  * [12] C. Févotte, N. Bertin, and J.-L. Durrieu, “Nonnegative matrix factorization with the Itakura–Saito divergence: With application to music analysis,” _Neural Computation_ , vol. 21, no. 3, pp. 793–830, 2009\.
  * [13] N. Ono, “Stable and fast update rules for independent vector analysis based on auxiliary function technique,” in _Proceedings of 2011 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_ , 2011, pp. 189–192.
  * [14] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation based on temporal structure of speech signals,” _Neurocomputing_ , vol. 41, no. 1-4, pp. 1–24, 2001.
  * [15] S. Nakamura, K. Hiyane, F. Asano, T. Nishiura, and T. Yamada, “Acoustical sound database in real environments for sound scene understanding and hands-free speech recognition,” in _Proceedings of the Second International Conference on Language Resources and Evaluation_ , 2000, pp. 965–968.
  * [16] T. Hasumi, T. Nakamura, N. Takamune, H. Saruwatari, D. Kitamura, Y. Takahashi, and K. Kondo, “Empirical Bayesian independent deeply learned matrix analysis for multichannel audio source separation,” in _Proceedings of European Signal Processing Conference (arXiv preprint arXiv:2106.03492)_ , 2021\.
  * [17] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” _arXiv preprint arXiv:1212.5701_ , 2012.
  * [18] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in blind audio source separation,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 4, pp. 1462–1469, 2006.