# Toward Deep Drum Source Separation

Alessandro Ilic Mezza,  Riccardo Giampiccolo,  Alberto Bernardini,  Augusto
Sarti
Dipartimento di Elettronica, Informazione e Bioingegneria
Politecnico di Milano
Milan, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In the past, the field of drum source separation faced significant challenges
due to limited data availability, hindering the adoption of cutting-edge deep
learning methods that have found success in other related audio applications.
In this manuscript, we introduce StemGMD, a large-scale audio dataset of
isolated single-instrument drum stems. Each audio clip is synthesized from
MIDI recordings of expressive drums performances using ten real-sounding
acoustic drum kits. Totaling 1224 hours, StemGMD is the largest audio dataset
of drums to date and the first to comprise isolated audio clips for every
instrument in a canonical nine-piece drum kit. We leverage StemGMD to develop
LarsNet, a novel deep drum source separation model. Through a bank of
dedicated U-Nets, LarsNet can separate five stems from a stereo drum mixture
faster than real-time and is shown to significantly outperform state-of-the-
art nonnegative spectro-temporal factorization methods.

_Keywords_ Deep Learning $\cdot$ Drums $\cdot$ Music Decomposition $\cdot$
Source Separation $\cdot$ U-Net

##  1 Introduction

††A revised and updated version of the present manuscript has been submitted
to Pattern Recognition Letters on Sept. 26, 2023.

Deep learning has been a breakthrough in the field of acoustic source
separation, with applications ranging from speech (Wang and Chen, 2018) to a
variety of environmental sounds (Kavalerov et al., 2019). In recent years,
neural-network-aided music demixing models (MDX) have shown tremendous
results. Open-Unmix (Stöter et al., 2019), Spleeter (Hennequin et al., 2020),
Meta-TasNet (Samuel et al., 2020), CrossNet-UMX (Sawata et al., 2021), and
Demucs (Défossez, 2021), to name a few, have indeed proved very capable at
extracting various stems from a mixed music track. MDX models are typically
trained to isolate four stems (Stöter et al., 2018; Mitsufuji et al., 2022;
Fabbro et al., 2023), i.e., “vocals”, “bass”, “drums”, and “other”, while
sometimes including also other instruments such as guitar and piano. In
practice, however, the drum kit is conventionally treated as a single
instrument despite being an ensemble of various percussive instruments in
itself. Unfortunately, this prevents the full inversion of the mixing process
performed by the sound engineer when producing a song. High-quality Drum
Source Separation (DSS) could have a significant impact on many creative
applications, as it may enhance remixing, remastering, and audio production by
providing precise control over individual drum element, allowing artists and
sound engineers to fine-tune their tracks beyond the capabilities of existing
MDX software. Additionally, it can be valuable in music analysis, facilitating
detailed studies of drum patterns and rhythms, as well as in music education
to isolate drum parts for instructional purposes. Furthermore, DSS may be
instrumental in advancing Automatic Drum Transcription (ADT) (Wu et al.,
2018).

Deep learning methods are data-hungry. However, as shown in Table 1, available
datasets of isolated drum stems are few and of limited size. This substantial
lack of data is possibly one of the main reasons why, so far, deep DSS has not
been investigated in the literature, unlike other drum-related tasks such as,
e.g., ADT for which deep learning approaches have been studied extensively
(Vogl et al., 2017; Choi and Cho, 2019; Callender et al., 2020; Ishizuka et
al., 2021). By contrast, the latest advances in DSS to date are based on
variants of nonnegative matrix factorization (NMF) and nonnegative matrix
factor deconvolution (NMFD), as they prove effective even when only a small
amount of data is available (Dittmar and Gärtner, 2014; Dittmar and Müller,
2016; Vande Veire et al., 2021; Cai et al., 2021).

With this work, our goal is to address the longstanding data scarcity problem
that held back research into more advanced data-driven DSS methods in the past
few decades, and demonstrate that deep neural networks are not only a viable
solution but also an efficient tool when it comes to DDS. In this manuscript,
we present StemGMD, a new large-scale audio dataset of isolated drum stems
synthesized from several hours of expressive performances. We leverage StemGMD
to develop LarsNet, a deep DSS model that separates five stems from a stereo
drum mixture. Each stem is extracted by a dedicated U-Net yielding a spectro-
temporal soft mask that is applied to the short-time Fourier transform (STFT)
of the mixture signal. In our experiments, LarsNet consistently outperform
state-of-the-art DSS methods based on NMF and NMFD, while greatly reducing
inter-channel cross-talk artifacts111Audio examples are available at
https://polimi-ispl.github.io/larsnet and computational time.

Table 1: Overview of existing drums datasets.

| Clips | Duration (h) | Classes/Mics⋆ | Drum kits | Real/Synth | Human | Mixture | Transcription | Isolated stems
---|---|---|---|---|---|---|---|---|---
MDB-Drums (Southall et al., 2017) | $23$ | $0.345$ | $21$ | $\leq 23$ | R | $\checkmark$ | Mono | TXT | $\times$
IDMT-SMT-Drums (Dittmar and Gärtner, 2014) | $608$ | $2.1$ | $3$ | N/A | R/S | $\times$ | Mono | XML |  $64$ mixtures
ENST-Drums (Gillet and Richard, 2006) | $456$ | $3.75$ |  ⋆$20/8^{\star}$ | $3$ | R | $\checkmark$ | Stereo | TXT | $\checkmark$
GMD (Gillick et al., 2019) | $1150$ | $13.6$ | $22$ | $1$ | S | $\checkmark$ | Mono | MIDI | $\times$
TMIDT (Vogl et al., 2018) | $4197$ | $259$ | $18$ | $57$ | S | $\times$ | Mono | TXT | $\times$
E-GMD (Callender et al., 2020) | $45\,537$ | $444.5$ | $22$ | $43$ | S | $\checkmark$ | Mono | MIDI | $\times$
StemGMD (ours) | $103\,500$ | $1224$ | $9$ | $10$ | S | $\checkmark$ | Stereo | MIDI | $\checkmark$

##  2 StemGMD

In 2019, Magenta released GMD, a large corpus comprising 13.6 hours of human-
performed drum tracks recorded by ten drummers playing on a Roland TD-11
electronic drum kit (Gillick et al., 2019). GMD contains MIDI files for each
performance, along with the corresponding full-kit audio mixtures. Later on,
Callender et al. (2020) introduced E-GMD, an ADT dataset containing about 444
hours of audio data gathered by re-recoding all GMD sequences using 43 drum
kits. Despite being valuable for many tasks, such as beat generation, drum
infilling, ADT, and groove transfer (Gillick et al., 2019), these datasets do
not contain isolated stems and are thus unsuited for training deep source
separation models. In this work, we expand the GMD family by introducing
StemGMD, a new large-scale corpus of isolated drum stems.

First, we applied the note mapping proposed in (Gillick et al., 2019) to the
raw MIDI data provided with GMD. Such mapping, summarized in Table 2,
preserves the velocity and timing of each note while reducing the 22 original
MIDI channels to nine canonical instruments (whose General MIDI note numbers
are reported in brackets), i.e., Kick Drum (36), Snare (38), High Tom (50),
Low-Mid Tom (47), High Floor Tom (43), Closed Hi-Hat (42), Open Hi-Hat (46),
Crash Cymbal (49), and Ride Cymbal (51).

Then, we synthesized each of the nine channels of the resulting MIDI clips as
16-bit/$44.1$ kHz stereo WAV files. We used high-fidelity Logic Pro X drum
libraries, selecting ten different acoustic drum kits in order to cover a wide
range of timbres. To ensure the superposition principle that underlies most
source separation techniques, we obtained the mixture signals simply by
summing the nine synthesized stems. This procedure preserves a one-to-one
correspondence between the audio tracks and the original MIDI files. Hence,
GMD metadata were kept for each clip, e.g., duration, genre, tempo, and
drummer ID, which we augmented with the drum kit ID. The resulting audio
dataset, which we call StemGMD, is made freely available online.222StemGMD:
https://zenodo.org/records/7860223

StemGMD comprises more than 136 hours of drum mixtures and totals
approximately 1224 hours of audio data. With 103 500 clips, to the best of our
knowledge, StemGMD is the largest publicly available dataset of drums.
Moreover, it is the first large-size dataset of isolated stems to account for
all pieces in a standard drum kit, including toms and cymbals.

The train, test, and validation folds from Magenta’s GMD are retained in
StemGMD. However, the recommended test fold amounts to more than ten hours.
Therefore, in this work, we use a more manageable test split named Eval
Session. StemGMD Eval Session comprises 400 drum mixtures (40 for each drum
kit) and spans several genres, as it was collected by tasking four drummers to
record the same set of ten beats in their own style (Gillick et al., 2019).

Table 2: MIDI note mapping (Gillick et al., 2019).

Original Pitch | Roland Mapping | General MIDI Mapping | StemGMD Mapping | StemGMD Pitch
---|---|---|---|---
36 | Kick | Bass Drum 1 | Kick Drum | 36
38 | Snare (Head) | Acoustic Snare | Snare | 38
40 | Snare (Rim) | Electric Snare | Snare | 38
37 | Snare X-Stick | Side Stick | Snare | 38
48 | Tom 1 | Hi-Mid Tom | High Tom | 50
50 | Tom 1 (Rim) | High Tom | High Tom | 50
45 | Tom 2 | Low Tom | Low-Mid Tom | 47
47 | Tom 2 (Rim) | Low-Mid Tom | Low-Mid Tom | 47
43 | Tom 3 (Head) | High Floor Tom | High Floor Tom | 43
58 | Tom 3 (Rim) | Vibraslap | High Floor Tom | 43
46 | HH Open (Bow) | Open Hi-Hat | Open Hi-Hat | 46
26 | HH Open (Edge) | N/A | Open Hi-Hat | 46
42 | HH Closed (Bow) | Closed Hi-Hat | Closed Hi-Hat | 42
22 | HH Closed (Edge) | N/A | Closed Hi-Hat | 42
44 | HH Pedal | Pedal Hi-Hat | Closed Hi-Hat | 42
49 | Crash 1 (Bow) | Crash Cymbal 1 | Crash Cymbal | 49
55 | Crash 1 (Edge) | Splash Cymbal | Crash Cymbal | 49
57 | Crash 2 (Bow) | Crash Cymbal 2 | Crash Cymbal | 49
52 | Crash 2 (Edge) | Chinese Cymbal | Crash Cymbal | 49
51 | Ride (Bow) | Ride Cymbal 1 | Ride Cymbal | 51
59 | Ride (Edge) | Ride Cymbal 2 | Ride Cymbal | 51
53 | Ride (Bell) | Ride Bell | Ride Cymbal | 51

##  3 LarsNet

This section presents a new deep DSS model dubbed LarsNet,333Pretrained
LarsNet models are available at https://github.com/polimi-ispl/larsnet which
is designed to separate five drum stems from a stereo drum mixture, i.e., kick
drum (KD), snare drum (SD), tom-toms (TT), hi-hat (HH), and cymbals (CY). More
specifically, TT includes High Tom, Low-Mid Tom, and High Floor Tom; HH
includes Open and Closed Hi-Hat; and CY includes Crash and Ride cymbals.

###  3.1 Model Architecture

Inspired by prior work on MDX (Jansson et al., 2017; Hennequin et al., 2020;
Défossez, 2021), LarsNet comprises five parallel U-Nets (Ronneberger et al.,
2015), i.e., one for each target stem. As shown in Fig. 1, the U-Nets operate
in the time-frequency domain. Each is fed a (portion of a) two-channel
magnitude STFT computed from the stereo waveform of a drum mixture and outputs
a stem-specific soft mask of the same size. The time-domain stem signal is
estimated by taking the inverse STFT (iSTFT) of the Hadamard product between
the complex-valued STFT of the mixture and the real-valued mask thus obtained.
The input magnitude STFT is first cropped to retain only the lowest $F$
frequency bands. Then, zero-padding is applied along the temporal dimension
before segmenting it into an integer number of chunks of $T$ time frames. The
resulting spectro-temporal patches $\mathbf{X}\in\mathbb{R}_{\geq 0}^{2\times
F\times T}$ are normalized using Batch Norm computed across frequency bands
(FreqBN), instead of across channels as it is usual in computer vision
applications (Ioffe and Szegedy, 2015). As shown in Fig. 2, each U-Net
comprises $13$ convolutional layers. Such layers have $5\times 5$ kernels,
$2\times 2$ stride, and $2\times 2$ padding, except for the last decoder layer
which has $4\times 4$ kernels, $2\times 2$ dilation, and $3\times 3$ padding.
The decoder yields a soft mask $\mathbf{M}\in[0,1]^{2\times F\times T}$
through a Sigmoid nonlinearity. The mask is zero-padded back to the original
STFT size, and the temporal dimension is reinstated by concatenating all
$T$-sized chunks. Finally, the time-domain signal is reconstructed by taking
the iSTFT of the masked magnitude after padding any frequency band above $F$
with zeros. Albeit more sophisticated phase estimation techniques exist, e.g.,
(Griffin and Lim, 1984; Kobayashi et al., 2022), we opt to simply use the
original mixture phase when transforming the signal back into the time domain
for efficiency’s sake.

###  3.2 Model Training

We train the U-Nets in parallel on five NVIDIA Titan V GPUs for $100\,000$
iterations using Adam, a learning rate of $0.0001$, and a batch size of $24$.
Each mixture-stem training pair consists of aligned stereo waveform segments
of $11.85$ s extracted from StemGMD with a stride of $2$ s, resulting in
$110\,000$ segments per epoch. We compute the STFT using a periodic Hann
window of length $4096$ and a hop size of $1024$ (ca. $23$ ms.) Since we are
dealing with percussive instruments whose onsets have impulsive spectral
characteristics, we set $F=2048$ to ensure a broadband estimate of every stem.
Additionally, to avoid padding the STFT patches at training time, we fix
$T=512$. The loss function is computed as the $L^{1}$-norm of the error
between the magnitude STFT patch of the ground truth stem
$\mathbf{X}_{i}\in\mathbb{R}_{\geq 0}^{2\times F\times T}$ and that of the
masked mixture, i.e.,

$\mathcal{L}=\lVert\mathbf{X}_{i}-\mathbf{M}_{i}\odot\mathbf{X}\rVert_{1}\,,$
(1)

where $\odot$ denotes the Hadamard product. Throughout the training, we apply
data augmentation as described in Section 4. We use six of the ten drum kits
included in StemGMD to train LarsNet, whereas the remaining four are held out
for evaluation purposes on unseen drum sounds (see Table 3.)

###  3.3 Wiener Filtering

Once the training is over, the magnitude STFT of each stem can be inferred
independently of the others as

$\hat{\mathbf{X}}_{i}=\mathbf{M}_{i}\odot\mathbf{X}\,.$ (2)

Additionally, LarsNet implements $\alpha$-Wiener filtering (Liutkus and
Badeau, 2015) as in (Dittmar and Müller, 2016). Namely, the $i$th soft mask is
computed as

$\tilde{\mathbf{M}}_{i}=\hat{\mathbf{X}}_{i}^{\alpha}\oslash\left(\sum_{j=1}^{N}\hat{\mathbf{X}}_{j}^{\alpha}+\epsilon\right)\,,$
(3)

and the magnitude STFT is estimated as
$\tilde{\mathbf{X}}_{i}=\tilde{\mathbf{M}}_{i}\odot\mathbf{X}$, where
$\epsilon$ avoids division by zero, $\oslash$ indicates the Hadamard division,
and the exponent $\alpha\in(0,2]$ is applied in an element-wise fashion. In
the following, we report the results of this latter LarsNet variant by setting
$\alpha=1$. Notice that, differently from the forward inference mechanism, (3)
combines the estimates of all $N=5$ stems, preventing full parallelization.

###  3.4 Target Stems

In this work, we separate five stems (KD, SD, TT, HH, CY) even if StemGMD
contains isolated tracks for nine different instruments (see Section 2). From
an application standpoint, we argue that there are limited practical scenarios
in which one might want to isolate, e.g., the High Tom from the Low-Mid Tom,
as such a distinction mainly depends on the diameter and tuning of the drums
that may vary greatly across drum kits. Furthermore, some drummers may use two
tom-toms; others may have more than three. Having a single TT class allows one
to isolate that entire family of drums regardless of the drum kit composition.
Similarly, the CY class may be considered a stereo “overhead” track where all
cymbals are recorded at once. In fact, the ride and crash cymbals are rarely
recorded using dedicated microphones unlike, e.g., the hi-hat, which is
typically close-miked. Finally, the hi-hat is a percussive instrument that can
be played in two different ways (open or closed) depending on whether the
pedal is pressed or not. In having a single HH class, we favored the
consistency with respect to the acoustic source rather than the sounds it may
produce, akin to having a microphone capturing the instrument.

Figure 1: LarsNet architecture. Figure 2: U-Net architecture.

##  4 Data Augmentation

Having access to isolated audio stems enables countless data augmentation
strategies and allows one to draw inspiration from common music production
practices. We propose to use the following six methods;444Other data
augmentation methods were considered but not applied in the present study,
such as Panning (PN), i.e., randomly modify the level of the left and right
channels of each stem according to a panning law, Reverb (REV), i.e.,
stochastically apply artificial reverberation with random parameters to each
stem independently of the others, and Random (RND), i.e., select stems
completely at random regardless of whether they come from the same drumming
performance or not. the order in which they are presented below corresponds to
the order in which they are applied in our implementation.

  1. 1.

Kit-swap augmentation (KS): for each drum pattern, we create a novel mixture
by adding together stems from randomly selected drum kits.

  2. 2.

Doubling augmentation (DB): in modern-day music production, layering multiple
drum hits in a Digital Audio Workstation (DAW) is common practice. Inspired by
this, for each stem, we compute a new track by averaging the same pattern from
different drum kits. To save on I/O operations, this method is limited to two
drum kits at a time.

  3. 3.

Pitch-shift augmentation (PS): to simulate a wider range of drums and cymbals,
each stem is pitch shifted by a random amount of semitones using SoX as
backend. Specifically, we randomly sample integer-valued shifts in the range
of $\pm 3$ semitones.

  4. 4.

Saturation augmentation (ST): we apply nonlinear processing to individual
stems to simulate compression and saturation, which are common when mixing
drums in a DAW. Namely, for each stem, we compute the hyperbolic tangent
($\tanh$) after scaling the input waveform by a uniformly distributed random
variable $\beta_{i}\sim\mathcal{U}_{[1,5]}$.

  5. 5.

Channel-swap augmentation (CS): left and right channels of an audio track are
randomly swapped.

  6. 6.

Remix augmentation (RX): Each stem is multiplied by a scalar
$\gamma_{i}\sim\mathcal{U}_{\left[0.1,1\right]}$, corresponding to a gain
variation in the range of $-20$ dB to $0$ dB. Albeit increasing the gain above
$0$ dB would have been possible, we opted against it to avoid potential
clipping.

If we apply all augmentations at once, the mixture signal

$\mathbf{x}[n]=\sum_{i=1}^{N}\mathbf{x}_{i}[n]$ (4)

becomes

$\mathbf{x}_{\text{aug}}[n]=\sum_{i=1}^{N}\gamma_{i}\tanh\\!\left(\beta_{i}\cdot\mathrm{shift}_{\pm
3}\\!\left(\frac{\mathbf{C}\mathbf{x}_{i}^{(k)}[n]+\mathbf{C}\mathbf{x}_{i}^{(k^{\prime})}[n]}{2}\right)\right)\\!,$
(5)

where $i$ denotes the stem index out of $N$ stems, $k$ indicates the index of
a randomly selected drum kit, $k\neq k^{\prime}$,
$\mathbf{C}=\begin{bmatrix}0&1\\\ 1&0\end{bmatrix}$,
$\beta_{i}\sim\mathcal{U}_{\left[1,5\right]}$,
$\gamma_{i}\sim\mathcal{U}_{\left[0.1,1\right]}$, and $\mathrm{shift}_{\pm
3}(\cdot)$ is an operator shifting the pitch of the input signal by a random
integer amount in the range of $[-3,3]$ semitones. In practice, each
augmentation method is applied stochastically and independently of the others.
If applied, KS and RX involve all stems in the mixture, whereas applying DB,
PS, ST, and CS depends on the outcome of a Bernoulli trial run for each voice.
The probability of every augmentation is as follows:
$\mathrm{Pr}(\mathrm{KS})=\mathrm{Pr}(\mathrm{CS})=0.5$, and
$\mathrm{Pr}(\mathrm{DB})=\mathrm{Pr}(\mathrm{PS})=\mathrm{Pr}(\mathrm{ST})=\mathrm{Pr}(\mathrm{RX})=0.3$.
On top of that, at training time, we disable data augmentation altogether with
a probability of 50%.

##  5 Evaluation

To assess the performance of our deep DSS model, we consider the signal-to-
distortion ratio ($\mathrm{SDR}$). In particular, we adopt the definition used
in previous MDX challenges (Mitsufuji et al., 2022; Fabbro et al., 2023),
which Défossez (2021) refers to as $\mathrm{nSDR}$. Namely, for the $i$th
stem, the metric is given by

$\mathrm{nSDR}_{i}=10\log_{10}\dfrac{\sum_{n}\lVert\mathbf{x}_{i}[n]\rVert^{2}+\epsilon}{\sum_{n}\lVert\mathbf{x}_{i}[n]-\hat{\mathbf{x}}_{i}[n]\rVert^{2}+\epsilon}\,,$
(6)

where $n$ is the time index, $\epsilon=10^{-7}$ is a small scalar introduced
to avoid numerical problems, $\mathbf{x}_{i}[n]$ is the $i$th ground truth
source, and $\hat{\mathbf{x}}_{i}[n]$ is the estimated stereo waveform. Then,
the overall $\mathrm{nSDR}$ score is given by the average over the $N$ stems
of every clip in the test set, namely

$\mathrm{nSDR}=\frac{1}{N}\sum_{i=1}^{N}\mathrm{nSDR}_{i}\,.$ (7)

Different from the more widespread formulation by Vincent et al. (2006) (later
referred to as $\mathrm{oSDR}$ for clarity’s sake), (7) is simpler and faster
to evaluate, which is convenient given the considerable size of large-scale
deep learning datasets. The $\mathrm{SDR}$ is well-defined as long as the
discrete-time energy of the target signal $\mathbf{x}_{i}[n]$ is greater than
zero. If not, the optimal ratio in (6), i.e., assuming no distortion, is
nonpositive and caps at $0$ dB. Hence, we divide the test set into two parts
by annotating each stem according to the corresponding MIDI file.
Specifically, we analyze the results for all those stems where at least one
MIDI note is present (nonzero-energy stems) separately from those with no drum
hits (zero-energy stems.) We test LarsNet against the baseline methods
described in Section 5.1. The three methods are evaluated on the Eval Session
fold of StemGMD, comprising mixtures from ten drum kits, four of which were
held out during training.

###  5.1 Baseline Methods

Dittmar and Gärtner (2014) proposed a low-latency method for drums
transcription and separation of KD, SD, and HH based on frame-wise nonnegative
matrix factorization with semi-adaptive bases (SAB-NMF). The method aims at
decomposing the magnitude STFT of the drum mixture into two nonnegative
matrices, one comprising a set of spectral templates (bases) and the other
containing the corresponding temporal activations. In particular, SAB-NMF
achieves this factorization by processing each short-time spectrum
independently of the others.

Dittmar and Müller (2016) proposed an alternative approach based on
nonnegative matrix factor deconvolution (NMFD) followed by $\alpha$-Wiener
filtering. Differently from other methods, the authors assume one has prior
knowledge about the drum score and presents an informed NMFD variant. However,
such a prior is rarely available when it comes to real-life source separation
problems. Hence, we implement the baseline decomposition from (Dittmar and
Müller, 2016), which closely resembles the classic NMFD formulation by
Smaragdis (2004).

Compared to (Dittmar and Gärtner, 2014) and (Dittmar and Müller, 2016), we
apply the multiplicative update rules (Lee and Seung, 2000) for $K=200$
iterations instead of $25$ and $30$, as this improved the $\mathrm{nSDR}$
significantly.

In our implementation, the spectral basis functions are pre-computed using a
partition of StemGMD containing isolated drum and cymbal hits, each
synthesized at ten different velocities ranging from 30 to 127 using the same
ten drum kits as in the main dataset. Moreover, whereas the original
references only considered mono files, we process the two stereo channels
independently of one another. As for the remaining hyperparameters, we refer
the reader to the original papers.

###  5.2 Results

Table 3 reports the $\mathrm{nSDR}$ for every drum kit and isolated stem in
StemGMD Eval Session. The “All” rows show the average $\mathrm{nSDR}$ over all
stems in a drum kit, while the “All” column lists the average across all drum
kits for a given stem.

First, none of the methods exhibits a noticeable drop in performance when
evaluated on the four held-out drum kits (marked with † in Table 3),
suggesting that all three methods can generalize in the face of unseen timbral
characteristics.

By looking at the $\mathrm{nSDR}$ of nonzero-energy stems, we notice that
LarsNet provides, on average, a performance increment of $+20.39$ dB for KD,
$+6.91$ dB for SD, $+20.63$ dB for TT, $+3.14$ dB for HH, and $+8.4$ dB for CY
compared to SAB-NMF. Similarly, we report an increment of $+3.92$ dB for KD,
$+11.84$ dB for SD, $+13.28$ dB for TT, $+3.33$ dB for HH, and $+9.78$ dB for
CY with respect to NMFD. When accounting for all drum kits and stems, LarsNet
yields an $\mathrm{nSDR}$ of $17.7$ dB, against the $10.97$ dB of NMFD and
$7.24$ dB of SAB-NMF. Respectively, the average $\mathrm{oSDR}$ scores
(Vincent et al., 2006) obtained using mir_eval (Raffel et al., 2014) are
$17.91$ dB for LarsNet, $11.02$ dB for NMFD, and $7.23$ dB for SAB-NMF,
differing at most by $0.21$ dB from the respective $\mathrm{nSDR}$ scores.

As for the results pertaining to zero-energy stems, both SAB-NMF and NMFD have
an average $\mathrm{nSDR}$ of $-25.56$ dB. This indicates a great deal of
cross-talk between drum channels, i.e., sound components from other stems tend
to leak into the TT, HH, and CY tracks. Conversely, LarsNet scores a perfect
$0$ dB for many of the ten drum kits, meaning that the proposed method
correctly outputs silence when no drum hits are present in the ground truth
stem. Overall, LarsNet average $\mathrm{nSDR}$ is $-0.84$ dB, corresponding to
a remarkable $+24.7$ dB improvement upon the baselines.

Table 3: nSDR computed on StemGMD Eval Session. The six drum kits denoted by
the superscript ⋆ were included in the training dataset, whereas the four drum
kits denoted by † were held out for evaluation.

| Method | Stem | No. | Drum kit
---|---|---|---|---
|  |  |  | Brooklyn | East Bay | Heavy | Portland | Retro Rock | SoCal | Bluebird† | Detroit Garage† | Motown Rev.† | Roots† | All
—————— Nonzero-energy stems —————— | SAB-NMF (Dittmar and Gärtner, 2014) | KD |  $400$ ($100\%$) | $4.63$ | $3.48$ | $6.14$ | $9.57$ | $4.81$ | $5.49$ | $5.82$ | $5.30$ | $7.20$ | $15.55$ | $6.80$
SD |  $400$ ($100\%$) | $13.03$ | $16.10$ | $17.56$ | $11.99$ | $18.32$ | $20.73$ | $16.86$ | $9.14$ | $10.27$ | $14.58$ | $14.86$
TT |  $40$ ($10\%$) | $-11.66$ | $-11.81$ | $-12.13$ | $-10.36$ | $-14.78$ | $-10.06$ | $-11.45$ | $-16.96$ | $-11.06$ | $-5.07$ | $-11.53$
HH |  $390$ ($97.5\%$) | $1.97$ | $6.50$ | $3.88$ | $-0.21$ | $6.61$ | $6.47$ | $1.14$ | $-3.34$ | $5.57$ | $4.26$ | $3.29$
CY |  $50$ ($12.5\%$) | $-5.42$ | $-2.98$ | $-1.04$ | $-9.22$ | $-0.97$ | $-2.11$ | $-1.67$ | $-7.41$ | $-7.03$ | $-5.27$ | $-4.31$
All | $1280$ | $5.54$ | $7.61$ | $8.17$ | $5.99$ | $8.74$ | $9.77$ | $7.01$ | $2.67$ | $6.54$ | $10.35$ | $7.24$
NMFD (Dittmar and Müller, 2016) | KD |  $400$ ($100\%$) | $21.98$ | $24.29$ | $12.33$ | $25.12$ | $24.67$ | $24.14$ | $28.26$ | $27.19$ | $23.08$ | $21.62$ | $23.27$
SD |  $400$ ($100\%$) | $11.07$ | $10.83$ | $11.83$ | $10.27$ | $4.86$ | $12.75$ | $10.02$ | $8.75$ | $7.17$ | $11.71$ | $9.93$
TT |  $40$ ($10\%$) | $-2.64$ | $-2.73$ | $-12.62$ | $-0.48$ | $-8.80$ | $-1.67$ | $0.23$ | $-1.85$ | $-5.55$ | $-5.70$ | $-4.18$
HH |  $390$ ($97.5\%$) | $4.51$ | $3.35$ | $2.75$ | $-1.43$ | $4.21$ | $3.47$ | $3.72$ | $2.95$ | $3.93$ | $3.56$ | $3.10$
CY |  $50$ ($12.5\%$) | $-6.49$ | $-5.34$ | $-5.02$ | $-7.37$ | $-3.81$ | $-4.08$ | $-6.96$ | $-7.16$ | $-3.40$ | $-7.30$ | $-5.69$
All | $1280$ | $11.37$ | $11.70$ | $7.80$ | $10.32$ | $10.09$ | $12.38$ | $12.83$ | $11.79$ | $10.35$ | $11.04$ | $10.97$
LarsNet (Ours) | KD |  $400$ ($100\%$) | $27.07$ | $26.91$ | $26.22$ | $31.17$ | $27.10$ | $25.99$ | $29.29$ | $27.54$ | $25.56$ | $25.07$ | $\mathbf{27.19}$
SD |  $400$ ($100\%$) | $21.48$ | $20.91$ | $20.84$ | $22.61$ | $23.07$ | $24.61$ | $23.01$ | $21.49$ | $17.42$ | $22.23$ | $\mathbf{21.77}$
TT |  $40$ ($10\%$) | $9.22$ | $10.25$ | $9.49$ | $8.37$ | $10.43$ | $11.92$ | $9.50$ | $7.65$ | $8.37$ | $5.84$ | $\mathbf{9.10}$
HH |  $390$ ($97.5\%$) | $6.80$ | $9.04$ | $5.69$ | $4.52$ | $7.70$ | $8.35$ | $6.03$ | $4.54$ | $5.59$ | $6.03$ | $\mathbf{6.43}$
CY |  $50$ ($12.5\%$) | $4.46$ | $5.75$ | $5.08$ | $4.39$ | $4.82$ | $5.46$ | $3.91$ | $4.08$ | $3.32$ | $-0.35$ | $\mathbf{4.09}$
All | $1280$ | $\mathbf{17.71}$ | $\mathbf{18.24}$ | $\mathbf{16.93}$ | $\mathbf{18.62}$ | $\mathbf{18.54}$ | $\mathbf{18.94}$ | $\mathbf{18.63}$ | $\mathbf{17.10}$ | $\mathbf{15.53}$ | $\mathbf{16.79}$ | $\mathbf{17.70}$
——— Zero-energy stems ——— | SAB-NMF (Dittmar and Gärtner, 2014) | TT |  $360$ ($90\%$) | $-37.33$ | $-37.09$ | $-36.77$ | $-36.38$ | $-33.89$ | $-30.49$ | $-35.29$ | $-33.49$ | $-32.85$ | $-26.06$ | $-33.96$
HH |  $10$ ($2.5\%$) | $-27.91$ | $-30.16$ | $-28.13$ | $-20.68$ | $-20.11$ | $-25.25$ | $-25.13$ | $-27.85$ | $-24.16$ | $-27.55$ | $-25.69$
CY |  $350$ ($87.5\%$) | $-20.60$ | $-14.13$ | $-7.98$ | $-25.97$ | $-12.57$ | $-11.15$ | $-10.92$ | $-23.23$ | $-25.06$ | $-17.44$ | $-16.91$
All | $720$ | $-29.07$ | $-25.83$ | $-22.66$ | $-31.10$ | $-23.34$ | $-21.01$ | $-23.30$ | $-28.42$ | $-28.95$ | $-21.89$ | $-25.56$
NMFD (Dittmar and Müller, 2016) | TT |  $360$ ($90\%$) | $-27.19$ | $-24.14$ | $-33.91$ | $-20.97$ | $-35.82$ | $-23.25$ | $-24.46$ | $-23.34$ | $-27.14$ | $-30.62$ | $-27.08$
HH |  $10$ ($2.5\%$) | $-24.67$ | $-28.37$ | $-27.26$ | $-25.54$ | $-22.82$ | $-23.34$ | $-21.59$ | $-23.88$ | $-26.43$ | $-26.67$ | $-25.06$
CY |  $350$ ($87.5\%$) | $-24.61$ | $-23.53$ | $-21.61$ | $-28.85$ | $-21.13$ | $-22.20$ | $-24.15$ | $-24.44$ | $-25.51$ | $-24.12$ | $-24.01$
All | $720$ | $-25.90$ | $-23.90$ | $-27.84$ | $-24.87$ | $-28.50$ | $-22.74$ | $-24.27$ | $-23.88$ | $-26.33$ | $-27.41$ | $-25.56$
LarsNet (Ours) | TT |  $360$ ($90\%$) | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $-8.99$ | $\mathbf{-0.90}$
HH |  $10$ ($2.5\%$) | $0.00$ | $-4.34$ | $-3.78$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $-7.24$ | $-5.48$ | $-23.65$ | $\mathbf{-4.45}$
CY |  $350$ ($87.5\%$) | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $-2.46$ | $-4.33$ | $\mathbf{-0.68}$
All | $720$ | $\mathbf{0.00}$ | $\mathbf{-0.06}$ | $\mathbf{-0.05}$ | $\mathbf{0.00}$ | $\mathbf{0.00}$ | $\mathbf{0.00}$ | $\mathbf{0.00}$ | $\mathbf{-0.10}$ | $\mathbf{-1.27}$ | $\mathbf{-6.93}$ | $\mathbf{-0.84}$

###  5.3 Real-Time Performance

LarsNet is implemented in Python using PyTorch. Despite not being optimized
for speed, the model achieves an average Real-Time Ratio (RTR) of $0.016$ on a
single NVIDIA TITAN V GPU ($62.5$ times faster than real-time) and $0.15$ on
an Intel Xeon E5-2687W ($6.6$ times faster than real-time). Conversely, on the
same CPU, SAB-NMF and NMFD achieve an RTR of $4.85$ (slower than real-time)
and $0.4$ (only $2.5$ times faster than real-time), respectively.

##  6 Conclusions

In this manuscript, we presented StemGMD, the first large-scale audio dataset
of isolated drum stems encompassing all the percussion instruments of a
canonical nine-piece acoustic drum kit, and LarsNet, a first-ever deep drum
source separation model. Based on a parallel arrangement of dedicated U-Nets,
LarsNet can extract five stems from a stereo drum mixture up to $62.5$ times
faster than real-time. At the same time, LarsNet is shown to significantly
outperform state-of-the-art methods based on nonnegative spectro-temporal
decomposition in terms of signal-to-distortion ratio and unwanted cross-talk
artifacts. Ultimately, LarsNet is intended to serve as a baseline for future
work on drum source separation, while StemGMD has the capability to foster
further research into the applications of deep learning to music demixing.

## References

  * Wang and Chen [2018] DeLiang Wang and Jitong Chen.  Supervised speech separation based on deep learning: An overview.  _IEEE/ACM Trans. Audio Speech Lang. Process._ , 26(10):1702–1726, 2018.
  * Kavalerov et al. [2019] Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R. Hershey.  Universal sound separation.  In _IEEE Workshop Appl. Signal Process. Audio Acoust. (WASPAA)_ , pages 175–179, 2019.
  * Stöter et al. [2019] Fabian-Robert Stöter, Stefan Uhlich, Antoine Liutkus, and Yuki Mitsufuji.  Open-Unmix – A reference implementation music source separation.  _J. Open Source Softw._ , 4(41):1667, 2019.
  * Hennequin et al. [2020] Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussallam.  Spleeter: A fast and efficient music source separation tool with pre-trained models.  _J. Open Source Softw._ , 5(50):2154, 2020.
  * Samuel et al. [2020] David Samuel, Aditya Ganeshan, and Jason Naradowsky.  Meta-learning extractors for music source separation.  In _2020 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , pages 816–820, 2020.
  * Sawata et al. [2021] Ryosuke Sawata, Stefan Uhlich, Shusuke Takahashi, and Yuki Mitsufuji.  All for one and one for all: Improving music separation by bridging networks.  In _2021 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , pages 51–55, 2021.
  * Défossez [2021] Alexandre Défossez.  Hybrid spectrogram and waveform source separation.  In _Proc. MDX Workshop_ , pages 1–13, 2021.
  * Stöter et al. [2018] Fabian-Robert Stöter, Antoine Liutkus, and Nobutaka Ito.  The 2018 signal separation evaluation campaign.  In _Latent Variable Analysis and Signal Separation_ , pages 293–305, 2018.
  * Mitsufuji et al. [2022] Yuki Mitsufuji, Giorgio Fabbro, Stefan Uhlich, Fabian-Robert Stöter, Alexandre Défossez, Minseok Kim, Woosung Choi, Chin-Yun Yu, and Kin-Wai Cheuk.  Music demixing challenge 2021.  _Frontiers in Signal Processing_ , 1, 2022.
  * Fabbro et al. [2023] Giorgio Fabbro, Stefan Uhlich, Chieh-Hsin Lai, Woosung Choi, Marco Martínez-Ramírez, Weihsiang Liao, Igor Gadelha, Geraldo Ramos, Eddie Hsu, Hugo Rodrigues, et al.  The sound demixing challenge 2023 — Music demixing track.  _arXiv preprint arXiv:2308.06979_ , 2023.
  * Wu et al. [2018] Chih-Wei Wu, Christian Dittmar, Carl Southall, Richard Vogl, Gerhard Widmer, Jason Hockman, Meinard Müller, and Alexander Lerch.  A review of automatic drum transcription.  _IEEE/ACM Trans. Audio Speech Lang. Process_ , 26(9):1457–1483, 2018.
  * Vogl et al. [2017] Richard Vogl, Matthias Dorfer, and Peter Knees.  Drum transcription from polyphonic music with recurrent neural networks.  In _2017 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , pages 201–205, 2017.
  * Choi and Cho [2019] Keunwoo Choi and Kyunghyun Cho.  Deep unsupervised drum transcription.  In _Proc. 20th Int. Soc. Music Inf. Retrieval Conf. (ISMIR)_ , pages 183–191, 2019.
  * Callender et al. [2020] Lee Callender, Curtis Hawthorne, and Jesse Engel.  Improving perceptual quality of drum transcription with the Expanded Groove MIDI dataset.  _arXiv preprint arXiv:2004.00188_ , 2020.
  * Ishizuka et al. [2021] Ryoto Ishizuka, Ryo Nishikimi, and Kazuyoshi Yoshii.  Global structure-aware drum transcription based on self-attention mechanisms.  _Signals_ , 2(3):508–526, 2021.
  * Dittmar and Gärtner [2014] Christian Dittmar and Daniel Gärtner.  Real-time transcription and separation of drum recordings based on NMF decomposition.  In _Int. Conf. Digital Audio Effects (DAFx-14)_ , pages 187–194, 2014.
  * Dittmar and Müller [2016] Christian Dittmar and Meinard Müller.  Reverse engineering the amen break — score-informed separation and restoration applied to drum recordings.  _IEEE/ACM Trans. Audio Speech Lang. Process_ , 24(9):1535–1547, 2016.
  * Vande Veire et al. [2021] Len Vande Veire, Cedric De Boom, and Tijl De Bie.  Sigmoidal NMFD: convolutional NMF with saturating activations for drum mixture decomposition.  _Electronics_ , 10(3):284, 2021.
  * Cai et al. [2021] Cheng-Yu Cai, Yu-Hui Su, and Li Su.  Dual-channel drum separation for low-cost drum recording using non-negative matrix factorization.  In _2021 Asia-Pacific Signal Inf. Process. Association Annu. Summit Conf. (APSIPA ASC)_ , pages 17–22, 2021.
  * Southall et al. [2017] Carl Southall, Chih-Wei Wu, Alexander Lerch, and Jason Hockman.  MDB Drums – an annotated subset of MedleyDB for automatic drum transcription.  In _Late-Breaking Demo Session, 18th Int. Soc. Music Inf. Retrieval Conf. (ISMIR)_ , 2017.
  * Gillet and Richard [2006] Olivier Gillet and Gaël Richard.  ENST-drums: an extensive audio-visual database for drum signals processing.  In _Proc. 7th Int. Soc. Music Inf. Retrieval Conf. (ISMIR)_ , pages 156–159, 2006.
  * Gillick et al. [2019] Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, and David Bamman.  Learning to groove with inverse sequence transformations.  In _Int. Conf. Mach. Learning (ICML)_ , volume 97, pages 2269–2279, 2019.
  * Vogl et al. [2018] Richard Vogl, Gerhard Widmer, and Peter Knees.  Towards multi-instrument drum transcription.  In _Int. Conf. Digital Audio Effects (DAFx-18)_ , pages 57–64, 2018.
  * Jansson et al. [2017] Andreas Jansson, Eric J. Humphrey, Nicola Montecchio, Rachel M. Bittner, Aparna Kumar, and Tillman Weyde.  Singing voice separation with deep U-net convolutional networks.  In _Proc. 18th Int. Soc. Music Inf. Retrieval Conf. (ISMIR)_ , pages 745–751, 2017.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Proc. 18th Mid. Image Comput. Comput.-Assisted Intervention Conf. (MICCAI)_ , pages 234–241, 2015.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Int. Conf. Mach. Learning (ICML)_ , pages 448–456, 2015.
  * Griffin and Lim [1984] Daniel W. Griffin and Jae Lim.  Signal estimation from modified short-time fourier transform.  _IEEE Trans. Acoust. Speech Signal Process._ , 32(2):236–243, 1984.
  * Kobayashi et al. [2022] Tomoki Kobayashi, Tomoro Tanaka, Kohei Yatabe, and Yasuhiro Oikawa.  Acoustic application of phase reconstruction algorithms in optics.  In _2022 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , pages 6212–6216, 2022.
  * Liutkus and Badeau [2015] Antoine Liutkus and Roland Badeau.  Generalized Wiener filtering with fractional power spectrograms.  In _2015 IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , pages 266–270, 2015.
  * Vincent et al. [2006] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte.  Performance measurement in blind audio source separation.  _IEEE Trans. Audio Speech Lang. Process._ , 14(4):1462–1469, 2006.
  * Smaragdis [2004] Paris Smaragdis.  Non-negative matrix factor deconvolution; extraction of multiple sound sources from monophonic inputs.  In _Ind. Compon. Anal. Blind Signal Separation (ICA BSS)_ , pages 494–499, 2004.
  * Lee and Seung [2000] Daniel Lee and H Sebastian Seung.  Algorithms for non-negative matrix factorization.  _Adv. Neural Inf. Process. Syst. (NIPS)_ , 13, 2000.
  * Raffel et al. [2014] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, and Daniel P W Ellis.  mir_eval: A transparent implementation of common MIR metrics.  In _Proc. 15th Int. Soc. Music Inf. Retrieval Conf. (ISMIR)_ , volume 10, page 2014, 2014.


#  Semi-Supervised Few-Shot Classification
with Deep Invertible Hybrid Models

Yusuke Ohtsubo, Tetsu Matsukawa, Einoshin Suzuki  Graduate School/Faculty of
Information Science and Electrical Engineering,
Kyushu University Fukuoka, Japan
<EMAIL_ADDRESS>{matsukawa<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a deep invertible hybrid model which integrates
discriminative and generative learning at a latent space level for semi-
supervised few-shot classification. Various tasks for classifying new species
from image data can be modeled as a semi-supervised few-shot classification,
which assumes a labeled and unlabeled training examples and a small support
set of the target classes. Predicting target classes with a few support
examples per class makes the learning task difficult for existing semi-
supervised classification methods, including self-training, which iteratively
estimates class labels of unlabeled training examples to learn a classifier
for the training classes. To exploit unlabeled training examples effectively,
we adopt as the objective function the composite likelihood, which integrates
discriminative and generative learning and suits better with deep neural
networks than the parameter coupling prior, the other popular integrated
learning approach. In our proposed model, the discriminative and generative
models are respectively Prototypical Networks, which have shown excellent
performance in various kinds of few-shot learning, and Normalizing Flow a deep
invertible model which returns the exact marginal likelihood unlike the other
three major methods, i.e., VAE, GAN, and autoregressive model. Our main
originality lies in our integration of these components at a latent space
level, which is effective in preventing overfitting. Experiments using mini-
ImageNet and VGG-Face datasets show that our method outperforms self-training
based Prototypical Networks.

###### Index Terms:

Composite Likelihood, Deep Generative Model, Discriminative and Generative
Learning, Few-Shot Learning, Semi-Supervised Learning

##  I Introduction

Suppose you wish to classify images into five kinds of new species given five
reference images for each new species and a large number of images of known
species. Few-shot classification [3, 13, 33, 30] models this kind of task
using a large training set which consists of labeled examples of known classes
and a small support set which consists of labeled examples of the target
classes.

Due to the high cost of obtaining labeled examples, semi-supervised learning
exploits unlabeled examples in addition to labeled examples in the training
set. Semi-supervised few-shot classification [23, 15], which is an extension
of the few-shot classification, also takes unlabeled training examples. Such
unlabeled training examples are considered useful in extracting common
features among different classes, e.g., unlabeled examples could contribute to
discovering spines for better vertebrate classification.

Existing methods for semi-supervised few-shot classification [23, 15] use such
unlabeled examples in the framework of self-training [25]. In self-training, a
model is first learnt from labeled training examples and used to predict the
class labels of the unlabeled training examples. Then these unlabeled examples
with their predicted class labels are added to the training set to learn the
next model and this process is iterated. If the class labels of the unlabeled
training examples are correctly predicted, the next model is expected to be
more accurate than the former one, which has been proved by the experiments
[23].

However, the opposite case in which unlabeled examples decrease classification
accuracy happens if the prediction is poor. Another series of experiments [23]
show that such a decrease happens when the unlabeled examples belong to
different classes from the labeled training classes. This procedure stems from
the fact that self-training methods are based on a discriminative model, which
requires identical classes to labeled examples. Unfortunately, the assumption
that the classes of the unlabeled examples are identical to those of the
labeled examples does not always hold true.

To solve this problem, we introduce a generative model, which allows us to
exploit unlabeled examples with no assumption of their classes, in addition to
a discriminative model. We integrate these models at the latent space level in
the framework of composite likelihood [17, 19]. Concretely speaking, we use,
as the discriminative model, Prototypical network [30], which can solve few-
shot classification, and, as the generative model, Normalizing Flow [1, 10] a
deep neural network capable of computing exact likelihood. Another advantage
is that our framework can be systematically described in the framework of the
maximum likelihood principle, whereas the self-training [23] deviates from it.

Note that there is a conjecture that a hybrid model of discriminative and
generative models exhibits better predictive performance when the labeled
training examples are scarce [20, 16]. Nevertheless, this conjecture has not
been confirmed on a practical problem such as few-shot classification.
Recently, a hybrid model [19] based on Normalizing Flow has been proposed.
However, it uses a regression model as the discriminative model and is thus
unable to cope with the few-shot classification problem directly. We expand
this approach [19] to a semi-supervised few-shot classification model and
confirm the advantage of the hybrid model on this problem.

Our contributions are summarized as follows:

  * •

We construct semi-supervised few-shot classification model without assuming
that unlabeled data share the same class with labeled data.

  * •

We expand a deep hybrid generative/discriminative model to a semi-supervised
few-shot classification model.

  * •

We experimentally show the intermediate model of the generative and
discriminative models achieve higher predictive performance than each of the
component models on the semi-supervised few-shot classification problem.

This paper is structured as follows. Section II explains related works and
background of this paper. Section III explains our target problem. Sections IV
and V explain our proposed hybrid model and experiments, respectively. Section
VI concludes.

##  II Related Work

###  II-A Semi-supervised Few-shot Classification

Fei-Fei et al. [3] initiated few-shot learning, and after Lake et al.’s work
[13], which uses a hierarchical model, the researches on the task became much
more active. Later, many methods based on neural networks are proposed. Such
methods can be classified into initialization based methods [4, 5] and
distance metric learning based methods [33, 30]. The former methods obtain
good initial model parameters using the training set so that they can
effectively cope with a small test set [4, 5]. The latter methods learn a
distance metric between a pair of images and estimate the test class based on
the distances to a small number of test examples, e.g., Matching Networks
[33]. Prototypical Networks [30] belong to the latter kind of methods, as they
learn a distance metric between a sample and a class prototype which is
defined as the mean of examples belonging to the class. Note that we can use
another distance metric learning based method instead of the Prototypical
Networks.

Semi-supervised methods can be classified into generative model based methods
[8] and self-training based methods [35, 25, 23, 15]. OPBN [8] belongs to the
former and can include unlabeled data in the likelihood function. It can
handle only binary classification and maximizes the lower-bound likelihood
because it cannot obtain the exact likelihood. Even if its exact likelihood
could be obtained, its extension to multiway classification by using
Prototypical Networks corresponds to a special case (pure generative model) of
our proposed method. Prototypical Networks with K-means (PNK) [23] belongs to
the self-training, which iteratively estimates the class labels of unlabeled
examples to improve the test accuracy of the classifier. In the self-training
based methods [23, 15], unlabeled training examples which belong to different
classes from the labeled training classes are considered to have no
contribution in the improvement. In our experiments, we will compare the
proposed method with PNK due to its simplicity for implementation.

###  II-B Deep Generative Model

A deep generative model obtains the marginal density with a neural network
[28]. It evolved into methods which efficiently estimate parameters with the
backpropagation algorithm, e.g., VAE [11, 24], GAN[6], Autoregressive model
[7] and Flow [1]. VAE [11, 24] maximizes the lower-bound of the log-
likelihood, and it has computational challenges [19]. Since GAN [6] minimizes
the Jensen-Shannon divergence instead of maximizing the likelihood, it is
difficult to adapt it in the framework of composite likelihood maximization.
Autoregressive model [7] computes likelihood by conditioning each dimension of
the input space on other dimensions sequentially, and thus its computation
cannot be parallelized. Consequently, it is time-inefficient in learning from
high-dimensional data including images [10]. Normalizing Flow [1] can obtain
exact log-likelihood and thus can be extended to a method which maximizes the
composite likelihood. Though we adopted Glow [10] as the architecture of
Normalizing Flow, we can also adopt other kinds of architectures [1, 2] in our
approach. Refer to an excellent survey [21] of deep invertible models for
details.

###  II-C Generative and Discriminative Models

Let us denote $x$ an example and $y$ a label, $\mathcal{X}^{\rm L}$ labeled
examples and $\mathcal{Y}$ its labels. Given a parametric model which is
governed by model paramters $\theta$, many methods in the maximum likelihood
approach either assume a generative model
$p(y,\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$, or a discriminative model
$p(y|\mbox{\boldmath$x$},\mbox{\boldmath$\theta$})$ as the likelihood function
to estimate the maximum likelihood estimate
$\hat{\mbox{\boldmath$\theta$}}_{\rm ML}$. Assuming a generative model allows
us to include unlabeled data $\mathcal{X}^{U}$ in addition to the labeled data
$\mathcal{X}^{L},\mathcal{Y}$ in the likelihood function [14], which results
in

$\displaystyle\hat{\mbox{\boldmath$\theta$}}_{\rm
ML}=\operatorname*{\arg\\!\max}_{\mbox{\boldmath$\theta$}}\;p(\mathcal{Y},\mathcal{X}^{\rm
L},\mathcal{X}^{\rm U}|{\mbox{\boldmath$\theta$}}).$ (1)

Assuming a discriminative model results in
$\hat{\mbox{\boldmath$\theta$}}_{\rm
ML}=\operatorname*{\arg\\!\max}_{\mbox{\scriptsize\boldmath$\theta$}}p(\mathcal{Y}|\mathcal{X}^{\rm
U},\mathcal{X}^{\rm L},\mbox{\boldmath$\theta$})$, though in practice it is
difficult to include unlabeled data $\mathcal{X}^{\rm U}$ in the likelihood
function. It is common to adopt the supervised learning setting and thus

$\displaystyle\hat{\mbox{\boldmath$\theta$}}_{\rm
ML}=\operatorname*{\arg\\!\max}_{\mbox{\boldmath$\theta$}}\>p(\mathcal{Y}|\mathcal{X}^{\rm
L},\mbox{\boldmath$\theta$}).$ (2)

There are pros and cons for the discriminative model
$p(y|\mbox{\boldmath$x$},\mbox{\boldmath$\theta$})$ and the generative model
$p(y,\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$. Vapnik [32] stated “one
should solve the problem (such as modeling
$p(y|\mbox{\boldmath$x$},\mbox{\boldmath$\theta$})$) directly and never solve
a more general problem (such as modeling $p(y,\mbox{\boldmath$x$})$) as an
intermediate step”. On the other hand, Ng et al. [20] proved the following
theorem for linear models. Though the optimal generalization loss, i.e., the
generalization loss with infinite training data, is lower for discriminative
models than generative models, the latter requires fewer data than the former
to attain it. A discriminative model yields a low generalization loss with
abundant training data [20]. On the other hand, a generative model, which
attains the optimal generalization loss with fewer data, happens to show a
lower generalization loss than a discriminative model when the training data
is scarce [20]. In other words, one should use a discriminative model when the
training data are abundant but he/she might achieve a lower generalization
loss with a generative model when the training data are scarce. In addition, a
generative model can exploit unlabeled data in the likelihood function [14] as
shown in Eq.(1).

###  II-D Hybrid Models

To integrate generative and discriminative models, parameter coupling prior
[14] and composite likelihood [17] are proposed. The parameter coupling prior
uses separate model parameters for generative and discriminate models and
considers their coupled prior distribution. The coupled prior distribution
regularizes training such that parameters of generative and discriminative
models become similar by means of Euclidean distance. We will use the neural
networks for the modeling of the likelihood. In this case, the Euclidean
distance between the parameters does not measure valuable differences between
the functions realized by the neural networks [31].

The composite likelihood, which is also known as pseudo-likelihood [16] or
multi-conditional learning [17], represents a weighted product of component
likelihood [16] and is better suited for integrating the likelihoods modeled
by neural networks. Liang et al conducted a theoretical analysis of the
behavior of the composite model. Hybrid models are extended to neural network
based methods [29, 12]. Several works maximize lower-bounds of the log-
likelihood of the model using stochastic gradient variational inference [11,
24]. Recently, a more direct method [19], which maximizes the composite
likelihood by using Normalizing Flow [1], a neural network capable of
conducting an inverse transformation from the latent space to the input space,
has been proposed. However, it uses a regression model as the discriminative
model and is thus unable to cope with new test classes in the few-shot
classification problem. We expand this approach [19] to a distance metric-
based semi-supervised few-shot classification model.

##  III Target Problem

Figure 1 shows the semi-supervised few-shot classification problem. The task
is to classify a test example $x$ into one of the test classes, given a large
training set $\mathcal{D}^{\rm train}=\\{\mathcal{X}^{\rm
L},\mathcal{Y},\mathcal{X}^{\rm U}\\}$ and a support set $\mathcal{S}$. We
refer to the classes included in the training and support sets as training and
test classes, respectively.

The training set consists of a set of $L$ labeled examples,
$\\{\mathcal{X}^{\rm
L},\mathcal{Y}\\}=\\{(\mbox{\boldmath$x$}_{l},y_{l})\\}_{l=1}^{L}$ and a set
of $U$ unlabeled examples $\mathcal{X}^{\rm
U}=\\{\mbox{\boldmath$x$}_{u}\\}_{u=1}^{U}$ where two kinds of training
examples $\mbox{\boldmath$x$}_{l},\mbox{\boldmath$x$}_{u}$ are in an identical
input space111 In this paper, the input space is image space, and we model the
mapping to a latent space by a deep neural network.  and $y_{l}$ is a class
label in labeled training classes $\mathcal{C}_{\rm L}$. It is common to
assume that each unlabeled example $\mbox{\boldmath$x$}_{u}$ belongs to one of
(unknown) unlabeled classes $\mathcal{C}_{\rm U}$. Usually, the unlabeled
classes $\mathcal{C}_{\rm U}$ include several classes of $\mathcal{C}_{\rm
L}$.

The support set $\mathcal{S}$ consists of labeled examples which are described
in the same input space and belonging to test class labels. The test classes
are different from both the training classes $\mathcal{C}_{\rm L}$ and
$\mathcal{C}_{\rm U}$. We tackle $C$-way $K$-shot classification, in which the
number of the test classes is $C$, and the support set contains $K$ examples
for each of the test classes.

Figure 1: Semi-supervised few-shot classification problem

##  IV Proposed Model

###  IV-A Our Model

From a probabilistic perspective, we model the few-shot classification and
estimate a predictive distribution of label $y$ over the $C$ test classes for
each test example $x$ given the training and support sets. In case of the
Bayesian inference, a predictive distribution of $y$ is given by
$p(y|{\mbox{\boldmath$x$}},\mathcal{S},\mathcal{D}^{\rm train})$. We introduce
a parametric model which is governed by model parameters $\theta$. In this
paper, we conduct maximum likelihood estimation on the training set and thus
the predictive distribution becomes
$p(y|{\mbox{\boldmath$x$}},\mathcal{S},\hat{\mbox{\boldmath$\theta$}}_{\rm
ML})$ in inference time.

In the training, we make use of the composite likelihood model [16, 17] which
is a weighted product of a discriminative model and a generative model. The
composite likelihood is given as:

$\displaystyle\mathcal{L}(\alpha,\beta)=p(\mathcal{Y}|\mathcal{X}^{\rm
L},\mbox{\boldmath$\theta$})^{\alpha}\;p(\mathcal{X}^{\rm L},\mathcal{X}^{\rm
U}|\mbox{\boldmath$\theta$})^{\beta},$ (3)

where $\alpha\geq 0$ and $\beta\geq 0$ are the weights of the discriminative
model and the weight of the marginal likelihood of $x$, respectively. Note
that $\mathcal{L}(1,0)$ and $\mathcal{L}(1,1)$ represent the likelihood
functions of a pure discriminative and generative models, respectively. Since
the influences of $\alpha$ and $\beta$ are relative in a maximum likelihood
estimation, for simplicity we set $\alpha=1$ and take $\beta$ as a variable.
Thus we hereafter denote $\mathcal{L}(1,\beta)$ by $\mathcal{L}$.

We introduce a latent space realized with an invertible neural network [1] for
modeling the marginal likelihood and the discriminative model. We assume a
latent space which is shared by the labeled and unlabeled training examples.
The shared latent space allows us to exploit the unlabeled training examples
regardless of their classes.

In the following, Section IV-B explains the marginal likelihood
$p(\mathcal{X}^{\rm L},\mathcal{X}^{\rm U}|\mbox{\boldmath$\theta$})$ which is
a component of the composite likelihood. Section IV-C explains the
discriminative model $p(\mathcal{Y}|\mathcal{X}^{\rm
L},\mbox{\boldmath$\theta$})$. Section IV-D explains the resulting hybrid
model. Section IV-E explains the details of the paramter estimation.

###  IV-B Marginal Density Model: Normalizing Flow

For our marginal density model, we use Normalizing Flow [1] a deep generative
model which can efficiently obtain marginal density222We described the reason
other deep generative models are inadequate as a component of the composite
likelihood in Section II-B. $p(\mathcal{X}^{\rm L},\mathcal{X}^{\rm
U}|\mbox{\boldmath$\theta$})$. We assume that the examples are mutually
independent, i.e., $p(\mathcal{X}^{\rm L},\mathcal{X}^{\rm
U}|\mbox{\boldmath$\theta$})=\prod_{l=1}^{L}p(\mbox{\boldmath$x$}_{l}|\mbox{\boldmath$\theta$})\prod_{u=1}^{U}p(\mbox{\boldmath$x$}_{u}|\mbox{\boldmath$\theta$})=\prod_{n=1}^{L+U}p(\mbox{\boldmath$x$}_{n}|\mbox{\boldmath$\theta$})$
where
$\\{\mbox{\boldmath$x$}_{n}\\}_{n=1}^{L}=\\{\mbox{\boldmath$x$}_{l}\\}_{l=1}^{L}$
and
$\\{\mbox{\boldmath$x$}_{n}\\}_{n=L+1}^{L+U}=\\{\mbox{\boldmath$x$}_{u}\\}_{u=1}^{U}$.
Normalizing Flow introduces latent variables $z$ and assumes a simple
distribution such as normal distribution as a prior distribution of the latent
variable $p_{z}(\mbox{\boldmath$z$})$. It models complex333For instance, it
can represent complex natural images. marginal density
$p(\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$ with a transformation using
a neural network
$\mbox{\boldmath$x$}=f_{\mbox{\scriptsize\boldmath$\theta$}}(\mbox{\boldmath$z$})$.
Here, the network is composed of an architecture which allows the inverse
transformation
$\mbox{\boldmath$z$}=f^{-1}_{\mbox{\scriptsize\boldmath$\theta$}}(\mbox{\boldmath$x$})$.
Under the transformation of variables by
$\mbox{\boldmath$x$}=f_{\mbox{\scriptsize\boldmath$\theta$}}(\mbox{\boldmath$z$})$,
marginal density $p(\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$ is given by
the following equation:

$\displaystyle
p(\mbox{\boldmath$x$}_{n}|\mbox{\boldmath$\theta$})=p_{z}(f_{\mbox{\scriptsize\boldmath$\theta$}}^{-1}(\mbox{\boldmath$x$}_{n}))\left|{\rm
det}\left(\frac{\partial
f_{\mbox{\scriptsize\boldmath$\theta$}}^{-1}({\mbox{\boldmath$x$}_{n}})}{\partial\mbox{\boldmath$x$}_{n}}\right)\right|,$
(4)

where $\left|{\rm det}\left(\frac{\partial
f_{\mbox{\tiny\boldmath$\theta$}}^{-1}(\mbox{\boldmath$x$})}{\partial\mbox{\boldmath$x$}}\right)\right|$
represents the absolute value of the Jacobian determinant of the
transformation $f^{-1}_{\mbox{\scriptsize\boldmath$\theta$}}(\cdot)$ at $x$.

Real-valued Non-Volume Preserving (RNVP) transformation [2] and Glow
transformation [10] are popular architectures of neural networks
$f_{\mbox{\scriptsize\boldmath$\theta$}}(\cdot)$ which allow the inverse
transformation. We used the Glow architecture for our experiments. For the
details of its architecture, see [10]. The crux here is that by the
introduction of a neural network
$\mbox{\boldmath$x$}=f_{\mbox{\scriptsize\boldmath$\theta$}}(\mbox{\boldmath$z$})$
which allows inverse transformation, we can obtain the gradient of the exact
marginal likelihood $\frac{\partial
p(\mbox{\boldmath$x$}|\mbox{\scriptsize\boldmath$\theta$})}{\partial\mbox{\scriptsize\boldmath$\theta$}}$,
which is necessary to optimize the model parameters in the composite
likelihood of Eq. (3).

###  IV-C Discriminative Model: Prototypical Networks

Deep Invertible Generalized Linear Model (DIGLM) [19] also uses Normalizing
Flow in the framework of the composite likelihood model. However, it uses a
regression model as the discriminative model
$p(\mathcal{Y}|\mathcal{X}^{L},\mbox{\boldmath$\theta$})$ and is thus unable
to cope with new test classes in the few-shot classification problem. As we
explained in Section II, there are several works which use neural networks to
tackle few-shot classification. Prototypical Networks [30] measure the
distance between an example $x$ and the prototype of each class and assign a
higher probability $p(y|\mbox{\boldmath$x$},\mbox{\boldmath$\theta$})$ to a
closer class to $x$. We adopt this model, which despite its simplicity, shows
high accuracy in experiments. They also enable easy comparison with a self-
training based method[23].

For optimizing model parameters $\theta$ from $\mathcal{X}^{\rm L}$, we employ
Stochastic Gradient Descend (SGD) for efficiency [30]. To form the prototype
of each class, a sample set $\mathcal{X}_{l}^{\mathcal{S}}$ is randomly chosen
from the labeled training set for each example $\mbox{\boldmath$x$}_{l}$. It
is common to form the sample set by mimicking the support set, i.e., the
sample set consists of $CK$ examples belonging to $C$ classes, in which the
number of examples for each class is $K$. The pair of the test example
$\mbox{\boldmath$x$}_{l}$ and the corresponding sample set
$\mathcal{X}^{\mathcal{S}}_{l}$, which corresponds to a minibatch in SGD, is
called an episode [30].

We define a prototype in the latent space obtained by the inverse
transformation
$\mbox{\boldmath$z$}_{l}=f^{-1}_{\mbox{\scriptsize\boldmath$\theta$}}(\mbox{\boldmath$x$}_{l})$
of Normalizing Flow. For each example $\mbox{\boldmath$x$}_{l}$, the prototype
$\mbox{\boldmath$t$}^{c}_{l}$ of class $c$ is defined as the mean of the
examples in class $c$ in the latent variable set
$\mathcal{Z}_{l}^{\mathcal{S}}$ of the sample set as follows.

$\displaystyle\mbox{\boldmath$t$}_{l}^{c}=\frac{1}{|\mathcal{Z}_{l}^{\mathcal{S}_{c}}|}\sum_{\mbox{\boldmath$z$}_{i}\in\mathcal{Z}_{l}^{\mathcal{S}_{c}}}\mbox{\boldmath$z$}_{i}.$
(5)

The probability that example $\mbox{\boldmath$x$}_{l}$ belongs to class $c$ is
defined with a softmax function which takes the distance to each class
prototype as the input.

$\displaystyle
p(y_{l}=c|\mbox{\boldmath$z$}_{l},\mathcal{Z}_{l}^{\mathcal{S}})=\frac{\exp(-\|\mbox{\boldmath$z$}_{l}-\mbox{\boldmath$t$}_{l}^{c}\|^{2}_{2})}{\sum_{c^{\prime}}\exp(-\|\mbox{\boldmath$z$}_{l}-\mbox{\boldmath$t$}_{l}^{c^{\prime}}\|^{2}_{2})}.$
(6)

Figure 2: Graphical model of the proposed method. The lower dashed rectangle
is the marginal likelihood $p(\mathcal{X}^{\rm L},\mathcal{X}^{\rm
U}|\mbox{\boldmath$\theta$})$ explained in Section IV-B. The upper dashed
rectangle is the discriminative model $p(\mathcal{Y}|\mathcal{X}^{\rm
L},\mbox{\boldmath$\theta$})$ explained in Section IV-C. The labeled example
$\mbox{\boldmath$x$}_{l}$ and its label $y_{l}$, the sample set
$\mathcal{X}_{l}^{\mathcal{S}}$, and the unlabeled example
$\mbox{\boldmath$x$}_{u}$ are half-toned as observed variables. The latent
variable $z$ takes the roles of both image generation and label estimation.
Since parameters $\theta$ are shared between the labeled and unlabeled data,
the unlabeled data contribute to the estimation of the latent variables for
the labeled data.

###  IV-D Hybrid Model

By introducing the marginal density and discriminative models and taking
logarithm to Eq. (3), we obtain the following log composite likelihood
function that our hybrid model maximizes:

$\displaystyle{\rm log}\mathcal{L}=\sum_{l=1}^{L}\log
p(y_{l}|\mbox{\boldmath$z$}_{l},\mathcal{Z}_{l}^{\mathcal{S}})+\beta\sum_{n=1}^{L+U}\log
p(\mbox{\boldmath$x$}_{n}|\mbox{\boldmath$\theta$}).$ (7)

The first term represents the log-likehoood of the discriminative model based
on Eq. (6), while the second term represents that of the marginal density
based on Eq. (4).

Figure 2 shows our graphical model, which integrates graphical models of
Normalizing Flow [10] and Prototypical Networks [30] for our hybrid model.
First, note that labeled example $\mbox{\boldmath$x$}_{l}$ is generated by
latent variables $\mbox{\boldmath$z$}_{l}$, which is realized by the
Normalizing Flow neural network with parameters $\theta$. Unlabeled example
$\mbox{\boldmath$x$}_{u}$ is generated similarly. From these facts we can
expect that the shared parameters $\theta$ are learnt such that they express
the essential characteristics of $\mbox{\boldmath$x$}_{u}$ and concurrently
$\theta$ are used to estimate $\mbox{\boldmath$z$}_{l}$ from labeled data
$\mbox{\boldmath$x$}_{l}$. Our advantage over Ren et al.’s self-training
method [23] lies in the fact that the above process is conducted with no
assumption on the classes of the unlabeled training examples.

Second, note that the process of estimating class $y_{l}$ of
$\mbox{\boldmath$z$}_{l}$ from the relationship between latent variables
$\mbox{\boldmath$z$}_{l}$ and set $\mathcal{Z}_{l}^{S}$ of the latent
variables in the support set. Together with the fact that labeled example
$\mbox{\boldmath$x$}_{l}$ is generated from $\mbox{\boldmath$z$}_{l}$, we can
expect that $\mbox{\boldmath$z$}_{l}$ takes the trade-off of the
discriminative characteristics for estimating $y_{l}$ and the generative
characteristics for generating $\mbox{\boldmath$x$}_{l}$ [20]444The generative
learning is called informative learning in [26]. and $\beta$ balances the two
kinds of characteristics. Note that Eq. (7) shows that our method degenerates
to Prototypical Networks [30] of Eq. (6) when $\beta=0$. When $\beta=1$, it
degenerates to a pure generative model (or the joint model).

###  IV-E Optimization

Since the optimization of $\beta$ should be based on the minimization
principle of the generalization loss, we could use WAIC [34], which
approximates the generalization loss, or a minimization by the leave-one-out
cross validation. However, WAIC requires sampling from the posterior
distribution of the parameters, which is unknown for Normalizing Flow. Leave-
one-out cross validation is notoriously slow when the training data is large.
Thus in the experiments we optimize $\beta$ based on the generalization loss
on a validation set, which is kept apart from the training and test sets.

Parameters $\theta$ are updated based on the backpropagation algorithm. For a
faster learning with less memory consumption, we update the parameters with
SGD, in which the log-likelihood function in Eq. (7) is iteratively maximized
in terms of the minibatch. For the term of the discriminative model, we form
the sample set, i.e., the labeled data in the mini-batch, by $CK$ examples
from the labeled training data as explained in Section IV-C. We firstly sample
$C$ classes from the training classes and then selected $K$ examples randomly
for each class. For the marginal density term, we sample unlabeled training
data randomly, which reflects that their class labels are unknown. The ratio
between the labeled and unlabeled examples in a minibatch is identical to that
in the whole training data so that the objective function on the minibatch
well approximates that on the entire training data.

##  V Experiments

In this Section, we evaluate our hybrid model on two datasets, i.e., mini-
ImageNet [33] and VGG-Face [22]. We first explain the details of our
implementation in Section V-A and then the datasets and the experimental
settings in Section V-B. To validate our modeling, which has no assumption on
the classes of the unlabeled examples, we evaluate their predictive
distributions by changing the number of the unlabeled training examples.
Section V-C evaluates them when none of the unlabeled training examples belong
to the labeled classes. Section V-D evaluates them when some of the unlabeled
training examples belong to labeled classes. Moreover, we measure the
influence of weight $\beta$ for the generative model in Section V-E.

###  V-A Implementation Details

Network architecture. We adopted a Glow [10] architecture, a kind of a
Normalizing Flow architecture, with $4$-blocks of $4$-affine coupling layers.
We resized each input image using Lanczos function555The size of resized image
is $80*80*3$ and $64*64*3$ pixels, respectively, for the mini-ImageNet and the
VGG-Face datasets. and normalized each RGB value so that their means and their
variances become $0$ and $1$, respectively. We used the normalized image as
the input of our Glow architecture. We introduced convolutional layers because
their effectiveness in the Glow architecture on image data has been confirmed
by experiments [10]. We had to use a shallower structure than the original
implementation ($3$-blocks of $32$-affine coupling layers) [10] due to our
memory constraint. Adopting a public implementation [18], we constructed an
affine coupling layer by a $3$-layer Highway network with $216$ hidden units.

Baseline method. We adopted a self-training based method Prototypical Networks
with soft K-means (PNK) [23] due to its simplicity. PNK adopts two strategies
in disregarding unlabeled training examples which are estimated not to belong
to the training classes. One strategy is to classify them into a distractor
cluster and the other one is to decrease their influence by masking. Since
masking exhibits better performance in most of the experiments by Ren et al.
[23], we adopt it in our baseline method. We modified the network structure of
PNK to the Glow architecture with $4$-blocks of $4$-affine coupling layers,
which our method adopts. Though our baseline method is not identical to the
corresponding one in [23], this modification allows us to evaluate the
differences of the two objective functions. Note that we adopted only PNK with
masking as the baseline method due to the high cost of re-implementing other
methods with Glow.

Parameter setting. We used ADAM[9] with the the learning rate $5*10^{-4}$, and
hyperparramers $\beta_{1}=0.9$, and $\beta_{2}=0.999$. We set the weight decay
to $5*10^{-5}$. These four hyper parameters were fixed before the execution
and were not tuned with the validation set.

We conducted a grid search on the validation set to adopt the pair of the
weight $\beta\in\\{10^{-4},10^{-5},...,10^{-9}\\}$ and the number of epochs
$\in\\{1,2,...,80\\}$ that achieves the highest evaluation measures. We
conduct the parameter search for each of the different settings of the
training samples. Similarly, we determined the number of epochs from the same
range for the baseline method.

###  V-B Datasets and Settings

mini-ImageNet [33]. The dataset is an edited version of ILSVRC-12 dataset
[27]. It consists of $60,000$ images belonging to $100$ classes, each of which
has $600$ images. We followed the protocol in [23] and assigned $64$ training
classes, $16$ validation classes, and $20$ test classes. Among the 64 training
classes, $50$ classes were selected as labeled classes $\mathcal{C}_{\rm L}$
and $14$ classes were hidden to generate the unlabeled training examples. For
the labeled training examples $\mathcal{X}^{\rm L}$, we selected $40\%$ of
examples, i.e., $240$ examples, per each labeled class.

VGG-Face [22]. The dataset consists of facial images of 2,622 persons, each of
whom corresponds to a class. Among the persons, 2,353 persons contain no less
than 100 images, and we used them in the experiments. We randomly selected 100
images for each person. The $2,353$ classes were divided into $1800$ training
classes, $200$ validation classes, and $353$ test classes. In the training
classes, $300$ classes were selected as labeled classes $\mathcal{C}_{L}$ and
$1500$ classes were hidden to generate the unlabeled training examples. As the
labeled training examples $\mathcal{X}^{\rm L}$, we selected $10\%$ of
examples, i.e., $10$ examples, per each labeled class.

Note that we change the setup of unlabeled examples $\mathcal{X}^{\rm U}$ and
classes $\mathcal{C}_{\rm U}$ in the following experiments in Sections V-C,
V-D, and V-E, while the labeled training examples $\mathcal{X}^{\rm L}$ are
common for all experiments.

Evaluation. To form the support set in the test set in $C$-way $K$-shot
classification, for each test example $x$, we randomly selected $C-1$ classes
from the test classes, and then randomly selected $K$ examples for each
class666In the process we did not include the test example $x$.. We kept the
condition $C=5$ and $K=5$ throughout the experiments.

We adopted as the evaluation measure the test accuracy and the negative cross
entropy (the higher, the better) between the estimated class distribution and
the true class labels 777The negative cross-entropy is also measured on the
test examples. . Note that the accuracy, despite its clear meaning, is only
based on the mode of the estimated class distribution. In contrast, the
negative cross entropy directly measures the estimated class distribution. We
believe that the latter measure is also valuable because it is more
informative and thus gives a basis for further detailed analyses.

###  V-C Case When the Unlabeled and Labeled Training Classes are Disjoint

First, we evaluate our method when the labeled and unlabeled classes are
disjoint from each other, i.e., $\mathcal{C}_{\rm U}\cap\mathcal{C}_{\rm
L}=\emptyset$. In this case, the assumption on the unlabeled examples of the
baseline method [23] is not satisfied. We validate our hypothesis that our
method improves predictive distributions even in such a case.

We construct unlabeled training examples $\mathcal{X}^{\rm U}$ from the
training examples which do not belong to labeled classes. As the unlabeled
classes $\mathcal{C}_{\rm U}$, we used $\\{0,2,6,10,14\\}$ classes, each of
which having $600$ examples for the mini-ImageNet dataset. For the VGG-Face
dataset, we used $\\{0,300,900,1500\\}$ classes, each of which having $10$
examples.

(a) mini-ImageNet (left: accuracy, right: negative cross entropy)

(b) VGG-Face (left: accuracy, right: negative cross entropy)

Figure 3: Predictive performance with increasing unlabeled classes when
$\mathcal{C}_{\rm U}\cap\mathcal{C}_{\rm L}=\emptyset$

####  V-C1 Results on mini-ImageNet

Figure 3 (a) shows the results on the mini-ImageNet dataset. In the Figure,
the red and black curves represent our proposed method and the baseline method
[23], respectively. The x-axis of each plot represents the number $|C_{U}|$ of
the unlabeled classes, and the left endpoint represents the case with no
unlabeled training example, i.e., supervised learning. We see that the
accuracies of the two methods are almost the same at the left endpoint. With
an increasing number of classes of the unlabeled training examples, the
accuracy and the negative cross entropy of the baseline method tend to
decrease. In contrast, those of our method are stable or slightly increase.
These results support our hypothesis that the performance of our method
improves or is at least stable by increasing number of the unlabeled training
examples even when their underlying classes are different from the labeled
training classes. The reason would be that our proposed method models the
latent variable of the unlabeled training examples with no assumption to their
classes (Fig. 2).

####  V-C2 Results on VGG-Face

Figure 3 (b) shows the results on the VGG-Face dataset. We see that the
accuracies are almost the same for different numbers of the unlabeled training
classes for both methods. The reason would lie in the fact that the training
and the test classes are highly similar, and thus the required number of the
unlabeled examples is smaller than in the mini-ImageNet dataset. From the left
endpoint of Fig. 3 (b-right), we see that the negative cross entropies of the
two methods are the same when there is no unlabeled training example. These
results are not surprising because $\beta$ was set to 0 by the parameter
setting on the validation set in our method, and as a result, both methods
became pure Prototypical Networks. Nevertheless, with the increasing number of
classes of the unlabeled training data, the negative cross entropy of the
baseline method deteriorates, whereas that of our method improves. These
results again support our hypothesis as in the mini-ImageNet dataset.

###  V-D Case When the Unlabeled and Labeled Training Classes Overlap

The experimental condition in Section V-C are in some sense unrealistic
because the classes of the unlabeled and labelled training examples are
disjoint. As a more realistic setting, we evaluate our method in the case when
the classes of the unlabeled and labeled training examples overlap, i.e.,
$\mathcal{C}_{U}\cap\mathcal{C}_{L}\neq\emptyset$. We verify our hypothesis
that the performance of our method improves or is at least stable in such a
case.

For the mini-ImageNet dataset, as the unlabeled training set $\mathcal{X}^{\rm
U}$, we first used the remaining $60\%$ of examples, i.e., 360 examples, from
each of the $50$ labeled classes, i.e., $\mathcal{C}_{\rm U}=\mathcal{C}_{\rm
L}$. Then we added $\\{0,2,6,10,14\\}$ classes which are not included in
$\mathcal{C}_{\rm L}$. We used $600$ examples per added class. Similarly, for
the VGG-Face dataset, as unlabeled training examples $\mathcal{X}^{\rm U}$, we
first used the remaining $20\%$ of the examples, i.e., 20 examples, from each
of the $300$ labeled classes. Then we added $\\{0,300,900,1500\\}$ classes
which are not included in the labeled classes. We used $10$ examples per added
class.

(a) mini-ImageNet (left: accuracy, right: negative cross entropy)

(b) VGG-Face (left: accuracy, right: negative cross entropy)

Figure 4: Predictive performance with increasing unlabeled classes when
$\mathcal{C}_{\rm U}\cap\mathcal{C}_{\rm L}\neq\emptyset$

####  V-D1 Results on mini-ImageNet

Figure 4 (a) shows the results on the mini-ImageNet dataset. The left endpoint
represents the case when the classes of the unlabeled examples and those of
the labeled examples are identical. Though this case satisfies the assumption
of the baseline method [23], i.e., $\mathcal{C}_{\rm U}=\mathcal{C}_{\rm L}$,
the proposed method exhibits slightly higher accuracy ($3.8\%$ higher) and
negative cross entropy than the baseline method. These results show the
superiority of our method even when the assumption of the baseline method [23]
holds. In increasing number of the unlabeled classes which is not included in
the labeled classes, the accuracy of our method slightly increases when more
than $6$ classes are added, and the cross entropy steadily increases. These
results again support our hypothesis, as in the previous section.

####  V-D2 Results on VGG-Face

Figure 4 (b) shows the results on the VGG-Face dataset. We see that the
accuracy and the negative cross entropy of the two methods are similar. Note
that at the left end, when all unlabeled examples belong to the labeled
training classes, i.e., $\mathcal{C}_{\rm U}=\mathcal{C}_{\rm L}$, the
baseline method slightly outperforms our method. These results are not
surprising because the case satisfies the assumption of the self-training of
the baseline method [23]. However, by adding examples of which underlying
classes are not included in the labeled class $\mathcal{C}_{\rm L}$ to
unlabeled training examples, the performance of the baseline method in both
evaluation measures degrades, whereas that of our method improves. These
results again support our hypothesis as in the mini-ImageNet dataset and the
experiments in the previous section.

###  V-E Influence of Weight $\beta$ of the Marginal Likelihood

We investigate the influence of weight $\beta$ of the marginal likelihood.
Theoretically, the discriminative model delivers lower generalization loss
with an infinite number of labeled examples, whereas a smaller number of the
labeled examples reinforces the tendency that the generative model achieves
lower generalization loss [20]. We confirm the advantage of our hybrid model,
which takes an intermediate model of a pure discriminative model ($\beta=0$)
and a pure generative model ($\beta=1$). Note that the $\beta=0$ corresponds
to pure Prototypical Networks (implemented with Normalizing Flow), and the
performance gain from $\beta=0$ shows the contribution of the marginal
density.

For this analysis, we used the mini-ImageNet dataset. For the labeled examples
and classes $\mathcal{C}_{\rm L}$, we used the common setting described in
Section V-B. For the unlabeled training set, we used all remaining examples,
i.e., 60% examples in a labeled class and all examples in an unlabeled class.

Figure 5: Predictive performance in terms of weight $\beta$ on marginal
likelihood $p(\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$. The cases of a
pure discriminative model ($\beta=0$, the upper dashed line) and that of a
pure generative model ($\beta=1$, the lower dashed line) are shown. Note that
the intermediate models between the dashed lines show better accuracies.

Figure 5 left shows the predictive accuracy in the test set in terms of weight
$\beta$ on marginal likelihood
$p(\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$. The cases of a pure
discriminative model ($\beta=0$) and that of a pure generative model
($\beta=1$) are shown with dashed lines. We see that the intermediate models
between the dashed lines show better accuracies and the negative cross
entropies. These results confirm the analysis by Liang et al. that the best
model that exhibits the lowest generalization loss exists between a
discriminative model and a generative model unless there is an infinite number
of training examples or the data is generated from marginal density
$p(\mbox{\boldmath$x$}|\mbox{\boldmath$\theta$})$ [16].

##  VI Conclusions

We have proposed a new method which combines Prototypical Networks and
Normalizing Flow in the framework of composite likelihood for the semi-
supervised few-shot classification problem. Experiments showed that by adding
more unlabeled examples, the proposed method improves predictive distributions
of the test classes even when there is no class overlap between the unlabeled
and the labeled training examples. These facts show the effectiveness of our
modeling which does not assume the classes of the unlabeled training examples
are the same with the classes of the labeled examples. Moreover, our
experiments also show that the semi-supervised classification model also
satisfies the conjecture that a hybrid model of discriminative and generative
models exhibits better predictive performance when the labeled training
examples are scarce [20, 16].

## References

  * [1] L. Dinh, D. Krueger, and Y. Bengio.  Nice: Non-Linear Independent Components Estimation.  arXiv preprint arXiv:1410.8516, 2014.
  * [2] L. Dinh, J. Sohl-Dickstein, and S. Bengio.  Density Estimation Using Real NVP.  In ICLR, 2016.
  * [3] L. Fei-Fei, R. Fergus, and P. Perona.  One-Shot Learning of Object Categories.  TPAMI, 28(4):594–611, 2006.
  * [4] C. Finn, P. Abbeel, and S. Levine.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.  In ICML, 2017.
  * [5] C. Finn, P. Abbeel, and S. Levine.  Probabilistic Model-Agnostic Meta-Learning.  In NeurIPS, 2018.
  * [6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative Adversarial Nets.  In NeurIPS, 2014.
  * [7] A. Graves.  Generating Sequences with Recurrent Neural Networks.  arXiv preprint arXiv:1308.0850, 2013.
  * [8] T. Karaletsos, S. Belongie, and G. Rätsch.  Bayesian Representation Learning with Oracle Constraints.  In ICLR, 2016.
  * [9] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  In ICLR, 2014.
  * [10] D. P. Kingma and P. Dhariwal.  Glow: Generative Flow with Invertible 1x1 Convolutions.  In NeurIPS, 2018.
  * [11] D. P. Kingma and M. Welling.  Auto-Encoding Variational Bayes.  In ICLR, 2014.
  * [12] V. Kuleshov and S. Ermon.  Deep Hybrid Models: Bridging Discriminative and Generative Approaches.  In UAI, 2017.
  * [13] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.  Human-Level Concept Learning through Probabilistic Program Induction.  Science, 350(6266):1332–1338, 2015.
  * [14] J. A. Lasserre, C. M. Bishop, and T. P. Minka.  Principled Hybrids of Generative and Discriminative Models.  In CVPR, 2006.
  * [15] X. Li, Q. Sun, Y. Liu, Q. Zhou, S. Zheng, T.-S. Chua, and B. Schiele.  Learning to Self-Train for Semi-Supervised Few-Shot Classification.  In NeurIPS, 2019.
  * [16] P. Liang and M. I. Jordan.  An Asymptotic Analysis of Generative, Discriminative, and Pseudolikelihood Estimators.  In ICML, 2008.
  * [17] A. McCallum, C. Pal, G. Druck, and X. Wang.  Multi-Conditional Learning: Generative/Discriminative Training for Clustering and Classification.  In AAAI, 2006.
  * [18] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan.  Do Deep Genrative Models Know What They Don’t Know.  In ICLR, 2019.
  * [19] E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, and B. Lakshminarayanan.  Hybrid Models with Deep and Invertible Features.  In ICML, 2019.
  * [20] A. Y. Ng and M. I. Jordan.  On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes.  In NeurIPS, 2002.
  * [21] G. Papamakarios, E. Nalisnick, D. Jimenez Rezende, S. Mohamed, and B. Lakshminarayanan.  Normalizing Flows for Probabilistic Modeling and Inference.  arXiv preprint arXiv:1912.02762, 2019.
  * [22] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep Face Recognition.  In BMVC, 2015.
  * [23] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel.  Meta-Learning for Semi-Supervised Few-Shot Classification.  In ICLR, 2018.
  * [24] D. J. Rezende, S. Mohamed, and D. Wierstra.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models.  In ICML, 2014.
  * [25] C. Rosenberg, M. Hebert, and H. Schneiderman.  Semi-Supervised Self-Training of Object Detection Models.  In WACV/MOTION, pages 29–36, 2005.
  * [26] Y. D. Rubinstein and T. Hastie.  Discriminative vs Informative Learning.  In KDD, 1997.
  * [27] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.C. Berg, and L. Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  IJCV, 115(3):211–252, 2015.
  * [28] R. Salakhutdinov and G. Hinton.  Deep Boltzmann Machines.  In AISTATS, 2009.
  * [29] R. Shu, H. H. Bui, and M. Ghavamzadeh.  Bottleneck Conditional Density Estimation.  In ICLR, 2017.
  * [30] J. Snell, K. Swersky, and R. Zemel.  Prototypical Networks for Few-Shot Learning.  In NeurIPS, 2017.
  * [31] S. Tulyakov, A. Fitzgibbon, and S. Nowozin.  Hybrid VAE: Improving Deep Generative Models Using Partial Observations.  In NeurIPS, 2017.
  * [32] V. Vapnik.  Statistical Learning Theory.  Wiley-Interscience, 1998.
  * [33] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al.  Matching Networks for One Shot Learning.  In NeurIPS, 2016.
  * [34] S. Watanabe.  Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory.  JMLR, 11:3571–3594, 2010.
  * [35] D. Yarowsky.  Unsupervised Word Sense Disambiguation Rivaling Supervised Methods.  In ACL, 1995.