11institutetext: OTH Regensburg, Prüfeninger Str. 58, 93049 Regensburg,
Germany 11email<EMAIL_ADDRESS>
# Non-Convergence and Limit Cycles in the Adam optimizer††thanks: This paper
presents results of the project ”LeaP – Learning Poses” supported by the
Bavarian Ministry of Science and Art under Kap. 15 49 TG 78.

Sebastian Bock 11 Martin Weiß 11

###### Abstract

One of the most popular training algorithms for deep neural networks is the
Adaptive Moment Estimation (Adam) introduced by Kingma and Ba. Despite its
success in many applications there is no satisfactory convergence analysis:
only local convergence can be shown for batch mode under some restrictions on
the hyperparameters, counterexamples exist for incremental mode. Recent
results show that for simple quadratic objective functions limit cycles of
period 2 exist in batch mode, but only for atypical hyperparameters, and only
for the algorithm without bias correction. We extend the convergence analysis
for Adam in the batch mode with bias correction and show that even for
quadratic objective functions as the simplest case of convex functions
2-limit-cycles exist, for all choices of the hyperparameters. We analyze the
stability of these limit cycles and relate our analysis to other results where
approximate convergence was shown, but under the additional assumption of
bounded gradients which does not apply to quadratic functions. The
investigation heavily relies on the use of computer algebra due to the
complexity of the equations.

###### Keywords:

Adam optimizer convergence computer algebra dynamical system limit cycle

##  1 Introduction

Adaptive Moment Estimation (Adam), originally presented by Kingma and Ba [7]
is probably the most widely used training algorithm for neural networks,
especially convolutional neural networks. Implementations exist in all popular
machine learning frameworks like Tensorflow or PyTorch. Despite its apparent
success the theoretical basis is weak: The original proof in [7] is wrong as
has been noted by several authors, see [1, 10]. Of course a faulty proof does
not imply that the Adam optimizer does not converge, and indeed local
convergence can be shown for batch mode under reasonable restrictions on the
hyperparameters, see [2]. Furthermore [9] give an example in incremental mode
where the regret does not converge, neither do the arguments of the objective
function.

Several results exist which show $\varepsilon$-bounds on the gradients,
$\left\lVert\nabla f(w_{t})\right\rVert<\varepsilon$ for $\varepsilon>0$
arbitrarily small for all $t$ is sufficiently large, see [5, Theorem 3.4.],
[3], [11, Theorem 3.3]. Other results show asymptotic bounds on the regret or
that the function values come close to the minimum,
$f(w_{t})-f(w_{\star})<\varepsilon$, see [3] for example. To the best of the
authors’ knowledge, [2] is the only (partial) result on weight convergence in
the standard mathematical definition $\lim_{t\to\infty}w_{t}=w_{\star}$,
however only in a local sense.

Contrary to these results [4] shows that 2-cycles exist for the Adam optimizer
without bias correction for the simple case of a scalar quadratic objective
function $f(w)=\frac{1}{2}w^{2}$. Quadratic objective functions are a natural
benchmark for any optimization algorithm in convex analysis: The standard
gradient descent algorithm converges for learning rate small enough, see [8],
so this behaviour should be replicated by more sophisticated gradient
motivated adaptive algorithms like Adam. However [4, Proposition 3.3] only
deals with the case $\beta_{1}=0$ which means that the first moments are not
adapted at all – this case hardly can be called Adam any more.

We extend the results of [4] to the general case of hyperparameters
$\alpha>0$, $0<\beta_{1}<1$, $0<\beta_{2}<1$, and show existence of 2-limit-
cycles for scalar objective functions $f(w)=\frac{1}{2}cw^{2}$, $c>0$, which
easily generalizes by diagonalization to strictly convex quadratic functions
$f(w)=\frac{1}{2}w^{\perp}Cw$, $C$ positive definite. This is done for the
Adam algorithm in batch mode only, but also for bias correction.

We give numerical evidence that for typical values of $\beta_{1},\beta_{2}$
near 1 these 2-cycles are unstable, and stable for $\beta_{1},\beta_{2}$ near
0. The analysis of the limit cycles is not exhaustive: more 2-cycles may
exist, and cycles of larger period. However our results suffice to clarify the
global non-convergence of Adam even for strictly convex functions under the
fairly standard assumptions of bounds on the Hessian like
$0<lI_{n}\leq\nabla^{2}f(w)\leq LI_{n}$ for all $w\in\mathbb{R}^{n}$.

The outline of the paper is as follows: In Section 2 we define our variant of
the Adam algorithm and explain the steps of our proof. In Section 3 these
steps are carried out. The Maple code used can be obtained from the publishers
web site111\- URL missing - We can provide the code for the referees of
course. Section 4 shows numerical simulations suggesting that a Hopf
bifurcation occurs, before we state some conclusions and relate our results to
other research.

Notation: With $\operatorname{Mat}_{n}$ we denote the set of all real
$n$-by-$n$ matrices. The symbol $\perp$ denotes the transpose of a vector or
matrix. The class of $k$-times continuously differentiable functions from
$\mathbb{R}^{n}$ to $\mathbb{R}^{m}$ is denoted by
$C^{k}(\mathbb{R}^{n},\mathbb{R}^{m})$, with $\nabla f$ the gradient and
$\nabla^{2}f$ the Hessian for scalar valued functions. Throughout this paper
we assume $f:\mathbb{R}^{n}\to\mathbb{R}^{n}$ at least $C^{1}$, $C^{2}$ for
some results. In the numerical tests we denote the machine accuracy with
$eps=2.2204*10^{-16}$, i.e. standard IEEE floating point numbers with double
precision.

##  2 Outline of the Proof

###  2.1 Definition of the Algorithm

In the course of time, variations of the Adam optimizer were developed. We use
the version of Adam shown in Algorithm 1 (The symbols $\otimes,\oplus$ and
$\oslash$ denote the component-wise multiplication and division of vectors, as
well as component-wise addition of vectors and scalar.). The main points worth
noting are:

Originally Kingma and Ba [7] use $\sqrt{v}\oplus\varepsilon$ and the bias
correction in $\hat{m}$ and $\hat{v}$. Other publications like [9, 3] do not
use an $\varepsilon$ to avoid division by zero as well as [12], but the latter
initialize $v_{0}=\varepsilon$ with essentially the same effect. We use the
variant with $\varepsilon$ in the denominator; otherwise the initial value
$v_{0}=0$ would have to be excluded in all results, and one could not talk
about stability of a fixed point $w_{\star}$ of the iteration corresponding to
a minimum of the objective function.

Also we use $\sqrt{v\oplus\varepsilon}$ as in [4, 2], instead of
$\sqrt{v}\oplus\varepsilon$ as in the original publication [7]. Our variant
has the advantage that the iteration is continuously differentiable for all
$v\geq 0$ whereas the $\varepsilon$ outside of the square root leads to a non
differentiable exception set. The numerical differences of the two variants
are marginal and described in more detail in [2].

All cited publications with the exception of [4] apply a bias correction in
the learning rate $\alpha_{t}$; we use the same bias correction as described
in [7, Section 2].

Algorithm 1 Adam Optimization

0: $\alpha\in\mathbb{R}^{+}$, $\varepsilon\in\mathbb{R}$
$\beta_{1},\beta_{2}\in(0,1)$, $w_{0}\in\mathbb{R}^{n}$ and the function
$f(w)\in C^{2}\left(\mathbb{R}^{n},\mathbb{R}\right)$

1: $m_{0}=0$

2: $v_{0}=0$

3: $t=0$

4: while $w$ not converged do

5: $m_{t+1}=\beta_{1}m_{t}+(1-\beta_{1})\nabla_{w}f(w_{t})$

6:
$v_{t+1}=\beta_{2}v_{t}+(1-\beta_{2})\nabla_{w}f(w_{t})\otimes\nabla_{w}f(w_{t})$

7:
$w_{t+1}=w_{t}-\alpha\frac{\sqrt{1-\beta_{2}^{t+1}}}{\left(1-\beta_{1}^{t+1}\right)}m_{t+1}\oslash\sqrt{v_{t+1}\oplus\varepsilon}$

8: $t=t+1$

9: end while

We denote $x=(m,v,w)$ the state of the Adam iteration, and interpret the
algorithm as discrete time dynamical system $x_{t+1}=T(t,x_{t};p)$ with
$T=T(t,x;p)=T(t,m,v,w;\alpha,\beta_{1},\beta_{2},\varepsilon)$ to express the
dependence of the iteration on the state $x=(m,v,w)$ and the hyperparameters
$p=(\alpha,\beta_{1},\beta_{2},\varepsilon)$.

We write Adam without bias correction in the same way as
$\bar{T}=\bar{T}(x;p)=\bar{T}(m,v,w;\alpha,\beta_{1},\beta_{2},\varepsilon)$.
This gives an autonomous dynamical system; the right hand side does not
explicitly depend on $t$. The difference between the two systems is denoted by
$\Theta(t,x;p)$, so we have analogous to [2]

$\displaystyle x_{t+1}=T(t,x_{t};p)=\bar{T}(x_{t};p)+\Theta(t,x_{t},p)$ (1)

with

$\displaystyle\bar{T}\left(x_{t};p\right)$
$\displaystyle=\begin{bmatrix}\beta_{1}m_{t}+\left(1-\beta_{1}\right)g\left(w_{t}\right)\\\
\beta_{2}v_{t}+\left(1-\beta_{2}\right)g\left(w_{t}\right)\otimes
g\left(w_{t}\right)\\\
w_{t}-\alpha\left(m_{t+1}\oslash\sqrt{v_{t+1}\oplus\varepsilon}\right)\end{bmatrix}$
(2)

and

$\displaystyle\Theta(t,x_{t};p)$ $\displaystyle=\begin{bmatrix}0\\\ 0\\\
-\alpha\left(\frac{\sqrt{1-\beta_{2}^{t+1}}}{1-\beta_{1}^{t+1}}-1\right)\left(m_{t+1}\oslash\sqrt{v_{t+1}\oplus\varepsilon}\right)\end{bmatrix}$
(3)

###  2.2 Steps of the Proof

We show that for the objective function $f(w)=\frac{1}{2}cw^{2}$ with $c>0$
2-cycles occur for a wide range of hyperparameters in the Adam iteration
without bias correction, and that iterations of the bias corrected algorithm
converge to this limit cycle if it is stable. We proceed in several steps,
analyzing simplified variants of Adam first, then adding complexity in each
step. The analysis uses Maple as a computer algebra system and some continuity
and disturbance arguments because the naive approach of applying the solve
command to find 2-cycles fails – the equations are too complicated.

  1. 1.

We start with the scalar case $f(w)=\frac{1}{2}cw^{2}$, $c>0$. We obtain
analytical expressions for 2-limit-cycles
$\bar{T}^{2}(x;\alpha,\beta_{1},\beta_{2},\varepsilon)=x$ of the autonomous
system with $\varepsilon=0$.

  2. 2.

Calculating the eigenvalues of $\bar{T}^{2}$ we find that these do not depend
on the learning rate $\alpha$ and the factor $c$ in the objective function.
For some typical values of the hyperparameters we give evidence that these
limit cycles are often attractive. We have not managed to give analytical
estimates for stable eigenvalues using CAS so far.

  3. 3.

Using the implicit function theorem we show that for a neighbourhood of
$\varepsilon_{\star}=0$ there exists a unique limit cycle of the autonomous
system with $\varepsilon>0$. By continuity of the eigenvalues, these limit
cycles are also attractive for $\varepsilon$ small enough.

  4. 4.

We apply a disturbance estimate to show that locally solutions of $T(t,x;p)$
converge to the limit cycles of $\bar{T}(x;p)$. The proof is essentially the
same as in [2, Theorem V.1.] and holds for cycles of any integer length.

##  3 Existence of 2-Limit-Cycles in Adam

Step 1: We show that limit cycles of period 2 exist for Adam without bias
correction, i.e. the autonomous system $\bar{T}$. A 2-cycle corresponds to a
non-constant solution of $\bar{T}(\bar{T}(x;p);p))=x$, so we try to solve this
system of equations with Maple. This fails, so we do not use arbitrary
parameters but fix $\varepsilon=0$. Now Maple succeeds and returns

$\tilde{m}=\frac{1}{2}{\frac{c\left({\it\beta_{1}}-1\right)^{2}\alpha}{\left({\it\beta_{1}}+1\right)^{2}}},\tilde{v}=\frac{1}{4}\,{\frac{{\alpha}^{2}\left(\beta_{1}^{2}-2\,{\it\beta_{1}}+1\right){c}^{2}}{\left({\it\beta_{1}}+1\right)^{2}}},\tilde{w}=\frac{1}{2}\,{\frac{\alpha\left({\it\beta_{1}}-1\right)}{{\it\beta_{1}}+1}}$
(4)

with $(-\tilde{m},\tilde{v},-\tilde{w})$ the other point on the 2-cycle. Note
that $\tilde{m}\neq 0$, so we have a 2-cycle indeed. We abbreviate these
points as $\tilde{x}_{1}$ and $\tilde{x}_{2}$. The $v$ components of
$\tilde{x}_{1}$ and $\tilde{x}_{2}$ are identical. This limit 2-cycle exists
for all $\beta_{1}\neq\pm 1$, that is for all reasonable Adam hyperparameters.
Maple also returns more 2-cycles depending on the roots of
$2\beta_{1}\beta_{2}-2\beta_{2}^{2}-2\beta_{1}+2\beta_{2}$, we have not
analyzed these. We could not determine cycles of greater period
$q\in{\mathbb{N}}$ by solving $\bar{T}^{q}(x)=x$.

Step 2: However we can investigate the stability of the limit cycle. This is
done using the Eigenvalues of the Jacobian of $\bar{T}^{2}(\tilde{x}_{1})$,
which are the same as those of $\bar{T}^{2}(\tilde{x}_{2})$. The Jacobian is
computed by Maple as

$\left[\begin{array}[]{ccc}-\left({\it\beta_{1}}+2\right){\it\beta_{1}}&2{\frac{\left({\it\beta_{1}}+1\right){\it\beta_{2}}}{{\it\alpha}{\it
c}}}&{\it
c}\left({\it\beta_{1}}-1\right)\left({\it\beta_{1}}+2{\it\beta_{2}}-1\right)\\\
\vskip 6.0pt plus 2.0pt minus
2.0pt\cr-2\left({\it\beta_{2}}-1\right){\it\alpha}{\it\beta_{1}}{\it
c}&3{{\it\beta_{2}}}^{2}-2{\it\beta_{2}}&2{\frac{{{\it
c}}^{2}\left({\it\beta_{1}}-1\right)\left({\it\beta_{1}}+3/2{\it\beta_{2}}-1/2\right)\left({\it\beta_{2}}-1\right){\it\alpha}}{{\it\beta_{1}}+1}}\\\
\vskip 6.0pt plus 2.0pt minus 2.0pt\cr
2{\frac{{\it\beta_{1}}\left({\it\beta_{1}}+1\right)\left(1-{\it\beta_{1}}-2{\it\beta_{2}}\right)}{{\it
c}\left({\it\beta_{1}}-1\right)}}&2{\frac{\left(2{\it\beta_{1}}+3{\it\beta_{2}}-1\right)\left({\it\beta_{1}}+1\right){\it\beta_{2}}}{{{\it
c}}^{2}{\it\alpha}\left({\it\beta_{1}}-1\right)}}&2{{\it\beta_{1}}}^{2}+\left(8{\it\beta_{2}}-6\right){\it\beta_{1}}+6{{\it\beta_{2}}}^{2}-6{\it\beta_{2}}+1\end{array}\right]$

and not easy to interpret. Using Maple we obtain a very lengthy expression for
the eigenvalues which does not depend on $\alpha$ or $c$. Details can be seen
in the supplementary code. This is surprising as most algorithms show
dependence on the learning rate, and one might assume that the behaviour at a
limit cycle is different at least for $\alpha\to 0$ and $\alpha\to\infty$. So
the stability of the limit cycle depends on $\beta_{1}$ and $\beta_{2}$ only.
If the limit cycle is unstable there are still chances that the Adam algorithm
converges.

Plotting the absolute values of the eigenvalues over $\beta_{1}$ and
$\beta_{2}$ we can see that these limit cycles are often attractive, see
Figure 1. Local attractivity holds if the absolute values are less than 1:
This is the case for the real eigenvalue, see left plot, but the magnitude
approaches 1 as $\beta_{2}$ approaches 1 – consider the standard value
$\beta_{2}=0$ suggested in [7]. The pair of complex conjugate eigenvalues can
be stable as well as unstable – unstable again for $\beta_{1}$ and $\beta_{2}$
near 1. This is good news: For typical $\beta_{1},\beta_{2}$ the limit cycle
will not turn up in numerical simulations as it is unstable. We have not
managed to give analytical estimates for stable eigenvalues using CAS so far.

Figure 1: Absolute magnitude of real and complex eigenvalues

Step 3: Now we show that the limit cycle also exists for $\varepsilon>0$
sufficiently small. We fix the hyperparameters $\alpha,\beta_{1},\beta_{2}$
and consider the function

$F(x,\varepsilon)=\bar{T}(\bar{T}(x;\alpha,\beta_{1},\beta_{2},\varepsilon);\alpha,\beta_{1},\beta_{2},\varepsilon)-x$

Consider a state $\tilde{x}$ on a 2-cycle for $\tilde{\varepsilon}=0$ as in
step 1, then $F(\tilde{x},0)=0$, that is we have a zero of $F$. If
$\frac{\partial F}{\partial x}(\tilde{x},0)$ is invertible, then the Implicit
Function Theorem shows that in a neighbourhood of $\tilde{\varepsilon}=0$
there exists a unique zero $x(\varepsilon)$ with
$F(x(\varepsilon),\varepsilon)=0$. This zero of $F$ corresponds to a 2-cycle
of $\bar{T}$ with hyperparameter $\varepsilon$. (We always have
$\varepsilon>0$ in Adam, but on a non-trivial 2-cycle we have shown that $v>0$
by the explicit term for $\tilde{x}_{1}$ and $\tilde{x}_{2}$ in (4), so even a
small reduction of $\varepsilon$ would be allowed.) Calculating
$\det\left(\frac{\partial F}{\partial x}(\tilde{x},0)\right)$ with Maple we
get

$4(\beta_{1}+\beta_{2})(\beta_{2}-1)(\beta_{1}+1)(2+\beta_{1}^{3}\alpha
c+(3\beta_{2}c\alpha-2\alpha c-2\beta_{2})\beta_{1}^{2}+3(\beta_{2}-1)(\alpha
c-2\beta_{2})\beta_{1})$

Here $\alpha c$ always appears in combination, with the interpretation that an
increase in the gradient of the objective can be compensated by a decrease in
the learning rate. The leading factors are non-zero because
$0<\beta_{1},\beta_{2}<1$, but for any given values of
$\alpha,\beta_{1},\beta_{2}$ one can zero the final factor with

$\hat{c}=\frac{2(\beta_{1}^{2}\beta_{2}+3\beta_{1}\beta_{2}^{2}-3\beta_{1}\beta_{2}-1)}{\alpha\beta_{1}(\beta_{1}^{2}+3\beta_{1}\beta_{2}-2\beta_{1}+3\beta_{2}-3)}$
(5)

assuming a non-zero denominator. So with the exception of
$\hat{c}(\alpha,\beta_{1},\beta_{2})$, the limit cycle exists for small
$\varepsilon$, and by continuous dependence of the eigenvalues on the matrix,
these limit cycles are also attractive.

Step 4: We apply a disturbance estimate to show that
$\bar{T}(x;\alpha,\beta_{1},\beta_{2},\varepsilon)$ and
$T(t,x;\alpha,\beta_{1},\beta_{2},\varepsilon)$ have asymptotically the same
limit cycles. The following theorem is a variation of [2, Theorem V.1.] and
holds for cycles of any integer length. The proof is very similar and omitted
for brevity. The difference between the variants is that here we do not use an
estimate of the type $\left\lVert\Theta(t,\tilde{x})\right\rVert\leq
C\beta^{t}\left\lVert\tilde{x}-x_{\star}\right\rVert$ where $x_{\star}$
appears but rather an exponentially decaying term $C\beta^{t}$. Consequently
we cannot show exponential stability of the 2-limit-cycle but only exponential
convergence of trajectories nearby, with the constant depending on the initial
value.

###### Theorem 3.1

Let $X\subset\mathbb{R}^{n}$ be a closed set, $\left\lVert\cdot\right\rVert$ a
norm on $\mathbb{R}^{n}$. Let $T:\mathbb{N}_{0}\times X\to X$ be a mapping
which is a contraction w.r.t. the second variable uniform in
$t\in\mathbb{N}_{0}$, i.e. there exists $L<1$ with

$\left\lVert T(t,x)-T(t,y)\right\rVert\leq L\left\lVert
x-y\right\rVert\quad\forall x,y\in X,t\in\mathbb{N}_{0}$

Furthermore assume that the difference between $T(t+1,\cdot)$ and $T(t,\cdot)$
is exponentially bounded: There exist $C\geq 0$, $0<\beta<1$ such that

$\left\lVert T(t+1,x)-T(t,x)\right\rVert\leq C\beta^{t}\quad\forall x\in
X,k\in\mathbb{N}_{0}$

Then $T$ has a unique fixed-point $x_{\star}$ in $X$:
$T(t,x_{\star})=x_{\star}$ for all $t\in\mathbb{N}_{0}$. For all $x_{0}\in X$,
the sequence defined by $x_{t+1}=T(t,x_{t})$, $t\in\mathbb{N}_{0}$, converges
to $x_{\star}$ exponentially.

To apply this theorem to the 2-limit-cycle we have to estimate the difference
between two iterations of Adam with and without bias correction:

$\left\lVert T(t+1,T(t,x))-\bar{T}(\bar{T}(x))\right\rVert$

We use the fact from [2] that

$\left\lVert\Theta(t,x)\right\rVert=\left\lVert
T(t,x)-\bar{T}(x)\right\rVert\leq C\beta^{t}\left\lVert
x-x_{\star}\right\rVert$

as well as

$\left\lVert\Theta(t,x)\right\rVert=\left\lVert
T(t,x)-\bar{T}(x)\right\rVert\leq C\beta^{t}\left\lVert x\right\rVert$

Using this we estimate

$\displaystyle\left\lVert T(t+1,T(t,x))-\bar{T}(\bar{T}(x))\right\rVert$
$\displaystyle\leq$ $\displaystyle\left\lVert
T(t+1,T(t,x))-\bar{T}(T(t,x))\right\rVert+$
$\displaystyle\left\lVert\bar{T}(T(t,x))-\bar{T}(\bar{T}(x))\right\rVert$
$\displaystyle\leq$ $\displaystyle C\beta^{t+1}\left\lVert
T(t,x)\right\rVert+L\left\lVert T(t,x)-\bar{T}(x)\right\rVert$

where $L$ is a local Lipschitz constant near the limit cycle. The Lipschitz
continuity exists because $\bar{T}$ continuously differentiable. We continue
the estimate

$\displaystyle\leq$ $\displaystyle C\beta^{t+1}\left\lVert
T(t,x)\right\rVert+LC\beta^{t}\left\lVert x\right\rVert$ $\displaystyle=$
$\displaystyle
C\beta^{t+1}\left\lVert\bar{T}(x)+\Theta(t,x)\right\rVert+LC\beta^{t}\left\lVert
x\right\rVert$ $\displaystyle\leq$ $\displaystyle
C\beta^{t+1}\left\lVert\bar{T}(x)\right\rVert+C^{2}\beta^{t+1}\beta^{t}\left\lVert
x\right\rVert+LC\beta^{t}\left\lVert x\right\rVert$ $\displaystyle\leq$
$\displaystyle\tilde{C}\beta^{t}\max\\{\left\lVert\bar{T}(x)\right\rVert,\left\lVert
x\right\rVert\\}$

with $\tilde{C}\geq\max\\{C^{2},LC\\}$. As we consider only states $x$ near
the limit cycle $\tilde{x}_{1}$, $\tilde{x}_{2}$ we can locally bound the term
$\left\lVert\bar{T}(x)\right\rVert$ by continuity of $\bar{T}$.

We summarize our findings in the following theorem.

###### Theorem 3.2

Consider the Adam-Optimizer as defined in Algorithm 1,
$f(w)=\frac{1}{2}cw^{2}$. Then the algorithm is locally convergent under the
assumptions stated [2]. However if $c\neq\hat{c}(\alpha,\beta_{1},\beta_{2})$
with $\hat{c}(\alpha,\beta_{1},\beta_{2})$ defined in (5), there exist
solutions that converge to the 2-limit-cycles of the algorithm without bias
correction; so the algorithm does not converge globally.

##  4 Numerical Simulations: Discrete Limit cycles

In [4, Proposition 3.3] the authors show the existence of a discrete limit
cycle for the Adam. This discrete limit cycle depends on the learning rate
$\alpha$ and $\beta_{1}=0$. Therefore we demand $0<\beta_{1}<1$ this limit
cycle does not affect the local convergence proof of [2]. But we found in some
numerical experiments few limit cycles which alter the convergence of Adam.

First, we will recall the hyperparameter bounding of [2]

$\displaystyle\frac{\alpha\max_{i=1}^{n}(\mu_{i})}{\sqrt{\epsilon}}\left(1-\beta_{1}\right)<2\beta_{1}+2$
(6)

with $\mu_{i}$ the $i$-th eigenvalue of the Hessian $\nabla^{2}f(w_{\star})$.
This bounding is marked in the Experiments with a red cross and depicts in
every of our Experiments the bifurcation position.

Table 1: Parameter of the different limit cycles | 1\. Experiment | 2\. Experiment | 3\. Experiment
---|---|---|---
$c$ | 10 | 1 | 1
$\alpha$ | 0.001 | 0.5 | 0.8
$\beta_{1}$ | 0.9 | 0.2 | 0.5
$\beta_{2}$ | 0.999 | 0.5 | 0.6
$\varepsilon$ | $10^{-8}$ | $10^{-6}$ | $0.01$
$m_{0}$ | $-1.281144718*10^{-5}$ | 0 | $0$
$v_{0}$ | $5.925207756*10^{-8}$ | 0 | $0$
$w_{0}$ | $2.434174964*10^{-5}$ | $eps$ | $eps$

In the first Experiment, we found a 2-limit-cycle lean on the parameters
suggested by [7]. A plot between the three main values $m_{t},v_{t},w_{t}$
looks like a fountain and we can see, that the values of $m_{t}$ and $w_{t}$
becoming more diffuse by an increasing $v_{t}$ (see Figure 2 left). By looking
closely to $w_{t}$, it attracts attention that $w_{t}$ is reaching the
solution $0$ but leaving it again (see Figure 2 right). Looking at the
eigenvalues of the corresponding Jacobian, it is noticeable that one of them
is greater than $1$. Therefore the solution is not a stable 2-limit-cycle. The
other real 2-limit-cycles are also not stable. Therefore we reach in
Experiment 1 a limit-cycle with a higher order than $2$ (see Figure 3).

Figure 2: Discrete limit cycle with the parameters suggested by [7]
(Experiment 1)

In the following we will only consider $w$ and $\alpha$. By the fact that
$m_{t}$ and $v_{t}$ are only auxiliary variables depending on the history of
$w$, they are less important than $w$ and $\alpha$. In order to clarify this
aspect, reference is made at this point to page $3$ in [7]. There is a
definition for $v_{t}$ without $v_{t-1}$ and thus a definition of $w$ without
$m$ and $v$ is possible. In addition, the following remark gives an insight
into the dependency from $m_{t}$ to the history of $w$ in 2-limit-cycles.

###### Remark 1

We assume a 2-limit-cycle and therefore we can write $m_{t}=m_{t+2}$. With
this knowledge, we can rewrite the $m_{t}$-update rule.

$\displaystyle\begin{pmatrix}-\beta_{1}&1\\\
1&-\beta_{1}\end{pmatrix}\begin{pmatrix}m_{t}\\\
m_{t+1}\end{pmatrix}=\left(1-\beta_{1}\right)\begin{pmatrix}g(w_{t})\\\
g(w_{t+1})\end{pmatrix}$

Defining $\beta_{1}\in(0,1)$ the system is uniquely solvable and thus $m_{t}$
does not have more information for the system than $w$. The same applies to
$v_{t}$.

If we are iterating over $\alpha$ from $10^{-4}$ to $0.01$ and declaring
$\varepsilon=10^{-6}$ we reach figure 3 and see a Hopf bifurcation. With
inequality (6) we can calculate exactly the coordinate of the bifurcation (see
the red cross in figure 3).

$\displaystyle\alpha_{Bifurcation}=\frac{(2\beta_{1}+2)\sqrt{\epsilon}}{\left(1-\beta_{1}\right)\max_{i=1}^{n}\left(\mu_{i}\right)}=0.0038$

Figure 3: Hopf bifurcation (Experiment 1)

In the second Experiment we can see that it is possible that even if we are
starting closely to the solution $w_{0}=eps$ we are ending in a stable cycle
far away.

Figure 4: Discrete limit cycle starting near the solution (Experiment 2)

By iterating over $\alpha$ from $10^{-4}$ to $0.01$ we can see a pitchfork
bifurcation of the Adam. With $\alpha=0.5$ stable, the eigenvalues of the
corresponding Jacobian are:

$\displaystyle\lambda_{1}$
$\displaystyle=0.0113983,\qquad\lambda_{2,3}=-0.7606667\pm
0.5465392\,\text{i}$

In the absolute value all three eigenvalues are lower than $1$ and so we reach
a stable 2-limit-cycle between $w_{1}=0.16666$ and $w_{2}=-0.16666$.

In contrast to the implicit function argument, Experiment 3 uses a
$\varepsilon=0.01>>eps$. In Figure 5 on the left side one can see that
starting at $\alpha=0.6$ Adam convergse to a 2-limit-cycle. At around
$\alpha=0.7$ Adam shows a chaotic behaviour. On the right side one can see the
behaviour of the parameters $m,v$ and $w$ at $\alpha=0.8$. It visualizes the
chaotic behaviour and reminds of the shape of a Lorenz system.

Figure 5: Discrete limit cycle with a large $\varepsilon$ (Experiment 3)

Even if we want to minimize a multidimensional problem we can detect such a
bifurcation. For example, with
$f(w):\mathbb{R}^{2}\rightarrow\mathbb{R},f(w)=w^{\perp}Cw$ and

$C=\begin{pmatrix}1.1184&0.5841\\\
0.5841&3.8816\end{pmatrix}=Q^{\perp}\begin{pmatrix}1&0\\\ 0&4\end{pmatrix}Q$

we obtain Figure 6.

In every of our experiments is the first bifurcation is exactly on the solved
inequality (6) (see the red cross in Figure 3, 4, 5 and 6). This result is
only an empirical proved assumption and an analytical proof would be
desirable.

Figure 6: Bifurcation in a multidimensional optimization

##  5 Conclusion and Discussion

The results can be extended easily by a diagonalization argument and change of
coordinates to strictly convex quadratic functions
$\frac{1}{2}w^{\perp}Cw+b^{\perp}w+a$ with $w\in\mathbb{R}^{n}$. But already
the scalar quadratic function shows that the Adam dynamics cannot be globally
convergent, even for strictly convex objective function. This implies that
there cannot be a global Lyapunov function. Our results seem in contradiction
to [5, 3, 11] where $\varepsilon$-bounded gradients are proven. However this
contradiction can be explained by the assumption $\left\lVert\nabla
f(w)\right\rVert\leq H$ in the cited publications, which we do not use, and
the choice of $\beta_{1}$, $\beta_{2}$ depending on the bound $\varepsilon$.
So there is still hope for a general convergence result under both assumptions

$0<lI_{n}\leq\nabla^{2}f(w)\leq LI_{n},\qquad\left\lVert\nabla
f(w)\right\rVert\leq H\qquad\forall w\in\mathbb{R}^{n}$

Furthermore we have investigated only one limit cycle of period 2. More limit
cycles of larger period might exist, so restrictions on objective function and
hyperparameters that eliminate this 2-limit-cycle might miss other or even
create other limit cycles. The computer algebraic methods used in this paper
seem hopeless even for period 3, so probably completely other methods are
necessary.

Our results also have no direct implication on efforts to prove convergence of
Adam in the incremental mode under additional assumptions. However the lack of
a Lyapunov function suggests that Lyapunov based proofs like [6] for the
stochastic heavy ball algorithm, cannot be transferred to Adam in the
stochastic or incremental setting.

## References

  * [1] Sebastian Bock  “Rotationsermittlung von Bauteilen basierend auf neuronalen Netzen (unpublished)”  Ostbayerische Technische Hochschule Regensburg, 2017
  * [2] Sebastian Bock and Martin G. Weiß  “A Proof of Local Convergence for the Adam Optimizer”, 2019
  * [3] Xiangyi Chen, Sijia Liu, Ruoyu Sun and Mingyi Hong  “On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization”, 2018
  * [4] André Belotto da Silva and Maxime Gazeau  “A general system of differential equations to model first order adaptive algorithms”, 2018  URL: https://arxiv.org/pdf/1810.13108
  * [5] Soham De, Anirbit Mukherjee and Enayat Ullah  “Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration”, 2018  URL: http://arxiv.org/pdf/1807.06766v3
  * [6] Sébastien Gadat, Fabien Panloup and Sofiane Saadane  “Stochastic heavy ball”  In _Electron. J. Statist._ 12.1, 2018, pp. 461–529  DOI: 10.1214/18-EJS1395
  * [7] Diederik P. Kingma and Jimmy Lei Ba  “Adam: A Method for Stochastic Optimization”  In _CoRR_ abs/1412.6980, 2014
  * [8] Jurij E. Nesterov  “Introductory lectures on convex optimization: A basic course” APOP 87, Applied optimization  Boston, Mass.: Kluwer Acad. Publ, 2004  URL: http://www.loc.gov/catdir/enhancements/fy0822/2003061994-d.html
  * [9] J. Reddi, Satyen Kale and Sanjiv Kumar  “On the Convergence of Adam and Beyond”  In _International Conference on Learning Representations_ , 2018  URL: https://openreview.net/forum?id=ryQu7f-RZ
  * [10] David Martínez Rubio  “Convergence Analysis of an Adaptive Method of Gradient Descent”, 2017
  * [11] Ding-Xuan Zhou  “Universality of Deep Convolutional Neural Networks”  In _CoRR_ abs/1805.10769, 2018
  * [12] Fangyu Zou et al.  “A Sufficient Condition for Convergences of Adam and RMSProp”, 2018


# Semi-supervised Contrastive Learning of Musical Representations

###### Abstract

Despite the success of contrastive learning in Music Information Retrieval,
the inherent ambiguity of contrastive self-supervision presents a challenge.
Relying solely on augmentation chains and self-supervised positive sampling
strategies can lead to a pretraining objective that does not capture key
musical information for downstream tasks. We introduce semi-supervised
contrastive learning (SemiSupCon), a simple method for leveraging musically
informed labeled data (supervision signals) in the contrastive learning of
musical representations. Our approach introduces musically relevant
supervision signals into self-supervised contrastive learning by combining
supervised and self-supervised contrastive objectives in a simpler framework
than previous approaches. This framework improves downstream performance and
robustness to audio corruptions on a range of downstream MIR tasks with
moderate amounts of labeled data. Our approach enables shaping the learned
similarity metric through the choice of labeled data that (1) infuses the
representations with musical domain knowledge and (2) improves out-of-domain
performance with minimal general downstream performance loss. We show strong
transfer learning performance on musically related yet not trivially similar
tasks - such as pitch and key estimation. Additionally, our approach shows
performance improvement on automatic tagging over self-supervised approaches
with only 5% of available labels included in pretraining.

##  1 Introduction

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning
structured representations of data without the need for costly and time-
consuming labeling. SSL approaches have achieved competitive performance on
downstream tasks with minimal labeled data in many domains [1, 2, 3, 4, 5, 6,
7, 8]. In the field of Music Information Retrieval (MIR), the complexity of
labeling for many tasks - due to the high technicality and subjectivity -
underscores the importance of such self-supervised methods [8, 9, 10, 5, 11,
12, 13, 14]. Instance-discriminative SSL specifically, such as contrastive
learning, has proven to be effective in learning meaningful representations
for a multitude of downstream tasks [8, 9, 15]. However, major design choices
such as positive mining strategies and augmentations are crucial to downstream
performance [16, 17, 8, 18, 19], and selecting a strategy for a given task
remains a challenge, prompting the reintroduction of supervision within the
SSL framework. In MIR, the key notion of “similarity” in contrastive learning
can derive from a variety of musical attributes. Guiding the model towards a
musically informed similarity metric is an objective that may be achieved by
leveraging supervised labeled data, i.e. _supervision signals_.

In this work, we propose a novel semi-supervised contrastive learning method,
SemiSupCon. Our method leverages both unlabeled and labeled data for
contrastive learning, an extension of Contrastive Learning of Musical
Representations (CLMR) in the music domain [8] and SupCon [20] in Computer
Vision. Our approach differs from previous attempts at combining self-
supervised contrastive learning with an auxiliary supervision signal in that
it is the first to our knowledge to implement a fully-contrastive semi-
supervised learning pipeline. The simple machinery of this method allows for
leveraging new supervision signals beyond labels within the contrastive
objective.

Briefly, the contributions of this work are the following: (1) We propose an
architecturally simple extension of self-supervised and supervised contrastive
learning to the semi-supervised case with the ability to make use of a variety
of supervision signals. (2) We show the ability of our method to shape the
representations according to the support supervision signal used for the
learning task with minimal performance loss on other tasks. (3) We propose a
representation learning framework with low-data regime potential and higher
robustness to data corruption. Our implementation and experiments are made
publicly available at https://github.com/Pliploop/SemiSupCon

##  2 Related work

Self-supervised learning aims to learn representations that capture the
semantic structure of data without labels in order to utilize these
representations on downstream tasks. Among self-supervised learning
approaches, Contrastive Learning teaches a model to identify augmented samples
originating from the same data point amongst distractor negative samples [1,
8]. Beyond its success in neighboring fields, MIR and audio representation
learning have largely benefited from Contrastive Learning approaches [8, 9,
21, 22, 2, 23]. From the implementation of CLMR, several works have expanded
on contrastive learning for music, with competitive results on many downstream
tasks and in multiple modalities [13, 10, 9, 24, 25]. One of the key
challenges of contrastive learning is establishing an effective positive
mining strategy to select positive and negative samples[17, 16, 18]. Previous
studies show that both the positive mining strategy and the augmentation chain
are crucial toward the performance on a given downstream task [16, 17, 19, 18]
\- an inappropriate sampling strategy can lead to treating similar samples as
negatives, to the detriment of downstream performance [26, 27, 28]. In MIR
specifically, even the temporal proximity of two positive segments within an
audio clip is influential on downstream performance depending on the task, as
shown in [18]. Previous works have attempted to design domain-appropriate
strategies for music and audio contrastive learning, including auxiliary
similarity metrics [29, 24, 30, 31], weak supervision [32, 15, 33, 34], as
well as music-specific preprocessing and augmentations [10, 8, 25].

Self-supervision is inherently limited by the ability of the positive mining
strategy to select semantically relevant positives. Some approaches have
attempted to reintroduce supervision signals for positive mining within the
contrastive objective to reduce noise induced by self-supervised pseudolabels.
SupCon [20] introduces supervised contrastive learning, which uses class
labels to mine positives. Other approaches have extended contrastive learning
to the semi-supervised regime by leveraging both labeled and unlabeled data.
However, these approaches often use complex machinery, such as auxiliary
classification modules or multiple losses [29, 35, 36, 37], making them
inflexible and difficult to balance with regard to the supervision signal.
Recently, in MIR, Akama et. al [29] employ contrastive learning as an
auxiliary loss for automatic tagging, with improved results over supervision
alone.

##  3 Methods

###  3.1 Self-Supervised contrastive learning

In the SSL setting for contrastive learning [8, 1], each sample in a
$N$-sample batch is augmented into two views through a stochastic augmentation
chain. Let $B$ be a batch of these augmented views $x_{i}$. Indices $i\in
I=\\{1,2...2N\\}$ represent the index of a data point in the batch (anchor).
$p(i)$ is the index of the augmented data point originating from the same
original sample as the anchor (positive sample). $N(i)$ is the set of
negatives: data points in the augmented batch excluding the anchor and
positives: $N(i)=I\setminus\\{i,p(i)\\}$. Let $z_{i}$ be the embedded
representation of the data point by an encoder $E:x\mapsto
E(x)\in\mathbb{R}^{d_{E}}$ and a projection head $g:E(x)\mapsto
g(E(x))=z_{i}\in\mathbb{R}^{d_{g}}$ into the contrastive latent space. In the
SSL setting, the objective function for the contrastive method is the
normalised temperature-scaled cross-entropy loss [1] between samples $i$ and
$p(i)$ for all pairs in the batch:

$\mathcal{L}_{ssl}^{i}=-\log\frac{\exp(sim(z_{i},z_{p(i)})/\tau)}{\sum\limits_{n\in
N(i)\cup\\{p(i)\\}}\exp(sim(z_{i},z_{n}/\tau))}$ (1)

Where $\tau$ is a temperature hyperparameter, $sim$ is a similarity function -
usually, cosine similarity [8, 1]. For the sake of brevity we notate
$\sigma_{i,j}=\exp(sim(z_{i},z_{j})/\tau)$ in the rest of this work.

###  3.2 Supervised contrastive learning

In the supervised setting [20], The set of supervised positives $P_{s}(i)$ are
now defined by the label information in the set of labels $y_{i}$:
$P_{s}(i)=\\{p\in I|y_{p}=y_{i}\\}\setminus i$. As in [20], the supervised
contrastive loss objective is given by:

$\mathcal{L}_{sl}^{i}=\frac{-1}{|P_{s}(i)|}\sum\limits_{p\in
P_{s}(i)}\log\frac{\sigma_{i,p}}{\sum\limits_{n\in N(i)\cup
P_{s}(i)}\sigma_{i,n}}$ (2)

The contrastive matrix M is constructed by leveraging class information
obtained by mining the labels, i.e. if two samples $x_{i}$ and $x_{j}$ are in
the same category then $\textbf{M}_{i,j}=1$.

###  3.3 Semi-supervised Contrastive Learning

Let $\mathcal{U}$ be a set of unlabeled samples, and $\mathcal{S}^{*}$ be a
set of labeled samples. We sample a proportion $p_{s}$ of the labeled dataset
for training such that $|\mathcal{S}|=p_{s}|\mathcal{S}^{*}|$. Let
$\mathcal{A}=\mathcal{U}\cup\mathcal{S}$ be the set of all data points seen
during training. During training, we use both labeled and unlabeled data
points by sampling batches $B$ comprised of proportions $b_{s}$ (resp.
$1-b_{s}$) of labeled (resp. unlabeled) samples. $P_{s}(i)=\varnothing$ if $i$
is the index of an unlabeled data point. We now define our semi-supervised
contrastive loss, with $P_{A}(i)=P_{s}(i)\cup P_{u}(i)$, where $P_{u}(i)$ is
the set of self-supervised positives ($\\{p(i)\\}$ in Eq. 1).:

$\mathcal{L}_{sem}^{i}=\frac{-1}{|P_{A}(i)|}\sum\limits_{p\in
P_{A}(i)}\log\Bigg{(}\frac{\sigma_{i,p}}{\sum\limits_{n\in N(i)\cup
P_{A}(i)}\sigma_{i,n}}\Bigg{)}$

(3)

With the inclusion of both sets of positives, we generalize to both labeled
and unlabeled data in our representation learning task: Note that if
$\mathcal{U}=\varnothing$ or $\mathcal{S}=\varnothing$, the semi-supervised
contrastive loss reverts back to the fully-supervised loss or the fully self-
supervised loss (as $P_{s}(i)=\varnothing$) respectively. The approach is
shown Figure 1.

Figure 1: Semi-Supervised Contrastive Learning. The sparsely labeled dataset
contains a mix of unlabeled data and labeled data. Given a batch, available
labels (blue and yellow tags) are used to augment M. Unlabeled samples
degenerate back to the self-supervised case. Loss is computed between the
pairwise similarity matrix from the encoded embeddings and the target matrix
using Equation 3

This approach differs from simply adding the supervised and self-supervised
contrastive losses together, as our objective maintains the number of samples
to discriminate against in the self-supervised setting by leveraging labeled
data as negatives for the self-supervised samples.

####  3.3.1 Extension to other supervision signals

The range of supervision signals this method can leverage are limited only by
the ability to construct the target contrastive matrix. In this, SemiSupCon
can make use of support data beyond single label multiclass tasks. To
demonstrate this, we devise two strategies for training on MagnaTagATune [38],
which are studied in Section 5.4. For a multi-label signal, if
$C\in\mathbb{N}$ labels coincide between two samples, we set the the
corresponding index in the target contrastive matrix $\textbf{M}_{i,j}=1$. the
criterion $C$ is a hyperparameter which is studied in Section 5.4. By default
we use $C=1$, i.e., if any labels coincide between two samples they are
considered as positives.

Further, we can construct a target continuous similarity metric factor
$\alpha_{i,j}$ which denotes the degree of “semantic similarity“ between the
samples by weighing the common classes by the total number of labels:

$\alpha_{i,j}=\frac{2C_{i,j}}{(C_{i}+C_{j})}$

$C_{i,j}$ is the number of common classes for $x_{i}$ and $x_{j}$, $C_{i}$ and
$C_{j}$ are the number of classes of $x_{i}$ and $x_{j}$. The similarity term
$\sigma_{i,j}$ is then weighted by $\alpha_{i,j}$ in Eq. 3.

##  4 Experiments and results

###  4.1 Datasets

For our experiments, we use The Free Music Archive (FMA) dataset [39] as a
self-supervised dataset, i.e., we do not use its labels. To match the scale of
the supervised datasets, we elect to use the medium subset, containing 25000
clips of 30 seconds of audio.

We utilize several labeled datasets as support labeled data for training and
for evaluation to demonstrate the cross-domain usefulness of SemiSupCon. For
automatic tagging and most of our experiments, we use MagnaTagATune (MTAT)
[38] as labeled data as a proxy evaluation of general music understanding. We
reproduce the canonical 12:3:1 train-test-validation splits [8]. We use MTG-
Jamendo (all subsets, including the top 50 tags, genre, mood/theme, and
instrument) [40] as another tagging dataset. We use NSynth [41] for pitch and
instrument classificiation of short snippets, and MedleyDB [42, 43] for
instrument classification with longer audio clips than NSyth. We use
Giantsteps [44] as a key classification dataset - as in [45], we use the
original dataset as our training set and the MTG-Giantsteps dataset as our
test set. For genre classification, we use the fault-filtered GTZAN dataset
[46, 47]. We use the VocalSet dataset [48] for two additional tasks: singer
identification and technique classification. Finally, we regress Arousal (A)
and Valence (V) on EmoMusic [49] as a downstream evaluation task only, with
the same train-test split as [45].

###  4.2 Model input, augmentation chain

As in [8, 9], we crop 2.7 second segments of mono 22050kHz audio as input to
the encoders, SampleCNN [50] or TUNe+ [9]. We sample and augment 2 adjacent
non-overlapping segments as positives. The dimensions of the encoders and the
2-layer ReLU-nonlinear projection head are $d_{E}=512$ and $d_{g}=128$. We
implement a stochastic augmentation chain similar to CLMR [8], TUNe [9], and
[10]. In order, we apply (Table 1):

Augmentation | probability | parameter | Min/Max | unit
---|---|---|---|---
Gain | 0.4 | Gain | -15‡ / 5† | dB
Polarity inv. | 0.6 | - | - | -
Colored Noise | 0.6 | Signal/noise ratio | 3‡ / 30‡ | dB
|  | Spectral decay | -2‡ / 2† | dB/octave
Filtering | (One of) |  |  |
Low pass | 0.3 | Cutoff | 0.15‡ / 7‡ | kHz
High pass | 0.3 | Cutoff | 0.2† / 2.4† | kHz
Band pass | 0.3 | center frequency | 0.2 / 4 † | kHz
|  | Bandwidth fraction | 0.5† / 2 | -
Band cut | 0.3 | center frequency | 0.2 / 4† | kHz
|  | Bandwidth fraction | 0.5† / 2 | -
Pitch shifting | 0.6 | transpose | -4‡ / 4† | semitones
Delay | 0.6 | reflection time | 100‡ / 500 | ms
|  | reflections | 1† / 3† | -
|  | attenuation | -6† / -3† | dB/reflection
|  | wet/dry factor | 0.25† / 1 | -

Table 1: Training augmentation chain. Only one amongst the four frequency
filters is applied at once. Ranges denoted with $\dagger$ (resp. $\ddagger$)
are subject to increasing (resp decreasing) in Subsection 5.3

###  4.3 Training and evaluation details

For our baseline models, we adopt a training setup similar to TUNe [9] and
CLMR [8]. Models are trained for 200k steps on semi-supervised batches sampled
from MagnaTagATune as labeled data and FMA-Medium as unlabeled data [38, 39]
using the Pytorch Adam optimiser with a learning rate of $1e^{-4}$. For
ablation and variation studies, we train our models for 50k steps unless
otherwise stated. All models are trained with $\tau=0.1$ with a non-augmented
batch size of 96 on a single RTX A5000 GPU unless otherwise specified. We
report steps instead of epochs to standardise the amount of data seen during
training.

To evaluate pretrained models, we freeze the encoder and discard the
projection head. Frozen representations are fed into a 2-layer ReLU-nonlinear
MLP for probing on downstream tasks. For probing, we use the Adam optimizer
with a learning rate $0.0003$ and an early stopping mechanism conditioned on
validation loss. For automatic tagging tasks, we report area under receiver-
operator curve (AUROC) and mean Average Precision (AP). For classification
tasks, we report top-1 accuracy except for key classification: the metric for
this task is a weighted score taking into account reasonable errors [45] \- We
use the mir_eval [51] implementation for evaluation . For emotion regression
we report $R^{2}$ values between predicted and actual values.

##  5 Results

###  5.1 Automatic tagging with semi-supervised contrastive learning

|  |  | AUROC $\uparrow$ | AP $\uparrow$
---|---|---|---|---
Ours | $b_{s}=p_{s}$ |  | SampleCNN | TUNe+ | $\ddagger$ | $\bigstar$
|  |  | ($\ddagger$) | ($\bigstar$) |  |
Self-Supervised | 0 |  | 88.8 | 88.9 | 41.6 | 41.6
Semi-Supervised | 0.05 |  | 89.4 | 89.4 | 42.5 | 42.1
0.1 |  | 89.5 | 89.4 | 42.2 | 42.2
0.25 |  | 89.5 | 89.4 | 42.5 | 42.8
0.5 |  | 89.7 | 89.5 | 42.9 | 43.3
0.75 |  | 89.9 | 89.8 | 43.3 | 43.5
0.5/1 |  | 89.8 | 89.8 | 43.1 | 43.0
Supervised | 1 |  | 90.3 | 90.1 | 44.3 | 44.6
Literature |  |  |  |  |  |
SampleCNN [8] | - |  | 89.3∗ (88.6† [8]) | 41.2∗ (34.4† [8])
CLMR${}_{\text{FMA}}$ [8] | - |  | 86.6† | 31.2†
TUNe+ [9] | - |  | 89.2† | 36.6†
MERT [5] | - |  | 91.0† | 39.3†

Table 2: Performance on automatic tagging. Results denoted by $\dagger$ are
reported in their original paper. In our experiment, we constrain
$p_{s}=b_{s}$ except for one run where $p_{s}=1,b_{s}=0.5$. We trained our own
end-to-end supervised SampleCNN with the same compute budget as SemiSupCon and
report results with *. Figure 2: Evolution of AUROC and AP on MTAT probing
with proportion of supervised MTAT data used for training.

We train a self-supervised baseline, a supervised contrastive baseline with
and without augmentations, an end-to-end supervised baseline using the
sampleCNN architecture, and five variants of our semi-supervised approach with
different proportions of labeled data ($p_{s}\in[0.01,0.05,0.1,0.2,0.5]$) for
Automatic Tagging on MTAT. MTAT labels augment the contrastive matrix M with
positives in the case of supervised or semi-supervised pretraining. We vary
the in-batch and global proportion of supervised data $b_{s}$ and $p_{s}$
simultaneously. We report results on the same task in the literature in Table
2 for comparable datasets and training scales.

When trained for 200k steps, the supervised contrastive model is competitive
with larger self-supervised approaches. Furthermore, it outperforms both our
implementation and the results claimed in CLMR for self-supervised contrastive
and end-to-end supervised models. Figure 2 shows the influence of
$p_{s}=b_{s}$ on AUROC and AP. As the proportion of supervised data increases,
so does the performance on the downstream evaluation. Including only 5% of
labeled data leads to an increase in performance from 88.8 to 89.4 in AUROC.
For our experiment with $p_{s}=1$ and $b_{s}=0.5$, both architectures perform
worse than $p_{s}=b_{s}=0.75$, as the model has seen 100k steps of supervised
data versus 150k.

Target Dataset | MTAT | Jamendo | NSynth | Giantsteps | GTZAN | VocalSet | MedleyDB | Emo
---|---|---|---|---|---|---|---|---
Subset | 50 | All | 50 | Genre | Mood | Inst. | Pitch | Inst. | Key | Genre | Tech. | Singer | Inst. | A/V
Metrics | AUROC | Acc. | Accw | Acc. |  $R^{2}_{V}$ / $R^{2}_{A}$
Self-Supervised |  |  |  |  |  |  |  |  |  |  |  |  |  |
FMA |  | 88.4 | 86.2 | 80.1 | 83.3 | 74.0 | 71.6 | 36.8 | 51.7 | 13.5 | 65.5 | 53.8 | 71.1 | 56.5 | 46.7/71.5
Semi-Supervised $b_{s}=0.5$ |  |  |  |  |  |  |  |  |  |  |  |  |  |
MTAT | 50 | 89.3 | 86.8 | 80.0 | 83.4 | 73.8 | 73.3 | 34.5 | 46.9 | 11.3 | 65.5 | 53.2 | 70.0 | 67.3 | 44.3/65.9
All | 89.1 | 87.5 | 80.3 | 83.2 | 74.1 | 73.0 | 34.0 | 51.0 | 14.9 | 68.2 | 52.4 | 72.9 | 72.8 | 41.6/76.2
Jamendo | 50 | 88.6 | 86.6 | 81.5 | 83.4 | 74.6 | 72.5 | 33.8 | 50.0 | 14.7 | 74.1 | 52.1 | 71.7 | 62.0 | 50.1/77.9
Genre | 88.6 | 86.3 | 80.5 | 84.0 | 74.6 | 71.5 | 33.4 | 50.2 | 14.6 | 72.8 | 52.0 | 74.6 | 66.3 | 48.2/70.3
Mood | 88.3 | 86.6 | 81.0 | 83.0 | 74.7 | 72.3 | 38.2 | 47.7 | 14.9 | 71.3 | 53.5 | 71.4 | 60.9 | 48.0/73.0
Instrument | 88.4 | 86.3 | 80.8 | 83.1 | 74.0 | 71.6 | 37.2 | 52.5 | 14.9 | 69.3 | 54.5 | 67.9 | 63.0 |  52.4/70.6
NSynth | Pitch† | 88.3 | 86.3 | 79.7 | 82.6 | 73.5 | 72.0 | 79.0 | 48.6 | 20.1 | 65.5 | 56.9 | 75.6 | 64.1 | 37.5/66.6
| Inst. | 88.6 | 85.7 | 79.6 | 82.7 | 73.3 | 71.7 | 26.6 | 59.6 | 16.0 | 67.2 | 57.3 | 72.3 | 66.3 | 40.3/75.0
GiantSteps | Key† | 87.7 | 85.0 | 79.0 | 82.1 | 73.0 | 70.5 | 50.8 | 51.3 | 22.3 | 69.3 | 54.1 | 71.4 | 61.2 | 39.6/63.6
GTZAN | Genre | 88.8 | 86.8 | 80.9 | 83.9 | 74.1 | 71.5 | 38.6 | 46.9 | 16.3 | 74.0 | 53.4 | 71.7 | 66.3 | 28.7/56.4
VocalSet | Technique | 88.7 | 86.7 | 79.6 | 82.5 | 73.3 | 71.0 | 46.0 | 53.5 | 12.1 | 63.5 | 63.0 | 77.8 | 67.3 | 41.5/70.1
Singer | 88.9 | 86.2 | 80.1 | 82.6 | 73.6 | 72.8 | 45.2 | 52.4 | 15.3 | 67.2 | 54.0 | 87.1 | 69.6 | 54.3/74.6
MedleyDB | Instrument | 88.6 | 87.0 | 80.2 | 82.6 | 73.6 | 73.8 | 32.0 | 48.8 | 13.2 | 62.1 | 58.6 | 74.3 | 62.0 | 41.6/74.8
SOTA |  | 92.7 | 95.4 | 84.3 | 88.0 | 78.6 | 78.8 | 94.4 | 78.2 | 74.3 | 86.9 | 76.9 | 87.5 | - | 61.7/76.3
| [12] | [34] | [13] | [14] | [13] | [52] | [5] | [53] | [54] | [55] | [5, 45] | [5, 45] |  | [14, 5]
CLMR [45] |  | 89.5 |  | 81.3 | 84.6 | 73.5 | 73.5 | 47.0 | 67.9 | 14.8 | 65.2 | 58.1 | 49.9 | - | 44.4/70.3

Table 3: Results for cross-task evaluation. Models are trained for 50k steps
on FMA [39] as the self-supervised dataset and support supervised datasets
(rows), and evaluated on target datasets (columns). Giantsteps†, NSynth† are
trained without pitch shifting augmentation. Results in bold are the best
results obtained for evaluation on a target dataset. SOTA results are included
for illustration purposes, but do not necessarily leverage comparable
methodologies.

###  5.2 Influence of pretraining labeled dataset

In this experiment, we pre-train multiple semi-supervised models using
datasets described in Section 4.1 as support labeled data and FMA as unlabeled
data - one model per dataset, each for 50000 steps. We then freeze all models
and train shallow MLP probes on all downstream tasks for each model. We train
a self-supervised baseline for comparison. Semi-supervised approaches are
trained with $b_{s}=0.5$ and $p_{s}=1$. Table 3 shows these results.

Semi-supervised training on the target dataset always surpasses the self-
supervised baseline by a significant margin when evaluating on the same
dataset - with minimal loss of performance on other downstream tasks.

Some complementary tasks improve performance on other downstream datasets,
proving semi-supervised contrastive learning a viable transfer learning
strategy. Expectedly, training on genre tagging data increases out-of-domain
performance on genre classification, instrument tagging on instrument
classification, etc. Training on mood data from MTG-Jamendo provides a
performance boost on emotion regression. A notable example is the improvement
in performance on NSynth pitch when training with key data as support labeled
data and vice versa. This demonstrates an improvement in the understanding of
_pitch_ by the model on tasks which are musically related but not trivial
transfer learning instances. Most importantly, this occurs without performance
loss on general music understanding, i.e. automatic tagging. Other musically
grounded examples are pitch pretraining improving instrument classification
performance and instrument pretraining improving emotion regression
performance.

###  5.3 Robustness to in-domain data corruption

Figure 3: Effect of corruption severity on downstream performance. Contrastive
models are more robust than Cross-entropy trained models.

In this section, we evaluate the robustness of our semi-supervised,
supervised, and self-supervised contrastive approaches to audio corruptions
compared to the end-to-end baseline. We train the probing head without
augmentation until convergence and evaluate the model _with_ augmentations
applied. We design different severity degrees of our augmentation chain (See
Subsection 4.2) by applying a modifier to the application probabilities: for
severity $s\in[0,1...4]$, we scale probabilities of application of each
augmentation by $s/2$ such that $s=2$ is the chain applied during training. We
sensibly multiply or divide the min and max values of each augmentation
hyperparameter (see Table 1) by $s/2$. We then evaluate all models with these
augmentation chains on MagnaTagATune. The results are shown in Figure 3.
Contrastive approaches are more robust to in-domain corruption than end-to-end
approaches - hypothetically because we train contrastive models to be
invariant to such transformations through the augmentation chain - which is
not an objective of the end-to-end supervised approach.

###  5.4 Multilabel positive mining strategy

In this experiment, we test multiple label-based positive mining strategies.
First by varying the number of common labels for mining positives - i.e.
$C\in\\{1,2,4,6\\}$. Further, we explore the “semantic weighing” strategy
described in Section 3.3.1, in which the target similarity between two tracks
is weighed by the number of common labels and the total number of labels. We
test these strategies on both semi-supervised and supervised contrastive
models. Results are reported in Table 4.

Positive strategy | Supervised | Semi-Supervised
---|---|---
Class criterion | AUROC | AP | AUROC | AP
$C=1$ | 90.1 | 44.2 | 89.3 | 41.3
$C=2$ | 90.1 | 43.9 | 89.0 | 41.6
$C=4$ | 89.3 | 42.8 | 89.0 | 41.3
$C=6$ | 88.9 | 42.3 | 89.0 | 41.5
Weighing | 90.6 | 45.3 | 88.9 | 41.6

Table 4: Multilabel positive mining strategy as described in Section 3.3.1.

For supervised approaches, the continuous target produced by semantic weighing
produces the best results, on par with 4x training steps with a criterion
$C=1$ (as shown in Table 2). In the supervised case, as the criterion
increases, performance deteriorates. We hypothesise that this could be because
it is an _easier_ task for the model to discern that two tracks with many
common tags are similar (higher $C$), as they likely share many attributes,
therefore providing a weaker training signal. Understanding what links two
tracks from a single tag is more challenging and appears to yield more robust
representations. The continuous “relative similarity” target created by the
weighing strategy is a more nuanced task and appears to be a stronger
supervision signal. This guides the model towards more robust representations,
which explains the higher performance. In the semi-supervised case, we
speculate that the binary self-supervision signal overpowers the continuous
target as a less nuanced objective with harsher penalties for failure. These
penalties could overpower softer penalties from the continuous target in the
loss, preventing optimal convergence. Future work should focus on
understanding and reconciling these aspects of the semi-supervised approach to
leverage other continuous signals.

###  5.5 Qualitative analysis

(a) Latent embeddings of the NSynth-pitch test set from a semi-supervised
model trained on FMA+NSynth-pitch

(b) Latent embeddings of the NSynth-pitch test set from a self-supervised
model trained on FMA

Figure 4: Exploration of the NSynth pitch latent space. Octaves are denoted by
size and pitch class by the color of the dot. Each dot is a full audio sample

The results reported in Section 5.2 show that performance on downstream tasks
improves when labels from a related task are used for model training, with
minimal loss of performance on other tasks. We hypothesise that the internal
latent representations are given structure relative to the supervision signal
while maintaining the semantic structure given by the self-supervision signal.
To illustrate this, we perform t-SNE dimension reduction on embeddings
produced by the semi-supervised model from Table 3 trained with NSynth (Figure
4(a)) as support labeled data and fully self-supervised (Figure 4(b))
evaluated on the test set of NSynth-pitch.

In the set of Figures 4, the latent spaces for the NSynth test set produced by
these two models are shown. When pretrained on NSynth-pitch, the latent space
is highly organized. Separability by class is much clearer than when
pretrained on FMA. We notice that several musical structures emerge in this
latent space. Notably, octaves go from low to high clockwise. Pitches that are
“similar” are close together, i.e., semitones and octaves of the same pitch
class.

##  6 Conclusion and Future work

We presented SemiSupCon, a simple method for leveraging both supervision and
self-supervision signals in contrastive representation learning. By leveraging
reduced amounts of labeled data during pretraining, SemiSupCon outperforms
end-to-end comparable supervised baselines on downstream tasks. We find that
SemiSupCon is more robust to data corruption at inference compared to end-to-
end supervised methods. Additionally, SemiSupCon can utilize various
supervision signals with minimal performance loss on out-of-domain tasks and
achieve performance transfer on similar tasks. While performance gains might
seem moderate on automatic tagging for instance, other downstream tasks show
more distinct improvements. Furthermore, The contrastive objective can lead to
explicitly structured latent spaces with emergent musical structures -
enhancing the musical interpretability of latent spaces by design of the
support supervision signal - i.e. labeling small amounts of data.

Future work will focus on exploring additional supervision signals and tasks
such as perceptual metrics, tempo estimation, and chord estimation. Other
avenues include leveraging the low-data proficiency of SemiSupCon for human-
in-the-loop representation learning. The architecture of SemiSupCon being very
flexible, it can be further adapted to multimodal approaches or hierarchical
representation learning. A more comprehensive exploration of the influence of
the proportion of labeled data and the exact effect of labels and contrastive
matrix sparsity on downstream performance will also be undertaken.

##  7 Acknowledgment

This work is supported by the EPSRC UKRI Centre for Doctoral Training in
Artificial Intelligence and Music (EP/S022694/1) and Universal Music Group.

## References

  * [1] T. Chen, S. Kornblith, M. Norouzi _et al._ , “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning (ICML)_. PMLR, 2020, pp. 1597–1607.
  * [2] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018.
  * [3] J.-B. Grill, F. Strub, F. Altché _et al._ , “Bootstrap your own latent-a new approach to self-supervised learning,” _Advances in neural information processing systems_ , vol. 33, pp. 21 271–21 284, 2020.
  * [4] X. Chen and K. He, “Exploring simple siamese representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)_ , 2021, pp. 15 750–15 758.
  * [5] L. Yizhi, R. Yuan, G. Zhang, Y. Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos _et al._ , “MERT: Acoustic music understanding model with large-scale self-supervised training,” in _The Twelfth International Conference on Learning Representations (ICLR)_ , 2023.
  * [6] Y. Gong, C.-I. Lai, Y.-A. Chung _et al._ , “SSAST: Self-supervised audio spectrogram transformer,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 10, 2022, pp. 10 699–10 709.
  * [7] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [8] J. Spijkervet and J. A. Burgoyne, “Contrastive Learning of Musical Representations,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2021, pp. 673–681.
  * [9] M. A. V. Vásquez and J. A. Burgoyne, “Tailed U-Net: Multi-scale music representation learning.” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022, pp. 67–75.
  * [10] H. Zhao, C. Zhang, B. Zhu _et al._ , “S3T: Self-supervised pre-training with swin transformer for music classification,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 606–610.
  * [11] Y. Li, R. Yuan, G. Zhang, Y. MA, C. Lin, X. Chen, A. Ragni, H. Yin, Z. Hu, H. He _et al._ , “MAP-Music2Vec: A simple and effective baseline for self-supervised music audio representation learning,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022.
  * [12] M. Won, Y.-N. Hung, and D. Le, “A foundation model for music informatics,” in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 1226–1230.
  * [13] M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. Ehmann, “Supervised and unsupervised learning of audio representations for music understanding,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022, pp. 256–263.
  * [14] R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representations for music information retrieval,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2021, pp. 88–96.
  * [15] I. Manco, E. Benetos, E. Quinton, and G. Fazekas, “Contrastive audio-language learning for music,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022, pp. 640–649.
  * [16] T. Xiao, X. Wang, A. A. Efros, and T. Darrell, “What should not be contrastive in contrastive learning,” in _The 9th International Conference on Learning Representations (ICLR)_ , 2021.
  * [17] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What Makes for Good Views for Contrastive Learning?” in _Advances in Neural Information Processing Systems_ , vol. 33. Curran Associates, Inc., 2020, pp. 6827–6839.
  * [18] J. Choi, S. Jang, H. Cho _et al._ , “Towards proper contrastive self-supervised learning strategies for music audio representation,” in _2022 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2022, pp. 1–6.
  * [19] M. C. McCallum, M. E. Davies, F. Henkel, J. Kim, and S. E. Sandberg, “On the effect of data-augmentation on local embedding properties in the contrastive learning of music audio representations,” _arXiv preprint arXiv:2401.08889_ , 2024.
  * [20] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” _Advances in neural information processing systems_ , vol. 33, pp. 18 661–18 673, 2020.
  * [21] H. Al-Tahan and Y. Mohsenzadeh, “CLAR: Contrastive learning of auditory representations,” in _International Conference on Artificial Intelligence and Statistics_. PMLR, 2021, pp. 2530–2538.
  * [22] Y.-A. Chung, Y. Zhang, W. Han _et al._ , “W2V-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2021, pp. 244–250.
  * [23] E. Fonseca, D. Ortego, K. McGuinness _et al._ , “Unsupervised contrastive learning of sound event representations,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 371–375.
  * [24] D. Yao, Z. Zhao, S. Zhang _et al._ , “Contrastive Learning with Positive-Negative Frame Mask for Music Representation,” in _Proceedings of the ACM Web Conference 2022_ , Apr. 2022, pp. 2906–2915.
  * [25] C. Garoufis, A. Zlatintsi, and P. Maragos, “Multi-Source Contrastive Learning from Musical Audio,” no. arXiv:2302.07077. arXiv, May 2023.
  * [26] H. Guo and L. Shi, “Ultimate Negative Sampling for Contrastive Learning,” in _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2023, pp. 1–5.
  * [27] S. Ge, S. Mishra, C.-L. Li, H. Wang, and D. Jacobs, “Robust Contrastive Learning Using Negative Samples with Diminished Semantics,” in _Advances in Neural Information Processing Systems_ , vol. 34. Curran Associates, Inc., 2021, pp. 27 356–27 368.
  * [28] T. Huynh, S. Kornblith, M. R. Walter, M. Maire, and M. Khademi, “Boosting Contrastive Self-Supervised Learning with False Negative Cancellation,” in _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. Waikoloa, HI, USA: IEEE, Jan. 2022, pp. 986–996.
  * [29] T. Akama, H. Kitano, K. Takematsu _et al._ , “Auxiliary self-supervision to metric learning for music similarity-based retrieval and auto-tagging,” _PLOS ONE_ , vol. 18, no. 11, p. e0294643, Nov. 2023.
  * [30] P. Manocha, Z. Jin, R. Zhang _et al._ , “CDPAM: Contrastive Learning for Perceptual Audio Similarity,” in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2021, pp. 196–200.
  * [31] P. Alonso-Jiménez, X. Favory, H. Foroughmand, G. Bourdalas, X. Serra, T. Lidy, and D. Bogdanov, “Pre-training strategies using contrastive learning and playlist information for music classification and similarity,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [32] X. Favory, K. Drossos, T. Virtanen, and X. Serra, “Coala: Co-aligned autoencoders for learning semantically enriched audio representations,” in _Self-supervision in Audio and Speech Workshop, International Conference on Machine Learning (ICML)_ , 2020.
  * [33] A. Ferraro, X. Favory, K. Drossos _et al._ , “Enriched Music Representations with Multiple Cross-modal Contrastive Learning,” _IEEE Signal Processing Letters_ , vol. 28, pp. 733–737, 2021.
  * [34] Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y. Li, and D. P. Ellis, “Mulan: A joint embedding of music audio and natural language,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022.
  * [35] B. Kim, J. Choo, Y.-D. Kwon, S. Joe, S. Min, and Y. Gwon, “Selfmatch: Combining contrastive self-supervision and consistency for semi-supervised learning,” in _34th conference on Neural Information Processing Systems (NEURIPS) Workshop_ , 2021.
  * [36] Y. Zhang, X. Zhang, J. Li, R. Qiu, H. Xu, and Q. Tian, “Semi-supervised contrastive learning with similarity co-calibration,” _IEEE Transactions on Multimedia_ , 2022.
  * [37] F. Yang, K. Wu, S. Zhang, G. Jiang, Y. Liu, F. Zheng, W. Zhang, C. Wang, and L. Zeng, “Class-aware contrastive semi-supervised learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)_ , 2022, pp. 14 421–14 430.
  * [38] E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2009, pp. 387–392.
  * [39] M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “FMA: A dataset for music analysis,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2017, pp. 316–323.
  * [40] D. Bogdanov, M. Won, P. Tovstogan _et al._ , “The mtg-jamendo dataset for automatic music tagging,” in _Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML)_ , 2019.
  * [41] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with wavenet autoencoders,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 1068–1077.
  * [42] R. M. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. P. Bello, “Medleydb: A multitrack dataset for annotation-intensive mir research.” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2014, pp. 155–160.
  * [43] R. M. Bittner, J. Wilkins, H. Yip, and J. P. Bello, “Medleydb 2.0: New data and a system for sustainable data collection,” _ISMIR Late Breaking and Demo Papers_ , p. 36, 2016.
  * [44] P. Knees, Á. Faraldo Pérez, H. Boyer, R. Vogl, S. Böck, F. Hörschläger, M. Le Goff _et al._ , “Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2015, pp. 364–70.
  * [45] R. Yuan, Y. Ma, Y. Li, G. Zhang, X. Chen, H. Yin, Y. Liu, J. Huang, Z. Tian, B. Deng _et al._ , “Marble: Music audio representation benchmark for universal evaluation,” _Advances in Neural Information Processing Systems_ , vol. 36, 2024.
  * [46] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” _IEEE Transactions on speech and audio processing_ , vol. 10, no. 5, pp. 293–302, 2002.
  * [47] B. L. Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,” _arXiv preprint arXiv:1306.1461_ , 2013.
  * [48] J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “Vocalset: A singing voice dataset.” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2018, pp. 468–474.
  * [49] M. Soleymani, M. N. Caro, E. M. Schmidt, C.-Y. Sha, and Y.-H. Yang, “1000 songs for emotional analysis of music,” in _Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia_ , 2013, pp. 1–6.
  * [50] J. Lee, J. Park, K. L. Kim, and J. Nam, “Samplecnn: End-to-end deep convolutional neural networks using very small filters for music classification,” _Applied Sciences_ , vol. 8, no. 1, p. 150, 2018.
  * [51] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “Mir_eval: A transparent implementation of common mir metrics.” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2014, p. 2014.
  * [52] P. Alonso-Jiménez, X. Serra, and D. Bogdanov, “Music representation learning based on editorial metadata from discogs,” in _Proc. of International Society for Music Information Retrieval (ISMIR)_ , 2022, pp. 825–33.
  * [53] L. Wang, P. Luc, Y. Wu _et al._ , “Towards Learning Universal Audio Representations,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2022, pp. 4593–4597.
  * [54] F. Korzeniowski and G. Widmer, “End-to-end musical key estimation using a convolutional neural network,” in _2017 25th European Signal Processing Conference (EUSIPCO)_. IEEE, 2017, pp. 966–970.
  * [55] P. Alonso-Jiménez, L. Pepino, R. Batlle-Roca, P. Zinemanas, D. Bogdanov, X. Serra, and M. Rocamora, “Leveraging pre-trained autoencoders for interpretable prototype learning of music audio,” _arXiv preprint arXiv:2402.09318_ , 2024.

Appendix for the paper "Semi-Supervised Contrastive Learning of Musical
Representations - preprint only

##  8 Appendix

###  8.1 Training details for probing experiments

In Section 5.2, we evaluate representations learned through semi-supervised
contrastive learning on a variety of datasets through shallow probing with MLP
probes. To do so, we Freeze the contrastive backbone and train an MLP for the
specified downstream task. Global training details are reported Section 5.2.
Here we go into probe architectures in more detail. We vary the number of
hidden layers, dropout and weight decay for each dataset to account for
overfitting. The values of hyperparameters were determined empirically through
prelimiary runs. Probe architectures for different datasets as well as details
on dataset scale are reported Table 5:

Dataset Details |  | Probe details
---|---|---
Dataset | Split | #classes | #samples |  | Probe layers output dims | Dropout | Weight Decay
|  |  | Train | Test | Validation | Total |  |  |  |
FMA |  | N/A | N/A | N/A | N/A | 25000 |  | N/A | N/A | N/A
MTAT | All | 288 | 18709 | 5329 | 1825 | 25863 |  | 512, 288 | 0.1 | $1e-6$
| Top50 | 50 | 15250 | 4332 | 1529 | 21111 |  | 512, 50 | 0.1 | $1e-6$
Jamendo | Top50 | 50 | 32136 | 11356 | 10888 | 54380 |  | 512,50 | 0.1 | $1e-6$
| Mood | 56 | 9949 | 4231 | 3802 | 17982 |  | 512, 56 | 0.1 | $1e-6$
| Genre | 87 | 32572 | 11479 | 11043 | 55094 |  | 512, 87 | 0.1 | $1e-6$
| Instr. | 40 | 14395 | 5115 | 5466 | 24976 |  | 512, 40 | 0.1 | $1e-6$
Nsynth | Pitch | 112 | 289205 | 4096 | 12678 | 305979 |  | 512, 112 | 0 | $1e-6$
| Instr. | 24 | 289205 | 4096 | 12678 | 305979 |  | 512, 24 | 0.1 | $1e-6$
Giansteps |  | 24 | 961 | 604 | 198 | 1763 |  | 24 | 0.2 | $1e-5$
GTZAN |  | 10 | 443 | 290 | 197 | 930 |  | 512, 10 | 0.1 | $1e-6$
VocalSet | Singer | 20 | 2467 | 577 | 566 | 3610 |  | 20 | 0.2 | $1e-6$
| Technique | 17 | 2282 | 894 | 403 | 3576 |  | 17 | 0.2 | $1e-6$
MedleyDB |  | 20 | 659 | 137 | 117 | 913 |  | 20 | 0.2 | $1e-6$
EmoMusic |  | 2* | 504 | 125 | 115 | 744 |  | 2 | 0.1 | $1e-5$

Table 5: Dataset and probe details for fine-tuning experiments. For Emomusic
(*) 2 classes are reported but are in fact regression targets (A/V)

###  8.2 Additional qualitative analysis

Here, we provide additional visualizations for qualitative analysis of the
embeddings obtained through cross-dataset training. The embeddings are
obtained from a model trained semi-supervisedly with FMA as the self-
supervised dataset and Nsynth-pitch as the supervised dataset. Embeddings are
obtained with the frozen model for the MagnaTagATune test set and t-SNE is
applied to obtain visualizations similar to 5.5.

(a) Latent embeddings of the NSynth-pitch test set from a semi-supervised
model trained on FMA+NSynth-pitch

(b) Latent embeddings of the NSynth-pitch test set from a self-supervised
model trained on FMA

Figure 5: Exploration of the NSynth pitch latent space. Octaves are denoted by
size and pitch class by the color of the dot. Each dot is a full audio sample

Where training with Nsynth as support labeled data greatly enhanced the
structure of pitch-related classes within the embedding space, we find that
the structure of other datasets (in this case, automatic tagging) is not lost,
and we even observe the same cluster structure, with some small details. This
corroborates our findings in Section 5.2 that performance can be greatly
improved on a dataset of interest without losing general performance and
understanding on other tasks through semi-supervised contrastive learning, a
key benefit of our method.

###  8.3 Additional ablation studies

####  8.3.1 Individual variation of $p_{s}$ and $b_{s}$

To investigate the influence of $p_{s}$ and $b_{s}$ on their own, we design
two experiments : In one, $p_{s}$ the proportion of supervised data is fixed
at $p_{s}=1$ and $b_{s}\in[0.05,0.1,0.25,0.5,0.75]$ is varied. In another,
$b_{s}$ is fixed at $b_{s}=0.5$ (exactly half of each batch is supervised
data) and $p_{s}\in[0.05,0.1,0.25,0.5,0.75]$ is varied. We use the SampleCNN
architecture for all our experiments. All models are pretrained with MTAT as
the support labeled data as well as the downstream task and FMA as the self-
supervised dataset. To better relate the in-batch proportion of supervised
data to a general property of the semi-supervised contrastive learning task,
we define the semi-supervised and supervised contrastive matrix sparsities
$s_{smsl}$ and $s_{sl}$:

$s_{smsl}=\frac{1}{|I|}\sum\limits_{i}|P_{A}(i)|$

and

$s_{sl}=\frac{1}{b_{s}|I|}\sum\limits_{i}|P_{s}(i)|$

Which represent proportion of positives in the contrastive matrix. as this
attribute of the system depends on many hyperparameters such as the task, the
positive sampling strategy, the in-batch supervised proportion, and the number
of augmentations, we report it in Table 6.

$p_{s}$ | $b_{s}$ |  $s_{sl}$ (%) |  $s_{smsl}$ (%) | AUROC | AP
---|---|---|---|---|---
1 | 0.05 | 87.5 | 99.5 | 88.4 | 40.6
1 | 0.1 | 77.1 | 99.3 | 88.4 | 40.2
1 | 0.25 | 61.5 | 97.2 | 88.2 | 40.4
1 | 0.5 | 74.7 | 93.4 | 89.0 | 41.4
1 | 0.75 | 77.3 | 87.1 | 89.3 | 42.0
|  |  |  |  |
$p_{s}$ | $b_{s}$ |  $s_{sl}$ (%) |  $s_{smsl}$ (%) | AUROC | AP
0.05 | 0.5 | 69.7 | 92.2 | 88.4 | 40.4
0.1 | 0.5 | 73.3 | 93.1 | 88.2 | 40.2
0.25 | 0.5 | 74.4 | 93.3 | 89.0 | 41.0
0.5 | 0.5 | 65.5 | 91.2 | 88.4 | 40.6
0.75 | 0.5 | 77.1 | 94.0 | 88.6 | 40.9

Table 6: Individual variation of $p_{s}$ and $b_{s}$: varying $b_{s}$ leads to
a clear improvement, while $p_{s}$ seems to lead to more fluctuations in
performance.

We find that varying $b_{s}$ leads to a consistent increase in performance as
$b_{s}$ increases, while the same does not hold true for $p_{s}$. This is
reasonable : When increasing $b_{s}$ for the same amount of labeled data and
training steps, the amount of labeled data the model sees during training
increases, thus reinforcing the influence of labeled data and bringing the
model closer to a fully-supervised contrastive setup. For varying $p_{s}$
however, it is not clear as to why the performance does not increase
consistently. Perhaps the noisiness of the data here is a factor that explains
why different subsets of labeled data (potentially with different
distributions to the test set) seen during training would lead to fluctuations
in downstream performance. This warrants further exploration to understand why
increasing the diversity of labeled data does not consistently improve
performance.

####  8.3.2 Influence of number of augmentations for same batch size

Because the inclusion of more positives into the training pipeline through
explicit labeling seems to improve the performance of the model significantly,
a possible intuition following this observation is that including more
positive samples in the contrastive matrix could lead to similar improvements.
For this study, we kept the global batch size constant at 192 and vary the
number of augmentations and the batch size without augmentations. We report
results on MTAT for fully supervised, self-supervised, and for $p_{s}=1$ and
$b_{s}=0.5$ for training.

|  |  |  | SSL | SL | Semi-SL
---|---|---|---|---|---|---
$M$ | $B$ | $s_{sl}$ | $s_{smsl}$ | AUROC | AP | AUROC | AP | AUROC | AP
2 | 96 | 74.7 | 93.5 | 88.4 | 40.1 | 90.1 | 44.1 | 89.3 | 41.0
4 | 48 | 69.4 | 88.3 | 88.5 | 40.6 | 90.4 | 44.8 | 89.2 | 42.0
8 | 24 | 60.1 | 82.7 | 88.6 | 40.3 | 90.5 | 44.9 | 89.6 | 41.9
16 | 12 | 51.0 | 76.1 | 88.1 | 39.5 | 89.3 | 41.9 | 88.3 | 40.2
32 | 6 | 40.1 | 73.1.4 | 87.4 | 38.4 | 88.9 | 40.7 | 87.4 | 38.4

Table 7: Influence of the number of augmentations $M$ on downstream MTAT
performance for SSL, SL, and SMSL models

Here, it is not fully clear how the number of augmentations improve
performance. However, there seems to be a sweet spot for a given batch size of
moderate number of augmentations. In this case, 8 augmentations for a batch
size of 24 unaugmented samples seems to be benefecial to performance across
the board for AUROC. AP benefits from 4-8 augmentations more than other
numbers of augmentations. Evidently this result is dependent on the batch size
as well as the augmentation strategy. In our case, we find that a moderate
amount of augmentation-sampled positives yields stronger representations,
contrary to the well-known a priori that only the number of negatives for each
sample in contrastive learning is an important factor for downstream
performance.

###  8.4 Additional adversarial robustness studies

.

We also test the robustness of contrastive and cross-entropy models when
subjected to novel corruption transformations at inference time. We implement
new augmentations with the torch-audiomentations library and the Spotify
pedalboard library, Including Chorus, Distortion, SpliceOut, Reverb,
TimeStretch, Compression, and Bitcrush.

Figure 6: Effect of novel corruptions on downstream AUROC performence -
relative performance variation when compared to uncorrupted baselines.

In most cases, except the notable exception of SpliceOut and Chorus,
contrastive approaches are more robust to out-of-domain corruption when
compared to cross-entropy approaches.