#  Banyan: Improved Representation Learning with Explicit Structure

Mattia Opper a N. Siddharth a,b
a University of Edinburgh; b The Alan Turing Institute
<EMAIL_ADDRESS>
###### Abstract

We present Banyan, an improved model to learn semantic representations by
inducing explicit structure over data. In contrast to prior approaches using
structure spanning single sentences, Banyan learns by resolving multiple
constituent structures into a shared one explicitly incorporating _global_
context. Combined with an improved message-passing scheme inspired by Griffin
De et al. (2024), Banyan learns significantly better representations, avoids
spurious false negatives with contrastive learning, and drastically improves
memory efficiency in such explicit-structured models. Using the Self-StrAE
Opper et al. (2023) framework, we show that Banyan (a) outperforms baselines
using sentential structure across various settings (b) matches or outperforms
unstructured baselines like GloVe ( + augmentations) and a RoBERTa medium ( +
SimCSE) pre-trained on 100M tokens, despite having just a _handful_ of (non-
embedding) parameters, and (c) also learns effective representations across
several low resource (Asian and African) languages as measured on SemRel
Ousidhoum et al. (2024b) tasks.

Banyan: Improved Representation Learning with Explicit Structure

Mattia Opper a and N. Siddharth a,b a University of Edinburgh; b The Alan
Turing Institute<EMAIL_ADDRESS>

##  1 Introduction

The principle of composition, states that in order to understand the meaning
of a whole, one must understand the meaning of its parts and the structure
that dictates how the parts combine. It is the bedrock of many linguistic and
cognitive theories of how humans are able to understand efficiently learn to
understand language and semantics more generally Chomsky (1956); Crain and
Nakayama (1987); Pallier et al. (2011); de Marneffe et al. (2006). Generally,
most NLP models only consider composition implicitly, i.e., as something
internal to the way representations are processed by the model. However, there
has been a longstanding line of work that has sought to create models that
operate on and learn explicit compositional structure. In general, the results
show that explicit structure is an efficient inductive bias. In particular, Hu
et al. (2021, 2022) and Sartran et al. (2022) demonstrate that learned
hierarchies lead to improved perplexity and superior single sentence GLUE Wang
et al. (2019) task performance over baseline transformer models Vaswani et al.
(2017). In a similar vein, Opper et al. (2023) show that an inductive bias for
hierarchy can allow the learning of effective multi-level semantic
representations even when pre-training occurs with limited data and parameter
scale. It is this application with which we concern ourselves.

The model that Opper et al. (2023) introduce is known as Self-StrAE (Self-
Structuring AutoEncoder). Self-StrAE is a recursive neural network Socher et
al. (2011, 2013), that learns both representations and structure. Though a
very simple model (see 2), Self-StrAE is surprisingly effective. This is due
to the power of the inductive bias behind it: letting representations define
their own compositional hierarchies. However, we argue that the formulation of
Opper et al. (2023) does not fully exploit the inductive bias. To remedy this,
we introduce Banyan, a revised formulation of the Self-StrAE with radically
improved performance. Our contributions are as follows:

  1. 1.

We introduce a novel form of structure: entangled trees which leads to
improved representations, and is significantly more compute efficient.

  2. 2.

We take inspiration from the recent resurgence in SSMs to replace the message
passing functions of the Self-StrAE further improving performance and reducing
compute cost.

  3. 3.

We test the resulting framework, which we call Banyan, across multiple
languages, including in the low resource setting, and demonstrate that
together with improved results, we are able to retain the data and parameter
efficiency of the Self-StrAE.

##  2 Preliminaries

Given our central theme involves the use of explicit structure, we begin with
a brief discussion of the Self-StrAE framework Opper et al. (2023). This gives
us a modelling basis which, given a sentence, can jointly induce both a
hierarchical binary-tree structure as well as concomitant multi-level
embeddings over the tree.

Self-StrAE involves three main components that acts over a sentence
$\mathbf{w}=\langle w_{n}\rangle_{n=1}^{N}$ represented as a sequence of
tokens. These are:  (a) a procedure to determine which tokens to merge and in
what order,  (b) message passing functions—composition and decomposition—that
merge and split embeddings respectively, and  (c) a reconstructive objective
that leverages both the induced structure and embeddings.  While we refer the
reader to Opper et al. (2023, §2 & Appendix B) for a detailed description of
these components, we briefly recap its operation to provide sufficient
background for the development of Banyan (§ 3).

At a high-level, Self-StrAE functions by first _embedding_ tokens to form an
initial frontier using an embedding matrix $\Omega_{\Psi}$. It then
interleaves a greedy pairwise-clustering mechanism using the cosine similarity
along with a parametric _composition function_ $C_{\Phi}$ to recursively merge
tokens upwards until it reduces to a single root embedding. Viewing the
resulting merge sequence as the induced binary tree for the sentence, it then
traverses back down this tree structure, recursively splitting embeddings at
every node using a _decomposition function_ $D_{\Theta}$ to recover embeddings
for the leaves. Now, the model can optionally use a dembedding function
$\Lambda_{\Gamma}$ to predict tokens $\widehat{w}_{n}$ from these leaf
embeddings. LABEL:fig:selfstrae illustrates the operation of the model on an
exemplar sentence.

Note that the model generates _two_ sets of embeddings; one set going up from
leaves to root, and another coming back down from root to leaves and denote
these $\bar{e}$ and $\underline{e}$ respectively We also note that an
embedding is typically viewed as
$e\in\operatorname{{}\mathbb{R}}^{U\times{}K}$ with $K$ independent
channels—of particular relevance to the composition and decomposition
functions which act independently over the channels. Tokens are denoted as the
vertices $w_{i}\in\Delta^{V}$ in a $V$-simplex for vocabulary size $V$. All
together, the functioning of the model is then characterised by:

$\displaystyle\Omega_{\Psi}(w_{i})$
$\displaystyle=w_{i}\,\Psi,\quad\Psi\in\operatorname{{}\mathbb{R}}^{V\times(U\star{}K)}$
(1) $\displaystyle C_{\Phi}(\bar{e}_{i},\bar{e}_{i+1})$
$\displaystyle=\textsc{hcat}(\bar{e}_{i},\bar{e}_{i+1})\,\Phi+\phi$ (2)
$\displaystyle\quad\Phi\in\operatorname{{}\mathbb{R}}^{2U\times
U},\phi\in\operatorname{{}\mathbb{R}}^{U}$ $\displaystyle
D_{\Theta}(\underline{e}_{i})$
$\displaystyle=\textsc{hsplit}(\underline{e}_{i}\,\Theta+\theta)$ (3)
$\displaystyle\quad\Theta\in\operatorname{{}\mathbb{R}}^{U\times
2U},\theta\in\operatorname{{}\mathbb{R}}^{2U}$
$\displaystyle\Lambda_{\Gamma}(\underline{e}_{i})$
$\displaystyle=\underline{e}_{i}\,\Gamma\quad\Gamma\in\operatorname{{}\mathbb{R}}^{(U\star{}K)\times
V}$ (4)

Given the nature of the model, a straightforward objective would be to simply
reconstruct the tokens, formulated for sentence $\mathbf{w}$ and prediction
$\widehat{\mathbf{w}}$ as

$\displaystyle\mathcal{L}_{\text{CE}}(\mathbf{w},\widehat{\mathbf{w}})=-\frac{1}{N}\sum_{n=1}^{N}w_{n}\cdot\log\widehat{w}_{n}.$
(5)

An alternate approach developed by Opper et al. (2023) leverages the multi-
level structure of the model to define a contrastive objective over a batch of
sentences $\\{\mathbf{w}_{b}\\}_{b=1}^{B}$ with a total of $M$ nodes (internal
+ leaves). Noting that the up and down trees share the same underlying
structure (modulo reversed edges), this objective draws together corresponding
up and down embeddings at a given tree position, whilst pushing away other
embeddings across the batch, using the cosine similarity metric. Denoting the
pairwise similarity matrix $A\in\operatorname{{}\mathbb{R}}^{M\times M}$
between upward embeddings $\langle\bar{e}_{i}\rangle_{i=1}^{M}$ and downward
embeddings $\langle\underline{e}_{i}\rangle_{i=1}^{M}$, and
$A_{i\bullet},A_{\bullet{}j},A_{ij}$ the $i\textsuperscript{th}$ row,
$j\textsuperscript{th}$ column, and $(i,j)\textsuperscript{th}$ entry of the
matrix respectively, the objective is defined as:

$\displaystyle\mathcal{L}_{\text{CO}}(\bar{e},\underline{e})=\frac{-1}{2M}\sum_{i=1}^{M}\log\left(\sigma_{\\!\tau}(A_{i\bullet})\,\sigma_{\\!\tau}(A_{\bullet{}i})\right)$
(6)

with tempered softmax $\sigma_{\\!\tau}(\cdot)$ (temperature $\tau$)
normalising over the unspecified ($\bullet$) dimension.

##  3 Model

A particularly interesting characteristic of learning with explicitly
structured models such as Self-StrAE or even earlier models such as the IORNN
Le and Zuidema (2014) is the dichotomy between the upward and downward
embeddings. Given how these models are constructed, the upward embeddings are
always _non-contextual_ : they do not encapsulate any of the surrounding
context. For example, following LABEL:fig:selfstrae, the upward embedding
$\bar{e}$ for the span ‹ate doughnuts› is always the same regardless of
context, no matter who did the eating. In contrast however, the downward
embeddings are always _contextual_ : they must encapsulate the surrounding
context by virtual of being decomposed from larger spans or units. For our
example, this implies that there are multiple downward embeddings
$\underline{e}^{y}$ for the given span, one for each $y\in\\{\text{Lisa,
Homer, \ldots}\\}$. To learn effective embeddings then, one must
_marginalised_ over these different downward embeddings to ensure that their
meaning resolves over all these contexts.

###  3.1 From trees to entangled trees

Given this need to resolve meaning across multiple contexts, it is obvious to
see for prior models that simply employing a cross-entropy objective over
tokens places no constraints that help achieve resolution. Note however that
Self-StrAE with the contrastive objective $(\mathcal{L}_{\text{CO}})$ provides
an indirect constraint. For a given node (i.e. span), by drawing together its
upward (non-contextual) embedding and _each_ downward (contextual) embedding,
it implicitly draws together all the contextual embeddings for that given
node. On the other hand, by being contrastive, this normally also pushes away
all other embeddings regardless of whether they correspond to the same span
elsewhere in the data, which can typically result in the objective suffering
from the presence of false negatives—terms that ought not to be pushed away,
but are.

Our innovation here is to address both these issues together by formulating
the process in terms of entangled trees—where entangling refers to reduction
of a set of disjoint tree structures into a single conjoined graph structure.
An example of this is shown in LABEL:fig:entangling with disjoint trees on the
left and resulting the entangled tree on the right. Here, as shown, all
instances of ‹night› and ‹some are born to› are captured by a single node
representing that constituent. We call our model Banyan on account of this
entangling.

#### Entangling

Constructing an entangled tree given a set of disjoint trees is a relatively
straightforward process and is formally specified in Algorithm 1. In contrast
to the agglomerative clustering employed in Self-StrAE, here we employ a
global frontier spanning all leaf nodes across the given data. The key
differences to the prior methods are mainly to do with constructing a graph
jointly with progressing the frontier and ensuring that new nodes are never
duplicated, for which we employ a node identity $s_{n}$ in addition to the
node embedding $e_{n}$.

Algorithm 1 Banyan: Entangled Compose

0: Global frontier $\langle(s_{n},e_{n})\rangle_{n=1}^{N}$, compose ($\circ$),
concat $(\diamond)$, similarity cSim$(e,e^{\prime})$

1: $\mathcal{A}\leftarrow\langle(s_{n},e_{n})\rangle_{n=1}^{N}$
$\triangleright$ initialise frontier

2: $(\mathcal{V},\mathcal{E})\leftarrow(\varnothing,\varnothing)$
$\triangleright$ initialise graph

3: while $\exists i:s_{i}\diamond s_{i+1}\not\in_{s}\mathcal{V}$ do

4:
${i^{\star}}\leftarrow\operatorname*{arg\,max}_{i}\textsc{cSim}(e_{i},e_{i+1})$
$\triangleright$ location of closest adjacent pair

5: $e_{p}=\circ(e_{i^{\star}},e_{i^{\star}+1})$ $\triangleright$ compute
composition

6: $\mathcal{V}\leftarrow\mathcal{V}\cup\\{(s_{i^{\star}}\diamond
s_{i^{\star}+1},e_{p})\\}$

7:
$\mathcal{E}\leftarrow\mathcal{E}\cup\\{{p}\sim{i^{\star}},{p}\sim(i^{\star}+1)\\}$
$\triangleright$ update graph

8:
$\mathcal{J}\leftarrow\\{j:(s_{j},s_{j+1})=(s_{i^{\star}},s_{i^{\star}+1})\\}$
$\triangleright$ locations of all occurrences of this pair

9:
$\mathcal{A}\leftarrow\mathcal{A}\setminus\\{\forall_{j\in\mathcal{J}}\;\mathcal{A}_{j},\mathcal{A}_{j+1}\\}$
$\triangleright$ delete occurrences from those locations

10:
$\mathcal{A}\leftarrow\mathcal{A}\cup_{\mathcal{J}}\\{(s_{i^{\star}}\diamond
s_{i^{\star}+1},e_{p})\\}$ $\triangleright$ insert composition into those
locations

11: return Graph $(\mathcal{V},\mathcal{E})$

#### Incorporating context

Following the entangling of trees described, the model proceeds in a similar
vein to Self-StrAE, by composing upwards from leaves to roots (multiple roots
corresponding to multiple trees), and then decomposing downwards back to the
leaves. With entangled trees, while traversing upwards each node is always
composed from the same two children, but on the way back down, things are
different as each separate context for a given node provides a different
downward embedding. This is shown in LABEL:fig:banyan focussing on a subgraph
of the entangled tree from LABEL:fig:entangling(right). Note that the node in
question (in blue) corresponds to the span ‹some are born to›, and has
downward embeddings that incorporate context both from ‹endless night› and
‹sweet delight›. This is exactly as desired, as Banyan allows explicit
aggregation to derive the downward embedding that resolves over the contexts.
For any upward embedding $\bar{e}$ whose span occurs in different contexts
$y\in\mathcal{Y}$, the corresponding downward embedding is derived by simply
averaging over the different contextual down embeddings; i.e.,
$\underline{e}=\frac{1}{\lvert\mathcal{Y}\rvert}\sum_{y}\underline{e}^{y}$.

#### Effectiveness and efficiency

Beyond the ability to explicitly incorporate context across data, entangled
trees also help the contrastive objective by avoiding false negatives since
they do not admit duplicate nodes by construction. Furthermore, the lack of
duplicate nodes also drastically impacts the memory footprint of the model as
one deals with the set of all nodes rather than counting each instance as its
own node. These effects becomes more pronounced when entangling a larger set
of instances as the likelihood of false negative and duplicates goes up
together.

#### Practical estimation

Given the advantages conferred by entangled trees, one would ideally want to
construct it over _all_ the available data. This however is not practically
feasible as the size of data typically grows exponentially with time. To
address this, we construct our model to estimate the given objective by taking
steps over _batches_ of data that are of a more manageable size, noting that
this estimator is unbiased. To see this is the case, note that entangled trees
only affects the downward embeddings directly, and that batching simply means
that the resolved embedding is an average over _samples_ instead of over all
the data (_population_)—the sample mean is always an unbiased estimator of the
population mean.

###  3.2 Simplified Message Passing

Complementary to the development of entangled trees to incorporate context, we
also explore avenues to improve the message passing with the composition $C$
and decomposition $D$ functions. The original formulations of these from Eqs.
2 and 3 employ concatenation and splitting along with simple single-layer
linear neural networks. A key question for such functions is how well they
propagate information (and gradients) through the recursive hierarchical
structure.

Following the recent resurgence in state-space models (SSMs) and in
particular, diagonally parametrised recurrent neural networks (RNNs) Peng et
al. (2023); Orvieto et al. (2023); De et al. (2024) that better handle
information flow through sequence recurrence, we adopt the same for our
functions. Under this new parameterisation, we now define

$\displaystyle C(\bar{e}_{i},\bar{e}_{i+1})$
$\displaystyle\\!=\\!(\bar{e}_{i}\\!\cdot\\!\sigma(\Phi_{l})+\bar{e}_{i+1}\\!\cdot\\!\sigma(\Phi_{r}))+\phi$
(7)
$\displaystyle\quad\Phi_{l},\Phi_{r},\phi\in\operatorname{{}\mathbb{R}}^{U}$
$\displaystyle D(\underline{e}_{i})=\bigl{(}$
$\displaystyle\underline{e}_{i}\\!\cdot\\!\sigma(\Theta_{l})+\theta_{l},\;\;\underline{e}_{i}\\!\cdot\\!\sigma(\Theta_{r})+\theta_{r}\bigr{)}$
(8)
$\displaystyle\quad\Theta_{l},\Theta_{r},\theta_{l},\theta_{r}\in\operatorname{{}\mathbb{R}}^{U}$

Here, $\sigma$ represents the sigmoid non-linearity applied to the parameters
for numerical stability following De et al. (2024).

These relatively simple changes have a pretty drastic effect, both in terms of
performance (see experiments) as well as memory footprint, with parameters now
reduced by a factor of $U$ compared to the functions from Eqs. 2 and 3

##  4 Experimental Setting

Table 1: Banyan compared with Self-StrAE pre-trained on 10 million tokens.
Higher is better. Results represent the average across four random
initialisations. Each of the plus rows represents an additional change to the
model. Banyan and Self-StrAE are differentiated in terms of the structure they
learn, +diag functions means additionally changing the message passing
functions, and ++CE loss means additionally switching the pre-training
objective.

Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score
---|---|---|---|---|---|---|---|---|---
Self-StrAE | 31.98 $\pm$ 0.58 | 53.88 $\pm$ 0.68 | 37.73 $\pm$ 0.70 | 55.23 $\pm$ 0.58 | 55.55 $\pm$ 0.47 | 39.53 $\pm$ 1.61 | 51.78 $\pm$ 0.29 | 50.05 $\pm$ 0.92 | 46.59 $\pm$ 0.43
\+ diag functions | 35.13 $\pm$ 0.33 | 56.05 $\pm$ 0.24 | 40.58 $\pm$ 0.05 | 58.83 $\pm$ 0.10 | 56.78 $\pm$ 0.21 | 44.10 $\pm$ 0.14 | 53.35 $\pm$ 0.17 | 52.65 $\pm$ 0.17 | 49.68 $\pm$ 0.06
++ CE loss | 47.10 $\pm$ 1.04 | 61.85 $\pm$ 1.44 | 58.60 $\pm$ 1.34 | 70.45 $\pm$ 0.57 | 62.45 $\pm$ 0.70 | 59.50 $\pm$ 0.53 | 59.00 $\pm$ 0.26 | 60.33 $\pm$ 0.26 | 59.91 $\pm$ 0.54
Banyan | 38.98 $\pm$ 0.39 | 61.75 $\pm$ 0.14 | 43.65 $\pm$ 0.46 | 58.21 $\pm$ 0.41 | 55.29 $\pm$ 0.23 | 46.15 $\pm$ 0.71 | 53.93 $\pm$ 0.16 | 52.53 $\pm$ 0.09 | 51.31 $\pm$ 0.13
\+ diag functions | 44.15 $\pm$ 0.002 | 62.80 $\pm$ 0.002 | 48.30 $\pm$ 0.001 | 64.60 $\pm$ 0.002 | 60.30 $\pm$ 0.001 | 49.80 $\pm$ 0.002 | 55.14 $\pm$ 0.001 | 57.70 $\pm$ 0.001 | 55.23 $\pm$ 0.001
++ CE loss | 51.20 $\pm$ 0.007 | 69.10 $\pm$ 0.002 | 63.20 $\pm$ 0.004 | 73.20 $\pm$ 0.002 | 66.60 $\pm$ 0.002 | 61.50 $\pm$ 0.002 | 55.50 $\pm$ 0.003 | 61.60 $\pm$ 0.002 | 62.70 $\pm$ 0.001

Table 2: Banyan compared with Self-StrAE on lexical evaluation. Higher is
better. Results are the average across four random intialisations. Additional
rows represent the same changes as described in Table 1.

Model | Simlex | Wordsim-S | Wordsim-R | Score
---|---|---|---|---
Self-StrAE | 13.80 $\pm$ 0.41 | 54.38 $\pm$ 0.78 | 52.85 $\pm$ 1.27 | 40.34 $\pm$ 0.66
\+ diag functions | 16.90 $\pm$ 0.29 | 57.15 $\pm$ 0.29 | 58.53 $\pm$ 0.62 | 44.18 $\pm$ 0.33
++ CE loss | 22.85 $\pm$ 1.01 | 60.15 $\pm$ 2.08 | 63.43 $\pm$ 0.86 | 48.81 $\pm$ 0.48
Banyan | 16.34 $\pm$ 0.87 | 57.10 $\pm$ 0.35 | 56.80 $\pm$ 0.47 | 43.41 $\pm$ 0.50
\+ diag functions | 18.04 $\pm$ 0.002 | 55.50 $\pm$ 0.001 | 60.00 $\pm$ 0.002 | 44.50 $\pm$ 0.001
++ CE loss | 16.57 $\pm$ 0.02 | 63.25 $\pm$ 0.03 | 69.00 $\pm$ 0.01 | 49.61 $\pm$ 0.02

(a) Changing Structure

(b) Changing Functions

(c) Changing Objective

(d) Combined Improvement

Figure 1: Ablations for changes to model, higher is better. In all plots
original and change.

We want to test whether our modifications to the inductive bias lead to more
effective acquisition of multi-level semantics. Consequently, we evaluate on a
series of tasks that measure correspondence between representation similarity
and human judgements on the word and sentence levels. As we want to focus on
the effects of our unsupervised pre-training regime, we utilise zeroshot
evaluation. On the word level, we use Simlex-999 Hill et al. (2015) and
WordSim-S/R Agirre et al. (2009). All tasks measure semantic similarity, but
do so on differing axes. To understand this, we must first qualify the
difference between semantic similarity and relatedness. Semantic similarity
measures the extent to which entities act the same way, e.g. ‘running’ and
‘singing’ are similar as they share the role verb. Semantic relatedness
measures conceptual association, e.g. ‘singing’ and ‘fame’ may be highly
related. Simlex measures similarity at the exclusion of relatedness, Wordsim S
measures similarity without penalising relatedness, and Wordsim R measures
relatedness. On the sentence level we utilise STS-12 through 16 Agirre et al.
(2012, 2013, 2014, 2015, 2016), the STS-B Cer et al. (2017), SICK-R Marelli et
al. (2014) and SemRel Ousidhoum et al. (2024b). Each measures slightly
different aspects of sentential semantics. We use an embedding dimension of
256 for all experiments; see A for additional hyperparameters.

##  5 Comparison to Self-StrAE

Table 3: Banyan compared with unstructured baselines on sentence level
semantics. Higher is better. Results represent the average across four random
initialisations. Only columns where there is no standard deviation overlap
between models are bolded.

Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score
---|---|---|---|---|---|---|---|---|---
GloVe | 31.61 $\pm$ 0.31 | 21.69 $\pm$ 0.12 | 27.37 $\pm$ 0.10 | 40.42 $\pm$ 0.09 | 29.27 $\pm$ 0.12 | 28.25 $\pm$ 0.08 | 50.20 $\pm$ 0.25 | 41.20 $\pm$ 0.43 | 33.75 $\pm$ 0.04
\+ stopword rm | 39.00 $\pm$ 0.57 | 41.61 $\pm$ 0.19 | 39.31 $\pm$ 0.18 | 51.06 $\pm$ 0.35 | 45.14 $\pm$ 0.14 | 48.40 $\pm$ 0.07 | 52.80 $\pm$ 0.04 | 42.37 $\pm$ 0.13 | 44.96 $\pm$ 0.10
RoBERTa | 42.77 $\pm$ 1.27 | 51.70 $\pm$ 1.30 | 45.67 $\pm$ 1.42 | 63.97 $\pm$ 0.81 | 59.60 $\pm$ 0.61 | 39.97 $\pm$ 0.95 | 52.93 $\pm$ 0.23 | 52.73 $\pm$ 0.58 | 51.08 $\pm$ 0.61
\+ SimCSE | 50.63 $\pm$ 1.45 | 62.23 $\pm$ 2.51 | 54.17 $\pm$ 2.10 | 68.77 $\pm$ 3.00 | 66.67 $\pm$ 1.40 | 53.53 $\pm$ 1.18 | 56.87 $\pm$ 1.16 | 59.27 $\pm$ 0.93 | 59.02 $\pm$ 1.45
Banyan | 51.20 $\pm$ 0.007 | 69.10 $\pm$ 0.002 | 63.20 $\pm$ 0.004 | 73.20 $\pm$ 0.002 | 66.60 $\pm$ 0.002 | 61.50 $\pm$ 0.002 | 55.50 $\pm$ 0.003 | 61.60 $\pm$ 0.002 | 62.70 $\pm$ 0.001

Table 4: Banyan compared with unstructured baselines on lexical semantics.
Higher is better. Results represents the average across four random
initialisations. Only columns where there is no standard deviation overlap
between models are bolded.

Model | Simlex | Wordsim-S | Wordsim-R | Score
---|---|---|---|---
GloVe | 27.47 $\pm$ 0.25 | 62.53 $\pm$ 0.42 | 51.00 $\pm$ 0.56 | 47.00 $\pm$ 0.38
RoBERTa | 29.23 $\pm$ 0.64 | 61.97 $\pm$ 2.38 | 46.00 $\pm$ 2.13 | 45.73 $\pm$ 1.71
Banyan | 16.57 $\pm$ 0.02 | 63.25 $\pm$ 0.03 | 69.00 $\pm$ 0.01 | 49.61 $\pm$ 0.02

For our first set of experiments, we compare Banyan against Self-StrAE Opper
et al. (2023). As the strength of the original model was on data efficiency,
we adopt the same pre-training regime of $\approx$10M tokens subsampled from
the human-written section of English Wikipedia.

Our first test involves simply changing the structure type from sentential
trees to entangled trees. Results are in Tables 1 and 2 and Fig. 1(a). Across
both the word and the sentence level, Banyan, which utilises entangled trees,
consistently outperforms the original Self-StrAE. However, it should be noted
that a fundamental difference between entangled trees and the standard tree
encoding setup is that the latter introduces false negatives in the
contrastive loss, as each repeated instances of a particular node in the batch
is treated as a negative to the other instances. Meanwhile, when using
entangled trees, because we are dealing with the set of all nodes such false
negatives cannot arise.

To assess whether it is entangled trees themselves rather than the absence of
false negatives causing the improvement, a simple test employs a pre-training
objective where false negatives have no effect. The cross entropy objective
$(\mathcal{L}_{\text{CE}})$ where the model simply reconstructs its input
tokens, suits this purpose. Initial attempts simply using
$\mathcal{L}_{\text{CE}}$ with Self-StrAE led to very unstable performance
between seeds (see Appendix B). To remedy this, we switched to the new message
passing paradigm described in § 3, utilising diagonal composition and
decomposition functions. We can now compare entangled trees with sentential
trees and avoid potential issues with false negatives as a confounding factor.
As before, we find entangled trees yield near universal benefits across the
board. The only difference between the two models is that Banyan uses
entangled trees, and while both structural variants show improvements from
diagonal message passing functions, Banyan still retains an edge, indicative
of the utility of entangled trees. As a final ablation, we also compare Banyan
and the original Self-StrAE using the new functions with the contrastive loss
(see Fig. 1(b)). While this underperforms compared to the cross entropy
objective, we again see that Banyan shows improved performance, cementing the
benefits of entangled trees.

Beyond the benefits afforded by entangled trees, the change to diagonal
functions brings two key benefits. Firstly, it helps avoid the contrastive
loss which, while conceptually elegant, in practice comes with a host of
problems (see Robinson et al. (2021)). Secondly, we find that with the new
functions, performance is either maintained or improved by increasing channel
number $K$ and correspondingly decreasing channel size $U$ (see Appendix C for
ablations). As the number of parameters is directly proportional to $U$, this
fact means that we can reduce the number of non-embedding parameters to 14
(our chosen setting has $K=128$ and $U=2$); a further order of magnitude
reduction compared with the already minimal Self-StrAE. We note that this is
the same pattern observed by Opper and Siddharth (2024), however, unlike their
findings we are able to achieve consistent good performance between seeds.

Summary: We make three changes on top of Self-StrAE:  (i) learning over
entangled trees,  (ii) message passing with diagonal functions, and  (iii)
switching from the contrastive to the cross-entropy objective.  Put together,
these changes result in a significantly more effective model.

##  6 Comparison to Unstructured Baselines

Next, we compare to baselines that do not have an inductive bias towards
explicit hierarchy. For these we chose GloVe Pennington et al. (2014), a
classic yet robust baseline which encodes the distributional hypothesis, and
RoBERTa Liu et al. (2019) a transformer encoder which may encode hierarchy
implicitly but does not do so explicitly. For RoBERTa we chose the ‘medium’
configuration as defined by Turc et al. (2019). Further hyperparameters can be
found in Appendix A. We also include augmentations to the baselines to make
them more challenging. For GloVe on the sentence level, we report results both
for simple averaging, and averaging with stopwords removed following Reimers
and Gurevych (2019)—intended to produce better sentence embeddings. For
RoBERTa we report results using both the standard model, and after an extra
round of SimCSE training Gao et al. (2021), as a further STS baseline. In both
cases, we generate sentence embeddings through mean pooling. To produce static
embeddings from RoBERTato use in lexical evaluation, we follow Bommasani et
al. (2020) and average the contextualised representations of all occurrences
of the word in the training set. Finally, since these baselines were not
designed with the data efficient focus of Self-StrAE we train them on the 100M
token Wiki-103 benchmark dataset Merity et al. (2016).

Results are shown in Tables 3 and 4. On the sentence level Banyan is only
rivalled by RoBERTa medium with SimCSE augmentation, and even here it retains
an edge despite being a _far_ smaller model (see Table 6) and having less pre-
training data. On the word level, the picture is more mixed. All models learn
to capture semantic similarity well, however, both RoBERTa and GloVe do so at
the expense of semantic relatedness—indicated by their higher Simlex score,
which penalises models that capture relatedness. Banyan on the other hand,
captures both similarity and relatedness well, with its performance on
relatedness surpassing performance on similarity. We believe that Banyan’s
strong ability to capture relatedness is a direct consequence of its inductive
bias. Related concepts might often be siblings in the tree structure, and so
if Banyan is capturing informative structure, it follows that it would learn
lexical relatedness well.

Summary: We compare Banyan against unstructured baselines to assess the
utility of modelling explicit interconnected tree structure. We find that
despite its minimal parameterisation Banyan manages to be performant,
demonstrating the utility of the inductive biases employed.

##  7 Performance Across Languages

Table 5: Banyan performance on sentence level semantic relatedness across
different languages. Higher is better. Results for Banyan are the average
across four random initialisations. Results for competition and XLM-R are
taken from Ousidhoum et al. (2024a).

@l@ *4l@[colortbl-like] Model Afrikaans Spanish Amharic Hindi
gray!40 Comp. Winners 78.9 70.9 76.4 79.7
gray!40 Comp. Average 73.23 65.04 50.6 59.8
Banyan 78.68 $\pm$ 0.30 60.95 $\pm$ 0.76 65.18 $\pm$ 0.46 61.83 $\pm$ 0.6
XLM-R 56.2 68.9 57.3 52.7

Having established that Banyan is able to efficiently learn the semantics on a
corpus of English data, what is still not clear is whether this is an artifact
of English, or whether Banyan can be effective across languages. In
particular, we are interested in low resource languages where Banyan’s data
efficiency and minimal parameterisation may prove particular assets (the model
can be pre-trained a few million tokens and does not require extensive
compute). However, we also need to evaluate performance in these languages,
and given their low resource nature, this is not straightforward. We turn to
the 2024 SemEval Task 1 Ousidhoum et al. (2024b), which evaluates sentential
semantic relatedness across several low resource African and Asian languages
(as well as a few European ones which are higher resource). Of the languages
we selected Afrikaans, Amharic, Hindi and Spanish to test Banyan. As a
baseline we utilise the official choice for the 2024 semantic relatedness
challenge Ousidhoum et al. (2024a) XLM-R Conneau et al. (2019) a 270M
parameter RoBERTa variant pre-trained on over 2TB of multilingual data. We
train Banyan from scratch on $\leq$ 1M sentences of pre-training data per
language. For further details see Appendix D. Results are in § 7. On all
languages apart from Spanish Banyan outperforms XLM-R; the exception for
Spanish is likely due to it being considerably higher-resource than the
others, and therefore more likely to make a large portion of XLM-R’s extensive
pre-training set. For the sake of transparency, we also include competition
average and winners’ results, noting however that the comparison is not
apples-to-apples as the vast majority of these fine-tuned existing sentence
embedding models like Reimers and Gurevych (2019), that were trained using
supervised sets, while Banyan is both smaller scale and unsupervised.
Surprisingly, Banyan generally performs above the average and even achieves
results comparable to SoTA for Afrikaans, despite its simplicity.

Summary: We show that Banyan can be effective across different languages
beyond English. As Banyan requires very little data scale to optimise, it can
do well even in a low resource setting.

##  8 Efficiency

Table 6: Number of parameters for our model and the baselines. Following
convention we exclude the embedding matrix and LM head from the count.

Model | Banyan | Self-StrAE |  RoBERTa (M) | XLM-R
---|---|---|---|---
Params | 14 | 1072 |  $\approx$10M |  $\approx$85M

Banyan is a significantly more efficient model than Self-StrAE for two primary
reasons. Firstly, the diagonal message passing functions are both easier to
compute and have fewer parameters than standard weight matrices ($2U$ rather
than $2U\times U$). This means a further order of magnitude reduction compared
with the already minimal Self-StrAE (see Table 6). Secondly, and crucially,
Banyan utilises entangled trees where the number of nodes grows at a
significantly reduced rate with batch size compared with sentential trees (see
Fig. 2). This is because the number of reused constituents also grows as batch
size increases, and entangled trees capture the set of all constituents, which
consequently does not grow as drastically. In practical terms because
entangled trees requires fewer nodes, and each node requires two distinct
embeddings ($\bar{e}$ and $\underline{e}$) to be held for it, reducing their
number leads to radical improvements in memory efficiency.

Figure 2: Total number of nodes in entangled trees vs sentential trees as
batch size grows

##  9 Related Work

Recursive Neural Networks (RvNNs): Similar to RNNs but traversing
hierarchically over a tree rather than a sequence. First popularised by Socher
et al. (2011, 2013) and have inspired numerous successor frameworks, to which
Self-StrAE and therefore by extension Banyan, also belong.

Learning Structure: Prior work on learning structure tends to fall into two
categories. The first centres on language modelling, where an unsupervised
pre-training objective, usually some form of masked language modelling, is
used to score a parsing module Drozdov et al. (2019, 2020); Hu et al. (2021,
2022). Common to these approaches is using a chart parser as the unsupervised
module, which has proved successful though highly memory intensive. The second
regards compositional generalisation tasks, commonly ListOps Nangia and Bowman
(2018), where learning the correct parse for given input constitutes success,
and where non-explicitly structured models commonly fail. Recent models either
use beam search Ray Chowdhury and Caragea (2023) or reinforcement learning
Havrylov et al. (2019), to learn parses, while the inner traversal is handled
by an RvNN module. Here, we use entangled agglomerative composition based on
representation similarity as it is more scalable/less resource intensive than
other methods and, for representation learning with structure, remarkably
effective.

Semantic Representations: The distributional hypothesis Harris (1954) posits
that the meaning of a word is derived from the company it keeps (i.e. context
dictates meaning). This principle has been leveraged by Word2Vec Mikolov et
al. (2013) and later GloVe Pennington et al. (2014) in order to learn static
word embeddings, where a pre-defined window around the word is utilised to
define the context for prediction. One issue with these approaches is that
they don’t capture change in meaning from context, which led to the
development of transformer embeddings. Though on the face of it these models
produce fairly mediocre embeddings Reimers and Gurevych (2019), they can be
enhanced by utilising contrastive learning Gao et al. (2021). Considering what
structure enables, one can see its benefits for learning both distributional
and contextual semantics. On the distributional side, structure provides a
more nuanced way to determine the context window, as the nearest structural
neighbours can have the greatest influence on the context window. On the
contextual side, ambiguity can be resolved by taking the representation of the
node which covers the contextualised meaning.

Representation Learning with Structure: An underexplored area, prior work
focuses on using dependency parses to define the context window for learning
lexical embeddings Levy and Goldberg (2014); Vashishth et al. (2019). More
recently, Opper et al. (2023) use constituency parses to learn representations
for multiple levels of hierarchy (from word to sentence). This is extended so
representations define their own structure, removing the need for an external
parser. This model, the Self-StrAE is the basis for our investigations here.

##  10 Discussion and Future Work

We present, Banyan which use explicit hierarchical structure in order to learn
representations. We identify two core features which make the model radically
more effective than the original Self-StrAE. Firstly, that modelling entangled
trees rather than individual sentential trees is more beneficial for
representation learning, as well as considerably more memory efficient.
Secondly, we find that lessons from the recent RNN resurgence can also be
applied in the recursive case, with diagonal message passing functions leading
to better representations. Banyan is significantly smaller than the majority
of baselines we used in this paper (see Table 6), but despite this is able to
compete above its weight. We believe this demonstrates the utility of
hierarchy when equipped with the appropriate inductive biases. For future
work, it would be interesting to see the effect of more flexible
architectures. The recent RNN/SSM literature already provides a pathway
towards scaling, and it would be interesting to see what an increased
parameter budget could achieve. We refrain from this question here in order to
keep the focus on the inductive bias. Secondly, our merge algorithm
(agglomerative clustering) despite its efficacy, remains very simple. It would
be interesting to see if it would possible to integrate self-attention, as it
already models a fully connected graph between inputs and all that is required
is to add a discretisation step.

##  11 Limitations

The same limitations which apply to the original Self-StrAE also apply to
Banyan. It has extremely few parameters, and makes locally greedy decisions
regarding merge. It is highly likely that a more sophisticated induction
algorithm and a more powerful model would lead to downstream performance. The
reason we adopt this setting is because we want to isolate the effects of the
inductive biases we use, and we know that performance for StrAE style models
has proven to be directly dependent on structure Opper et al. (2023). We
believe that we have demonstrated the utility of the biases advocated for in
this paper: explicit hierarchy, specifically, entangled trees. However, our
aim is to conduct basic research, not to directly compete with SoTA.

##  12 Ethics Statement

In this paper, we investigate what inductive biases enable more resource
efficient representation learning. We believe this can provide considerable
benefits in making research more accessible, both in terms of compute and
language coverage.

## References

  * Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009.  A study on similarity and relatedness using distributional and WordNet-based approaches.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , pages 19–27.
  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015.  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 252–263, Denver, Colorado. Association for Computational Linguistics.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014.  SemEval-2014 task 10: Multilingual semantic textual similarity.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 81–91, Dublin, Ireland. Association for Computational Linguistics.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California. Association for Computational Linguistics.
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013.  *SEM 2013 shared task: Semantic textual similarity.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , pages 32–43, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Agirre et al. (2012) Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_ , SemEval ’12, page 385–393, USA. Association for Computational Linguistics.
  * Bommasani et al. (2020) Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4758–4781, Online. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Workshop on Semantic Evaluation (SemEval)_ , pages 1–14.
  * Chomsky (1956) N. Chomsky. 1956.  Three models for the description of language.  _IRE Transactions on Information Theory_ , 2(3):113–124.
  * Conneau et al. (2019) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Unsupervised cross-lingual representation learning at scale.  _CoRR_ , abs/1911.02116.
  * Crain and Nakayama (1987) Stephen Crain and Mineharu Nakayama. 1987.  Structure dependence in grammar formation.  _Language_ , 63(3):522–543.
  * De et al. (2024) Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. 2024.  Griffin: Mixing gated linear recurrences with local attention for efficient language models.
  * de Marneffe et al. (2006) Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006.  Generating typed dependency parses from phrase structure parses.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Drozdov et al. (2020) Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O’Gorman, Mohit Iyyer, and Andrew McCallum. 2020.  Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4832–4845, Online. Association for Computational Linguistics.
  * Drozdov et al. (2019) Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019.  Unsupervised latent tree induction with deep inside-outside recursive autoencoders.  _CoRR_ , abs/1904.02142.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Harris (1954) Zellig S. Harris. 1954.  Distributional structure.
  * Havrylov et al. (2019) Serhii Havrylov, Germán Kruszewski, and Armand Joulin. 2019.  Cooperative learning of disjoint syntax and semantics.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1118–1128, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Heinzerling and Strube (2018) Benjamin Heinzerling and Michael Strube. 2018.  BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  SimLex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hu et al. (2022) Xiang Hu, Haitao Mi, Liang Li, and Gerard de Melo. 2022.  Fast-R2D2: A pretrained recursive neural network based on pruned CKY for grammar induction and text representation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2809–2821, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hu et al. (2021) Xiang Hu, Haitao Mi, Zujie Wen, Yafang Wang, Yi Su, Jing Zheng, and Gerard de Melo. 2021.  R2D2: Recursive transformer based on differentiable tree for interpretable hierarchical language modeling.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4897–4908, Online. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA.
  * Le and Zuidema (2014) Phong Le and Willem Zuidema. 2014.  Inside-outside semantics: A framework for neural models of semantic composition.  In _NIPS 2014 Workshop on Deep Learning and Representation Learning_.
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Dependency-based word embeddings.  In _Association for Computational Linguistics (ACL)(Volume 2: Short Papers)_ , pages 302–308.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.  _CoRR_ , abs/1609.07843.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _International Conference on Learning Representations (ICLR)_.
  * Nangia and Bowman (2018) Nikita Nangia and Samuel R. Bowman. 2018.  Listops: A diagnostic dataset for latent tree learning.  _CoRR_ , abs/1804.06028.
  * Opper et al. (2023) Mattia Opper, Victor Prokhorov, and Siddharth N. 2023.  StrAE: Autoencoding for pre-trained embeddings using explicit structure.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7544–7560, Singapore. Association for Computational Linguistics.
  * Opper and Siddharth (2024) Mattia Opper and N. Siddharth. 2024.  Self-strae at semeval-2024 task 1: Making self-structuring autoencoders learn more with less.
  * Orvieto et al. (2023) Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023.  Resurrecting recurrent neural networks for long sequences.
  * Ousidhoum et al. (2024a) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, and Saif M. Mohammad. 2024a.  Semrel2024: A collection of semantic textual relatedness datasets for 14 languages.
  * Ousidhoum et al. (2024b) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, and Saif M. Mohammad. 2024b.  SemEval-2024 task 1: Semantic textual relatedness for african and asian languages.  In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_. Association for Computational Linguistics.
  * Pallier et al. (2011) Christophe Pallier, Anne-Dominique Devauchelle, and Stanislas Dehaene. 2011.  Cortical representation of the constituent structure of sentences.  _Proceedings of the National Academy of Sciences_ , 108(6):2522–2527.
  * Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Xiangru Tang, Bolun Wang, Johan S. Wind, Stansilaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. 2023.  Rwkv: Reinventing rnns for the transformer era.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Ray Chowdhury and Caragea (2023) Jishnu Ray Chowdhury and Cornelia Caragea. 2023.  Beam tree recursive cells.  In _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 28768–28791. PMLR.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Robinson et al. (2021) Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. 2021.  Can contrastive learning avoid shortcut solutions? _CoRR_ , abs/2106.11230.
  * Sartran et al. (2022) Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, and Chris Dyer. 2022.  Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale.  _Transactions of the Association for Computational Linguistics_ , 10:1423–1439.
  * Socher et al. (2011) Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011.  Semi-supervised recursive autoencoders for predicting sentiment distributions.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 151–161.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1631–1642.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: The impact of student initialization on knowledge distillation.  _CoRR_ , abs/1908.08962.
  * Vashishth et al. (2019) Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, and Partha Talukdar. 2019.  Incorporating syntactic and semantic information in word embeddings using graph convolutional networks.  In _Association for Computational Linguistics (ACL)_ , pages 3308–3318.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 5998–6008.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations (ICLR)_.
  * Wang et al. (2020) Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2020.  Deep graph library: A graph-centric, highly-performant package for graph neural networks.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.

##  Appendix A Hyperparameters

We trained Self-StrAE and Banyan for 15 epochs (circa 15k steps- sufficient
for convergence) using the Adam optimizer Kingma and Ba (2015) with a learning
rate of 1e-3 for cross entropy and 1e-4 for contrastive loss, using a batch
size of 512. We applied dropout of 0.2 on the embeddings and 0.1 on the
composition and decomposition functions. The temperature hyper-parameter for
the contrastive objective was set to 0.2 and the r value 0.1. These settings
were obtained by a grid search over the values r= 1.0, 0.1, 0.01, batch size =
128, 256, 512, 768, $\tau$ = 0.2, 0.4, 0.6, 0.8 and learning rate 1e-3, 5e-4,
and 1e-4. To process the graphs we used DGL Wang et al. (2020). The GloVe
baseline was trained for 15 epochs with a learning rate of 1e-3, and a window
size of 10. RoBERTa medium was trained for 200,000 steps, with a 10% of which
were used for warmup, a learning rate of 5e-5 and a linear schedule. We used
relative key-query positional embeddings. The configuration for RoBERTa medium
is 8 layers, 8 attention heads and 2048 dimensional feedforward layers. We
used the Transformers library to train the model Wolf et al. (2020). For
SimCSE training, we used the default parameters and implementation for
unsupervised RoBERTa training from Gao et al. (2021). Training was performed
using a single A40 (40gb GPU).

##  Appendix B Cross Entropy Instability

Without our changed functions, the cross entropy objective demonstrated
significant instability between seeds. We show these results in Tables 8 and
7.

Table 7: Cross entropy instability with old functions. Results are the average
across four random seeds.

Model | STS-12 | STS-13 | STS-14 | STS-15 | STS-16 | STS-B | SICK-R | SemRel | Score
---|---|---|---|---|---|---|---|---|---
Banyan old functions | 29.4 $\pm$ 19.4 | 42.2 $\pm$ 26.6 | 35.8 $\pm$ 26.08 | 45.8 $\pm$ 28.6 | 43.2 $\pm$ 23.3 | 36.5 $\pm$ 25.4 | 41.9 $\pm$ 19.05 | 39.8 $\pm$ 24.5 | 39.33 $\pm$ 24.0

Table 8: Cross entropy instability with old functions. Results are the average
across four random seeds.

Model | Simlex | Wordsim-S | Wordsim-R | Score
---|---|---|---|---
Banyan old functions | 11.6 $\pm$ 10.1 | 32.4 $\pm$ 38.8 | 34.5 $\pm$ 30.0 | 26.2 $\pm$ 26.2

##  Appendix C The k and u balance

The change to diagonal composition functions allows us to reduce the number of
total parameters while maintaining performance. This is because the number of
parameters is directly proportional to channel size u. We show ablations for
this finding in Table 9.

Table 9: Performance Depending on k and u values using new functions. Scores
are the average of four random seeds.

k | u | Lex Score | STS Score
---|---|---|---
4 | 64 | 42.9 $\pm$ 0.01 | 43.5 $\pm$ 0.04
8 | 32 | 43.2 $\pm$ 0.02 | 48.6 $\pm$ 0.01
16 | 16 | 47.02 $\pm$ 0.03 | 62.2 $\pm$ 0.01
32 | 8 | 49.2 $\pm$ 0.01 | 62.9 $\pm$ 0.01
64 | 4 | 48.7 $\pm$ 0.01 | 62.9 $\pm$ 0.01
128 | 2 | 49.61 $\pm$ 0.02 | 62.7 $\pm$ 0.001
256 | 1 | 48.7 $\pm$ 0.01 | 62.9 $\pm$ 0.001

##  Appendix D Data Sources

For Afrikaans, Spanish and Amharic we obtained corpora from Leipzig Corpora
Collection111For both Spanish, Afrikaans and Hindi we selected the mixed
corpus and took a uniform subsample to reduce size to the requisite scale if
necessary.. Meanwhile for Amharic we utilised a MiT licenced pre-training set
of 1 million sequences available on the Huggingface hub at this link. We
utilise a pre-trained BPE tokenizer for each language from the BPEMB Python
package Heinzerling and Strube (2018). Though the package also provides pre-
trained embeddings, we solely use the tokenizer and learn embeddings from
scratch.


2021

1]Institute of Medicial Technology and Intelligent Systems, Technische
Universitaet Hamburg, Germany 2] Department of Otorhinolaryngology, Head and
Neck Surgery and Oncology 3] Clinic and Polyclinic for Diagnostic and
Interventional Radiology and Nuclear Medicine 4] Population Health Research
Department, University Heart and Vascular Center 5] Clinic and Polyclinic for
Neurology

# Self-supervised learning for classifying paranasal anomalies in the
maxillary sinus

Debayan Bhattacharya  Finn Behrendt  Benjamin Tobias Becker  Lennart Maack
Dirk Beyersdorff  Elina Petersen  Marvin Petersen  Bastian Cheng  Dennis
Eggert  Christian Betz  Anna Sophie Hoffmann  Alexander Schlaefer  [  [  [  [
[  University Medical Center Hamburg-Eppendorf, Hamburg, Germany

###### Abstract

00footnotetext: *Equal contribution00footnotetext: For correspondence send
email to<EMAIL_ADDRESS>
Purpose: Paranasal anomalies, frequently identified in routine radiological
screenings, exhibit diverse morphological characteristics. Due to the
diversity of anomalies, supervised learning methods require large labelled
dataset exhibiting diverse anomaly morphology. Self-supervised learning (SSL)
can be used to learn representations from unlabelled data. However, there are
no SSL methods designed for the downstream task of classifying paranasal
anomalies in the maxillary sinus (MS).

Methods: Our approach uses a 3D Convolutional Autoencoder (CAE) trained in an
unsupervised anomaly detection (UAD) framework. Initially, we train the 3D CAE
to reduce reconstruction errors when reconstructing normal maxillary sinus
(MS) image. Then, this CAE is applied to an unlabelled dataset to generate
coarse anomaly locations by creating residual MS images. Following this, a 3D
Convolutional Neural Network (CNN) reconstructs these residual images, which
forms our SSL task. Lastly, we fine-tune the encoder part of the 3D CNN on a
labelled dataset of normal and anomalous MS images.

Results: The proposed SSL technique exhibits superior performance compared to
existing generic self-supervised methods, especially in scenarios with limited
annotated data. When trained on just 10% of the annotated dataset, our method
achieves an Area Under the Precision-Recall Curve (AUPRC) of 0.79 for the
downstream classification task. This performance surpasses other methods, with
BYOL attaining an AUPRC of 0.75, SimSiam at 0.74, SimCLR at 0.73 and Masked
Autoencoding using SparK at 0.75.

Conclusion: A self-supervised learning approach that inherently focuses on
localizing paranasal anomalies proves to be advantageous, particularly when
the subsequent task involves differentiating normal from anomalous maxillary
sinuses. Access our code at https://github.com/mtec-tuhh/self-supervised-
paranasal-anomaly

###### keywords:

Paranasal anomaly, self supervided learning, maxillary sinus, CNN,
classification

##  1 Introduction

The paranasal sinuses, air-filled spaces within the craniofacial complex, vary
significantly and include the maxillary, frontal, sphenoid, and ethmoid
sinuses 9910059601802121 . Common pathologies like retention cysts, polyps,
and mucosal thickening are identifiable through radiological screenings
Bal2014-uw ; Varshney2015-qi ; Van_Dis1994-xz . However, their diagnosis is
challenging due to their incidental nature and the variability in sinus
appearance Hansen.2014 . Research underscores their prevalence and the
importance of accurate diagnosis in patient care Tarp.2000 . 3D imaging from
computed tomography (CT) and magnetic resonance images (MRI) is vital for
precise diagnosis, and misdiagnosis can lead to patient distress and increased
healthcare costs Brierley.2017 ; Gutmann.2013 . The anatomical variability of
the sinuses Papadopoulou2021-ch  necessitates careful application of deep
learning for reliable diagnoses .

Convolutional Neural Networks (CNN) are recognized for diagnosing paranasal
pathologies, evidenced in sinusitis classification Jeon.2021 ; Kim.2019 ,
differentiating inverted papilloma from carcinomas Liu.2022 , and detecting MS
fungal ball and chronic rhinosinusitis in CT scans
10.1371/journal.pone.0263125 . Prior studies have explored contrastive
learning and cross-entropy loss for MS anomaly classification
10.1007/978-3-031-16437-8_41 , and MS extraction techniques from MRI
bhattacharya2023multiple . However, all of the aforementioned methods use
supervised learning. Given the difficulty in obtaining well-labelled datasets
in clinical settings 10.1145/3439950 , and the relative ease of acquiring
unlabelled data, self-supervised learning (SSL), which learns representations
from unlabelled data to improve the downstream task, has not yet been explored
for paranasal anomaly classification. SSL efficiently utilizes unlabelled data
through tasks like non-linear compression 9412239 ; Xie2023 , denoising
10.5555/1756006.1953039 , feature alignment from augmented images
NEURIPS2020_f3ada80d ; 9578004 ; Huang2023  and inpainting masked regions of
images tian2023designing . However, these methods are designed to improve the
performance of models exposed to 2D natural images. Hence, they lack a
specific focus on enhancing MS anomaly classification from 3D MRI. Our aim is
to design an SSL task that enables the models trained on it to achieve maximum
data efficiency in classifying paranasal anomalies. We hypothesize anomaly
segmentation within MS could be a good SSL task. Without ground truth
segmentation masks, we use a UAD framework, applied in brain Baur2021-uz ;
9761443  and paranasal anomaly detection
https://doi.org/10.48550/arxiv.2211.01371 , to localize MS anomalies. A 3D
Convolutional Autoencoder (CAE) trained on a labelled normal dataset is used
to reconstruct MS volumes and localize anomalies in an unlabelled dataset by
failing to reconstruct anomalies leading to reconstruction errors. These
errors, serving as pseudo segmentation masks are used in the SSL task to
localize anomalies. We investigate if a 3D CNN, predicting these errors as SSL
task, can improve feature discrimination between anomalous and normal MS in
our labelled dataset. Our SSL task leverages available normal MS data,
essential for supervised downstream task training.

Overall, our main contributions can be summed up as follows:

  * •

We present a self-supervised method that improves the downstream
classification of normal vs anomlous MS. Our self-supervision task explicitly
learns to coarsely localize anomalies by reconstructing the residual volumes
generated through the UAD-trained autoencoder. This distinguishes our approach
from the compared methods, where anomaly localization is not a primary focus
for the self-supervision task.

  * •

Our self-supervised method effectively utilizes labelled healthy MS data
reserved for downstream tasks. Hence, we explore how varying the CAE training
set impacts downstream classification performance.

  * •

We investigate post-processing strategies and loss function used in the self-
supervision task for learning better transferable features for the downstream
task.

##  2 Methods

Figure 1: a) Extraction of MS volumes from cranial MRI b) Exemplarary coronal
images of normal MS volume and MS with mucosal thickening, polyp and cyst
anomaly c) Our CAE architecture. Here, k refers to kernel size, s refers to
stride, p refers to padding, c refers to channel where, for example, 1/16
refers to input channel of 1 and output channel of 16. Each stage of the
encoder and decoder is formed using 3D convolution followed by batch
normlalisation and leaky ReLU. Upsample refers to trilinear upsampling. d)
Generation of residual volume required for the self-supervision task using our
CAE e) Our self-supervision task where the encoder and decoder is trained to
reconstruct the residual volume f) Downstream task where the self-supervision
trained encoder is trained to classify between normal and anomalous MS.
Figure 2:  Our data processing pipeline comprises several steps: a) The
labelled dataset  $\ignorespaces D_{l}$ b) Splitting  $\ignorespaces D_{l}$
into training, validation, and test subsets for downstream classification of
normal versus anomalous MS. c) Normal MS samples from the labelled training
set form  $\ignorespaces D_{l}^{n}$ , used to train the 3D CAE  $\ignorespaces
A(.)$ within the UAD framework. d) Unlabelled dataset  $\ignorespaces D_{u}$
e) This trained 3D CAE  $\ignorespaces A(.)$ generates residual volumes from
the unlabelled dataset  $\ignorespaces D_{u}$ e) Unlabelled dataset of
residual volumes f) The 3D CNN undergoes self-supervised training to
reconstruct these residual volumes. g) The 3D CNN’s encoder is initialized
with weights from the SSL task, then undergoes supervised training for the
final task of classifying normal versus anomalous MS, using the training set
created in step a).

###  2.1 Description of Dataset

As part of the Hamburg City Health Study (HCHS) Jagodzinski2019-ct , cranial
MRI scans were obtained from individuals aged 45-74 years to evaluate
neuroradiological parameters. The scans were acquired using fluid attenuated
inversion recovery (FLAIR) sequences in the NIfTI format at the University
Medical Center Hamburg-Eppendorf. The MRI scans had a resolution of 173 mm x
319 mm x 319 mm. The labelled dataset consisted of 1067 patients. Among the
patients, 489 exhibited no pathologies in their left and right MS, while 578
had at least one MS presenting polyp, cyst or mucosal thickening pathology.
All these anomalies were grouped into the ”anomaly” class. Our unlabelled
dataset consists of 1559 patient MRIs. The diagnoses were established by two
ENT specialists and one radiologist specialized in ENT. Figure 1 b shows
coronal slices highlighting the diverse set of anomalies that are present in
our dataset.

###  2.2 Dataset preprocessing

In our dataset preprocessing, as outlined in previous work
bhattacharya2023multiple ; 10.1007/978-3-031-16437-8_41 , we first align MRIs
with a fixed sample from our dataset. Centroid locations of left and right MS
regions were recorded for 20 patients, guiding the extraction of MS volumes
from larger cranial MRIs. This step isolates the relevant MS volumes for our
task of classifying healthy and anomalous MS. We then used the mean centroid
location from these 20 recordings to extract left and right MS volumes from
all cranial MRIs in our dataset. The extracted volumes, sized 64 mm x 64 mm x
64 mm, cover the entire MS. Figure 1 a illustrates this extraction process.

Each cranial MRI yielded one left and one right MS volume. To enhance
symmetry, right MS volumes were horizontally flipped to match the left ones.
All volumes were normalized to an intensity range of 0 to 1. We employed five-
fold cross-validation for evaluation, ensuring diverse labelled datasets (10%,
20%, 40%, 60%, 80%) maintain the anomaly-to-normal ratio. The separation of
training, validation, and test sets was strictly maintained, with left or
right MS volumes from the same patient assigned to only one set. Table 1
details our dataset division across these sets.

Class | Training Set | Validation Set | Test Set
---|---|---|---
# Normal MS | 708 | 176 | 380
# Anomalous MS | 487 | 122 | 261
Table 1: Statistics of our labelled dataset  $\ignorespaces D_{l}$

###  2.3 Architecture

Our CAE, depicted in Figure 1 c, uses 3D convolutional operations with a
latent bottleneck dimension of 512. The CNN architecture is U-Net inspired,
featuring a 3D ResNet18 encoder  $\ignorespaces E(.)$ 8578773  with four
stages and channel dimensions of 64, 128, 256, and 512. The decoder
$\ignorespaces D(.)$ mirrors the encoder, with reverse channel dimensions and
trilinear upsampling. Skip connections are used to pass encoder features to
the decoder. For Bootstrap your own latent (BYOL), SimSiam, and SimCLR
training, only the encoder  $\ignorespaces E(.)$ is used, with an MLP attached
to project the final layer features to a dimension of 512.

###  2.4 Autoencoder training and inferrence on unlabelled dataset

Consider  $\ignorespaces D_{l}$ to be our labelled dataset containing normal
and anomalous MS and  $\ignorespaces D_{u}$ to be our unlabelled dataset.
Further, let  $\ignorespaces D_{l}^{n}\subset D_{l}$ be a dataset consisting
of only normal MS volumes. Let  $\ignorespaces x\in R^{64\times 64\times 64}$
be an MS volume in  $\ignorespaces D_{l}$ . Let the autoencoder be represented
as  $\ignorespaces A(.)$ such that  $\ignorespaces x^{\prime}=A(x)$ represents
the reconstructed MS volume. We train the autoencoder using L1 reconstruction
loss which may be written as  $\ignorespaces\|x-x^{\prime}\|$ on
$\ignorespaces D_{l}^{n}$ . Once trained, we use the autoencoder
$\ignorespaces A(.)$ to generate residual volumes on  $\ignorespaces D_{u}$ .
Figure 1 d illustrates our residual volume generation method.

###  2.5 Transfer Learning

Since transfer learning (TL) is a method to achieve data efficiency, we also
trained our models initialised with transfer learning weights. However, since
our downstream task involves MRI and is in 3D domain, ImageNet 5206848
weights may not be appropriate. Hence, the model weights we utilized as
initial weights were obtained through training on eight diverse public 3D
segmentation datasets, covering both MRI and CT modalities. We believe these
weights are more suitable than those derived from natural image training and
therefore employed them as the basis for our 3D CNN. For further information
on the transfer learning model, please see the GitHub repository
111https://github.com/Tencent/MedicalNet.

###  2.6 Self-Supervised Training

With the residual volumes generated for  $\ignorespaces D_{u}$ , we train
$\ignorespaces E(.)$ and  $\ignorespaces D(.)$ to reconstruct the residual
volumes again. This, in effect, makes the encoder and decoder learn features
relevant for anomaly localisation within the unlabelled dataset
$\ignorespaces D_{u}$ . We train  $\ignorespaces E(.)$ and  $\ignorespaces
D(.)$ using  $\ignorespaces L_{recon}$ which in our case is binary cross
entropy (BCE) loss. Figure 1 e illustrates our self-supervised training task.
We evaluated our self-supervised learning method against Autoencoder (AE),
Denoising Autoencoder (DAE), BYOL, SimSiam, SimCLR and Sparse masKed modeling
with hierarchy (SparK). These methods use similar encoders  $\ignorespaces
E(.)$ and decoders  $\ignorespaces D(.)$ , with BYOL, SimSiam, and SimCLR
employing an additional MLP for feature projection. Pretraining with the SparK
framework requires sparse encoder  $\ignorespaces E^{\prime}(.)$ and a special
light decoder which contains 3 convolutional blocks and 3 upsampling blocks
tian2023designing . Patch size 8  $\ignorespaces\times$ 8
$\ignorespaces\times$ 8 and masking ratio of 60% was used during pretraining.
Detailed description and implementation details of our state-of-the-art (SOTA)
SSL methods is provided in the supplementary material section 1-7. More
details about the other masking ratios and patch sizes tested for SparK can be
found in the supplementary material section 11.

###  2.7 Finetuning

Having successfully trained the  $\ignorespaces E(.)$ and  $\ignorespaces
D(.)$ using self-supervision, we move onto the finetuning phase. We discard
$\ignorespaces D(.)$ and focus on training  $\ignorespaces E(.)$ by leveraging
samples from the labelled dataset  $\ignorespaces D_{l}$ . For TL models, we
initialise  $\ignorespaces E(.)$ with transfer learning weights. Next, we
introduce a MLP as an additional component, responsible for projecting the
encoder features from their original dimension of 512 to an intermediate
dimension of 256. Subsequently, the MLP maps these features to a final
dimension of 2, corresponding to the number of classes. We finetune
$\ignorespaces E(.)$ using BCE loss.

Figure 2 illustrates the data processing pipeline and elucidates how the
different components fit into our overall method.

###  2.8 Implementation details

Our PyTorch and PyTorch Lightning-based code accommodates a maximum batch size
of 256 on NVIDIA A6000 with 48GB VRAM for self-supervised pretraining. We
optimize models using LARS ginsburg2018large  with a learning rate of 0.2
across 500 epochs, incorporating a 20-epoch linear warmup and cosine
annealing. For finetuning, AdamW loshchilov2018decoupled  is employed with a
constant rate of 1e-4 for 100 epochs at a batch size of 16. Models yielding
the lowest validation loss are preserved for final evaluation with the test
set. The CAE was trained on 708 normal MS volume samples without augmentation.
For self-supervised methods and MS anomaly classification, we applied data
augmentations such as random affine transformations, flipping, and Gaussian
noise. The DAE specifically used Gaussian noise with a mean of 0 and standard
deviation of 0.6 at 100% probability, while other augmentations were applied
50% of the time. Supplementary material offers comprehensive descriptions and
visualizations of SOTA SSL methods.

##  3 Results

Comparison to state of the art

Results in Table 2 show our method outperforming others in AUROC, AUPRC, and
F1 scores across different labelled dataset scenarios (10%, 20%, 100% of
$\ignorespaces D_{l}$ ). Our method demonstrated notable improvements in AUROC
(3.34% and 4.93% over SimSiam) and AUPRC (5.33% over BYOL and 5.12% over AE)
for 10% and 20% dataset scenarios, respectively. SparK trained models perform
generally poorer compared to the other SSL and TL methods with the performance
gap between SparK MAE and our method widening with increased training set
percentage. Our method had AUPRC 8.21% higher than the TL method when
finetuned on a 10% training set. Pretraining models using our method
significantly boosted AUPRC by 14.49% and AUROC by 9.45% compared to no
pretraining when trained on a 10% training dataset. At 100% dataset
finetuning, our method achieved the highest scores, with AE and SimSiam
showing similar performance. Compared to no pretraining, our method improved
AUPRC by 3.33%. Figure 3 illustrates AUPRC and AUROC trends with increasing
training set percentages, respectively. Our method excels in settings with 40%
or less training data but aligns with SOTA performance beyond that.

Table 2:  The table displays the mean and 95% confidence intervals of metrics
evaluating model performance in the downstream classification task. These
models, trained with varying portions of  $\ignorespaces D_{l}$ , were
initialized using different SSL methods before supervised training.

Method | Training Set Percentage  $\ignorespaces D_{l}$ | AUROC | AUPRC | F1
---|---|---|---|---
No pretraining | 10% | 0.74 (0.64-0.84) | 0.69 (0.56-0.82) | 0.64 (0.59-0.69)
Transfer Learning | 10% | 0.77 (0.72-0.82) | 0.73 (0.66-0.79) | 0.63 (0.57-0.69)
AE | 10% | 0.73 (0.68-0.79) | 0.68 (0.62-0.74) | 0.55 (0.43-0.67)
DAE | 10% | 0.74 (0.73-0.76) | 0.68 (0.66-0.69) | 0.62 (0.60-0.64)
BYOL | 10% | 0.79 (0.76-0.81) | 0.75 (0.70-0.79) | 0.63 (0.59-0.69)
SimSiam | 10% | 0.77 (0.72-0.83) | 0.74 (0.68-0.79) | 0.62 (0.53-0.72)
SimCLR | 10% | 0.78 (0.74-0.81) | 0.73 (0.68-0.78) | 0.63 (0.59-0.68)
SparK MAE | 10% | 0.78 (0.77-0.80) | 0.75 (0.73-0.76) | 0.65 (0.63-0.67)
Ours | 10% | 0.81 (0.74-0.88) | 0.79 (0.71-0.87) | 0.67 (0.58-0.77)
No pretraining | 20% | 0.81 (0.79-0.82) | 0.78 (0.76-0.79) | 0.67 (0.65-0.69)
Transfer Learning | 20% | 0.84 (0.79-0.88) | 0.81 (0.74-0.88) | 0.68 (0.62-0.75)
AE | 20% | 0.81 (0.76-0.86) | 0.78 (0.72-0.83) | 0.67 (0.60-0.74)
DAE | 20% | 0.79 (0.77-0.81) | 0.74 (0.70-0.79) | 0.67 (0.64-0.70)
BYOL | 20% | 0.82 (0.80-0.84) | 0.79 (0.77-0.82) | 0.70 (0.68-0.71)
SimSiam | 20% | 0.84 (0.82-0.86) | 0.81 (0.78-0.84) | 0.70 (0.67-0.74)
SimCLR | 20% | 0.81 (0.79-0.83) | 0.77 (0.74-0.81) | 0.68 (0.67-0.69)
SparK MAE | 20% | 0.80 (0.78-0.82) | 0.76 (0.73-0.79) | 0.67 (0.65-0.68)
Ours | 20% | 0.85 (0.83-0.87) | 0.82 (0.81-0.83) | 0.72 (0.70-0.75)
No pretraining | 100% | 0.90 (0.89-0.91) | 0.89 (0.88-0.90) | 0.80 (0.78-0.82)
Transfer Learning | 100% | 0.92 (0.91-0.93) | 0.91 (0.90-0.93) | 0.82 (0.80-0.83)
AE | 100% | 0.92 (0.91-0.93) | 0.91 (0.90-0.93) | 0.82 (0.80-0.84)
DAE | 100% | 0.90 (0.88-0.92) | 0.89 (0.88-0.91) | 0.79 (0.77-0.82)
BYOL | 100% | 0.89 (0.89-0.90) | 0.88 (0.87-0.89) | 0.78 (0.76-0.81)
SimSiam | 100% | 0.92 (0.91-0.93) | 0.91 (0.90-0.92) | 0.81 (0.79-0.83)
SimCLR | 100% | 0.90 (0.88-0.91) | 0.89 (0.87-0.91) | 0.79 (0.77-0.80)
SparK MAE | 100% | 0.87 (0.85-0.88) | 0.86 (0.84-0.87) | 0.75 (0.73-0.76)
Ours | 100% | 0.93 (0.91-0.94) | 0.92 (0.90-0.93) | 0.83 (0.80-0.86)

Figure 3:  (LEFT) AUPRC trend vs training set percentage (RIGHT) AUROC trend
vs training set percentage

Effect of varying the CAE training set

The effectiveness of our self-supervised task is contingent on the CAE’s
proficiency in reconstructing healthy MS volumes. Inaccurate reconstructions
yield unreliable residuals, affecting self-supervision. To assess the impact
of training set size, the CAE was trained with different proportions (20%,
40%, 60%, 80%, 100%) of the healthy MS dataset  $\ignorespaces D_{l}^{n}$ .
After training, the CAE processed dataset  $\ignorespaces D_{u}$ to produce
residual volumes, which were refined using a median filter with a kernel size
of 5. Subsequent supervised training utilized 10% of our labelled dataset
$\ignorespaces D_{l}$ . Table 3 presents improvements in the downstream task
metrics correlating with increased healthy MS training set sizes, suggesting
that larger normal dataset  $\ignorespaces D_{l}^{n}$ enhance normal MS
representation learning and improve anomaly localization.

Table 3: The table shows the mean and 95% confidence intervals of metrics for
evaluating model performance in downstream classification. The CAE was trained
on varying proportions of the normal MS volumes dataset (  $\ignorespaces
D_{l}^{n}$ ), then used to generate residual volumes from the unlabelled
dataset (  $\ignorespaces D_{u}$ ). Each model was initialized using our
proposed SSL method.

Training Set Percentage  $\ignorespaces D_{l}^{n}$ | AUROC | AUPRC | F1
---|---|---|---
20% | 0.76 (0.70-0.81) | 0.72 (0.67-0.77) | 0.60 (0.50-0.69)
40% | 0.77 (0.73-0.80) | 0.72 (0.66-0.78) | 0.63 (0.57-0.68)
60% | 0.78 (0.75-0.80) | 0.74 (0.71-0.77) | 0.65 (0.62-0.68)
80% | 0.80 (0.76-0.84) | 0.76 (0.72-0.81) | 0.67 (0.63-0.72)
100% | 0.81 (0.74-0.88) | 0.79 (0.71-0.87) | 0.67 (0.58-0.77)

##  4 Discussion

Tailoring SSL tasks to specific downstream tasks offers distinct advantages
ozbulak2023know . Current SOTA SSL methods NEURIPS2020_f3ada80d ; 9578004 ;
Huang2023 , primarily developed for 2D image classification on datasets like
ImageNet, do not address the unique challenges of 3D MRI modalities and the
specifics of paranasal anomalies. Our SSL task is specifically tailored to
address the challenges associated with 3D environments, MRI modality, and the
classification of paranasal anomalies.

We conjecture that segmentation of anomalies as a SSL task, requiring
knowledge of anomaly locations, enhances the learning of class-discriminative
features for distinguishing normal and anomalous MS. Our SSL task is a
segmentation task therefore, it requires segmentation masks highlighting
anomalies. To avoid the high costs of annotation, we use a CAE trained in the
UAD framework for generating approximate annotations, effective in localizing
paranasal anomalies https://doi.org/10.48550/arxiv.2211.01371 . This CAE
training utilizes labelled normal datasets, typically accessible in supervised
settings. Unlike generic SOTA SSL methods, which do not prioritize anomaly
localization, our approach demonstrates improved AUROC and AUPRC (as shown in
Table 2), suggesting that effective anomaly localization can enhance
classification performance, even with limited labelled data. Methods like BYOL
and SimSiam, which aim to maximize agreement between augmented views, are less
effective for paranasal anomaly classification. SimCLR’s performance shortfall
is likely due to smaller batch sizes, a necessity given the impracticality of
large batches in 3D settings, despite SimCLR’s recommendation of 4096
10.5555/3524938.3525087 . Our method is more suited for such constrained
computational resources. AE and DAE, focusing on compression-decompression and
denoising, do not guarantee discriminative feature learning for downstream
classification 10.1145/1273496.1273592 , and were found less effective in our
context. When the entire training set is used, our method, AE, and SimSiam
yield comparable results, with ours marginally outperforming. We also explored
MAE-style pretraining using SparK. However, the results suggest that fine-
tuning performance is notably weaker, particularly when fine-tuning with a
training set percentage 40% and above. These findings imply that generating
masked regions contributes to representation learning; however, the acquired
representations do not appear to enhance downstream classification. It is
noteworthy that the SparK framework was initially developed and evaluated for
2D natural images. Although we adapted the framework for 3D applications, our
findings underscore the necessity for further methodological advancements to
effectively support tasks in the 3D domain. Further, TL models exhibit
comparable performance to SSL methods when fine-tuning on training sets
exceeding 20%. This suggests that transfer learning methods remain viable for
paranasal anomaly classification given an ample supply of labeled samples.
However, in the scenario of an extremely limited labeled dataset, such as 10%,
our method outperforms TL, indicating that the representations acquired by our
approach are especially advantageous in low-data environments. Overall,
compared to approaches without pretraining, our tailored SSL task consistently
shows superior downstream classification performance, underlining its
efficacy.

Our analysis regarding the impact of the CAE training set size shown in Table
3 has demonstrated that the inclusion of a substantial cohort of normal MS
volumes yields notable benefits for both the self-supervision task and the
subsequent downstream task suggesting that better anomaly localisation by the
CAE and thereby better representation learning by the CNN in the self-
supervision task. We also analysed the influence of the loss function and
post-processing used in the self-supervision task which can be found in the
supplementary material section 8 and 9.

Our study has limitations that require further investigation. It is based on a
single-center, MRI-only study, so multi-center studies with varied imaging
modalities are needed for generalizability. Our methods rely on a cohort of
healthy MS volumes, unlike other self-supervised tasks. We focused on
convolutional autoencoders, not exploring models like variational autoencoders
Generative Adversarial Networks, or transformer-based architectures and
diffusion models , which might offer better anomaly localization. We compared
L1, L2, and BCE loss functions but not others like the Structural Similarity
Index or perceptual loss. Future research should examine these aspects and
apply this self-supervision approach to other domains, like brain anomaly
detection.

##  5 Conclusion

We developed a novel self-supervision task that focuses on anomaly
localization to better classify paranasal anomalies in the maxillary sinus,
addressing the lack of methods that effectively use unlabelled datasets to
learn discriminative features for this purpose. Our approach uses an
autoencoder trained on healthy MS volumes to generate residual volumes from an
unlabelled dataset. These residuals serve as coarse segmentation masks for
localizing anomalies. By training a CNN to reconstruct these volumes, it
implicitly learns anomaly localization, thereby developing transferable
features for the downstream classification task. Our method outperforms
existing self-supervision techniques, proving its effectiveness in this
specific domain.

Acknowledgements

This work has not been submitted for publication anywhere else. This work is
funded partially by the i3 initiative of the Hamburg University of Technology.
The authors also acknowledge the partial funding by the Free and Hanseatic
City of Hamburg (Interdisciplinary Graduate School) from University Medical
Center Hamburg-Eppendorf. This work was partially funded by Grant Number
KK5208101KS0 (Zentrales Innovationsprogramm Mittelstand, Arbeitsgemeinschaft
industrieller Forschungsvereinigungen).

Ethical approval declarations The study protocol received approval from the
local ethics committee (Landesärztekammer Hamburg, PV5131) and was approved by
the Data Protection Commissioners for the University Medical Center of the
University Hamburg-Eppendorf and the Free and Hanseatic City of Hamburg. It is
registered on ClinicalTrial.gov (NCT03934957) and adheres to Good Clinical
Practice, Good Epidemiological Practice, and ethical principles outlined in
the Declaration of Helsinki.

Conflicts of Interest

The authors declare no conflict of interest.

## References

  *   * (1) Marieb, E.N.: Essentials of Human Anatomy & Physiology. Third edition. Redwood City, Calif. : Benjamin/Cummings Pub. Co., [1991] ©1991. https://search.library.wisc.edu/catalog/9910059601802121
  * (2) Bal, M., Berkiten, G., Uyanık, E.: Mucous retention cysts of the paranasal sinuses. Hippokratia 18(4), 379 (2014)
  * (3) Varshney, H., Varshney, J., Biswas, S., Ghosh, S.K.: Importance of CT scan of paranasal sinuses in the evaluation of the anatomical findings in patients suffering from sinonasal polyposis. Indian J Otolaryngol Head Neck Surg 68(2), 167–172 (2015)
  * (4) Van Dis, M.L., Miles, D.A.: Disorders of the maxillary sinus. Dent Clin North Am 38(1), 155–166 (1994)
  * (5) Hansen, A.G., Helvik, A.-S., Nordgård, S., Bugten, V., Stovner, L.J., Håberg, A.K., Gårseth, M., Eggesbø, H.B.: Incidental findings in mri of the paranasal sinuses in adults: a population-based study (hunt mri). BMC ear, nose, and throat disorders 14(1), 13 (2014). https://doi.org/10.1186/1472-6815-14-13
  * (6) Tarp, B., Fiirgaard, B., Christensen, T., Jensen, J.J., Black, F.T.: The prevalence and significance of incidental paranasal sinus abnormalities on mri. Rhinology 38(1), 33–38 (2000)
  * (7) Brierley, J., Gospodarowicz, M.K., Wittekind, C. (eds.): TNM Classification of Malignant Tumours, Eighth edition edn. John Wiley & Sons Inc, Chichester West Sussex UK and Hoboken NJ (2017)
  * (8) Gutmann, A.: Ethics. the bioethics commission on incidental findings. Science (New York, N.Y.) 342(6164), 1321–1323 (2013). https://doi.org/10.1126/science.1248764
  * (9) Papadopoulou, A.-M., Chrysikos, D., Samolis, A., Tsakotos, G., Troupis, T.: Anatomical variations of the nasal cavities and paranasal sinuses: A systematic review. Cureus 13(1), 12727 (2021)
  * (10) Jeon, Y., Lee, K., Sunwoo, L., Choi, D., Oh, D.Y., Lee, K.J., Kim, Y., Kim, J.-W., Cho, S.J., Baik, S.H., Yoo, R.-E., Bae, Y.J., Choi, B.S., Jung, C., Kim, J.H.: Deep learning for diagnosis of paranasal sinusitis using multi-view radiographs. Diagnostics (Basel, Switzerland) 11(2) (2021). https://doi.org/10.3390/diagnostics11020250
  * (11) Kim, Y., Lee, K.J., Sunwoo, L., Choi, D., Nam, C.-M., Cho, J., Kim, J., Bae, Y.J., Yoo, R.-E., Choi, B.S., Jung, C., Kim, J.H.: Deep learning in diagnosis of maxillary sinusitis using conventional radiography. Investigative radiology 54(1), 7–15 (2019). https://doi.org/10.1097/RLI.0000000000000503
  * (12) Liu, G.S., Yang, A., Kim, D., Hojel, A., Voevodsky, D., Wang, J., Tong, C.C.L., Ungerer, H., Palmer, J.N., Kohanski, M.A., Nayak, J.V., Hwang, P.H., Adappa, N.D., Patel, Z.M.: Deep learning classification of inverted papilloma malignant transformation using 3d convolutional neural networks and magnetic resonance imaging. International forum of allergy & rhinology (2022). https://doi.org/10.1002/alr.22958
  * (13) Kim, K.-S., Kim, B.K., Chung, M.J., Cho, H.B., Cho, B.H., Jung, Y.G.: Detection of maxillary sinus fungal ball via 3-d cnn-based artificial intelligence: Fully automated system and clinical validation. PLOS ONE 17(2), 1–19 (2022). https://doi.org/10.1371/journal.pone.0263125
  * (14) Bhattacharya, D., Becker, B.T., Behrendt, F., Bengs, M., Beyersdorff, D., Eggert, D., Petersen, E., Jansen, F., Petersen, M., Cheng, B., Betz, C., Schlaefer, A., Hoffmann, A.S.: Supervised contrastive learning to classify paranasal anomalies in the maxillary sinus. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, pp. 429–438. Springer, Cham (2022)
  * (15) Bhattacharya, D., Behrendt, F., Becker, B.T., Beyersdorff, D., Petersen, E., Petersen, M., Cheng, B., Eggert, D., Betz, C., Hoffmann, A.S., Schlaefer, A.: Multiple Instance Ensembling For Paranasal Anomaly Classification In The Maxillary Sinus (2023)
  * (16) Pang, G., Shen, C., Cao, L., Hengel, A.V.D.: Deep learning for anomaly detection: A review. ACM Comput. Surv. 54(2) (2021). https://doi.org/10.1145/3439950
  * (17) Pihlgren, G., Sandin, F., Liwicki, M.: Pretraining image encoders without reconstruction via feature prediction loss. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 4105–4111. IEEE Computer Society, Los Alamitos, CA, USA (2021). https://doi.org/10.1109/ICPR48806.2021.9412239. https://doi.ieeecomputersociety.org/10.1109/ICPR48806.2021.9412239
  * (18) Xie, Y., Thuerey, N.: Reviving autoencoder pretraining. Neural Computing and Applications 35(6), 4587–4619 (2023). https://doi.org/10.1007/s00521-022-07892-0
  * (19) Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A.: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010)
  * (20) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M.: Bootstrap your own latent - a new approach to self-supervised learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 21271–21284. Curran Associates, Inc., ??? (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
  * (21) Chen, X., He, K.: Exploring simple siamese representation learning. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745–15753 (2021). https://doi.org/10.1109/CVPR46437.2021.01549
  * (22) Huang, S.-C., Pareek, A., Jensen, M., Lungren, M.P., Yeung, S., Chaudhari, A.S.: Self-supervised learning for medical image classification: a systematic review and implementation guidelines. npj Digital Medicine 6(1), 74 (2023). https://doi.org/10.1038/s41746-023-00811-0
  * (23) Tian, K., Jiang, Y., qishuai diao, Lin, C., Wang, L., Yuan, Z.: Designing BERT for convolutional networks: Sparse and hierarchical masked modeling. In: The Eleventh International Conference on Learning Representations (2023). https://openreview.net/forum?id=NRxydtWup1S
  * (24) Baur, C., Denner, S., Wiestler, B., Navab, N., Albarqouni, S.: Autoencoders for unsupervised anomaly segmentation in brain MR images: A comparative study. Med Image Anal 69, 101952 (2021)
  * (25) Behrendt, F., Bengs, M., Rogge, F., Krüger, J., Opfer, R., Schlaefer, A.: Unsupervised anomaly detection in 3d brain mri using deep learning with impured training data. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–4 (2022). https://doi.org/10.1109/ISBI52829.2022.9761443
  * (26) Bhattacharya, D., Behrendt, F., Becker, B.T., Beyersdorff, D., Petersen, E., Petersen, M., Cheng, B., Eggert, D., Betz, C., Hoffmann, A.S., Schlaefer, A.: Unsupervised Anomaly Detection of Paranasal Anomalies in the Maxillary Sinus. arXiv (2022). https://doi.org/10.48550/ARXIV.2211.01371. https://arxiv.org/abs/2211.01371
  * (27) Jagodzinski, A.: Rationale and design of the hamburg city health study. Eur J Epidemiol 35(2), 169–181 (2019)
  * (28) Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6450–6459. IEEE Computer Society, Los Alamitos, CA, USA (2018). https://doi.org/10.1109/CVPR.2018.00675. https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00675
  * (29) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
  * (30) Ginsburg, B., Gitman, I., You, Y.: Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling (2018). https://openreview.net/forum?id=rJ4uaX2aW
  * (31) Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=Bkg6RiCqY7
  * (32) Ozbulak, U., Lee, H.J., Boga, B., Anzaku, E.T., Park, H.-m., Messem, A.V., Neve, W.D., Vankerschaver, J.: Know your self-supervised learning: A survey on image-based generative and discriminative training. Transactions on Machine Learning Research (2023). Survey Certification
  * (33) Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. ICML 2020. JMLR.org
  * (34) Raina, R., Battle, A., Lee, H., Packer, B., Ng, A.Y.: Self-taught learning: Transfer learning from unlabeled data. In: Proceedings of the 24th International Conference on Machine Learning. ICML ’07, pp. 759–766. Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1273496.1273592. https://doi.org/10.1145/1273496.1273592