# Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders
Learn More With Less

Mattia Opper a N. Siddharth a,b
a University of Edinburgh; b The Alan Turing Institute
<EMAIL_ADDRESS>
###### Abstract

This paper presents two simple improvements to the Self-Structuring
AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to
the vocabulary as an auxiliary objective improves representation quality.
Secondly, we demonstrate that increasing the number of independent channels
leads to significant improvements in embedding quality, while simultaneously
reducing the number of parameters. Surprisingly, we demonstrate that this
trend can be followed to the extreme, even to point of reducing the total
number of non-embedding parameters to seven. Our system can be pre-trained
from scratch with as little as 10M tokens of input data, and proves effective
across English, Spanish and Afrikaans.

Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn
More With Less

Mattia Opper a and N. Siddharth a,b a University of Edinburgh; b The Alan
Turing Institute<EMAIL_ADDRESS>

##  1 Introduction

Natural language is generally understood to be compositional. To understand a
sentence, all you need to know are the meanings of the words and how they fit
together. The mode of combination is generally conceived as an explicitly
structured hierarchical process which can be described through, for example, a
parse tree. Recent work by Opper et al. (2023) presents the Self-StrAE (Self-
Structuring AutoEncoder), a model which learns embeddings such that they
define their own hierarchical structure and extend to multiple levels (i.e.
from the subword to the sentence level and beyond). The strengths of this
model lie in its parameter and data efficiency achieved through the inductive
bias towards hierarchy.

Learning embeddings such that they meaningfully represent semantics is crucial
for many modern NLP applications. For example, retrieval augmented generation
Lewis et al. (2020) is predicated on the fact that the correct contexts for a
given query can be determined. The semantic relation between a query and a
context is encompassed by the notion of semantic relatedness. They are not
equivalent to one another (i.e. paraphrases), but are close in meaning in a
broader, more contextual sense. The focus of task one of this year’s SemEval
Ousidhoum et al. (2024a, b) is capturing this notion of semantic relatedness,
with a particular focus on African and Asian languages generally characterised
by a lack of NLP resources.

In this work, we investigate whether Self-StrAE can learn embeddings which
capture semantic relatedness, when trained from scratch on moderately sized
pre-training corpora. We turn to the competition in order to examine whether
the model can even compare with dedicated STR systems. In order to determine
whether Self-StrAE can provide an alternative approach in low resource
settings where systems that rely on large pre-trained transformers Vaswani et
al. (2017) may not have sufficient scale to prove effective. We show that with
two simple changes, Self-StrAE’s performance can be substantially improved.
Moreover, we demonstrate that the the resulting system is not limited to
English, but can work equally well (if not better) for both Spanish and
Afrikaans 111Code available at: https://github.com/mopper97/Self-StrAE.

$\bar{e}_{0}$$C_{\Phi}$$\bar{e}_{1}$ $\Omega_{\Psi}$ $w_{1}$
Homer$\bar{e}_{2}$$C_{\Phi}$$\bar{e}_{3}$ $\Omega_{\Psi}$ $w_{2}$
ate$\bar{e}_{4}$ $\Omega_{\Psi}$ $w_{3}$
doughnuts0.70.40.6$\underaccent{\bar}{e}_{0}$$D_{\Theta}$$\underaccent{\bar}{e}_{1}$
$\Lambda_{\Gamma}$ $\hat{w}_{1}$
Homer$\underaccent{\bar}{e}_{2}$$D_{\Theta}$$\underaccent{\bar}{e}_{3}$
$\Lambda_{\Gamma}$ $\hat{w}_{2}$
ate$\underaccent{\bar}{e}_{4}$ $\Lambda_{\Gamma}$ $\hat{w}_{3}$
doughnutsroot embedding shared

Figure 1: Self-StrAE forward pass. Red lines indicate cosine similarity
between adjacent nodes. Shared colours indicate shared parameters.

##  2 Model and Objectives

###  2.1 Model

The core architecture at the heart of this paper is the Self-StrAE. A model
that processes a given sentence to generate both multi-level embeddings and a
structure over the input. The forward pass begins by first embedding tokens to
form an initial frontier, using the embedding matrix $\Omega_{\Psi}$. This is
followed by iterative application of the following update rule:

  1. 1.

Take the cosine similarity between adjacent embeddings in the frontier.

  2. 2.

Pop the most similar pair.

  3. 3.

Merge the pair into a single parent representation, and insert into the
frontier.

  4. 4.

If len(frontier) = 1, stop

Merge is handled by the recursively applied composition function $C_{\Phi}$,
which takes the embeddings of two children and produces that of the parent.
The process is illustrated in 1. In the figure, the highest cosine similarity
is between the embeddings of ’ate’ and ’doughnuts’, so these two embeddings
are merged first. At the next step, ’Homer’ and ’ate doughnuts’ are merged as
they have the highest similarity of the remaining embeddings. At this point
the frontier has shrunk to a single embedding and the root has been reached.

If we consider the merge history at the root, we can see that it has come to
define a tree structure over the input. This structure is passed to the
decoder, which then generates a second set of embeddings, starting from the
root and proceeding to the leaves. The decoder achieves this through recursive
application of the decomposition function $D_{\Theta}$, which takes the
embedding of a parent and produces the embeddings of the two children. Once
the decoder reaches the leaves, it can optionally output discrete tokens
through use of a dembedding function $\Lambda_{\Gamma}$.

We denote embeddings produced during composition as $\bar{e}$ and produced
during decomposition as $\underaccent{\bar}{e}$. For a vocabulary of size V,
each embedding $e\in\mathbb{R}^{E}$ consists of k independent channels of size
u. With this notation established, we can now define the four core components
of a Self-StrAE.

Embedding:

$\Omega_{\Psi}(w_{i})=w_{i}\Psi$, where $\Psi\in\mathbb{R}^{V\times E}$

Composition:

$C_{\Phi}(\bar{e}_{c1},\bar{e}_{c2})=hcat(\bar{e}_{c1},\bar{e}_{c2})\Phi+\phi$

where $\Phi\in\mathbb{R}^{2u\times u}$ and $\phi\in\mathbb{R}^{u}$

Decomposition:

$D_{\Theta}(\underaccent{\bar}{e}_{p})=hsplit(\underaccent{\bar}{e}_{p}\Theta+\theta)$

where $\Theta\in\mathbb{R}^{u\times 2u}$ and $\theta\in\mathbb{R}^{2u}$

Dembedding:

$\Lambda_{\Gamma}(\underaccent{\bar}{e}_{i})=\underaccent{\bar}{e}_{i}\Gamma$
where $\Gamma\in\mathbb{R}^{E\times V}$

Note that in the above the dembedding layer is treated as a separate parameter
matrix to the embedding layer, however, it can just as easily be weight tied
to increase efficiency.

###  2.2 Objectives

There are a few options for pre-training Self-StrAE. The simplest solution is
to have the model reconstruct the leaf tokens, which can be achieved by simply
employing cross entropy over the output of the dembedding layer. For a given
sentence $s_{j}=\langle w_{i}\rangle_{i=1}^{T_{j}}$, this objective is
formulated as:

$\displaystyle\mathcal{L}_{\text{CE}}=-\frac{1}{T_{j}}\sum_{i=1}^{T_{j}}w_{i}\cdot\log\hat{w}_{i}.$
(1)

An alternative approach adopted by Opper et al. (2023) is to use contrastive
loss as the reconstruction objective. For a given batch of sentences $s_{j}$,
the total number of nodes (internal + leaves) in the associated structure is
denoted as $M$. This allows for the construction of a pairwise similarity
matrix $A\in\operatorname{{}\mathbb{R}}^{M\times M}$ between normalised upward
embeddings $\langle\bar{e}_{i}\rangle_{i=1}^{M}$ and normalised downward
embeddings $\langle\underaccent{\bar}{e}_{i}\rangle_{i=1}^{M}$, using the
cosine similarity metric (where embeddings are flattened to be of shape E).
Denoting $A_{i\bullet},A_{\bullet{}j},A_{ij}$ the $i\textsuperscript{th}$ row,
$j\textsuperscript{th}$ column, and $(i,j)\textsuperscript{th}$ entry of a
matrix respectively, the objective is defined as:

$\displaystyle\mathcal{L}_{\text{cont}}=\frac{-1}{2M}\\!\\!\left[\\!\sum_{i=1}^{M}\log\sigma_{\\!\tau}(A_{i\bullet})\\!+\\!\sum_{j=1}^{M}\log\sigma_{\\!\tau}(A_{\bullet{}j})\\!\right]\\!$
(2)

where $\sigma_{\\!\tau}(\cdot)$ is the tempered softmax (temperature $\tau$),
normalising over the unspecified (∙) dimension.

A final option is to combine these two objectives, applying the cross entropy
reconstruction over leaves and the contrastive objective over all other nodes,
where constructing a vocabulary is intractable due to the number of possible
combinations. The contrastive objective remains identical except that A is now
defined as pairwise similarity matrix
$A\in\operatorname{{}\mathbb{R}}^{I\times I}$, where I is the number of
internal nodes of the structure. In its simplest form, this objective, which
we will henceforth refer to as CECO, can then be defined as:

$\displaystyle\mathcal{L}_{\text{CECO}}=\frac{1}{2}(\mathcal{L}_{CE}+\mathcal{L}_{cont})$
(3)

##  3 Experiments

###  3.1 Setup

For all experiments, we utilise a pre-training set of $\approx$10 million
tokens. We make this choice because Self-StrAE is intended to be data
efficient, especially if it is to be useful for low resource languages where
scale may not be available. For English the data was sourced from a subset of
Wikipedia, while for Afrikaans and Spanish we obtained corpora from Leipzig
Corpora Collection222For both Spanish and Afrikaans we selected the mixed
corpus and took a uniform subsample to reduce size to the requisite scale.. We
utilise a pre-trained BPE tokenizer for each language from the BPEMB Python
package Heinzerling and Strube (2018). Though the package also provides pre-
trained embeddings, we solely use the tokenizer and learn embeddings from
scratch.

During the course of model development, we utilised additional evaluation sets
as a further guide. For English, we used Simlex Hill et al. (2015) and
Wordsim353 Agirre et al. (2009) as measures of how well the model captures
lexical semantics, and STS-12 Agirre et al. (2012), STS-16 Agirre et al.
(2016) and STS-B Cer et al. (2017). For Afrikaans, due to lack of resources,
we utilised a Dutch translation of STS-B Huertas-García et al. (2021) as the
two languages are closely related. For Spanish, we utilised a Spanish
translation of STS-B from the same source, as well as the labelled train and
dev sets from SemRel 2024 Ousidhoum et al. (2024a). While these sets contain
labels, we apply the model fully unsupervised and solely use them for zeroshot
evaluation.

We train Self-StrAE for 15 epochs using the Adam optimizer at a learning rate
of 1e-3 Kingma and Ba (2015). We set the embedding dimension to 256, with a
batch size of 512 and $\tau$ of 1.2. We conducted our primary experiments on
English and then applied the same system design to Spanish and Afrikaans.

Figure 2: Uniformity and Alignment plot for contrastive, cross entropy and
CECO pre-training objectives. Results taken across four random seeds. Lower is
better for both measures.

###  3.2 Which Objective is Best?

Objective | Simlex | Wordsim S | Wordsim R | STS-12 | STS-16 | STS-B | SemRel (Dev)
---|---|---|---|---|---|---|---
Contrastive | 13.80 $\pm$ 0.41 | 54.33 $\pm$ 0.78 | 52.40 $\pm$ 0.87 | 31.93 $\pm$ 1.03 | 52.48 $\pm$ 0.44 | 40.05 $\pm$ 2.01 | 50.13 $\pm$ 0.88
CE | 13.77 $\pm$ 9.43 | 46.43 $\pm$ 24.00 | 51.23 $\pm$ 23.04 | 17.68 $\pm$ 4.88 | 25.40 $\pm$ 15.60 | 22.43 $\pm$ 15.12 | 32.95 $\pm$ 14.93
CECO | 19.15 $\pm$ 2.39 | 58.33 $\pm$ 3.31 | 62.65 $\pm$ 2.76 | 41.20 $\pm$ 4.04 | 58.40 $\pm$ 1.35 | 48.35 $\pm$ 1.36 | 54.40 $\pm$ 0.81

Table 1: Comparison of Objective Performance. Results are taken across four
random intialisations. Models are trained on English.

k | u | Simlex | Wordsim S | Wordsim R | STS-12 | STS-16 | STS-B | SemRel (Dev) | # Params
---|---|---|---|---|---|---|---|---|---
8 | 32 | 17.50 $\pm$ 2.12 | 58.45 $\pm$ 1.04 | 62.10 $\pm$ 2.29 | 31.00 $\pm$ 2.67 | 52.53 $\pm$ 3.33 | 41.90 $\pm$ 2.09 | 49.30 $\pm$ 0.59 | 4192
32 | 8 | 17.28 $\pm$ 5.94 | 44.83 $\pm$ 27.11 | 49.10 $\pm$ 25.47 | 33.28 $\pm$ 17.49 | 46.75 $\pm$ 30.85 | 41.35 $\pm$ 25.57 | 43.95 $\pm$ 30.50 | 280
64 | 4 | 16.15 $\pm$ 9.82 | 48.63 $\pm$ 20.95 | 51.30 $\pm$ 23.05 | 38.88 $\pm$ 22.39 | 49.48 $\pm$ 31.05 | 43.05 $\pm$ 28.91 | 46.13 $\pm$ 30.35 | 88
128 | 2 | 17.33 $\pm$ 7.12 | 52.85 $\pm$ 19.33 | 55.15 $\pm$ 19.85 | 39.63 $\pm$ 20.83 | 50.38 $\pm$ 31.92 | 46.63 $\pm$ 27.95 | 47.78 $\pm$ 30.92 | 22
256 | 1 | 12.00 $\pm$ 12.84 | 42.80 $\pm$ 23.35 | 45.05 $\pm$ 24.58 | 29.18 $\pm$ 24.68 | 39.65 $\pm$ 32.22 | 37.35 $\pm$ 29.55 | 40.63 $\pm$ 29.07 | 7
8 | 32 | 19.4 | 59.4 | 64.3 | 27.6 | 56 | 44.5 | 50.1 | 4192
32 | 8 | 21.6 | 57.2 | 61.6 | 44.3 | 63.3 | 54.1 | 58.8 | 280
64 | 4 | 21.7 | 62.8 | 66.1 | 49.9 | 65.6 | 57.4 | 61.3 | 88
128 | 2 | 18.4 | 65.1 | 67.2 | 49 | 67.2 | 60.9 | 63.2 | 22
256 | 1 | 20.7 | 63.2 | 66.3 | 50.1 | 66.2 | 61.6 | 63.6 | 7

Table 2: Impact of number of independent channels on performance. Results are
taken across four random initialisations. Models are trained on English. Top
half of the table represents average performance, the bottom half contains the
best performing initialisation. # Params is the number of non-embedding
parameters.

The first thing we want to establish is which objective is most suitable for
training Self-StrAE, as the original version only utilises contrastive loss.
For parity with the original implementation, we treat the embeddings as square
matrices (i.e. k = u) in this experiment.

Figure 2 show the uniformity and alignment analysis Wang and Isola (2020) of
the representations learned by the different objectives. Uniformity describes
the extent to which embeddings are spread around the space, while alignment
characterises how similar positive target pairs are to each other. To be
successful, representations should optimise both properties. We can observe
that while the cross entropy objective leads to uniformity, it is
comparatively poor at optimising alignment. This essentially implies that the
decoder embeddings deviate from those of the encoder. Alignment is clearly a
desireable property, as the results in table 1 show. The contrastive loss
leads to both better sentence level representations and to more stable
performance.

However, the best setting of all is CECO (the combination of cross entropy and
contrastive). There are two factors worth considering that may explain this
finding. Firstly, including reconstruction of discrete labels inherently
provides additional meaningful information compared to just organising the
representations alone. Secondly, at the token level the contrastive loss is
most susceptible to noise (e.g. the word ’the’ may occur frequently in the
batch, but each repeated instance will be treated as a false negative), and
under such conditions the objective has been shown to lead to feature
suppression Robinson et al. (2021).

Summary: We find that combining cross entropy and contrastive loss leads to
better representations than applying each objective individually, and
consequently use this approach going forward.

###  3.3 How many channels?

Each embedding in Self-StrAE is treated as consisting of k independent
channels of size u. This is intended to allow the representations to capture
different senses of meaning. However, in the original paper the number of
channels is set to be the square root of u, and not explored further.
Consequently, we wanted to see what the optimal balance between the number of
channels and their size was. Results are shown in 2. Surprisingly, we found
that as the number of channels increased (and consequently u decreased)
performance improves quite dramatically, even to the limit of treating each
value in the embedding as independent. Furthermore, because the number of non-
embedding parameters (i.e. the composition and decomposition functions) is
directly tied to the channel size u, decreasing model complexity improves
embedding quality.

However, it should be noted that this decrease in complexity comes with a
tradeoff in terms of reliability. The smaller the size of the channel, the
more variance we observed between random initialisations, with some
initialisations failing to learn any meaningful representations whatsoever. We
have found a solution that is able to maintain performance and ensure
stability between seeds, but we leave discussion of this to the appendix, as
we do not yet have a clear picture of what exactly is causing instability and
wish to avoid speculation. We do however wish to emphasise that the problem is
tractable and there is ample scope for further development, and direct the
interested reader to A for more information.

Summary: Increasing the number of channels while decreasing their size leads
to significant improvements in performance, though at the cost of some
instability between seeds. For our submisson to SemRel we used the setting k =
128, u = 2 as this allowed for an acceptable failure rate while not
compromising performance (roughly 1 in 4 seeds fail). Consequently, our system
utilises only 22 non-embedding parameters.

Language | NL STS-B (Dev) | NL STS-B (Test) | Afr SemRel (Dev) | Afr SemRel (Test) | Competition Rank
---|---|---|---|---|---
Afrikaans | 52.8 | 64.5 | 23.4 | 76.5 | 2
Language | ESP STS-B (Test) | ESP SemRel (Train) | ESP SemRel (Dev) | ESP SemRel (Test) | Competition Rank
Spanish | 61.5 | 58.5 | 68.7 | 63.5 | 6

Table 3: Self-StrAE Performance on Spanish and Afrikaans. Results correspond
to those of the submitted systems, which we selected using the best run from
four random initialisations.

###  3.4 Performance Across Languages

So far our experiments have only considered English. We now examine whether
the framework is language agonistic, and pre-train Self-StrAE on both Spanish
and Afrikaans. As before we pretrain on a small scale data (described in 3.1).

Results are in 3. We can see that the improvements to Self-StrAE hold across
different languages and are not the result of some quirk in our English pre-
training set. In fact performance is either comparable or better than on
English. The results on Afrikaans are particularly interesting as the model
performs significantly better on this language. Whether this is due to how the
test set was created or to underlying features of the language provides an
interesting question for future work. Moreover, the Afrikaans model, despite
never having been trained on Dutch, is able to generalise fairly well to it,
shown by the results on the translated STS-B sets.

##  4 Related Work

Recursive Neural Networks: Self-StrAE belongs to the class of recursive neural
networks first popularised by Socher et al. (2011, 2013). Recursive neural
networks are extremely similar to recurrent neural networks, they differ
because they process inputs hierarchically rather than sequentially (e.g.
going up a parse tree).

Learning Structure and Representations: Recursive neural networks require
structure as input. An alternative approach is to train a model that learns
structure and the network at the same time. Recent unsupervised examples
include Drozdov et al. (2019, 2020); Hu et al. (2021). However, these
mechanisms generally use search to determine structure making them highly
memory intensive. Self-StrAE differs from these as it asks the representations
to define their own structure, making it much more resource efficient, though
less flexible in certain aspects.

Contrastive Loss: Contrastive loss is an objective which optimises the
representation space directly. In broad terms this objective requires the
representations of a positive pair to be as similar to each other as possible,
while minimising similarity to a set of negative examples. The closest
examples of this objective, for the approach employed in this paper, are Chen
et al. (2020); Shi et al. (2020); Radford et al. (2021).

##  5 Conclusion

We show that two simple changes can make Self-StrAE significantly more
performant: adding a discrete reconstruction objective and increasing the
number of independent channels. The latter also has the added benefit of
reducing the number of parameters in the model, and surprisingly means that
simpler is better. More broadly, we believe these findings demonstrate the
potential of an inductive biases towards explicit structure. Self-StrAE, at
present, is a very simple model. The only thing it really has going for it is
the inductive bias which tasks embeddings with organising themselves
hierarchically. While the gap between Self-StrAE and SoTA systems still
remains, the fact that it is able to perform at all demonstrates the promise.
Moreover, the fact that the two simple changes demonstrated in this paper can
lead to such improvements indicates that the full potential of the inductive
bias has yet to be reached, and it is likely that further refinements can lead
to even more substantial benefits. Finally, because this model does not
require significant scale to optimise pursuing further improvements may
provide substantial benefits for low resource languages where pre-training
data is scarce.

##  6 Limitations

The results in this paper represent steps towards an improved model rather
than a complete picture. We still do not fully understand what causes the
instability in training when the number of channels increased, and though we
can provide a solution (see A), further analysis is needed. The performance of
contrastive loss can depend quite heavily on how positive and negative
examples are defined and it is likely that the explanation rests there.
Secondly, while we have shown that Self-StrAE can be applied to languages
other than English the results are limited to Indo-European languages. An
interesting avenue for future work would be investigating a broader spectrum
of languages, and whether specific characteristics can be identified which
influence how well the model performs.

##  7 Acknowledgements

MO was funded by a PhD studentship through Huawei-Edinburgh Research Lab
Project 10410153. We thank Victor Prokhorov, Ivan Vegner and Vivek Iyer for
their valuable comments and helpful suggestions during the creation of this
work.

## References

  * Agirre et al. (2009) Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009.  A study on similarity and relatedness using distributional and WordNet-based approaches.  In _North American Chapter of the Association for Computational Linguistics (NAACL)_ , pages 19–27.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California. Association for Computational Linguistics.
  * Agirre et al. (2012) Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  In _Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_ , SemEval ’12, page 385–393, USA. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Workshop on Semantic Evaluation (SemEval)_ , pages 1–14.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International Conference on Learning Representations (ICLR)_ , volume 119, pages 1597–1607.
  * Drozdov et al. (2020) Andrew Drozdov, Subendhu Rongali, Yi-Pei Chen, Tim O’Gorman, Mohit Iyyer, and Andrew McCallum. 2020.  Unsupervised parsing with S-DIORA: Single tree encoding for deep inside-outside recursive autoencoders.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4832–4845, Online. Association for Computational Linguistics.
  * Drozdov et al. (2019) Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019.  Unsupervised latent tree induction with deep inside-outside recursive auto-encoders.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1129–1141, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Heinzerling and Strube (2018) Benjamin Heinzerling and Michael Strube. 2018.  BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  SimLex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hu et al. (2021) Xiang Hu, Haitao Mi, Zujie Wen, Yafang Wang, Yi Su, Jing Zheng, and Gerard de Melo. 2021.  R2D2: Recursive transformer based on differentiable tree for interpretable hierarchical language modeling.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4897–4908, Online. Association for Computational Linguistics.
  * Huertas-García et al. (2021) Álvaro Huertas-García, Javier Huertas-Tato, Alejandro Martín, and David Camacho. 2021.  Countering misinformation through semantic-aware multilingual models.  In _Intelligent Data Engineering and Automated Learning – IDEAL 2021_ , pages 312–323, Cham. Springer International Publishing.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA.
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  _CoRR_ , abs/2005.11401.
  * Opper et al. (2023) Mattia Opper, Victor Prokhorov, and Siddharth N. 2023.  StrAE: Autoencoding for pre-trained embeddings using explicit structure.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7544–7560, Singapore. Association for Computational Linguistics.
  * Ousidhoum et al. (2024a) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaima Hourrane, Gopichand Kanumolu, Lokesh Madasu, Samuel Rutunda, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Hailegnaw Getaneh Tilaye, Krishnapriya Vishnubhotla, Genta Winata, Seid Muhie Yimam, and Saif M. Mohammad. 2024a.  Semrel2024: A collection of semantic textual relatedness datasets for 14 languages.
  * Ousidhoum et al. (2024b) Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnubhotla, Seid Muhie Yimam, and Saif M. Mohammad. 2024b.  SemEval-2024 task 1: Semantic textual relatedness for african and asian languages.  In _Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)_. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning (ICML)_ , pages 8748–8763. PMLR.
  * Robinson et al. (2021) Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. 2021.  Can contrastive learning avoid shortcut solutions? _CoRR_ , abs/2106.11230.
  * Shi et al. (2020) Yuge Shi, Brooks Paige, Philip Torr, and N. Siddharth. 2020.  Relating by contrasting: A data-efficient framework for multimodal generative models.  In _International Conference on Learning Representations (ICLR)_.
  * Socher et al. (2011) Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011.  Semi-supervised recursive autoencoders for predicting sentiment distributions.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 151–161.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1631–1642.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _CoRR_ , abs/1706.03762.
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  _CoRR_ , abs/2005.10242.

##  Appendix A Stabilising High Channel Self-StrAE

One solution we have found to the instability issue is modifying the
objective. This formulation, loosely inspired by SimCSE Gao et al. (2021),
runs the same input through the model twice, with different dropout masks
applied each time. The objective is cross entropy reconstruction for the
leaves, and contrastive loss between the two different sets of decoder
embeddings for the non-terminals. Currently we have two theories as to why
this might work:

  * •

Better negatives: because the decoder embeddings represent the contextualised
meaning of node rather than it’s local one, the issue of false negatives is
somewhat mitigated.

  * •

Encoder consistency: because we ask the two sets of decoder embeddings to be
similar to each other the encoder is encouraged to produce the same structure
regardless of dropout mask. It may be that this pressure towards regularity
leads to the improved consistency.

Results are shown in 4. For lack of a better term we refer to this alternative
objective as StrCSE. In its current form we do not consider this objective to
be well formed, and solely provide it here as a possible starting point for
further research.

Objective | Simlex | Wordsim S | Wordsim R | STS-12 | STS-16 | STS-B | SemRel (Dev)
---|---|---|---|---|---|---|---
Contrastive | 13.80 $\pm$ 0.41 | 54.33 $\pm$ 0.78 | 52.40 $\pm$ 0.87 | 31.93 $\pm$ 1.03 | 52.48 $\pm$ 0.44 | 40.05 $\pm$ 2.01 | 50.13 $\pm$ 0.88
CE | 13.77 $\pm$ 9.43 | 46.43 $\pm$ 24.00 | 51.23 $\pm$ 23.04 | 17.68 $\pm$ 4.88 | 25.40 $\pm$ 15.60 | 22.43 $\pm$ 15.12 | 32.95 $\pm$ 14.93
CECO | 19.15 $\pm$ 2.39 | 58.33 $\pm$ 3.31 | 62.65 $\pm$ 2.76 | 41.20 $\pm$ 4.04 | 58.40 $\pm$ 1.35 | 48.35 $\pm$ 1.36 | 54.40 $\pm$ 0.81
CECO k=128 u=2 | 17.33 $\pm$ 7.12 | 52.85 $\pm$ 19.33 | 55.15 $\pm$ 19.85 | 39.63 $\pm$ 20.83 | 50.38 $\pm$ 31.92 | 46.63 $\pm$ 27.95 | 47.78 $\pm$ 30.92
StrCSE k=128 u=2 | 21.68 $\pm$ 1.88 | 59.06 $\pm$ 2.38 | 64.08 $\pm$ 0.91 | 49.46 $\pm$ 0.59 | 66.18 $\pm$ 0.24 | 61.30 $\pm$ 0.76 | 62.88 $\pm$ 0.42

Table 4: StrCSE compared with other objectives. Results are taken over four
random intialisations. Training data is English.


# NUAA-QMUL at SemEval-2020 Task 8: Utilizing BERT and DenseNet for Internet
Meme Emotion Analysis

Xiaoyu Guo
Nanjing University of
Aeronautics and Astronautics
<EMAIL_ADDRESS>
Jing Ma
Nanjing University of
Aeronautics and Astronautics
<EMAIL_ADDRESS>
Arkaitz Zubiaga
Queen Mary
University of London
<EMAIL_ADDRESS>

###### Abstract

This paper describes our contribution to SemEval 2020 Task 8: Memotion
Analysis. Our system learns multi-modal embeddings from text and images in
order to classify Internet memes by sentiment. Our model learns text
embeddings using BERT and extracts features from images with DenseNet,
subsequently combining both features through concatenation. We also compare
our results with those produced by DenseNet, ResNet, BERT, and BERT-ResNet.
Our results show that image classification models have the potential to help
classifying memes, with DenseNet outperforming ResNet. Adding text features is
however not always helpful for Memotion Analysis.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International Licence. Licence details:
http://creativecommons.org/licenses/by/4.0/.
Source code for our model and for the training procedure is published on
https://github.com/xxxxxxxxy/Memotion-Analysis.

The growing ubiquity of Internet memes on social media platforms such as
Facebook, Instagram, and Twitter reinforces the importance of processing
multimodal content. Despite memes being so deeply ingrained in the Internet
culture, it is still hard to leverage AI methods to understand the underlying
meanings expressed by memes. This, in turn, makes it challenging to identify
certain kinds of content in these memes, such as malicious information posted
on social media. To tackle this problem, SemEval 2020 Task 8 held a meme
emotion analysis task[Sharma et al., 2020]. It consisted of three sub-tasks:
(1) predicting the sentiment of Internet memes as positive, neutral or
negative, (2) identifying the type of humor expressed as sarcastic, humorous,
offensive or motivation, and (3) quantifying the extent to which a particular
emotion is being expressed.

Internet memes can comprise a variety of modalities such as textual, visual,
and audio. Adhering to the task guidelines at hand, we approach the problem by
considering only the image-with-caption class of meme, which is commonly found
on social media. This paper describes a system that combines BERT
(Bidirectional Encoder Representation from Transformers) and DenseNet (Dense
Convolutional Network) to classify Internet memes. We find that the
performance of Multi-modal Deep Neural Network is better than one model Deep
Neural Network and the performance of ResNet (Residual Network) is worse than
DenseNet.

This paper is organized as follows. In section Background, we describe
Memotion Analysis task, the characteristics of the Internet memes, prior work
on multi-modal classification, prior research exploring data analysis
approaches to study memes, and our contributions in this paper. Then in
Section 3, we propose BERT-DenseNet, a deep learning model to classify memes.
We then use the models to study memes in task A, task B, and task C in Section
4. In Section 5, we display the results of our experiments, including
comparative experiments. Finally, we conclude the findings of this research
and suggest directions for future work.

##  2 Background

Task A of this competition consists of classifying an Internet meme as
positive, negative, or neutral. Task B consists of identifying whether an
Internet meme is sarcastic, humorous, offensive, or motivation. A meme can
have more than one category in this case. Task C consists of quantifying the
scales of every type of humor. The system should classify it as positive for
task A, identify it as sarcastic, humorous, not offensive and not motivational
for task B, and quantify it as general (the scale of sarcasm), hilarious (the
scale of humor) and not offensive (the scale of offensive) for task C.

A dataset with 6,992 annotated memes is used in this task [Sharma et al.,
2020]. The dataset contains a set of meme images, the corresponding textual
content OCR’d and manually corrected, and the corresponding class. A meme is
not only about the funny picture, the Internet culture, or the emotion that
passes along, but also about the richness and uniqueness of its language: It
is often highly structured with a special writing style and forms interesting
and subtle connotations that resonate among the readers [Wang and Wen, 2015].
For example, the Boromir memes with the caption of “one does not simply…” are
derived from the movie Lord of the Rings. It shows that the same example can
lead to the production of several memes making use of the same background
image but different texts, which gives it different meanings and contexts.

Although the language of Internet memes is a relatively new research topic,
our work is broadly related to studies on multi-modal classification [Kalva et
al., 2007, Merdivan et al., 2019, Zhou et al., 2016, Zhu et al., 2019] and
multi-modal sentiment analysis [Baecchi et al., 2016, Hu and Flaxman, 2018,
Zhongqiang et al., 2017]. Most recently, Sun et al. [Sun et al., 2019] learned
text embeddings using BERT, the current state-of-the-art in text encoders.
However, none of the above studies have applied multi-modal approaches for
memes sentiment analysis.

Previous research on memes mainly focuses on the automatic generation of memes
[Costa et al., 2015, Oliveira et al., 2016, Peirson et al., 2018, Veale et
al., 2015], but not on the analysis as we do in this task. In a typical
automatic generation work, Wang & Wen [Wang and Wen, 2015] combined textual
and visual information for predicting and generating popular meme
descriptions. Beskow, Kumar, & Carley [Beskow et al., 2020] detected and
characterized Internet memes with multi-modal deep learning. However, fewer
studies focus on sentiment and emotion analysis of memes. One of the few works
who did focus on sentiment analysis is that by Costa et al. [Costa et al.,
2015], who developed a Maximum Entropy classifier for memes sentiment
classification, which was however in an early stage of development.

Based on previous work, we take a multi-modal approach to do memes sentiment
analysis. We propose to use Keras functional API111https://keras.io/getting-
started/functional-api-guide/ to model the multimodal dependencies among
images, text, and corresponding labels.

We then introduce BERT [Devlin et al., 2019] and DenseNet [Huang et al., 2017]
to the model. In empirical experiments, we show that our model outperforms
strong discriminative baselines by large margins except for the second sub-
task (B). Our contributions are three-fold:

  1. 1.

We perform sentiment analysis of memes combining NLP, CV, and deep learning
techniques, and show that combining the visual and textual signals helps
classifying memes by sentiment;

  2. 2.

We find that the performance of DenseNet is better than ResNet, and provide an
analysis of this result;

  3. 3.

Our proposed deep learning model outperforms competitive baselines except for
task B.

##  3 System overview

We approach the task of Memotion Analysis as eight classification sub-tasks
(one for task A, four for task B and three for task C). Four classification
sub-tasks from task B are binary but others are multiclass. Our model is
depicted in Figure 1, with inputs at the top and outputs at the bottom. In a
nutshell, our model combines BERT that processes the caption text and DenseNet
that extracts features from images.

Figure 1: The BERT-DenseNet model. Softmax activation is used for tasks A and
C. Sigmoid activation is for task B.

Data Pre-processing: Firstly, we delete quotation marks because this would
raise an error when reading CSV files. Secondly, we copy text from
OCR_extracted_text to corrected_text when the value of corrected_text is null
and the value of OCR_extracted_text is not null. We check the corresponding
image and add the text to corrected_text if the value of OCR_extracted_text
and corrected_text are both null. Thirdly, we use WordPiece tokenisation,
which can break a word into more than one sub-word. This kind of tokenisation
is beneficial for memes because they contain several out of vocabulary words.

Text Encoding: To encode text in the data, we use pre-trained BERT [Devlin et
al., 2019], which is a transformer-based language model that conditions
jointly on the left and right of a given word. Typically, the BERT encoder is
fine-tuned for a particular task by learning an additional task-specific
weight layer. In this paper, the layers of BERT, pre-trained on a large corpus
of Wikipedia+Book corpus data, are set to be trainable, and additional fine-
tuning is not performed. The reason why we choose this encoder is motivated by
the success of BERT in achieving state-of-the-art results in several NLP tasks
such as sentiment analysis, machine reading comprehension, question-answering,
etc. The input text in Section 4 is encoded using BERT-cased_L-12_H-768_A-12.
After encoding with BERT, the input text is transformed into indices (index of
the word) and segments (which sentence the word belongs to). These two parts
are then sent to BERT model and the [CLS] vector is fetched with the Lambda
function. This process leads to text embeddings of size 768.

Image Encoding: Images are converted into arrays before encoding. Image
features from data are extracted using DenseNet, which builds feature-maps for
memes. The image embedding for each meme is the result of the concatenation of
feature vectors extracted from the image. The image encoding is then processed
into 2 dimensions with Global Max Pooling. The resulting image embeddings are
of size 1024. In this procedure, IOError occurs when the system reads images
that are corrupted; this error occurred with a single image in the dataset.
This was solved by setting LOAD_TRUNCATED_IMAGES to true.

The next step is to combine these two embeddings with the concatenate function
provided by Keras. After the combination layer, there is a dense layer
followed by a dropout layer. There is no need to add too many layers on the
top of the model because BERT is so complex that it is competent to extract
text features. Besides, the extra layer would have a bad influence on the pre-
trained weights of BERT and DenseNet because our added layer will be
initialized randomly. If we put several layers on the system, the performance
would be reduced, and the model may not convergent. Finally, as the activation
functions, we use softmax for task A and task C and sigmoid for task B, which
are used for outputting the predicted class.

##  4 Experimental setup

We tried three different groups of classifiers: (1) unimodal classification
using only text (2) unimodal classification using only machine vision, and (3)
multimodal classification using text and vision.

Data: We use the publicly released dataset for experiments. As there are eight
classification tasks in total, different tasks have different proportions of
labels. The details of the splitting train/dev/test are shown in the tables
below (Table 1-3). In total, the training, dev, and test data contain 5,192,
1,800, and 1,878 mentions, respectively.

| Train | Dev | Test
---|---|---|---
Positive | 3,089 | 1,071 | 1,111
Neutral | 1,634 | 567 | 594
Negative | 469 | 162 | 173
Sum | 5,192 | 1,800 | 1,878
Table 1:  Data splitting of task A.  | Funny or not | Sarcastic or not | Offensive or not | Motivational or not
---|---|---|---|---
|  Train |  Dev |  Test |  Train |  Dev |  Test |  Train |  Dev |  Test |  Train |  Dev |  Test
Yes |  3,973 |  1,368 |  1,433 |  4,044 |  1,404 |  1,457 |  3,181 |  1,098 |  1,171 |  1,837 |  630 |  1,188
No |  1,219 |  432 |  445 |  1,148 |  396 |  421 |  2,011 |  702 |  707 |  3,355 |  1,170 |  690
Sum |  5,192 |  1,800 |  1,878 |  5,192 |  1,800 |  1,878 |  5,192 |  1,800 |  1,878 |  5,192 |  1,800 |  1,878
Table 2: Data splitting of task B. | Scale of funny | Scale of sarcastic | Scale of offensive
---|---|---|---
Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
Not | 1,219 | 432 | 445 | 1,148 | 396 | 421 | 2,011 | 702 | 707
Slightly | 1,822 | 630 | 654 | 2,607 | 900 | 937 | 1,926 | 666 | 709
Mildly | 1,662 | 576 | 605 | 1,151 | 396 | 424 | 1,088 | 378 | 387
Very | 489 | 162 | 174 | 286 | 108 | 96 | 167 | 54 | 75
Sum | 5,192 | 1,800 | 1,878 | 5,192 | 1,800 | 1,878 | 5,192 | 1,800 | 1,878
Table 3: Data splitting of task C.

Parameter settings: We try different hyper-parameters and chose those which
yielded the best result. Specifically, our submitted model is trained in batch
mode using the Adam optimizer [Kingma and Ba, 2015] with a learning rate of
1.00E-05. Each batch covers 16 mentions. A dropout rate of 0.2 is applied on
the dropout layer and an L2 regularization of 0.02 is used on the dense layer
which is adjacent to the last layer. Also, to balance the ratio of different
labels and reduce potential overfitting, we add a class weight parameter
according to the actual ratio observed in the training data when fitting the
model. Additionally, hyper-parameters of comparative experiments (including
re-training the BERT-DenseNet model) are shown below in Table 4. Optimizer,
batch-size, and class weight are the same as the submitted model.

Type | Model | Learning rate | Dropout | L2 regularization
---|---|---|---|---
Text | BERT | 1.00E-06 | 0 | 0.01
Vision | DenseNet | 1.00E-06 | 0 | 0.01
ResNet | 1.00E-06 | 0 | 0.01
Multi-modal | BERT-DenseNet | 1.00E-06 | 0.3 | 0.04
BERT-ResNet | 1.00E-06 | 0.3 | 0.04
Table 4: Hyper-parameters of comparative experiments

External library: Keras, the python deep learning library, is used to build
the whole model structure. It offers an easy way to load
DenseNet222https://keras.io/applications/#densenet, but there is no official
method to call BERT. Consequently, keras-
bert333https://github.com/CyberZHG/keras-bert,an external library and a great
implementation of BERT with Keras that could load official pre-trained models,
is used in the process of implementing the system.

Evaluation: The organizers proposed to evaluate the performance with the
macro-F1 score. For task B and C, the final score is produced by averaging the
macro-F1 score for each of the subtasks.

##  5 Results

Among all participating systems in this SemEval task, our model achieved the
14th score both on the evaluation of task A and task C. Our score on task B is
however below the baseline, which we look to improve in future work. Table 5
gives our results, the highest score in this competition and baseline.

Models | Task A | Task B | Task C
---|---|---|---
Highest score | 0.3547 | 0.5183 | 0.3225
BERT-DenseNet | 0.3452 | 0.4421 | 0.3097
Baseline | 0.2176 | 0.5002 | 0.3009
Table 5: Macro-F1 scores obtained for the highest score model, our final
submitted model and baseline.

Sub-tasks of task B are all binary classification tasks and we expected it to
be easier to achieve performance improvements in this case. Surprisingly,
however, we get the worst result in task B. Actually, the final submitted
result of task B overfits and all of the results are almost positive. But it
achieves the best scores among all of our submissions. We believe that
changing the parameter to reduce the overfitting effect will yield better
results.

Experiments with other models (including re-training the BERT-DenseNet model)
are shown below in Table 6. The re-trained BERT-DenseNet model, while ending
with lower scores in task A, achieves higher performance than the final
submitted model (in Table 5) in task B and task C.

Type | Models | Task A | Task B | Task C
---|---|---|---|---
| Highest score | 0.3547 | 0.5183 | 0.3225
Text | BERT | 0.1574 | 0.4798 | 0.2749
Vision | DenseNet | 0.3344 | 0.5120 | 0.3209
ResNet | 0.3186 | 0.4965 | 0.3129
Multi-modal | BERT-DenseNet | 0.3137 | 0.4999 | 0.3127
BERT-ResNet | 0.3305 | 0.4946 | 0.3149
| Baseline | 0.2176 | 0.5002 | 0.3009
Table 6: Macro-F1 scores obtained for other models.

We can observe that: (i) DenseNet outperforms ResNet; (ii) introducing BERT
would reduce the performance of DenseNet and has little impact on ResNet;
(iii) BERT, while yielding SOTA results in a lot of NLP tasks, is not suitable
for processing text of memes.

The reasons why DenseNet outperforms ResNet in this SemEval task are two-fold:

  1. 1.

Characteristics of DenseNet: The features extracted by DenseNet are
concatenated while ResNet sums the features. Moreover, DenseNets allow layers
access to feature-maps from all of its preceding layers. These two
characteristics encourage feature reuse throughout the network and lead to
more compact models;

  2. 2.

Dataset: In total, 6,992 mentions are provided by the organizers. The limited
availability of data makes it more difficult to extract sufficient features.
Meanwhile, our model is deep. As a result, overfitting may occur easily.
Evidence from [Huang et al., 2017] shows that DenseNet is better for small
datasets, which is why we opted for this option in our approach.

The results show that DenseNet achieves the best scores in task A among all of
our submissions. Therefore, we cannot conclude that uni-modal approaches are
better for characterizing internet memes than multi-modal approaches while
introducing BERT would reduce the performance of DenseNet. Intuitively, more
features encourage models to learn more information from data. We may put more
effort into how to connect multi-modal features in that our proposed model
concatenates text-based features with image-based features without exploring
other options.

##  6 Conclusion

In this paper, we describe our entry for SemEval 2020 Task 8: Memotion
Analysis. We present a method for using deep learning to classify memes by
emotion. We find that image classification models help classifying memes, with
DenseNet outperforming ResNet. Adding text-based features is not always
helpful for Memotion Analysis. Our proposed model outperforms baselines except
for task B. We are looking to improve performance scores for task B by further
tweaking our method and by testing additional parameters.

We should stress that the classifier we trained is still in an early stage of
development. We aim to improve it by: (i) enriching the meme datasets from
social media and trying to keep the balance among classes; (ii) considering
additional features (e.g. adult slang); (iii) trying other methods for
combining features. Our present work is limited to one type of meme made of
image and text, and we aim to take other types of memes (e.g. video and audio
memes) into consideration in the future.

##  7 Acknowledgments

This work was supported by International Cooperation Office of Nanjing
University of Aeronautics and Astronautics and College of Economics and
Management of Nanjing University of Aeronautics and Astronautics. Xiaoyu Guo
conducted part of this work while visiting Queen Mary University of London.

## References

  * [Baecchi et al., 2016]  Claudio Baecchi, Tiberio Uricchio, Marco Bertini, and Alberto Del Bimbo.  2016\.  A multimodal feature learning approach for sentiment analysis of social network multimedia.  Multimedia Tools and Applications, 75(5):2507–2525.
  * [Beskow et al., 2020]  David M. Beskow, Sumeet Kumar, and Kathleen M. Carley.  2020\.  The evolution of political memes: Detecting and characterizing internet memes with multi-modal deep learning.  Information Processing and Management, 57(2):102170.
  * [Costa et al., 2015]  Diogo Costa, Hugo Gonçalo Oliveira, and Alexandre Miguel Pinto.  2015\.  In reality there are as many religions as there are papers-first steps towards the generation of internet memes.  In ICCC, pages 300–307.
  * [Devlin et al., 2019]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2019\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  * [Hu and Flaxman, 2018]  Anthony Hu and Seth Flaxman.  2018\.  Multimodal sentiment analysis to explore the structure of emotions.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 350–358.
  * [Huang et al., 2017]  Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  2017\.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.
  * [Kalva et al., 2007]  Pedro R. Kalva, Fabricio Enembreck, and Alessandro L. Koerich.  2007\.  WEB image classification based on the fusion of image and text classifiers.  Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, 1(Icdar):561–565.
  * [Kingma and Ba, 2015]  Diederik P. Kingma and Jimmy Lei Ba.  2015\.  Adam: A method for stochastic optimization.  3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pages 1–15.
  * [Merdivan et al., 2019]  Erinc Merdivan, Anastasios Vafeiadis, Dimitrios Kalatzis, Sten Hanke, Joahannes Kroph, Konstantinos Votis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, Raouf Hamzaoui, and Matthieu Geist.  2019\.  Image-Based Text Classification using 2D Convolutional Neural Networks.  In 2019 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), pages 144–149.
  * [Oliveira et al., 2016]  Hugo Gonçalo Oliveira, Diogo Costa, and Alexandre Miguel Pinto.  2016\.  One does not simply produce funny memes!–explorations on the automatic generation of internet humor.  In Proceedings of the Seventh International Conference on Computational Creativity (ICCC 2016). Paris, France, pages 243–250.
  * [Peirson et al., 2018]  V Peirson, L Abel, and E Meltem Tolunay.  2018\.  Dank learning: Generating memes using deep neural networks.  arXiv preprint arXiv:1806.04510.
  * [Sharma et al., 2020]  Chhavi Sharma, Deepesh Bhageria, William Paka, Scott, Srinivas P Y K L, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck.  2020\.  SemEval-2020 Task 8: Memotion Analysis-The Visuo-Lingual Metaphor!  In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics.
  * [Sun et al., 2019]  Zhongkai Sun, Prathusha K Sarma, William Sethares, and Erik P Bucy.  2019\.  Multi-Modal Sentiment Analysis Using Deep Canonical Correlation Analysis.  Proc. Interspeech 2019, pages 1323–1327.
  * [Veale et al., 2015]  Tony Veale, Alessandro Valitutti, and Guofu Li.  2015\.  Twitter: The best of bot worlds for automated wit.  In International Conference on Distributed, Ambient, and Pervasive Interactions, pages 689–699. Springer.
  * [Wang and Wen, 2015]  William Yang Wang and Miaomiao Wen.  2015\.  I can has cheezburger? a nonparanormal approach to combining textual and visual information for predicting and generating popular meme descriptions.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 355–365.
  * [Zhongqiang et al., 2017]  W U Zhongqiang, Zhang Yaowen, and Shang Lin.  2017\.  Multi- view sentiment classification of microblogs based on semantic features.  CAAI Transactions on Intelligent Systems, 12(5):745–751.
  * [Zhou et al., 2016]  Ye Zhou, Xin Lu, Junping Zhang, and James Z. Wang.  2016\.  Joint image and text representation for aesthetics analysis.  MM 2016 - Proceedings of the 2016 ACM Multimedia Conference, pages 262–266.
  * [Zhu et al., 2019]  Fangyi Zhu, Zhanyu Ma, Xiaoxu Li, Guang Chen, Jen Tzung Chien, Jing Hao Xue, and Jun Guo.  2019\.  Image-text dual neural network with decision strategy for small-sample image classification.  Neurocomputing, 328:182–188.