11institutetext: German Research Center for Artificial Intelligence (DFKI),
Kaiserslautern, Germany 22institutetext: RPTU Kaiserslautern-Landau, Germany
22email<EMAIL_ADDRESS>

# Improving Disease Detection from Social Media Text via Self-Augmentation and
Contrastive Learning

Pervaiz Iqbal Khan 1122 0000-0002-1805-335X Andreas Dengel 1122
0000-0002-6100-8255 Sheraz Ahmed 11 0000-0002-4239-6520

###### Abstract

Detecting diseases from social media has diverse applications, such as public
health monitoring and disease spread detection. While language models (LMs)
have shown promising performance in this domain, there remains ongoing
research aimed at refining their discriminating representations. In this
paper, we propose a novel method that integrates Contrastive Learning (CL)
with language modeling to address this challenge. Our approach introduces a
self-augmentation method, wherein hidden representations of the model are
augmented with their own representations. This method comprises two branches:
the first branch, a traditional LM, learns features specific to the given
data, while the second branch incorporates augmented representations from the
first branch to encourage generalization. CL further refines these
representations by pulling pairs of original and augmented versions closer
while pushing other samples away. We evaluate our method on three NLP datasets
encompassing binary, multi-label, and multi-class classification tasks
involving social media posts related to various diseases. Our approach
demonstrates notable improvements over traditional fine-tuning methods,
achieving up to a $2.48\%$ increase in F1-score compared to baseline
approaches and a $2.1\%$ enhancement over state-of-the-art methods.

###### Keywords:

Language models Self-augmentation Contrastive learning Disease detection
Social media text classification.

##  1 Introduction

Depression is a widespread mental disorder affecting approximately 280 million
individuals globally. Tragically, over 700,000 people per year commit suicide
due to depression, making it the fourth leading cause of death among
individuals aged 15 to 29111https://www.who.int/news-room/fact-
sheets/detail/depression. The condition significantly impacts both personal
and professional lives. Alarmingly, reports published in August 2021 revealed
that 1.6 million individuals in England were on waiting lists for mental
health support, with approximately 8 million unable to access treatment from
specialists due to not meeting the criteria for treatment [5].

The utilization of the internet and social media platforms such as
Reddit222https://www.reddit.com/ and X333https://twitter.com has surged in
recent years. Individuals use these platforms not only to provide feedback on
services and products but also to share emotions and health-related
information. Analyzing these emotions is crucial for the early detection of
depression and other diseases. Early detection can mitigate losses by
facilitating better resource planning and the establishment of health-related
policies. This prompts the question: can social media content aid in disease
detection? Natural language processing (NLP) offers an effective means to
analyze emotions from text. However, the task of disease detection poses
challenges, as disease words and emotions are sometimes used figuratively
rather than literally. Overcoming this challenge necessitates learning
appropriate word representations based on their context.

Transformer-based models such as BERT [4], RoBERTa [22], and XLNet [33] excel
at learning suitable representations and have achieved state-of-the-art (SotA)
performance in various NLP tasks. These models are pretrained on large amounts
of unlabeled text data using auxiliary tasks such as Masked Language Modeling
and Next Sentence Prediction. They can then be fine-tuned on labeled datasets
to further enhance representations specific to the downstream task. Various
approaches [18, 16, 14]. These methods inject perturbations into the embedding
matrix, hidden representations, and parameter-space of the models using
techniques such as Gaussian noise and the Fast Gradient Sign Method [7].

In this paper, we propose a novel and effective approach called self-
augmentation to generate adversarial examples during the fine-tuning process
of these models, thereby enhancing their text representation capabilities. The
model is fine-tuned on two streams: the first stream is fine-tuned
conventionally by minimizing the classification loss, while the second stream
augments its intermediate states with extracted representations from the first
stream and then completes its fine-tuning pass. Contrastive loss is then
employed to further align the intermediate representations of both streams to
improve representations. The overall fine-tuning objective is to minimize two
classification losses and one contrastive loss.

The main contributions of this paper are summarized as follows:

  * •

We conducted experimentation on three public social media text datasets,
spanning binary-class, multi-class, and multi-label classification tasks. Our
approach achieved SotA results on these datasets, demonstrating the
effectiveness of our proposed method in real-world scenarios. Additionally, we
presented a thorough analysis of the experimental results, including
visualizations of representations and ablation studies, to provide insights
into the performance and behavior of our approach.

  * •

We proposed a novel method called self-augmentation, which generates
adversarial examples to improve text representation learning of LMs. By
jointly training clean and adversarial examples, our approach enhances the
discriminative features learned by LMs, leading to improved performance on
downstream tasks. This contribution addresses the need for robust and
generalizable text representations in NLP applications, particularly in the
context of social media data.

  * •

We incorporated contrastive learning (CL) method to further enhance the
representation of text. By leveraging CL, our approach encourages the model to
learn semantically meaningful representations by maximizing the agreement
between similar instances and minimizing the agreement between dissimilar
instances. This contribution enhances the generalization capabilities of our
proposed method, leading to more accurate predictions.

##  2 Related Work

This section provides an overview of existing approaches in the literature for
disease detection from social media platforms such as Reddit, X, etc. We
present various methodologies ranging from traditional machine learning and
deep learning techniques to more recent advancements such as adversarial
training and contrastive learning with an aim to provide a comprehensive
understanding of the domain.

###  2.1 Disease Detection from Social Media

Karisani et al. [13] curated a dataset comprising 7,192 English Tweets
focusing on health-related discussions and disease-related terms. They
proposed the WESPAD approach, which involved partitioning and distorting
embeddings to enhance generalization on unseen data and address data scarcity
issues.

Jiang et al. [12] introduced another English Tweets dataset with 12,331
labeled examples, utilizing non-contextual embeddings and Long Short-Term
Memory Networks (LSTMs) [9] to classify tweets into personal experience tweets
(PET) or non-PET. They also incorporated feature engineering techniques,
improving classification results compared to traditional models like support
vector machines, decision trees, and k-nearest neighbors.

Iyer et al. [10] presented FeatAug+, a method that calculated word
similarities using an external data source, Sentiment140. These similarity
scores were concatenated with language-based features and fed into a
convolutional neural network (CNN)-based classifier.

Biddle et al. [3] expanded the PHM-2017 dataset [13] by incorporating tweets
related to ten additional diseases, along with figurative mentions as a new
label. They leveraged sentiment information and contextual embeddings to
achieve improved performance compared to previous methods.

Khan et al. [15] enhanced health mention classification by employing an XLNet-
based language model, while Khan et al. [17] conducted a comparative analysis
of various transformer-based language models, highlighting the superior
performance of the RoBERTa [22] model on the health mention classification
task.

Luo et al. [23] introduced a COVID-19 tweets dataset and proposed a dual-CNN
classifier, comprising an auxiliary network (A-Net) to address class imbalance
issues in the dataset.

Naseem et al. [27] utilized domain-specific language models and linguistic
features to enhance text representation, employing a Bi-LSTM with an attention
module focused on disease or symptom terms. They also introduced PHS-BERT
[28], a domain-specific language model, and evaluated its performance on
various social media datasets.

Naseem et al. [26] released the Reddit Health Mention Dataset (RHMD),
consisting of Reddit posts related to fifteen diseases. They developed HMCNET,
combining emotional and domain-specific representations to classify given
posts.

Rahman et al. [30] introduced DepressionEmo, a multi-label classification
dataset derived from Reddit posts focusing on eight emotions related to
depression. They experimented with machine learning and deep learning models,
achieving the highest performance with a BART [21] classifier.

Haque et al. [8] released a dataset for distinguishing posts related to
depression versus suicidal ideation. They proposed a deep learning model
coupled with an unsupervised label correction method, demonstrating the
effectiveness of their approach.

Garg et al. [5] constructed an annotated dataset for identifying risk factors
of mental disturbance using Reddit posts. They established baseline models
based on transformer architecture as well as LSTM and GRU for future research.

Poswiata et al. [29] categorized social media posts into various depression
severity levels using BERT [4], RoBERTa [22], and XLNet [33] models.

Ghosal et al. [6] utilized fastText embeddings for contextual analysis,
followed by TF-IDF for term relevance, and applied an XGBoost classifier to
detect mental illness from social media posts.

###  2.2 Disease detection using Contrastive Adversarial Training

Khan et al. [18] introduced the Contrastive Adversarial Training (CAT) method,
utilizing Fast Gradient Sign Method (FGSM) [7] to generate adversarial
examples. Their approach involved two forward passes during LM training: one
with clean training examples and another with adversarial examples.
Additionally, they employed a contrastive loss that incorporated sentence-
level representations of both clean and adversarial examples to enhance word
representations. The overall training objective aimed to minimize a weighted
loss comprising two classification losses for clean and adversarial examples,
respectively, along with a contrastive loss.

In another work, Khan et al. [16] proposed a CAT-based method by augmenting
Gaussian noise with a mean of $0$ and a standard deviation of $1$ in the
hidden state representations of LMs. They found that adding noise to earlier
layers of models yielded better results than adding noise in the middle or
later layers. Additionally, Khan et al. [14] introduced the Random Weighted
Perturbation (RWP) method, which perturbed the parameters of LMs based on
their parameter distribution. They jointly trained models with clean and
perturbed parameters, incorporating a contrastive loss similar to that in Khan
et al. [18]. Furthermore, they proposed a meta-predictor leveraging the
predictive performance of five language models.

Naseem et al. [25] proposed a self-training method to generate new training
examples to address label scarcity. Additionally, they employed a supervised
contrastive loss to learn distinguishable representations between literal and
figurative usage of disease or symptom terms.

This paper introduces a novel method termed self-augmentation for generating
adversarial examples. Drawing inspiration from prior research [18, 16, 14],
our approach involves concurrently training LMs with both clean and
adversarial examples. Through extensive experimentation, we demonstrate that
this approach consistently outperforms various baseline and SotA methods on
benchmark datasets.

##  3 Method

Our proposed method is illustrated in Fig. 1 and comprises three main
components.

The first component utilizes a transformer-based model fine-tuned for text
classification. This model converts input text into feature embeddings and
categorizes them into predefined categories. The second component also employs
a transformer-based model, utilizing the same input text as the first
component. However, during its fine-tuning process, the intermediate layer
representations are explicitly augmented with representations from the first
component, resulting in the generation of an adversarial version. The third
and final component involves a shared projection network and contrastive loss.
This component aims to bring the input sample and its perturbed version closer
together while simultaneously pushing away other samples within the batch of
inputs.

Now, we provide a detailed description of each of these components:

Figure 1: The pipeline of our proposed method, consisting primarily of three
components: a transformer model with standard settings, a transformer model
with self-augmentation, and contrastive loss.

###  3.1 Transformer Model

Let $\digamma$ be a transformer model that serves as the backbone of our
approach, responsible for mapping input tokens to embedding representations.
We leverage transformer-based encoders such as BERT [4] and RoBERTa [22].
These models are fine-tuned on our datasets to learn representations for the
input and minimize classification loss. The classification loss quantifies the
disparity between the model’s predictions, and the actual label. Cross-entropy
is employed as the classification loss in our experiments.

###  3.2 Self Augmentation

The transformer model $\digamma$ comprises a series of L encoder layers, with
each layer denoted as $L_{i}$ and its corresponding hidden representation as
$H_{i}$ during the fine-tuning process. Additionally, let $C$ denote a copy of
the model $\digamma$, with its layers represented as $L_{j}$ and hidden
representations as $H_{j}$.

During the fine-tuning process of $\digamma$ and $C$, we implement self-
augmentation by incorporating $H_{i}$ into the hidden representation $H_{j}$
of $C$. Mathematically, this augmentation is expressed as:

$H_{j}=H_{j}\oplus H_{i}$ (1)

Here, the symbol $\oplus$ represents element-wise summation. The term “self-
augmentation” signifies that the augmentation originates from the model
itself. Both models $\digamma$ and $C$ are trained concurrently with identical
inputs, differing only in the augmentation of hidden representations in $C$.
Subsequently, these models complete their forward passes and return two
classification losses.

###  3.3 Training Objective

####  3.3.1 Classification Loss

We employ cross-entropy as the classification loss for both models, measuring
the disparity between predictions and the actual label.

####  3.3.2 Contrastive Loss

Inspired by [18, 14], our approach incorporates a contrastive learning method
[34] based on the redundancy injection principle. This method operates on two
inputs: the clean input and its perturbed version, aiming to learn similar
representations for corresponding pairs. Essentially, it learns a square
matrix $M$ to depict the correlation between clean and perturbed inputs, where
values range from -1 to +1, indicating anti-correlation and perfect
correlation, respectively.

In our experiments, the inputs to the contrastive loss are denoted as $H_{i}$
and $H_{j}$. Following the methodology of prior studies [18, 14], we pass
$H_{i}$ and $H_{j}$ through a shared multi-layer perceptron (MLP) based
projection network. This network projects them onto a similar representation
space, subsequently passing them to the contrastive loss function. We adopt
configurations for the projection network akin to those used in previous works
[18, 14].

The overall training loss minimized is a weighted sum of two classification
losses and the contrastive loss, formulated as:

$L=\frac{(1-\alpha)}{2}(L_{CE_{1}}+L_{CE_{2}})+\alpha L_{c}$ (2)

Here, $L_{CE_{1}}$ and $L_{CE_{2}}$ denote two classification losses, and
$L_{c}$ reprresents contrastive loss. The parameter $\alpha$ serves as a
trade-off parameter, governing the weights assigned to the three losses.

##  4 Experiments

In Table 1, we present the statistics of the datasets used for training and
evaluating our proposed method. Detail of each of the datasets is as follows:

###  4.1 Benchmark Datasets

####  4.1.1 Dreaddit

The dataset comprises posts sourced from the Reddit platform, covering a range
of categories including anxiety, domestic violence, PTSD, relationships,
stress, assistance for survivors of abuse, homelessness, food pantry, and
almost homeless. Each post is labeled as either depression or non-depression.
In total, the dataset contains 3,553 posts, with 2,838 posts allocated for
training and 715 for testing. We divided the training set into 20% validation
set, and used remaining 80% as training set.

####  4.1.2 Reddit Health Mention Dataset (RHMD):

The dataset comprises 10,015 Reddit posts covering 15 diseases, including
asthma, OCD, diabetes, depression, cough, heart attack, PTSD, headache,
cancer, allergy, Alzheimer’s, stroke, addiction, migraine, and fever. Each
post is labeled into one of four classes: personal health mention (PHM), non-
personal health mention (NPHM), figurative mention (FM), and hyperbolic
mention (HM). It’s worth noting that we utilized a publicly available version
of the dataset where HM and FM are combined. We divided dataset into 70%
training set and 15% each of the validation and test sets.

####  4.1.3 DepressionEmo:

The dataset is a multi-label dataset comprising 6,037 Reddit posts, each
associated with one or more of 8 depression-related emotions: anger, cognitive
dysfunction, emptiness, hopelessness, loneliness, sadness, suicide intent, and
worthlessness. The distribution of posts across these emotions is as follows:
21.2% for sadness, 19.1% for hopelessness, 13.6% for worthlessness, 12.7% for
loneliness, 11.1% for anger, 10.3% for emptiness, 6.8% for suicide intent, and
5.2% for cognitive dysfunction. For model training and evaluation, the dataset
is divided into 4,225 training examples, 906 validation examples, and 906 test
examples.

Table 1: Statistics of Benchmark Datasets. BC, MC, and ML denote binary classification, multi-class classification, and multi-label classification tasks, respectively. Dataset | No. of Samples | Type | No. of Classes/Labels
---|---|---|---
Dreaddit [31] | 3,553 | BC | 2
RHMD [26] | 10,015 | MC | 3
DepressionEmo [30] | 6,037 | ML | 8

###  4.2 Training details

To determine the optimal hyperparameters for our proposed method, we conducted
a grid search. Specifically, we experimented with varying batch size
$b\in\\{16,32\\}$ and $\alpha\in\\{0.1,0.2,0.3,0.4,0.5\\}$. Additionally, we
explored the selection of $L_{i}$ and $L_{j}$ from the range of
$\\{0,3,6,9,12,15,18,21\\}$. For optimization, we employed the Adam optimizer
[19]. The model was trained for 20 epochs with a fixed learning rate of
$1e^{-5}$, and training was halted if no improvement in F1-score on the
validation set was observed for 5 consecutive epochs. We also extracted the
$H_{i}$ and $H_{j}$ hidden states and projected them to lower dimensions,
using a 3-layered shared projection network. This network consisted of the
first two hidden layers with a size of 1024 each, followed by a final layer
with a size of 300. ReLU was employed as the non-linear activation function,
and 1-D batch normalization was applied after each layer. Both projected
representations were passed to the contrastive loss function. Following prior
works [18, 16, 14], default parameters were utilized for the contrastive loss
function.

###  4.3 Results

(a) Baseline vs. Proposed (RHMD).

(b) Baseline vs. Proposed (Dreaddit).

(c) Baseline vs. Proposed (DepressionEmo).

Figure 2: Performance Comparison of the Proposed Approach with Baseline Methods across 3 Datasets. Table 2: Comparison of Proposed Method with SotA Results in terms of Precision, Recall, and F1-Scores, Reported on Test Sets. Dataset | Method | Precision | Recall | F1-score |
---|---|---|---|---|---
Dreaddit [31] | MentalBERT [11] | - | 81.82 | 81.82 |
Alghamdi et al. [2] | - | - | 82.88 |
PHS-BERT [28] | - | - | 82.89 |
KC-Net [32] | 84.1 | 83.3 | 83.5 |
Proposed (+RoBERTa) | 84.44 | 84.46 | 84.45 |
DepressionEmo [30] | SVM [30] | 72.0 | 41.0 | 47.0 |
Light GBM [30] | 48.0 | 80.0 | 58.0 |
XGBoost [30] | 63.0 | 56.0 | 59.0 |
GAN-BERT [30] | 69.0 | 72.0 | 70.0 |
BERT [30] | 72.0 | 77.0 | 74.0 |
BART [30] | 70.0 | 81.0 | 76.0 |
Proposed (+RoBERTa) | 76.4 | 80.31 | 78.16 |

In Fig.2, we present a comparative analysis of the performance of our proposed
method against baseline models across three datasets. As illustrated in
Fig.2(a), our proposed method demonstrates a noteworthy improvement over the
BERT baseline, achieving a 0.95% increase in F1-score on the RHMD dataset.
Additionally, when coupled with RoBERTa, our proposed method achieves an
F1-score of 81.49%, surpassing its baseline by 0.57%. Fig.2(b) depicts the
performance on the Dreaddit dataset, where the BERT baseline model achieves an
F1-score of 80.26%. In contrast, our proposed method outperforms the baseline
by 0.69%, while the performance is further elevated to 3.48% with RoBERTa. In
Fig.2(c), the visualization showcases the performance of our proposed method
on the DepressionEmo dataset, exhibiting improvements of 2.15% and 0.83% in
F1-score over the respective baselines. These results strongly validate the
superiority of our proposed method.

Table 2 provides a detailed performance comparison of our method with the SotA
on the Dreaddit and DepressionEmo datasets. On the Dreaddit dataset, our
proposed method with RoBERTa achieves a notable F1-score improvement of 0.95%
over the SotA, primarily enhancing recall by 1.16%. Similarly, on the
DepressionEmo dataset, our method achieves an F1-score of 78.16%, surpassing
the existing SotA by 2.16%. While our method exhibits higher recall than
precision, akin to existing deep learning based methods, it significantly
enhances precision by 6.4%.

Furthermore, to provide a comprehensive comparison with existing literature,
we report average results over 10-fold cross-validations on the RHMD dataset
in Table 3. Our proposed method with RoBERTa achieves the second-best results
compared to the SotA methods. Although the highest-performing method [14]
involves an ensemble of 5 trained models, resulting in increased inference
time, our method slightly outperforms the model by Khan et al. [18]. However,
its margin of improvement is significantly higher than other methods in the
literature.

Table 3: Proposed vs. SotA results in terms of Precision, Recall, and F1-scores for RHMD dataset. Results are averaged across 10-folds. Model | Precision | Recall | F1-score
---|---|---|---
BERT [4] | 63.0 | 68.0 | 65.0
BioBERT [20] | 65.0 | 62.0 | 63.0
CT-BERT [24] | 65.0 | 68.0 | 67.0
PHS-BERT [28] | 67.0 | 69.0 | 68.0
WESPAD [13] | 60.0 | 60.0 | 59.0
FeatAug+ [10] | 51.0 | 51.0 | 51.0
JiangLSTM [12] | 63.0 | 63.0 | 63.0
BERT-MTL [1] | 69.0 | 65.0 | 67.0
BiLSTM-Senti [3] | 67.0 | 68.0 | 68.0
BiLSTM-Attn+Senti [27] | 70.0 | 71.0 | 71.0
HMCNET [26] | 75.0 | 75.0 | 75.0
Naseem et al. (+BERT) [25] | 78.0 | 79.0 | 79.0
Naseem et al. (+PHS-BERT) [25] | 81.0 | 81.0 | 81.0
Khan et al. [18] | 83.43 | 83.27 | 83.23
Khan et al. [14] | 83.82 | 83.91 | 83.8
Proposed (+RoBERTa) | 83.40 | 83.29 | 83.3

###  4.4 Analysis of the Proposed Method

Table 4: Optimal Layer Combination and Weighting Parameter $\alpha$ on Validation Sets for 3 Datasets. Dataset | Model | $\alpha$ | $L_{i}$ | $L_{j}$ | F1-score |
---|---|---|---|---|---|---
Dreaddit [31] | Proposed(+BERT) | 0.1 | 18 | 21 | 84.76 |
Proposed(+RoBERTa) | 0.1 | 18 | 3 | 86.56 |
DepressionEmo [30] | Proposed(+BERT) | 0.2 | 18 | 0 | 78.90 |
Proposed(+RoBERTa) | 0.5 | 21 | 9 | 80.08 |
RHMD [26] | Proposed(+BERT) | 0.4 | 9 | 21 | 82.52 |
Proposed(+RoBERTa) | 0.4 | 0 | 15 | 84.44 |

####  4.4.1 Effect of Layer Combination

In Table 4, we present the layer combinations and the effect of $\alpha$ for
our proposed method across three datasets. Notably, on the Dreaddit dataset,
combining later layers from both models yields favorable results for the
proposed method leveraging BERT. Conversely, the proposed method employing
RoBERTa performs well when the later layer is combined with an earlier layer.
The optimal value of $\alpha$ for both models is found to be 0.1. For the
DepressionEmo dataset, combining later layers from the first model with later
layers from the second model proves effective, although the optimal value of
$\alpha$ varies for each model. On the other hand, for the RHMD dataset,
combining earlier layers from the first model with later layers from the
second model demonstrates superior performance, with the optimal value of
$\alpha$ identified as 0.4 for both models. A notable trend observed in the
Dreaddit and DepressionEmo datasets is that later layers from the first model
enhance performance when combined with the second model. One possible
explanation for this phenomenon is the limited number of training samples in
these datasets, posing challenges to the learning capability of the models.
Leveraging already well-learned semantic representations from later layers
improves syntactic features for earlier layers.

Conversely, the RHMD dataset contains relatively larger training examples.
Here, the combination of syntactic features learned in earlier layers with
semantic features from later layers enhances overall performance.

####  4.4.2 Ablation Study

To analyze the effect of various components of the proposed method, we conduct
ablation studies. It is clear from Table 5 that the F1-score for Dreaddit
dataset is decreased by 1.9% if CL component is dropped. The F1-score is
further decreased by 0.58% if self-augmentation component is dropped. Overall
F1-score is reduced by 2.48% if the proposed components are dropped. We
observe different pattern for DepressionEmo dataset. The addition of self-
augmentation actually decreases the F1-score over RoBERTa baseline by 0.38%.
One possible explanation for this behaviour is the complexity of dataset
itself. This complexity comes from the smaller sample size in the dataset.
Multi-labeling makes it further complex as unique combination of labels is
small. However, adding CL component improves the text representation, and
F1-score is increased by 0.83% compared to RoBERTa baseline. For RHMD dataset,
incorporating self-augmentation causes a minor improvement of F1-score by
0.12%. Possible justification of this minute difference is the complexity of
the dataset due to various number of diseases, i.e., 15 in the dataset. Hence,
per-disease samples are small as compared to overall dataset. However similar
to DepressionEmo dataset, adding CL causes increase in F1-score over the BERT
baseline.

####  4.4.3 Visualization of Text Representations

In Fig. 3, embedding are visualized for Dreaddit and DepressionEmo datasets.
For dreaddit dataset, our proposed method with BERT learns different
representations from its baseline by effectively separating depression and
non-depression classes. On the other hand, proposed model with RoBERTa spreads
the representations while maintaining the decision boundary between two
classes. For the DepressionEmo dataset, our proposed method demonstrates
embeddings with a smaller spread compared to the baseline methods.
Furthermore, the representations generated by our proposed methods exhibit
numerous vacant spaces in the representation space, indicating a stronger
ability to learn decisive boundaries between classes. Similarly, in Fig. 4,
the representations learned by our proposed methods are notably more compact,
with embeddings of various classes distinctly spaced apart from one another.

Table 5: Ablation Study: Analyzing the Impact of Adding Self-augmentation (SA) and Contrastive Learning (CL) on Test Sets. Dataset | Method | Precision | Recall | F1-score |
---|---|---|---|---|---
Dreaddit [31] | RoBERTa (Baseline) | 81.91 | 82.45 | 81.97 |
RoBERTa (+SA) | 82.49 | 82.94 | 82.55 |
RoBERTa (+Proposed) | 84.44 | 84.46 | 84.45 |
DepressionEmo [30] | RoBERTa (Baseline) | 76.38 | 78.76 | 77.33 |
RoBERTa(+SA) | 73.99 | 80.50 | 76.95 |
RoBERTa (+Proposed) | 76.40 | 80.31 | 78.16 |
RHMD [26] | BERT (Baseline) | 80.13 | 80.12 | 80.12 |
BERT (+SA) | 80.16 | 80.40 | 80.24 |
BERT (+Proposed) | 81.02 | 81.33 | 81.07 |

(a) BERT baseline (Dreaddit).

(b) BERT Proposed (Dreaddit).

(c) RoBERTa baseline (Dreaddit).

(d) RoBERTa Proposed (Dreaddit).

(e) BERT baseline (DepressionEmo).

(f) BERT Proposed (DepressionEmo).

(g) RoBERTa baseline (DepressionEmo).

(h) RoBERTa Proposed (DepressionEmo).

Figure 3: Embedding Visualizations for Dreaddit and DepressionEmo Datasets.
The embeddings are visualized for the entire validation set of the Dreaddit
dataset, and a subset (for clearer visualization) of the DepressionEmo
dataset.

(a) BERT baseline.

(b) BERT Proposed.

(c) RoBERTa baseline.

(d) RoBERTa Proposed.

Figure 4: Embedding Visualizations for RHMD Dataset.

##  5 Conclusion

In this paper, we have advanced the field of disease detection from social
media text by improving the text representations capabilities of LMs by
proposing a new method. We proposed self-augmentation technique for generating
adversarial examples and then unified adversarial training with contrastive
learning methods, where it demonstrated notable improvements in models
performance. Through extensive experiments across 3 benchmark datasets
comprising diverse types of classification tasks on social media text, we have
demonstrated the efficacy of our proposed approach. Our method has achieved
SotA performance on these datasets by enhancing discriminative features of
LMs. Our future investigations will focus on evaluating the scalability and
performance of this method on LMs with billions of parameters.

## References

  * [1] Aduragba, O.T., Yu, J., Cristea, A., Long, Y.: Improving health mention classification through emphasising literal meanings: A study towards diversity and generalisation for public health surveillance. In: Proceedings of the ACM Web Conference 2023. pp. 3928–3936 (2023)
  * [2] Alghamdi, Z., Kumarage, T., Karami, M., Alatawi, F., Mosallanezhad, A., Liu, H.: Studying the influence of toxicity and emotion features for stress detection on social media. In: ECSM 2023 10th European Conference on Social Media. Academic Conferences and publishing limited (2023)
  * [3] Biddle, R., Joshi, A., Liu, S., Paris, C., Xu, G.: Leveraging sentiment distributions to distinguish figurative from literal health reports on twitter. In: Proceedings of The Web Conference 2020. pp. 1217–1227 (2020)
  * [4] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [5] Garg, M., Shahbandegan, A., Chadha, A., Mago, V.: An annotated dataset for explainable interpersonal risk factors of mental disturbance in social media posts. arXiv preprint arXiv:2305.18727 (2023)
  * [6] Ghosal, S., Jain, A.: Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier. Procedia Computer Science 218, 1631–1639 (2023)
  * [7] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
  * [8] Haque, A., Reddi, V., Giallanza, T.: Deep learning for suicide and depression identification with unsupervised label correction. In: Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part V 30. pp. 436–447. Springer (2021)
  * [9] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [10] Iyer, A., Joshi, A., Karimi, S., Sparks, R., Paris, C.: Figurative usage detection of symptom words to improve personal health mention detection. arXiv preprint arXiv:1906.05466 (2019)
  * [11] Ji, S., Zhang, T., Ansari, L., Fu, J., Tiwari, P., Cambria, E.: Mentalbert: Publicly available pretrained language models for mental healthcare. arXiv preprint arXiv:2110.15621 (2021)
  * [12] Jiang, K., Feng, S., Song, Q., Calix, R.A., Gupta, M., Bernard, G.R.: Identifying tweets of personal health experience through word embedding and lstm neural network. BMC bioinformatics 19(8), 67–74 (2018)
  * [13] Karisani, P., Agichtein, E.: Did you really just have a heart attack? towards robust detection of personal health mentions in social media. In: Proceedings of the 2018 World Wide Web Conference. pp. 137–146 (2018)
  * [14] Khan, P.I., Asim, M.N., Dengel, A., Ahmed, S.: A unique training strategy to enhance language models capabilities for health mention detection from social media content. arXiv preprint arXiv:2310.19057 (2023)
  * [15] Khan, P.I., Razzak, I., Dengel, A., Ahmed, S.: Improving personal health mention detection on twitter using permutation based word representation learning. In: International Conference on Neural Information Processing. pp. 776–785. Springer (2020)
  * [16] Khan, P.I., Razzak, I., Dengel, A., Ahmed, S.: A novel approach to train diverse types of language models for health mention classification of tweets. In: Artificial Neural Networks and Machine Learning–ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6–9, 2022, Proceedings, Part II. pp. 136–147. Springer (2022)
  * [17] Khan, P.I., Razzak, I., Dengel, A., Ahmed, S.: Performance comparison of transformer-based models on twitter health mention classification. IEEE Transactions on Computational Social Systems (2022)
  * [18] Khan, P.I., Siddiqui, S.A., Razzak, I., Dengel, A., Ahmed, S.: Improving health mention classification of social media content using contrastive adversarial training. IEEE Access 10, 87900–87910 (2022)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [20] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
  * [21] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
  * [22] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
  * [23] Luo, L., Wang, Y., Liu, H.: Covid-19 personal health mention detection from tweets using dual convolutional neural network. Expert Systems with Applications 200, 117139 (2022)
  * [24] Müller, M., Salathé, M., Kummervold, P.E.: Covid-twitter-bert: A natural language processing model to analyse covid-19 content on twitter. Frontiers in artificial intelligence 6, 1023281 (2023)
  * [25] Naseem, U., Kim, J., Khush, M., Dunn, A.G.: A linguistic grounding-infused contrastive learning approach for health mention classification on social media. In: Proceedings of the 17th ACM International Conference on Web Search and Data Mining. pp. 529–537 (2024)
  * [26] Naseem, U., Kim, J., Khushi, M., Dunn, A.G.: Identification of disease or symptom terms in reddit to improve health mention classification. In: Proceedings of the ACM Web Conference 2022. pp. 2573–2581 (2022)
  * [27] Naseem, U., Kim, J., Khushi, M., Dunn, A.G.: Robust identification of figurative language in personal health mentions on twitter. IEEE Transactions on Artificial Intelligence (2022)
  * [28] Naseem, U., Lee, B.C., Khushi, M., Kim, J., Dunn, A.G.: Benchmarking for public health surveillance tasks on social media with a domain-specific pretrained language model. arXiv preprint arXiv:2204.04521 (2022)
  * [29] Poświata, R., Perełkiewicz, M.: Opi@ lt-edi-acl2022: Detecting signs of depression from social media text using roberta pre-trained language models. In: Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. pp. 276–282 (2022)
  * [30] Rahman, A.B.S., Ta, H.T., Najjar, L., Azadmanesh, A., Gönül, A.S.: Depressionemo: A novel dataset for multilabel classification of depression emotions. arXiv preprint arXiv:2401.04655 (2024)
  * [31] Turcan, E., McKeown, K.: Dreaddit: A reddit dataset for stress analysis in social media. arXiv preprint arXiv:1911.00133 (2019)
  * [32] Yang, K., Zhang, T., Ananiadou, S.: A mental state knowledge–aware and contrastive network for early stress and depression detection on social media. Information Processing & Management 59(4), 102961 (2022)
  * [33] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. In: Advances in neural information processing systems. pp. 5754–5764 (2019)
  * [34] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International Conference on Machine Learning. pp. 12310–12320. PMLR (2021)