# SIDU-TXT: An XAI Algorithm for NLP with a Holistic Assessment Approach

Mohammad N.S. Jahromi111These authors contributed equally to this work.
<EMAIL_ADDRESS>Satya. M. Muddamsetty222These authors contributed equally
to this work<EMAIL_ADDRESS>Asta Sofie Stage Jarlner<EMAIL_ADDRESS>Anna
Murphy Høgenhaug<EMAIL_ADDRESS>Thomas Gammeltoft-Hansen<EMAIL_ADDRESS>Thomas
B. Moeslund<EMAIL_ADDRESS>
###### Abstract

Explainable AI (XAI) aids in deciphering ’black-box’ models. While several
methods have been proposed and evaluated primarily in the image domain, the
exploration of explainability in the text domain remains a growing research
area. In this paper, we delve into the applicability of XAI methods for the
text domain. In this context, the ’Similarity Difference and Uniqueness’
(SIDU) XAI method, recognized for its superior capability in localizing entire
salient regions in image-based classification is extended to textual data. The
extended method, SIDU-TXT, utilizes feature activation maps from ’black-box’
models to generate heatmaps at a granular, word-based level, thereby providing
explanations that highlight contextually significant textual elements crucial
for model predictions. Given the absence of a unified standard for assessing
XAI methods, this study applies a holistic three-tiered comprehensive
evaluation framework—Functionally-Grounded, Human-Grounded and Application-
Grounded—to assess the effectiveness of the proposed SIDU-TXT across various
experiments. We find that, in sentiment analysis task of a movie review
dataset, SIDU-TXT excels in both functionally and human-grounded evaluations,
demonstrating superior performance through quantitative and qualitative
analyses compared to benchmarks like Grad-CAM and LIME. In the application-
grounded evaluation within the sensitive and complex legal domain of asylum
decision-making, SIDU-TXT and Grad-CAM demonstrate comparable performances,
each with its own set of strengths and weaknesses. However, both methods fall
short of entirely fulfilling the sophisticated criteria of expert
expectations, highlighting the imperative need for additional research in XAI
methods suitable for such domains.

###### keywords:

NLP Explainability, Holistic XAI Evaluation , Sentiment Analysis , Legal Text
Analysis

[label1]organization=Visual Analysis and Perception Laboratory (VAP), Aalborg
University, postcode=9000 Aalborg, country=Denmark

[label2]organization=Faculty of Law, Center of Excellence for Global Mobility
Law. University of Copenhagen,postcode=2300 København S, country=Denmark

##  1 Introduction

Deep learning has substantially advanced Natural Language Processing (NLP),
enabling significant progress in tasks ranging from sentiment analysis [1] to
machine translation and and chatbot development [2]. Despite these advances, a
critical challenge persists—the ’black-box’ nature of such models. While the
field of NLP continues to evolve, the deployment of these models in products
and services of critical importance necessitates adherence to regulatory
frameworks such as the GDPR and the forthcoming AI Act. This underscores the
need for Explainable AI (XAI), ensuring that these advanced systems are
transparent, accountable, and in line with public expectations [3, 4]. The
subjective nature of explanation quality, coupled with the intricacies of
evaluations, further complicates the pursuit of transparency. XAI, especially
as applied to text, lacks a standardized framework, leading to diverse
interpretations of what constitutes understandability, comprehensibility, and
interpretability [5]. These terms, often used interchangeably, reflect the
need for clarity and uniformity in the evaluation of XAI methodologies. The
current landscape of XAI evaluation is marked by a patchwork of approaches
[6], each with its own merits and limitations, underscoring the necessity of
developing more robust and tailored approaches that resonate with human
intuition and expert judgment. This is particularly important in high-risk
applications, e.g., those in the legal and medical domains, where the
consequences of model outcomes can be significant. In parallel, for such
applications, gaining trust from domain experts through assessing and
verifying the decision quality of ’black-box’ models is inevitable [8].

In the realm of XAI research, one popular approach to make these models
understandable to human end-users is to create a separate model that
identifies the importance of given input features to the model being
explained. This category of methods is referred to as ’post-hoc’
explainability [11]. The model agnostic LIME [12] and class activation based
GRAD-CAM [13] are well-known examples of this explanation class that are
widely used for image input. In spirit of these methods, the ’Similarity
Difference and Uniqueness’ (SIDU) method [14], a class activation-based
explanation approach, was developed to effectively localize all the important
features in an image responsible for the prediction. Comprehensive
quantitative and expert-driven qualitative analyses have demonstrated that
this method not only outperforms both LIME and GRAD-CAM but also invokes
greater trust from human users. [14].

While these methods have set a standard in the visual domain, the text domain
presents its own unique set of challenges and opportunities for
explainability. Text data, with its complex syntactic and added semantic
layers, requires an approach that can unravel the contributions of individual
words and contextual relationships to a model’s output. Inspired by the
success of XAI methods in the visual domain, this work extends the class
activation-based explanation algorithm ’SIDU’ to the text domain, which we
refer to as ’SIDU-TXT’. This adaptation aims to provide interpretability
comparable to image-based tasks by generating ’heatmap’ representations at a
granular level for a given input text sequence. These ’heatmaps’ emphasize the
significance of individual words or contexts, effectively highlighting the
textual elements that are crucial for the model’s predictions. In
transitioning from the visual to the textual realms, our approach aligns with
existing scholarly observations that the main challenge in XAI for NLP lies in
crafting explanations that are intuitively comprehensible to humans in the
context of text analysis [15, 16]. Thus, we have modified the SIDU algorithm
to fine-tune the interpretability for NLP tasks. The key refinement in SIDU-
TXT is the introduction of feature importance weights that represent
similarity difference, and uniqueness scores, which are used to identify and
select a subset of feature masks that carry the most semantic information for
the given text features.

Moving beyond specific XAI tools, we confront the broader question of
evaluating such methods in the NLP landscape. An inaccurate or low-fidelity
explanation model can significantly limit trust in the explanation and, by
extension, in the ’black-box’ model it aims to explain. Currently, there is no
consensus on how best to assess XAI techniques. However, an effective
explanation should be faithful, justifiable, and comprehensible to humans, as
well as invoke further trust in expert users. Recognizing the need for a
comprehensive evaluation of these aspects in NLP, we advocate for a holistic
approach to the XAI assessment, encompassing Functionally-Grounded, Human-
Grounded, and Application-Grounded framework [17], as illustrated in Fig 3.
Concretely, we use sentiment analysis as a typical NLP task to evaluate SIDU-
TXT through Functionally-Grounded and Human-Grounded approaches. For
Application-Grounded assessment, we adopt the more complex task of asylum case
classification in the sensitive legal domain. This comprehensive strategy
ensures a thorough examination of the model’s performance across various
dimensions, from quantitative faithfulness to qualitative human expert
involvement, ultimately fostering greater alignment with human intuition and
enhancing the trustworthiness of explanations provided by SIDU-TXT. Our main
contributions can be summarized as follows:

  1. 1.

We present SIDU-TXT, a powerful class activation-based XAI method optimized
for NLP, which elucidates the decision-making process of ’black-box’ models.
By leveraging the top feature activation masks extracted from the models,
along with their corresponding importance weights, SIDU-TXT discloses the most
influential features driving the model’s predictions. This method provides
granular and insightful explanations at the word and context levels,
significantly enhancing transparency and interpretability.

  2. 2.

A comprehensive quantitative assessment on the IMDB movie review dataset [18]
for sentiment analysis is conducted to assess the faithfulness/fidelity. Our
approach shows a significant improvement in both insertion and deletion
metrics compared to established XAI methods such as LIME and GRAD-CAM,
indicating greater precision in capturing and retaining critical textual
features.

  3. 3.

We employ a Human-Grounded evaluation to analyze the justifiability and
comprehensibility of XAI methods This process involved using human-provided
explanations to assess the models at both token and sentence levels. The
findings demonstrate that SIDU-TXT excels in these criteria, achieving a
higher Jaccard similarity score in justifiability and superior mean precision,
recall, and F1-score in comprehensibility.

  4. 4.

For the first time, we apply an XAI model to complex and lengthy texts from
the legal domain, with a particular focus on asylum decision-making where
trust in algorithmic judgment is paramount. This challenging application
emphasizes the adaptability of SIDU-TXT to high-stakes domains where
interpretability is crucial. Our study demonstrates that involving domain
experts in the evaluation process significantly bolsters the trust and
reliability of an AI system’s decision-making capabilities.

##  2 Related Work

This section explores two intertwined dimensions in the realm of explainable
AI (XAI) for the text domain. The first dimension, as covered in Subsection
2.1, presents an overview of various XAI methods that have been developed to
provide clarity on how models process and analyze text data. The second
dimension, outlined in Subsection 2.2, delves into the diverse strategies and
challenges associated with evaluating XAI methods.

###  2.1 XAI Methods for Text

In the domain of XAI, various methods have been developed to demystify the
decision-making processes of deep learning models in text classification, as
highlighted in several surveys and studies [11, 25, 26]. Among these, post-hoc
or feature attribution methods stand out as particularly popular. They seek to
pinpoint the most influential components of a given input text using
techniques such as input perturbation [12], gradient analysis, and relevance
propagation [27]. A recent survey specifically targeting NLP-focused XAI
categorizes contemporary literature, mainly encompassing local post-hoc, local
self-explaining, global post-hoc, and global self-explaining methods [11].
These ’black-box’ explanation methods comprise a variety of techniques,
including saliency mask generation methods such as LIME or Grad-CAM, as well
as architecture-specific methods like attention mechanisms [28]. While these
approaches are often employed in the context of image data, their adaptability
for text models has also been demonstrated, aiming to enhance transparency in
model predictions. Well-established methods such as LIME and Grad-CAM,
traditionally associated with Convolution Neural Networks (CNNs) for text
classification and examined in earlier survey papers [29], are pertinent to
our discussion. Our SIDU-TXT method, which generates saliency-based heatmaps,
will employ these established methods as a benchmark for comparison.

###  2.2 Evaluation of XAI Methods

Evaluating the effectiveness of XAI methods involves a spectrum of techniques,
each with unique benefits and limitations. Initial approaches often focus on
faithfulness metrics (Functionally-Grounded), which may include strategies
like word deletion to observe the impact of token (word) removal on prediction
confidence [8, 30, 31]. These methods provide a quantitative means to assess
explanation quality but may not fully capture the root cause of a model’s
decision, leaving the critical ‘why’ question only partially answered [32].
Other quantitative methods involve using relevance scores to create document
vectors, subsequently evaluated with traditional machine learning techniques
[34]. However, while feature importance metrics are widely utilized, their
limitations are especially pronounced in the context of NLP in demonstrating
the underlying rationale behind a linguistic phenomenon [33]. This gap
highlights that the evaluation of XAI is still evolving and that there is a
need for more nuanced approaches that can more clearly explain the model’s
reasoning in a way that aligns with human understanding. Further methods, such
as hybrid documents and morphosyntactic agreements, attempt to verify the
accuracy of explanations without direct human involvement [35]. Yet, the
ultimate goal of ’explanation evaluation’ is not simply to select the most
likely or accurate explanation from the model’s perspective, but rather to
identify explanations that possess pragmatic value—those that are the most
useful and relevant to end-users, especially in sensitive real-world
applications like the legal domain, where the implications of AI’s decisions
are critical. In contrast, Human-Grounded evaluations bring in expert-derived
relevance benchmarks and involve tasks where humans infer classifier outputs
based on highlighted words, adding a layer of a qualitative dimension to the
assessment process [8, 30, 36]. We will embrace a dual approach, integrating
both quantitative Functional-Grounded evaluation and qualitative human-expert
judgment to validate the effectiveness of the SIDU-TXT method compared to
algorithms like LIME and GRAD-CAM. This holistic evaluation strategy not only
adheres to conventional metrics but also seeks to capture the pragmatic
aspects of explanations in real-world contexts.

##  3 SIDU-TXT

The advent of deep neural networks in NLP has brought about transformative
changes, yet the interpretability of these models remains a significant
challenge. Here, we introduce SIDU-TXT, a novel approach to XAI that builds
upon our previous success with SIDU in the visual domain, tailored to meet the
intricate demands of text interpretation. SIDU-TXT is designed to illuminate
the decision-making process of NLP models, providing granular insights into
the textual elements that drive predictions. When adapting SIDU to text input,
our focus shifts to the nuanced world of text classification. We aim at a
method that identifies and emphasizes the subtle yet critical language
elements—ranging from individual words to phrases, or N-grams—that are pivotal
in text-based decision-making of a text classification model. This capability
sets SIDU-TXT from current state-of-the-art methods, which often struggle to
highlight the specific sequence of words or phrases that provide essential
contextual information when applied to the domain of textual data.

The core concept of SIDU-TXT centers on deriving feature text masks from the
last convolution layers of a CNN model. It then utilizes the most significant
$K$ masks to compute two key metrics: similarity differences (SID) and
uniqueness (U). SID assesses the importance of text features by measuring the
change in the model’s output when a feature is altered, indicating its
influence on the prediction. Uniqueness evaluates the distinctiveness of a
feature’s contribution compared to all other features, with more unique
features receiving higher importance. The final explanation, a heatmap,
emerges from a weighted combination of these feature masks, offering a visual
representation of the text’s influential elements as discerned by the model.

SIDU-TXT begins by extracting feature text masks from the final convolutional
layer of a deep CNN model, denoted as ’F’. Unlike in image processing, where
convolution layers capture spatial visual features, in NLP, these layers are
interpreted as capturing syntactic and semantic patterns in textual data. This
distinction is important, as the nature of text data demands a different
approach to feature extraction and interpretation. The layers, denoted by
dimensions ${}^{\prime}n\times N^{\prime}$, play a central role in pinpointing
textual features that are relevant for a specific class ’c’, formalized as
$f^{c}_{i}=\left[f^{c}_{1},f^{c}_{2},\ldots,f^{c}_{N}\right]$. For each
textual feature activation $f^{c}_{i}$, we derive a corresponding binary mask
$B^{c}_{i}$ through a thresholding operation:

$B^{c}_{i=1,...,N}=H(f^{c}_{i=1,...,N}>\tau)$ (1)

Where, $B^{c}_{i}$ denotes the binary mask for the ith feature, and $\tau$
represents the threshold value, and the expression $f^{c}_{i}>\tau$ is a
condition that a function $H$ uses to set each element of $B^{c}_{i}$ to ’1’
if $f^{c}_{i}$ exceeds $\tau$, and ’0’ otherwise. This process effectively
filters the activation maps, retaining only those features that surpass the
threshold, thereby highlighting the most influential text components for class
’c’. Note that, $\tau$ is a hyperparameter set experimentally to balance
between interpretability and noise reduction in heatmap visualization.

The binary text mask $B^{c}_{i}$ is then upscaled to match the input text’s
sequence length $T$ using bilinear interpolation, ensuring that the feature
activations correspond to the actual text sequence. The resulting up-sampled
mask, denoted as $L^{c}_{i}$, contains values within the range $[0,1]$. This
up-sampled mask is further converted back into a binary mask, represented as
$M^{c}_{i}$. To effectively distill the most influential text features as
learned by the model via its convolution layer for a given text input, the
binary mask $M^{c}_{i}$ is combined with the input text token vector $T$
through point-wise multiplication. This generates the feature activation text
mask $A^{c}_{i}$, defined as:

${A^{c}_{i}=T\odot M^{c}_{i},}$ (2)

where, $T$ is an input text, and $M^{c}_{i}$ is an up-sampled binary mask. The
procedure for feature activation’s text mask generation is illustrated in
Figure 1.

Figure 1: The block diagram depicts the process of generating a feature
activation text mask, serving as input for the SIDU-TXT XAI method. Within the
diagram, $\langle$unk$\rangle$ symbolizes an unknown word, typically employed
in NLP to indicate a missing word in a sentence. This token is mathematically
assigned a value of 0.

After generating the upscaled feature text activation masks, the prediction
score vectors $P^{c}_{i}$ and $P^{c}_{org}$ are computed for all feature
activation text masks $M^{c}_{i}$, and the original input text $T$ using the
’black-box’ model. We then proceed to determine the similarity difference
between each prediction score $P^{c}_{i}$ associated with an input feature
activation text mask and the prediction score $P^{c}_{org}$ of the original
input text $T$. The similarity difference $SID^{c}_{i}$ between these two
vectors signifies the relevance of the feature activation text mask concerning
the original input text $T$ and it is defined by utilizing the Gaussian kernel
as:

$SID^{c}_{i}=\exp\left(\frac{-\|P^{c}_{org}-P^{c}_{i}\|}{2\sigma^{2}}\right)$
(3)

where $\sigma$ represents the spread of the kernel, effectively controlling
the sensitivity of the similarity measure to differences between the
prediction scores. This step aims to minimize the influence of less
significant textual elements with low weights while highlighting those textual
elements genuinely responsible for the model’s predictions. Following the
calculation of the similarity difference measure, we also compute a uniqueness
measure $U^{c}_{i}$ to identify distinct textual feature masks among all the
prediction score vectors of the feature activation text masks. The uniqueness
measure is defined as:

$U^{c}_{i}=\sum_{j=1}^{N}\|P^{c}_{i}-P^{c}_{j}\|,{~{}~{}~{}~{}~{}~{}~{}i=1,2,...,N}$
(4)

The SIDU weights are computed as the product of the similarity differences and
uniqueness measures and is given by

$W^{c}_{i}=SID^{c}_{i}\cdot U^{c}_{i},$ (5)

where $SID^{c}_{i}$ is the similarity difference and $U^{c}_{i}$ is the
uniqueness measure given in Eq. (3) and Eq. (4), respectively. Upon computing
the weights, we then introduce a step tailored to the textual domain:
selecting the most informative masks based on their weights $W^{c}_{i}$. This
process entails arranging the weights in descending order and focusing on the
top-ranking masks. For instance, if we opt for the top $K$ weights, only the
top $K$ masks are considered, where each mask can represent different aspects
of language use, such as semantic nuances or syntactic structures. The
motivation for this selective approach stems from our careful examination of
all generated feature activation masks. Masks with higher weights are found to
signify features rich in semantic information, essential for the model’s
decision-making process. On the other hand, masks with lower weights tend to
contain less important information. This step contrasts with image processing
methods, where the significance of spatial and visual features follows a
different pattern of relevance.

Ultimately, the final explanatory output of SIDU-TXT, $S^{c}_{i}$ ,is derived
by calculating a weighted sum of the feature activation text masks with the
highest weights:

$S^{c}_{i}=\frac{1}{K}\sum_{i=1}^{K}W^{c}_{i}\cdot M^{c}_{i},$ (6)

where $K$ represents the number of top masks used for text explanation, a
hyperparameter that primarily influences the the granularity of the
explanation generated by SIDU-TXT. An overview of SIDU-TXT is summarized in
Figure 2.

Figure 2: Block diagram illustrates the procedure of generating the
explanations for the text data using SIDU-TXT.

##  4  Assessing XAI Methods

This section lays the groundwork for a comprehensive evaluation framework,
drawing from interdisciplinary insights into what constitutes an effective
explanation. Building upon the challenges and opportunities discussed in our
related work, we articulate a set of criteria any XAI method should strive to
meet: fidelity/faithful to the model’s decision process, justifiability,
comprehensibility in the eyes of users, and trustworthy by the expert
audience. These criteria, informed by the selected taxonomy [8], will guide
our empirical assessment in the upcoming experiments section. The approach, as
illustrated in Fig 3, provides a versatile framework for XAI NLP evaluation
and allows for the domain-specific customization of experiments, which makes
it suitable for our objective. We will explore these criteria in the
subsequent subsections: 4.1 Functionally-Grounded Evaluations, which focus on
proxy tasks without human involvement; 4.2 Human-Grounded Evaluation,
involving real humans in simplified tasks; and 4.3 Application-Grounded
Evaluation, engaging human experts in domain-specific tasks.

Figure 3: Three aspects of empirical assessment within the XAI Evaluation
Framework for text.

###  4.1 Functionally-Grounded Evaluations: No Humans, Proxy tasks

Functionally-Grounded Evaluation focuses on assessing XAI methods without the
direct involvement of human participants. This approach, while efficient in
bypassing the complexities of human-subject experiments, challenges us to
identify appropriate proxies that accurately reflect decision-making features.
By utilizing formal definitions of explanations, we define a proxy function to
evaluate the faithfulness of XAI methods. To assess the faithfulness of XAI
explanations, we employ Insertion and Deletion metrics, adapted from image
processing applications to the text domain. These metrics gauge the impact of
highlighted text on model predictions, offering insights into the
explanations’ influence on classification scores. A significant change in
probability scores upon inserting or deleting words indicates the degree of
faithfulness of the explanation method.

The practical implementation of this evaluation involves training a text
classification model on a specified text corpus. Upon classification of input
text, the chosen XAI method is applied to elucidate the model’s predictions,
focusing on the most significant text fragments. This procedure highlights our
approach to validate the faithfulness of explanations generated by XAI
methods.

###  4.2 Human-Grounded Evaluation: Humans in Simplified Tasks

In the realm of Human-Grounded evaluation, the focus shifts to conducting
practical, human-centric experiments that mirror the fundamental aspects of
the intended application. This methodology is especially beneficial when
direct experiments with the target audience are impractical. By involving non-
expert humans in these evaluations, we gain the advantage of a potential
participant pool and reduced costs, as the need for specialized domain experts
is eliminated. Key to this evaluation is the task of assessing the
justifiability and comprehensibilty of explanations provided by XAI methods.
In these tasks, human reasoning and intuition serve as benchmarks for
evaluating the explanations generated by XAI systems. Essentially, we seek to
determine if these explanations resonate with the human understanding of the
decision-making process. To this end, we design tasks that involve systematic
data collection to measure the alignment between human annotations and XAI
explanations. Participants engage in activities such as identifying key
elements—tokens and sentences—that they perceive as influential in the
decision-making context of the model. The effectiveness of XAI methods is then
evaluated based on their ability to mirror these human-selected elements. This
approach not only tests the justifiability of the XAI methods but also
enriches our understanding of how closely machine-generated explanations align
with human thought processes.

###  4.3 Application-Grounded Evaluation: Human experts in Domain-specific
Tasks

In real-world scenarios, the effectiveness of XAI is often measured by its
contribution to crucial objectives like enhancing safety or promoting trust.
This evaluation approach, particularly relevant in contexts where specific
applications are targeted, involves detailed analyses by human experts within
real-world scenarios. It is especially valuable in fields such as medical
diagnostics or legal judgments, where the impact of model decisions can be
substantial.This section aims to evaluate the trustworthiness of XAI methods
through the lens of domain-specific experts. It seeks to determine if the
explanations provided by these methods can help experts grasp the decision-
making process of the underlying models. Such understanding is vital in high-
stakes areas where model decisions carry significant consequences.

To conduct this evaluation, we utilize models trained on domain-specific tasks
and apply different XAI methods to generate explanations for the model’s
predictions. These explanations, along with the predicted outcomes and their
associated probabilities, are presented to domain experts for assessment. The
experts are charged with identifying which XAI method best clarifies the
model’s reasoning in a manner consistent with their expert judgment. An XAI
method favored by the experts would indicate its effectiveness in enhancing
understanding and building trust in the ’black-box’ model.

##  5 Experiments and Results

This section presents the experimental study of our SIDU-TXT method, employing
the threefold XAI assessment framework detailed in Section 4, aimed at
evaluating the model’s interpretative capabilities. Initially, we describe the
datasets utilized for our analysis, highlighting their specific features and
how they align with our research goals. We then describe the training process
of the AI models, tailored for two distinct applications: sentiment analysis
of the IMDB movie reviews and asylum decision prediction. The subsequent
subsections delve into the results derived from our Functionally-Grounded,
Human-Grounded, and Application- Grounded evaluations. These results offer
further insights into the performance and interpretability of the XAI methods
employed, highlighting their practical efficacy in practical implementations.

###  5.1 Datasets

  1. 1.

The IMDB (Internet Movie Database) dataset is a widely-used resource for
sentiment analysis in NLP [18]. It consists of a large collection of movie
reviews, each labeled as either ’positive’ or ’negative’, making it an ideal
benchmark for binary sentiment classification tasks. The dataset typically
comprises 50,000 reviews, evenly split into training and testing sets. Each
review is a variable-length text covering a diverse range of movies, offering
rich linguistic content for analysis. The reviews in the IMDB dataset are
user-generated, reflecting a variety of writing styles and opinions, which
adds to the complexity and realism of sentiment analysis tasks. This dataset
provides a comprehensive platform for developing and evaluating models aimed
at understanding and classifying human emotions and opinions expressed in
written form.

  2. 2.

The Danish asylum decisions dataset, provided by the Refugee Appeals Board,
consists of 14,987 asylum case files from January 1995 to January 2021. It
reflects the distinct nature of the Danish two-tiered asylum process. Each
file offers detailed information about asylum seekers, including personal
backgrounds and specifics of their asylum claims [19]. The dataset is enriched
with narrative accounts, legal reasoning, and various supplementary documents,
making it a rich resource for NLP-based legal analysis. The dataset comprises
cases with diverse outcomes: 15% are granted, 83% rejected and 2% are sentback
cases (pending cases). These figures underline the dataset’s diversity and
complexity, offering a realistic representation of asylum decision-making in
Denmark. This comprehensive data collection serves as an effective tool for
modeling and understanding the nuances of asylum adjudication using deep
learning techniques in NLP.

###  5.2 Training a Model for Sentiment Analysis and Asylum Decision
Prediction

Before delving into the XAI analysis, a predictive AI model is needed. In our
study, we leverage Convolutional Neural Networks (CNNs) to address two
distinct text classification challenges: sentiment analysis of the IMDB movie
reviews and decision prediction for Danish asylum cases. CNNs effectively
process text by translating it into dense vector representations through an
embedding layer, followed by the extraction of features using convolution
layers to identify key patterns like n-grams. The extracted features are
processed through fully connected layers, where softmax or sigmoid functions
are used to determine the probability of each class, enabling precise class
differentiation. This comprehensive process makes CNNs well-suited for our
varied text classification tasks. In addition, the hyperparameter tuning
strategy and hardware specification are briefly discussed.

For each task, we design a specific CNN model, tailored to the unique
characteristics and requirements of the respective datasets. This involves
customizing the architecture and training process to best fit the nature of
the data and the objectives of the analysis. The following subsections detail
the development, training, and evaluation of each model, highlighting the
nuances that make each model suited for its specific task.

####  5.2.1 IMDB Sentiment Analysis Model

In the sentiment analysis of the IMDB movie review dataset, we develop a CNN
model using TensorFlow [20], optimized for distinguishing between positive and
negative reviews. The model employs GloVe pre-trained embeddings [21] in its
initial embedding layer with a dimensional of 300, leveraging these
established word vectors to capture the semantic richness of the reviews. This
layer is followed by a convolutional layer with 128 filters and a kernel size
of 5, using ’ReLU’ activation to extract text features relevant to sentiment
analysis. To enhance the model’s generalizability and reduce over-fitting, we
incorporate dropout layers with a rate of 0.2, following the convolutional and
dense layers. The convolutional layer’s output undergoes dimensionality
reduction through a ’GlobalAveragePooling1D’ layer, leading to a dense layer
of 64 neurons with ’ReLU’ activation. The final output layer comprises a
single neuron with a ’sigmoid’ activation function, making it suitable for
binary classification. For our training and validation purposes, we utilize
the pre-structured split of the IMDB dataset, which comprises 50,000 reviews
divided equally into training and testing sets. From the training portion, we
further allocate 15% for validation, effectively utilizing the dataset for a
comprehensive evaluation of our model. The training was conducted over 30
epochs with a batch size of 32. The model was compiled using binary cross-
entropy as the loss function and optimized with an ’Adam optimizer’ set to a
learning rate of 0.0002, and its performance is primarily evaluated based on
accuracy. This architecture, combining GloVe embeddings with a carefully
structured CNN, provides a robust framework for effective sentiment
classification. The model is trained on the balanced dataset and achieved an
overall accuracy, precision, recall, and F1-score of
$90\%,~{}89.577\%,~{}89.567\%$ and $89.566\%$ respectively, on the test
dataset.

####  5.2.2  Asylum Case Decision Prediction Model

In the context of asylum case decision prediction, the CNN model is
specifically trained on the Danish asylum dataset, as described in Section
5.1, to categorize case files as either ’Granted’ or ’Rejected’. The dataset
is partitioned into 80% for training, 10% for validation, and 10% for testing.
Pre-processing steps include the removal of punctuation, white spaces, and
other insignificant symbols, as well as the lemmatization of text. Unlike
typical preprocessing methods, stop words are retained due to their potential
contextual relevance in legal texts.

The embedding layer, trained with the full vocabulary of the dataset, is set
to produce 300-dimensional feature vectors. This is followed by a
convolutional layer with 128 filters, kernel size 4, and ’ReLU’ activation,
aimed at identifying key legal text features. To mitigate overfitting, dropout
layers at a rate of 0.2 are employed, along with a GlobalAveragePooling1D
layer to reduce output dimensionality. The next stage consists of a dense
layer with 64 neurons (activated by ’ReLU’), and the final output is produced
by a dense layer with 2 neurons using ’Softmax’ activation, suitable for
categorical classification.

For optimization, we employ categorical cross-entropy as the loss function and
the Adam optimizer with a learning rate of 0.0002 and momentum of 0.9. The
training spans over 30 epochs with a batch size of 8, balancing computational
demands with model efficiency. An early stopping strategy is utilized based on
validation loss, complemented by dropout layers for additional regularization.
The asylum dataset has class imbalance, we introduced the class weights. Class
weights alter the loss function directly by penalizing the classes with
varying weights. The trained CNN-NLP predictive model achieved overall
precision, recall and F1-score of $82.948\%,~{}85.188\%$ and $83.184\%$
respectively.

####  5.2.3 Hyperparameter Optimization and Hardware Specifications

In the development of our models for IMDB sentiment analysis and asylum case
decision prediction, we utilize Keras Tuner for hyperparameter optimization.
Keras Tuner employs the Hyperband algorithm [37], an advanced optimization
method that efficiently balances exploration and exploitation of the
hyperparameter space. This approach allows us to effectively and rapidly
identify optimal configurations, including layer structures and learning
rates, within the constraints of our computational resources.

Both models are developed and trained on a system equipped with an Nvidia RTX
2080Ti GPU, providing 11GB of GPU memory. This hardware was crucial for
managing the demanding computational needs of deep learning models,
particularly in light of the extensive datasets and sophisticated model
architectures we work with. The selection of this hardware was instrumental in
achieving the best balance between high model performance and efficient
training.

For the SIDU-TXT XAI analysis, we focus on two important hyperparameters: the
threshold $\tau$ and the number of top masks $K$. We set the threshold at
$\tau=0.5$, for converting sample a value chosen experimentally to optimize
the interpretability of text data within the heatmap. The selection of this
threshold significantly affects the granularity of the interpretability,
impacting how words are highlighted and their contextual meanings are
represented. Unlike in image processing, where threshold variations often have
minimal impact, in text analysis, this setting plays a key role in the quality
of the heatmap visualization. Given this importance, our approach emphasizes
balancing interpretability with noise reduction in heatmap visualization. This
threshold setting, while optimal in our experiments, may be subject to further
refinement for different NLP tasks to achieve more precise interpretability,
respectively. In a similar mechanism, we select the top $K$ masks, setting $K$
to $10$, to generate an output visual explanation rich in contextual
relevance. Masks with lower SIDU weights, offering limited context, are
excluded from the final explanatory visualization, ensuring focus on the most
significant textual features.

###  5.3 Results

####  5.3.1 Functionally-Grounded Evaluation Results

This section presents the results of our Functionally-Grounded evaluation
outlined in Section 4.1. Our focus here is on quantitatively assessing the
faithfulness of XAI methods through a series of deletion and insertion
experiments. These experiments aim to measure how the removal or addition of
key text elements impacts model predictions. Such an approach provides a
practical assessment of the XAI methods’ fidelity to the underlying decision-
making process of the model. To conduct this evaluation, we utilize the model
described in 5.2.1. The evaluation involves systematically deleting words
(setting the word vector to zero) and observing the consequent effect on
classification scores. Additionally, we inserted words in sequence, according
to their relevance scores, to track their impact on model predictions. The
faithfulness of the XAI methods is quantified using the Area Under Curve (AUC)
metric for each text sample, with results derived from 100 randomly selected
texts from the IMDB validation dataset. Our findings, as represented in Table
1, demonstrate the comparative performance of our proposed method against
established XAI methods LIME and GRAD-CAM. The results reveal that our method
surpasses both LIME and GRAD-CAM in terms of the insertion and deletion
metrics, with GRAD-CAM following in performance. LIME’s approach, based on
random perturbations, exhibited limitations in providing contextual relevance
due to its inherent randomness in feature instance generation. This contrasts
with SIDU-TXT, where feature text activation masks are derived from meaningful
internal model activations, offering a more insightful and contextually
relevant analysis.

Table 1: Faithfulness evaluation of XAI methods measured in terms of mean AUC.  XAI METHOD | Insertion (mean AUC) $\uparrow$ | Deletion (mean AUC) $\downarrow$
---|---|---
SIDU-TXT | 0.5513 | 0.1537
GRAD-CAM | 0.4308 | 0.2073
LIME | 0.4228 | 0.2431

####  5.3.2 Human-Grounded Evaluation Resutls

Building on the framework established in the Human-Grounded Evaluation
section, we now present our quantitative findings. These results are derived
from comparing the XAI-generated explanations with human annotations, a key
aspect of our methodology to assess the justifiability of XAI methods.
Employing lexical similarity measures, we quantitatively analyze the alignment
between the explanations provided by XAI methods and the words highlighted by
human annotators. This approach offers a concrete metric to evaluate the
extent to which machine-generated explanations correspond with human
reasoning. Our study begins with the systematic collection of data to assess
the alignment between human sentiment analysis and the visual heatmap provided
by XAI methods. We collect $50$ short movie reviews from the IMDB database as
the basis for this analysis. The aim is to compile a set of human annotations
to act as a standard for comparison against the sentiment assessments
generated by XAI. Four annotators, each reviewing the same set of 50 reviews,
participated in the study. They each read the reviews and classify them as
either ’positive’ or ’negative’ based on their assessment of the overall
sentiment. They are tasked with identifying up to 10 words or phrases that
influence their sentimental decision. In addition, they highlight up to five
sentences that they believe best exemplify the sentiment of the review. Figure
4 shows an example of an annotation provided by one participant for a review
from our dataset of 50 movie reviews. The final set of annotations is formed
by creating a union of the inputs from all four participants. This union
process involves compiling all unique words and sentences identified by any
participant, ensuring no duplicates are included in the collective set. This
aggregated data provides a robust benchmark for the subsequent evaluation of
XAI methods, offering a substantial measure of the consistency between AI-
generated interpretations and collective human judgment both at a token level
and sentence level.

• Input Text : “Hobgoblins is a very cheap and badly done Gremlins rip-off.
That’s the best thing one can say about this stinkpile. Pretty much everyone
in the cast was chosen for their looks and not their acting ability. It was
very painful to watch. Avoid this one at all costs.” • Label the Review:
Positive Negative • Characterizing Words: Based on your label judgment, list
up to 10 words from the review that characterize its sentiment. These are
words that you think significantly support your chosen label. Cheap, badly
done, stinkpile, painful, avoid • Significant Sentences or Contexts: Highlight
or pick up to 5 sentences or contexts in the review that support your selected
label. These are sentences or parts of sentences that you believe most clearly
convey the sentiment of the review. “Hobgoblins is a very cheap and badly done
Gremlins rip-off.” “That’s the best thing one can say about this stinkpile.”
“It was very painful to watch. Avoid this one at all costs.”

Figure 4: Annotation sample from one participant for a movie review in the
dataset.

In the next step, to quantitatively compare the words highlighted by XAI
methods with human-collected annotations, one can utilize lexical similarity
measures common in NLP applications. These measures assess the similarity
between two text sets through the intersection of their word sets,
irrespective of language. A lexical similarity score of 1 denotes complete
overlap, indicating identical texts, whereas a score of 0 means there are no
shared words. Such measures are useful in NLP for comparing word or phrase
sets identified by different annotators or processes. The Jaccard similarity
index and cosine similarity are among the popular methods used for such
evaluation purposes [38]. Our study employs the Jaccard similarity coefficient
to facilitate direct quantitative comparison between influential words
identified by human annotators and those highlighted by XAI techniques. We
apply a threshold to the XAI scores to include only those words also selected
by human annotators. This approach means that, for example, if humans
identified up to 5 keywords, only the corresponding XAI scores exceeding the
threshold are considered. At a threshold of 0, all words scored by the XAI
method and identified by humans are included. As the threshold increases,
stricter criteria are applied, and only words with an XAI score above the
threshold are compared. For instance, at a threshold of 0.8, only words
matching those identified by humans and with an XAI score above 0.8 are
selected for comparison. The average Jaccard similarity results between 50
human-annotated movie reviews at the token level and XAI-generated token
scores are presented in Figure 5.

Figure 5: Average Jaccard Similarity Scores between Human-Annotated token and
XAI-Generated token score for $50$ Movie Reviews.

As it can be seen in Figure 5, the average Jaccard similarity decreases as the
threshold increases for all three methods. This is because a smaller subset of
words meets the criteria to be included in the comparison set between the XAI
and the ground truth– reducing the likelihood of overlap with the human-
annotated set. At lower thresholds, SIDU-TXT exhibits the highest average
Jaccard similarity, suggesting that when considering a broader set of words,
SIDU-TXT aligns more closely with human judgment compared to the other
methods. LIME maintains a moderate level of similarity across all thresholds,
suggesting a consistent but less pronounced alignment with human annotations.
On the other hand, while Grad-CAM shows steady performance at different
threshold levels, it might not detect as wide a range of important words as
SIDU-TXT. This highlights SIDU-TXT’s sensitivity to a broader range of
influential tokens, which is indicative of its superior alignment with human
judgment in the sentiment analysis tasks we evaluated. The observed decline in
similarity for all methods as the threshold increases also raises important
considerations about the selection of a threshold for practical applications.
A lower threshold may capture a larger, potentially more comprehensive set of
influential words but at the risk of including less relevant tokens.
Conversely, a higher threshold may result in a more focused set of words but
could miss out on tokens that humans find influential. In conclusion, SIDU-
TXT, particularly at a lower threshold, emerges as a strong candidate for
scenarios where a comprehensive extraction of influential words is paramount.

Given the complexities discussed in XAI evaluation within the literature,
which we have also addressed in the Related Work section, where factors that
may be faithful to the model’s rationale may not always be intuitive to human
reasoning, in addition to our quantitative investigations, we further employ a
qualitative study in our case of sentiment analysis. This involves examining
the heatmaps generated by SIDU-TXT, LIME, and Grad-CAM, assessing them
qualitatively against the annotations we collected. Such a comparison allows
us to observe the degree of agreement between the AI-generated interpretations
and the human perspective.

LIME’s heatmap is produced through a process that randomly samples the
vicinity of the input data and generates perturbations. This method assesses
the model’s behavior with these variations to identify both positive and
negative contributions to the final prediction. By nature, LIME’s technique
for text analysis involves a duality: it highlights contributions that
increase (in orange) or decrease (in blue) the model’s confidence in its
decision. The intensity of the color shades on the highlighted words in the
heatmap illustrates the strength of each word’s contribution, whether positive
or negative, to the model’s prediction. Grad-CAM generates its heatmap by
applying the gradient-based localization technique to the last convolutional
layer of the neural network. This process involves backpropagating the signal
from the output layer to the text input level, thus producing a visualization
that highlights words according to their impact on the model’s decision-making
process. SIDU-TXT, like Grad-CAM, utilizes the last convolutional layer of the
given model. However, instead of tracing gradients, SIDU-TXT employs a
Similarity and Uniqueness metric on a selected feature mask, as detailed in
Section 4, to generate the heatmap of highlighted words. For both Grad-CAM and
SIDU-TXT methods, shades of orange are utilized within the heatmaps to denote
the strength of the scores assigned to words in each method, respectively.
These shades correspond to a spectrum of ten intervals that represent the XAI
methods’ scoring range. The more intense the color shade, the closer the score
is to 1, indicating a higher influence of the word on the model’s prediction,
and conversely, a paler color suggests a lower influence. Table 2 and Table 3
illustrate the qualitative comparison of XAI methods with respect to human
token-level annotations for two samples (negative and positive) in sentiment
analysis. To ensure consistency with human annotators who were instructed to
identify up to $10$ influential words, we adapt our analysis to focus on the
same number of words across all XAI methods. This offers a more standardized
basis for comparison and maintains alignment with the human-provided
annotations.

For our analysis, here we consider the ’Negative sentiment’ example and the
corresponding heatmaps generated by the XAI methods as presented in Table 2.

Table 2: Comparative heatmaps of the top 10 words from ’Negative sentiment’ IMDB dataset reviews. Yellow in Human Annotation denotes equal importance across words. LIME shows negative significance in varying shades of blue and positive in orange, while deeper shades in GRAD-CAM and SIDU-TXT indicate words of greater importance for the XAI methods. Method | Heatmaps
---|---
Human Annotation |
LIME |
Grad-CAM |
SIDU-TXT |

In case of LIME, the majority of words are highlighted in shades of orange,
suggesting a positive contribution to the sentiment classification. This is in
contrast to the review’s overall negative sentiment and human annotation.
Notably, the word ”badly” is highlighted in blue, partially aligning with the
human-annotated negative sentiment. The word ”avoid” is also marked in blue
but with less intensity, suggesting a lesser importance. However, the presence
of other blue-highlighted words, which appear to be randomly chosen and
irrelevant to the sentiment, adds to the confusion. This mixed representation
of sentiment by LIME, with a dominance of positive attributions in a negative
context, underscores the challenge in ensuring that XAI evaluations provide
clear and contextually relevant interpretations for human being. In case of
Grad-CAM, the heatmap slightly better aligns with human judgment, spotlighting
negative terms such as ”badly done” and ”avoid this” in orange shades.
However, it also applies this highlighting to neutral terms emphasizing same
challenge as in the case of LIME. SIDU-TXT’s heatmap, on the other hand,
aligns more closely with the words chosen by human annotators, strongly
highlighting clear negative words like ’badly done’ and ’painful’ with the
most intense shades of orange. It also picks up additional words that fit the
negative sentiment of the review, even if they are not picked up by human
annotators (e.g., cheap). Like LIME and Grad-CAM, SIDU-TXT may occasionally
highlight irrelevant words, but it marks these less important words with a
lower intensity. This shows that SIDU-TXT is is capable of recognizing the
relative insignificance of these tokens to a certain extent. A similar
analysis and conclusion can be drawn for SIDU-TXT concerning the ’Positive
Sentiment’ example displayed in Table 3, where both LIME and Grad-CAM methods
exhibit a slight improvement over their performance with the negative example.

Table 3: Comparative heatmaps of the top 10 words from ’Positive sentiment’ IMDB dataset reviews. Yellow in Human Annotation denotes equal importance across words. LIME shows negative significance in varying shades of blue and positive in orange, while deeper shades in GRAD-CAM and SIDU-TXT indicate words of greater importance for the XAI methods. Method | Heatmaps
---|---
Human Annotation |
LIME |
Grad-CAM |
SIDU-TXT |

In summary, while LIME and Grad-CAM provide valuable insights, they
occasionally incorporate extraneous words that may lead to misinterpret the
sentiment context. This proves once again the inherent challenges in
generating explanations that are fully in agreement with human reasoning, a
persistent and complex issue in NLP domain. SIDU-TXT while not being a
bulletproof method against similar challenges, appears to have a good grasp of
the key terms that convey sentiment, particularly negative sentiment in this
case. It further offers a more refined analysis that aligns better with human
rationales at the token level.

In the second experiment, we conducted a quantitative comparison of sentence-
level annotations. The primary goal is to evaluate the alignment between XAI
methods and human-generated sentence-level rationales. Given that a sentence
or phrase often conveys more meaningful context than a single word (token),
our focus is on assessing how effectively XAI methods highlight the
surrounding context, a group of words, around the most crucial words that
contribute meaningful information. The key criterion is whether the XAI method
can effectively emphasize sequences of words or phrases, rendering the
explanation comprehensible to humans.

For the quantitative evaluation at the sentence level, employing human
sentence-level annotations, we treat it as a binary classification problem. We
measure the comprehensibility in terms of precision, recall, and F1-score for
the human-annotated data samples. Table 4, provides a summary of the mean
precision, recall, and F1-score for the XAI methods.

Table 4: Sentence-level evaluation with Human ground truth Method | Mean Precision $\uparrow$ | Mean Recall$\uparrow$ | Mean F1-SCORE$\uparrow$
---|---|---|---
SIDU-TXT | 0.570 | 0.515 | 0.504
GRAD-CAM | 0.399 | 0.484 | 0.383
LIME | 0.289 | 0.387 | 0.287

Observing the results, SIDU-TXT methods exhibit high precision, recall, and
F1-score, closely followed by Grad-cam. This is evident from the showcased
samples in Table 2 and Table 3. The examples illustrate that SIDU can
effectively highlight sequences of words adjacent to the most crucial tokens,
collectively conveying meaningful information. In contrast, GRAD-CAM and LIME
struggle to emphasize sequences of words adjacent to the most crucial tokens.
In conclusion, we assert that a robust explanation method should be capable of
highlighting sequences of words alongside the most crucial words, ensuring a
comprehensible explanation.

####  5.3.3 Application-Grounded Evaluation Results

In our Application-Grounded evaluation, we conduct a qualitative study with
legal experts to assess the interpretability of heatmaps generated by the
SIDU-TXT, Grad-CAM and LIME XAI methods in the sensitive context of asylum
case decision-making. This evaluation is key to understanding the real-world
effectiveness of these methods, focusing on the experts’ trust in the models’
interpretability.

Three researchers, who are experts within both academic and have practitioner
experience in asylum law, are presented with heatmaps from these XAI methods
for three asylum case files. Each of these case files contains, on average,
5000 words. The experts’ assessments focus on the overall explanation quality
of each heatmap, particularly how effectively each method explains predictions
and highlights legally relevant words and sentences. This involved examining
the heatmaps’ ability to capture crucial aspects of asylum cases, such as the
grounds for claiming asylum, applicant characteristics, and the credibility of
evidence. To ensure unbiased evaluations, the experts are not informed about
the names of the methods, and the order of presentation is randomized.

SIDU-TXT demonstrated effectiveness in identifying legally relevant
information, including details about the asylum claim, travel routes, facts
about the motive, and applicant characteristics such as citizenship, religion,
and ethnicity. However, it occasionally missed critical details, like
documentation of the applicant’s experiences, thus affecting its overall
completeness in explainability from a traditional legal perspective.

Conversely, Grad-CAM exhibited a more comprehensive approach, with its heatmap
effectively highlighting detailed descriptions of facts and evidence,
including medical assessments and event chronologies. However, this model also
presented challenges, notably in the form of a significant proportion of stop
words and false positives, which impacted its precision.

LIME predominantly highlighted stop words, lacking contextual information,
although it captured some aspects related to credibility.

In summary, while SIDU-TEXT and Grad-CAM, both belonging to the same class of
activation-based XAI methods, succeeded in gaining the expert’s trust to
varying degrees in highlighting relevant information, both methods exhibited
areas of underperformance, indicating the need for further advancing XAI
methods for complex real-world applications. The difference in their emphasis
on legal reasoning and applicant characteristics highlights an epistemological
gap between internal model logic and external validity criteria. This divided
judgment, particularly in such intricate and challenging decision-making
tasks, experts underscores the need for continuous improvements in model
development to bridge this gap.

##  6 Conclusion and Future works

In this paper, we extend the novel ’Similarity Difference and Uniqueness’
(SIDU) XAI method [14], originally developed for explaining CNN models in the
image domain, to the realm of NLP. Specifically, SIDU-TXT investigates textual
predictions through granular, word-based heatmaps by utilizing feature
activation maps of convolution layers in the 1D CNN model. In this adaptation,
we acknowledge the importance of striking a balance between the semantic
interpretation of contextual text presented in the visual heatmap and the
filtering out of noisy, irrelevant non-contextual elements. Consequently,
SIDU-TXT focuses on contextually rich features by selecting the top $K$
feature activation masks. This approach provides a deeper understanding of how
the proposed XAI model interprets textual data within the decision-making
pipeline, ensuring that only the most relevant and informative features are
highlighted.

To evaluate the performance of the proposed SIDU-TXT method, we conducted
comprehensive and holistic evaluations focusing on the faithfulness,
justifiability, comprehensibility, and trustworthiness of explanations using
three distinct categories of XAI assessments. They are: Functionally-Grounded
(to assess the faithfulness of explanations), Human-Grounded (to assess the
justifiability and comprehensibility), and Application-Grounded evaluations
(to gain trust from domain experts, examplified in the law domain). For a
faithful analysis of the XAI explanations that measure the fidelity of the
highlighted features to the underlying decision-making process in the
sentiment prediction task, we employ Functionality-Grounded evaluation
metrics, specifically insertion and deletion tests. The results demonstrate
that SIDU-TXT outperforms both LIME and Grad-CAM in terms of these metrics.
Additionally, to assess how closely the generated explanations align with
human justifications, we performed a lexical similarity analysis at both the
word and sentence levels using the Jaccard similarity coefficient. To
understand this agreement between the influential features (words/sentences)
identified by laypersons and those highlighted by the XAI methods, we
collected human ground-truth sentiment annotations from 50 short movie reviews
drawn from the IMDB database. A comparative study of the features highlighted
by human annotators and the XAI methods revealed that, in movie sentiment
analysis, SIDU-TXT yields Jaccard similarity closer to $1$ at the word level
and more closely aligns with human annotations. This close alignment is
further qualitatively validated by comparing the top $10$ highlighted words in
each XAI method, drawing similar conclusions as in the lexical analysis.
Moreover, for comprehensibility in our Human-Grounded evaluation, we perform a
quantitative analysis of the contextual similarity between human intuition and
the XAI methods at the sentence level. To achieve this, our approach involves
assessing similarity via binary classification and computing metrics such as
precision, recall, and F1-score. The analysis indicates that SIDU-TXT
surpasses Grad-CAM and LIME, effectively suggesting that it can more
efficiently exploit and highlight sequences of contextual words adjacent to
crucial words that provide comprehensible explanations. In the final phase of
our evaluation framework, the Application-Grounded, we qualitatively assess
the power of interpretability in complex real-world applications to gain the
trust of experts in the sensitive legal domain. We employ legal experts to
qualitatively assess the heatmaps generated by SIDU-TXT, LIME, and Grad-CAM in
the asylum case decision-making process. The qualitative assessment reveals
that both SIDU-TXT and Grad-CAM outperform LIME in gaining trust from experts,
yet each shows strengths and weaknesses in different aspects. However, it is
evident that both methods still fall short of the desired level of performance
required for an XAI model to be fully utilized in real-life complex NLP tasks.
This underscores the need for further research and development in XAI methods
to bridge the epistemological gap between current capabilities and the demands
of human-centered, high-stakes decision-making environments. These challenges
also echo the intricate nature of interpretability in the NLP domain, as
discussed in the Related Work section. Specifically, our comprehensive
evaluation framework demonstrates SIDU-TXT’s superiority in certain contexts,
such as sentiment analysis of IMDB movie reviews, but also reveals the
difficulty in drawing similar conclusions for more complex tasks like asylum
case analyses compared to downstream sentiment analysis tasks. One potential
reason for this observed behavior, aside from the inherently challenging
semantics of the problem and the high expectations for precision and
comprehension in such specific applications, may lie in the foundational model
used: for sentiment analysis, we employ a pre-trained Glove embedding, which
facilitates better contextual understanding, while in the asylum case study,
the absence of such a robust, language and domain-specific foundation model
due to limitations in available legal text embeddings, could contribute to the
challenges encountered.

Consequently, in future work, we should aim to enhance SIDU-TXT’s application
in complex NLP tasks by expanding our qualitative evaluations with more case
studies with the help of experts and testing the method across various NLP
tasks for broader adaptability, and language-specific embeddings as well as
developing domain-specific. These steps are crucial for improving
interpretability and reliability in sensitive domains, addressing current
limitations, and paving the way for more effective XAI methods.

## Declarations

  * •

Funding : Research for this publication is principally conducted as part of
the Villum Interdisciplinary grant for the project Explainable Artificial
Intelligence and Fairness in Asylum Law (XAIfair) and with support from
research funded by the Nordic Research Council grant no. 105178 - Nordic
Refugee Determination: Advancing Data Science in Migration Law (NoRDASiL), the
Volkswagen Foundation Algorithmic Fairness for Asylum Seekers, Refugees (AFAR)
and the Danish National Research Foundation Grant no. DNRF169 - Centre of
Excellence for Global Mobility Law. It is also a part of the Independent
Research Fund Denmark grant for the project Contestable Artificial
Intelligence-defining, evaluating and communicating AI contestability in
healthcare, law and finance. Additionally, funding is provided by the
Responsible AI for Value Creation project (REPAI) funded by the Grundfos
Foundation.

## References

  * [1] J. Zhang, P. Lertvittayakumjorn, and Y. Guo.  Integrating semantic knowledge to tackle zero-shot text classification.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Volume 1, pp. 1031–1040, 2019.
  * [2] D. Bahdanau, K. Cho, and Y. Bengio.  Neural Machine Translation by Jointly Learning to Align and Translate.  _International Conference on Learning Representations_ , volume. abs/1409.0473, 2014.
  * [3] A. Bibal et al.  Is attention explanation? An introduction to the debate.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , Volume 1, 2022.
  * [4] K. McGregor Richmond, S. M. Muddamsetty, T. Gammeltoft-Hansen, H. Palmer Olsen, and T. B. Moeslund.  Explainable AI and Law: An Evidential Survey.  _Digital Society_ , Springer, 2(3), 2023.
  * [5] F. Cerutti et al.  Interpretability of deep learning models: A survey of results.  In _Proceedings of the IEEE UIC/ATC_ , pp. 1–6, 2017.
  * [6] O. M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom.  e-SNLI: Natural Language Inference with Natural Language Explanations.  _Advances in Neural Information Processing Systems_ , vol. 2018-December, pp. 9539–9549, 2018.
  * [7] A. Adadi and M. Berrada.  Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI).  _IEEE Access_ , vol. 6, pp. 52138–52160, 2018.
  * [8] F. Doshi-Velez and B. Kim.  Towards A Rigorous Science of Interpretable Machine Learning.  _arXiv preprint arXiv:1702.08608v2_ , Feb. 2017.  Available: https://arxiv.org/abs/1702.08608v2
  * [9] P. Lertvittayakumjorn and F. Toni.  Human-grounded Evaluations of Explanation Methods for Text Classification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_ , pp. 5195–5205, 2019  DOI: 10.18653/v1/D19-1523.
  * [10] C. J. Mahoney et al.  A framework for explainable text classification in legal document review.  In _2019 IEEE International Conference on Big Data_ , IEEE, 2019.
  * [11] M. Danilevsky et al.  A Survey of the State of Explainable AI for Natural Language Processing.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pp 447–459, 2020.
  * [12] M. T. Ribeiro, S. Singh, and C. Guestrin.  Model-agnostic interpretability of machine learning.  In _ICML Workshop on Human Interpretability in Machine Learning_ , 2016.
  * [13] R. R. Selvaraju et al.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2017.
  * [14] S. M. Muddamsetty et al.  Visual explanation of black-box model: Similarity Difference and Uniqueness (SIDU) method.  _Pattern Recognition_ , 127:108604, 2022.
  * [15] H. Hajiyan, H. Davoudi, and M. Ebrahimi.  A Comparative Analysis of Local Explainability of Models for Sentiment Detection.  In _Proceedings of the Future Technologies Conference_ , Springer International Publishing, 2022.
  * [16] S. Jain and B. C. Wallace.  Attention is not Explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_ , 2019.
  * [17] F. Doshi-Velez and B. Kim.  Considerations for evaluation and generalization in interpretable machine learning.  In _Explainable and Interpretable Models in Computer Vision and Machine Learning_ , pages 3–17, Springer, 2018.
  * [18] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pp. 142–150, 2011.
  * [19] UN High Commissioner for Refugees (UNHCR).  Nordic Asylum Practice in relation to Religious Conversion: Insights from Denmark, Norway and Sweden.  December 2023, PPLA/2023/02.  Available: https://www.refworld.org/docid/6571d6614.html
  * [20] M. Abadi et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  2015\.  Software available from tensorflow.org.  Available: https://www.tensorflow.org/
  * [21] J. Pennington, R. Socher, and C. D. Manning.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1532–1543, 2014.
  * [22] X. Zhang, J. Zhao, and Y. LeCun.  Character-level convolutional networks for text classification.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15_ , pp. 649–657, 2015.
  * [23] B. Gambäck and U. K. Sikdar.  Using convolutional neural networks to classify hate-speech.  In _Proceedings of the First Workshop on Abusive Language Online_ , pp. 85–90, 2017.
  * [24] Y. Kim.  Convolutional neural networks for sentence classification.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1746–1751, 2014.
  * [25] J. Li, W. Monroe, and D. Jurafsky.  Understanding neural networks through representation erasure.  _International Conference on Learning Representations_ , volume. abs/1612.08220, 2016.
  * [26] H. Liu, Q. Yin, and W. Y. Wang.  Towards explainable NLP: A generative explanation framework for text classification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 5570–5581, 2019.
  * [27] L. Arras, G. Montavon, K.-R. Müller, and W. Samek.  Explaining recurrent neural network predictions in sentiment analysis.  In _Proceedings of the EMNLP’17 Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis (WASSA)_, pp. 159–168, 2017.
  * [28] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi.  A Survey Of Methods For Explaining Black Box Models.  _ACM Computing Surveys_ , vol. 51, 2018.
  * [29] S. Luo et al.  Local interpretations for explainable natural language processing: A survey.  _arXiv preprint arXiv:2103.11072_ , 2021.  Available: https://arxiv.org/abs/2103.11072
  * [30] D. Nguyen.  Comparing automatic and human evaluation of local explanations for text classification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 1069–1078, 2018.
  * [31] M. Du, N. Liu, F. Yang, S. Ji, and X. Hu.  On Attribution of Recurrent Neural Network Predictions via Additive Decomposition.  In _The Web Conference 2019 - Proceedings of the World Wide Web Conference_ , pp. 383–393, May 2019.  DOI: 10.1145/3308558.3313545
  * [32] E. Cambria et al.  A survey on XAI and natural language explanations.  _Information Processing & Management_, 60(1):103111, 2023.
  * [33] M. Nauta et al.  From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI.  _ACM Computing Surveys_ , 55(13s):1-42, 2023.
  * [34] W. Xiong et al.  Looking deeper into deep learning model: Attribution-based explanations of TextCNN.  In _NIPS 2018 Workshop on Challenges and Opportunities for AI in Financial Services_ , 2018.  DOI: 10.48550/arXiv.1811.03970.
  * [35] N. Poerner, H. Schütze, and B. Roth.  Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , pp. 340–350, 2018.
  * [36] S. Mohseni and E. D. Ragan.  A human-grounded evaluation benchmark for local explanations of machine learning.  _International Conference on Learning Representations_ , 2018.
  * [37] L. Li et al.  Hyperband: A novel bandit-based approach to hyperparameter optimization.  _The Journal of Machine Learning Research_ , 18(1):6765-6816, 2017.
  * [38] M. K. Vijaymeena and K. Kavitha.  A survey on similarity measures in text mining.  _Machine Learning and Applications: An International Journal_ , 3(2):19-28, 2016.


# Field-like torque induced synchronized oscillation in spin torque
oscillators

Rajasekaran Arun1 Ramupillai Gopal2 Vallipalayam Kuppusamy Chandrasekar2
Muthusamy Lakshmanan1 1Department of Nonlinear Dynamics, School of Physics,
Bharathidasan University, Tiruchirapalli-620024, India
2Centre for Nonlinear Science & Engineering, School of Electrical &
Electronics Engineering, SASTRA Deemed University, Thanjavur- 613 401, India.

###### Abstract

Synchronization of two spin torque oscillators coupled in parallel through
current is investigated by taking into account the influence of field-like
torque. It is shown that the presence of such torque can cancel out the effect
of damping and thus can induce synchronized oscillations in response to the
direct current. Further, our results show that the existence of positive
field-like torque enhances the frequency of the synchronized oscillations
without affecting the power. The validity of the above results is confirmed by
numerical and analytical studies based on the Landau-Lifshitz-Gilbert-
Slonczewski equation.

nonlinear dynamics,spintronics,synchronization

##  I Introduction

Synchronization phenomenon in spin torque oscillators (STOs) has been the
subject of active research in recent years due to its potential applications
to generate microwave power in nanoscale devices uraz ; mancoff ; gro ; kaka ;
persson . A number of significant efforts have been made to study
magnetization dynamics and synchronization of STOs driven by spin polarized
current slon:96 , injection locking adler , external ac excitation Li:06 ;
tibo , spin waves kend:14 , magnetic fields subash:13:15 ; subash1 ; gopal
and electrical couplings rippard:05 ; piko . The synchronization of STOs
greatly enhances the output microwave power when compared with the low output
power of an individual STO. Also it is more desirable for an enhancement of
efficiency, quality factor and oscillation frequency of the practical STO
devices such as high density microwave signal processors and chip-to-chip
communication system rippard:05 ; geor ; zhou ; nakada ; zeng:12 . Moreover,
synchronization of STOs has also been identified in new applications such as
wireless communication, brain-inspired computing and microwave assisted
magnetic reading chio:14 ; loca:14 ; gro1 ; kudo ; torr . In particular, it
has been observed that an STO with the configuration of perpendicularly
magnetized free layer and in-plane magnetized pinned layer is suitable for
high emission of power, narrow line width and wide frequency tunability
zeng:12 ; mangin ; houss ; ripp ; kubota:13 . The oscillation properties of
this STO have also been investigated both experimentally and theoretically in
Refs. kubota:13 ; tani:13 . Further, the existence and stability of the
synchronized state and the conditions to synchronize the individual
precessions have also been studied in an array of $N$ serially connected
identical STOs coupled through current, also clearly demonstrated in Ref.
turtle . Recently, the mutual synchronization between two parallelly connected
STOs, coupled by current, has also been identified tomo:18 .

However, the major issues in the system of coupled STOs are the formation of
multistability and the decrease of frequency with respect to current. The
nature of multistability prevents the system to exhibit stable synchronized
oscillations for all initial conditions. Removing this multistability and
making the system to exhibit stable synchronized oscillations are challenging
tasks and have not yet been fully clarified. Also, a decrease in the frequency
of synchronized oscillations while increasing current limits the enhancement
of frequency beyond some value.

In this paper, we study the mutual synchronization between two parallelly
coupled STOs in the presence of field-like torque tani:14 ; tani1:14 ; tani:15
; guo:15 . By solving the Landau-Lifshitz-Gilbert-Slonczewski(LLGS) equation
with the configuration of perpendicularly magnetized free layer and in-plane
magnetized pinned layer, the analytical formula for the frequency of
synchronized oscillations is derived tani:14 ; tani:15 ; tomo:18 . The
existence of multistability is confirmed and the impact of field-like torque
on the STOs for the various strengths of coupling is observed. In the absence
of field-like torque the two STOs show the existence of multistable states
that include synchronized oscillations and steady state. The presence of
field-like torque removes the multistability and makes the system to oscillate
with in-phase synchronization. Further, the frequency of synchronized
oscillations is also enhanced in the presence of field like torque. The onset
of multistability in the absence of such a torque and the onset of
monostability due to it are analytically verified.

##  II Model description of two parallelly coupled STOs

Figure 1: (Color online) Time evolution of $m_{1x}$ and $m_{2x}$ for (a)
$\beta$=0 and (b) $\beta$=0.1 when the currents passing through the first,
second oscillators and the applied field are instantaneously switched off at
500 ns and switched on at 1500 ns. Time evolution of $m_{1x}$ and $m_{2x}$ for
(c) $\beta$=0 and (d) $\beta$=0.1 when the currents passing through the first
oscillator, second oscillator and applied field are cut off at 504 ns, 500 ns
and 496 ns respectively and switched on simultaneously at 1500 ns. Here the
currents passing through the two oscillators are given values 2.0 mA and the
coupling strength is 0.5. The inset figures show the corresponding time
evolution of $m_{z}$.

We consider a system that consists of two parallelly coupled spin-torque
oscillators. Each oscillator consists of a perpendicularly magnetized free
layer, where the direction of magnetization is allowed to change and an in-
plane magnetized pinned layer where the direction of magnetization is fixed
along the positive x-direction. Both free and pinned layers are separated by a
nonmagnetic conducting layer. The two free layers are labeled as $j=1,2$ and
the material parameters of the two oscillators are kept identical for
simplicity. The unit vector along the direction of free layer’s magnetization
is given by ${\bf m}_{j}=(m_{jx},m_{jy},m_{jz})$. The z axis is kept
perpendicular to the plane of the free layer and ${\bf e}_{x}$,${\bf e}_{y}$
and ${\bf e}_{z}$ are unit vectors along positive x,y and z directions
respectively. The unit vector along the direction of magnetization of the
pinned layers is given by ${\bf P}(={\bf e}_{x})$. The magnetization of the
free layers $(j=1,2)$ is governed by the following LLGS equation,

$\displaystyle\frac{d{\bf m}_{j}}{dt}=-\gamma{\bf m}_{j}\times{\bf
H}_{eff,j}+\alpha{\bf m}_{j}\times\frac{d{\bf m}_{j}}{dt}$
$\displaystyle+\gamma H_{sj}{\bf m}_{j}\times({\bf m}_{j}\times{\bf
P})+\gamma\beta H_{Sj}{\bf m}_{j}\times{\bf P},~{}j=1,2.$ (1)

Here ${\bf H}_{eff,j}$ is the effective field, given by ${\bf
H}_{eff,j}=[H_{a}+(H_{k}-4\pi M_{s})m_{jz}]{\bf e}_{z}$, which includes
externally applied field $H_{a}$, crystalline anisotropy field $H_{k}$ and
shape anisotropy field (or demagnetizing field) $4\pi M_{s}$. $M_{s}$ is the
saturation magnetization, $\gamma$ is the gyromagnetic ratio, $\alpha$ is the
Gilbert damping parameter, $\beta$ is the strength of the field-like torque
and $H_{Sj}$ is the strength of the spin-transfer torque, given by

$H_{Sj}=\frac{\hbar\eta I_{j}}{2eM_{s}V(1+\lambda{\bf m}_{j}.{\bf P})}.$ (2)

In Eq.(2) $\hbar=h/2\pi$ ($h$ \- Planck’s constant), $V$ is the volume of the
free layer, $e$ is the electron charge, $\eta$ and $\lambda$ are dimensionless
parameters which determine the magnitude and the angular dependence of the
spin transfer torque respectively. $I_{j}$ is the total current passing
through the free layer which is given by tomo:18

$I_{j}=I_{0}+I_{j}^{coupling}=I_{0}+I_{0}\chi[m_{jx}(t)-m_{j^{\prime}x}(t)],$
(3)

where $j,j^{\prime}=1,2,~{}j\neq j^{\prime}$ and $I_{j}^{coupling}$ is the
current injected from the free layer $j^{\prime}$ to $j$. In Eq.(3) $I_{0}$ is
the current flowing through the free layer when there is no coupling between
the oscillators. The second term in Eq.(3) describes the current flowing
through the connection between the two STOs and $\chi$ is the coupling
strength which characterizes the energy loss in the connector. The oscillating
electric current generated by the STO is proportional to
$[2V_{i}/(R_{P}+R_{AP})][1+\triangle R({\bf m}_{j}.{\bf P})/(R_{P}+R_{AP})]$
as pointed out in tomo:18 , which implies that the electric current generated
by the oscillator depends upon the component of the free layer’s magnetization
along the pinned layer’s magnetization. Here $V_{i}$ is the external voltage,
$R_{P}$ and $R_{AP}=R_{P}+\triangle R$ are the resistances of the STO when the
magnetization of the free layer is parallel and antiparallel to the
magnetization of the pinned layer, respectively.

##  III Effect of field-like torque

###  III.1 Destabilization of steady state by field-like torque

To understand the dynamics of the magnetization of the free layer, Eq.(1) is
numerically solved by Runge-Kutta 4th order step-halving method for the
material parameters kubota:13 ; tani:13 ; tomo:18  $M_{s}=1448.3$ emu/c.c.,
$H_{k}=18.6$ kOe, $\eta$ = 0.54, $\lambda$ = $\eta^{2}$, $\gamma$ = 17.64
Mrad/(Oe s), $\alpha$ = 0.005, and $V=\pi\times 60\times 60\times 2$ nm3.
Throughout our study $H_{a}$ is fixed as 2.0 kOe. The dynamics of the coupled
spin torque oscillators is more complicated than that of a single oscillator.
Here we show that the dynamics of the two oscillators can be altered when
there is a lack of simultaneity between the currents passing through the
individual oscillators and the external magnetic field when they are switched
off at different times with even nanosecond difference. As an example,
initially the currents to the first and second oscillators are switched on at
$\tau_{I_{1},1}^{on}$ and $\tau_{I_{2},1}^{on}$, respectively, and the field
at $\tau_{H_{a},1}^{on}$. After the oscillators attain synchronized
oscillations, the currents and field are switched off at
$\tau_{I_{1},1}^{off}$, $\tau_{I_{2},1}^{off}$ and $\tau_{H_{a},1}^{off}$.
After some time they are again switched on at $\tau_{I_{1},2}^{on}$,
$\tau_{I_{2},2}^{on}$ and $\tau_{H_{a},2}^{on}$, respectively. Figs.1(a)-(d)
show the time evolution of $m_{1x}$ and $m_{2x}$ for the initial conditions
chosen for $0.99<m_{1z},m_{2z}<1.00$, when $I_{0}$ = 2.0 mA, $\chi$ = 0.5. The
corresponding values of $m_{1z}$ and $m_{2z}$ are plotted as inset figures.
Figs.1(a) and (b) have been plotted for $\beta$=0 and $\beta$=0.1,
respectively, when
$\tau_{I_{1},1}^{on}=\tau_{I_{2},1}^{on}=\tau_{H_{a},1}^{on}=0s$,
$\tau_{I_{1},1}^{off}=\tau_{I_{2},1}^{off}=\tau_{H_{a},1}^{off}=500~{}ns$ and
$\tau_{I_{1},2}^{on}=\tau_{I_{2},2}^{on}=\tau_{H_{a},2}^{on}=1500~{}ns$.
Figs.1(a) & (b) show that irrespective of whether the field-like torque is
present or not, both the oscillators reach steady state after 500 ns and
regain synchronized oscillations after 1500 ns as confirmed in Ref.tomo:18 .
To mimic the actual experimental situations, where the currents passing
through the individual oscillators and external field may not reach the value
zero simultaneously at the moment when they are switched off, Figs. 1(c)&(d)
are plotted for
$\tau_{I_{1},1}^{on}=\tau_{I_{2},1}^{on}=\tau_{H_{a},1}^{on}=0~{}s$,
$\tau_{I_{1},1}^{off}=504~{}ns,~{}\tau_{I_{2},1}^{off}=500~{}ns,~{}\tau_{H_{a},1}^{off}=496~{}ns$
and $\tau_{I_{1},2}^{on}=\tau_{I_{2},2}^{on}=\tau_{H_{a},2}^{on}=1500~{}ns$.
It is observed that in the absence of field-like torque, the oscillations of
the two oscillators damp out after 500 ns to the steady states at different
hemispheres, formed by the unit vector ${\bf m}$ around the origin, and
continue in the same steady states even after the currents and field are
applied at 1500 ns as shown in Fig.1(c). On the other hand in the presence of
positive field-like torque, both the oscillators attain the steady state at
the northern hemisphere after the currents and field are switched off around
500 ns and reach synchronized oscillations after the currents and field are
switched on at 1500 ns. From Figs.1(a) & (c) it is understood that the lack of
simultaneity in switching off the currents and field transforms the system
from getting synchronized oscillatory state to steady state. However, the
presence of field-like torque destabilizes the steady state at the southern
hemisphere and makes the magnetization vectors of the two oscillators to stay
in the northern hemisphere and exhibit synchronized oscillations after the
currents and field are switched on. It has also been verified that the
positive field-like torque makes synchronized oscillations even when
$\tau_{I_{1},1}^{on},\tau_{I_{2},1}^{on},\tau_{H_{a},1}^{on}$ and
$\tau_{I_{1},2}^{on},\tau_{I_{2},2}^{on},\tau_{H_{a},2}^{on}$ differ by
nanoseconds. Also, we checked that a negative field-like torque does not
produce synchronized oscillations when the currents and field are switched on
again after switching off at different times.

From the above studies we understand that there is a definite possibility for
the oscillators to reach steady states in different hemispheres, and therefore
it is essential to verify the existence of multistability and the possibility
of its removal by suitable means. Hence, Eq.(1) is numerically solved for 100
numbers of randomly chosen initial conditions, chosen from both the
hemispheres, and the corresponding probabilities for steady state (SS) and
synchronized oscillations(SYN) are computed. The values of SS and SYN are
plotted against current in Figs. 2(a) and 2(b) for $\beta=0$ and $\beta=0.61$
respectively, when $\chi=0.5$. Fig.2(a) shows that, in the absence of field-
like torque, there is a nonzero probability for both steady state and
synchronized oscillations beyond a critical current strength, whereas in the
presence of positive field-like torque the system exhibits synchronized
oscillations only, as shown in Fig.2(b).

Figure 2: (Color online) Probabilities of synchronized oscillations (SYN) and
steady state (SS) in the (a) absence ($\beta=0$) and (b) presence
($\beta=0.61$) of field-like torque when $\chi=0.5$.

###  III.2 Frequency and power of synchronized oscillations

Further, the frequency of synchronized oscillations is derived by transforming
Eq.(1) into spherical polar coordinates using the transformations ${\bf
m}_{j}=(\sin\theta_{j}\cos\phi_{j},\sin\theta_{j}\sin\phi_{j},\cos\theta_{j})$
as follows:

$\displaystyle(1+\alpha^{2})\frac{d\theta_{j}}{dt}=-2\pi\alpha
F\sin\theta_{j}$ $\displaystyle-\gamma
H_{Sj}\left[(\alpha-\beta)\sin\phi_{j}+(1+\alpha\beta)\cos\theta_{j}\cos\phi_{j}\right],$
(4) $\displaystyle(1+\alpha^{2})\sin\theta_{j}\frac{d\phi_{j}}{dt}=2\pi
F\sin\theta_{j}$ $\displaystyle+\gamma
H_{Sj}\left[(1+\alpha\beta)\sin\phi_{j}-(\alpha-\beta)\cos\theta_{j}\cos\phi_{j}\right],$
(5)

where $F=({\gamma}/{2\pi})[H_{a}+(H_{k}-4\pi M_{s})\cos\theta_{j}]$. From
Eqs.(4) and (5), it can be numerically verified that in the synchronized state
the values of $\theta_{1}$ and $\theta_{2}$ are the same and can be
approximated to a constant value tomo:18 ; tani:14 . Also, $\phi_{1}=2\pi ft$
and $\phi_{1}-\phi_{2}=2n\pi$, $n$=0,$\pm$1,$\pm$2…. tomo:18 . Here, $f$ is
the frequency of the synchronized oscillations tomo:18  derived from Eq.(5) as

$\displaystyle f(\theta)=\left(\frac{1}{1+\alpha^{2}}\right)$
$\displaystyle\left[F+\frac{(\beta-\alpha)\gamma\hbar\eta
I_{0}\cos\theta}{4\pi
eM_{s}V\lambda\sin^{2}\theta}\left(1-\frac{1}{\sqrt{1-\lambda^{2}\sin^{2}\theta}}\right)\right].$
(6)

The frequency and power of the synchronized oscillations against current in
the absence ($\beta=0$) and presence ($\beta=0.61$ and $\beta=-0.61$) of
field-like torque have been plotted in Figs. 3(a) and 3(b) respectively for
$\chi$ = 0.5. The solid line in Fig.3(a) corresponds to numerically computed
frequency and the open circles correspond to analytically computed frequency
from Eq.(6). From Fig.3(a) it is observed that the frequency of the
synchronized oscillations is enhanced by positive field-like torque and
decreased by negative field-like torque.

Figure 3: (Color online) (a) The frequency of synchronized oscillations in the
absence and presence of field-like torque when $\chi$ = 0.5. The solid line
and open circle correspond to the frequency computed by numerical and
analytical (Eq.(6)) calculations, respectively. (b) The power spectrum of the
oscillations in the absence ($\beta=0$) and presence ($\beta$=0.61 and -0.61)
of field-like torque when $I_{0}$ = 2.0 mA and $\chi$ = 0.5.

In order to elucidate the experimental consequences of enhancement of the
frequency of synchronized oscillations due to field-like torque, we have
plotted the spectral power in the frequency domain in Fig.3(b) for $\beta=0$,
$\beta=0.61$ and $\beta=-0.61$ when $I_{0}$ = 2.0 mA and $\chi$ = 0.5. It is
evident that there is no change in the power due to the field-like torque and
the frequency is indeed enhanced without any loss in the power. Though the
enhancement of the frequency is in the MHz range, there is no loss in the
power due to field-like torque.

###  III.3 Removal of multistability by field-like torque

To analyze the impact of field like torque on coupling strength, we plot the
probabilities of SYN and SS for 100 randomly chosen initial conditions for
$I_{0}$ = 8mA. Figure 4(a) shows that in the absence of field-like torque the
probability of synchronized oscillations(steady state) reduces(increases) from
1(0) when the coupling strength is increased. This evidences that the system
does not exhibit synchronized oscillations for all initial conditions beyond
some critical value of coupling strength in the absence of field-like torque.
From Fig.4(b) it is observed that the oscillators do not get synchronized for
all initial conditions in the absence of field-like torque. However, beyond
certain critical value of positive field-like torque both the oscillators
oscillate synchronously for all initial conditions, which is confirmed from
Fig.4(b) where the probability for synchronized state reaches 1 when the
strength of field-like torque is increased beyond the critical value
($\beta_{c}=0.33$).

Figure 4: (a) The probabilities of synchronized oscillations(red) and steady
state (black) against coupling strength in the absence of field-like torque at
$I_{0}$ = 8 mA. (b) The probabilities of synchronized oscillations(red) and
steady state (black) against field-like torque at $\chi$ = 0.6, $I_{0}$ = 8
mA. The vertical lines correspond to the critical values $\chi_{c}$ = 0.29 and
$\beta_{c}$ = 0.33 obtained from Eq.(8) and Eq.(9) respectively.

###  III.4 Steady states and critical values of and $\beta$ and $\chi$ for
synchronized oscillations

The steady state solution of the system (1) is found around
$\phi_{1}^{*}=\phi_{2}^{*}\approx 3\pi/2$, and

$\displaystyle\theta_{1}^{*}\approx\sin^{-1}\left(\frac{H_{S0}}{H_{a}+G}\right),~{}\theta_{2}^{*}\approx\pi-\sin^{-1}\left(\frac{H_{S0}}{H_{a}-G}\right),$

where $H_{S0}=\hbar\eta I_{0}/2eM_{s}V$ and $G=H_{k}-4\pi M_{s}$. From the
linear stability analysis, in the absence of field-like torque the steady
state is found to be stable when laksh

$\sum_{i=1}^{2}\left({\frac{\partial
f_{i}}{\partial\theta_{i}}+{\frac{\partial
g_{i}}{\partial\phi_{i}}}}\right)_{\theta_{1}^{*},\theta_{2}^{*},\phi_{1}^{*},\phi_{2}^{*}}<0.$
(7)

Here, $f_{i}$ and $g_{i}$ are derived from Eqs.(4) and (5) as
$\dot{\theta}_{i}=f_{i}(\theta_{1},\theta_{2},\phi_{1},\phi_{2}),~{}\dot{\phi}_{i}=g_{i}(\theta_{1},\theta_{2},\phi_{1},\phi_{2}),~{}i=1,2.$
From the condition (7), the critical value of coupling strength $\chi_{c}$
above which the system exhibits stable steady state solution in the absence of
field-like torque ($\beta=0$), is derived as

$\displaystyle\chi_{c}=\lambda+\frac{\alpha}{2H_{S0}}\left[2GU-2\alpha
H_{a}\tau_{-}-G\tau_{+}\right],$ (8)

where $\tau_{\pm}=(\sqrt{1-T_{+}}\pm\sqrt{1-T_{-}})$,
$T_{\pm}={H_{S0}^{2}}/{(H_{a}\pm G)^{2}}$ and $U=(T_{+}+T_{-}-1)$.

However, in the presence of field-like torque, the critical value of
$\beta_{c}$ above which the steady state loses the stability, so that the
synchronized state is the only stable state, can be found to be

$\displaystyle\beta_{c}=\frac{\alpha
G[\tau_{+}-2U]-2H_{S0}(\lambda-\chi)+2H_{a}\alpha\tau_{-}}{2H_{S0}\alpha(\lambda-\chi)+H_{a}\tau_{-}+G\tau_{+}}.$
(9)

The values of $\chi_{c}$ and $\beta_{c}$ match well with the numerical values,
as confirmed by the vertical lines in Figs. 4(a,b).

###  III.5 Stability of synchronized oscillations in the presence of field-
like torque

In the absence of field-like torque the stability of the synchronized
oscillations has already been confirmed by Taniguchi $et~{}al$ tomo:18 .
However, in the presence of positive field-like torque the stability of the
synchronized oscillations is confirmed by perturbing $\phi_{1}$ as
$\phi_{1}=\phi_{2}+\delta\phi$ after synchronization is reached, and the time
evolution of $\delta\phi$ is analysed over $n$ periods of oscillations. By
substituting $\phi_{1}=2\pi ft+\delta\phi(t),~{}\phi_{2}=2\pi ft$ and
$\theta_{1}=\theta_{2}=\theta$ in Eq.(5) and after averaging over $n$
oscillations we can obtain tomo:18

$\displaystyle\frac{1}{nT}\int_{0}^{nT}\frac{d\delta\phi}{dt}\approx-\frac{\chi\gamma
H_{S0}(1+\alpha\beta)}{(1+\alpha^{2})nT}\int_{0}^{nT}\delta\phi.$ (10)

The solution of Eq.(10) is
$\delta\phi(t)\approx\delta\phi(0)~{}exp({-[\chi\gamma
H_{S0}(1+\alpha\beta)/(1+\alpha^{2})]t})$, indicating the small
deviation$(\delta\phi)$ between $\phi_{1}$ and $\phi_{2}$ exponentially
decreases to zero as time increases. This implies that the presence of field-
like torque does not affect the stability of the synchronized oscillatory
state of the two parallelly coupled spin torque oscillators.

##  IV Conclusion

In conclusion, the synchronization between two parallelly coupled spin torque
oscillators in the presence of field-like torque has been investigated
theoretically, with a physical configuration of perpendicularly magnetized
free-layer and in-plane magnetized pinned layer. The numerical simulation of
the LLGS equation has revealed that the existence of field-like torque can
cancel out the damping effect and thus can induce synchronized oscillations
with respect to applied current. One can also note that the multistable
behavior in coupled STOs can be efficiently removed by introducing the field-
like torque and the frequency of the synchronized oscillations gets enhanced
by positive field-like torque.

## Acknowledgements

The work of V.K.C. forms part of a research project sponsored by CSIR Project
No. 03/ 1444/18/EMR II. M.L. wishes to thank the Department of Science and
Technology for the award of a SERB Distinguished Fellowship under Grant
No.SB/DF/04/2017 in which R. Arun is supported by a Research Associateship.

## References

  * (1) J. Grollier, V. Cros, and A. Fert, Phys. Rev. B 73, 060409(R) (2006).
  * (2) S. Urazhdin, P. Tabor, V. Tiberkevich and A. Slavin, Phys. Rev. Lett. 105, 104101 (2010).
  * (3) F.B. Mancoff, N. D. Rizzo, B. N. Engel and S. Techrani, Nature 437, 393 (2005).
  * (4) S. Kaka, M. R. Pufall, W. H. Rippard, T. J. Silva, S. E. Russek, and J. A. Katine, Nat. Lett. 437, 389 (2005).
  * (5) J. Persson, Y. Zhou, and J. Akerman, J. Appl. Phys. 101, 09A503 (2007).
  * (6) J. C. Slonczewski, J. Magn. Magn. Mater. 159, L261-68 (1996).
  * (7) R. Adler, Proc. IEEE 61, 1380 (1973).
  * (8) Z. Li, Y. C. Li, and S. Zhang, Phys. Rev. B 74, 054417 (2006).
  * (9) V. Tiberkevich, A. Slavin, E. Bankowski, and G. Gerhart, Appl. Phys. Lett. 95, 262505 (2009).
  * (10) T. Kendzioczyk, S. O. Demokritov, and T. Kunn, Phys. Rev. B 90, 054414 (2014).
  * (11) B. Subash, V. K. Chandrasekar, and M. Lakshmanan, Euro. Phys. Lett. 102, 17010 (2013).
  * (12) B. Subash, V. K. Chandrasekar, and M. Lakshmanan, Euro. Phys. Lett. 109, 17009 (2015).
  * (13) R. Gopal, B. Subash, V. K. Chandrasekar, and M. Lakshmanan, IEEE Trans. Magn. (2019) (Accepted, DOI: 10.1109/TMAG.2019.2908954).
  * (14) W. H. Rippard, M. R. Pufall, S. Kaka, T. J. Silva, S. E. Russek, and J. A. Katine, Phys. Rev. Lett. 95, 067203 (2005).
  * (15) M. Zaks and A. Pikovsky, Scientific Reports 7, 4648 (2017).
  * (16) B. Georges, J. Grollier, M. Darques, V. Cros, C. Deranlot, B. Marcilhac, G. Faini, and A. Fert, Phys. Rev. Lett. 101, 017201 (2008).
  * (17) Y. Zhou and J. Akerman, Appl. Phys. Lett. 94 , 112503 (2009).
  * (18) K. Nakada, S. Yakata, and T. Kimura, J. Appl. Phys.111, 07C920 (2012).
  * (19) Z. M. Zeng, P. K. Amiri, I.N. Krivorotov, H. Zhao, G. Finocchio, J. P. Wang, J. A. Katine, Y. Huai, J. Langer, K. Galatsis, K. Wang and H. Jiang, ACS Nano 6, 6115 (2012).
  * (20) H. S. Chio, S. Y. Kang, S. J. Cho, I.Y Oh, M. Shin, H. Park, C. Jang, B.C. Min, S.Y. Park and C. S. Park, Sci. Rep.4, 5486(2014).
  * (21) N. Locatelli, V. Cros, and J. Grollier, Nat. Mater. 13, 11 (2014).
  * (22) J. Grollier, D. Querlioz, and M. D. Stiles, Proc. IEEE 104, 2024 (2016).
  * (23) K. Kudo and T. Morie, Appl. Phys. Express 10, 043001 (2017).
  * (24) J. Torrejon, M. Riou, F. A. Araujo, S. Tsunegi, G. Khalsa, D. Querlioz, P. Bortolotti, V. Cros, K. Yakushiji, A. Fukushima, H. Kubota, S. Yuasa, Mark D. Stiles and J. Grollier, Nature 547, 428 (2017).
  * (25) S. Mangin, D. Ravelosona, J. A. Katine, M. J. Carey, B. D. Terris, and E. E. Fullerton, Nat. Mater. 5, 210 (2006).
  * (26) D. Houssameddine, U. Ebels, B. Dela$\ddot{\textrm{e}}$t, B. Rodmacq, I. Firastrau, F. Ponthenier, M. Brunet, C. Thirion, J.-P. Michel, L. Prejbeanu-Buda, M.-C. Cyrille, O. Redon, and B. Dieny, Nat. Mater. 6, 447 (2007).
  * (27) W. H. Rippard, A. M. Deac, M. R. Pufall, J. M. Shaw, M. W. Keller, S. E. Russek, G. E. W. Bauer, and C. Serpico, Phys. Rev. B 81, 014426 (2010).
  * (28) H. Kubota, K. Yakushiji, A. Fukushima, S. Tamaru, M. Konoto, T. Nozaki, S. Ishibashi, T. Saruya, S. Yuasa, T. Taniguchi, H. Arai and H. Imamura, Appl. Phys. Express 6, 103003 (2013).
  * (29) T. Taniguchi, H. Arai, S. Tsunegi, S. Tamaru, H. Kubota, and H. Imamura, Appl. Phys. Express 6, 123003 (2013).
  * (30) J. Turtle, P.-L. Buono, A. Palacios, C. Dabrowski, V. In, P. Longhini, Phys. Rev. B 95 144412 (2017).
  * (31) T. Taniguchi, S. Tsunegi, and H. Kubota, Appl. Phys. Express 11,013005 (2018).
  * (32) T. Taniguchi, H. Arai, H. Kubota, and H. Imamura, IEEE Trans. Magn. 50, 1400404 (2014).
  * (33) T. Taniguchi, S. Tsunegi, H. Kubota and H. Imamura, App. Phys. Lett. 104, 152411 (2014).
  * (34) T. Taniguchi, S. Tsunegi, H. Kubota, and H. Imamura, J. Appl. Phys. 117, 17C504 (2015).
  * (35) Y.Y. Guo, H. B, Xue, and Z. J. Liu, AIP Advances 5, 057114 (2015).
  * (36) M. Lakshmanan and S. Rajasekar, Nonlinear Dynamics: Integrability, Chaos and Patterns, Springer-Verlag Berlin Heidelberg (2003).