# Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment
Proposal

Rahul Pankajakshan1, Sumitra Biswal* 2, Yuvaraj Govindarajulu*2, Gilad
Gressel1
1Center for Cybersecurity Systems and Networks, Amrita Vishwa Vidyapeetham,
Amritapuri, India
{rahulp<EMAIL_ADDRESS>2AIShield, Bosch Global Software
Technologies, Bengaluru, India
{sumitra.biswal<EMAIL_ADDRESS>
###### Abstract

The rapid integration of Large Language Models (LLMs) across diverse sectors
has marked a transformative era, showcasing remarkable capabilities in text
generation and problem-solving tasks. However, this technological advancement
is accompanied by significant risks and vulnerabilities. Despite ongoing
security enhancements, attackers persistently exploit these weaknesses,
casting doubts on the overall trustworthiness of LLMs. Compounding the issue,
organisations are deploying LLM-integrated systems without understanding the
severity of potential consequences. Existing studies by OWASP and MITRE offer
a general overview of threats and vulnerabilities but lack a method for
directly and succinctly analysing the risks for security practitioners,
developers, and key decision-makers who are working with this novel
technology. To address this gap, we propose a risk assessment process using
tools like the OWASP risk rating methodology which is used for traditional
systems. We conduct scenario analysis to identify potential threat agents and
map the dependent system components against vulnerability factors. Through
this analysis, we assess the likelihood of a cyberattack. Subsequently, we
conduct a thorough impact analysis to derive a comprehensive threat matrix. We
also map threats against three key stakeholder groups: developers engaged in
model fine-tuning, application developers utilizing third-party APIs, and end
users. The proposed threat matrix provides a holistic evaluation of LLM-
related risks, enabling stakeholders to make informed decisions for effective
mitigation strategies. Our outlined process serves as an actionable and
comprehensive tool for security practitioners, offering insights for resource
management and enhancing the overall system security.

###### Index Terms:

LLM, Security, AI, GenAI, Threat modelling, risk assessment

**footnotetext: These authors contributed equally to this work.

##  I Introduction

Large Language Models (LLMs) have become increasingly prevalent in various
sectors, including industry and academia. They are undergoing transformative
integration at an impressive pace, showcasing remarkable human-like
capabilities in text generation, language understanding, sentiment analysis,
and summarization. They also demonstrate appreciable abilities in code
generation, problem-solving, and reasoning tasks. The accessibility and
affordability of LLM inference APIs and the availability of open-source models
have facilitated their widespread adoption, resulting in a continuous
emergence of a plethora of applications.

However, LLMs embody a dual nature, wielding the potential to be both a
powerful tool for progress and innovation and a concerning source of harm and
misuse. They introduce a new set of risks, which are evolving and not
understood well enough to be effectively managed. These include attacks like
prompt injection to hijack the model and perform unauthorized actions, as well
as planting backdoors in training data for potentially malicious activities
once deployed. Despite significant efforts to align the models and implement
defensive mechanisms to make LLMs more helpful and less harmful [11, 9],
attackers have found ways to circumvent these guardrails [16, 26].
Nevertheless, organisations and users are adopting this technology without
understanding its security and privacy implications. For instance, an employee
at Samsung accidentally leaked proprietary code via ChatGPT [21], while
Amazon’s recently implemented LLM-based chatbot, Q, inadvertently disclosed
confidential information and generated severely hallucinatory responses [22].
This low maturity level and bug-ridden experience are cause for concern and
significantly negatively impact their trustworthiness and usability in the
future.

The Open Web Application Security Project (OWASP), with the help of industry
and academic experts, has compiled a detailed list of the top 10
vulnerabilities and threats to LLM applications [20]. MITRE has introduced the
ATLAS threat matrix, which outlines the adversarial tactics and techniques
used to attack AI systems, based on the popular ATT&CK framework[19]. These
studies have provided a comprehensive understanding of the various attack
methods in the LLM ecosystem. However, they lack focus on identifying the
severity of risks faced by different stakeholders within the LLM ecosystem.
Unlike traditional IT systems, there is an absence of well-established risk
assessment or threat modeling processes for LLM-based systems. This absence
poses challenges for security professionals in evaluating cyber risks and
implementing appropriate mitigation strategies to secure their systems
effectively.

In this study, we demonstrate the applicability of tools like the OWASP Risk
Rating Methodology, commonly utilized for evaluating risks in conventional IT
systems, to assess the severity of risks specific to LLM-based systems. Our
approach involves a three-point method: first, we conduct a scenario analysis,
considering factors such as the motive and skill level of potential threat
agents. Next, we map dependent system components against vulnerabilities,
taking into consideration factors such as the ease of system exploitation and
vulnerability discovery. By integrating scenario analysis with dependency
mapping, we estimate the likelihood of a threat. The final step in our
approach involves performing an impact analysis, aimed at comprehending both
the technical and business implications. Following our detailed risk analysis,
we distilled our findings into a threat matrix, providing stakeholders with a
quick-reference tool that can be tailored to their needs. We’ll demonstrate
this risk assessment process on a hypothetical use case to show the usability
and relevance of the process.

We argue that this risk assessment process, along with the threat matrix,
would provide readers involved in the risk assessment of their LLM-based
systems with a valuable and actionable tool for resource management and
understanding the overall security posture of their system. Moreover, such an
approach empowers security professionals and developers to make informed
decisions regarding risk mitigation by gaining insights into the specific
threats that may impact them and optimise resource allocation and management.

##  II Background

###  II-A Rise of LLMs

LLMs represent a significant advancement in natural language processing. These
deep learning algorithms, trained on massive datasets of text and code, boast
capabilities far exceeding their predecessors in understanding and generating
human-like text. The transformer architecture, introduced by Vaswani et al. in
2017, serves as the foundational framework for large language models,
facilitating their ability to capture intricate dependencies in sequential
data through self-attention mechanisms[5].

The training process of LLMs mainly involves two key phases: pretraining and
fine-tuning. In the pretraining phase, the model undergoes self-supervised
learning on a diverse dataset, predicting the next word in a sentence based on
context. This self-supervised approach and techniques like Masked Language
Modelling (MLM) and Casual Language Model (CLM) enable the model to grasp
complex language patterns implicitly. This results in a pre-trained LLM with a
generalised understanding of language patterns. Subsequently, in the fine-
tuning phase, the model is adapted to specific tasks or domains using smaller,
task-specific datasets. Fine-tuning employs strategies like Reinforcement
Learning from Human Feedback (RLHF) [4], where the model refines its
capabilities through iterative adjustments based on human-provided feedback.
This process enhances the LLM’s proficiency in performing targeted
applications such as text summarization, translation, or sentiment analysis,
ensuring its adaptability and effectiveness in real-world scenarios.

###  II-B Risk Assessment

Risk assessment is a crucial component in information security management,
providing organisations with a systematic approach to identifying, analysing,
and mitigating potential risks. Various frameworks have been established over
the years to guide the risk assessment and management process, each with its
unique perspective and methodologies. Some prominent frameworks include the
ENISA Risk Management Framework [31], the NIST Risk Management Framework [3],
and the ISO 27001 [32].

However, they all follow a common approach involving the following key steps:

  1. 1.

Risk Identification: Initiate the process by identifying potential threats,
vulnerabilities, and assets susceptible to security breaches.

  2. 2.

Risk Analysis: Analyze the identified threats using a risk rating methodology.
Estimate the likelihood and impact of risks to assess their significance.

  3. 3.

Risk Evaluation: Prioritize risks based on their potential impact, aligning
with the organisation’s objectives.

  4. 4.

Risk Treatment: Take action to manage the identified risks. Develop and
implement strategies to mitigate, transfer, or accept risks, considering risk
criticality and aligning with the organization’s security policies and
tolerance.

  5. 5.

Monitoring and Review: Continuously assess and update the risk management
process to address evolving threats and vulnerabilities.

In addition to the frameworks, risk assessments can be categorised into
different types based on their methodologies:

  * •

Qualitative Risk Assessment: Focuses on subjective judgment and expert
opinions to assess risks based on qualitative criteria such as high, medium,
or low.

  * •

Quantitative Risk Assessment: Utilises numerical data and statistical methods
to quantify risks’ potential impact and likelihood, providing a more precise
analysis.

  * •

Semi-Quantitative Risk Assessment: Combines elements of both qualitative and
quantitative assessments to provide a balanced approach, incorporating expert
judgment and numerical data.

###  II-C OWASP Top 10 for LLM

The OWASP Top 10 for LLM Applications is a living document that serves as a
comprehensive guide for developers and security teams navigating the unique
security challenges posed by large language models, such as prompt injection,
data poisoning, and model theft. It identifies the top ten critical
vulnerabilities, details example attack scenarios, and recommends specific
mitigation strategies to help developers build secure and reliable LLM
applications.

###  II-D Risk Rating

Risk rating entails estimating the likelihood of an attack occurring and,
should it happen, evaluating its potential impact. Several well-established
and widely recognized methodologies exist to assist in calculating both the
likelihood and impact, such as NIST SP 800-30[2] and the Harmonized Threat and
Risk Assessment (TRA) Methodology from the Canadian Centre for Cyber Security
[1]. Alternatively, the OWASP Risk Rating Methodology[33] offers a more
straightforward and simplified approach.

The OWASP Risk Rating Methodology is a structured approach that assists
organizations in evaluating and prioritizing potential risks associated with
software and web application security. It adheres to a standard model for risk
calculation, expressed as:

$\text{Risk}=\text{Likelihood}\times\text{Impact}$ (1)

####  II-D1 Likelihood factors

The factors applicable to estimating the likelihood of risk are categorized
into two distinct groups:

  * •

Threat Agent Factors: These factors encompass (i) the skill level of the
threat actor, (ii) their motive to exploit, (iii) the opportunities or
resources required for the threat actor to discover and exploit the
vulnerability, and (iv) the size of the threat actor group.

  * •

Vulnerability Factors: This group includes factors such as the (i) ease of
discovering the vulnerability by threat agents, (ii) the ease of exploiting
the vulnerability once discovered, (iii) awareness of the vulnerability among
threat agents, and the likelihood of a successful (iv) intrusion detection if
the exploit occurs.

####  II-D2 Impact factors

The factors used to determine impact are also categorized into two groups:

  * •

Technical Impact Factors: These factors concentrate on the traditional
cybersecurity areas of concern, namely, (i) loss of confidentiality, (ii) loss
of integrity, (iii) loss of availability, and (iv) loss of accountability.

  * •

Business Impact Factors: These factors are crucial to the organisation and
encompass considerations such as the (i) financial damage resulting from the
exploit, the (ii) impact on the business’s reputation, (iii) the exposure
introduced by non-compliance, and the (iv) severity of privacy violations.

The factors are scored on a scale from 0 to 9. The likelihood score is
determined by averaging the scores of the threat agent factor and the
vulnerability factors, while the impact score is derived by averaging the
scores of the technical and business impact factors. The values of these
factors can also be weighted according to specific requirements. The
likelihood and impact levels chart, as well as the overall risk severity
chart, can be configured according to the specific needs of the organization,
or standard tables proposed by OWASP can be adopted.

##  III LLM Risks

This section is dedicated to enumerating all the risks and threats associated
with LLMs, adhering to the taxonomy outlined by the OWASP in their Top Ten for
LLMs version 1.1.0 [20]. This approach is employed to ensure comparability and
mitigate redundancy in the identification and categorisation of potential
security concerns related to LLMs.

###  III-A LLM01: Prompt Injection

Prompt Injection is an LLM vulnerability that enables an attacker to
manipulate the LLM’s output by carefully crafted prompts, leading to the
generation of texts that usually violate the LLM’s developer-set usage
policies. There exist two primary categories of prompt injections:

  * •

Direct Prompt Injections: Colloquially known as “jailbreaking”, these entail
the manipulation of the system prompt through overwriting or revealing, often
leading to partial IP loss. This may involve crafting prompts with the
specific intent of circumventing safety and moderation features imposed on
LLMs by their creators.

  * •

Indirect Prompt Injections: Occurs when an LLM accepts input from external
sources susceptible to control by an attacker, such as websites or files. In
this context, attackers can deceive the LLM into interpreting input from LLM
as “commands” rather than “data” for processing, consequently inducing
unexpected behaviour in LLM-based applications or compromising the security of
the entire system [15].

Automated tools for jailbreaking LLMs have been developed[13], as well as
multi-prompt injection techniques[18] are discussed in the existing
literature. Moreover, universal and transferable adversarial suffixes have
emerged as effective methods for jailbreaking various models [12].

###  III-B LLM02: Insecure Output Handling

General-purpose LLMs undergo training on a substantial portion of the
internet. When employed for downstream tasks in any application or plug-ins,
developers must exercise caution in their utilisation, as these models can
generate outputs that may be harmful to the user or the application itself.
Insecure Output Handling specifically pertains to the absence of sufficient
validation or sanitisation of LLM outputs before they are used for downstream
tasks. If the outputs of LLMs are not managed properly, it could lead to
security risks like Cross-Site Scripting and Cross-Site Request Forgery in web
browsers. Attackers can also exploit the LLM outputs for privilege escalation,
and remote code execution on backend systems [23].

###  III-C LLM03: Training Data Poisoning

Training data poisoning involves deliberately manipulating the data used to
train these models with malicious intent. Adversaries strategically inject
deceptive or biased examples into the training dataset at the pre-training or
fine-tuning stage, aiming to influence the model’s learning process. Attackers
can introduce backdoors, biases or other vulnerabilities that can degrade the
model’s security, performance, and trustworthiness [27].

###  III-D LLM04: Model Denial of Service

LLMs are very resource-intensive to train and run. An attacker can interact
with LLMs leading it to consume resources excessively resulting in a decline
in the quality of service or even denial of service to other users as well as
higher compute costs. Attackers can craft prompts that are computationally
complex in terms of context length or language patterns.

###  III-E LLM05: Supply Chain Vulnerabilities

In the context of LLMs, the supply chain refers to the entire process from
data collection and model training to deployment. It involves various
components such as the training data, pre-trained models, and the deployment
infrastructure. Each component can be vulnerable, the crowd-sourced training
data could be poisoned, the pre-trained model could be compromised or the
third-party packages used to develop the LLM could be insecure.

###  III-F LLM06: Sensitive Information Disclosure

LLMs are pre-trained on diverse datasets that include snippets of real-world
data. During the generation process, these models can inadvertently produce
responses that disclose sensitive details. Conversational agents like OpenAI’s
ChatGPT and Google’s Gemini collect user prompts during conversations to
enhance their model’s performance. However, this practice introduces a
security and privacy concern, as the model may unintentionally generate
outputs that reveal confidential or private information. Moreover, using
carefully crafted prompts, an attacker could exploit this vulnerability to
reveal or expose sensitive details intentionally.

###  III-G LLM07: Insecure Plugin Design

Often LLM plugins accept user input as free text, which can be easily
exploited by an attacker. The LLM plugins that are designed without proper
access control or input validation can result in SQL injection or remote code
execution.

###  III-H LLM08: Excessive Agency

LLM-based systems make decisions based on a user’s prompt or the input they
receive from another integrated component. If the degree of freedom or
authorisation granted to the LLM is excessive, attackers can exploit this
vulnerability to compromise the LLM-based system. However, an attacker need
not exploit this vulnerability to be harmful. Any unsuspecting user input or
unintended action from a system component can lead the model to produce
ambiguous or unexpected output, causing the system to behave unexpectedly. For
instance, an LLM-based file summarizer utilizes a third-party plugin for
reading files from the user. However, this plugin also possesses the
capability to modify and delete files. If a user detects an error in the LLM’s
generated response, they may report the mistake to the application, directing
the LLM to potentially modify or delete the files [20].

###  III-I LLM09: Overreliance

LLMs can “hallucinate”, generating information that can be factually
incorrect, unsafe, or inappropriate [14, 28]. When these models are relied
upon to generate source code, there is a risk of introducing unnoticed
security vulnerabilities, which pose a significant threat to the safety and
security of applications as well its users. Relying on such information or
code without adequate oversight can result in security breaches, spread of
misinformation, communication breakdowns, legal complications, and damage to
one’s reputation.

###  III-J LLM10: Model Theft

Model theft is the illegal act of copying or extracting weights or parameters
or data from closed-source LLM models to create functional equivalents[24].
This activity can lead to substantial economic losses and harm to brand
reputation, posing a threat to competitive advantage. Attackers may exploit
the proprietary information within the model or use the model itself for
malicious purposes.

##  IV Methodology

In this section, we will outline the methodology employed to derive the threat
matrix. We will begin by discussing the scope of the identified stakeholder
groups. Following that, we will delve into the risk analysis process that we
propose, which is rooted in the OWASP risk rating methodology. This risk
assessment process underpins our threat matrix, ensuring each risk is
systematically analyzed and rated.

###  IV-A Stakeholders to LLM

The stakeholders involved in large language models range from prominent
corporations funding and advancing these models to individuals within the
broader populace who might be unaware of their existence. This paper focuses
on stakeholders directly affected by potential security lapses in large
language models. We specifically highlight three key groups of stakeholders:
developers and organisations engaged in fine-tuning open-source pre-trained
models such as Llama-2 family [25] or Mistral family[17] for various
downstream tasks, application developers who depend on third-party LLM
inference APIs such as GPT-3.5, GPT-4 provided by OpenAI, and end users who
utilise these systems.

####  IV-A1 LLM Fine-tuning Developers

The significant financial investment required for developing foundational
models from scratch includes expenditures on cutting-edge computing
infrastructure for model training, such as thousands of powerful GPUs,
extensive datasets, and the recruitment of specialized research and
development teams. Unfortunately, this level of investment is feasible for
only a limited number of organisations. This significant financial barrier
presents a formidable challenge for smaller organizations, limiting their
capacity to compete on an equal footing.

Despite these challenges, open-source large language model families,
comprising both pre-trained and fine-tuned models, are increasingly becoming
accessible for public and commercial use [25, 17]. This accessibility creates
an opportunity for less resource-constrained entities in LLM research and
development to fine-tune their versions for various tasks. Techniques like
Low-Rank Adaptation (LoRA) [8] have also helped preserve performance while
reducing memory requirements and accelerating training. Fine-tuning typically
involves data collection, establishing training infrastructure, and leveraging
expertise in machine learning. Through this process, the model’s knowledge
becomes refined and tailored to the specific domain of the application,
resulting in more relevant and nuanced outputs.

####  IV-A2 LLM API Integration Developers

The emergence of ChatGPT and its rapid rise in popularity has spurred the
development of applications and plugins based on LLMs. These applications
primarily utilize public LLM inference APIs. These APIs allow developers to
send input data or prompts to the LLM and receive its output without the need
to manage the model or its training process directly. While this approach
offers convenience, developers rely on the API provider for maintaining and
enhancing the underlying LLM, limiting their ability to customize its
behaviour.

These applications span various domains, such as content creation assistants,
coding or problem-solving tools, and even support for HR recruitment
processes. OpenAI has further facilitated this trend through the introduction
of the ‘GPT store’, creating a platform for the launch of millions of custom
versions of ChatGPT or simply ‘GPTs’.

However, in the rush to adopt cutting-edge technology, there is a significant
risk that security considerations are being overlooked, either due to cost
constraints or a lack of awareness, given the newness of this technology. For
instance, MathGPT is an LLM-based application intended to aid users in solving
mathematical problems. A user was able to inadvertently expose its OpenAI
GPT-3.5 API key through code execution, exploiting a vulnerability related to
prompt injection [23].

####  IV-A3 End Users

End users constitute a pivotal stakeholder group in the domain of LLM. They
are the ultimate consumers of applications, services, or platforms powered by
LLMs, interacting with technology that influences various aspects of their
lives, including education, communication, business, and entertainment. Users
depend on LLM-based applications to receive assistance in writing, access
health-related information, and make informed decisions in finance. These
models have become integral to contemporary life, permeating multiple facets
and delivering convenience and efficiency across diverse walks of life. It is
crucial to prioritise the safety, privacy, and security of these end users to
maintain their trust and uphold a positive outlook towards large language
models.

###  IV-B Risk analysis process

The risk analysis process starts with the initial step of risk identification,
wherein all potential threats are enumerated based on the taxonomy proposed by
OWASP in their Top Ten for LLM.

In assessing each identified risk, we utilize both the risk factors and
recommendations outlined in the OWASP Risk Rating Methodology to determine its
criticality. This process can be characterized as a semi-quantitative risk
assessment. The overall risk assessment process we advocate adopts a three-
step approach to determine the values for likelihood and impact factors:

  1. 1.

Scenario Analysis: In this phase, targeted scenarios are crafted, and the
rationale behind threat agent factors is deliberated upon. The strategy we are
following in this step involves identifying the threat agents who are most
likely to exploit the vulnerability and examining the worst-case scenario.

  2. 2.

Dependency Mapping: All system components associated with the vulnerability
are identified and mapped against the vulnerability factors, such as ease of
discovery, ease of exploitation, awareness of the threat agent regarding the
vulnerability, and the likelihood of detecting the intrusion when the exploit
occurs. The values for vulnerability factors need to be estimated based on the
threat agent factors. Once a clear understanding of threat agents and
vulnerability factors is obtained, the likelihood of the attacker targeting
the vulnerability is estimated using the standard risk calculation equation
(1).

  3. 3.

Impact Analysis: This crucial step involves assessing the technical and
business impact if the exploit occurs. The technical impact is initially
assessed by contemplating worst-case scenarios, providing a foundation for
estimating the subsequent impact on the business operations.

##  V Threat Matrix

In this section, we present a generic threat matrix in Table I. We map the
OWASP Top Ten LLM-specific threats against different stakeholders, providing
practitioners with a reference sheet for conducting risk assessments. This
matrix equips stakeholders with the insights needed for targeted risk
mitigation, summarising the comprehensive analysis in an accessible format.

In this threat matrix, we have outlined the general causes and consequences of
each LLM-relevant risk, along with corresponding suggested controls and
mitigation, after conducting an extensive review of existing literature. We
also indicate whether the risk falls within the realm of traditional
cybersecurity risks. For instance, as mentioned in the matrix, the
vulnerability ‘Prompt Injection’ typically is caused by insufficient control
over input and the LLM’s inherent nature to assist users with their queries.
The potential consequences of exploiting this vulnerability vary from
reputation harm to user harm through indirect prompt injection. Mitigation
techniques that may be utilized include static input validation and dynamic
output filtering. Since these vulnerabilities have emerged with the advent of
LLMs and other foundational models, it is placed outside the scope of
traditional cybersecurity risks. However, this risk affects all identified
stakeholder groups we are targeting.

The probability, impact, and risk rating values within the matrix have
deliberately been left blank to maintain generality and serve their purpose as
a template. We will now demonstrate how to calculate the values for the
probability and impact field using a hypothetical scenario for the ‘LLM fine-
tuning developers’ stakeholder group. This process will aid in deriving the
final risk rating.

S.No | Risk Description | Cause & Consequences | Likelihood | Impact |  | Risk
---
Rating
Controls/Mitigation |  | Traditional
---
Cybersec
Concerned Stakeholders
LLM01 |  | Prompt Injection
---
| Caused by: lack of control/validation on
---
LLM’s input, LLM’s implicit nature or
design/architecture
Consequences: Reputation loss, Partial IP loss,
Performance degradation, User harm
|  |  |  | Static: Use trusted/reputed LLM service
---
provider, Input validation and filtering
Dynamic: Adaptive trust boundaries for
input source, Monitoring of LLM outputs, Red
teaming, LLM Response monitoring/filtering
No |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
End users
LLM02 |  | Insecure
---
Plugin Design
| Caused by: improper access control, poor
---
design and implementation
Consequences: Compromised System
|  |  |  | Static: Input sanitisation, parameterisation,
---
validation, protect against all REST API security
risks
Dynamic: proper authorisation and authentication
Yes |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
LLM03 |  | Training
---
Data Poisoning
| Caused by: Poor vetting/verification of training
---
data and data source
Consequences: Reputation loss, Model integrity
loss, Financial Damage, Misinformation and
bias, Performance degradation, User harm
|  |  |  | Static: Exhaustive analysis and sanitisation of all
---
unvetted training dataset.
No |  | LLM Fine-tuning Developers
---
End users
LLM04 |  | Model Denial
---
of Service
| Caused by: Poor design and implementation,
---
improper input validation
Consequences: Financial and Reputation loss
|  |  |  | Static: Use proper input validation and filtering,
---
rate-limiting, usage limit per user, Adversarial
input detection
Dynamic: resource utilisation monitoring
Yes |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
End users
LLM05 |  | Supply Chain
---
Vulnerabilities
| Caused by: Poor security review and vetting of
---
3rd party components used
Consequences: Variable - Compromised System,
Performance degradation
|  |  |  | Static: Use only trusted/reputed 3rd party
---
softwares and components.
Yes |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
LLM06 |  | Sensitive Information
---
Disclosure
| Caused by: Incomplete training data sanitization,
---
training data memorisation
Consequences: Privacy violation, Reputation
damage, Partial IP loss, User harm
|  |  |  | Static: Training data monitoring to weed out
---
sensitive information, Differential privacy
mechanisms, Encrypt sensitive information
Dynamic:
No |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
End users
LLM07 |  | Insecure Output
---
Handling
| Caused by: General purpose LLM’s ability to
---
generate arbitrary code and text, improper
input validation or output scrutiny
Consequences: IP loss, Compromised system
and data, User harm
|  |  |  | Static: Proper validation/filtering of output,
---
output encoding to mitigate code execution,
rate limiting
No |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
End users
LLM08 | Excessive Agency |  | Caused by: design and implementation choices,
---
improper access control
Consequences: Variable - Compromised System
|  |  |  | Static: Limit the permissions of LLMs ,
---
use components with granular functionalities
than open-ended ones,
Dynamic: implement proper authorisation
No |  | LLM Fine-tuning Developers
---
LLM API Integration Developers
End users
LLM09 | Overreliance |  | Caused by: blindly trusting LLM generated content
---
without review
Consequences: Misinformation, implementation of
incorrect solutions
|  |  |  | Static: User awareness
---
Dynamic: Output validation and review,
No | End Users
LLM10 | Model Theft |  | Caused by: Weak access control, insider threats,
---
model inversion
Consequences: Reputation loss, Model integrity
loss, Financial Damage, Misinformation, privacy
violation
|  |  |  | Static: Model obfuscation
---
Dynamic: Strong access controls and
authentication, Regular auditing
No | LLM Fine-tuning Developers

TABLE I:  The generic threat matrix for LLM enumerates the risks identified in
OWASP’s Top Ten for LLM, detailing causes, consequences, and control measures,
both static and dynamic. Each risk is also categorized as a traditional cyber
risk or not. Likelihood, impact, and risk ratings should be calculated using
the proposed assessment process. Furthermore, the matrix maps these risks
against the respective stakeholder groups, facilitating easy reference and
analysis.

##  VI Use case Analysis

We will conduct a risk assessment for a university virtual assistant. This
virtual assistant is created by fine-tuning an open-source pre-trained LLM
using a dataset containing course and administrative information from the
university.

###  VI-A University Virtual Assistant

The University Virtual Assistant is a chatbot developed by a prominent
university to help students and faculty with details regarding the course
materials, university policies, and other administrative information. This
assistant is developed by fine-tuning an open-source pre-trained LLM with
databases containing lecture materials from courses with a high number of
attendees, course details, and administrative instructions from the
university. The university has developed the bot to enhance administrative
efficiency by automating routine tasks and freeing up staff resources. Also,
the university aims to improve the student experience by offering personalized
assistance with coursework and ensuring availability 24/7.

###  VI-B System Design and Analysis

####  VI-B1 System Description

Authenticated students and faculty of the university can utilise the virtual
assistant to inquire about course details, requirements, and lecture
materials. It also offers information regarding campus resources, services,
administrative procedures, policies, and general details about the university
and its surroundings.

The pre-trained model is fine-tuned using a dataset comprising student and
faculty-related questions with human-reviewed responses. Additionally, the
model has access to a knowledge base that is regularly updated with
information from the university to assist in answering queries. Access levels
to the knowledge base are differentiated, with students having access only to
information from level 1 for their specific requirements. On the other hand,
Access Level 2 contains sensitive information intended for use as contextual
data by faculty and administrative staff, such as income from students or PII
of students and staff, such as gender and race. Figure 1 illustrates the
system design of the University Virtual Assistant with the security components
highlighted.

####  VI-B2 Security Overview

The system is designed with a focus on usability and accessibility, with an
overall security posture of moderate strength. The system includes an LLM user
input validation and filtering library that has not undergone rigorous testing
for vulnerabilities such as role-promoting injection, which threat actors with
this knowledge could exploit. The library performs a similarity search using a
set of historical prompt injection attacks or attempts. However, it lacks the
capability to detect any variations. The system lacks multi-factor
authentication and has a modest password policy that can be vulnerable to
brute-force attacks.

The training dataset used for fine-tuning the model is benchmarked using only
a single dataset for detecting obvious bias and toxicity; there is no
extensive vetting or verification process with multiple benchmark datasets in
place during the fine-tuning of the model. While data sources are audited
manually, there is no real-time monitoring for malicious activity or anomaly
detection. The system relies on well-established libraries and packages but
does not update them automatically with available patches. Basic rate limiting
with adjustable thresholds is enforced to prevent DoS attacks.

User activities are logged, but advanced statistical analysis in real-time to
detect suspicious activities is not performed. User awareness campaigns on
basic security and system functionalities are conducted. In short, we imagine
a university with limited resources making the best effort - the maturity
level of security operations is slightly naive combined with a lack of
resources.

Figure 1: The system design of the university virtual assistant emphasizes the
diverse security measures implemented for each component. An authenticated
user’s prompt undergoes optimization and validation/filtering for prompt
injection within the prompt optimization module. Contextual information for
the query is accessed based on authorization. The query and context serve as
inputs to the LLM, and the resulting response from the LLM is logged and
cached. Before being sent to the user, the response undergoes filtering. A
rate-limiting mechanism has been implemented at the Gateway to safeguard
against DoS attacks.

| Vulnerability
---
Description
Scenario analysis | Dependency mapping | Impact Analysis
| Prompt Injection
---
| Threat Agent: An authenticated student
---
or someone who has bypassed authentication.
Skill Level: Network and programming skills
\- The potential threat agent or a university student is
expected to have good programming skills and
knowledge of LLM.
Motive: Possible rewards
\- Threat agent might be interested in academic exploits
or as a learning experiment.
Opportunity: Some access or resources required
\- Basic user account access required.
Size: Authenticated users
\- Any of the authenticated student or attackers
who has bypassed authentication can be the attacker.
Method: Crafts malicious prompts
| Dependent Components:
---
1) Prompt validation/filtering mechanism
\- vulnerable to role prompting
2) Access controls
\- weak password policy
3) LLM’s inherent nature
-LLM’s are aligned to be helpful, engaging and responsive
4) LLM output/user activity monitoring
\- logged without review
Ease of exploit: Easy
\- Component 1,2, and 3 makes the exploit easy
Ease of discovery: Automated tools available
\- Tools available to discover the existence of prompt injection.
Enabled by components 1,2, and 3.
Awareness: Public knowledge
\- It is public knowledge that there is no currently a robust defence
against prompt injection attacks.
Intrusion Detection: Logged without review
\- Enabled by component 4.
| Technical Impact:
---
Loss of confidentiality: Minimal non-sensitive data disclosed
\- System prompt leaking, Access unauthorised information
Loss of accountability: Possibly traceable
\- since output logs are maintained, it is possible to trace the
user account responsible for the exploit.
Business Impact:
Privacy Violation: Thousands of people
\- Information about thousands of students could be extracted.
Non-Compliance: Clear violation
\- Major privacy violation
Financial Damage: Significant effect on annual profit
\- Indirect financial damage could arrive in the form of paying
fines for compliance issues and privacy violations.
Reputation Damage: Loss of goodwill
\- Major damage to reputation possible due and people will lose
trust in the application.
| Training Data
---
Poisoning
| Threat Agent: A malicious actor with access to
---
a popular training/finetuning dataset.
Skill Level: Network and programming skills
\- Attacker should have in-depth knowledge of LLM’s
architecture and training process to poison the training
data as well as sophisticated social engineering skills to
smuggle the poisoned training data into the target system.
Motive: Possible rewards
\- Threat agent might be a disgruntled student interested
in reputational damage or performance degradation or
trying to spread misinformation or bias. The exploit will
only materialise if the system incorporates the poisoned
data into their model.
Size: Anonymous Internet users
\- The attacker could be any internet user as there are possibly
multiple instances of poisoned data available in the internet.
Method: Threat agent uploads a poisoned version of a
popular training dataset on HuggingFace.
| Dependent Components:
---
1) Data sources
\- vetting or auditing of the data source is manually, so
infiltration of a poisoned dataset into the system is unlikely, however,
they remain susceptible to human error.
2) LLM output/user activity monitoring
\- logged without review
3) Access logs auditing
\- Access logs are maintained and analysed regularly for unauthorised
or anomalous behaviour.
Ease of exploit: Difficult
\- Component 1 makes it difficult to exploit.
Ease of discovery: Difficult
\- Depends on poisoned data infiltrating the system. Component 1
and 3 makes it difficult.
Awareness: Hidden
\- Threat agents are aware of the existence of the vulnerability,
but it is not known whether the target system is vulnerable
because of the security practice of component 1.
Intrusion Detection: Logged without review
\- If the exploit happens, it is difficult to
detect without active LLM output monitoring. Enabled by
component 2.
| Technical Impact:
---
Loss of integrity: Extensively seriously corrupt data
\- Training data poisoning usually involves injecting bias or
misinformation in the training data in large quantity.
Loss of accountability: Completely anonymous
\- As the threat agent has sophisticated skills to poison training
data, it is very likely they’ll remain anonymous when uploading
the data for public use.
Business Impact:
Financial Damage: Significant effect on annual profit
\- Model training/finetuning is costly. If the model is finetuned on
poisoned data, it cannot be fixed with continual learning. Financial
damage could also arrive in the form of compliance violation dues.
Privacy Violation: Thousands of people
\- Poisoned model also can contain backdoors which will potentially
override guardrails. Student/faculty information can be extracted.
Non-Compliance: Clear violation - Major privacy violation.
Reputation Damage: Brand damage - The model becomes completely
compromised. An attack of this nature results in brand damage.

TABLE II: Risk analysis for the University Virtual Assistant use case
detailing an assessment of Prompt Injection and Training Data Poisoning
vulnerabilities. This analysis involves scenario analysis to estimate threat
agent factors, dependency mapping to align system components with
vulnerability factors, and impact analysis with documented rationale.

| Prompt Injection | Training Data Poisoning
---|---|---
| Skill level | 6 - Network and programming skills | Skill level | 6 - Network and programming skills
| Motive | 4 - Possible reward | Motive | 4 - Possible reward
Threat Agent Factors | Opportunity | 7 - Some access or resources required | Opportunity | 0 - Full access or expensive resources required
| Size | 6 - Authenticated users | Size | 9 - Anonymous Internet users
| Ease of discovery | 9 - Automated tools available | Ease of discovery | 3 - Difficult
| Ease of exploit | 5 - Easy | Ease of exploit | 3 - Difficult
Vulnerability Factors | Awareness | 9 - Public knowledge | Awareness | 1 - Unknown
| Intrusion detection | 8 - Logged without review | Intrusion detection | 8 - Logged without review
| Likelihood Score: | 6.75 | Likelihood Score: | 4.25
| Likelihood: | High | Likelihood: | Medium
| Loss of Confidentiality | 5 - Extensive critical data disclosed | Loss of Confidentiality | 0
| Loss of Integrity | 0 | Loss of Integrity | 7 - Extensive seriously corrupt data
Technical Impact Factors | Loss of Availability | 0 | Loss of Availability | 0
| Loss of Accountability | 7 - Possibly traceable | Loss of Accountability | 9 - Completely anonymous
| Technical Impact Score: | 3 | Technical Impact Score: | 4
| Financial damage | 7 - Significant effect on annual profit | Financial damage | 7 - Significant effect on annual profit
| Reputation damage | 5 - Loss of goodwill | Reputation damage | 9 - Brand damage
Business Impact Factors | Non-compliance | 5 - Clear violation | Non-compliance | 5 - Clear violation
| Privacy violation | 7 - Thousands of people | Privacy violation | 7 - Thousands of people
| Business Impact Score: | 6 | Business Impact Score: | 7
| Final Impact Score: | 4.5 | Final Impact Score: | 5.5
| Impact: | Medium | Impact: | Medium
| Risk Severity: | HIGH | Risk Severity: | MEDIUM

TABLE III: Calculating the Risk Rating using the OWASP Risk Rating Methodology
for the University Virtual Assistant use case. The analysis reveals a high-
risk rating for ‘Prompt Injection’ and a medium-risk rating for ‘Training Data
Poisoning’.

###  VI-C Risk Analysis

In this subsection, we will demonstrate the process of conducting risk
analysis as discussed in section IV-B for two risks: prompt injection and
training data poisoning. Table II presents a detailed breakdown, including
threat agent-specific scenario analysis, dependency mapping of affected system
components and processes, and an assessment of the technical and business
impacts. Subsequently, leveraging this analysis we determine likelihood and
impact values and calculate the overall risk rating as outlined in Table III
referring to the standard likelihood and impact chart, as well as the overall
risk severity chart from OWASP risk rating methodology.

Our analysis of ‘Prompt Injection’ has revealed a high likelihood of
occurrence due to the ease of discovering and exploiting the vulnerability,
coupled with its status as public knowledge. Additionally, we have identified
a moderate impact on the business and system, primarily stemming from
potential financial and reputation damage, as well as the risk of confidential
information leakage.

Similarly, for ‘Training Data Poisoning’, our analysis indicates that the
likelihood of occurrence is influenced by the fact that anonymous internet
users can perpetrate poisoned datasets and detecting such poisoning is
challenging even after it has occurred. The impact of this risk is determined
to be moderate, as it could compromise the model, resulting in financial
losses and damage to the brand’s reputation, given the compromised privacy of
users. The threat matrix reflects these findings, assigning a high-risk rating
to ‘Prompt Injection’ and a medium rating for ‘Training Data Poisoning’, as
illustrated in Table IV.

S.No | Risk Description | Cause & Consequences | Likelihood | Impact |  | Risk
---
Rating
Controls/Mitigation
LLM01 |  | Prompt Injection
---
and Jailbreaking
| Caused by:
---
lack of control/validation
on LLM’s input,
LLM’s implicit nature or
design/architecture
Consequences:
Reputation loss,
Partial IP loss,
Performance degradation,
User harm
High | Medum | High |  | Static:
---
Use trusted/reputed
pre-trained models,
Robust input
validation and filtering
Dynamic:
Adaptive trust
boundaries for input
source,
Monitoring of LLM
outputs,
Red teaming,
LLM Response
monitoring/filtering
LLM03 |  | Training
---
Data Poisoning
| Caused by:
---
Poor vetting/verification
of training data and data
source
Consequences:
Reputation loss,
Model integrity loss,
Financial Damage,
Misinformation and bias,
Performance degradation,
User harm
Medium | Medium | Medium |  | Static:
---
Exhaustive analysis
and sanitisation of all
unvetted training
dataset and data source

TABLE IV: The threat matrix for the University Virtual Assistant use case
reflects the calculated likelihood, impact, and risk rating for prompt
injection and training data poisoning following the completion of the risk
analysis.

Enhancements in component security and robustness typically correlate with a
decrease in the likelihood of successful attacks. However, it is important to
maintain the independence of threat agent and impact factors from such
changes, as adherence to worst-case scenarios mitigates the risk of oversight.
For instance, the implementation of a more robust prompt validation and
filtering library, hardened with multi-layered prompt injection detection
mechanisms, alongside response filtering to preempt potential prompt injection
responses, can potentially increase the level of difficulty associated with
prompt injection attacks. This would indeed decrease the likelihood of an
attack occurring. However, if an adversary were to surpass these constraints,
the impact would remain unchanged. Hence, in evaluating the impact, we solely
consider the potential outcomes if the threat agent were to successfully
exploit the vulnerability.

##  VII Related work

There are two primary avenues of research within the field of LLM security.
The majority of studies introduce innovative techniques for compromising
language models, while counter-studies aim to effectively mitigate or
safeguard against such attacks. However, there is a scarcity of research
focusing on the risk assessment procedures necessary for developers or
security practitioners to evaluate the security resilience of their systems.
Notable works falling into the former category include those from MITRE and
OWASP.

MITRE’s Adversarial Threat Landscape for Artificial-Intelligence Systems
(ATLAS) serves as a structured knowledge base detailing adversary tactics,
techniques, and procedures (TTPs) tailored specifically to AI systems. ATLAS
offers valuable insights into potential AI security threats by delineating how
adversaries might exploit vulnerabilities throughout the AI development and
deployment lifecycle. These TTPs are elucidated through real-world case
studies and attack illustrations, enhancing both comprehension and practical
applicability. Additionally, brief mitigation strategies are provided to aid
in addressing identified threats.

Cui et al.[29] classified risks associated with an LLM system into four
modules: input, language model, toolchain, and output. They are also
conducting risk assessments by evaluating the LLMs using benchmarking datasets
to measure their robustness, truthfulness, and potential biases. Mauri and
Damiani introduced STRIDE-AI [10], a threat modeling approach based on the
widely recognized STRIDE threat modeling from Microsoft. They utilized the
Failure Modes and Effects Analysis (FMEA) process to identify potential
failure modes within the ML lifecycle and map their causes and effects.
Similar to our work, Wilhjelm and Younis explored the application of
traditional threat modeling techniques for Machine Learning Based Systems
(MLBS) [7]. They employed Data Flow Diagrams (DFDs) and STRIDE to identify
threats, utilizing the Microsoft SDL AI/ML Bug Bar to rank threats based on
their impact. Finally, they referenced the Microsoft AML attack library to
propose mitigation strategies.

Q. Zou et al. proposed ML System Security Analysis (ML-SSA), a system
architecture-centered security analysis comprising two main graphs [12]. The
first graph captures all cause-and-effect relationships relevant to assessing
the likelihood of adversarial consequences in an ML system. The second graph
delineates typical dependencies found in software systems, including those
introduced by the supply chain perspective of an ML system. Berryville
Institute of Machine Learning identified 78 generic ML risks and subsequently
highlighted the top 10 risks from this set [6]. They categorised the
components in a generic ML system and mapped them to associated risks.

Kapoor and Bommasani et al. [30] anaylses the benefits and risks posed by open
foundation model. They introduce a framework that focuses on the concept of
marginal risk, aiding in the identification of additional risks society faces
due to open foundation models compared to pre-existing technologies or other
relevant benchmarks. This framework involves several key steps: identifying
the threat, comprehending the existing risk and its defenses in the absence of
open foundation models, presenting evidence of the marginal risks posed by
open foundation models, assessing the ease with which we can defend against
these new risks, and finally outlining the assumptions and uncertainties
related to the risks under consideration. They use this framework to analyse
past studies that analysed cyber risks possed by open foundation models and
found them to be incomplete. Based on their study, they put forward
recommendations to AI developers, researchers investigating AI risks, and
policymakers and regulators.

##  VIII Conclusion and Future Work

This study demonstrates the applicability of established risk assessment
methodologies, such as the OWASP risk rating method, to the unique context of
LLM-based systems. By combining scenario analysis, dependency mapping, and
impact analysis, we provide a systematic framework for identifying and
prioritising risks arising from LLM integration. The hypothetical use case
showcases the practical utility of this approach, empowering security
professionals and developers to make informed decisions about risk mitigation
and resource allocation. In the scenario we assessed, we found that prompt
injection presents a significant threat with a high-risk rating, whereas
training data poisoning has a medium-risk rating. A security practitioner
could utilize this information to prioritize mitigation efforts for prompt
injection over training data poisoning. This might involve recommending
adjustments to the prompt validation and filtering library being utilized, or
implementing a more stringent LLM response filtering mechanism.

Our risk assessment process, culminating in the creation of the threat matrix,
equips each stakeholder group with the knowledge to navigate LLM-related risks
effectively. This helps organisations understand the unique risks faced by
each stakeholder group, enabling them to prioritize mitigation strategies and
tailor security measures accordingly.

Despite the promise of LLMs, their associated risks and still-evolving nature
highlight the need for a comprehensive security approach. We believe this work
can serve as a foundation for more robust risk assessment practices within the
rapidly developing field of LLM applications. Continuous refinement of the
threat matrix will be necessary in response to emerging risks and changes in
defensive and attack strategies. Real-world case studies evaluating the
effectiveness of the proposed process and identifying potential shortcomings
could further refine the process and improve its applicability.

In essence, our study serves as a foundation for ongoing discussions and
advancements in securing LLMs, aiming to pave the way for a more resilient and
trustworthy integration of these powerful language models in future
applications.

## References

  * [1] Communications Security Establishment (Canada) Royal Canadian Mounted Police.  “Harmonized threat and risk assessment (TRA) methodology .”  Communications Security Establishment : Royal Canadian Mounted Police, 2007, 2007  URL: https://publications.gc.ca/site/eng/9.845156/publication.html
  * [2] NIST  “NIST SP 800-30 Rev. 1 Guide for Conducting Risk Assessments”  In _NIST_ NIST, 2012  URL: https://csrc.nist.gov/pubs/sp/800/30/r1/final
  * [3] NIST  “NIST Risk Management Framework (RMF)”  In _NIST_ NIST, 2016  URL: https://csrc.nist.gov/projects/risk-management/about-rmf
  * [4] Paul F Christiano et al.  “Deep reinforcement learning from human preferences”  In _Advances in neural information processing systems_ 30, 2017
  * [5] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [6] Gary McGraw, Harold Figueroa, Victor Shepardson and Richie Bonett  “An architectural risk analysis of machine learning systems: Toward more secure machine learning”  In _Berryville Institute of Machine Learning, Clarke County, VA. Accessed on: Mar_ 23, 2020
  * [7] Carl Wilhjelm and Awad A Younis  “A threat analysis methodology for security requirements elicitation in machine learning based systems”  In _2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C)_ , 2020, pp. 426–433  IEEE
  * [8] Edward J Hu et al.  “Lora: Low-rank adaptation of large language models”  In _arXiv preprint arXiv:2106.09685_ , 2021
  * [9] Yuntao Bai et al.  “Training a helpful and harmless assistant with reinforcement learning from human feedback”  In _arXiv preprint arXiv:2204.05862_ , 2022
  * [10] Lara Mauri and Ernesto Damiani  “Modeling threats to AI-ML systems using STRIDE”  In _Sensors_ 22.17 MDPI, 2022, pp. 6662
  * [11] Long Ouyang et al.  “Training language models to follow instructions with human feedback”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 27730–27744
  * [12] Qingtian Zou et al.  “Attacks on ML Systems: From Security Analysis to Attack Mitigation”  In _International Conference on Information Systems Security_ , 2022, pp. 119–138  Springer
  * [13] Gelei Deng et al.  “Jailbreaker: Automated jailbreak across multiple large language model chatbots”  In _arXiv preprint arXiv:2307.08715_ , 2023
  * [14] Isabel O Gallegos et al.  “Bias and fairness in large language models: A survey”  In _arXiv preprint arXiv:2309.00770_ , 2023
  * [15] Kai Greshake et al.  “More than you’ve asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models”  In _arXiv e-prints_ , 2023, pp. arXiv–2302
  * [16] Yangsibo Huang et al.  “Catastrophic jailbreak of open-source llms via exploiting generation”  In _arXiv preprint arXiv:2310.06987_ , 2023
  * [17] Albert Q Jiang et al.  “Mistral 7B”  In _arXiv preprint arXiv:2310.06825_ , 2023
  * [18] Haoran Li et al.  “Multi-step jailbreaking privacy attacks on chatgpt”  In _arXiv preprint arXiv:2304.05197_ , 2023
  * [19] MITRE  “ATLAS Machine Learning Threat Matrix”  In _MITRE_ MITRE, 2023  URL: https://atlas.mitre.org/matrices/ATLAS/
  * [20] OWASP  “OWASP Top 10 for Large Language Model Applications”  In _OWASP_ OWASP, 2023  URL: https://owasp.org/www-project-top-10-for-large-language-model-applications/
  * [21] Siladitya Ray  “Samsung bans chatgpt among employees after sensitive code leak”  In _Forbes_ Forbes Magazine, 2023  URL: https://www.forbes.com/sites/siladityaray/2023/05/02/samsung-bans-chatgpt-and-other-chatbots-for-employees-after-sensitive-code-leak/?sh=3a06ed686078
  * [22] Schiffer  “Amazon’s Q has “severe hallucinations” and leaks confidential data in public preview, employees warn”  In _Platformer_ Platformer, 2023  URL: https://www.platformer.news/amazons-q-has-severe-hallucinations/
  * [23] Ludwig-Ferdinand Stumpp  “Achieving Code Execution in MathGPT via Prompt Injection”  In _MITRE_ MITRE, 2023  URL: https://atlas.mitre.org/studies/AML.CS0016/
  * [24] Rohan Taori et al.  “Alpaca: A strong, replicable instruction-following model”  In _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ 3.6, 2023, pp. 7
  * [25] Hugo Touvron et al.  “Llama 2: Open foundation and fine-tuned chat models”  In _arXiv preprint arXiv:2307.09288_ , 2023
  * [26] Yotam Wolf, Noam Wies, Yoav Levine and Amnon Shashua  “Fundamental limitations of alignment in large language models”  In _arXiv preprint arXiv:2304.11082_ , 2023
  * [27] Jiashu Xu et al.  “Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Models”  In _arXiv preprint arXiv:2305.14710_ , 2023
  * [28] Yue Zhang et al.  “Siren’s song in the ai ocean: A survey on hallucination in large language models”  In _arXiv preprint arXiv:2309.01219_ , 2023
  * [29] Tianyu Cui et al.  “Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems”  In _arXiv preprint arXiv:2401.05778_ , 2024
  * [30] Kapoor and Bommasani al.  “On the Societal Impact of Open Foundation Models”  In _Stanford CRFM_ Stanford CRFM, 2024  URL: https://crfm.stanford.edu/open-fms/
  * [31] ENISA  “ENISA Risk Assessment”  In _ENISA_ ENISA  URL: https://www.enisa.europa.eu/topics/risk-management/current-risk/risk-management-inventory/rm-process/risk-assessment
  * [32] ISO  “ISO 27001:2022 Information security management systems”  In _ISO_ ISO  URL: https://www.iso.org/standard/27001
  * [33] OWASP  “OWASP Risk Rating Methodology”  In _OWASP_ OWASP  URL: https://owasp.org/www-community/OWASP_Risk_Rating_Methodology


#  Determination of the Néel vector in rutile altermagnets through x-ray
magnetic circular dichroism: the case of MnF2

A. Hariki  Department of Physics and Electronics, Graduate School of
Engineering, Osaka Metropolitan University, 1-1 Gakuen-cho, Nakaku, Sakai,
Osaka 599-8531, Japan  T. Okauchi  Department of Physics and Electronics,
Graduate School of Engineering, Osaka Metropolitan University, 1-1 Gakuen-cho,
Nakaku, Sakai, Osaka 599-8531, Japan  Y. Takahashi  Department of Physics and
Electronics, Graduate School of Engineering, Osaka Metropolitan University,
1-1 Gakuen-cho, Nakaku, Sakai, Osaka 599-8531, Japan  J. Kuneš  Department of
Condensed Matter Physics, Faculty of Science, Masaryk University, Kotlářská 2,
611 37 Brno, Czechia

###### Abstract

We present a numerical simulation of the X-ray magnetic circular dichroism
(XMCD) at the $L_{2,3}$ edge of Mn in altermagnetic MnF2 using a combination
of density functional + exact diagonlization of an atomic model. We explore
how the dichroic spectra vary with the light propagation vector and the Néel
vector. We show how XMCD in rutile structures can be employed to determine the
orientation of the Néel vector. An exact relationship between the XMCD spectra
for different orientation of the Néel vector valid in the absence on the
valence spin-orbit coupling and core-valence multipole interaction is derived
and its approximate validity demonstrated by numerical calculation for the
full Hamiltonian.

Introduction. Altermagnets, a new species on the magnetism evolutionary tree,
have split from antiferromagnets recently [1, 2]. Owing to their space-group
symmetry altermagnets facilitate the presence of spin-polarized bands [3, 4,
1, 5, 6, 7, 8, 2, 9, 10, 11], anomalous Hall effect [12, 5, 13, 14, 15, 9, 16,
17], odd magneto-optical effects [14, 18, 19] and number of other phenomena
with odd Néel vector dependence. The key ingredient is the rotation symmetry
that connects atoms on different magnetic sublattices, instead of translation
or inversion, and makes the time-reversed states with the opposite sign of the
Néel vector macroscopically distinct.

Much of the early studies on altermagnetism focused on RuO2 [3, 20, 21, 22,
23], a metal with the rutile structure. Despite considerable theoretical and
experimental effort the altermagetism of RuO2 is far from understood and
magnetic order in bulk RuO2 remains controversial [24, 25, 26, 27]. Moreover a
large magnetic field is needed to tilt the magnetic moments away from the
$[001]$ easy-axis direction in order to allow finite odd magneto-optical
effects such as circular dichroism [18, 28] or the anomalous Hall effect [20].

These effects are facilitated by the spin-orbit coupling (SOC), which allows
reciprocal-space spin textures (spin-polarized bands) of non-relativistic
origin to affect the charge response of an altermagnet. However, SOC acts also
in the opposite direction and coupling to the orbital degrees of freedom
breaks the spin SU(2) symmetry giving rise to the magneto-crystalline
anisotropy or a weak ferromagnetism due to canting of the local moments. This
disturbs the purely altermagnetic signal and must be taken into account in the
analysis of the experimental data [20].

The x-ray circular dichroism (XMCD) has the advantage that SOC is naturally
separated into a large core contribution and a small valence contribution.
While the latter is responsible for the above ’parasitic’ effects, it has a
marginal effect on the XMCD spectra in lighter elements such as $3d$ metals
[19, 29]. Given the above uncertainty concerning the magnetism of RuO2 it is
desirable to establish the behavior of XMCD in a isostructural material.

MnF2 is a perfect candidate [6]. It Crystallizes in rutile structure and its
magnetism is well studied. It orders antiferromangetically around 67 K [30]
with magnetic moments are aligned along the $c$-axis [31] similar to RuO2. In
contrast to metallic RuO2, MnF2 is a Mott insulator where the Mn2+
configuration gives rise to a large spin moment of $S\approx 5/2$ and an
orbital singlet. Together with weak SOC of Mn this results in a negligible
single-ion anisotropy. The magneto-crystalline anisotropy with $[001]$ easy
axis arises from the long-range dipolar interaction [32, 33]. The spin-flop
transition, at which the moments rotate approximately into the $ab$-plane,
takes place in the field $\sim 9$ T [33, 34] oriented along the $c$-axis.

Methods. We perform a density functional theory (DFT) calculation for the
experimental structure of MnF2 [35] using the Wien2K package [36]. The energy
levels of the Mn 3$d$ orbitals are derived from the DFT bands using the
Wannier90 and wien2wannier packages [37, 38], see the Supplementary Material
(SM) for the computational details [39]. Since MnF2 is a large-gap Mott
insulator, the Mn2+ atomic model adequately accounts for the Mn $L_{2,3}$-edge
XAS spectrum dominated by the intra-atomic multiplet effects as shown by the
early studies by de Groot et al. [40]. Recent experimental and theoretical
work on isoelectronic MnTe [19], which employed both the atomic model as well
as dynamical mean-field theory, came to the same conclusion. Therefore we use
the atomic model where the lattice information is encoded in the sublattice-
dependent crystal field. The staggered spin polarization described by the Néel
vector $\mathbf{L}=\mathbf{S}_{1}-\mathbf{S}_{2}$ is generated by adding a
Zeeman field of 0.01 eV in the desired direction, which is sufficient to
achieve the saturated moment of approximately $5~{}\mu_{B}$. We adopt the
Hund’s coupling $J=0.86$ eV, which is the typical value for Mn2+ systems [19,
41], and account for the SOC in the Mn 2$p$ and 3$d$ shells, as well as the
Slater integrals for the core-valence interaction (2$p$–3$d$), based on atomic
Hartree-Fock calculation as in Refs. [42, 43].

Figure 1: The rutile structure of MnF2 with various orientations of local spin
moments.

The XMCD spectrum $\Delta F(\omega)=F^{+}(\omega)-F^{-}(\omega)$ is the
difference of absorption of the right and left circularly polarized light
propagating along the direction $\hat{\mathbf{k}}$, which is obtained by the
Fermi golden rule

$F^{\pm}(\omega;\hat{\mathbf{k}},\mathbf{L})=\sum_{f}\left|\expectationvalue{f_{\mathbf{L}}|\hat{T}^{\pm}_{\hat{\mathbf{k}}}|i_{\mathbf{L}}}\right|^{2}\\!\delta\left(\omega\\!-\\!E_{fi;\mathbf{L}}\right).$
(1)

Here $|i_{\mathbf{L}}\rangle$ and $|f_{\mathbf{L}}\rangle$ are the eigenstates
of the Hamiltonian, $E_{fi;\mathbf{L}}$ is the excitation energy, and
$\hat{T}^{\pm}_{\hat{\mathbf{k}}}$ are the dipole operators for the right- and
left-hand polarization with respect to the propagation vector
$\hat{\mathbf{k}}$ 111In the following we use the hat symbol
$\hat{\mathbf{v}}$ to indicate the direction of vector $\mathbf{v}$, i.e.,
$\mathbf{v}=a\hat{\mathbf{v}}$ for some $a>0$.. Thanks to the immobility of
the core hole the x-ray absorption spectrum is a sum over the site
contributions. In the dipole approximation the dependence on $\mathbf{L}$ and
$\hat{\mathbf{k}}$ has the form ${\Delta
F(\omega;\hat{\mathbf{k}},\mathbf{L})=2\operatorname{Im}\mathbf{h}_{\mathbf{L}}(\omega)\cdot\hat{\mathbf{k}}}$.
The axial vector
${\mathbf{h}(\omega)=(\sigma^{a}_{zy}(\omega),\sigma^{a}_{xz}(\omega),\sigma^{a}_{yx}(\omega))}$
representing the antisymmetric part of the conductivity tensor
$\boldsymbol{\sigma}(\omega)$ is the finite frequency equivalent of the Hall
vector [45, 1, 18].

Figure 2: The XAS calculated for the two circular polarizations (red and blue)
at the Mn $L_{2,3}$ edge together with the XMCD intensities (shaded) for
different orientations of the Néel vector $\mathbf{L}$ and x-ray propagation
vector $\bf k$. The calculated spectral intensities are broadened by
Lorentizan of 0.15 eV (HWHM). The experimental Mn $L_{2,3}$-edge XAS spectrum
taken from Ref. 40 is shown for comparison. Figure 3:  The orientation of the
Hall vector $\mathbf{h}$ (black arrow) and the local spin moments (blue
arrows) in the $ab$-plane of the rutile structure. Figure 4: The Mn
$L_{3}$-edge XMCD intensities calculated for various angles $\varphi$, with 0∘
and 45∘ corresponding to $\hat{\mathbf{L}}=[110]$ and $[010]$, respectively.
The valence SOC and CV interaction is switched off in (a,b). The XMCD
intensities in the panels (a,b) collapse on a single curve upon division by
$\cos(2\varphi)$ and $sin(2\varphi)$, respectively, as required by (4). (c,d)
The same calculations as in (a,b) with full Hamiltonian. (e,f) The spectra
from (c,d) divided by $\cos(2\varphi)$ and $sin(2\varphi)$ demonstrate that
(4) is approximately fulfilled also with the full Hamiltonian.

Results. The symmetry of the rutile structure [5, 18] implies that XMCD is not
allowed for $\mathbf{L}\parallel c$, i.e., $\mathbf{h}=0$. However, it is
allowed if $\mathbf{L}$ has a finite projection in the $ab$-plane. First, we
consider an idealized situation with magnetic moments entirely in the
$ab$-plane, i.e., $\mathbf{L}\perp c$ and $\mathbf{m}_{1}+\mathbf{m}_{2}=0$.
Figure 2 shows the XAS and XMCD spectra at the Mn $L_{2,3}$ edges calculated
for the Néel vectors $\hat{\mathbf{L}}=[110]$ and $[010]$. The corresponding
Hall vectors are parallel ($\hat{\mathbf{h}}=[110]$) and perpendicular
($\hat{\mathbf{h}}=[100]$) to $\mathbf{L}$, respectively [18]. The XAS line
shape agrees well with the experimental data [40]. Relatively large XMCD
intensities are predicted at both the Mn $L_{2}$ and $L_{3}$ edges in the
present theory, Figs. 2ab.

The similarity of the spectra in Figs. 2ab is not accidental. As shown in Ref.
18 the two spectra are related by symmetry if the valence SOC and the
multipole part of the core-valence (CV) interaction are neglected. This raises
questions about the general orientation of $\mathbf{L}$ in the $ab$-plane. We
will address this question by extending the analysis of Ref. 18.

In the absence of the valence SOC and the CV interaction the valence spin is
decoupled from the rest of the system and the states (including the excited
ones) corresponding to different orientation of $\mathbf{L}$ are related by a
valence spin rotation. Following Ref. 18 we fix the coordinate system so that
$\hat{\mathbf{x}}=\hat{\mathbf{k}}$ and $\hat{\mathbf{z}}=[001]$ as depicted
in Fig. S3 [39], and use dipole operators in the helicity basis along
$\hat{\mathbf{z}}$, which is also the quantization axis for spin. Eq. 1 for
this coordinates becomes

$\begin{split}\Delta
F(\varphi,\alpha)=&\sum_{f}\expectationvalue{f_{\varphi,\alpha}|\hat{T}^{+}-\hat{T}^{-}|i_{\varphi,\alpha}}\expectationvalue{i_{\varphi,\alpha}|\hat{T}^{0}|f_{\varphi,\alpha}}\\\
&\times\delta\left(\omega\\!-\\!E_{fi}\right)+c.c.\\\
\equiv&(T^{+}-T^{-})\overline{T^{0}}+c.c.,\end{split}$ (2)

where the angles $\varphi$ and $\alpha$ capture the orientation of the MnF6
octahedra and the local moment with respect to the light propagation vector
$\hat{\mathbf{k}}$. The dependence of single site XMCD on $\varphi$ and
$\alpha$ has a simple form [18, 39]

$\begin{split}\Delta
F(\varphi,\alpha)=&(e^{i\alpha}T_{\uparrow}^{+}-e^{-i(2\varphi-\alpha)}T_{\uparrow}^{-})\overline{T_{\downarrow}^{0}}\\\
&+(e^{i(2\varphi-\alpha)}T_{\downarrow}^{+}-e^{-i\alpha}T_{\downarrow}^{-})\overline{T_{\uparrow}^{0}}+c.c.\end{split}$
(3)

Upon summation over the two Mn sites we get

$\begin{split}&\tfrac{1}{2}\left(\Delta F(\varphi,\alpha)+\Delta
F(\varphi+\tfrac{\pi}{2},\alpha+\pi)\right)\\\
&=e^{i(2\varphi-\alpha)}T_{\downarrow}^{+}\overline{T_{\uparrow}^{0}}-e^{-i(2\varphi-\alpha)}T_{\uparrow}^{-}\overline{T_{\downarrow}^{0}}+c.c.\\\
&=A(\omega)\cos(2\varphi-\alpha),\end{split}$ (4)

where the bottom line comes from the fact that (4) must vanish for $\varphi=0$
and $\alpha=\tfrac{\pi}{2}$. In Fig. 4 we check the validity of (4) by
explicit calculation in Figs. 4ab.

Eq. 4 has a simple geometrical interpretation. If one starts with $\mathbf{L}$
pointing along $[110]$ or $[1\bar{1}0]$ and rotates it along the $c$-axis, the
corresponding Hall vector $\mathbf{h}(\omega)$ rotates by the same angle in
the opposite direction, see Fig. 3b, while its $\omega$-dependence remains
unchanged, i.e., $\hat{\mathbf{h}}$ is a mirror image of $\hat{\mathbf{L}}$
with respect to $(110)$ or $(1\bar{1}0)$ plane 222The choice of $(110)$ vs
$(1\bar{1}0)$ plane fixes the sign of the effect and depends on the assignment
of $\mathbf{m}_{1}$ and $\mathbf{m}_{2}$ to the lattice sites.

$\mathbf{h}_{\mathbf{L}}(\omega)=A(\omega)\mathcal{M}_{[110]}\hat{\mathbf{L}}.$
(5)

This formula applies to any structure with the magnetic sublattices related by
four-fold rotation axis and another rotation axis perpendicular to it, which
determines the zero of $\varphi$ in (4). Without the latter condition a
$B(\omega)\sin(2\varphi-\alpha)$ contribution to (4) may appear.

Next, we assess the validity of Eq. 4 in the presence of both the valence SOC
and the multipole part of the core-valence interaction. To this end we vary
the orientation of $\mathbf{L}$ within the $ab$-plane and compute the XMCD
spectra for $\hat{\mathbf{k}}=\hat{\mathbf{L}}$ and
$\hat{\mathbf{k}}=\hat{\mathbf{L}}\times\hat{\mathbf{z}}$. This corresponds to
fixing $\alpha=0$ and $\alpha=90^{\circ}$, respectively, in the geometry of
Fig. 3. The corresponding Mn $L_{3}$-edge XMCD spectra in Figs. 4cd indeed
follow the $\cos 2\varphi$ and $\sin 2\varphi$ dependencies. Upon dividing by
the trigonometric functions the spectra collapse on almost identical curves
shown in Figs. 4ef. It is not clear why the rescaled spectra group into two
groups. The accurate adherence of the XMCD spectra to Eq. 5 enables the
determination of the Néel vector orientation within the $ab$-plane through the
measurement of XMCD’s angular dependence.

Though the easy axis of MnF2 is parallel to [001], the Néel vector
$\mathbf{L}$ can be flopped into the $ab$-plane by a field of 9–10 T [34, 33]
along the $c$-axis, which results in canting of the Mn moments
$\mathbf{S}_{1}$ and $\mathbf{S}_{2}$ along the $c$-axis. The net
magnetization along the $c$-axis for fields close to the spin-flop transition
was estimated to $\mu_{z}=0.3~{}\mu_{B}$ and 0.5 $\mu_{B}$ in the two Mn sites
[33]. In Fig. 5(a), we simulate the effect of canting on the XMCD spectra for
$\hat{\mathbf{L}}=[110]$. The net magnetization along the $c$-axis gives rise
to a finite $h_{z}$ component of the Hall vector. The magnitude of XMCD for
$\hat{\mathbf{k}}=[001]$ in Fig. 5(b) is comparable to the purely
altermagnetic effect for $\hat{\mathbf{k}}=[110]$ despite the out-of-plane
component of the local magnetic moments being order of magnitude smaller than
in-plane one. Similar to the experimental observation on MnTe [19] the
altermagnetic ($\hat{\mathbf{k}}=[110]$) and ferromangetic
($\hat{\mathbf{k}}=[001]$) components of the XMCD spectra exhibit distinct
shapes.

Finally, we calculate the linear dichroism (XMLD) in Fig. 6, which provides
the standard tool to determine the direction of the Néel vector up to a sign.
The shape of the XMLD spectra is distinct from the XMCD spectra and thus
possible signal contamination due to an imperfect polarization in experimental
realization should be easy to eliminate.

Figure 5: The XMCD intensities at the Mn $L_{2,3}$ edge for (left) x-ray
propagation vector $\hat{\mathbf{k}}=[110]$ and (right)
$\hat{\mathbf{k}}=[001]$ for the Neél vector $\hat{\mathbf{L}}=[110]$ with a
small tilt of the Mn magnetic moment along the $z$-axis of
$\mu_{z}=0.5~{}\mu_{B}$ at Mn1 (top) and $\mu_{z}=0.5~{}\mu_{B}$ at Mn2
(middle). The total XMCD intensities are shown in the bottom panels. Figure 6:
The XMLD intensities at the Mn $L_{2,3}$ edge for the x-ray propagation vector
$\hat{\mathbf{k}}=[001]$ and the Néel vector $\hat{\mathbf{L}}=[110]$ (left)
and $\hat{\mathbf{L}}=[010]$ (right). The XMLD is defined for the two x-ray
polarization vectors $\boldsymbol{\epsilon}~{}||~{}[110]$ (red) and
$\boldsymbol{\epsilon}~{}||~{}[1\overline{1}0]$ (blue). A small tilt of the Mn
magnetic moment along the $z$-axis with the values ($\mu_{z}$) indicated in
the panels is considered in the simulation. The total XMLD intensities are
shown in the bottom panels.

Discussion. We compare XMCD in MnF2 and $\alpha$-MnTe. Both compounds are
$S=5/2$ altermagnetic Mn2+ insulators and the presence or absence of the
effect (orientation of the XMCD Hall vector $\mathbf{h}$) depends strongly on
the orientation of the Néel vector $\mathbf{L}$, but behaves differently when
$\mathbf{L}$ is rotated in the $ab$-plane. In $\alpha$-MnTe, $\mathbf{h}$
points along the $c$-axis and changes sign [16, 19] at six nodal points. In
MnF2, the XMCD spectrum remains approximately unchanged with $\mathbf{h}$
turning in the $ab$-plane in the opposite sense to the rotation of
$\mathbf{L}$. In both compounds the valence SOC has a minor impact on the XMCD
spectra. On the other hand, the role of CV multipole interaction is very
different. In $\alpha$-MnTe, with $\mathbf{h}\perp\mathbf{L}$ geometry, XMCD
completely vanishes if CV interaction (and valence SOC) is absent thanks to
presence of the local (3-fold) rotation axis parallel to $\mathbf{h}$ [19]. In
MnF2 the $\mathbf{h}\perp\mathbf{L}$ geometry takes place for
$\mathbf{L}\parallel[100]$, however, there is no local rotation axis parallel
to $\mathbf{h}$ and thus XMCD can exists even in absence of the CV
interaction. Therefore the key interaction for appearence of XMCD in
$\alpha$-MnTe is a mere minor perturbation in MnF2 333It shifts the peak
positions, but does not affect the magnitute of XMCD much.. This is also
reflected in the maximal magnitudes of the effect, which is in MnF2 is about
five times larger than in $\alpha$-MnTe.

We have calculated the x-ray magnetic circular dichroism (XMCD) in the spin-
flop phase of MnF2. The spectral dependence of the in-plane x-ray Hall vector
of altermagnetic origin is distinct from the field-induced out-of-plane
component. We have found a simple relationship between the light propagation
vector maximizing the XMCD ($\mathbf{h}$) and the in-plane Néel vector
$\mathbf{L}$, which allows a unique determination of $\mathbf{L}$ from the
angular dependence of XMCD and thus opens the possibilities such as domain
imaging [48].

###### Acknowledgements.

We thank Karel Výborný, Andriy Smolyanyuk and Anna Kauch for discussions and
critical reading of the manusript. This work was supported by JSPS KAKENHI
Grant Numbers 21K13884, 21H01003, 23K03324, 23H03817 (A.H.), and by the
project Quantum materials for applications in sustainable technologies
(QM4ST), funded as project No. CZ.02.01.01/00/22_008/0004572 by Programme
Johannes Amos Commenius, call Excellent Research and by the Ministry of
Education, Youth and Sports of the Czech Republic through the e-INFRA CZ
(ID:90254)(J.K.).

## References

  * Šmejkal  _et al._ [2022a] L. Šmejkal, J. Sinova, and T. Jungwirth, Emerging Research Landscape of Altermagnetism, Phys. Rev. X 12, 040501 (2022a).
  * Šmejkal  _et al._ [2022b] L. Šmejkal, J. Sinova, and T. Jungwirth, Beyond Conventional Ferromagnetism and Antiferromagnetism: A Phase with Nonrelativistic Spin and Crystal Rotation Symmetry, Phys. Rev. X 12, 031042 (2022b).
  * Ahn  _et al._ [2019] K.-H. Ahn, A. Hariki, K.-W. Lee, and J. Kuneš, Antiferromagnetism in RuO2 as $d$-wave Pomeranchuk instability, Phys. Rev. B 99, 184432 (2019).
  * Hayami  _et al._ [2019] S. Hayami, Y. Yanagi, and H. Kusunose, Momentum-Dependent Spin Splitting by Collinear Antiferromagnetic Ordering, J. Phys. Soc. Jpn. 88, 123702 (2019).
  * Šmejkal  _et al._ [2020] L. Šmejkal, R. González-Hernández, T. Jungwirth, and J. Sinova, Crystal time-reversal symmetry breaking and spontaneous Hall effect in collinear antiferromagnets, Sci. Adv. 6, eaaz8809 (2020).
  * Yuan  _et al._ [2020] L.-D. Yuan, Z. Wang, J.-W. Luo, E. I. Rashba, and A. Zunger, Giant momentum-dependent spin splitting in centrosymmetric low-$Z$ antiferromagnets, Phys. Rev. B 102, 014422 (2020).
  * Yuan  _et al._ [2021] L.-D. Yuan, Z. Wang, J.-W. Luo, and A. Zunger, Prediction of low-Z collinear and noncollinear antiferromagnetic compounds having momentum-dependent spin splitting even without spin-orbit coupling, Phys. Rev. Mater. 5, 014409 (2021).
  * Hayami  _et al._ [2020] S. Hayami, Y. Yanagi, and H. Kusunose, Bottom-up design of spin-split and reshaped electronic band structures in antiferromagnets without spin-orbit coupling: Procedure on the basis of augmented multipoles, Phys. Rev. B 102, 144441 (2020).
  * Mazin  _et al._ [2021] I. I. Mazin, K. Koepernik, M. D. Johannes, R. González-Hernández, and L. Šmejkal, Prediction of unconventional magnetism in doped FeSb2, Proc. Natl. Acad. Sci. U.S.A. 118, e2108924118 (2021).
  * Liu  _et al._ [2022] P. Liu, J. Li, J. Han, X. Wan, and Q. Liu, Spin-Group Symmetry in Magnetic Materials with Negligible Spin-Orbit Coupling, Phys. Rev. X 12, 021016 (2022).
  * [11] J. Yang, Z.-X. Liu, and C. Fang, Symmetry invariants and classes of quasi-particles in magnetically ordered systems having weak spin-orbit coupling, arXiv:2105.12738 .
  * Šmejkal  _et al._ [2022] L. Šmejkal, A. H. MacDonald, J. Sinova, S. Nakatsuji, and T. Jungwirth, Anomalous Hall antiferromagnets, Nat. Rev. Mater. 7, 482 (2022).
  * Samanta  _et al._ [2020] K. Samanta, M. Ležaić, M. Merte, F. Freimuth, S. Blügel, and Y. Mokrousov, Crystal Hall and crystal magneto-optical effect in thin films of SrRuO3, J. Appl. Phys. 127, 213904 (2020).
  * Naka  _et al._ [2020] M. Naka, S. Hayami, H. Kusunose, Y. Yanagi, Y. Motome, and H. Seo, Anomalous Hall effect in $\kappa$-type organic antiferromagnets, Phys. Rev. B 102, 075112 (2020).
  * Hayami and Kusunose [2021] S. Hayami and H. Kusunose, Essential role of the anisotropic magnetic dipole in the anomalous Hall effect, Phys. Rev. B 103, L180407 (2021).
  * Gonzalez Betancourt  _et al._ [2023] R. D. Gonzalez Betancourt, J. Zubáč, R. Gonzalez-Hernandez, K. Geishendorf, Z. Šobáň, G. Springholz, K. Olejník, L. Šmejkal, J. Sinova, T. Jungwirth, S. T. B. Goennenwein, A. Thomas, H. Reichlová, J. Železný, and D. Kriegner, Spontaneous Anomalous Hall Effect Arising from an Unconventional Compensated Magnetic Phase in a Semiconductor, Phys. Rev. Lett. 130, 036702 (2023).
  * Naka  _et al._ [2022] M. Naka, Y. Motome, and H. Seo, Anomalous Hall effect in antiferromagnetic perovskites, Phys. Rev. B 106, 195149 (2022).
  * Hariki  _et al._ [2024a] A. Hariki, Y. Takahashi, and J. Kuneš, X-ray magnetic circular dichroism in RuO2, Phys. Rev. B 109, 094413 (2024a).
  * Hariki  _et al._ [2024b] A. Hariki, A. Dal Din, O. J. Amin, T. Yamaguchi, A. Badura, D. Kriegner, K. W. Edmonds, R. P. Campion, P. Wadley, D. Backes, L. S. I. Veiga, S. S. Dhesi, G. Springholz, L. Šmejkal, K. Výborný, T. Jungwirth, and J. Kuneš, X-ray Magnetic Circular Dichroism in Altermagnetic $\alpha$-MnTe, Phys. Rev. Lett. 132, 176701 (2024b).
  * Feng  _et al._ [2022] Z. Feng, X. Zhou, L. Šmejkal, L. Wu, Z. Zhu, H. Guo, R. González-Hernández, X. Wang, H. Yan, P. Qin, X. Zhang, H. Wu, H. Chen, Z. Meng, L. Liu, Z. Xia, J. Sinova, T. Jungwirth, and Z. Liu, An anomalous Hall effect in altermagnetic ruthenium dioxide, Nat. Electron. 5, 735 (2022).
  * Lovesey  _et al._ [2022] S. W. Lovesey, D. D. Khalyavin, and G. van der Laan, Magnetic properties of RuO2 and charge-magnetic interference in bragg diffraction of circularly polarized X-rays, Phys. Rev. B 105, 014403 (2022).
  * Bai  _et al._ [2022] H. Bai, L. Han, X. Y. Feng, Y. J. Zhou, R. X. Su, Q. Wang, L. Y. Liao, W. X. Zhu, X. Z. Chen, F. Pan, X. L. Fan, and C. Song, Observation of Spin Splitting Torque in a Collinear Antiferromagnet RuO2, Phys. Rev. Lett. 128, 197202 (2022).
  * [23] Y. Guo, J. Zhang, Z. Zhu, Y.-y. Jiang, L. Jiang, C. Wu, J. Dong, X. Xu, W. He, B. He, Z. Huang, L. Du, G. Zhang, K. Wu, X. Han, D.-f. Shao, G. Yu, and H. Wu, Direct and Inverse Spin Splitting Effects in Altermagnetic RuO2, Adv. Sci. n/a, 2400967.
  * Berlijn  _et al._ [2017] T. Berlijn, P. C. Snijders, O. Delaire, H.-D. Zhou, T. A. Maier, H.-B. Cao, S.-X. Chi, M. Matsuda, Y. Wang, M. R. Koehler, P. R. C. Kent, and H. H. Weitering, Itinerant Antiferromagnetism in RuO2, Phys. Rev. Lett. 118, 077201 (2017).
  * Zhu  _et al._ [2019] Z. H. Zhu, J. Strempfer, R. R. Rao, C. A. Occhialini, J. Pelliciari, Y. Choi, T. Kawaguchi, H. You, J. F. Mitchell, Y. Shao-Horn, and R. Comin, Anomalous Antiferromagnetism in Metallic RuO2 Determined by Resonant X-ray Scattering, Phys. Rev. Lett. 122, 017202 (2019).
  * Smolyanyuk  _et al._ [2024] A. Smolyanyuk, I. I. Mazin, L. Garcia-Gassull, and R. Valentí, Fragility of the magnetic order in the prototypical altermagnet RuO2, Phys. Rev. B 109, 134424 (2024).
  * Hiraishi  _et al._ [2024] M. Hiraishi, H. Okabe, A. Koda, R. Kadono, T. Muroi, D. Hirai, and Z. Hiroi, Nonmagnetic Ground State in RuO2 Revealed by Muon Spin Rotation, Phys. Rev. Lett. 132, 166702 (2024).
  * Sasabe  _et al._ [2023] N. Sasabe, M. Mizumaki, T. Uozumi, and Y. Yamasaki, Ferroic Order for Anisotropic Magnetic Dipole Term in Collinear Antiferromagnets of $({t}_{2g}{)}^{4}$ System, Phys. Rev. Lett. 131, 216501 (2023).
  * Kuneš and Oppeneer [2003] J. Kuneš and P. M. Oppeneer, Anisotropic X-ray magnetic linear dichroism at the ${L}_{2,3}$ edges of cubic Fe, Co, and Ni: $\mathit{Ab}$ $\mathit{initio}$ calculations and model theory, Phys. Rev. B 67, 024431 (2003).
  * Stout and Adams [1942] J. W. Stout and H. E. Adams, Magnetism and the Third Law of Thermodynamics. The Heat Capacity of Manganous Fluoride from 13 to 320°K., J. Am. Chem. Soc. 64, 1535 (1942).
  * Erickson [1953] R. A. Erickson, Neutron Diffraction Studies of Antiferromagnetism in Manganous Fluoride and Some Isomorphous Compounds, Phys. Rev. 90, 779 (1953).
  * Keffer [1952] F. Keffer, Anisotropy in the Antiferromagnetic MnF2, Phys. Rev. 87, 608 (1952).
  * Felcher and Kleb [1996] G. P. Felcher and R. Kleb, Antiferromagnetic domains and the spin-flop transition of MnF2, Europhysics Letters 36, 455 (1996).
  * King and Rohrer [1979] A. R. King and H. Rohrer, Spin-flop bicritical point in MnF2, Phys. Rev. B 19, 5864 (1979).
  * Stout and Reed [1954] J. W. Stout and S. A. Reed, The crystal structure of MnF2, FeF2, CoF2, NiF2 and ZnF2, J. Am. Chem. Soc. 76, 5279 (1954).
  * [36] P. Blaha, K. Schwarz, G. Madsen, D. Kvasnicka, and J. Luitz,  _WIEN2k, An Augmented Plane Wave + Local Orbitals Program for Calculating Crystal Properties (Karlheinz Schwarz, Techn. Universitat Wien, Austria, 2001), ISBN 3-9501031-1-2_.
  * Mostofi  _et al._ [2014] A. A. Mostofi, J. R. Yates, G. Pizzi, Y.-S. Lee, I. Souza, D. Vanderbilt, and N. Marzari, An updated version of wannier90: A tool for obtaining maximally-localised Wannier functions, Comput. Phys. Commun. 185, 2309 (2014).
  * Kuneš  _et al._ [2010] J. Kuneš, R. Arita, P. Wissgott, A. Toschi, H. Ikeda, and K. Held, Wien2wannier: From linearized augmented plane waves to maximally localized Wannier functions, Comput. Phys. Commun. 181, 1888 (2010).
  * [39] See Supplementary Material for details at …
  * de Groot  _et al._ [1990] F. M. F. de Groot, J. C. Fuggle, B. T. Thole, and G. A. Sawatzky, 2$p$ X-ray absorption of 3$d$ transition-metal compounds: An atomic multiplet description including the crystal field, Phys. Rev. B 42, 5459 (1990).
  * Anisimov  _et al._ [1991] V. I. Anisimov, J. Zaanen, and O. K. Andersen, Band theory and Mott insulators: Hubbard U instead of Stoner I, Phys. Rev. B 44, 943 (1991).
  * Hariki  _et al._ [2017] A. Hariki, T. Uozumi, and J. Kuneš, LDA+DMFT approach to core-level spectroscopy: Application to $3d$ transition metal compounds, Phys. Rev. B 96, 045111 (2017).
  * Hariki  _et al._ [2020] A. Hariki, M. Winder, T. Uozumi, and J. Kuneš, $\mathrm{LDA}+\mathrm{DMFT}$ approach to resonant inelastic X-ray scattering in correlated materials, Phys. Rev. B 101, 115130 (2020).
  * Note [1] In the following we use the hat symbol $\hat{\mathbf{v}}$ to indicate the direction of vector $\mathbf{v}$, i.e., $\mathbf{v}=a\hat{\mathbf{v}}$ for some $a>0$.
  * Wimmer  _et al._ [2019] S. Wimmer, S. Mankovsky, J. Minár, A. N. Yaresko, and H. Ebert, Magneto-optic and transverse-transport properties of noncollinear antiferromagnets, Phys. Rev. B 100, 214429 (2019).
  * Note [2] The choice of $(110)$ vs $(1\bar{1}0)$ plane fixes the sign of the effect and depends on the assignment of $\mathbf{m}_{1}$ and $\mathbf{m}_{2}$ to the lattice sites.
  * Note [3] It shifts the peak positions, but does not affect the magnitute of XMCD much.
  * Amin  _et al._ [2024] O. J. Amin, A. Dal Din, E. Golias, Y. Niu, A. Zakharov, S. C. Fromage, C. J. B. Fields, S. L. Heywood, R. B. R. B. Cousins, J. Krempasky, J. H. Dil, A., D. Kriegner, B. Kiraly, R. P. Campion, A. W. Rushforth, K. W. Edmonds, S. S. Dhesi, L. Šmejkal, T. Jungwirth, and P. Wadley, Altermagnetism imaged and controlled down to the nanoscale (2024), arXiv:2405.02409 .