# Understanding Users’ Interaction with Login Notifications
(Extended Version)

Philipp Markert , Leona Lassak , Maximilian Golla⋆ , and Markus Dürmuth‡
Ruhr University Bochum, ⋆ Max Plank Institute for Security and Privacy, ‡
Leibniz University Hannover

###### Abstract

Login notifications are intended to inform users about recent sign-ins and
help them protect their accounts from unauthorized access. The notifications
are usually sent if a login occurs from a new location or device, which could
indicate malicious activity. They mostly contain information such as the
location, date, time, and device used to sign in. Users are challenged to
verify whether they recognize the login (because it has been them or someone
they know) or to proactively protect their account from unwanted access by
changing their password. In two user studies, we explore users’ comprehension,
reactions, and expectations of login notifications. We utilize two treatments
to measure users’ behavior in response to login notifications sent for a login
they initiated themselves or based on a malicious actor relying on statistical
sign-in information. Users feel relatively confident identifying legitimate
logins but demonstrate various risky and insecure behaviors when it comes to
malicious sign-ins. We discuss the identified problems and give
recommendations for service providers to ensure usable and secure logins for
everyone.

##  1 Introduction

To protect accounts from unauthorized access, login notifications are intended
to inform users about recent sign-ins. Depending on the service, the
notifications are only sent if the login occurred from an unknown location or
new device, which could indicate malicious activity.

Notifications are often delivered via email and include details about the
device (browser and OS), approximate location, date, and time of the sign-in.
Users need to decide whether the reported login is legitimate or malicious and
are recommended to change the password in case the login is unfamiliar. Logins
can be confused to be malicious when users share accounts and friends or
family log in unknowingly. While the notification is intended to protect users
and provide a feeling of security, it can also be perceived as burdening and
overwhelming by requiring a decision based on technical jargon and
highlighting negative consequences.

Previous work [28] focused on challenge-based notifications and studied
incident-response information-seeking and mental models about attackers. In
contrast, we focus on _granted access_ notifications informing users about a
recent sign-in and analyze users’ comprehension, expectations, and reaction to
the notification.

In this work, we collected and analyzed 67 login notifications sent by real-
world services and developed a _baseline_ notification that we employed in two
user studies. In the first study ($n=220$), we asked participants to imagine
having an account they recently have or have _not_ signed in to (depending on
their treatment) and then showed participants a notification that either
contained their current _real_ sign-in information or was prefilled with
information from an attacker relying on statistical data (i.e., California,
USA; Chrome on Windows). The second study ($n=229$), disguised as a
psychological test, let users create an account they had to sign in to during
the multi-stage study. Participants then either received a legitimate
notification to their email upon signing in themselves (_Legit_) or
unexpectedly received a notification prefilled with statistical sign-in
information after around one week (_Malicious_). We sought to answer the
following questions:

  1. RQ1

_[Comprehension & Reaction] Do users understand why they received the
notification and which factors may have caused receiving it? Which actions do
users take in response, and is resolving the situation a priority?_

We find that participants correctly understand that “a login” caused receiving
the notification. However, they are unaware of or misinterpret the trigger and
are thus unsure how to react appropriately. Unique to Study 1, participants
either claim they would change the password (Legit) or report they would
mistakenly ignore the notification (Malicious). Comparing both studies, we
find a large gap between participants’ intended and actual behavior regarding
password changes.

  2. RQ2

_[Decision-Making & Execution] Do the currently employed notifications help
users distinguish malicious and legitimate logins? Which information helps
account owners with their decision, and do current notifications appropriately
guide users in resolving the situation?_

Based on device and location, participants can correctly attribute
notifications caused by their own logins, but they are confused when the
notification is unexpected (Malicious) and struggle to identify the correct
reaction.

  3. RQ3

_[Perception & Expectation] How do login notifications make users feel? When
do they expect notifications to be sent, and how does prior experience affect
their decision?_

Notifications about malicious logins evoke (more) negative emotions, but
participants who changed their password also felt empowered by taking action
to protect their account. Interestingly, more than 90% of the participants
expect services to send login notifications because it makes them feel
protected.

Our results suggest that login notifications are not, _per se_ , a net
positive for online account security. We find only $\frac{1}{5}$ of the
participants who should have changed their password to protect their account
actually did. While participants appreciate when companies decide to monitor
their accounts for incidents, services that send notifications for every login
can cause warning fatigue. We find malformed login notifications and current
anti-phishing advice problematic and give clear recommendations for service
providers to improve their notifications. While login notifications can help
to reinforce account security, we think that protecting their accounts by
identifying malicious logins should not be the sole responsibility of the user
and highlight the need for better notifications.

##  2 Related Work

Next, we outline how our research extends related work.

###  2.1 Login Notifications & Challenges

Related to our work is a qualitative interview study ($n=67$) by Redmiles
[28]. It explores the account security incident response at Facebook by
interviewing users who experienced a login incident. Different from our work,
Redmiles focused on “secondary authentication” notifications that prompt users
to enter a code to access their accounts. Redmiles interviewed participants
from 5 countries and reported on incident-response information-seeking and
mental models about attackers. Regarding the notifications’ effectiveness,
Redmiles identified a lack of key information to be problematic, e.g., the
likelihood that the notification is about a legitimate threat. In contrast,
our work studies a different notification (see Section 3) and focuses on
users’ comprehension, expectations, and reaction to the notification and not
on regaining access or mental models about attackers. Markert et al. [21]
studied administrators’ configuration of risk-based authentication (RBA). The
predefined notifications were slightly customized, and only few administrators
opted to disable them completely. Generally, participants lacked consensus
about which information to include. Further aspects were the need for more
context and explanation to prevent phishing attacks or the inaccuracy of IP-
based location estimation. Wardle [45] measured the time it takes for leaked
credentials to be abused. For this, Wardle created accounts on web services,
intentionally leaked them online, and then used login notifications, among
other signals, to measure the time between the leak of the credentials and
their first malicious abuse. A study by Doerfler et al. [6] evaluated the
efficacy of login challenges in preventing account takeovers, finding that up
to 94% of phishing-rooted hijacking attempts and 100% of automated hijacking
attempts can be prevented. Still, Gavazzi et al. [9] found that only about 20%
of popular websites employ risk-based measures. Wiefling et al. [48] showed
that verification codes sent via email are the de facto standard for login
challenges enforced by RBA. In a subsequent study, they demonstrated that
providing this code in the subject can reduce the login time [49]. Jubur et
al. [14] presented an attack to bypass push-based two-factor authentication
(2FA) notifications.

###  2.2 Security Warning & Notification Design

There is a large body of literature on security warning design [5, 31, 43].
The most prominent applications are notifications in the context of TLS [2,
8], phishing [27], and malware [3], as well as domains like warnings for
developers [12] or countering misinformation [15]. For user authentication,
there is work on breach notifications [13, 50], password-reuse notifications
[11, 38], notifications to promote the use of 2FA [29, 10], or FIDO2 [16], or
protect users from using common PINs [20].

While certain design patterns discussed in these works, like the use of
opinionated design, proved to be effective for certain applications [8], we
did not observe such patterns in login notifications. This is likely owed to
the uncertainty of whether a true threat is present. Similarly, the use of
alarming language or wording highlighting urgency was low. Interestingly, only
a handful of notifications in our analysis (see Section 3) tried to explain
why the notification is sent and attempted to address contextual
misunderstandings [31, 21]. Personalization was the only best practice
strictly applied [10] by referring to “your account” and including the account
name to create trust and direct the notification.

##  3 Login Notifications in the Wild

Login notifications intend to inform users about recent sign-ins and often
include technical details such as the login time, used device, or approximate
sign-in location. However, they are not sent for every login. While
theoretically significant location or device changes trigger notifications,
the probabilistic nature involving factors like sign-in history and user
behavior makes it difficult to predict when notifications are actually sent.
Some services sent notifications for every login, others only in case of
significant location and device changes causing a higher risk level. For
example, we noticed receiving fewer sign-in notifications if the affected
account had two-factor authentication enabled.

Sign-In Information |  $0\%$$20\%$$40\%$$60\%$$80\%$$100\%$Time ZoneTimeOperating SystemIP AddressCityStateCountryBrowserAccount Name59%77%62%44%36%30%68%62%74%
---|---
Components |  $0\%$$20\%$$40\%$$60\%$$80\%$$100\%$Unsubscribe OptionLegal FooterClosingPhishing MentionedOpportune MomentHelp PageWhy NotificationMalicious InstructionsLegit InstructionsHeadline73%64%97%21%33%24%17%50%67%8%
Wording & Design |  $0\%$$20\%$$40\%$$60\%$$80\%$$100\%$Information in BoxLogoHTMLApproximate LocationNeutral Subject65%26%89%80%23%
Figure 1: The information included in login notifications for a granted access
($n=67$) sent by real-world services.

###  3.1 Notification Types

Based on the type of information they convey, services usually distinguish
between three different notification types [21]:

  1. (1)

Granted Access: The notification inform about a granted access. Some services
send such notifications for every sign-in, while others follow a risk-based
approach.

  2. (2)

Additional Challenge: These notifications inform about a new sign-in attempt
for which an additional challenge needs to be solved (i.e., insert a code or
click a link).

  3. (3)

Blocked Access: The notification informs users about a blocked access, which
can happen because the risk-based authentication system ranks the sign-in as
too risky.

For the remainder of this work, we focus on the first type, i.e.,
notifications informing the user about a granted access. Sending this type can
be done by every organization, as it does not require an advanced risk
assessment (i.e., basic logic and the ability to display login details are
enough). Moreover, we limited our dataset to email-based notifications. While
notifications can also be via other channels, e.g., SMS or push notifications,
establishing them requires additional effort.

Figure 2: The _baseline_ login notification, as used in our user studies, is
derived from ($n=67$) real-world notifications.

###  3.2 Analysis

To find a representative _baseline_ notification, we collected more than $80$
login-related emails sent by real-world services by actively enumerating over
$500$ existing accounts. To trigger the notification, we signed in using the
Tor browser, which is often classified as suspicious activity, and monitored
our inbox. We also searched through account remediation pages [24] and
community support forums [4], and learned about them via friends and
colleagues, in both cases, we created an account on the service to obtain a
notification ourselves. Our collection is limited to the top Tranco list [17]
websites (as of June 2023), with about $\frac{1}{3}$ being in the top
$100$/$1,000$/$50,000$ respectively. The dataset is biased towards English
language websites (few non-English notifications have been translated).

For analysis, two authors categorized $67$ of the emails as _granted access_
notifications. They then independently analyzed various aspects: which sign-in
information the notifications includes (i.e., login time, location, device),
what the main components are (i.e., headline, malicious instructions), salient
design and wording decisions (i.e., logo, highlighting of sign-in details,
neutral language), and metadata such as sender and subject. We summarize our
findings in Figure 1. Please refer to Appendix D and E for the full details.

Sign-In Information As depicted in Figure 1, the majority of notifications
included the login 7  _Time_ ($77\%$), 5  _Account Name_ ($74\%$), 7
_Country_ ($68\%$), 7  _Browser_ ($62\%$), and 7  _Operating System_ ($62\%$).
Less frequently, the notifications also included the 7  _Time Zone_ ($59\%$)
or a login _IP Address_ ($44\%$). The small number of notifications including
the login _City_ ($36\%$) or _State_ ($30\%$), is explained by geographical
differences between the U.S. and Europe. Notifications about logins in the
U.S. mostly focused on states; European notifications often focused on cities.

Components Throughout all notifications, we noted the presence of a 4
_Headline_ ($73\%$) that was often ($76\%$) different from the email subject.
Another key component is the instructions describing how users should react in
response to receiving the notification. While only $64\%$ provided
instructions in the 8  _Legitimate_ case, more than $97\%$ explained how to
react in the 9  _Malicious_ case if the user does not recognize the login. The
large majority ($66\%$) recommended changing the password. Fewer (high-ranked)
web services included a button to report the login as malicious or legit on a
separate web page ($9\%$) displaying account remediation steps. Similarly, a
small number ($9\%$) suggested to visit the account activity page. Prominent
among financial services was the option to contact support ($4\%$). A
dedicated _Why Notification_ component was included in $21\%$ of the
notifications. It primarily creates context and explains to users why they
received the notification. It often gives examples of legitimate (i.e., new
device) and malicious causes (“someone unauthorized gained access”) that might
have triggered the notification. $33\%$ included a link to a dedicated _Help
Page_ (note: _regular_ support links in the email footer were not counted).

About $24\%$ of the emails tried to use the _Opportune Moment_ to tell the
user about other options to secure their account (i.e., enabling 2FA). The
dangers of _Phishing_ and methods to double-check the legitimacy of the
notification were mentioned in $17\%$ of the emails, with the most prominent
suggestion to not click the “change password” link and instead sign in to the
website by manually pasting or typing in the URL. About half of the
notifications included a 2418241810  _Closing_ ($50\%$) text that often
thanked the user and included the name of a “{service} account team.” A footer
with 2418241811  _Legal_ information was included in $67\%$ of the emails, and
an _Unsubscribe_ link was present in $8\%$ of the notifications.

Wording & Design Using affinity diagramming, we identified the wording of most
email subjects as _Neutral_ ($65\%$), with a strong focus on “ _New login
to{service}_.” In some cases it is alarming ($23\%$), like “ _Security alert_
” or a prompt ($9\%$), like “ _Please review this sign in!_.” In two cases it
was a question ($3\%$), e.g., “ _Did you recently sign into{service}_?” Almost
all emails ($92\%$) referred to 6 “your account” to emphasize the importance
of the notification. A few services tried to address the inaccuracies of IP-
based location estimation by describing it as _Approximate_ ($26\%$). Most of
the notifications ($89\%$) were sent as _HTML_ emails; the rest were sent in
plaintext. For a “corporate look-and-feel,” $80\%$ of all notifications
included a 3  _Logo_ , with an even split between a centered or left
alignment. Interestingly, $23\%$ of the emails opted to display the sign-in
information in a visually detached box, most likely to draw the user’s
attention to the login details.

Selecting a Representative Login Notification Our data-driven _baseline_ login
notification (see Figure 2) includes all components used by at least $50\%$ of
the notifications. It uses a neutral subject and a slightly modified headline.
We adjusted our email sender, opted for an HTML email, and included a logo.
Also, we included the affected account name and referred to “your account.” We
included the most popular sign-in details, as well as legitimate and malicious
instructions for users to take after receiving the notification. As our study
sample was U.S.-based, we included the 7  _State_ in the sign-in details. The
email also included a closing and footer with fictional legal information.

##  4 Self-Report Study: Method

This section describes our self-report studies’ methodology. We first explain
the study structure and treatments. Next, we describe the recruitment process,
demographics, limitations, and ethical considerations. This initial self-
report study aims to explore tendencies and identify themes to inform the
design of the subsequent measurement study.

###  4.1 Study Protocol

We ran a quantitative online study with the following structure utilizing our
baseline notification (see Section 3).

  1. (1)

Consent: We briefed participants and asked for consent.

  2. (2)

Scenario: Next, we described the scenario for the study telling participants
to imagine that they are Jo Doe<EMAIL_ADDRESS>and have an account with
the company AcmeCo which is “like other accounts you may have, such as for
online shopping or social media.” Based on their (randomly) assigned treatment
(see Section 4.2), we told participants on a subsequent page to imagine that
they recently signed into this AcmeCo account (Legit) or have not signed into
the account for a while (Malicious).

  3. (3)

Notification: After describing the circumstances, participants saw a login
notification sent by AcmeCo (Figure 2). The login information depicted in the
notification was either derived from the currently used system (Legit) or
showed a login from California using Chrome on Windows (Malicious). From now
on, up to the questions about prior experiences, a two-column layout displayed
the notification on the left and the questions on the right.

  4. (4)

Reaction: To understand how participants felt in reaction to being exposed to
the notification, we used I-PANAS-SF [39]. Afterward, we asked participants to
list three actions they would take after receiving the notification (SQ1), how
concerned they would feel (SQ2–SQ3), and their priority of taking action
(SQ4–SQ5).

  5. (5)

Understanding: Questions SQ6–SQ9 investigated if participants understand what
the notification is telling them and whether it appropriately guides them.

  6. (6)

Expectation: Next, we asked what participants think would be the consequence
of ignoring (SQ10–SQ11) and if they expect real companies to send similar
emails (SQ12–SQ13), it also contained an attention check SAC.

  7. (7)

Prior Experience: If they have received similar login notifications in the
past (SQ14), we asked if and why they decided to read them (SQ15–SQ16) and
situations where such notifications helped them (SQ17–SQ18).

  8. (8)

Demography: Next, we asked for demographics, including their age, gender
identity, education, and any technical background (SD1–SD4). The study closed
by asking participants if they participated honestly.

###  4.2 Treatments

We used two treatments to cover the cases that can trigger a notification: a
Legit and a Malicious login. While a legit login is initiated by the user (or
anyone who was intentionally given access), a malicious login is initiated by
an attacker who got hold of the password, e.g., via a breach.

Legit ($n=110$): To mimic a legit login, participants in this treatment should
imagine that they recently logged into their AcmeCo account. The login details
for the notification were obtained from the current session, i.e., their
operating system and browser (based on the HTTP User-Agent string), as well as
state and country derived from the IP address.

Malicious ($n=110$): Participants in this treatment were told to imagine they
had not logged into their AcmeCo account for a while and were all shown a
notification for a login from California using Chrome on Windows. These login
details were selected as we expect an attacker to know the victim resides in
the USA. Logging in from the state with the highest population [40], as well
as the browser and operating system with the highest market share [35, 36]
minimizes the risk of being detected. We consider more sophisticated attacks
[6, 22] that involve relaying login details and spoofing the user’s sign-in
information out of scope, as it makes recognizing such logins very difficult.

###  4.3 Recruitment and Demographics

Using Prolific, we recruited $n=220$ participants, 110 for each treatment. A
power analysis ($d=0.4,\alpha=0.05,power=0.8$) yielded 100 as the necessary
treatment size to which we added a 10% buffer to account for errors. We
required participants to be over the age of 18 and reside in the U.S. to
fulfill the assumption of the attacker in the malicious treatment. We did not
exclude any participants as all of them passed the attention check (SAC) and
indicated to have answered honestly; we also did not identify any
inconsistencies. The self-report study was compensated with $2.50 and took
participants, on average, 12 minutes to complete.

The demographics for the self-report study are depicted in Table 1 in Appendix
C. Participants split equally between male- and female-identifying
participants; seven identified as non-binary, and four preferred not to
disclose their gender. In terms of age and education, 49% of the participants
were younger than 35 years, and 50% had a Bachelor’s or Master’s degree.
Lastly, 73% of the participants stated not to have a technical background.

###  4.4 Limitations

Although we carefully considered various aspects of the study, it has its
limitations which we will outline in the following. First, we conducted an
online study and asked participants to describe their behavior, opinion, and
perception based on a fictitious scenario. As research has shown [46, 47], it
is difficult to determine if this reflects reality even if none of the
participants indicated to have answered dishonestly at the end of the study.
To draw our conclusion based on measured instead of intended behavior and
experienced situations, we only used this study to inform the design of a
subsequent study to delve further into the topic. Second, the self-report
study’s sample was comprised of younger and higher-educated participants,
which is typical for Prolific or other crowdsourcing platforms. We note that
results may change for different populations, which should be investigated in
succeeding studies. Third, results may suffer from self-report biases (e.g.,
social desirability). To mitigate this, we did not explain that this was a
study about usability or security. Fourth, like many human-subject studies,
there is the potential for a bias in question wording. To circumvent this, we
piloted the study and tried to keep the questions short and clear. The full
survey instrument can be found in Appendix A. Fifth, we only recruited US-
based participants, which can have culture-based influences on the results.

###  4.5 Ethical Considerations

When the self-report study was conducted, none of the authors worked at an
institution with an Institutional Review Board (IRB) that could oversee it. We
followed the principles of the Menlo Report [41] “Respect for Persons,”
“Beneficence,” and “Justice.” We designed the study to minimize any potential
harm while maximizing benefits. At the beginning of the study, we described
the study procedure carefully, including all risks, and asked participants for
their consent. Participants could withdraw from the study at any point without
risking losing their compensation. All collected data were stored and
processed in accordance with the General Data Protection Regulation (GDPR).

##  5 Self-Report Study: Results

Next, we present the results of our self-report study. As self-reported data
may be exaggerated and biased, we mostly refrain from reporting exact numbers
and only report general trends and themes to address this limitation.

The qualitative coding was done by two of the researchers, who started by
separately coding 10% of the answers. Afterward, they agreed on a joint
codebook (see Tables 5–8) and used it to code the remaining 90%. The agreement
between the two coders for this first study was high ($\kappa=0.82$). When
quoting individual participants, e.g., L61-N, one can derive their treatment
(Legit or Malicious) and password change behavior (Would Not Change or Would
Change). Similarly, we re-use the color codes like  A Was Me from Figure 3
within the text when referring to participants’ explanation.

###  5.1 RQ1: Comprehension & Reaction

Comprehension Questions SQ6 and SQ7 asked participants to describe what the
notification is telling them and why they may have received it. All 220
participants correctly described that a new login happened, yet, about a
third, both in the legit and the malicious group, assumed that the login must
have been abnormal. Some respondents speculated that the device or the
location is unknown. Missing cookies, e.g., when using private browsing, were
brought up by few participants, and just one person mentioned an unusual login
time as a reason. However, all of those deviations may trigger a notification.
We further discuss this discrepancy between users’ understanding and the
actual implementation in Section 8.

Legit |  Would Not ChangeWould Change$0$%$20$%$40$%$60$%$80$%$100$%AECDB
---|---
Malicious |  Would Not ChangeWould Change$0$%$20$%$40$%$60$%$80$%$100$%Was MeProb. MeFatigueOtherUnsureNot MeADB

Figure 3: Breakdown (based on SQ1), into those who reported they would or
would not change their password.

Reaction Legit The 110 participants in the legit group split equally between
participants who would ignore the notification and those who would change
their password (SQ1).

The majority of participants who would ignore it argued that they  A
definitely or  E probably received it because they logged in themselves:

  * “ _I live in Texas, use Chrome on Windows, and have just logged into this account. It’s probably me._ ” (L61-N)

Fewer participants thought it was them causing the notification, but
additionally expressed  C fatigue:

  * “ _Delete the email because I know I am the person who signed in, […] I unsubscribe from AcmeCo entirely because I am tired of receiving these emails constantly._ ” (L80-N)

Of participants who said they would change their password (unnecessarily), the
majority were  D unsure whether it was them logging in. They said they would
change the password to eliminate the possibility of someone having access:

  * “ _It looks like someone else logged in. […] Even if I’m not sure, might do this just in case._ ” (L87-C)

The remaining  B participants were certain that someone else would know their
password:

  * “ _I would think my password was compromised._ ” (L56-C)

Taking action is a higher priority for participants who said that they would
change their password than for those who do not. Likewise, their agreement to
SQ10 saying that “ignoring this email from AcmeCo would have consequences” is
also higher.

Generally, the ratings of participants who reported that they would change
their password in the legit group are more similar to those participants in
the malicious group who reported that they would change their password, which
highlights the strong influence of a misinterpreted notification.

Reaction Malicious In the malicious group, the majority described that they
would change their password, only a few said they would ignore it. Of the
former,  B most justified their reaction by (correctly) saying that someone
else just logged in:

  * “ _Someone else got into my account, so I need to change the password right away._ ” (M56-C)

Again, there are also  D participants who expressed some form of uncertainty,
yet, in contrast to the legit group, they mostly questioned the legitimacy of
the email:

  * “ _I would hover over the email to ensure it wasn’t a phishing email […] after ensuring it was legit and safe, I would change my password_ ” (M87-C)

Of the participants in the malicious group who said they would ignore it,  A
most described that it must have been them logging in, misinterpreting the
situation:

  * “ _It’s safe to assume that my account is not compromised. […] It’s good to see that my safety is a concern._ ” (M92-N)

Only  F few participants argued differently: M25-N mentions phishing, similar
to many other participants, but would “send the email to spam” and not do
anything else. M51-N, on the other hand, would “ignore it. It is my business,
not theirs.”

Similar to the legit treatment, the intend to take action in response to the
notification was of higher priority for participants who supposed to be
compromised than those who did not. Likewise, of those who intended to change
their password, the majority _agreed_ or _strongly agreed_ that ignoring the
notification would have consequences.

Summary While users understand that the notification informs them about a new
login, their comprehension of potential triggers deviates from what can be
actual reasons. We find a tendency to misinterpret the notification:
participants either stated they would unnecessarily change the password
(Legit) or they would mistakenly ignore the notification (Malicious). Note
this study is limited to self-reported, hypothetical reactions. Thus, we
further investigate these findings in the subsequent study (see Section 6).

###  5.2 RQ2: Decision-Making & Execution

Decision-Making To identify the type of information (location, date, device)
participants based their decision on, we specifically checked the open
responses. Most participants consider the device as a factor (explicitly the
browser or OS). However, responses vary noticeably between the 4 groups. While
4 of 5 participants in the Legit (Would Not Change) group mentioned the device
when describing how they made their decision, slightly fewer did in the Legit
(Would Change) group. In the malicious treatment, the device was mentioned
less than half of the time by those who would not change and even more seldom
by those who stated they would.

Although intuitively the location should be the easiest, hence, most popular
indicator for identifying the notification’s intent, it is only reported as
the second most frequent across all groups. Again, participants in the legit
Legit (Would Not Change) treatment included it most often, followed by Legit
(Would Change). In the malicious treatment, 1 in 3 for the Would Not Change
and Would Change groups said the same.

Lastly, only some participants mentioned the date and/or time in their
responses. When comparing the groups, ratios range from a few for the
Malicious (Would Change) group to about half for the Legit (Would Not Change)
group, which fits the overall picture: participants in the legit group who
expressed not to change their password refer to the login factors the most,
followed by the Legit (Would Change) and the Malicious (Would Not Change)
group. The participants who refer to any of the 3 types of information the
least are those in the Malicious (Would Change) group.

A second view on this is given by SQ18, where we asked participants if they
ever had a real-world situation in which they learned about an unrecognized
login via such a notification. Of those who did, half described that the
location was the determining factor. Only a few said that the information
about the device helped them, and only twice it was the time.

Execution Question SQ8 asked participants if the notification explained how to
resolve the situation. The majority across all groups agrees or strongly
agrees. When explaining their answer (SQ9), most participants focused on how
to resolve the situation for a malicious login, even if they do not intend to
change their password:

  * “ _It gave me steps to take if this was not me_ ” (L41-N)

Another group explicitly mentions both outcomes:

  * “ _It explains to ignore the email if it was me, and change my password if it wasn’t._ ” (L101-N)

Participants who criticize the notification report that they miss information
in some form:

  * “ _Changing my password is the very first line of defense, but I wish there was more information about what steps to take, or further assistance from AcmeCo._ ” (M26-C)

This highlights an important aspect, as account remediation usually consists
of more steps than just changing the password. Others [23, 44] also observed a
lack of advice provided by websites. In Section 8, we discuss the risk that
arises from this lack of information.

Lastly, some participants referred to the change your password link, which is
provided in the notification. Some noted:

  * “ _The steps to take were very clear. It also provided a link to make it easy to change my password if I needed to._ ” (L42-C)

Other participants perceived the link negatively, saying that they do not
trust it and describe it as bad practice:

  * “ _It shouldn’t be suggesting that I click the link in the email as this is a trick often used by scammers._ ” (M35-C)

Both positions provide reasonable arguments which come down to a weighting of
usability and security. We further contrast both perspectives in Section 8.

Summary Based on the open-ended questions, the device and the location appear
to be influential types of information when participants make their decision.
The Legit (Would Not Change) group generally refers to login information the
most, Malicious (Would Change) participants the least. As we identified a
mismatch between the responses in the study and real-world experiences, we
employ the second study investigating to what extent the different types of
information actually drive their decision processes. In terms of the
execution, most participants, irrespective of the treatment and the described
action, agree that the notification provides them with sufficient information.
Participants who commented negatively mentioned a lack of information or
disliked the change your password link.

###  5.3 RQ3: Perception & Expectation

Perception Using the PANAS, we investigated how the notifications make
participants feel and asked them about their concern (SQ2). Participants in
the Malicious (Would Change) group felt the most positive followed by Legit
(Would Change). Malicious (Would Not Change) and Legit (Would Not Change)
ranked the lowest, which shows that participants feel more positive if they do
something in reaction to the notification. This may explain why participants
(Legit) tended to change their password even if it may not be necessary.

The situation for the negative affect is more distinct. Participants in the
malicious group who described to change their password feel more negative than
those in all other groups. This stark contrast may be explained by the fact
that the majority of them report being hacked, while only a few in the Legit
(Would Change) have a similar assumption and all others just try to “better be
safe than sorry.” The Legit (Would Not Change) and Malicious (Would Not
Change) group feel less negative, even compared to Legit (Would Change). This
aligns with the participant’s understanding of the notification as information
about their own login.

Participants’ responses regarding the concern (SQ2–SQ3) match the previous
observations. The Malicious (Would Change) has the highest concern level, with
the majority being moderately or extremely concerned. This is higher compared
to Legit (Would Change), where some are moderately, and few are extremely
concerned. Participants who described not to change their password, reported
the lowest concern.

Expectation In response to SQ12, nearly all participants agree or strongly
agree that notifications should be sent by real companies. They feel that the
notifications protect accounts, alert customers, and show that companies are
invested in security. The only negative aspect mentioned was annoyance:

  * “ _People who log into the site a lot are going to get spammed. It would be better if they only sent you a notification when they detect a strange login._ ” (L32-N)

Still, this is only reported by few participants. Others were not annoyed and
reported to receive such notifications regularly (SQ14). More than three times
as many said they get them many times, and the largest portion described to
receive them occasionally. Similarly, more than half said they read them
always, some often, few sometimes. Hence, there appears to be some decline
which could be explained by participants differentiating expected and
unexpected notifications:

  * “ _If I signed in from a second device and then received the email, I knew it was me, so I would ignore it. But when I didn’t sign in, I would pay attention and read the email._ ” (L16-N)

Summary Participants who would change their password have higher positive
affect ratings, which may be explained by them securing their accounts. Those
that assumed to be hacked also felt most negatively. Participants who reported
they would not change their password reacted significantly more indifferent.
We use Study 2 to observe participants’ feelings when confronted with a real
notification. Almost all participants expect services to send login
notifications, leaving them feeling protected. They claimed to always read
them or at least when the notification arrives unexpectedly. Only a few
reported being annoyed by such notifications.

##  6 Measurement Study: Method

In this second study, we expand on the self-reported results by measuring user
behavior on an actual website. The following outlines the protocol,
treatments, recruitment, ethics, and limitations.

###  6.1 Study Protocol

In contrast to Study 1, participants in this study should receive a
notification for a concrete account. To resemble a real-world setting, the
protocol had to fulfill four criteria:  (1) a _real account_ gets created,
(2) participants are _unaware_ that the study is about login notifications,
(3) participants receive the notification in their _personal email account_ ,
(4) reactions to login notifications are _measurable_.

For this, we invited participants to take part in a multi-stage study about
changes in the cognitive ability of mental rotation over time [34, 42]. This
framing allowed us to inform people about the length of the commitment without
revealing our interest and justified the account creation on our website. The
task also was a strong cognitive distractor that prevented participants from
drawing too much attention to the authentication task.

Similar to Study 1, we used 2 treatments and the baseline notification (see
Figure 2) adapted to the branding of our study’s website: The legit group
($n=110$) received a notification only after they logged in themselves. The
location, date, and device in the notification were derived from the metadata
of their login. The malicious group ($n=119$) received a notification
unexpectedly at a time when they had not interacted with the account for
multiple days. This resembled a login attempt by a malicious actor. The
location (“California, USA”) and device (“Chrome on Windows”) were selected to
have the highest statistical chance of matching any user in our U.S.-based
sample [40, 35, 36]. We did not allow mobile devices and used a tracking pixel
to see if the email was opened.

Stage 1: The first page on the study’s website explained the mental rotation
test. To ensure participants would regularly check their email and understand
the value of the account, after giving their consent, they saw a privacy
notice, which highlighted the importance of the account as it would be used to
store the study data, name, and email address. It also explained that the
email would be used to send invitations to subsequent stages, and the
compensation in form of Amazon gift cards. After the account creation,
participants solved 5 mental rotation tests and provided demographic
information (MD1–MD4). At the end, participants in the legit treatment were
informed that invitations to Stage 2 would be sent in approx. 7 days; in the
malicious group, the note said 14 days.

Stage 2: After 7 days, participants in the legit group received an email
inviting them to return to our website to conduct another mental rotation
test. To do so, they had to log into their account, which triggered a login
notification. Participants in the malicious group expected their next email
after 14 days. However, to imitate a malicious login, we sent them an
(unexpected) login notification filled with our statistical sign-in data 7
days after they completed the first stage.

Stage 3: For the legit group, invitations to the final Stage 3 were sent 48
hours after they completed Stage 2; in the malicious group, 48 hours after
they received a notification for a login they did not initiate. We chose this
time frame to give participants enough time to react. After logging into Stage
3, participants were debriefed and told about the purpose of the study. This
was followed by our questionnaire (see Appendix B). From then on, the
notification we sent was shown on the left side of their screen for reference.

  1. (1)

Email: First, we asked participants if they remember receiving the
notification (MQ0); if not, they were forwarded to a different section (see
Appendix B). Participants who triggered the tracking pixel in the notification
or who changed their password skipped this question.

  2. (2)

I-PANAS-SF: To learn about the feelings and emotions in reaction to the
notification, we again utilized the Positive and Negative Affect Schedule
(I-PANAS-SF) [39].

  3. (3)

Reaction: Next, we asked how thoroughly participants read the notification
(MQ1) and how and why they chose to react to it (MQ2a–MQ3a). Participants who
changed their password were specifically asked about any other actions
(MQ2b–MQ3b).

  4. (4)

Content & Design: To better understand the reactions, MQ4 asked about
influencing factors like metadata, content, and design. MQ5 specifically asked
about the helpfulness of the account name, location, date, and device.

  5. (5)

Time & Location: MQ6–MQ10 investigated the time when and location where the
notification was read. With MQ7, we verified if the location, which had been
derived automatically, was actually accurate or could have led to confusion,
and MAC2 was an attention check.

  6. (6)

Comprehension & Expectation: With MQ11, we captured if participants understood
why they received the notification. MQ12 and MQ13 asked participants when they
expect real companies to send notifications.

  7. (7)

Prior Experience: We concluded with three questions covering negative
experiences with security incidents (MQ14), as well as their opinion on
regular (MQ15) and event-driven password changes (MQ16).

###  6.2 Recruitment & Demographics

We used Cint for the recruitment of the measurement study, as Prolific could
not support all our requirements. For Stage 1, we recruited 625 participants,
about 3 times more than the desired number of completions as recommended by
the panel provider. After filtering 12 participants who failed the attention
check (MAC1), 613 participants remained. At the end of Stage 3, we had 252
completions. This high number of dropouts is almost exclusively owed to
participants who did not return after the first stage of the study. Lastly, we
removed 23 participants as they provided unrelated answers or failed the
second attention check (MAC2) for a final number of $n=229$. Stage 1 took, on
average, 2.5 minutes and was compensated with $3.00 USD. Stage 3 took, on
average, 6 minutes and was compensated with $4.00 USD. Participants in the
legit group received an additional $1.00 USD for the completion of Stage 2,
which took 2 minutes on average. Regarding the demographics, we observe a
shift towards male-identifying participants (65%), the age distribution is
diverse, ranging from 14% to 21% for all age groups between 25 and 74. Most
participants had a high school (33%), Bachelor’s (26%), or trade degree (23%)
and did not have a technical background (82%). The full details are given in
Table 2 in Appendix C.

###  6.3 Ethical Considerations

At the time we conducted the measurement study, none of the authors worked at
an institution with an IRB. However, we carefully followed the guidelines
provided in the Menlo Report, including a risk-benefit evaluation, developing
the protocol with peers familiar with conducting user studies, and following
the legal requirements. The study included deception and sent a login
notification to participants’ personal email accounts, which could have caused
more anxiety than just imagining to have received a login notification. We
actively tried to avoid this situation by conducting our self-report study
first. However, as half of the participants stated they would (unnecessarily)
change their password, we decided to verify this potentially user-burdening
behavior.

To protect participants from unnecessary risks, we implemented several
safeguards:  i) Our panel provider offered the study only to participants that
agreed to studies that might involve deception.  ii) The affected spatial
reasoning account had no subjective value to the participants and only allowed
to access the email address.  iii) All participants have been debriefed (also
the ones that decided to withdraw early or drop out). In particular, we told
them about the true purpose of the study, and in case they belonged to the
_malicious_ treatment that “This sign-in did not take place; at no time was
your account at risk,” and asked them whether they prefer to leave the study
early (while being fully compensated), which nobody did.  iv) We provided an
optional contact address and feedback form that we closely monitored (we have
not received any complaints).  v) We shared a website (also accessible from
outside of the study), that participants could visit and share to learn more
about login notifications and related account security measures.  vi) We
created a distinct email account for sending the notifications that applied
all state-of-the-art email security features, which can prevent email spoofing
attacks. We also allowed participants to reply to the notification and ask for
assistance. Finally, all email addresses were only stored encrypted, separated
from the study responses, and were deleted after the study in accordance with
GDPR.

###  6.4 Limitations

In addition to the limitations from Study 1, we relied on a controllable
artificial account setting for this study which might lack ecological
validity. However, only 7 participants mention the non-real-world setting as a
reason for not reacting to the notification. We expect more participants to
change their password if the notification was sent for an account with a
higher subjective value. In regard to the tracking pixel, we tested it with
Gmail, Outlook, and Yahoo! and analyzed the web server access log to check if
any automated scans may already trigger the pixel. While it was not the case,
we cannot rule out that this holds for all scenarios.

##  7 Measurement Study: Results

Next, we present the results of the measurement study. We have applied the
same structure, coding methods, and statistical testing as with our self-
report study (see Section 5). Inter-coder agreement for this second study was
high ($\kappa=0.77$). The resulting codebook can be found in Table 9–11 in
Appendix F. In contrast to Section 5, we will report on exact numbers to
provide more detailed insights based on measured user behavior. We will
discuss the similarities and difference between the two study findings and
their methods in Section 8.

###  7.1 RQ1: Comprehension & Reaction

Comprehension When asked why they have received the notification (MQ11), 85%
(93) in the legit and 79% (94) of the participants in the malicious treatment
realized that a new login happened. Very few who gave a different explanation
believed it was a phishing attempt (3; 1%), most simply did not understand
what has happened at all (39; 17%):

  * “ _I had no idea, which is why I deleted it._ ” (M93-N)

Those in the legit treatment who mapped the notification to a new login
usually perceived it as a simple info email (42; 38%), followed by those who
saw it as a prompt to review the login (28; 26%). Fewer responses (15; 13%)
explicitly mention that the login must have been abnormal. In the malicious
treatment, most participants, who understood that a new login happened,
described that they were (potentially) compromised (46; 36%). Another 19% (22)
perceived it as an informative but non-critical email. The remainder (13; 11%
each) either mentioned that the system rated the login as unusual or wants
them to review the login.

Legit |  Not Changed$0$%$20$%$40$%$60$%$80$%$100$%FAEHI
---|---
Malicious |  Not ChangedChanged$0$%$20$%$40$%$60$%$80$%$100$%Don’tRememberWas MeLowValueFeelProtectedSpontaneousDon’tUnderstandNot MeFatigueSuspiciousUnsureFEIAHCDB

Figure 4: Breakdown of participants (measured), into those participants who
have or have not changed their password.

We observed a lower comprehension (compared to the first study) of what might
have caused the notification, especially in the malicious group. One
explanation might be the temporal connection between logging in and receiving
the notification. From MQ6, we know that about two-thirds read it immediately,
most of the others within few hours. Hence, participants in the legit
treatment had indeed a connection, and their understanding was substantially
better. Similarly, we did not observe differences between treatments in the
self-report study, where both had identical circumstances. This influence of
contextual factors was already observed by prior work on warning design [31,
12] and could be achieved by including a _Why Notification_ section. Some
websites already do (see Section 3), and we will elaborate on this in the
discussion.

Reaction General Out of the total 229 participants, 48 participants, 23 in the
legit and 25 in the malicious treatment,  F cannot remember the notification.
Still, 26 of them triggered the tracking pixel, so they must have at least
opened the notification. Among the large majority of participants who saw the
login notification (181; 79%), it was very rare that they completely ignored
its content. In response to MQ1, just 6% said that they only read the subject.
About 90% read the notification completely or at least skimmed the body.

Reaction Legit Unlike the self-report study suggested, where half of the
participants in legit treatment said they would change their password (see
Section 5.1), none in the measurement study actually did. As shown in Figure
4, the majority of participants (60; 55%) explained their reaction in response
to MQ3b by saying  A it was their own login. Another 12, i.e., 11%, described
it as a  E spontaneous reaction, e.g., M42-N: “I just didn’t think much of
it.” Previously, we saw that some participants do not understand what the
notification is saying, which was the driving reason for  H 6% (6) to ignore
it. Finally, we recorded themes aligning with the findings from the self-
report study with participants who were  I suspicious about the legitimacy of
the notification (4; 4%), felt  C fatigued (3; 3%), or  G protected (3; 3%).

Reaction Malicious In the malicious group, only 26 of the 119 participants,
i.e., 22%, changed their password; all of them correctly saying  B it was not
them logging in. The reasons given by the other 78% (93) in response to MQ3b
for not changing their password have mostly been given by participants in the
legit treatment:  E spontaneous reaction (15; 13%), notification looked  I
suspicious (14; 12%), or was  H not understood (8; 7%),  C being fatigued (6;
5%) or  D unsure how to react (6; 5%). Finally, there are two justifications
that are owed to the study design: participants describing they  A logged in
themselves although they did not (11; 9%), likely an example of social
desirability, and those who assigned a low value to the account (7; 6%):

  * “ _This account has no value, it was not a streaming or banking account or amazon account_ ” (M74-N)

This justification can be reasonable, but users need to keep in mind that an
attacker can also target other accounts that verbatim or partially reused the
compromised password [25].

Summary About 80% saw the notification. Participants in the legit treatment
who triggered it themselves understood what it was telling them and reacted
accordingly. In the malicious treatment where participants did not have this
context, only 22% changed their password, and they had more difficulties
explaining the circumstances. Hence, the number of password changes in the
malicious treatment is substantially lower than expected, while we could not
confirm the tendency to unnecessary password changes.

###  7.2 RQ2: Decision-Making & Execution

|  Affected Account Name (i.e., Email Address)
---|---
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
| Location
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
| Date
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
| Device
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$%$20$%$40$%$60$%$80$%$100$%Not at allHelpfulSlightlyHelpfulSomewhatHelpfulModeratelyHelpfulExtremelyHelpful

Figure 5: Helpfulness of the details for deciding (MQ5).

Participants in the self-report study said that the notification tells them
how to resolve the situation. Hence, we did not further test this. Instead, we
will focus on the decision-making process, as participants seem to struggle
with determining whether it was them or not, especially for malicious logins.

Helpfulness of Login Information Foremost, we wanted to get more insights into
the helpfulness of the displayed login information. We know from the self-
report study that information about the device and location are considered by
participants when deciding how to react to a login notification. In Figure 5,
we can see that for those in the Legit and Malicious (Change) group, all
information is about equally helpful: 22–35% find the different types
moderately and 38–62% even extremely helpful. Participants in the Malicious
(No Change) group, in contrast, appear to have a less distinct opinion as
ratings are more equally distributed, ranging from 8–30%. A Kruskal-Wallis
test also showed significant differences for all types of information when
comparing Malicious (No Change) to Legit and Malicious (Change), respectively.
This uncertainty of participants in the Malicious (No Change) group regarding
the displayed information aligns with the previous section, where we found
that those participants misattributed or did not understand the cause of the
notification.

Effect of Other Factors In addition to the already-known influence of the
login information, we were also interested in the effect of other exogenous
and endogenous factors. Figure 6 gives an overview. Generally speaking, the
content (e.g., provided information, instructions, wording) and prior
experience in dealing with such notifications had the highest effect on
participants’ reactions, with 42% expressing a moderate or major effect on
average. Followed by that is the metadata (e.g., sender, subject, time of
arrival) with 29%. All other factors seemed to have a less distinct influence,
with 18% (appeared to be phishing) to 23% (was expected) of the participants
reporting a moderate or major effect.

When comparing the groups, Legit is the one where most participants reported a
factor having no effect. The Malicious (Change) group, on the other hand, is
the one where participants describe the strongest influences of the factors.
Using a Kruskal-Wallis test with Bonferroni-correction for pairwise
comparisons, we found that the metadata had a significantly higher effect for
Malicious (Change) participants compared to Malicious (No Change) participants
($\chi^{2}(2)=6.65,p<0.05$). The same is true for the email content
($\chi^{2}(2)=7.73,p<0.05$). Thus, to nudge more users to change their
password upon receiving potentially malicious login notifications, focusing on
designing the content and metadata is vital.

Influence of Negative Experiences Overall, 30% of participants described
falling victim to a security breach within the last two years (MQ14). In the
malicious treatment, 42% of those who changed their password reported prior
negative experiences. Only 32% of those who did not change their password said
so. The difference is not statistically significant,
$\chi^{2}(2)=2.61,p=0.271$ but suggests that prior breach experience increases
the likelihood of users changing their password upon receiving a notification.

Summary Based on the first study, we assumed the device and location to be
more helpful than the account name and the date. However, when compared side-
by-side, we can conclude that all factors are equally essential. What we did
observe is that the helpfulness of the information for the Malicious (No
Change) participants is significantly lower, which further explains the issues
of this group when determining what happened. Regarding other factors, the
content of the notification, its metadata, and prior experience in dealing
with it had the highest effect across all treatments. Negative experience
tends to influence the reaction as well; other aspects appeared to be less
crucial.

|  Exogenous: Metadata
---|---
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Exogenous: Content
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Exogenous: Design
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Endogenous: Experience with Notifications
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Endogenous: Negative Experience with Incidents
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Endogenous: Appeared to Be Phishing
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$$20$$40$$60$$80$$100$
|  Endogenous: Was Expected
Legit (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (No Change) |  $0$$20$$40$$60$$80$$100$
Malicious (Change) |  $0$%$20$%$40$%$60$%$80$%$100$%NoEffectMinorEffectMediumEffectModerateEffectMajorEffect

Figure 6: Influence of factors on participants reaction (MQ4).

###  7.3 RQ3: Perception & Expectation

Perception In the self-report study, the PANAS revealed that participants who
would change their password felt significantly more positive (M: 15.6, SD:
3.2) but also more negative (M: 12.7, SD: 4.1). Participants who described
ignoring the notification scored the lowest. The second study backs up both
findings. The average positive _affect_ of the Malicious (Change) group is
15.0 (SD: 4.5) but only 11.6 (SD: 5.0) and 12.6 (SD: 5.6) for the Malicious
(No Change) and Legit, respectively. Using a Kruskal-Wallis test (Bonferroni
corrected), we were also able to confirm the significance between the two
malicious groups, $\chi^{2}(2)=8.29,p<0.05$. For the negative _affect_ ,
Malicious (Change) averages 9.8 (SD: 4.1), Malicious (No Change) (M: 8.1, SD:
4.3), and Legit (M: 5.7, SD: 1.6). Again, Kruskal-Wallis was used yielding
significance between both malicious groups and Legit ($p<0.01$); the
comparison between the two malicious groups nearly did,
$\chi^{2}(2)=5.26,p=0.0654$.

Expectation More than 90% of the participants in the self-report study expect
real-world services to send notifications. Participants who expressed
disagreement reported receiving them constantly and being annoyed. Moreover,
the second study showed so far that there is a substantial number of
participants who have not changed their password although they should, some of
them mentioning that it was a spontaneous reaction which this fatigue may also
explain. Hence, we used MQ12 to learn when users expect to receive
notifications.

A majority of participants (151; 66%) expressed they want to receive
notifications after suspicious account activity. On average, 60% want to be
notified if a login takes place from a new device, 47% for logins from a new
location, 31% if they have not logged in for a while, and 22% for logins that
take place at an unusual time of the day. Only 9% want to receive a login for
every login, and even less (9; 3%) do not want to receive login notifications
at all.

Summary We can confirm that participants who changed their password felt both
more positively and negatively, probably because they assumed some form of
compromise but also had a sense of achievement after preventing it by changing
the password. The other groups had lower scores, aligning with them not
expecting any harm. The self-report study showed that participants expect
services to send login notifications. With the new findings, we can further
specify this by saying that participants want to be notified after suspicious
logins, logins from new devices, and logins from new locations. Fewer
participants expect to receive notifications based on temporal deviations.

##  8 Discussion

Next, we discuss the takeaways and give recommendations.

###  8.1 Effectiveness of Login Notifications

Above all, we intend to understand if login notifications, as used by numerous
services nowadays, achieve their goal of increasing the security of users’
accounts. In the case of the malicious logins, we saw that more than 80% in
the self-report study and still about 20% of users in the measurement study
react to the notification by changing their password. Hence, email
notifications can serve as an enhancement to account security, as every
additional password change serves the goal of preventing a malicious actor
from gaining access.

Still, there are also negative aspects that should be mentioned. For example,
one may argue that such notifications are burdening and blaming. If service
providers, with all the information about the login history, are uncertain,
how should the user be able to determine the legitimacy of a login? Of course,
from a service provider’s perspective allowing login and hedging it with a
notification rather than blocking it makes sense. Still, it highlights the
need for robust security measures implemented by service providers themselves
[18].

Contrary to concerns about burdening users, it can be argued that users took
the appropriate action in the legitimate cases. Therefore, users are not
burdened, as they correctly follow the instructions when prompted by a
legitimate login notification. Moreover, users tend to feel more protected and
satisfied when they receive notifications. The reassurance they provide
contributes to a positive user experience and reinforces trust in the service.

Finally, although login notifications are generally beneficial, their
implementation has room for improvement. Further research and development
should focus on refining notification systems to ensure maximum effectiveness
and usability. For example, research on click-through rates of SSL/TLS
warnings increased comprehension and adherence by improving warning design
[8]. One approach to facilitate appropriate reaction may be to match users’
understanding of the login notifications and the actual implementation of
these notifications. As we saw in Sections 5 and 7, participants appear to
expect and need contextual factors to determine what has caused the
notification. Services could adhere to this expectation by including a
distinct _Why Notification_ component in the notification, e.g., by explicitly
saying that a login happened from a previously unseen device. Our initial
analysis only found this contextualizing section in about $20\%$ of the
notifications. Moreover, services need to provide more steps than just
changing the password. Rather, they should initiate a thorough remediation
process, including removing unknown sessions, checking related accounts, or
enabling 2FA [23, 44].

###  8.2 Self-Reported vs. Measured Findings

When we compare the findings of the self-report study with the measured
behavior, we find that participants’ actual reactions differ from the
responses in the hypothetical scenario. This is in line with prior work [30]
and the main reason why we would encourage measuring behavior, particularly
regarding participants’ reactions (RQ1) to the notification. Here, we observed
the most pronounced difference, with fewer participants actually changing
their password than originally anticipated. Prior work suggests that behavior
is more correlated if decisions are either made proactively or because users
are actively prompted for a decision [47]. Hence, using neutral instead of
alarming language in our notification headline and subject can be a
contributing factor. In the limitation sections, we have outlined other
potential reasons for this observation, like social desirability biases.
Smaller are the differences related to comprehension (RQ1) and decision-making
processes (RQ2); identical are those about perceptions and expectations (RQ3).
Here, we see the negative affection for the malicious treatment and the more
neutral affection for the legitimate treatment in both studies.

###  8.3 Expectation vs. Fatigue

More than 90% of the participants expect services to send login notifications,
and finding a balance between sending them too often and too rarely is
crucial. We found that participants want to be notified when a login takes
place from a new device or location, but also if a login appears “suspicious,”
which can be accomplished with advanced logic provided by risk-based
algorithms [48, 49]. Time-related notifications (i.e., a login after a long or
at an unusual time) are less demanded.

For service providers, sending notifications more often than necessary,
following a “better safe than sorry”-mentality may be tempting. Yet, for
users, this leads to a well-documented phenomenon in the area of security
warnings: fatigue [7, 37, 2]. This fatigue is most likely caused by
unnecessary login notifications, i.e., those that do not convey a real risk
teach users that all notifications are unimportant. The situation is further
aggravated by services like Mozilla, Tumblr, Etsy, and others that send
notifications for every single login.

###  8.4 Good Intentions & Questionable Advice

In both studies, about 15% of the participants questioned the legitimacy of
the notification. Prior work explains how to best advise users [32], yet, most
of the 10 notifications that include information about phishing at all do not
follow it.

For example, questionable advice is given by Twitter (and three other major
services), which suggests that the presence of a padlock icon will “let you
know a site is secure” and that users should check for the presence of
“https://” and “{domain}” in the hyperlink. Interestingly, in May 2023, Google
announced removing the padlock icon in Chrome, as HTTPS should nowadays be
considered the default state [1]. Similarly remarkable, Amazon suggests better
copying and pasting the “It wasn’t me”-link into a browser, “just to be safe.”
Spotify advises users to verify the email was sent from “@spotify.com,” which
is only expedient if the email server and DNS are configured correctly.
Related work demonstrated how vulnerable this email ecosystem is [33]. A good
example is some of PayPal’s advice [26] that explicitly mentions to “not rely
on the padlock symbol and the ‘s’ in https.” Interestingly, LinkedIn opted to
add a security footer message [19] to their login notification that includes
the affected user’s name and profession to authenticate official emails.

###  8.5 Recommendations

Below, we give recommendations for service providers.

Include Information in Metadata We found that the metadata is an influential
factor, and 75% of the participants paid attention to the email subject.
Hence, in addition to the most important information, the subject should
already provide context for deciding how to react. Generic clauses, e.g.,
“Your account has been logged into.” as used by Tumblr, do not achieve this,
which is why email subjects like “New login to Twitter from {browser} on {OS}”
are preferable. Similarly, websites should make use of the email sender’s name
so that recipients can quickly parse the information about the sender; in our
initial analysis, we found 5 services that did not.

Describe What Happened What triggers a notification, e.g., an “unusual login”
is often unclear to participants. Services could easily address this issue by
explaining what triggered the notification, yet, only 12% of the evaluated
emails currently provide examples of common triggers. Explaining the
circumstances would also help to create context, which is especially important
when users receive unexpected notifications and struggle to assess the
situation correctly.

Specify Instructions Notifications need to include instructions for both
outcomes, i.e., legit and malicious logins. For the legit case, most services
describe to ignore the message. For malicious logins, the recommendation needs
to prompt users to visit the website and change or reset the password or, even
better, initiate a thorough remediation process. Most services facilitate this
by including a link which is a controversial practice. Of course, it
simplifies the process, and 10% in our study explicitly appreciated the
shortcut. However, also 10% of the participants were suspicious due to the
presence of a link and did not change their password. Hence, services should
carefully evaluate, including a link.

Provide Comprehensible Details We found that all types of information (account
name, location, time, and device) have a positive influence. Still, services
should avoid too technical details. Negative examples are sign-ins on
smartphones displayed using the model number instead of the name, e.g.,
“SM-S908B/DS” instead of “Samsung Galaxy S22.” Operating systems are also
often reported with their full version number, e.g., “iOS 16.0.1.” The least
recommendable example is the raw User-Agent string, which we found in 4
emails.

Respect Privacy Most services employed tracking technologies in their emails
like tracking pixels, URL parameters, and third-party redirects. However,
participants reported being concerned and wondered “what else they’re
tracking.” Adhering to a privacy-by-design concept as suggested by GDPR or the
California Privacy Act would address this concern. Currently, the only
services that do so are those 7 that send plaintext notifications, e.g.,
Cloudflare, GitHub, or Nintendo.

By addressing the identified areas for improvement, service providers can
continue to strengthen account security and foster user trust.

## References

  * [1] David Adrian, Serena Chen, Joe DeBlasio, Emily Stark, and Emanuel von Zezschwitz.  Chromium Blog: An Update on the Lock Icon, May 2023.  https://blog.chromium.org/2023/05/an-update-on-lock-icon.html, as of .
  * [2] Devdatta Akhawe and Adrienne Porter Felt.  Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness.  In USENIX Security Symposium, SSYM ’13, pages 257–272, Washington, District of Columbia, USA, July 2013. USENIX.
  * [3] Hazim Almuhimedi, Adrienne Porter Felt, Robert W. Reeder, and Sunny Consolvo.  Your Reputation Precedes You: History, Reputation, and the Chrome Malware Warning.  In Symposium on Usable Privacy and Security, SOUPS ’14, pages 113–128, Menlo Park, California, USA, July 2014. USENIX.
  * [4] Apple, Inc.  Apple Support Community: Notification of Sign In with Apple ID, January 2022.  https://discussions.apple.com/thread/253581666, as of .
  * [5] Lujo Bauer, Cristian Bravo-Lillo, Lorrie Cranor, and Elli Fragkaki.  Warning Design Guidelines.  Technical Report CMU-CyLab-13-002, Carnegie Mellon University, February 2013.
  * [6] Periwinkle Doerfler, Kurt Thomas, Maija Marincenko, Juri Ranieri, Yu Jiang, Angelika Moscicki, and Damon McCoy.  Evaluating Login Challenges as a Defense Against Account Takeover.  In The World Wide Web Conference, WWW ’19, pages 372–382, San Francisco, California, USA, May 2019. ACM.
  * [7] Serge Egelman, Lorrie Faith Cranor, and Jason Hong.  You’ve Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings.  In ACM Conference on Human Factors in Computing Systems, CHI ’08, pages 1065–1074, Florence, Italy, April 2008. ACM.
  * [8] Adrienne Porter Felt, Alex Ainslie, Robert W. Reeder, Sunny Consolvo, Somas Thyagaraja, Alan Bettes, Helen Harris, and Jeff Grimes.  Improving SSL Warnings: Comprehension and Adherence.  In ACM Conference on Human Factors in Computing Systems, CHI ’15, pages 2893–2902, Seoul, Republic of Korea, April 2015. ACM.
  * [9] Anthony Gavazzi, Ryan Williams, Engin Kirda, Long Lu, Andre King, Andy Davis, and Tim Leek.  A Study of Multi-Factor and Risk-Based Authentication Availability.  In USENIX Security Symposium, SSYM ’23, Anaheim, California, USA, August 2023. USENIX.
  * [10] Maximilian Golla, Grant Ho, Marika Lohmus, Monica Pulluri, and Elissa M. Redmiles.  Driving 2FA Adoption at Scale: Optimizing Two-Factor Authentication Notification Design Patterns.  In USENIX Security Symposium, SSYM ’21, pages 109–126, Virtual Conference, August 2021. USENIX.
  * [11] Maximilian Golla, Miranda Wei, Juliette Hainline, Lydia Filipe, Markus Dürmuth, Elissa Redmiles, and Blase Ur.  “What was that site doing with my Facebook password?” Designing Password-Reuse Notifications.  In ACM Conference on Computer and Communications Security, CCS ’18, pages 1549–1566, Toronto, Ontario, Canada, October 2018. ACM.
  * [12] Peter Leo Gorski, Yasemin Acar, Luigi Lo Iacono, and Sascha Fahl.  Listen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs.  In ACM Conference on Human Factors in Computing Systems, CHI ’20, pages 1–13, Honolulu, Hawaii, USA, April 2020. ACM.
  * [13] Jun Ho Huh, Hyoungshick Kim, Swathi S.V.P. Rayala, Rakesh B. Bobba, and Konstantin Beznosov.  I’m Too Busy to Reset My LinkedIn Password: On the Effectiveness of Password Reset Emails.  In ACM Conference on Human Factors in Computing Systems, CHI ’17, pages 387–391, Denver, Colorado, USA, May 2017. ACM.
  * [14] Mohammed Jubur, Prakash Shrestha, Nitesh Saxena, and Jay Prakash.  Bypassing Push-Based Second Factor and Passwordless Authentication with Human-Indistinguishable Notifications.  In ACM Asia Conference on Computer and Communications Security, ASIA CCS ’21, pages 447–461, Virtual Conference, May 2021. ACM.
  * [15] Ben Kaiser, Jerry Wei, Elena Lucherini, Kevin Lee, J. Nathan Matias, and Jonathan Mayer.  Adapting Security Warnings to Counter Online Disinformation.  In USENIX Security Symposium, SSYM ’21, pages 1163–1180, Virtual Conference, August 2021. USENIX.
  * [16] Leona Lassak, Annika Hildebrandt, Maximilian Golla, and Blase Ur.  “It’s Stored, Hopefully, on an Encrypted Server”: Mitigating Users’ Misconceptions About FIDO2 Biometric WebAuthn.  In USENIX Security Symposium, SSYM ’21, pages 91–108, Virtual Conference, August 2021. USENIX.
  * [17] Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński, and Wouter Joosen.  Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation.  In Symposium on Network and Distributed System Security, NDSS ’19, San Diego, California, USA, February 2019. ISOC.
  * [18] Xu Lin, Panagiotis Ilia, Saumya Solanki, and Jason Polakis.  Phish in Sheep’s Clothing: Exploring the Authentication Pitfalls of Browser Fingerprinting.  In USENIX Security Symposium, SSYM ’22, pages 1651–1668, Boston, Massachusetts, USA, August 2022. USENIX.
  * [19] LinkedIn, Inc.  Security Footer Message in LinkedIn Emails, March 2023.  https://www.linkedin.com/help/linkedin/answer/a1339250, as of .
  * [20] Philipp Markert, Daniel V. Bailey, Maximilian Golla, Markus Dürmuth, and Adam J. Aviv.  On the Security of Smartphone Unlock PINs.  ACM Transactions on Privacy and Security, 24(4):30:1–30:36, September 2021.
  * [21] Philipp Markert, Theodor Schnitzler, Maximilian Golla, and Markus Dürmuth.  “As soon as it’s a risk, I want to require MFA”: How Administrators Configure Risk-based Authentication.  In Symposium on Usable Privacy and Security, SOUPS ’22, pages 483–501, Boston, Massachusetts, USA, August 2022. USENIX.
  * [22] Ariana Mirian, Joe DeBlasio, Stefan Savage, Geoffrey M. Voelker, and Kurt Thomas.  Hack for Hire: Exploring the Emerging Market for Account Hijacking.  In The World Wide Web Conference, WWW ’19, pages 1279–1289, San Francisco, California, USA, May 2019. ACM.
  * [23] Lorenzo Neil, Elijah Bouma-Sims, Evan Lafontaine, Yasemin Acar, and Bradley Reaves.  Investigating Web Service Account Remediation Advice.  In Symposium on Usable Privacy and Security, SOUPS ’21, pages 359–376, Virtual Conference, August 2021. USENIX.
  * [24] Netflix, Inc.  I Received an Email Stating There Was a New Sign-in to My Account, June 2023.  https://help.netflix.com/en/node/100775, as of .
  * [25] Bijeeta Pal, Tal Daniel, Rahul Chatterjee, and Thomas Ristenpart.  Beyond Credential Stuffing: Password Similarity Models using Neural Networks.  In IEEE Symposium on Security and Privacy, SP ’19, pages 866–883, San Francisco, California, USA, May 2019. IEEE.
  * [26] PayPal, Inc.  PayPal Security Center: How to Identify Fake Messages, June 2023.  https://www.paypal.com/us/security/learn-about-fake-messages, as of .
  * [27] Justin Petelka, Yixin Zou, and Florian Schaub.  Put Your Warning Where Your Link Is: Improving and Evaluating Email Phishing Warnings.  In ACM Conference on Human Factors in Computing Systems, CHI ’19, pages 518:1–518:15, Glasgow, Scotland, United Kingdom, May 2019. ACM.
  * [28] Elissa M. Redmiles.  “Should I Worry?” A Cross-Cultural Examination of Account Security Incident Response.  In IEEE Symposium on Security and Privacy, SP ’19, pages 920–934, San Francisco, California, USA, May 2019. IEEE.
  * [29] Elissa M. Redmiles, Everest Liu, and Michelle L. Mazurek.  You Want Me To Do What? A Design Study of Two-Factor Authentication Messages.  In Who Are You?! Adventures in Authentication Workshop, WAY ’17, pages 1–5, Santa Clara, California, USA, August 2017.
  * [30] Elissa M. Redmiles, Ziyun Zhu, Sean Kross, Dhruv Kuchhal, Tudor Dumitras, and Michelle L. Mazurek.  Asking for a Friend: Evaluating Response Biases in Security User Studies.  In ACM Conference on Computer and Communications Security, CCS ’18, pages 1238–1255, Toronto, Ontario, Canada, October 2018. ACM.
  * [31] Robert W. Reeder, Adrienne Porter Felt, Sunny Consolvo, Nathan Malkin, Christopher Thompson, and Serge Egelman.  An Experience Sampling Study of User Reactions to Browser Warnings in the Field.  In ACM Conference on Human Factors in Computing Systems, CHI ’18, pages 512:1–512:13, Montreal, Quebec, Canada, April 2018. ACM.
  * [32] SECUSO Research Group, KIT.  How to Detect Fraudulent and Phishing Messages, March 2022.  https://secuso.aifb.kit.edu/betr-nachrichten-flyer2EN, as of .
  * [33] Kaiwen Shen, Chuhan Wang, Minglei Guo, Xiaofeng Zheng, Chaoyi Lu, Baojun Liu, Yuxuan Zhao, Shuang Hao, Haixin Duan, Qingfeng Pan, and Min Yang.  Weak Links in Authentication Chains: A Large-scale Analysis of Email Sender Spoofing Attacks.  In USENIX Security Symposium, SSYM ’21, pages 3201–3217, Virtual Conference, August 2021. USENIX.
  * [34] Roger N. Shepard and Jacqueline Metzler.  Mental Rotation of Three-Dimensional Objects.  Science, 171(3972):701–703, February 1971.
  * [35] StatCounter.  Desktop Browser Market Share Worldwide – June 2023, June 2023.  https://gs.statcounter.com/browser-market-share/desktop/worldwide, as of .
  * [36] StatCounter.  Desktop Operating System Market Share Worldwide – June 2023, June 2023\.  https://gs.statcounter.com/os-market-share/desktop/worldwide, as of .
  * [37] Joshua Sunshine, Serge Egelman, Hazim Almuhimedi, Neha Atri, and Lorrie Faith Cranor.  Crying Wolf: An Empirical Study of SSL Warning Effectiveness.  In USENIX Security Symposium, SSYM ’09, pages 399–416, San Diego, California, USA, June 2009. USENIX.
  * [38] Kurt Thomas, Jennifer Pullman, Kevin Yeo, Ananth Raghunathan, Patrick Gage Kelley, Luca Invernizzi, Borbala Benko, Tadek Pietraszek, Sarvar Patel, Dan Boneh, and Elie Bursztein.  Protecting Accounts From Credential Stuffing With Password Breach Alerting.  In USENIX Security Symposium, SSYM ’19, pages 1556–1571, Santa Clara, California, USA, August 2019. USENIX.
  * [39] Edmund R. Thompson.  Development and Validation of an Internationally Reliable Short-Form of the Positive and Negative Affect Schedule.  Journal of Cross-Cultural Psychology, 38(2):227–242, March 2007\.
  * [40] U.S. Census Bureau.  U.S. and World Population Clock, June 2023.  https://www.census.gov/popclock/, as of .
  * [41] U.S. Department of Homeland Security.  The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research, August 2012.  https://www.caida.org/publications/papers/2012/menlo_report_actual_formatted/, as of .
  * [42] Steven G. Vandenberg and Allan R. Kuse.  Mental Rotations, a Group Test of Three-Dimensional Spatial Visualization.  Perceptual and Motor Skills, 47(2):599–604, October 1978.
  * [43] Meridel Walkington.  Designing Better Security Warnings, March 2019.  https://blog.mozilla.org/ux/2019/03/designing-better-security-warnings/, as of .
  * [44] Kathryn Walsh, Faiza Tazi, Philipp Markert, and Sanchari Das.  My Account Is Compromised – What Do I Do? Towards an Intercultural Analysis of Account Remediation for Websites.  In Workshop on Inclusive Privacy and Security, WIPS ’21, pages 1–6, Virtual Conference, August 2021.
  * [45] David Wardle.  How Long Does It Take To Get Owned?  Technical Report RHUL–ISG–2019–4, Royal Holloway University of London, March 2019.
  * [46] Rick Wash, Emilee J. Rader, Ruthie Berman, and Zac Wellmer.  Understanding Password Choices: How Frequently Entered Passwords Are Re-Used across Websites.  In Symposium on Usable Privacy and Security, SOUPS ’16, pages 175–188, Denver, Colorado, USA, July 2016. USENIX.
  * [47] Rick Wash, Emilee J. Rader, and Chris Fennell.  Can People Self-Report Security Accurately?: Agreement Between Self-Report and Behavioral Measures.  In ACM Conference on Human Factors in Computing Systems, CHI ’17, pages 2228–2232, Denver, Colorado, USA, May 2017. ACM.
  * [48] Stephan Wiefling, Luigi Lo Iacono, and Markus Dürmuth.  Is This Really You? An Empirical Study on Risk-Based Authentication Applied in the Wild.  In International Conference on ICT Systems Security and Privacy Protection, IFIP SEC ’19, pages 134–148, Lisbon, Portugal, June 2019. IFIP.
  * [49] Stephan Wiefling, Tanvi Patil, Markus Dürmuth, and Luigi Lo Iacono.  Evaluation of Risk-based Re-Authentication Methods.  In International Conference on ICT Systems Security and Privacy Protection, IFIP SEC ’20, pages 280–294, Virtual Conference, September 2020\. IFIP.
  * [50] Yixin Zou, Shawn Danino, Kaiwen Sun, and Florian Schaub.  You ‘Might’ Be Affected: An Empirical Analysis of Readability and Usability Issues in Data Breach Notifications.  In ACM Conference on Human Factors in Computing Systems, CHI ’19, pages 194:1–194:14, Glasgow, Scotland, United Kingdom, May 2019. ACM.

## Appendix

##  Appendix A Self-Report Study: Survey Instrument

Scenario Description

In the following survey, you will be asked to imagine that your name is Jo Doe
(jo.doe@gmail.com). You have an online account with a major company called
AcmeCo which you regularly access with all the devices you own. Imagine that
this account is important to you, and that it is like other accounts you may
have, such as for online shopping or social media.

A Notification from AcmeCo
Imagine, you {recently signed into your AcmeCo account. / have not signed into
your AcmeCo account for a while.}
In your inbox, you see the following email.

Notification

We displayed an interactive mockup email interface with our baseline
notification shown in Figure 2.

I-PANAS-SF
Now we would like to know how you feel in reaction to the email shown on the
left. The list below consists of a number of words that describe different
feelings and emotions. Read each item and then mark the appropriate answer on
the list. Indicate to what extent you feel this way right now, that is, at the
present moment.

| Very slightly |  |  |  |
---|---|---|---|---|---
| or not at all | A little | Moderately | Quite a bit | Extremely
| (1) | (2) | (3) | (4) | (5)
Upset | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Hostile | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Alert | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Ashamed | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Inspired | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Nervous | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Determined | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Attentive | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Afraid | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$
Active | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$

Reaction

  1. SQ1

Please list 3 actions you might take after receiving this email and explain
why you might take those actions.

    1. 1

Action:  Explanation:

    2. 2

Action:  Explanation:

    3. 3

Action:  Explanation:

  2. SQ2

I would feel about receiving this email from AcmeCo.
$\circ$ Not at all concerned $\circ$ Slightly concerned
$\circ$ Somewhat concerned $\circ$ Moderately concerned
$\circ$ Extremely concerned

  3. SQ3

Why?
Answer:

  4. SQ4

For me, taking action in response to this email from AcmeCo would be
$\circ$ Not a priority $\circ$ Low priority $\circ$ Medium priority $\circ$
High priority $\circ$ Very high priority

  5. SQ5

Why?
Answer:

Understanding

  1. SQ6

In your own words, please describe what this email is telling you.
Answer:

  2. SQ7

In your own words, please describe all of the factors that may have caused you
to receive this email.
Answer:

  3. SQ8

I feel that this email from AcmeCo explained to me how to resolve the
situation.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  4. SQ9

Why?
Answer:

Expectation

  1. SQ10

I feel that ignoring this email from AcmeCo would have consequences.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  2. SQ11

Why?
Answer:

  1. SAC

Please select ‘Agree’ as the answer to this question.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  1. SQ12

I think real companies should send emails like this one when necessary.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  2. SQ13

Why?
Answer:

Prior Experience

  1. SQ14

I have received emails similar to this one in the past.
$\circ$ Never $\circ$ A few times $\circ$ Occasionally $\circ$ Many times
$\circ$ Regularly

If participant stated to have received similar notifications in SQ14:

We will now ask you about your personal experiences with similar emails you
have received in the past. Based on these experiences, please answer the
questions below:

  1. SQ15

I  read them.
$\circ$ Never $\circ$ Rarely $\circ$ Sometimes $\circ$ Often $\circ$ Always

  2. SQ16

Why?
Answer:

  3. SQ17

Have you ever received a similar email that helped you to learn about an
unrecognized sign-in? For example, a sign-in that concerned you.
$\circ$ Yes $\circ$ No $\circ$ Don’t know

If participant answered ‘Yes’ in SQ17:

  4. SQ18

Referring to the situation where a similar email helped you to learn about an
unrecognized sign-in. Please describe the situation, how you knew that
something was wrong, and how you solved it.
Answer:

Demography

  1. SD1

Select your age.
$\circ$ 18–24 $\circ$ 25–34 $\circ$ 35–44 $\circ$ 45–54 $\circ$ 55–64 $\circ$
65–74 $\circ$ 75+ $\circ$ Prefer not to answer

  2. SD2

Which of these best describes your current gender identity?
$\circ$ Woman $\circ$ Man $\circ$ Non-binary
$\circ$ Prefer to self-describe:
$\circ$ Prefer not to answer

  3. SD3

What is the highest degree or level of school you have completed?
$\circ$ No schooling completed $\circ$ Some high school, no diploma
$\circ$ High school graduate, diploma, or equivalent $\circ$ Some college
$\circ$ Trade, technical, or vocational training $\circ$ Associate’s degree
$\circ$ Bachelor’s degree $\circ$ Master’s degree $\circ$ Professional degree
$\circ$ Doctorate $\circ$ Prefer not to answer

  4. SD4

Which of the following best describes your educational background or job
field?
$\circ$ I have an education in, or work in, the field of computer science,
computer engineering or IT.
$\circ$ I do not have an education in, nor do I work in, the field of computer
science, computer engineering or IT.
$\circ$ Prefer not to answer

One More Thing
Please indicate if you’ve honestly participated in this survey and followed
instructions completely. You will not be penalized/rejected for indicating
‘No’ but your data may not be included in the analysis:
$\circ$ Yes $\circ$ No

##  Appendix B Measurement Study: Survey Instrument

Stage 1: Enrollment

Participants solved 5x spatial reasoning tests.

Demography

  1. MD1

Select your age.
$\circ$ 18–24 $\circ$ 25–34 $\circ$ 35–44 $\circ$ 45–54 $\circ$ 55–64 $\circ$
65–74 $\circ$ 75+ $\circ$ Prefer not to answer

  2. MD2

Which of these best describes your current gender identity?
$\circ$ Woman $\circ$ Man $\circ$ Non-binary
$\circ$ Prefer to self-describe:
$\circ$ Prefer not to answer

  3. MD3

What is the highest degree or level of school you have completed?
$\circ$ No schooling completed $\circ$ Some high school, no diploma
$\circ$ High school graduate, diploma, or equivalent $\circ$ Some college
$\circ$ Trade, technical, or vocational training $\circ$ Associate’s degree
$\circ$ Bachelor’s degree $\circ$ Master’s degree $\circ$ Professional degree
$\circ$ Doctorate $\circ$ Prefer not to answer

  1. MAC1

Please select ‘Agree’ as the answer to this question.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  1. MD4

Which of the following best describes your educational background or job
field?
$\circ$ I have an education in, or work in, the field of computer science,
computer engineering or IT.
$\circ$ I do not have an education in, nor do I work in, the field of computer
science, computer engineering or IT.
$\circ$ Prefer not to answer

Stage 2: Recall

Participants solved 5x spatial reasoning tests.

Stage 3: Questionnaire
Debriefing
Participants were debriefed and told about the actual purpose of the study.

Email
The individual login notification we sent to the participant is displayed for
later reference (see Figure 2, but re-branded to match the SRS study).

If participant has not changed their password.

  1. MQ0

Do you remember receiving this email?
$\circ$ Yes $\circ$ No

Participants who selected ‘No’ in MQ0 were forwarded to MQ10.

I-PANAS-SF
Now we would like to know how you felt in reaction to the email. The list
below consists of a number of words that describe different feelings and
emotions. Read each item and then mark the appropriate answer on the list.
Indicate to what extent you felt this way when you noticed the email.
We asked I-PANAS-SF from the Self-Report Study again.

Reaction

  1. MQ1

Did you read this email when you received it? (email as shown on the left)
$\circ$ I did not read it at all $\circ$ I only read the subject but not the
body
$\circ$ I read the subject and skimmed the body $\circ$ I fully read it

  1. MQ2a

In reaction to this email, you decided to change your password.
Please describe any other actions you took.
Answer:

  1. MQ3a

Why did you react this way, i.e., change your password and take the other
actions you described.
Answer:

If participant has not changed their password.

  1. MQ2b

What did you do in reaction to it?
Answer:

  1. MQ3b

Why did you react this way?
Answer:

Content & Design

  1. MQ4

How much did the following factors influence your reaction?
Answer choice per item: No effect (1) – Major effect (5).
$\bowtie$ Email metadata (e.g., sender, subject, time of arrival)
$\bowtie$ Email content (e.g., information, instructions, wording)
$\bowtie$ Email design (e.g., structure, color, font size)
$\bowtie$ Experience in dealing with such emails
$\bowtie$ Negative experience with security and privacy incidents
(e.g., data breach, identity theft) $\bowtie$ Email appeared to be phishing
$\bowtie$ Expected to receive such an email
$\bowtie$ Other:
Answer choices were randomly ordered.

  2. MQ5

Please rate how helpful the following information was for deciding how to
react to this email?
Ans. choice per item: Not at all helpful (1) – Extremely helpful (5).
$\bowtie$ Affected account name (i.e., email address)
$\bowtie$ Location $\bowtie$ Date $\bowtie$ Device

Time & Location

  1. MQ6

When did you read the email?
$\circ$ I never read it $\circ$ Immediately after I noticed it
$\circ$ Less than 1 hour after I noticed it $\circ$ A few hours after I
noticed it $\circ$ One day after I noticed it $\circ$ More than one day after
I noticed it $\circ$ I do not remember

If participant has not selected “Never” in MQ6:

  2. MQ7

In which US state have you been when you read the email?
Dropdown with all 50 US states + District of Columbia
If somewhere outside the USA:

  3. MQ8

Where did you read the email?
$\circ$ At home $\circ$ At work $\circ$ On the go
$\circ$ Somewhere else:  $\circ$ I do not remember

  1. MAC2

Please select ‘Agree’ as the answer to this question.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  1. MQ9

In case you received the email at a different location or different time,
would your reaction to it been any different?
$\circ$ Yes $\circ$ No $\circ$ Do not know

If participant selected “Yes” in MQ8:

  2. MQ10

What would you have done differently, if you had received the email at a
different location or different time?
Answer:

Comprehension

  1. MQ11

In your opinion, why have you received this email?
Answer:

Expectation

  1. MQ12

In your opinion, when should real companies send emails like this one? (Select
all that apply)
$\square$ Never $\square$ After every detected sign-in which suggests that
something is suspicious or wrong $\square$ After every detected sign-in when I
have not signed in for a while $\square$ After every detected sign-in from a
new device $\square$ After every detected sign-in at an unusual time of the
day (e.g., in the middle of the night) $\square$ After every detected sign-in
from a new location $\square$ After every detected sign-in $\square$ Other:

If participant selected “Never” in MQ12:

  2. MQ13

In your opinion, why do you think real companies should never send emails like
this one?
Answer:

Prior Experience

  1. MQ14

Have you had any negative experiences with a security or privacy incident
within the last two years (e.g., data breach, identity theft)?
$\circ$ Yes $\circ$ No

  2. MQ15

Regularly changing my password (e.g., every 90 days) increases the security of
my account.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

  3. MQ16

Changing my password after it has been breached increases the security of my
account.
$\circ$ Strongly disagree $\circ$ Disagree $\circ$ Neither agree or disagree
$\circ$ Agree $\circ$ Strongly agree

One More Thing
Please indicate if you’ve honestly participated in this survey and followed
instructions completely. You will not be penalized/rejected for indicating
‘No’ but your data may not be included in the analysis:
$\circ$ Yes $\circ$ No

##  Appendix C Demographics

###  C.1 Self-Report Study

Table 1: Demographics of the self-report study. | Male | Female | Other | Total
---|---|---|---|---
| No. | % | No. | % | No. | % | No. | %
Age | 105 | 48 | 104 | 47 | 11 | 5 | 220 | 100
18–24 | 16 | 7 | 9 | 4 | 2 | 1 | 27 | 12
25–34 | 37 | 17 | 39 | 18 | 6 | 3 | 82 | 37
35–44 | 25 | 11 | 29 | 13 | 1 | 0 | 55 | 25
45–54 | 14 | 6 | 14 | 6 | 1 | 0 | 29 | 13
55–64 | 10 | 5 | 6 | 3 | 0 | 0 | 16 | 7
65–74 | 2 | 1 | 6 | 3 | 0 | 0 | 8 | 4
75+ | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 1
Education | 105 | 48 | 104 | 47 | 11 | 5 | 220 | 100
High School | 14 | 6 | 11 | 5 | 1 | 0 | 26 | 12
Some College | 19 | 9 | 20 | 9 | 2 | 1 | 41 | 19
Trade | 4 | 2 | 5 | 2 | 0 | 0 | 9 | 4
Associate’s | 10 | 5 | 8 | 4 | 2 | 1 | 20 | 9
Bachelor’s | 36 | 16 | 40 | 18 | 2 | 1 | 78 | 35
Master’s | 17 | 8 | 14 | 6 | 2 | 1 | 33 | 15
Professional | 3 | 1 | 2 | 1 | 0 | 0 | 5 | 2
Doctorate | 1 | 0 | 4 | 2 | 1 | 0 | 6 | 3
Background | 105 | 48 | 104 | 47 | 11 | 5 | 220 | 100
Technical | 10 | 5 | 39 | 18 | 1 | 0 | 50 | 23
Non-Technical | 92 | 42 | 61 | 28 | 8 | 4 | 161 | 73
Prefer not to say | 3 | 1 | 4 | 2 | 2 | 1 | 9 | 4

###  C.2 Measurement Study

Table 2: Demographics of the measurement study. | Male | Female | Other | Total
---|---|---|---|---
| No. | % | No. | % | No. | % | No. | %
Age | 149 | 65 | 79 | 34 | 1 | 0 | 229 | 100
18–24 | 4 | 2 | 6 | 3 | 0 | 0 | 10 | 4
25–34 | 17 | 7 | 15 | 7 | 0 | 0 | 32 | 14
35–44 | 27 | 12 | 17 | 7 | 0 | 0 | 44 | 19
45–54 | 24 | 10 | 15 | 7 | 0 | 0 | 39 | 17
55–64 | 35 | 15 | 13 | 6 | 0 | 0 | 48 | 21
65–74 | 31 | 14 | 11 | 5 | 0 | 0 | 42 | 18
75+ | 11 | 5 | 2 | 1 | 1 | 0 | 14 | 6
Education | 149 | 65 | 79 | 34 | 1 | 0 | 229 | 100
High School | 47 | 21 | 31 | 14 | 0 | 0 | 78 | 34
Trade | 39 | 17 | 12 | 5 | 1 | 0 | 52 | 23
Bachelor’s | 34 | 15 | 24 | 10 | 0 | 0 | 58 | 25
Master’s | 23 | 10 | 10 | 4 | 0 | 0 | 33 | 14
Doctorate | 4 | 2 | 2 | 1 | 0 | 0 | 6 | 3
Prefer not to say | 2 | 1 | 0 | 0 | 0 | 0 | 2 | 1
Background | 149 | 65 | 79 | 34 | 1 | 0 | 229 | 100
Technical | 10 | 4 | 25 | 11 | 0 | 0 | 35 | 15
Non-Technical | 134 | 59 | 53 | 23 | 1 | 0 | 188 | 82
Prefer not to say | 5 | 2 | 1 | 0 | 0 | 0 | 6 | 3

##  Appendix D Real-World Notifications: Features

Table 3: Information contained in notifications sent by real-world services.

Rank | Domain | Account Name | Browser | Country | State | City | IP | OS | Time | Time Zone | Instructions Legit | Instructions Malicious
---|---|---|---|---|---|---|---|---|---|---|---|---
1 | google.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑
| workspace.google.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕
2 | facebook.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
6 | microsoft.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
7 | twitter.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑
9 | instagram.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
10 | cloudflare.com | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
13 | apple.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
14 | linkedin.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
15 | netflix.com | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
17 | wikipedia.org | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑
20 | amazon.com | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
25 | yahoo.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑
32 | github.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑
36 | pinterest.com | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑
63 | vk.com | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑
72 | mozilla.org | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑
80 | spotify.com | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
82 | tumblr.com | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑
83 | paypal.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
97 | ebay.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
99 | dropbox.com | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
103 | csdn.net | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑
104 | imdb.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
125 | soundcloud.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
155 | twitch.tv | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
157 | etsy.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑
164 | booking.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
171 | sourceforge.net | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
179 | researchgate.net | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑
180 | oracle.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
186 | slack.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑
206 | weebly.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
236 | samsung.com | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑
322 | grammarly.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
328 | fiverr.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
344 | snapchat.com | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
381 | yelp.com | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
392 | binance.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑
524 | netease.com | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑
541 | gitlab.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
545 | atlassian.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
563 | uber.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
753 | airbnb.com | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
885 | nintendo.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑
924 | xing.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑
1205 | wayfair.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
1327 | deezer.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
1387 | lyft.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑
1413 | battle.net | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑
2476 | 1password.com | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌕
2645 | porkbun.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑
3179 | faceit.com | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑
3605 | plex.tv | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑
4189 | dhl.de | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑
4250 | dashlane.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑 | 🌕 | 🌑
5383 | logmein.com | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌑
8544 | maxmind.com | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
10625 | check24.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑
16460 | myunidays.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌕 | 🌑
16993 | n26.com | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
19535 | neteller.com | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
25667 | splitwise.com | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑
27539 | decathlon.com | 🌕 | 🌑 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌑
31988 | netatmo.com | 🌑 | 🌕 | 🌑 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑 | 🌕 | 🌑 | 🌑
40161 | stacksocial.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌑 | 🌑 | 🌑
48031 | traderepublic.com | 🌑 | 🌕 | 🌕 | 🌕 | 🌕 | 🌕 | 🌑 | 🌕 | 🌕 | 🌕 | 🌑

##  Appendix E Real-World Notifications: Email Metadata

Table 4: Sender, email address, and subject of the notifications sent by real-
world services.

Rank | Domain | Display Name | Email Address | Subject
---|---|---|---|---
1 | google.com | Google |<EMAIL_ADDRESS>| Security alert
| workspace.google.com | Google Workspace Alerts |<EMAIL_ADDRESS>| Alert: Suspicious login
2 | facebook.com | noreply |<EMAIL_ADDRESS>| Did you use Facebook from somewhere new?
6 | microsoft.com | Microsoft account team |<EMAIL_ADDRESS>| Microsoft account unusual sign-in activity
7 | twitter.com | Twitter |<EMAIL_ADDRESS>| New login to Twitter from {browser} on {OS}
9 | instagram.com | Instagram |<EMAIL_ADDRESS>| New login to Instagram from {browser} on {OS}
10 | cloudflare.com | Cloudflare |<EMAIL_ADDRESS>| Your Cloudflare account has been accessed from a new IP Address
13 | apple.com | Apple |<EMAIL_ADDRESS>| Your Apple ID was used to sign in to iCloud on a {device}
14 | linkedin.com | LinkedIn |<EMAIL_ADDRESS>| {Name}, please verify your new device
15 | netflix.com | Netflix |<EMAIL_ADDRESS>| A new device is using your account
17 | wikipedia.org | Wikipedia |<EMAIL_ADDRESS>| Login to Wikipedia as {account name} from a device you have not recently used
20 | amazon.com | amazon.com |<EMAIL_ADDRESS>| amazon.com, action needed: Sign-in
25 | yahoo.com | Yahoo |<EMAIL_ADDRESS>| Unexpected sign-in attempt
32 | github.com | GitHub |<EMAIL_ADDRESS>| [GitHub] Please review this sign in
36 | pinterest.com | Pinterest |<EMAIL_ADDRESS>| New login on your Pinterest account
63 | vk.com | VK |<EMAIL_ADDRESS>| Someone has accessed your account from {OS} through {browser}, {country}
72 | mozilla.org | Firefox Accounts |<EMAIL_ADDRESS>| New sign-in to Firefox
80 | spotify.com | Spotify |<EMAIL_ADDRESS>| New login to Spotify
82 | tumblr.com | Tumblr |<EMAIL_ADDRESS>| Your account has been logged into.
83 | paypal.com |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>| Login from a new device
97 | ebay.com | eBay |<EMAIL_ADDRESS>| A new device is using your account
99 | dropbox.com | Dropbox |<EMAIL_ADDRESS>| We noticed a new sign in to your Dropbox
103 | csdn.net | CSDN |<EMAIL_ADDRESS>| [CSDN] Notification of remote login
104 | imdb.com | imdb.com |<EMAIL_ADDRESS>| imdb.com, action needed: Sign-in
125 | soundcloud.com | SoundCloud Login |<EMAIL_ADDRESS>| SoundCloud sign-in detected from a new device
155 | twitch.tv | Twitch |<EMAIL_ADDRESS>| Your Twitch Account - Successful Log-in
157 | etsy.com | Etsy |<EMAIL_ADDRESS>| {Name}, did you recently sign into Etsy?
164 | booking.com | - |<EMAIL_ADDRESS>| New sign in to your account
171 | sourceforge.net | SourceForge |<EMAIL_ADDRESS>| Foreign login to your SourceForge.net account
179 | researchgate.net | ResearchGate |<EMAIL_ADDRESS>| New login from {browser} on {OS}
180 | oracle.com | Oracle |<EMAIL_ADDRESS>| New Device Login Detected with Your Account
186 | slack.com | Slack |<EMAIL_ADDRESS>| Slack account sign in from a new device
206 | weebly.com | - |<EMAIL_ADDRESS>| New login from {browser} on {OS}
236 | samsung.com | Samsung Account |<EMAIL_ADDRESS>| New sign in to your Samsung account
322 | grammarly.com | Grammarly Security |<EMAIL_ADDRESS>| New Login to Grammarly
328 | fiverr.com | Fiverr |<EMAIL_ADDRESS>| New login on your Fiverr account
344 | snapchat.com | Team Snapchat |<EMAIL_ADDRESS>| New Snapchat Login
381 | yelp.com | Yelp |<EMAIL_ADDRESS>| New login to your Yelp account ({account name})
392 | binance.com | Binance |<EMAIL_ADDRESS>| [Binance] Login Attempted from New IP address {IP} - {time}({timezone})
524 | netease.com | NetEase Account Center |<EMAIL_ADDRESS>| NetEase mailbox account abnormal login reminder
541 | gitlab.com | GitLab |<EMAIL_ADDRESS>| gitlab.com sign-in from new location
545 | atlassian.com | Atlassian |<EMAIL_ADDRESS>| Unusual login attempts on your Atlassian account
563 | uber.com | Uber |<EMAIL_ADDRESS>| New device sign-in
753 | airbnb.com | Airbnb |<EMAIL_ADDRESS>| Account activity: New login from {browser}
885 | nintendo.com | - |<EMAIL_ADDRESS>| [Nintendo Account] New sign-in
924 | xing.com | XING |<EMAIL_ADDRESS>| New login on XING: {browser} {OS}
1205 | wayfair.com | Wayfair |<EMAIL_ADDRESS>| New device sign-in
1327 | deezer.com | - |<EMAIL_ADDRESS>| Access from new PC
1387 | lyft.com | Lyft |<EMAIL_ADDRESS>| New Login
1413 | battle.net | Blizzard Entertainment |<EMAIL_ADDRESS>| Help us keep your Blizzard Account safe with a security check
2476 | 1password.com | 1Password |<EMAIL_ADDRESS>| New 1Password sign-in from {browser}
2645 | porkbun.com | Porkbun Support |<EMAIL_ADDRESS>| porkbun.com $|$ account security notice - successful login
3179 | faceit.com | FACEIT |<EMAIL_ADDRESS>| Login from a new IP
3605 | plex.tv | Plex |<EMAIL_ADDRESS>| New sign-in to your Plex account
4189 | dhl.de | - |<EMAIL_ADDRESS>| Successful login to your DHL account with a new device or browser
4250 | dashlane.com | Dashlane |<EMAIL_ADDRESS>| New device added to Dashlane
5383 | logmein.com | LogMeIn.com Auto-Mailer |<EMAIL_ADDRESS>| LogMeIn Audit Notification - Login from an unfamiliar location
8544 | maxmind.com | - |<EMAIL_ADDRESS>| MaxMind Notification: Unrecognized Device Login
10625 | check24.com | CHECK24 Accounts |<EMAIL_ADDRESS>| New Login to Your Customer Account
16460 | myunidays.com | UNiDAYS |<EMAIL_ADDRESS>| Important: UNiDAYS Log-in Notification
16993 | n26.com | N26 |<EMAIL_ADDRESS>| Action needed: Unusual login to your N26 account
19535 | neteller.com | NETELLER |<EMAIL_ADDRESS>| New device has been detected
25667 | splitwise.com | Splitwise |<EMAIL_ADDRESS>| New sign-in to your Splitwise account
27539 | decathlon.com | DECATHLON Service |<EMAIL_ADDRESS>| DECATHLON: New login to your account
31988 | netatmo.com | Legrand - Netatmo - Bticino |<EMAIL_ADDRESS>| Someone has logged into your account
40161 | stacksocial.com | StackSocial |<EMAIL_ADDRESS>| Account Activity Notification
48031 | traderepublic.com | Trade Republic |<EMAIL_ADDRESS>| Registration from a new device

##  Appendix F Codebook

Table 5: Codebook for SQ1, SQ5, and SQ6 used in Section 5.1 RQ1: Comprehension & Reaction. Code |  Freq. |  Description | Example
---|---|---|---
SQ1: Please list 3 actions you might take after receiving this email and
explain why you might take those actions.
Change PW |  151 |  Participant would change the password. |  _“Someone else got into my account, so I need to change the password right away.”_ (M56-C)
Ignore |  69 |  Participant would ignore the email. |  _“Ignore the email. It was me that signed in.”_ (L78-N)
Contact AcmeCo |  64 |  Participant would contact AcmeCo. |  _“I would try and contact them and get a solution to this urgent problem.”_ (M54-C)
Review Activity |  59 |  Participant would check what has been done with the account. |  _“I would look for strange activity on the account.”_ (M47-C)
Review Details |  37 |  Participant would check the shown login information. |  _“Check location, date, device. If it is my own location, date, and device, then I can safely ignore this message.”_ (L12-N)
Prevent Phishing |  33 |  Participant would check the legitimacy of the notification. |  _“Check the sender of the email. I would want to be sure the email was legitimate.”_ (M16-C)
Change Other PWs |  25 |  Participant would change passwords of other accounts. |  _“Change passwords for other important accounts as well”_ (M87-C)
Check Other Accounts |  22 |  Participant would check the other accounts. |  _“I would look to see if other accounts of mine had been compromised.”_ (M28-C)
Add Additional Security |  19 |  Participant would try to add security measures. |  _“Look for additional security options e.g. 2FA”_ (L8-C)
Ask Family/Friends |  12 |  Participant would ask family and/or friends if they have used the account. |  _“Ask my husband if he signed in.”_ (L42-C)
Delete Account |  10 |  Participant would consider deleting the account. |  _“My account has been taken over, and might as well make a new account and terminate this account.”_ (M7-C)
Log Out (All) Devices |  10 |  Participant would try to end the session of some or all devices. |  _“I would logout all devices if the option is available.”_ (M61-C)
Remember Recent Logins |  9 |  Participant would try to recall past actions. |  _“I will do my best to remember whether I have login recently.”_ (L71-C)
Verify Old PW |  7 |  Participant would log in to check if the password still works. |  _“Check to see if my login still works to see if my password’s been changed.”_ (M16-C)
SQ5: Why? (SQ4: For me, taking action in response to this email from AcmeCo
would be [Priority Level])
Secure Account |  105 |  Participant would want to secure the account. |  _“To make my information secure if it wasn’t me logging in.”_ (M46-C)
It Was Me |  37 |  Participant would know why it was sent. |  _“It wouldn’t be a priority if I recognize the device, data, and location.”_ (L1-N)
Serious Consequences |  31 |  Participant would assume that an attacker could cause serious damage. |  _“It could have serious consequences.”_ (M53-C)
Security Is Important |  11 |  Behaving securely is generally important to the participant. |  _“Protecting my accounts is important”_ (M31-C)
Just Info |  8 |  Participant would see the notification as a heads-up. |  _“I assume it’s simply a courtesy email.”_ (L95-N)
Account Is Important |  9 |  The account is important to the participant. |  _“This account is important to me, so I want to make sure it’s not compromised.”_ (M83-C)
Annoyed by Notification |  4 |  Participant would be annoyed by the notification. |  _“I will not waste my time pursuing something that is not a real security threat to my account.”_ (L83-N)
Identify Damage |  4 |  Participant would want to understand what happened. |  _“I would need to get to the bottom of the notification”_ (L37-C)
SQ6: In your own words, please describe what this email is telling you.
New Login |  157 |  Participant describes that there was a new login. |  _“It’s alerting me that my account has been signed into.”_ (L61-N)
Abnormal Login |  63 |  Participant describes that there was a new login which deviates from previous ones. |  _“The main factor is a security issue associated with an unknown login to my account.”_ (L72-N)
Abnormal Login: Device |  38 |  Participant describes that there was a new login from an unusual device. |  _“There was a sign in to my account not recognized as me or one of my devices”_ (L6-N)
Abnormal Login: Location |  15 |  Participant describes that there was a new login from an unusual location. |  _“someone signed in from California and I’m not normally in that area”_ (M91-N)
Abnormal Login: Cookies |  3 |  Participant describes that there was a new login without cookies. |  _“I just logged in after not logging in for a while, so the cached session was probably expired, and it thought I was a new login.”_ (M87-N)
Abnormal Login: Time |  1 |  Participant describes that there was a new login at an unusual time. |  _“A new sign-in from a different location or a different time signing in.”_ (L91-N)
Table 6: Codebook for SQ7 and SQ11 used in Section 5.1 RQ1: Comprehension & Reaction. Code |  Freq. |  Description | Example
---|---|---|---
SQ7: In your own words, please describe all of the factors that may have
caused you to receive this email.
Hack |  134 |  An unauthorized person logged into the account. |  _“There could have been a breach into my account and jeopardized my security.”_ (M54-C)
New Device |  90 |  Login with a new device. |  _“Signing in on a different device than normally used”_ (L40-N)
It Was Me |  53 |  Participant logged in. |  _“I logged in and triggered the email.”_ (L20-C)
New Location |  32 |  Login from a new location. |  _“Because someone signed into my account from a new location.”_ (M38-C)
Account Inactivity |  12 |  Login after a longer period of inactivity. |  _“I logged in to an account that I haven’t tried in a long time and it wanted to give me notice”_ (M62-N)
Deleted Cookies |  11 |  Previously set cookies were deleted. |  _“I probably have cleared all of my cookies so the site didn’t recognize this device and though it was a new device.”_ (L28-N)
Phishing |  11 |  Notification is a phishing attempt. |  _“It is a phishing email… they probably just got a list of everyone who has AcmeCo emails and sent it out to all of them to try to get passwords.”_ (M25-N)
Unknown Login |  9 |  An unknown login happened |  _“The main factor is a security issue associated with an unknown login to my account.”_ (L72-C)
Shared Account |  9 |  Someone who the account is shared with logged in. |  _“If it wasn’t me, then it might have been a loved one with my account access.”_ (L93-C)
Error |  8 |  An error occurred on AcmeCo’s side. |  _“It could be an error, the email may have been sent to me by a glitch in the system.”_ (M57-C)
Notifications Enabled |  8 |  The participant opted in to receive such notifications. |  _“I have alerts set up to send this notice to me automatically every time my AcmeCo account is accessed (even if it’s me who logged in)”_ (L41-N)
SQ11: Why? (SQ10: I feel that ignoring this email from AcmeCo would have
consequences. [Agreement Level])
Someone Has Access |  115 |  An attacker has access to the account. |  _“Someone potentially may have hacked my account”_ (L6-C)
It Was Me |  37 |  It was the participant logging in. |  _“There would be no consequences of ignoring it if the sign-in was me.”_ (L1-N)
Identity Compromised |  36 |  Own identity could be compromised. |  _“My personal info could be stolen and used for nefarious purposes.”_ (M7-C)
Depends |  27 |  It depends on whether the login was legit or malicious. |  _“It depends on whether this was me or not.”_ (L90-N)
Unclear |  9 |  Participant has no clear opinion on the consequences of ignoring. |  _“There is no telling what is happening.”_ (M94-C)
Compromise Other Accounts |  7 |  Ignoring the notification could also compromise other accounts. |  _“Someone else has my account and can use it to steal other accounts associated with this email.”_ (M11-C)
Lock Out |  7 |  Access to the account could be lost entirely. |  _“I feel that someone could get me locked out of my account.”_ (M28-C)
Just Info |  6 |  Notification is just a heads-up. |  _“It is a courtesy email.”_ (L95-N)
Check Login |  5 |  Login needs to be checked. |  _“I should at least ensure there was no fraud.”_ (L55-C)
Table 7: Codebook for SQ9 and SQ18 used in Section 5.2 RQ2: Decision-Making & Execution. Code |  Freq. |  Description | Example
---|---|---|---
SQ9: Why? (SQ8: I feel that this email from AcmeCo explained to me how to
resolve the situation. [Agreement Level])
Explains Malicious |  133 |  Participant appreciates that instructions for the malicious case are explained. |  _“It gives me the option to change my password.”_ (L41-N)
Explains Both |  53 |  Participant appreciates that instructions for the legit and malicious case are explained. |  _“It explains to ignore the email if it was me, and change my password if it wasn’t.”_ (L101-N)
Missing |  33 |  Participant misses information or advice. |  _“Changing my password is the very first line of defense, but I wish there was more information about what steps to take, or further assistance from AcmeCo.”_ (M26-C)
Provides Link |  22 |  Participant appreciates that a link is provided to change the password. |  _“The steps to take were very clear and easy to follow. It also provided a link to make it easy to change my password if I needed to.”_ (L42-C)
Notification Untrusted |  10 |  Participant describes the notification to be untrustworthy. |  _“It shouldn’t be suggesting that I click the link in the email as this is a trick often used by scammers.”_ (M35-C)
SQ18: Referring to the situation where a similar email helped you to learn
about an unrecognized sign-in.
Please describe the situation, how you knew that something was wrong, and how
you solved it.
Malicious |  134 |  It was not the participant signing in. |  _“I changed my password for my email once because someone got a hold of it somehow.”_ (L42-C)
Location |  68 |  Login took place at an unknown location. |  _“I knew it because the login happened from a place I did not even visit and immediately changed my password.”_ (L43-C)
Location: Continent |  19 |  Login took place on a different continent. |  _“I received a notification that someone from another continent logged into my account and I immediately knew this was not me.”_ (L50-C)
Location: Country |  17 |  Login took place in a different country. |  _“I received an email that someone logged into my account from another country so I changed my password”_ (L69-N)
Location: State |  8 |  Login took place in a different state. |  _“I had a login notification from California when I live in Texas. Immediately I knew someone hacked into my account.”_ (M61-C)
Unexpected |  32 |  Notification came unexpected. |  _“I knew something was wrong because I received a sign-in attempt notification from a website I hadn’t used in a while, and hadn’t even tried signing in to lately.”_ (M95-N)
Device |  17 |  Login was done with an unknown device. |  _“It was my instagram account, I hadn’t logged in on that device they described, I changed my password”_ (M20-C)
Account Sharing |  10 |  Login was done by someone who the account is shared. |  _“My kids sometimes use my accounts - Amazon, Netflix, Disney plus - I ask them to confirm it was them signing in - which is fine.”_ (L89-C)
It Was Me |  10 |  It was actually the participant logging in. |  _“I received a similar email from Amazon about a new sign in. It turned out that I used a new device that it didn’t recognize.”_ (L10-C)
Time |  2 |  Login took place at an unusual time. |  _“I knew it was me because the sign in came at a time, and from a location, which did not at all correlate to my real world location, and I dealt with the situation as described in this study - I changed my password and ensured that 2FA was enabled.”_ (M27-C)
Table 8: Codebook for SQ3, SQ13 and SQ16 used in Section 5.3 RQ3: Perception & Expectation. Code |  Freq. |  Description | Example
---|---|---|---
SQ3: Why? (SQ2: I would feel [Concern Level] about receiving this email from
AcmeCo.)
Someone Has Access |  99 |  The account is accessed by someone else. |  _“Cause someone else is using my account”_ (M60-C)
It Was Me |  41 |  Participant knows why it was sent. |  _“Because I already know that I just signed in”_ (L90-N)
Privacy Invasion |  28 |  Participant mentions the private information which may be saved on the account. |  _“i don’t want someone to have access to my personal info”_ (M39-C)
PW Exposed |  13 |  Participant refers to the risk of an exposed password. |  _“Someone is logging into my account which means they have my login info”_ (M31-C)
Someone Tried |  12 |  Participant focuses on someone trying to sign into the account. |  _“Somebody else tried to log in to my account”_ (L85-C)
Only Information |  8 |  Participant sees the notification as a heads-up. |  _“It seems like a normal security warning”_ (L56-N)
Feeling of Security |  5 |  Participant appreciates the information. |  _“Because I like the added security function and appreciate the notice.”_ (L28-N)
Annoyed |  5 |  Participant is annoyed by the notification. |  _“It’s not a problem. It is just annoying.”_ (L32-N)
Phishing |  4 |  Participant sees the notification as phishing. |  _“I would feel moderately concerned because it is a phishing email.”_ (M25-N)
SQ13: Why? (SQ12:I think real companies should send emails like this one when
necessary. [Agreement Level])
Protect Accounts |  111 |  Notifications are a form of protection. |  _“It’s helpful and protects the persons data.”_ (M45-C)
Alert Users |  95 |  Notifications alert users to unknown logins. |  _“To alert customers about possible security breach”_ (M67-C)
Show Protection |  15 |  Sending notifications shows that companies are security conscious. |  _“It shows that they care about users security.”_ (L22-C)
Annoying |  10 |  Notifications are annoying. |  _“These emails are incredibly annoying.”_ (L90-N)
Quick Action |  9 |  Notifications allow to take a quick action in case something unauthorized happens. |  _“You can immediately stop access to your account from people who exhibit criminal behavior”_ (M91-C)
SQ16: Why? (SQ15: I [Frequency Level] read them.)
Account Security |  110 |  Reading the notification assure that the account is secure. |  _“Because it keeps my accounts safe and protected by monitoring this.”_ (L82-C)
Know Sign-in |  33 |  Reading the notification depends on whether the login is known. |  _“If I signed in from a second device and then received the email, I knew it was me, so I would ignore it. But when I didn’t sign in, I would pay attention and read the email.”_ (L16-N)
Important |  46 |  Reading the notifications is important. |  _“I read them, cos they are important.”_ (M36C)
Check Details |  36 |  Notification is read to check the login details. |  _“Just in case it wasn’t me. I look at the date, location, and time to determine.”_ (L20-N)
Table 9: Codebook for MQ2a, MQ2b, MQ3a, MQ3b, and MQ11 used in Section 7.1 RQ1: Comprehension & Reaction. Code |  Freq. |  Description | Example
---|---|---|---
MQ2a: In reaction to this email, you decided to change your password. Please
describe any other actions you took.
MQ2b: What did you do in reaction to it?
Nothing |  121 |  Participant did nothing. |  _“After I read it, I didn’t do anything as it was me who signed in.”_ (L17-N)