# Discovering an Aid Policy to Minimize Student Evasion Using Offline
Reinforcement Learning

Leandro M. de Lima  Graduate Program in Computer Science, PPGI
Federal University of Espirito Santo, UFES
Vitória, Brazil
<EMAIL_ADDRESS>Renato A. Krohling Graduate Program in Computer
Science, PPGI
Production Engineering Department, LABCIN
Federal University of Espirito Santo, UFES
Vitória, Brazil
<EMAIL_ADDRESS>
###### Abstract

High dropout rates in tertiary education expose a lack of efficiency that
causes frustration of expectations and financial waste. Predicting students at
risk is not enough to avoid student dropout. Usually, an appropriate aid
action must be discovered and applied in the proper time for each student. To
tackle this sequential decision-making problem, we propose a decision support
method to the selection of aid actions for students using offline
reinforcement learning to support decision-makers effectively avoid student
dropout. Additionally, a discretization of student’s state space applying two
different clustering methods is evaluated. Our experiments using logged data
of real students shows, through off-policy evaluation, that the method should
achieve roughly 1.0 to 1.5 times as much cumulative reward as the logged
policy. So, it is feasible to help decision-makers apply appropriate aid
actions and, possibly, reduce student dropout.

©2021 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

Globally the gross enrollment ratio in tertiary education increased from
$19\%$ in 2000 to $38\%$ in 2018 according to Unesco [1], but the expansion of
the higher education system may not result in an increase in graduated
professionals. In 2017, on average $33\%$ of the students fail to successfully
complete the programs they undertake [2]. Especially in developing countries
that have fragile economies, as in Latin America, effectiveness in
transforming tertiary enrollment rates increase into the supply of high-
skilled workers is key for their development. Evasion in tertiary education
rates are important indicators of that efficiency and their results must be
improved.

Student evasion incurs wasted resources, frustrations of expectations, and
loss of personal, professional, and social potential. In public education
institutions, it also represents an onus for society, especially for the
financial waste it entails. Institutional policies and actions are extremely
important to prevent that outcome. Identifying which students are at risk of
evasion and what actions to take for each of them to minimize it is a complex
problem that affects educational institutions. There are several reasons for
students to abandon their undergraduate studies and those reasons, for the
most cases, remain undetected by educational institutions until the moment a
student initiates a transfer request, leave of absence request, or drops out.
Institutions that can identify students with high evasion risk, and manage to
successfully overcome student complains early on, may increase their
graduation rates and contribute, ultimately, to the progress of society as a
whole [3].

###  I-A Related Works

Researches in the educational data mining field mainly focus on
classification, clustering, and visual data mining[4] techniques to solve
educational issues, as predicting students at risk. Most of the previous work
in the student evasion field focuses on predictive tasks [5, 6, 7]. Those
approaches are an important step towards effective monitoring and help those
students, but simply predicting who is at risk is not enough to solve the
problem and minimize evasion. For such cases, it is necessary to identify a
policy that defines the sequence of effective aid actions for each of the
students at risk. However, adequate aid actions at the right moment still
mostly depend on the specialist’s correct decision due to a lack of
appropriate decision support tools. Our approach is similar to [8], but their
work is focused in identify critical decisions in interactive learning
environments proposing a reinforcement learning based framework.

Many deep reinforcement learning (RL) algorithms have shown great results in
sequential decision-making problems and this work is based on those methods to
help decision-makers minimize student evasion. Deep reinforcement learning
algorithms, as Deep Q-Network (DQN)[9], Advantage Actor-Critic (A2C)[9], Trust
Region Policy Optimization (TRPO)[10], Stochastic Lower Bounds Optimization
(SLBO) [11], have shown advantages in learning human behaviors and strategies
in several areas, providing good results in problems with high-dimensional
action and state space. Usually, (online) reinforcement learning typically is
employed in problems that can directly apply actions and observe the
environment to acquire more data. However, data from previous interactions
could have been already collected. Besides that, in some situations, there are
cost, safety, or ethical issues in the usage of artificial intelligence (AI)
decisions without a known AI quality to gather more data, as in health or
educational field. Offline (or batched) reinforcement learning [12, 13] seeks
to overcome these limitations in exploration.

Off-policy reinforcement learning methods, as DQN[9], use data generated by an
exploratory policy to learn an optimal policy [14]. Usually, those methods use
an online sample collection to mitigate distributional shift issues, despite
that, off-policy RL can learn reasonably well in offline problems[15].

As the suggested actions often cannot be evaluated in the environment in
offline problems, off-policy policy evaluation (OPE) can evaluate newly
learned policies without interacting with the environment. OPE is an active
field of research and multiples estimators can be applied, e.g. Sequential
Weighted Doubly-Robust (SWDR) or Model Guided Importance Sampling Combining
(MAGIC) [16].

This work is also related to imitation learning and inverse reinforcement
learning as they also try to learn how to mimic other policies. In imitation,
learning the goal consists of finding a policy that directly mimics the
observed policy, but it also suffers from distributional shift[17]. Inverse
reinforcement learning tries to find a reward function that explains the
policy behavior that generated the data[18]. However, both are inappropriate
approaches as they require either access to an oracle, further online data
collection, or an explicit distinction between expert and non-expert data[13].
Distributional shift also affects the model-based algorithms that suffer from
it in the same way as the value function, since out-of-distribution state-
action tuples can result in inaccurate prediction [15].

The proposed decision support method can not act directly in the environment.
So, its recommendations can be reviewed or changed by the decision-maker
before they are applied. Similar methodologies were applied to sepsis
treatment [19, 20]. Offline reinforcement learning can also be applied in many
other sequential decision-making problems as in healthcare, spoken dialogue
systems, self-driving cars, and robotics[15].

The goal of this work is to propose a decision support method for selection of
aid actions for students based on a deep reinforcement learning approach
aiming to help decision-makers of an educational institution. A method that
recommends actions to aid students at risk of evasion to reduce the
institutional evasion rate. Also, a novel case study is presented to
illustrate the offline RL approach to automatically suggesting actions to aid
those students with an analysis comparing the performance of the application
using different evaluation methods and comparing the impact of different
clustering methods on application performance.

The remainder of this paper is organized as follows. In Section II, key
concepts of student dropout and offline RL are defined. Additionally, we
provide an explanation of how to model the problem as a Markov decision
process (MDP) and define a discrete state space for student dropout data. In
Section III, we propose a method to properly tackle the aforementioned
problem. Next, the experimental results are presented in Section IV. Finally,
Section V summarizes key findings of the experimental results and their
implications. It also presents new directions for further investigation and
future work.

##  II Problem Statement

Student dropout or evasion are referenced here as any student that cannot
complete their initiated studies and will not be awarded an academic degree.
Our approach is based on a program perspective and any reason that caused a
student to not graduating in a program is considered as dropout, e.g.
transfer, withdrawal, poor performance.

This research aims to reduce the dropout rate by providing the most
appropriate help for each student, according to their profile. Also, it is
intended that the decision-maker has a reduction in the choice’s workload,
complexity, and pressure of who needs help, what action to take and when to
apply that action to minimize student evasion.

###  II-A Markov decision process

Time-varying state space can be described as a Markov decision process (MDP).
In student evasion problem, it is not possible to directly observe the
underlying state, therefore, that problem can be more precisely defined as a
partially observable Markov decision process (POMDP). Due to simplification,
we consider the state observation as a full representation of the state, so
the student evasion problem is defined here as a fully-observed MDP.

A MDP[21] is characterized by $M=\\{S,A,P,R\\}$:

  * •

State space $S$. At each step $t$ (e.g. academic term), $s_{t}\in S$ defines a
student state.

  * •

Action space $A$. At each step $t$, an agent takes action $a_{t}\in A$ to
modify the environment. In the student evasion problem, it consists of all
combinations of the existence of a study plan and the types of student aid.

  * •

Transition function $P(s_{t+1}|s_{t},a_{t})$. The probability $P:S\times
A\times S\to\mathbb{R}$ of seeing state $s_{t+1}$ after taking action $a_{t}$
at state $s_{t}$.

  * •

Reward function $R(s_{t},a_{t})\in\mathbb{R}$. A function $R:S\times
A\to\mathbb{R}$ that returns the observed immediate feedback $r_{t}$ after
transition $(s_{t},a_{t})$ at time $t$. Transitions to desired states generate
positive rewards, transitions to undesired states negative ones.

In student evasion context, each student is represented as a trajectory
$H=\\{(s_{1},a_{1},r_{1}),(s_{2},a_{2},r_{2}),\dots,(s_{T},a_{T},r_{T})\\}$,
that is a sequence of state, action and reward tuples $(s_{t},a_{t},r_{t})$
over $T$ academic terms. The agent seeks maximize the expected discounted
return $R_{t}=\sum^{T}_{\tau=t}\gamma^{\tau-t}r_{\tau}$, where
$\gamma\in[0,1]$ is a discount factor that trades-off the importance of
immediate and future rewards.

###  II-B Offline (batch) reinforcement learning

The goal in general reinforcement learning is to learn an optimal policy $\pi$
that observes the current state $s$ and selects an action $a$ which maximizes
the sum of expected rewards. Yet, offline reinforcement learning can be
defined as the task of learning the best possible policy from a fixed set
$\mathcal{D}$ of a priori-known transition samples or as a data-driven
reinforcement learning.

Deployment efficiency measure [22] in RL counts the number of changes in the
data-collection policy during learning, i.e., an offline RL setup corresponds
to a single deployment allowed for learning. Reinforcement learning methods
can be classified through data and interaction perspectives [23]. What is
named here as offline RL can be defined as pure batch algorithms, being
classified as batch (data perspective) and offline (interaction perspective)
methods. We refer to it as offline RL, since the use of a ”batch” in an
iterative learning algorithm can also refer to a method that consumes a batch
of data, updates a model, and then obtains online a different batch, as
opposed to a traditional online learning algorithm, which consumes one sample
at a time [15]. The interaction differences are shown in Fig. 1.

Figure 1: Online and Offline RL interaction difference. An online agent can
perform a selected action and observe its impact in the environment through
state and reward signals. An offline agent passively receives a batch of
logged interactions that consist of actions, states, and rewards signals.

Given that only a fixed (finite) set of samples are available, note that the
objective of offline RL is to learn the best possible policy from the given
data and not an optimal policy, as in general reinforcement learning. Since
there is no possibility of improving exploration, which is the main difficulty
in offline RL. Samples distribution is another challenge that the offline RL
approach needs to tackle. Generally, RL assumes that experiences samples are
from a representative distribution of the environment, but the agent in
offline RL has no control or knowledge of how they are sampled. While we
cannot expect offline RL to discover actions that are better than any action
in logged data, we can expect it to effectively utilize the compositional
structure inherent in any temporal process [15].

Offline RL needs to learn a policy that does something differently, presumably
better, from the behavior pattern observed in the experience samples [15].
Therefore, changes in visited states and actions mean be evaluated on a
distribution different from the training one, which forces offline RL to not
assume that the data is independent and identically distributed (i.i.d.).
Recently, fully offline reinforcement learning methods like Random Ensemble
Mixture (REM) [12], Deep Q-learning from Demonstrations (DQfD) [24],
Bootstrapping Error Accumulation Reduction (BEAR) [25], Batch-Constrained deep
Q-learning (BCQ) [13, 26], and Behavior Regularized Actor Critic (BRAC) [27]
consider different approaches and techniques to surpass those limitations.
Behavior-Regularized Model-ENsemble (BREMEN) [22] is a model-based algorithm
that can be used in a fully offline setup, but its main goal is to be
efficient considering the number of changes in the data-collection policy
needed during learning (deployment efficiency) and sampling efficient by using
a mixed (online and offline) approach. Similar to BREMEN, Advantage Weighted
Actor Critic (AWAC) [28] is focused on effectively fine-tuning using online
experiences after an offline pre-training period.

In deep reinforcement learning, for large or continuous state and action
spaces we can represent the various components of agents, such as policies
$\pi(s,a)$ or state-action values $Q(s,a)$, as an approximation with neural
networks, e.g. Dueling Double Deep Q-Network (Dueling DDQN) [29, 30]. In
Dueling DDQN, the classical DQN [9] is extended using Dueling Networks and
Double Q-learning[31] techniques.

Each policy $\pi$ has a corresponding state-action function
$Q_{\pi}(s,a)=\mathbb{E}_{\pi}[R_{t}|s,a]$, the expected return when following
the policy after taking action $a$ in state $s$, and a state value function
$V_{\pi}(s)=\mathbb{E}_{a\sim\pi(s)}[Q_{\pi}(s,a)]$, that measures how good it
is to be in a particular state $s$. In Double Q-Learning, an approximation of
the deep Q-function $Q_{\pi}(s,a;\theta)=Q_{\pi}^{\theta}(s,a)$ with
parameters $\theta$ is used. However, the value function $Q_{\pi}^{\theta}$ is
updated using the target $Q^{\bar{\theta}}_{\pi}(s^{\prime},a^{\prime})$ as

$y^{\text{DDQN}}_{i}=R_{t+1}+\gamma_{t+1}Q_{\pi}^{\bar{\theta}}(s^{\prime},\underset{a^{\prime}}{\mathrm{arg\,max\,}}Q_{\pi}^{\theta_{i}}(s^{\prime},a^{\prime})),$
(1)

where $\bar{\theta}$ represents the parameters of a fixed and separate target
network. Moreover, $s^{\prime}$ and $a^{\prime}$ are state and action at
$t+1$, respectively.

The parameters of the neural network are optimized by using stochastic
gradient descent to minimize the loss

$\mathcal{L}(\theta)=\|y^{\text{DDQN}}_{i}-Q_{\pi}^{\theta}(s,a)\|^{2}.$ (2)

The gradient of the loss is back-propagated only into the parameters $\theta$
of the online network, which is also used to select actions. Periodically, the
target network parameters $\bar{\theta}$, which is not directly optimized, are
updated as $\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}$, where
$\tau$ is the target update rate.

Dueling Networks represents the value function $V_{\pi}(s)$ and the advantage
function $A_{\pi}(s,a)=Q_{\pi}(s)-V_{\pi}(s)$ as two separated streams with a
single deep model whose output combines the two to produce a state-action
value $Q_{\pi}(s,a)$ [29]. The streams are constructed such that they have the
capability of providing separate estimates of the value and advantage
functions.

The learning algorithm is provided with a static dataset
$\mathcal{D}=\\{(s^{i}_{t},a^{i}_{t},s^{i}_{t+1},r^{i}_{t})\\}_{i=1}^{m}$ as a
set of $m$ transitions and must learn the best policy it can using this
dataset. When training the Q-network, instead of only using the current
experience as prescribed by standard temporal difference learning, the network
is trained by sampling mini-batches of experiences from $\mathcal{D}$
uniformly at random. The sequence of losses thus takes the form

$\mathcal{L}_{i}(\theta_{i})=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{U}(\mathcal{D})}[(y^{\text{DDQN}}_{i}-Q_{\pi}^{\theta_{i}}(s,a;\theta_{i}))^{2}].$
(3)

###  II-C Discrete state space

As some algorithms are not suitable for continuous state space and
simplification, the state space is discretized through clustering. K-means
clustering algorithm [32] can discretize the state space and the number of
clusters can be defined by using the Bayesian information criterion (BIC) or
Akaike information criterion (AIC) [19].

Clustering with the K-means method, and its variants, can only produce convex
clusters[33]. Alternatives can be applied as the OPTICS [34] method, which is
a density-based clustering algorithm suitable for an arbitrary shape of
cluster[35].

In this work, discrete state space is created using X-means[36], a K-means
variant, with BIC, therefore there is no need to manually define a specific
number of clusters. Moreover, another discrete state space is created using
OPTICS for comparison.

###  II-D Off-policy policy evaluation (OPE)

Offline reinforcement learning algorithms are trained with offline data,
however, the agent must perform in real situations. Although, in some
problems, evaluation can be performed through simulation, often the same
restrictions that led to the use of offline RL apply in evaluation. So, it is
important to predict how well the new policy will perform before deploying it.

Off-policy policy evaluation tackles the performance prediction problem
producing an estimate that minimizes some concept of error [16]. Alternatives
to OPE are, for example, crowd-sourced human labeling agent actions [37], an
expert qualitative analysis [20] and policy ranking [27].

Recently, supervised learning has been applied to large and diverse training
datasets available [38]. There are few initiatives to establish an offline RL
benchmark as [39, 27, 40], but the field still lacks realistic evaluation
protocols [39].

Off-policy policy evaluation is an active field of research and multiples
estimators can be applied. Two of them are Sequential Weighted Doubly-Robust
(SWDR) and Model and Guided Importance Sampling Combining (MAGIC) [16]
estimators to evaluate and compare offline RL methodologies. Both are Doubly-
Robust (DR) based methods specifically designed for evaluating policies on RL
problems where the horizon of episodes is longer than one. Besides the low
bias advantage inherited from the DR method [41], if either action
propensities or the reward function is accurate, those estimators balance the
bias-variance trade-off while maintaining asymptotic consistency [16].

For a description of MAGIC and SWDR methods for evaluation, let an approximate
model of an MDP, $\widehat{r}^{\pi}(s,a,t)$ denotes the model’s prediction of
the reward $t$ steps later, $R_{t}$, if $S_{0}=s$, $A_{0}=a$, and the policy
$\pi$ is used to generate the subsequent actions. Let

$\widehat{r}^{\pi}(s,t)=\displaystyle\sum_{a\in\mathcal{A}}\pi(a|s)\widehat{r}^{\pi}(s,a,t)$

be a prediction of $R_{t}$ if $S_{0}=s$, $A_{0}=a$, and the policy $\pi$ is
used to generate actions $A_{0},A_{1},\dots$, for all
$(s,t,\pi)\in\mathcal{S}\times\mathbb{N}_{\geq 0}\times\Pi$. We can also
define the estimated state value $\widehat{v}^{\pi}(s)$ and the estimate
state-action value $\widehat{q}^{\pi}(s)$ as, respectively,

$\widehat{v}^{\pi}(s)=\sum^{\infty}_{t=0}\gamma\widehat{r}^{\pi}(s,t),$

$\widehat{q}^{\pi}(s)=\sum^{\infty}_{t=0}\gamma\widehat{r}^{\pi}(s,a,t).$

Given a historical dataset $D=\\{H_{i}|\pi_{i}\\}^{n}_{i=1}$ as a set of $n$
trajectories and known policies, called behavior policies, that generated
them. A partial importance sampling estimator called off-policy $j$-step
return[16], $g^{(j)}(D)$, which uses an importance sampling based method to
predict the outcome by using an evaluated policy $\pi_{e}$ up to $R_{j}$ is
generated, and the approximate model estimator to predict the outcomes
thereafter is proposed in[16]. That is, for all $j\in\mathbb{N}_{\geq-1}$ and
using Weighted doubly-robust (WDR) method, let

$\begin{split}g^{(j)}(D)=\displaystyle\sum^{n}_{i=1}\displaystyle\sum^{j}_{t=0}\gamma^{t}w^{i}_{t}R^{H_{i}}_{t}+\displaystyle\displaystyle\sum^{n}_{i=1}\gamma^{j+1}w^{i}_{j}\widehat{v}^{\pi_{e}}(S^{H_{i}}_{j+1})\\\
-\displaystyle\sum^{n}_{i=1}\displaystyle\sum^{j}_{t=0}\gamma^{t}(w^{i}_{t}\widehat{q}^{\pi_{e}}(S^{H_{i}}_{t},A^{H_{i}}_{t})-w^{i}_{t-1}\widehat{v}^{\pi_{e}}(S^{H_{i}}_{t})),\end{split}$
(4)

where $n=|D|$, $S^{H}_{t}$ denotes the state at time $t$ during trajectory $H$
and $w^{i}_{j}$ is the weighted importance sampling

$w^{i}_{j}=\displaystyle\frac{\rho^{i}_{t}}{\displaystyle\sum^{n}_{j=1}\rho^{j}_{t}}$

and
$\rho_{t}(H,\pi_{e},\pi_{b})=\prod^{t}_{i=0}\frac{\pi_{e}(A^{H}_{i}|S^{H}_{i})}{\pi_{b}(A^{H}_{i}|S^{H}_{i})}$,
is an importance weight, which is the probability of the first $t$ steps of
$H$ under the evaluation policy $\pi_{e}$ divided by its probability under the
behavior policy $\pi_{b}$. For brevity, $\rho_{t}(H_{i},\pi_{e},\pi_{i})$ is
written here as $\rho^{i}_{t}$.

Notice that $g^{(-1)}(D)$ is a purely model-based estimator, $g^{(\infty)}(D)$
is the WDR estimator, and the other off-policy $j$-step returns are partial
WDR estimators that blend between these two extremes. So, the WDR estimator
can be defined as

$\text{WDR}(D)=g^{(\infty)}(D)=\displaystyle\lim_{j\to\infty}g^{(j)}(D).$

A Blending Importance Sampling and Model (BIM) estimator is defined as
$BIM(D)=\mathbf{x}^{\top}g(D)$, where
$\mathbf{x}=(x_{-1},x_{0},x_{1},\dots)^{\top}$ is a weight vector and
$g(D)=(g^{(-1)}(D),g^{(0)}(D),\dots)^{\top}$. So, we estimate $\mathbf{x}^{*}$
by minimizing an approximation of $MSE(\mathbf{x}^{\top}g(D),v(\pi_{e}))$. For
the approximation, we use a subset of the returns, $\\{g^{(j)}(D)\\}$, for
$j\in\mathcal{J}$, where $|\mathcal{J}|<\infty$. For all $j\notin\mathcal{J}$,
we assign $\mathbf{x}_{j}=0$. We also always include $-1$ and $\infty$ in
$\mathcal{J}$. Let $\mathbf{g}_{\mathcal{J}}(D)\in\mathbb{R}^{|\mathcal{J}|}$
be the elements of $\mathbf{g}(D)$ whose indexes are in $\mathcal{J}$, the
returns that will not necessarily be given weights of zero. Also let
$\mathcal{J}_{j}$ denote the $j$th element in $\mathcal{J}$. Before redefining
the BIM estimator, we need to introduce the bias approximation
$\widehat{\mathbf{b}}_{n}$ and the covariance approximation
$\widehat{\Omega}_{n}$, when there are $n$ trajectories in $D$.

After computing the percentile bootstrap $10\%$ confidence interval, $[l,u]$,
for the mean of $g^{(\infty)}(D)$, which we ensure includes $\text{WDR}(D)$,
the bias approximation is defined as

$\widehat{\mathbf{b}}_{n}(j)\leftarrow\begin{cases}g^{(\mathcal{J}_{i})}(D)-u,&\quad\text{if
}g^{(\mathcal{J}_{i})}(D)>u\\\ g^{(\mathcal{J}_{i})}(D)-l,&\quad\text{if
}g^{(\mathcal{J}_{i})}(D)<l\\\ 0,&\quad\text{ otherwise.}\end{cases}$

The covariance approximation $\widehat{\Omega}_{n}$ is defined as

$\begin{split}\widehat{\Omega}_{n}(i,j)=\displaystyle\frac{n}{n-1}\displaystyle\sum^{n}_{k=1}(g^{(\mathcal{J}_{i})}_{k}(D)-\overline{g}^{(\mathcal{J}_{i})}_{k}(D))\\\
\times(g^{(\mathcal{J}_{j})}_{k}(D)-\overline{g}^{(\mathcal{J}_{j})}_{k}(D)),\end{split}$
(5)

where

$\overline{g}^{(\mathcal{J}_{i})}_{k}(D)=\displaystyle\frac{1}{n}\displaystyle\sum^{n}_{k=1}g^{(\mathcal{J}_{i})}_{k}(D).$
(6)

That approximation can be summarized and redefine BIM estimator as

$\text{BIM}(D,\widehat{\Omega}_{n},\widehat{\mathbf{b}}_{n})=(\widehat{\mathbf{x}}^{*})^{\top}\mathbf{g}_{\mathcal{J}}(D),$

where

$\widehat{\mathbf{x}}^{*}\in\underset{\mathbf{x}\in\Delta^{|\mathcal{J}|}}{\mathrm{arg\,min\,}}\mathbf{x}^{\top}[\widehat{\Omega_{n}}+\widehat{\mathbf{b}}_{n}\widehat{\mathbf{b}}^{\top}_{n}]\mathbf{x}.$

Both estimators are designed to evaluate policies acting sequentially and are
step-wise estimators. In SWDR a low bias can be achieved if either action
propensities or the reward function is accurate. It uses weighted importance
sampling and balances bias-variance trade-off while maintaining asymptotic
consistency [42]. MAGIC also balances bias-variance trade-off using SWDR
combined with a purely model-based estimator (blending importance sampling and
model (BIM)) [16].

##  III Methodology

This work uses the student evasion data from undergraduate students who
started and completed the course between $2008$ and $2018$ at the Federal
University of Espírito Santo (Ufes). The data consists of pseudo-anonymous
academic, social, and demographic information presented in $13150$ observed
semesters from a total of $1342$ students. Each observed semester is
represented by the subjects taken in the period and has a total of $37$
features. Each student is represented by a sequence of academic terms. At the
end of this sequence, the way of evasion is known and determined the student’s
outcomes, i.e, success or not.

An initial state space is defined as a $10$ continuous dimension state,
consisting only of academics information and after a manual aggregation of all
courses taken in each term, as listed in Table I. After that, a discretization
of the state features was used to represent the state space, where the
discrete state space is the identifier of the cluster to which it belongs. So,
it is a simplified one-dimension discrete state space defined through the
clustering of those $10$ initial features. Two clustering algorithms are
evaluated in this work by creating different discrete state spaces using
X-means and OPTICS.

TABLE I: State features Feature |  Description
---|---
CH CURSO |  Total major’s course hours
NUM PERIODOS SUGERIDO |  Major’s suggested terms
NUM MAX PERIODOS |  Major’s maximum allowed terms
MEDIA FINAL mean |  Mean of aggregated grades of all courses in the term
MEDIA FINAL std |  Standard deviation of aggregated grades of all courses in the term
CH DISCIPLINA mean |  Mean of aggregated hours per course of all courses in the term
CH DISCIPLINA std |  Standard deviation of aggregated hours per course of all courses in the term
NUM FALTAS mean |  Mean of aggregated student’s frequency of all courses in the term
NUM FALTAS std |  Standard deviation of aggregated student’s frequency of all courses in the term
COD DISCIPLINA count |  Number of completed courses

The discretized states are represented in Fig. 2, where the cluster centroids
are expressed through a reduction of dimensionality to 3D using PCA. The
frequency at which a given state occurs in the dataset is represented by the
size of the circle, the higher the more frequent. The occurrence of
trajectories that end in evasion is highlighted by the color of each circle,
according to the scale in the percentile described in the sidebar of the
figure. A spontaneous gradient in the dropout rate supports the validity of
this state representation as it indicates that, in fact, there is an
association between state membership and dropout rate. The OPTICS algorithm
does not produce a clustering explicitly [34], so, it does not define a center
for each cluster and it is not possible to present a similar representation of
its states.

Figure 2: State clusterization using X-means. Circle size represents state
frequency in the dataset. Circle color represents the dropout rate in that
state.

A one-dimension discrete action state describes actions deployed to the
student in that term. The action space consists of all possible combinations
of two action features, which are the existence of a mandatory supervised
study plan for that student ($2$ options) and the types of available student
aid ($5$ options). In that manner, each action is represented by an index of
all $10$ possibilities. In Ufes dataset only one action is possible per action
type at each semester.

The reward function is a sparse function that returns non-zero signals only if
the current state is the last state in the student trajectory and the student
has a successful outcome. It is defined as:

$R(s_{t},a_{t})=\begin{cases}1,&\text{if }s_{t}\text{ is a terminal
academic}\\\ &\quad\quad\text{successful state}\\\ 0,&\text{otherwise.}\\\
\end{cases}$ (7)

To create a new policy from the logged policy in the dataset, we used the
Dueling Double DQN [30] algorithm. This off-policy reinforcement learning
algorithm can be setup to work online or offline. We deployed it in offline
setup as needed for the aforementioned problem according to Algorithm 1. The
performance of the newly learned policies is evaluated by off-policy policy
evaluation of the reward. For that, we used MAGIC and SWDR methods described
by Algorithm 2 and Algorithm 3, respectively.

Algorithm 1 Dueling Double DQN[30]

0:

  * •

$\mathcal{D}$ \- dataset of transitions;

  * •

$\theta$ \- initial network parameters,

  * •

$\tau$ \- target update rate;

  * •

$N_{b}$ \- training batch size;

  * •

$\bar{N}$ \- target network replacement frequency.

1: $\bar{\theta}\leftarrow\theta$

2: for $t\in\\{0,1,\dots\\}$ do

3: Sample a minibatch of $N_{b}$ tuples
$(s,a,r,s^{\prime})\sim\mathcal{U}(\mathcal{D})$.

4: Construct target values, one for each of the $N_{b}$ tuples.

5:
$a^{max}(s^{\prime};{\bar{\theta}})=\underset{a^{\prime}}{\mathrm{arg\,max\,}}Q_{\pi}^{\bar{\theta}}(s^{\prime},a^{\prime})$

6: if $s^{\prime}$ is terminal then

7: $y_{j}=r$

8: else

9:
$y_{j}=R_{t+1}+\gamma_{t+1}Q_{\pi}^{\theta}(s^{\prime},a^{max}(s^{\prime};{\bar{\theta}}))$

10: end if

11: Do a gradient descent step according to Eq. 3 loss every $\bar{N}$ steps.

12: Update target parameters
$\bar{\theta}\leftarrow\tau\theta+(1-\tau)\bar{\theta}$ every $\bar{N}$ steps.

13: Calculate OPEs according to Algorithm 2 and Algorithm 3.

14: end for

15: return Newly learned policy $\pi$

Algorithm 2 Model and Guided Importance Sampling Combining (MAGIC) [16]

0:

  * •

$D$: Historical data;

  * •

$pi_{e}$: Evaluation policy;

  * •

Approximate model that allows for computation of $\hat{r}^{\pi_{e}}(s,a,t)$;

  * •

$\mathcal{J}$: The set of return lengths to consider;

  * •

$k$: The number of bootstrap resampling.

1: Calculate $\widehat{\Omega}_{n}$ according to Eq. 5.

2: Allocate $D_{(.)}$ so that for all $i\in\\{1,\dots,k\\},D_{i}$ can hold $n$
trajectories.

3: for $i\in\\{1,\dots k\\}$ do

4: Load $D_{i}$ with $n$ uniform random samples drawn from $D$ with
replacement.

5: end for

6: $\mathbf{v}=sort(g^{(\infty)}(D_{(.)})$

7: $l\leftarrow\min\\{\text{WDR}(D),\mathbf{v}(\lfloor 0.05n\rfloor)\\}$

8: $u\leftarrow\max\\{\text{WDR}(D),\mathbf{v}(\lceil 0.5n\rceil)\\}$

9: for $j\in\\{1,\dots|\mathcal{J}|\\}$ do

10:
$\widehat{\mathbf{b}}_{n}(j)\leftarrow\begin{cases}g^{(\mathcal{J}_{i})}(D)-u,&\quad\text{if
}g^{(\mathcal{J}_{i})}(D)>u\\\ g^{(\mathcal{J}_{i})}(D)-l&\quad\text{if
}g^{(\mathcal{J}_{i})}(D)<l\\\ 0,&\quad\text{ otherwise.}\end{cases}$

11: end for

12:
$\mathbf{x}\leftarrow\underset{\mathbf{x}\in\Delta^{|\mathcal{J}|}}{\mathrm{arg\,min\,}}\mathbf{x}^{\top}[\widehat{\Omega_{n}}+\widehat{\mathbf{b}}_{n}\widehat{\mathbf{b}}^{\top}_{n}]\mathbf{x}$

13: return $\mathbf{x}^{\top}\mathbf{g}_{\mathcal{J}}(D)$

Algorithm 3 Sequential weighted doubly-robust (SWDR) [16]

0:

  * •

$D$: Historical data;

  * •

$\pi_{e}$: Evaluation policy;

  * •

Approximate model that allows for computation of $\hat{r}^{\pi_{e}}(s,a,t)$;

  * •

$k$: The number of bootstrap resampling.

1: $\mathcal{J}=\\{\infty\\}$

2: return $\text{MAGIC}(D,\pi_{e},\hat{r}^{\pi_{e}}(s,a,t),\mathcal{J},k)$

##  IV Experimental Results

In our tests, for each clustering method, we defined $3$ different discrete
datasets. Then, we performed one trial for each discrete dataset. We also have
performed a limited tuning starting with the values used in the paper [30].
The offline setup of the Dueling Double DQN [30] algorithm was used to create
a new policy from the logged policy. The Q-network is a fully-connected layer
of size $128$ followed by two parallel fully-connected layers of size $32$,
creating a dueling architecture. The network is trained for $25$ epochs with
minibatch size $512$. The optimizer used was ADAM with a learning rate of
$0.01$ and a learning rate decay of $0.999$. The target update rate is $0.1$
and the discount factor is $0.99$. All experiments run in a computer with the
following specifications: i5-8265U processor, 8 GB DDR4 RAM system memory, $1$
TB HD (SATA 3.1 $5400$ RPM) and $240$ GB SSD (PCIe NVMe Ger 3.0 x2) storage.
Each discretization took $127.945$ seconds on average and each trial policy
training took $38.117$ seconds on average.

In Fig. 3a and Fig. 3b there is the OPE cumulative reward score of the
policies using X-means and OPTICS, respectively, to discretize state space.
Both show a solid line as the average of the performances along with the
training steps and the translucent error bands as the $95\%$ confidence range.
The OPE cumulative reward score (value axis) on the graph represents how many
times the performance of the new policy represents the performance of the
logged policy. Therefore, the dashed line ($value=1$) represents the point
where the learned policy is equivalent to the logged policy, values above it
mean that the new policy performs better and values below represent worse
performance.

(a)

(b)

Figure 3: OPE cumulative reward score (value axis), mean (solid line) and 95%
confidence (translucent error bands) over all trials, using X-means
LABEL:sub@fig_CPE_reward_xmeans and OPTICS LABEL:sub@fig_CPE_reward_optics to
discretize the state space. Notice that the RL model should achieve roughly
$1.0$ to $1.5$ times as much cumulative reward as the logged policy in both
scenarios.

In both figures, the SWDR estimator shows a value close to $1.5$, which is
$50\%$ of improvement over logged policy, with the OPTICS method having a
narrower $95\%$ confidence. However, the MAGIC estimator shows that the
learned policies performed approximately equivalent to the logged policy. It
also means that there was possible to learn a policy using both clustering
methods to discretize the state space. Despite that, there is no guarantee of
the effectiveness of the learned policies due to overestimation. There is the
possibility of the occurrence of distributional shift, inducing overestimation
if the learned policy does not stay close to the behavior policy [43].
Difficulties in properly estimate the policy performance before deploying it
is a major issue in applying offline RL in real-world scenarios[15]. That can
explain the expressive difference between estimators.

A comparison between the actions taken by the new policies and the logged
policy is shown in Fig. 4a and Fig. 4b, using X-means and OPTICS for
clustering, respectively. Logged policy action frequencies are expressed as
the mean of the logged actions occurrence evaluated in all seeds. Note that
all learned policies focus on using almost only two actions, which may
indicate that other actions do not have an impact on improving the reward or,
due to their low occurrence (low exploration), it was not possible to learn
their proper use.

(a)

(b)

Figure 4: Count of occurrences of each possible action in evaluation set after
training using X-means LABEL:sub@fig_actions_bar_xmeans and OPTICS
LABEL:sub@fig_actions_bar_optics clustering. Logged policy actions frequency
is expressed as the mean of the logged actions occurrence evaluated in all
seeds.

##  V Conclusion

Recent developments in the machine learning field provide new ways to address
problems that are still present, such as student dropout. In this paper, a
methodology is proposed to reduce student dropout through the use of a
decision support method to select appropriate aid actions using offline
reinforcement learning.

Our analyzes were performed using real data from students at a Brazilian
university and show promising results in terms of the ability to produce an AI
policy, at least equivalent to the logged one, as it should achieve roughly
$1.0$ to $1.5$ times as much cumulative reward as the logged policy. However,
due to overestimation in offline RL when the learned policy loses similarity
to the policy that generated the dataset, there is no guarantee of the learned
policy effectiveness. Selecting algorithms that encourage policies to stay
close to the behavior policy can reduce that problem.

The proposed discretization of the state space seems suitable to learn a
policy in offline RL context. This work also makes a comparison of the impact
of two different clustering methods for that discretization on the result,
showing similar performance.

For future work, an investigation to identify features that are the most
informative using PCA analysis and which features are most suitable for the
state space and the possibility of a continuous state space approach might
improve performance. Providing interpretability for agent’s decisions through
explainable reinforcement learning and understanding their impacts seems
promising. A further investigation in the cases in which students were able to
avoid dropouts and clarify what actions and spans are appropriate can be
insightful for decision-makers.

## References

  * [1] Unesco, “Education: gross enrolment ratio by level of education.” [Online]. Available: http://data.uis.unesco.org/index.aspx?queryid=142
  * [2] OECD, _Education at a glance 2019_. Paris: OECD Publishing, 2019.
  * [3] R. Balaniuk, H. A. do Prado, R. da Veiga Guadagnin, E. Ferneda, and P. R. Cobbe, “Predicting evasion candidates in higher education institutions,” in _International Conference on Model and Data Engineering_. Springer, 2011, pp. 143–151.
  * [4] H. Aldowah, H. Al-Samarraie, and W. M. Fauzy, “Educational data mining and learning analytics for 21st century higher education: A review and synthesis,” _Telematics and Informatics_ , vol. 37, pp. 13–49, 2019.
  * [5] A. Sales, L. Balby, and A. Cajueiro, “Exploiting academic records for predicting student drop out: a case study in Brazilian higher education,” _Journal of Information and Data Management_ , vol. 7, no. 2, pp. 166–166, 2016.
  * [6] L. Kemper, G. Vorhoff, and B. U. Wigger, “Predicting student dropout: a machine learning approach,” _European Journal of Higher Education_ , vol. 10, no. 1, pp. 28–47, 2020.
  * [7] M. Alban and D. Mauricio, “Predicting university dropout through data mining: a systematic literature,” _Indian Journal of Science and Technology_ , vol. 12, no. 4, pp. 1–12, 2019.
  * [8] S. Ju, G. Zhou, T. Barnes, and M. Chi, “Pick the moment: identifying critical pedagogical decisions using long-short term rewards,” in _Proceedings of International Conference on Educational Data Mining_ , 2020.
  * [9] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [10] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International Conference on Machine Learning_ , 2015, pp. 1889–1897.
  * [11] Y. Luo, H. Xu, Y. Li, Y. Tian, T. Darrell, and T. Ma, “Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees,” _arXiv preprint arXiv:1807.03858_ , 2018.
  * [12] R. Agarwal, D. Schuurmans, and M. Norouzi, “An optimistic perspective on offline reinforcement learning,” _arXiv preprint arXiv:1907.04543_ , 2019\.
  * [13] S. Fujimoto, E. Conti, M. Ghavamzadeh, and J. Pineau, “Benchmarking batch deep reinforcement learning algorithms,” _arXiv preprint arXiv:1910.01708v1_ , 2019.
  * [14] R. S. Sutton and A. G. Barto, _Reinforcement learning: an introduction_. MIT press, 2018.
  * [15] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [16] P. Thomas and E. Brunskill, “Data-efficient off-policy policy evaluation for reinforcement learning,” in _International Conference on Machine Learning_ , 2016, pp. 2139–2148.
  * [17] S. Reddy, A. D. Dragan, and S. Levine, “SQIL: imitation learning via reinforcement learning with sparse rewards,” _arXiv preprint arXiv:1905.11108_ , 2019.
  * [18] L. Luceri, S. Giordano, and E. Ferrara, “Detecting troll behavior via inverse reinforcement learning: a case study of Russian trolls in the 2016 US election,” in _Proceedings of the International AAAI Conference on Web and Social Media_ , vol. 14, 2020, pp. 417–427.
  * [19] M. Komorowski, L. A. Celi, O. Badawi, A. C. Gordon, and A. A. Faisal, “The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care,” _Nature medicine_ , vol. 24, no. 11, pp. 1716–1720, 2018.
  * [20] A. Raghu, M. Komorowski, I. Ahmed, L. Celi, P. Szolovits, and M. Ghassemi, “Deep reinforcement learning for sepsis treatment,” _arXiv preprint arXiv:1711.09602_ , 2017.
  * [21] M. L. Puterman, “Markov decision processes,” _Handbooks in operations research and management science_ , vol. 2, pp. 331–434, 1990.
  * [22] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu, “Deployment-efficient reinforcement learning via model-based offline optimization,” _arXiv preprint arXiv:2006.03647_ , 2020.
  * [23] S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” in _Reinforcement learning_. Springer, 2012, pp. 45–73.
  * [24] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband _et al._ , “Deep Q-learning from demonstrations,” in _AAAI Conference on Artificial Intelligence_ , 2018.
  * [25] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, “Stabilizing off-policy Q-learning via bootstrapping error reduction,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 11 784–11 794.
  * [26] S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” _arXiv preprint arXiv:1812.02900_ , 2018.
  * [27] Y. Wu, G. Tucker, and O. Nachum, “Behavior regularized offline reinforcement learning,” _arXiv preprint arXiv:1911.11361_ , 2019.
  * [28] A. Nair, M. Dalal, A. Gupta, and S. Levine, “Accelerating online reinforcement learning with offline datasets,” _arXiv preprint arXiv:2006.09359_ , 2020\.
  * [29] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in _International Conference on Machine Learning_ , 2016, pp. 1995–2003.
  * [30] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: combining improvements in deep reinforcement learning,” in _AAAI Conference on Artificial Intelligence_ , 2018.
  * [31] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” _arXiv preprint arXiv:1509.06461_ , 2015.
  * [32] D. Arthur and S. Vassilvitskii, “K-means++: the advantages of careful seeding,” in _Proceedings of ACM-SIAM Symposium on Discrete Algorithms_ , USA, 2007, p. 1027–1035.
  * [33] P. Mitra, S. K. Pal, and M. A. Siddiqi, “Non-convex clustering using expectation maximization algorithm with rough set initialization,” _Pattern Recognition Letters_ , vol. 24, no. 6, pp. 863–873, 2003.
  * [34] M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander, “OPTICS: ordering points to identify the clustering structure,” _ACM SIGMOD Record_ , vol. 28, no. 2, pp. 49–60, 1999.
  * [35] A. Nagpal, A. Jatain, and D. Gaur, “Review based on data clustering algorithms,” in _IEEE Conference on Information & Communication Technologies_, 2013, pp. 298–303.
  * [36] D. Pelleg, A. W. Moore _et al._ , “X-means: extending k-means with efficient estimation of the number of clusters.” in _International Conference on Machine Learning_ , vol. 1, 2000, pp. 727–734.
  * [37] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard, “Way off-policy batch deep reinforcement learning of implicit human preferences in dialog,” _arXiv preprint arXiv:1907.00456_ , 2019.
  * [38] I. Goodfellow, Y. Bengio, and A. Courville, _Deep learning_. MIT press, 2016.
  * [39] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine, “D4RL: datasets for deep data-driven reinforcement learning,” _arXiv preprint arXiv:2004.07219_ , 2020.
  * [40] C. Gulcehre, Z. Wang, A. Novikov, T. L. Paine, S. G. Colmenarejo, K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru _et al._ , “RL unplugged: benchmarks for offline reinforcement learning,” _arXiv preprint arXiv:2006.13888_ , 2020.
  * [41] M. Dudík, J. Langford, and L. Li, “Doubly robust policy evaluation and learning,” in _International Conference on Machine Learning_ , 2011, p. 1097–1104.
  * [42] J. Gauci, E. Conti, Y. Liang, K. Virochsiri, Y. He, Z. Kaden, V. Narayanan, X. Ye, Z. Chen, and S. Fujimoto, “Horizon: Facebook’s open source applied reinforcement learning platform,” _arXiv preprint arXiv:1811.00260_ , 2018\.
  * [43] T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas, “Hyperparameter selection for offline reinforcement learning,” _arXiv preprint arXiv:2007.09055_ , 2020.


COSINE-100 Collaboration

# Search for inelastic WIMP-iodine scattering with COSINE-100

G. Adhikari  Department of Physics and Wright Laboratory, Yale University, New
Haven, CT 06520, USA  N. Carlin  Physics Institute, University of São Paulo,
05508-090, São Paulo, Brazil  J. J. Choi  Department of Physics and Astronomy,
Seoul National University, Seoul 08826, Republic of Korea  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  S. Choi  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Republic of Korea  A. C. Ezeribe  Department
of Physics and Astronomy, University of Sheffield, Sheffield S3 7RH, United
Kingdom  L. E. França  Physics Institute, University of São Paulo, 05508-090,
São Paulo, Brazil  C. Ha  Department of Physics, Chung-Ang University, Seoul
06973, Republic of Korea  I. S. Hahn  Department of Science Education, Ewha
Womans University, Seoul 03760, Republic of Korea  Center for Exotic Nuclear
Studies, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
IBS School, University of Science and Technology (UST), Daejeon 34113,
Republic of Korea  S. J. Hollick  Department of Physics and Wright Laboratory,
Yale University, New Haven, CT 06520, USA  E. J. Jeon  Center for Underground
Physics, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
J. H. Jo  Department of Physics and Wright Laboratory, Yale University, New
Haven, CT 06520, USA  H. W. Joo  Department of Physics and Astronomy, Seoul
National University, Seoul 08826, Republic of Korea  W. G. Kang  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  M. Kauer  Department of Physics and Wisconsin IceCube
Particle Astrophysics Center, University of Wisconsin-Madison, Madison, WI
53706, USA  B. H. Kim  Center for Underground Physics, Institute for Basic
Science (IBS), Daejeon 34126, Republic of Korea  H. J. Kim  Department of
Physics, Kyungpook National University, Daegu 41566, Republic of Korea  J. Kim
Department of Physics, Chung-Ang University, Seoul 06973, Republic of Korea
K. W. Kim  Center for Underground Physics, Institute for Basic Science (IBS),
Daejeon 34126, Republic of Korea  S. H. Kim  Center for Underground Physics,
Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea  S. K. Kim
Department of Physics and Astronomy, Seoul National University, Seoul 08826,
Republic of Korea  W. K. Kim  IBS School, University of Science and Technology
(UST), Daejeon 34113, Republic of Korea  Center for Underground Physics,
Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea  Y. D. Kim
Center for Underground Physics, Institute for Basic Science (IBS), Daejeon
34126, Republic of Korea  Department of Physics, Sejong University, Seoul
05006, Republic of Korea  IBS School, University of Science and Technology
(UST), Daejeon 34113, Republic of Korea  Y. H. Kim  Center for Underground
Physics, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
Korea Research Institute of Standards and Science, Daejeon 34113, Republic of
Korea  IBS School, University of Science and Technology (UST), Daejeon 34113,
Republic of Korea  Y. J. Ko  Center for Underground Physics, Institute for
Basic Science (IBS), Daejeon 34126, Republic of Korea  D. H. Lee  Department
of Physics, Kyungpook National University, Daegu 41566, Republic of Korea  E.
K. Lee  Center for Underground Physics, Institute for Basic Science (IBS),
Daejeon 34126, Republic of Korea  H. Lee  IBS School, University of Science
and Technology (UST), Daejeon 34113, Republic of Korea  Center for Underground
Physics, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
H. S. Lee  Center for Underground Physics, Institute for Basic Science (IBS),
Daejeon 34126, Republic of Korea  IBS School, University of Science and
Technology (UST), Daejeon 34113, Republic of Korea  H. Y. Lee  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  I. S. Lee  Center for Underground Physics, Institute for
Basic Science (IBS), Daejeon 34126, Republic of Korea  J. Lee  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  J. Y. Lee  Department of Physics, Kyungpook National
University, Daegu 41566, Republic of Korea  M. H. Lee  Center for Underground
Physics, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
IBS School, University of Science and Technology (UST), Daejeon 34113,
Republic of Korea  S. H. Lee  IBS School, University of Science and Technology
(UST), Daejeon 34113, Republic of Korea  Center for Underground Physics,
Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea  S. M. Lee
Department of Physics and Astronomy, Seoul National University, Seoul 08826,
Republic of Korea  Y. J. Lee  Department of Physics, Chung-Ang University,
Seoul 06973, Republic of Korea  D. S. Leonard  Center for Underground Physics,
Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea  N. T.
Luan  Department of Physics, Kyungpook National University, Daegu 41566,
Republic of Korea  B. B. Manzato  Physics Institute, University of São Paulo,
05508-090, São Paulo, Brazil  R. H. Maruyama  Department of Physics and Wright
Laboratory, Yale University, New Haven, CT 06520, USA  R. J. Neal  Department
of Physics and Astronomy, University of Sheffield, Sheffield S3 7RH, United
Kingdom  J. A. Nikkel  Department of Physics and Wright Laboratory, Yale
University, New Haven, CT 06520, USA  S. L. Olsen  Center for Underground
Physics, Institute for Basic Science (IBS), Daejeon 34126, Republic of Korea
B. J. Park  IBS School, University of Science and Technology (UST), Daejeon
34113, Republic of Korea  Center for Underground Physics, Institute for Basic
Science (IBS), Daejeon 34126, Republic of Korea  H. K. Park  Department of
Accelerator Science, Korea University, Sejong 30019, Republic of Korea  H. S.
Park  Korea Research Institute of Standards and Science, Daejeon 34113,
Republic of Korea  K. S. Park  Center for Underground Physics, Institute for
Basic Science (IBS), Daejeon 34126, Republic of Korea  S. D. Park  Department
of Physics, Kyungpook National University, Daegu 41566, Republic of Korea  R.
L. C. Pitta  Physics Institute, University of São Paulo, 05508-090, São Paulo,
Brazil  H. Prihtiadi  Center for Underground Physics, Institute for Basic
Science (IBS), Daejeon 34126, Republic of Korea  S. J. Ra  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  C. Rott  Department of Physics, Sungkyunkwan University,
Suwon 16419, Republic of Korea  Department of Physics and Astronomy,
University of Utah, Salt Lake City, UT 84112, USA  K. A. Shin  Center for
Underground Physics, Institute for Basic Science (IBS), Daejeon 34126,
Republic of Korea  D. F. F. S. Cavalcante  Physics Institute, University of
São Paulo, 05508-090, São Paulo, Brazil  A. Scarff  Department of Physics and
Astronomy, University of Sheffield, Sheffield S3 7RH, United Kingdom  N. J. C.
Spooner  Department of Physics and Astronomy, University of Sheffield,
Sheffield S3 7RH, United Kingdom  W. G. Thompson  Department of Physics and
Wright Laboratory, Yale University, New Haven, CT 06520, USA  L. Yang
Department of Physics, University of California San Diego, La Jolla, CA 92093,
USA  G. H. Yu  Department of Physics, Sungkyunkwan University, Suwon 16419,
Republic of Korea  Center for Underground Physics, Institute for Basic Science
(IBS), Daejeon 34126, Republic of Korea

###### Abstract

We report the results of a search for inelastic scattering of weakly
interacting massive particles (WIMPs) off 127I nuclei using NaI(Tl) crystals
with a data exposure of 97.7 kg$\cdot$years from the COSINE-100 experiment.
The signature of inelastic WIMP-127I scattering is a nuclear recoil
accompanied by a 57.6 keV $\gamma$-ray from the prompt deexcitation, producing
an energetic signature compared to the typical WIMP nuclear recoil signal. We
found no evidence for this inelastic scattering signature and set a 90 $\%$
confidence level upper limit on the WIMP-proton spin-dependent, inelastic
scattering cross section of $1.2\times 10^{-37}{\rm cm^{2}}$ at the WIMP mass
500 ${\rm GeV/c^{2}}$.

##  I Introduction

An abundance of astronomical observations indicates the existence of invisible
dark matter in the Universe Clowe:2006eq ; Ade:2015xua . Among the various
candidates for the particle dark matter, the weakly interacting massive
particle (WIMP) PhysRevLett.39.165 ; Goodman:1984dc  is a prominent candidate
that is well motivated by theoretical models beyond the standard model of
particle physics Jungman:1995df . Several direct Billard:2021uyg  and indirect
Conrad:2017pms  detection experiments, as well as collider production
experiments dmcol , have tested this hypothesis, but no clear evidence has
been yet observed Workman:2022ynf .

Most direct detection experiments have focused on WIMP-nucleus elastic
scatterings reactions, which result in nuclear recoils with energies of about
10 keVnr (kilo-electron-volt nuclear recoil energy) that are quenched to about
1 keVee (kilo-electron-volt electron-equivalent energy) due to the detector
response Joo:2018hom . This low energy is close to the threshold of many dark
matter detectors PhysRevLett.118.021303 ; Agnese:2017njq ; Aprile:2017iyp ;
DarkSide:2018bpj ; XENON:2018voc ; CRESST:2019jnq ; Zyla:2020zbs .

On the other hand, when WIMP scatters from a target nucleus in a detector, it
may excite the nucleus to a higher energy state. In this so called WIMP-
nucleus inelastic scattering scenario ELLIS1988375 ; Vergados:2013raa ;
PhysRevC.102.035501 , the subsequent deexcitation of the nucleus emits a
characteristic $\gamma$-ray that is detected in addition to the nuclear recoil
energy transferred from the WIMP-nucleus interaction. In this scenario, the
inelastic interaction produces a relatively high-energy signature that is well
above the detector’s threshold. Since the nuclear excitation in inelastic
scattering is a spin-dependent interaction, the observation of WIMP inelastic
scattering would provide insight into spin properties of WIMP. In the model of
Ref. Arcadi:2019hrw , inelastic scattering channels can have better
sensitivity for some ranges of WIMP model parameters.

Previous searches for WIMP inelastic scattering have focused on excited states
of 127I FUSHIMI1994400  or 129Xe xmass:20191 ; PhysRevD.96.022008 ;
PhysRevD.103.063028 . In the case of 129Xe, the inelastic nuclear excitation
of the 39.6 keV state relies on the WIMP-neutron spin-dependent interaction.
With a 26 % natural abundance of 129Xe, the exclusion limits on the WIMP-
neutron inelastic scattering channels have been obtained xmass:20191 ;
PhysRevD.103.063028 .

The 127I nucleus has complementary advantages over the 129Xe nucleus; it has
almost 100 % natural abundance and odd proton numbers sensitive to WIMP-proton
spin-dependent interaction. The first excited level of 127I is a 7/2+ state of
57.6 keV above the 5/2+ ground state, with a half-life of 1.9 ns
Vergados:2013raa . Therefore, the WIMP-127I inelastic interaction signal would
be a combination of the nuclear recoil and the 57.6 keV deexcitation energies.
The ELEGANTS-V NaI(Tl) experiment FUSHIMI1994400  reported limits on the rate
of events rate with energies in this region. In this paper, we report a search
for the WIMP-127I inelastic scattering events from the COSINE-100 NaI(Tl)
crystal detectors in the signal region of 35–85 keVee.

##  II Experiment

The COSINE-100 experiment Adhikari:2017esn  is located at the Yangyang
underground laboratory in South Korea, with an overburden of approximately 700
m Kims:2005dol ; Kim:2012rza . The COSINE-100 detector, as shown in Fig. 1,
consists of an array of eight ultra-pure NaI(Tl) crystals, total weight of 106
kg cosinebg ; cosinebg2 . The NaI(Tl) crystal assemblies are immersed in an
active veto detector comprised of 2,200 L of linear alkylbenzene (LAB)-based
liquid scintillator (LS) that attenuates or tags the influence of radioactive
backgrounds observed by crystals Park:2017jvs ; Adhikari:2020asl . The LAB-LS
is surrounded by a 3 cm-thick layer of oxygen-free copper, a 20 cm thick lead
shield, and plastic scintillator pannels that tag and veto cosmic ray muons
Prihtiadi:2017inr ; Prihtiadi:2020yhz . Each crystal is optically coupled to
two photomultiplier tubes (PMTs).

Figure 1: Schematic of the COSINE-100 detector. The NaI(Tl) detectors are
immersed in the 2,200 L LAB-LS that is surrounded by layers of copper and lead
shielding.

An event is triggered when a signal corresponding to one or more
photoelectrons occurs in both PMTs of a crystal that are within a 200 ns time
window. If at least one crystal satisfies the trigger condition, signals from
all crystals and the LAB-LS are recorded. The signals from the crystal PMTs
are 8 $\mu$s long waveforms that start 2.4 $\mu$s earlier than the trigger
position, and are processed by 500 MHz flash analog-to-digital (FADC)
converters. In addition to the 5${}^{\textrm{th}}$ stage dynode readouts for
high-energies (50–3 MeV), the low-energy (0–100 keV) anode readouts are stored
by the FADCs. The anode readouts provide sufficient energy information for
events between 35 keVee and 85 keVee. The LAB-LS and plastic scintillator
signals are processed by charge-sensitive flash analog-to-digital converters.
Muon events are triggered by coincident signals from at least two plastic
scintillator panels. A trigger and clock board reads the trigger information
from individual boards and generates a global trigger and time
synchronizations for all of the modules. Details of the COSINE-100 data
acquisition system are described in Ref. Adhikari:2018fpo .

The analysis presented here utilizes data from October 2016 to July 2018,
corresponding to 1.7 years of exposure, which was used for our first annual
modulation search Adhikari:2019off  and the model-dependent WIMP dark matter
search that was based on the shape of the energy spectra COSINE-100:2021xqn .
During the 1.7-year data-taking period, no significant environmental anomalies
or unstable detector performance were observed. Three of the eight crystals
are excluded from this analysis due to their high background, high noise, and
low light yield, resulting in a total effective mass of 61.3 kg
Adhikari:2017esn ; Adhikari:2018ljm .

##  III WIMP-127I inelastic scattering signals

An inelastic scattering event that occurs in 127I can result in the nuclear
recoil together with the emission of the 57.6 keV $\gamma$-ray from the
deexcitation. The contribution to the scintillation signal from the nuclear
recoil depends on the velocity distribution of the WIMPs in the galatic dark
matter halo and the nuclear form factor for the spin dependent interaction.
The differential nuclear recoil rate per unit energy of the nuclear components
is described as follows Zyla:2020zbs ,

$\displaystyle\frac{d{R}}{dE_{\rm
nr}}=\frac{\rho_{\chi}}{2m_{\chi}\mu^{2}}\sigma\int^{v_{\rm max}}_{v_{\rm
min}}d^{3}f({\bf v},t),$ (1)

where ${R}$ is the event rate per unit target mass and unit time, $E_{\rm nr}$
is the nuclear recoil energy, $\rho_{\chi}$ is the local dark matter density,
$m_{\chi}$ and $\mu$ are the WIMP mass and the reduced mass of WIMP and 127I
nuclei, respectively, and $\sigma$ is the WIMP-nucleus scattering cross
section for the inelastic interaction. The integral is performed from the
minimum velocity $v_{\rm min}$ of the inelastic scattering to the maximum
velocity $v_{\rm max}$, which is the same as the galactic escape velocity
$v_{\rm esc}$. Because of the excited energy $E_{\rm ex}$, $v_{\rm min}$ is
increased from the minimum velocity of the typical nuclear recoil ($v_{0}$) as
follows,

$\displaystyle v_{\rm min}=v_{0}+\frac{v^{2}_{\rm thr}}{4v_{0}},$ (2)

where $v_{0}=\sqrt{\frac{m_{\rm target}E_{\rm nr}}{2\mu^{2}}}$, $m_{\rm
target}$ $=$ the mass of 127I, and $v^{2}_{\rm thr}=\frac{2\ E_{\rm
ex}}{\mu}$.

This cross section can be expressed in terms of the cross section of the WIMP-
proton spin-dependent interaction $\sigma_{\rm p}$ by

$\displaystyle\sigma=\frac{4}{3}\frac{\pi}{2J+1}\left(\frac{\mu}{\mu_{\rm
p}}\right)^{2}S(E_{\rm nr})\sigma_{\rm p},$ (3)

where $J=5/2$ is the spin of the ground-state 127I nucleus, $\mu_{\rm p}$ is
the reduced mass of the WIMP and proton, and $S(E_{\rm nr})$ is the nuclear
form factor of inelastic WIMP-127I scattering. We use a recent calculation of
$S(E_{\rm nr})$ for the 127I inelastic interaction in Ref. PhysRevC.102.035501
. We assume the standard halo model of the WIMP velocity distribution, $f({\bf
v},t)$ Lewin:1995rx ; Freese:2012xd ,

$\displaystyle f({\bf
v},t)=\begin{cases}\frac{1}{N_{\mathrm{esc}}}&\left(\frac{3}{2\pi\sigma_{v}^{2}}\right)^{3/2}e^{-3[{\bf
v}+{\bf v}_{\mathrm{E}}(t)]^{2}/2\sigma_{v}^{2}},\\\ &\mbox{for }\left|{\bf
v}+{\bf v}_{\mathrm{E}}(t)\right|<v_{\rm esc}\\\
0,&\mbox{otherwise,}\end{cases}$ (4)

where $N_{\mathrm{esc}}$ is a normalization constant, ${\bf v}_{\mathrm{E}}$
is the Earth velocity relative to the WIMP dark matter and $\sigma_{v}$ is the
velocity dispersion. The standard halo model parameterization is used with the
local dark matter density $\rho_{\chi}=0.3$ GeV/cm3, $v_{\mathrm{E}}$ = 232
km/s, $\sqrt{2/3}\sigma_{v}$ = 220 km/s, and $v_{\rm esc}$ = 544 km/s
Smith:2006ym .

Because of the short half-life of 1.9 ns of the 57.6 keV excited 127I state,
the energy deposited in the detector will be the sum of the nuclear recoil
energy and the deexcitation energy of 57.6 keV. Since the nuclear recoil
energy is quenched to the visible electron-equivalent energy with
approximately 5–10 % levels Joo:2018hom , the visible energy is expressed as
follows,

$\displaystyle E_{\rm vis}=f(E_{\rm nr})\times E_{\rm nr}+E_{\rm ex}.$ (5)

Here, $f(E_{\rm nr})$ is the energy-dependent quenching factor for nuclear
recoils SMITH1990203 . In this search, we use the measured quenching factor of
iodine from Ref. Joo:2018hom  with an empirical model described in Ref.
Ko:2019enb . Figure 2 shows the simulated energy spectra for the WIMP masses
of 50 GeV/c2, 500 GeV/c2, and 5,000 GeV/c2 based on Eq. 1. In these energy
spectra, energy resolutions of individual crystal detectors cosinebg2  are
taken into account. The nuclear recoil energy is more relevant for high-mass
WIMPs, and long tails of the energy spectra at high energy are evident.

Figure 2: The expected energy spectra for inelastic scattering of WIMP-127I in
the COSINE-100 detector are shown for WIMP masses of 50 ${\rm GeV/c^{2}}$, 500
${\rm GeV/c^{2}}$, and 5,000 ${\rm GeV/c^{2}}$. The spectra include the
nuclear recoil energy and the deexcitation energy.

##  IV Data analysis

###  IV.1 Event selection

In order to suppress cosmic-ray muon-induced events, the crystal hit events
that are coincident with muon candidate events in the muon detector
Prihtiadi:2017inr ; Prihtiadi:2020yhz  within 30 ms are rejected.
Additionally, we require that the leading edges of the trigger pulses start
later than 2.0 $\mu$s after the start of the recording, that the waveforms
from the hit crystal contain more than two single photoelectrons, and that the
integral waveform area below the baseline does not exceed a certain limit.
These criteria reject muon-induced phosphor events and electronic interference
noise events.

A multiple-hit event is one in which more than one crystal has a signal with
more than four photoelectrons in an 8 $\mu$s time window or has an LS signal
above an 80 keVee threshold within 4 $\mu$s of the hit crystal
Adhikari:2020asl . A single-hit event is classified as one where only one
crystal has a hit, and none of the other detectors meets the above criteria.
In this analysis, only single-hit events are used.

In the signal region around 57.6 keV energy, there is a few-percent-level
contamination from the high energy tail of PMT-induced noise events. However,
these events are efficiently rejected with the boosted decision tree-based
discriminant, as described in Refs. Adhikari:2020xxj ; COSINE-100:2021poy .
The selection efficiencies for the scintillation events are estimated from
calibration data using a 60Co source to be more than 99 % in 35–85 keVee
signal region.

###  IV.2 Backgrounds

Geant4 Agostinelli:2002hh -based simulations are used to understand the
contribution of each background component Adhikari:2017gbj ; cosinebg ;
cosinebg2 . The fraction of each of these is determined from a simultaneous
fit to the single-hit and multiple-hit events cosinebg2 . For the single-hit
data, we consider the fit energy range between 35 and 85 keVee that covers the
dominant inelastic signal range of 50–70 keVee. Figure 3 presents the crystal
4 energy spectrum in the results from the Ref. cosinebg2  background model
superimposed. In this ROI, the dominant backgrounds are the internal 210Pb,
which produces a 46.5 keV $\gamma$-ray with a few keV Auger electrons, and
129I, which emits a 39.6 keV $\gamma$-ray and a $\beta$ particle. Table 1
presents the expected background composition and the observed data in the
50–70 keVee energy region of the single-hit events for the five crystals. As
shown in this table, the observed data agrees well with the sum of the
expected backgrounds.

Figure 3: Measured energy spectrum in the ROI of crystal 4 (black points) and its background model (blue solid line) with the 68% (yellow band) and 95% (green band) confidence intervals are presented. The expected contributions to the background from 210Pb (red dashed line), 129I (green dotted-dashed line), and other components (black dotted line) are indicated. In the 50–70 keVee energy region, 210Pb and 129I are the dominant background components. Table 1: The background expectations and observed data in the 50–70 keVee energy range for single hits of the 1.7 years COSINE-100 exposure are summarized, with only statistical uncertainties being considered.  | crystal 2 | crystal 3 | crystal 4 | crystal 6 | crystal 7
---|---|---|---|---|---
Internal |  210Pb |  424500 $\pm$ 2400 |  155800 $\pm$ 4200 |  290000 $\pm$ 9100 |  486300 $\pm$ 8200 |  420900 $\pm$ 5100
40K |  8400 $\pm$ 200 |  3700 $\pm$ 100 |  8800 $\pm$ 100 |  2100 $\pm$ 200 |  2300 $\pm$ 200
Others |  100 $\pm$ 10 |  100 $\pm$ 10 |  100 $\pm$ 10 |  100 $\pm$ 11 |  100 $\pm$ 10
External |  238U |  87200 $\pm$ 1900 |  68600 $\pm$ 1600 |  24900 $\pm$ 2100 |  71100 $\pm$ 1600 |  67300 $\pm$ 1700
228Th |  11600 $\pm$ 2500 |  17500 $\pm$ 1900 |  38700 $\pm$ 2800 |  34300 $\pm$ 2400 |  29300 $\pm$ 2100
Others |  6400 $\pm$ 600 |  3400 $\pm$ 600 |  13400 $\pm$ 1000 |  12200 $\pm$ 900 |  8100 $\pm$ 1200
Cosmogenic |  129I |  157700 $\pm$ 3200 |  148200 $\pm$ 3300 |  331000 $\pm$ 5300 |  230300 $\pm$ 4200 |  313100 $\pm$ 5100
127mTe |  1900 $\pm$ 300 |  2900 $\pm$ 100 |  2100 $\pm$ 100 |  800 $\pm$ 100 |  800 $\pm$ 100
Others |  100 $\pm$ 8 |  2500 $\pm$ 280 |  32500 $\pm$ 900 |  18100 $\pm$ 1600 |  13600 $\pm$ 1600
Total (expected) |  707900 $\pm$ 5100 |  402700 $\pm$ 5900 |  742800 $\pm$ 9800 |  855300 $\pm$ 9800 |  855500 $\pm$ 8000
Data | 716352 | 410655 | 746285 | 856789 | 864034

We consider various sources of systematic uncertainties in the background and
signal models. Errors associated with the energy resolution, the energy scale,
and the background modeling technique are accounted for in the shapes of the
signal and background probability density functions, as well as in rate
changes as described in Ref. COSINE-100:2021xqn . These quantities are allowed
to vary within their uncertainties as nuisance parameters in the data fit used
to extract the signal. The largest systematic uncertainty comes from the error
associated with the 210Pb background modeling, which is due to its dominant
contribution and its fast shape change in the ROI as shown in Fig. 3. This
error includes uncertainties in the energy resolution, energy scale, and depth
profiles of 210Pb on the surface of the NaI(Tl) crystals, which were studied
with a 222Rn contaminated crystal Yu:2020ntl  and varied within their
uncertainties.

###  IV.3 Signal fit

To search for evidence of the WIMP-127I inelastic scattering signals, a
Bayesian approach with a likelihood function based on Poisson probability,
described in Ref. COSINE-100:2021xqn , is used. The likelihood fit is applied
to the measured single-hit energy spectra between 35 and 85 keVee for several
WIMP masses. Each crystal is fitted with a crystal-specific background model
and a crystal-correlated WIMP signal for the combined fit by multiplying the
five crystals’ likelihoods. The means and uncertainties for background
components, which are determined from modeling Adhikari:2017gbj ; cosinebg ;
cosinebg2 , are used to set Gaussian priors for the background. The systematic
uncertainties are included in the fit as nuisance parameters with Gaussian
priors.

Prior to the data fit, the fitter was tested with simulated event samples.
Each simulated dataset is prepared by Poisson random extraction of the modeled
background spectrum, assuming a background-only hypothesis. Marginalization to
obtain the posterior probability density function (PDF) for each simulated
sample is performed to set the 90 % confidence level upper limits. The 1,000
simulated experiments result in $\pm$1$\sigma$ and $\pm$2$\sigma$ bands of the
expected median sensitivity.

|
---|---
(a) | (b)
Figure 4: (a) The black-filled circles represent the data points showing the
summed energy spectra from the five crystals. The solid blue line indicates
the result of the fit with a 500 GeV/c2 WIMP mass signal. The expected signal
shape of the 500 GeV/c2 WIMP mass is a red solid line, assuming WIMP-proton
spin-dependent cross sections of $5\times 10^{-36}$ cm2, which is 30 times
higher than the 90 % confidence level upper limit. (b) It is an example of the
posterior probability density function (PDF) and cumulated density function
(CDF) for the 1.7-year of COSINE-100 data and a WIMP mass of 500 GeV/c2. The
posterior PDF is scaled so that the maximum value is unity. In this PDF, the
best fit (i.e. the most probable cross section) points to a null signal.
Therefore, we set the 90 % confidence level upper limit. The exclusion limit
at a 90 % confidence level is obtained from the CDF matched with 0.9. The
yellow and green areas represent the 1 $\sigma$ and 2 $\sigma$ confidence
intervals, respectively.

Fits to the data are performed for each of the 12 WIMP masses considered in
the same way as the simulated data. As an example, the data fit with a WIMP
mass of 500 ${\rm GeV/c^{2}}$ is presented in Fig. 4. The summed event
spectrum for the five crystals is shown in Fig. 4 (a), which corresponds to
the null observation. For comparison, the expected signal for the WIMP mass
500 GeV/c2 with spin-dependent cross section of $3.6\times 10^{-36}$cm2 is
overlaid. This cross section is 30 times higher than the measured 90 %
confidence level upper limit. Figure 4 (b) shows the posterior PDF and its
cumulative distribution function of data for this example.

Figure 5: The observed 90 % confidence level exclusion limits on the WIMP-
proton spin-dependent cross section from the 1.7-year of COSINE-100 data are
shown, along with the $\pm$1 $\sigma$ and $\pm$2 $\sigma$ bands for the
expected sensitivity under the background-only hypothesis. The limits are
compared with a WIMP interpretation of the ELEGANTS-V upper limit of 57.6 keV
event rate for the WIMP-127I inelastic scattering hypothesis.

No excess of events that could be attributed to WIMP-127I inelastic
interactions is found in the 12 different WIMP masses considered. The
posterior probabilities of the signal were consistent with zero in all cases,
and 90% confidence level limits are determined, as an example shown in Fig. 4
(b). Figure 5 shows the 90% confidence level exclusion limits from the
COSINE-100 data with $\pm$1$\sigma$ and $\pm$2$\sigma$ expected bands of
exclusion limits from the simulated experiments.

For comparison, we interpret a 90 % confidence level upper limit of ELEGANTS-V
from the inelastic scattering event rate of 9.8$\times$10-2 counts/kg/day
FUSHIMI1994400  as the WIMP-127I inelastic scattering cross section, as shown
in Fig. 5. Although extracted event rates depend on the WIMP mass, we assume
the same event rates of the ELEGANTS-V interpretations considering similar
shapes of signal spectra due to the dominant deexcitation energy of 57.6 keV.
Our results improve the exclusion limit by order of magnitude from the
previous search for the same channel and are the most stringent result to date
in the spin-dependent WIMP-proton cross section via the inelastic scattering
channel.

Because of the dominant background from 210Pb of 46.5 keV $\gamma$ with Auger
electrons to ROI, the inelastic WIMP-127I scattering signal for the low-mass
WIMP is encompassed with the 210Pb background as one can see in Fig. 2. It is
also affected by the systematic uncertainty of the 210Pb modeling, especially
from the energy scale that increases uncertainties of the event rates near the
signal region and increases the fluctuation of the expected limit bands, as
one can see in Fig. 5. Compared with the ELEGANTS-V interpretation, which
assumes flat background, the low-mass WIMP limit from this work reflect the
influence of the the 210Pb background.

Our R&D program for the development of low-background NaI(Tl) crystal
detectors has resulted in NaI(Tl) detectors with significantly reduced 210Pb
background COSINE:2020egt ; 10.3389/fphy.2023.1142765 , which will be applied
to the COSINE-200 experiment Ko:2022pmu . With the realization of the
COSINE-200 experiment, the sensitivities to search for the WIMP-127I inelastic
interaction will be enhanced.

##  V Conclusion

We performed a search for WIMP-127I inelastic scattering events from 57.6 keV
deexcitation $\gamma$ with the nuclear recoil in the 1.7 years COSINE-100
data. The single-hit energy spectrum was fitted with signal and background
models in the energy range of 35–85 keVee. We found no evidence of the
WIMP-127I inelastic interaction signals, allowing us to set 90 % confidence
level exclusion limits on the WIMP-proton spin-dependent interaction cross
section. The best limit is 1.2$\times$10-37cm2 of a WIMP mass of 500 GeV/c2.
It is the most stringent limit for the spin-dependent WIMP-proton interaction
using WIMP-nucleus inelastic scattering process.

###### Acknowledgements.

We thank the Korea Hydro and Nuclear Power (KHNP) Company for providing
underground laboratory space at Yangyang and the IBS Research Solution Center
(RSC) for providing high performance computing resources. This work is
supported by: the Institute for Basic Science (IBS) under project code
IBS-R016-A1, NFEC-2019R1A6C1010027, NRF-2021R1A2C3010989,
NRF-2021R1I1A3041453, and NRF-2021R1A2C1013761, Republic of Korea; NSF Grants
No. PHY-1913742, DGE-1122492, WIPAC, the Wisconsin Alumni Research Foundation,
United States; STFC Grant ST/N000277/1 and ST/K001337/1, United Kingdom; Grant
No. 2021/06743-1 and 2022/12002-7 FAPESP, CAPES Finance Code 001, CNPq
131152/2020-3, Brazil.

## References

  * (1) D. Clowe et al., A direct empirical proof of the existence of dark matter, Astrophys. J. 648, L109 (2006).
  * (2) P. A. R. Ade et al., (Planck Collaboration), Planck 2015 results. XIII. Cosmological parameters, Astron. Astrophys. 594, A13 (2016).
  * (3) B. W. Lee and S. Weinberg, Cosmological lower bound on heavy-neutrino masses, Phys. Rev. Lett. 39, 165 (1977).
  * (4) M. W. Goodman and E. Witten, Detectability of Certain Dark Matter Candidates, Phys. Rev. D 31, 3059 (1985).
  * (5) G. Jungman, M. Kamionkowski, and K. Griest, Supersymmetric dark matter, Phys. Rept. 267, 195–373 (1996).
  * (6) J. Billard et al., Direct detection of dark matter—APPEC committee report*, Rept. Prog. Phys. 85, 056201 (2022).
  * (7) J. Conrad and O. Reimer, Indirect dark matter searches in gamma and cosmic rays, Nature Phys. 13, 224–231 (2017).
  * (8) O. Buchmueller, C. Doglioni, and L.-T. Wang, Search for dark matter at colliders, Nature Phy. 13, 217 (2017).
  * (9) R. L. Workman and Others, (Particle Data Group Collaboration), Review of Particle Physics, PTEP 2022, 083C01 (2022).
  * (10) H. W. Joo, H. S. Park, J. H. Kim, S. K. Kim, Y. D. Kim, H. S. Lee, and S. H. Kim, Quenching factor measurement for NaI(Tl) scintillation crystal, Astropart. Phys. 108, 50–56 (2019).
  * (11) D. S. Akerib et al., (LUX Collaboration), Results from a Search for Dark Matter in the Complete LUX Exposure, Phys. Rev. Lett. 118, 021303 (2017).
  * (12) R. Agnese et al., (SuperCDMS Collaboration), Results from the Super Cryogenic Dark Matter Search Experiment at Soudan, Phys. Rev. Lett. 120, 061802 (2018).
  * (13) E. Aprile et al., (XENON Collaboration), First Dark Matter Search Results from the XENON1T Experiment, Phys. Rev. Lett. 119, 181301 (2017).
  * (14) P. Agnes et al., (DarkSide Collaboration), Low-Mass Dark Matter Search with the DarkSide-50 Experiment, Phys. Rev. Lett. 121, 081307 (2018).
  * (15) E. Aprile et al., (XENON Collaboration), Dark Matter Search Results from a One Ton-Year Exposure of XENON1T, Phys. Rev. Lett. 121, 111302 (2018).
  * (16) A. H. Abdelhameed et al., (CRESST Collaboration), First results from the CRESST-III low-mass dark matter program, Phys. Rev. D 100, 102002 (2019).
  * (17) P. Zyla et al., (Particle Data Group Collaboration), Review of Particle Physics, PTEP 2020, 083C01 (2020).
  * (18) J. Ellis, R. Flores, and J. Lewin, Rates for inelastic nuclear excitation by dark matter particles, Physics Letters B 212, 375–380 (1988).
  * (19) J. D. Vergados, H. Ejiri, and K. G. Savvidy, Theoretical direct WIMP detection rates for inelastic scattering to excited states, Nucl. Phys. B 877, 36–50 (2013).
  * (20) R. Sahu, D. K. Papoulias, V. K. B. Kota, and T. S. Kosmas, Elastic and inelastic scattering of neutrinos and weakly interacting massive particles on nuclei, Phys. Rev. C 102, 035501 (2020).
  * (21) G. Arcadi, C. Döring, C. Hasterok, and S. Vogl, Inelastic dark matter nucleus scattering, JCAP 12, 053 (2019).
  * (22) K. Fushimi, H. Ejiri, R. Hazama, N. Kudomi, K. Nagata, H. Ohsumi, K. Okada, and J. Tanaka, Search for exotic nuclear transition by using the large volume NaI detector of ELEGANTS V., Nucl. Phys. B (Proc. Suppl.) 35, 400–402 (1994).
  * (23) T. Suzuki et al., (XMASS Collaboration), Search for WIMP-129Xe inelastic scattering with particle identification in XMASS-I, Astropart. Phys. 110, 1–7 (2019).
  * (24) E. Aprile et al., (XENON Collaboration), Search for WIMP inelastic scattering off xenon nuclei with XENON100, Phys. Rev. D 96, 022008 (2017).
  * (25) E. Aprile et al., (XENON Collaboration), Search for inelastic scattering of WIMP dark matter in XENON1T, Phys. Rev. D 103, 063028 (2021).
  * (26) G. Adhikari et al., (COSINE-100 Collaboration), Initial Performance of the COSINE-100 Experiment, Eur. Phys. J. C 78, 107 (2018).
  * (27) H. S. Lee et al., (KIMS Collaboration), First limit on WIMP cross section with low background CsI(Tl) crystal detector, Phys. Lett. B 633, 201–208 (2006).
  * (28) S. C. Kim et al., New Limits on Interactions between Weakly Interacting Massive Particles and Nucleons Obtained with CsI(Tl) Crystal Detectors, Phys. Rev. Lett. 108, 181301 (2012).
  * (29) P. Adhikari et al., (COSINE-100 Collaboration), Background model for the NaI(Tl) crystals in COSINE-100, Eur. Phys. J. C 78, 490 (2018).
  * (30) G. Adhikari et al., (COSINE-100 Collaboration), Background modeling for dark matter search with 1.7 years of COSINE-100 data, Eur. Phys. J. C 81, 837 (2021).
  * (31) J. S. Park et al., (KIMS Collaboration), Performance of a prototype active veto system using liquid scintillator for a dark matter search experiment, Nucl. Instrum. Meth. A 851, 103 (2017).
  * (32) G. Adhikari et al., The COSINE-100 liquid scintillator veto system, Nucl. Instrum. Meth. A 1006, 165431 (2021).
  * (33) H. Prihtiadi et al., (COSINE-100 Collaboration), Muon detector for the COSINE-100 experiment, JINST 13, T02007 (2018).
  * (34) H. Prihtiadi et al., (COSINE-100 Collaboration), Measurement of the cosmic muon annual and diurnal flux variation with the COSINE-100 detector, JCAP 02, 013 (2021).
  * (35) G. Adhikari et al., (COSINE-100 Collaboration), The COSINE-100 Data Acquisition System, JINST 13, P09006 (2018).
  * (36) G. Adhikari et al., (COSINE-100 Collaboration), Search for a dark matter-induced annual modulation signal in NaI(Tl) with the COSINE-100 experiment, Phys. Rev. Lett. 123, 031302 (2019).
  * (37) G. Adhikari et al., (COSINE-100 Collaboration), Strong constraints from COSINE-100 on the DAMA dark matter results using the same sodium iodide target, Sci. Adv. 7, abk2699 (2021).
  * (38) G. Adhikari et al., (COSINE-100 Collaboration), An experiment to search for dark-matter interactions using sodium iodide detectors, Nature 564, 83 (2018).
  * (39) J. Lewin and P. Smith, Review of mathematics, numerical factors, and corrections for dark matter experiments based on elastic nuclear recoil, Astropart. Phys. 6, 87 (1996).
  * (40) K. Freese, M. Lisanti, and C. Savage, Colloquium: Annual modulation of dark matter, Rev. Mod. Phys. 85, 1561 (2013).
  * (41) M. C. Smith et al., The RAVE Survey: Constraining the Local Galactic Escape Speed, Mon. Not. Roy. Astron. Soc. 379, 755–772 (2007).
  * (42) P. Smith and J. Lewin, Dark matter detection, Physics Reports 187, 203–280 (1990).
  * (43) Y. J. Ko et al., (COSINE-100 Collaboration), Comparison between DAMA/LIBRA and COSINE-100 in the light of Quenching Factors, JCAP 1911, 008 (2019).
  * (44) G. Adhikari et al., (COSINE-100 Collaboration), Lowering the energy threshold in COSINE-100 dark matter searches, Astropart. Phys. 130, 102581 (2021).
  * (45) G. Adhikari et al., (COSINE-100 Collaboration), Searching for low-mass dark matter via the Migdal effect in COSINE-100, Phys. Rev. D 105, 042006 (2022).
  * (46) S. Agostinelli et al., (GEANT4 Collaboration), GEANT4: A Simulation toolkit, Nucl. Instrum. Meth. A 506, 250 (2003).
  * (47) G. Adhikari et al., (KIMS Collaboration), Understanding NaI(Tl) crystal background for dark matter searches, Eur. Phys. J. C 77, 437 (2017).
  * (48) G. H. Yu, C. Ha, E. J. Jeon, K. W. Kim, N. Y. Kim, Y. D. Kim, H. S. Lee, H. K. Park, and C. Rott, Depth profile study of 210Pb in the surface of an NaI(Tl) crystal, Astropart. Phys. 126, 102518 (2021).
  * (49) B. J. Park et al., (COSINE Collaboration), Development of ultra-pure NaI(Tl) detectors for the COSINE-200 experiment, Eur. Phys. J. C 80, 814 (2020).
  * (50) H. Lee, B. J. Park, J. J. Choi, O. Gileva, C. Ha, A. Iltis, E. J. Jeon, D. Y. Kim, K. W. Kim, S. H. Kim, S. K. Kim, Y. D. Kim, Y. J. Ko, C. H. Lee, H. S. Lee, I. S. Lee, M. H. Lee, S. J. Ra, J. K. Son, and K. A. Shin, Performance of an ultra-pure nai(tl) detector produced by an indigenously-developed purification method and crystal growth for the cosine-200 experiment, Frontiers in Physics 11, (2023).
  * (51) Y. J. Ko and H. S. Lee, Solar and supernova neutrino physics with future NaI(Tl) dark matter search detectors, arXiv:2210.01386.