# DeepLocalization: Using change point detection for Temporal Action
Localization

Mohammed Shaiqur Rahman
Iowa State University
<EMAIL_ADDRESS>Ibne Farabi Shihab
Iowa State University
<EMAIL_ADDRESS>Lynna Chu
Iowa State University
<EMAIL_ADDRESS>Anuj Sharma
Iowa State University
<EMAIL_ADDRESS>
###### Abstract

In this study, we introduce DeepLocalization, an innovative framework devised
for the real-time localization of actions tailored explicitly for monitoring
driver behavior. Utilizing the power of advanced deep learning methodologies,
our objective is to tackle the critical issue of distracted driving—a
significant factor contributing to road accidents. Our strategy employs a dual
approach: leveraging Graph-Based Change-Point Detection for pinpointing
actions in time alongside a Video Large Language Model (Video-LLM) for
precisely categorizing activities. Through careful prompt engineering, we
customize the Video-LLM to adeptly handle driving activities’ nuances,
ensuring its classification efficacy even with sparse data. Engineered to be
lightweight, our framework is optimized for consumer-grade GPUs, making it
vastly applicable in practical scenarios. We subjected our method to rigorous
testing on the SynDD2 dataset, a complex benchmark for distracted driving
behaviors, where it demonstrated commendable performance—achieving 57.5%
accuracy in event classification and 51% in event detection. These outcomes
underscore the substantial promise of DeepLocalization in accurately
identifying diverse driver behaviors and their temporal occurrences, all
within the bounds of limited computational resources.

##  1 Introduction

According to the National Highway Traffic Safety Administration (NHTSA),
distracted driving was responsible for 3,522 fatalities in 2021, alongside
causing injuries to an additional 362,415 individuals in motor vehicle
accidents [3]. Distracted driving encompasses any activity that diverts
attention from the primary task of driving safely. This includes but is not
limited to engaging in phone conversations or texting, eating and drinking,
interacting with passengers, and adjusting the vehicle’s stereo,
entertainment, or navigation systems. Essentially, any action that detracts
from the focus on safe driving falls under this category [3]. Drivers often
engage in secondary activities to avoid feeling drowsy; however, these actions
lead to distractions.

The consequences of distraction are significant, often resulting in crashes,
some of which are fatal. Substant economic repercussions exist beyond the
physical harm inflicted on those involved in accidents. The NHTSA has reported
that crashes incurred a total cost of 340 billion dollars in 2019.
Specifically, distraction-related crashes contributed approximately 98.2
billion dollars to this figure, representing 29% of the total traffic-related
economic costs. As such, there is a pressing need to mitigate the damages
caused by distracted driving and enhance road safety. One viable solution is
the implementation of a real-time driver behavior monitoring system. Such a
system would alert drivers upon detecting any form of distraction. Given their
low maintenance and cost-effectiveness, in-vehicle cameras are optimal for
real-time driver behavior monitoring [24].

In recent years, the surge in computational capabilities, coupled with
advancements in deep learning algorithms, has significantly bolstered the
prominence of computer vision technologies, especially in the realm of driver
behavior classification. Unlike traditional machine learning algorithms—which
rely on manually defined key features and specific classifiers for image
classification, a process that varies across different datasets and
complicates application in diverse scenarios—the advent of deep learning
techniques, such as Convolutional Neural Networks (CNNs), has marked a
transformative shift. CNNs possess the unique ability to autonomously extract
features from any given input data, thereby achieving superior accuracy in
both the detection and classification processes. This evolution has spurred
extensive research, yielding several CNN models like ResNet50 [12], AlexNet
[13], and InceptionV3 [22], among others, that stand out for their enhanced
performance in image classification tasks. Consequently, these models often
serve as foundational or baseline models for researchers embarking on image
classification or detection projects tailored to their respective projects’
specific requirements and constraints.

Parallel to these technological advances, the field has seen an uptick in
naturalistic driving studies (NDS), aimed at examining distracted driver
behavior within a realistic driving environment. Such studies are invaluable,
as the data garnered can shed light on various aspects of the driver’s state,
including drowsiness, alertness, and distraction, thus providing deep
behavioral insights. Notably, the Second Strategic Highway Research Program
Naturalistic Driving Study (SHRP 2) [23], undertaken by the Virginia Tech
Transportation Institute (VTTI), stands out as the largest of its kind,
amassing a staggering 2 Petabytes of driving data. Analysis of the SHRP 2 data
revealed that the majority of crashes are closely associated with factors
attributed to the driver (such as error, impairment, fatigue, and
distraction), constituting nearly 90 of all recorded incidents [9]. However,
the utility of NDS is not without challenges, primarily due to susceptibility
to noise and other data quality issues. To address these concerns, the
introduction of SynDD2 [21] in the 7th AI City Challenge [18] represents a
notable effort to mitigate such data quality problems, paving the way for more
accurate and reliable analysis of driver behavior.

Determining the driver’s behavior involves recognizing the driver’s behavior
(activity recognition) and finding the duration of such behavior (TAL-
Temporal Activity Localization). The CNN-based models [11, 10, 27] are used
for the former, and for the latter various approaches have been explored such
as Graph Convolutional Networks [32], 1D temporal segments [4], sliding window
[26]. Although there has been various research for TAL, using a change point
algorithm has not been explored. Hence, we propose an approach combining a
Graph-Based Change-Point Detection [7, 6] for temporal action localization and
video LLM model [17] for classification.

The contributions of our work are as follows:

  1. 1.

Exploring a novel approach for finding the start and end times of an activity.
Our approach doesn’t require any training, the algorithm only needs the key
points to find action proposals.

  2. 2.

The overall framework is novel from generating action proposals to action
classification. It is lightweight which is optimized for consumer-grade GPUs.

The rest of the paper is organized as follows: Section 2 reviews the existing
literature. Sections 3 and 4 detail our methodology and introduce our proposed
approach with results. Finally, in Section 5, we discuss our findings and
suggest avenues for future research.

##  2 Related Works

In the past few years, there has been a significant increase in research on
driver action recognition. The main objective of this research is to identify
and predict risky driving behaviors that lead to accidents caused by
distracted driving. Many researchers are developing models and exploring
different methods to achieve this goal, significantly focusing on using
supervised learning techniques.

Temporal Action Localization: Temporal Action Localization (TAL) is a dynamic
field that identifies the precise start and end times of actions within video
segments. This process typically involves the initial generation of numerous
candidate segments as action proposals, followed by the classification of
these proposals into their respective action categories. To enhance the
precision of action localization, previous studies have adopted a variety of
strategies, such as refining detection boundaries [15, 33], creating frame-
wise action labels to delineate the temporal limits of actions [30], using
graphs structures [28, 31] and improving the quality of action proposals
through the aggregation of context at both the boundary and proposal levels
[5]. Despite the broad application of these methods across different research
areas, there has been limited exploration into the use of graph-based change
point detection algorithms for generating action proposals. The innovative
work presented in [7] showcases the application of such an algorithm to
identify intervals within data where there is a significant shift in
distribution, a technique that has shown promise in the context of action
proposal identification in videos.

Video recognition using LLM: The integration of visual models with Large
Language Models (LLMs) has garnered significant attention following the
success of ChatGPT. This emerging field has seen remarkable advancements,
notably the pioneering work demonstrated by Yuan et al. [29], which laid the
groundwork for subsequent innovations. Among these, the Querying Transformer
proposed by Li et al. [14] stands out for its innovative approach to mapping
images into the text embedding space, enhancing the interaction between visual
data and language models. Further developments by Liu et al. [16] and Dai et
al. [8] have introduced methods that facilitate visual conversation, expanding
the capabilities of LLMs in understanding and generating responses based on
visual inputs. In our research, we have chosen to focus on the work by Maaz et
al. [17] for activity recognition within video data. This approach is
distinguished by its adapted LLM framework, incorporating the visual encoder
from CLIP [19] alongside the Vicuna language decoder [34]. This combination
not only leverages the strengths of both models but also addresses the
challenge of capturing temporal dynamics and ensuring video-to-frame
coherence. Consequently, it offers an optimal solution for analyzing
activities in video content, especially in scenarios where the duration of
activities varies unpredictably across the dataset.

Figure 1: Showing camera positions [20]

(a) Dashboard

(b) Rear-view

(c) Right-side window

Figure 2: Showing different camera views Figure 3: Showing framework pipeline

##  3 Methodology

Our goal is to predict various distracted driver behaviors with their start
and end times using the data from in-vehicle cameras positioned at the
Dashboard, near the rearview mirror, and on the top right-side window corner
(Figure 1 shows the camera positions while Figure 2 shows the images from the
three camera views). To achieve this goal, we have subdivided it into two sub-
tasks: Temporal Action Recognition (TAL), which identifies the start and end
times of activities, and Activity Recognition, which recognizes the type of
activity like texting, yawning, and more. Therefore, we propose an
architectural framework to render these tasks.

As shown in Figure 3, our proposed framework consists of two major modules:
Event detection and event classification. The Event Detection module extracts
the key points related to the drivers’ heads, hands, and bodies for each data
frame. Using these key points, the gseg2 [6] algorithm finds an interval where
their underlying distribution differs from the rest of the sequence. This
interval serves as the start and end time of an activity performed by the
driver, which is then fed as input to the event classification module to
identify the type of that event. The proposed framework pipeline works as
follows: feed the different intervals/events (TAL) from the first module as
input to the second module for classifying those events.

We have utilized deep learning models for each module to effectively carry out
the specific tasks. For the first module, we have employed YOLOv7 [25], a pre-
trained model recognized for its robust object detection capabilities and pose
estimation for the key-point extraction. For the subsequent module, we have
used the VideoChatGPT model [17].

###  3.1 Event detection

This module performs two major tasks: First, it extracts the key points from
the video for each frame. Secondly, using these extracted key points, the
change point detection algorithm finds an interval in the video data frames
that is different from the rest of the sequence of data frames.

####  3.1.1 Key-Point Extraction

We have utilized YOLO (You Only Look Once) [25], a renowned real-time object
detection and image segmentation model, to extract the key points.
Specifically, we have used yolov7 pose estimation model to identify the
location of key points in each frame, along with their respective confidence
scores. The locations of these points represented a set of 2D [x, y]
coordinates, with confidence scores deciding their visibility. We only
selected points which had confidence scores greater than 50%. Moreover,these
points were normalized as discussed in section 4.1. These key points are
obtained from each frame, focusing on individuals’ facial, hand, and body
within the video. They play a vital role in providing essential information to
the Interval Selection module, which employs these key points to find a
statistically significant interval different from the given data sequence. As
shown in Figure 4, the key points are generated for each camera view.

Figure 4: Showing key points extraction

####  3.1.2 Interval Selection

We have executed Graph-Based Change-Point Detection for Changed Interval
algorithm (gseg2) [2] to find intervals from the given data set. This
algorithm is a part of the R package and has the following usage: gseg2(n, E,
statistics=c(”all”,”o”,”w”,”g”,”m”), l0=0.05*n, l1=0.95*n, pval.appr=TRUE,
skew.corr=TRUE, pval.perm=FALSE, B=100) where ’n’ is the number of
observations in the sequence, ’E’ is the edge matrix (a ”number of edges” by
two matrices) for the similarity graph. Each row contains the node indices of
an edge, ’l0’ is the minimum length of the interval to be considered as a
changed interval, ’l1’ is the maximum length of the interval to be considered
as a changed interval, ’pval.appr’ if it is TRUE, the function outputs p-value
approximation based on asymptotic properties, ’skew.corr’ is applicable only
when ’pval.appr’=TURE. If skew.corr is TRUE, the p-value approximation will
incorporate skewness correction, ’pval.perm’ if it is TRUE, the function
outputs p-value from doing B permutations, ’B’ is useful only when
pval.perm=TRUE. The default value for B is 100.

The algorithm needed data in a specific format which is explained in section
4.2. We experimented with different values of n, l0, and l1 for our case and
left others as default. We have presented the results corresponding to the
best combinations. The algorithm outputs results for four statistics: Original
edge-count scan statistic (”o”), Weighted edge-count statistic (”w”),
Generalized edge-count statistic (”g”), and Max-type edge-count statistic
(”m”). We have used Max-type edge-count statistics for finding the interval as
we have key points corresponding to human posture that change with time. After
getting the proposed intervals from all the camera views, we applied post-
processing to select common intervals among the camera views.

###  3.2 Event Classification

Video-LLM showcases the impressive capabilities of learning a wide range of
visual concepts and exhibits exceptional performance on different few-shot
tasks using a pre-trained model. We have selected VideoChatGPT, a version of
Video-LLM, as our fundamental model for detecting distracted behaviors in
video clips. We chose this model because of its ability to be pretty accurate
with limited resources and its proven superior performance over other Video-
LLMs, mainly when resources are scarce. Our strategy involves redefining event
classification into a video question-answering (VQA). We trained VideoChatGPT
with a comprehensive question covering all sixteen identified events to ensure
the model is efficiently tailored to our needs. We also recognize the
challenge posed by the small size of our dataset on distracted behaviors. It
is another reason to use a pre-trained Video-LLM, which reduces the risk of
overfitting during the VQA process. These strategies aim to avoid overfitting
and catastrophic forgetting, maintaining the model’s robustness and
generalization ability. In summary, our objective is to leverage
VideoChatGPT’s capabilities for video understanding tasks, focusing on
classifying distracted actions while effectively navigating the hurdles of
limited resources and the risk of overfitting.

##  4 Experiments

###  4.1 SynDD2

This dataset was introduced in the 7th AI City Challenge [18]. It contains
videos of distracted driving activities recorded from three in-vehicle cameras
in synchronization positioned at the Dashboard, Rearview, and Right-side
window. The dataset has data from participants who performed 16 distracted
driving activities, as listed in Table 1. These 16 activities have a random
order, and each of them lasts for a random duration of time. Thus, classifying
each activity with its start and end times (localization) becomes challenging.

ID | Activity Classes
---|---
1 | Normal Forward Driving
2 | Drinking
3 | Phone Call(right)
4 | Phone Call(left)
5 | Eating
6 | Text (Right)
7 | Text (Left)
8 | Reaching behind
9 | Adjust control panel
10 | Pick up from floor (Driver)
11 | Pick up from floor (Passenger)
12 | Talk to passenger at the right
13 | Talk to passenger at backseat
14 | Yawning
15 | Hand on head
16 | Singing and dancing with music
Table 1: Showing SynDD2: Activity classes

Each video in the dataset has a frame rate of 30 FPS and a resolution of
1920*1080. Each video has its corresponding annotation files (for the training
data set), which contain information such as activity type, start and end
times of each activity type, participant’s appearance, etc.

###  4.2 Experimentation: Event Detection

Figure 3 shows our overall approach. First, we extracted the key points of the
videos of the Dashboard, Rearview, and Right-side camera views by using the
Yolov7 pose estimation model [25]. The model provides 17 crucial key points
along with their associated confidence scores, representing the visibility of
these key points related to the human posture, spanning from the head to the
toes. These key points include the nose, eyes, ears, and joints (shoulders,
wrists, elbows, knees, etc). For our specific case, not all key points prove
valuable; hence, we opted for a subset of key points relevant to the hands and
head.

####  4.2.1 Data Preprocessing

We preprocessed the data by normalizing the key points to maintain consistency
across multiple videos. This involved dividing each key point by the frame
size and standardizing their scale. This was done during the key point
extraction step.

####  4.2.2 Data Preparation for gseg2

The gseg2 algorithm required an edge matrix as input, and to create that we
have used KMST (K-Means Spanning Tree) algorithm. This algorithm takes the
input data in matrix format and calculates the distance. We have applied
Euclidean distance to compute the distance of each key point from the other.
For the given points $\mathbf{p}=(x_{1},y_{1})$ and $\mathbf{q}=(x_{2},y_{2})$
the Euclidean distance is given by:

$d(\mathbf{p},\mathbf{q})=\sqrt{(x_{1}-x_{2})^{2}+(y_{1}-y_{2})^{2}}$

Since our data sets contain multiple key points, we have calculated the
pairwise Euclidean distances between each pair of points in the dataset. This
resulted in a distance matrix where each element represented the Euclidean
distance between the corresponding pair of points in the dataset. Moreover, to
find the optimal k-value for KMST, we experimented with multiple k-values and
concluded that making the graph denser (higher k-value) gave better results.
Table 2 shows results corresponding to various k-values. In the table, actual-
start and actual-end are the ground truth times in seconds while p-start and
p-end are predicted times intervals predicted by gseg2 algorithm.

K value | actual-start | actual-end | p-start | p-end
---|---|---|---|---
10 | 236 | 241 | 233.9 | 236.067
12 | 236 | 241 | 233.9 | 236.067
15 | 236 | 241 | 233.9 | 236.067
17 | 236 | 241 | 233.9 | 236.067
19 | 236 | 241 | 233.9 | 236.067
20 | 236 | 241 | 233.9 | 236.067
23 | 236 | 241 | 233.9 | 236.067
25 | 236 | 241 | 233.9 | 236.067
26 | 236 | 241 | 237.3 | 240
27 | 236 | 241 | 237.3 | 240
28 | 236 | 241 | 237.3 | 240
30 | 236 | 241 | 237.3 | 240
Table 2: Showing k-values with predicted intervals

Finally, this matrix is fed to the gseg2 algorithm, which outputs the
statistically significant change point interval. Subsequently, the interval
serves as a distracted activity’s start and end time, which is classified by
the next module.

###  4.3 Experimentation: Event Classification

For VQA, using prompt engineering, we have found the following question works
the best for our task. In this process, we have crafted three questions, and
they are :

1\. ”Based on the following activities: Normal Forward Driving, Drinking,
Phone Call (right), Phone Call (left), Eating, Text (Right), Text (Left),
Reaching behind, Adjusting control panel, Pick up from floor (Driver), Pick up
from floor (Passenger), Talk to passenger at the right, Talk to passenger at
backseat, Yawning, Hand on head, and Singing or dancing with music, which
activity is being performed in the video?”

2.”Is the driver simulating any of the following activities? 1. Normal Forward
Driving, 2. Pretending to drink a beverage, 3. Simulating a phone call with
the right hand, 4. Simulating a phone call with the left hand, 5. Pretending
to eat food, 6. Simulating texting with the right hand, 7. Simulating texting
with the left hand, 8. Pretending to reach behind the seat, 9. Simulating
adjusting the control panel, 10. Pretending to pick up an object from the
floor on the driver’s side, 11. Pretending to pick up an object from the floor
on the passenger’s side, 12. Simulating talking to a passenger seated on the
right side, 13. Simulating talking to a passenger seated in the backseat, 14.
Simulating yawning, 15. Pretending to place a hand on the head, 16. Simulating
singing or dancing to music. Please provide a ’yes’ or ’no’ response for each
activity.”

3.”Is the driver simulating any of the following activities? 1. Normal Forward
Driving, 2. Pretending to drink a beverage, 3. Simulating a phone call with
the right hand, 4. Simulating a phone call with the left hand, 5. Pretending
to eat food, 6. Simulating texting with the right hand, 7. Simulating texting
with the left hand, 8. Pretending to reach behind the seat, 9. Simulating
adjusting the control panel, 10. Pretending to pick up an object from the
floor on the driver’s side, 11. Pretending to pick up an object from the floor
on the passenger’s side, 12. Simulating talking to a passenger seated on the
right side, 13. Simulating talking to a passenger seated in the backseat, 14.
Simulating yawning, 15. Pretending to place a hand on the head, 16. Simulating
singing or dancing to music. Please provide the activity the driver is doing.”

We utilized this question for all the videos we trimmed based on the start and
end time calculated from our Key Point Change Detection process. The testing
dataset has 30 videos comprising 450 activities, comprising 15 activities per
video. The model answered every activity based on the start and end time. If
the model detects no activity from the list provided in the question, it
responds with ”no.” We verified the accuracy of each question using the ground
truth.

###  4.4 Evaluation metric.

We have evaluated our approach using the AICITY CHALLENGE evaluation [1]. The
AICITY evaluation is defined as follows: Given a ground-truth activity g with
start time gs and end time ge, the aim is to find its closest predicted
activity match as that predicted activity p belonging to the same class as g
with the highest overlap score os. Additional condition is imposed that start
time ps and end time pe fall within the range [gs – 10s, gs \+ 10s] and [ge –
10s, ge + 10s], respectively. The overlap between g and p is calculated as the
ratio between the time intersection and the time union of the two activities,
i.e.,

Figure 5: showing overlap score

##  5 Results

###  5.1 Performance of event detection

This module’s main objective is to predict an activity’s intervals, start and
end times. We experimented with the gseg2 algorithm for different values of n
(number of data samples), l0, and l1, as shown in Table 3. The results are
displayed for 30 video samples, which had a total of 450 activity intervals.

n-data samples | Accurate predictions | Accuracy % | L0, L1
---|---|---|---
1 minute | 242 | 51 | 0.1, 0.90
2 minutes | 235 | 49 | 0.1, 0.90
3 minutes | 228 | 47 | 0.1, 0.90
4 minutes | 176 | 42 | 0.1, 0.90
Table 3: Showing results for different n

We have used the data samples from 1, 2, 3, and 4 minutes, and to address
scenarios where activities extend beyond these time intervals, such as an
activity commencing just before the 1-minute and continuing into the next
minute, we re-executed the algorithm using data samples starting from 30
seconds for 1 minute. Similar approaches were applied for other duration’s to
accommodate boundary conditions.

We have evaluated our approach using the AICITY CHALLENGE evaluation [1].
Although the AICITY CHALLENGE evaluation is designed for final results
(activity recognition with start and end times), we have leveraged it to
assess the accuracy of intervals. This unconventional use of the evaluation
metric aims to ensure our approach’s robustness. We reasoned that if our
approach accurately predicted start and end times, it would likely also enable
the activity recognition algorithm to identify activity types effectively.

(a) Hand on Head

(b) Phone call-left

(c) Picking from floor

(d) Phone call-right

Figure 6: Showing few classes where the change-point detection produced
correct action proposals

(a) Yawning

(b) Text left

(c) Talking

(d) Text right

Figure 7: Showing few classes where the change-point detection could not
produced correct action proposals

We have specifically used the condition which states that start time ps and
end time pe are in the range [gs – 10s, gs \+ 10s] and [ge – 10s, ge + 10s],
respectively, to find the accuracy of the predicted intervals. We used this
approach to find the accuracy of action proposals of the last years top
performing team, the results are shown in Table 4.

Approach | Accuracy %
---|---
Meituan-IoTCV [35] | 40
change-point-detection (ours) | 51
Table 4: Showing action proposal prediction accuracy

####  5.1.1 Error Analysis of the change point algorithm

The algorithm have worked better for few classes while it failed for others.
Specifically, the algorithm picks the classes that involves movements of hands
or body by a large amount. For example the classes ”Hand on head” ,”phone
calls”, ”picking from floor” is mostly predicted by the algorithm while
”texting” ”yawning”, ”singing”,”talking” is not predicted most of the time.
One of the reason for this is the presence of noise in the data (key points)
which makes it difficult to differentiate anomaly from normal.

Figure 6 shows the classes where the algorithm performed well. The changes in
key points are visible for the left hand and the right hand. Similarly, Figure
7 shows classes where the model didn’t perform well because of the noise in
the data or because the activity didn’t involve much movement of hand or head.
For example, ”talking”, ”singing” and ”yawning” do not involve any movements.

Overall, the change point algorithm worked well because it was able to locate
the activity start and times given that there was noise in the data and some
of the activities did not cause any changes in the key points.

###  5.2 Performance of event classification

We conducted experiments to evaluate the performance of our event
classification module using VideoChatGPT, which is a tailored version of
Video-LLM. VideoChatGPT is designed to detect distracted behaviors in video
clips and followed the exact same way it is in the official repository. To
assess the accuracy of the model for classifying the identified distracted
driving activities, we crafted three comprehensive questions using prompt
engineering in previous section. The questions cover all sixteen activities
and are designed to evaluate the model’s ability to classify the activities
accurately based on start and end times determined by the Key Point Change
Detection process. AS we mentioned before, the testing dataset consisting of
30 videos, each containing 15 activities, resulting in a total of 450
activities. For each activity, the model provided a response based on the
specified time intervals. If the model was unable to detect any activity from
the provided list, it responded with ”no.” We verified the accuracy of each
question against the ground truth.

Question Number | Accurate Predictions | Accuracy(%)
---|---|---
1 | 223 | 49.5
2 | 249 | 55.3
3 | 259 | 57.5
Table 5: Showing Model Performance on three questions

In Table 5, it is clear that the accuracy peak was reached with the third
question, achieving an impressive 57.5%. This milestone underscores the
substantial impact of our refined prompt engineering technique on bolstering
VideoChatGPT’s classification capabilities. The discernible upward trend in
accuracy rates from the initial to the third question highlights the model’s
enhanced comprehension of the tasks at hand, thanks to the iterative
refinement of the prompts. These findings underscore VideoChatGPT’s promising
utility in accurately classifying distracted driving behaviors, showcasing its
potential in real-world applications. To the best of our knowledge, this is
the first time Video-LLMs have been used for this task. Therefore, we did not
report a baseline score.

###  5.3 Conclusion

Our DeepLocalization model is an effective solution for detecting and locating
distracted driver behavior in real time. The approach we use involves key-
point extraction, change-point detection, and video language modeling to
precisely identify and temporally localize a wide range of driver activities.
Our experimental results on the SynDD2 dataset demonstrate the effectiveness
of our approach in addressing the complex problem of distracted driving with
limited resources. In the future, to improve the accuracy of event detection
and classification, we plan to introduce a more robust key point detection
algorithm, which is currently the bottleneck in achieving better accuracy.
Additionally, fine-tuning for one or two epochs is a viable way to improve
accuracy even on a consumer-grade GPU.

## References

  * [1] https://www.aicitychallenge.org/2024-data-and-evaluation/.
  * [2] https://www.rdocumentation.org/packages/gSeg/versions/0.1/topics/gseg2.
  * noa [2023] Distracted Driving | Transportation Safety | Injury Center | CDC, 2023.
  * Chao et al. [2018] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the Faster R-CNN Architecture for Temporal Action Localization.  pages 1130–1139, 2018.
  * Chen et al. [2021] Guo Chen, Yin-Dong Zheng, Limin Wang, and Tong Lu.  Dcan: Improving temporal action detection via dual context aggregation, 2021.
  * Chen and Zhang [2015] Hao Chen and Nancy Zhang.  Graph-based change-point detection.  _The Annals of Statistics_ , 43(1):139 – 176, 2015.
  * Chu and Chen [2019] Lynna Chu and Hao Chen.  Asymptotic distribution-free change-point detection for multivariate and non-Euclidean data.  _The Annals of Statistics_ , 47(1):382 – 414, 2019.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  * Dingus et al. [2016] Thomas A Dingus, Fang Guo, Suzie Lee, Jonathan F Antin, Miguel Perez, Mindy Buchanan-King, and Jonathan Hankey.  Driver crash risk factors and prevalence evaluation using naturalistic driving data.  _Proceedings of the National Academy of Sciences of the United States of America_ , 113(10):2636–2641, 2016.
  * Feichtenhofer [2020] Christoph Feichtenhofer.  X3D: Expanding Architectures for Efficient Video Recognition.  pages 203–213, 2020.
  * Feichtenhofer et al. [2019] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  SlowFast Networks for Video Recognition.  pages 6202–6211, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  pages 770–778, 2016.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in Neural Information Processing Systems_ , 25:1097–1105, 2012.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * Lin et al. [2018] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation, 2018.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * Maaz et al. [2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.  Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023.
  * Naphade et al. [2023] Milind Naphade, Shuo Wang, David C. Anastasiu, Zheng Tang, Ming-Ching Chang, Yue Yao, Liang Zheng, Mohammed Shaiqur Rahman, Meenakshi S. Arya, Anuj Sharma, Qi Feng, Vitaly Ablavsky, Stan Sclaroff, Pranamesh Chakraborty, Sanjita Prajapati, Alice Li, Shangru Li, Krishna Kunadharaju, Shenxin Jiang, and Rama Chellappa.  The 7th AI City Challenge.  pages 5537–5547, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Rahman et al. [2023a] Mohammed Shaiqur Rahman, Archana Venkatachalapathy, Anuj Sharma, Jiyang Wang, Senem Velipasalar Gursoy, David Anastasiu, and Shuo Wang.  Synthetic distracted driving (syndd1) dataset for analyzing distracted behaviors and various gaze zones of a driver.  _Data in Brief_ , 46:108793, 2023a.
  * Rahman et al. [2023b] Mohammed Shaiqur Rahman, Jiyang Wang, Senem Velipasalar Gursoy, David Anastasiu, Shuo Wang, and Anuj Sharma.  Synthetic distracted driving (syndd2) dataset for analyzing distracted behaviors and various gaze zones of a driver, 2023b.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the Inception Architecture for Computer Vision.  pages 2818–2826, 2016.
  * Transportation Research Board [2015] Transportation Research Board.  Second strategic highway research program naturalistic driving study (shrp 2).  2015\.
  * Venkatachalapathy et al. [2023] Archana Venkatachalapathy, Mohammed Shaiqur Rahman, Aditya Raj, Jennifer Merickel, Anuj Sharma, Jiyang Wang, and Senem Velipasalar.  Deep Insight: A Cloud Based Big Data Analytics Platform For Naturalistic Driving Studies: Data Management, Analytics, and Automated Annotations.  _International Journal of Automotive Engineering_ , 14(3):66–76, 2023.
  * Wang et al. [2023] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors.  pages 7464–7475, 2023.
  * Wang et al. [2021] Lining Wang, Haosen Yang, Wenhao Wu, Hongxun Yao, and Hujie Huang.  Temporal Action Proposal Generation with Transformers, 2021.  arXiv:2105.12043 [cs].
  * Xu et al. [2021] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer.  VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding, 2021.
  * Xu et al. [2020] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-tad: Sub-graph localization for temporal action detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10156–10165, 2020.
  * Yuan et al. [2021] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.  Florence: A new foundation model for computer vision, 2021.
  * Yuan et al. [2017] Zehuan Yuan, Jonathan C. Stroud, Tong Lu, and Jia Deng.  Temporal action localization by structured maximal sums.  In _Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017_ , pages 3215–3223, United States, 2017. Institute of Electrical and Electronics Engineers Inc.  Funding Information: This work is partially supported by a University of Michigan graduate fellowship, the Natural Science Foundation of China under Grant No. 61672273, No. 61272218, and No. 61321491, and the Science Foundation for Distinguished Young Scholars of Jiangsu under Grant No. BK20160021. Publisher Copyright: © 2017 IEEE.; 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 ; Conference date: 21-07-2017 Through 26-07-2017.
  * Zeng et al. [2019a] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7094–7103, 2019a.
  * Zeng et al. [2019b] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph Convolutional Networks for Temporal Action Localization.  pages 7094–7103, 2019b.
  * Zhao et al. [2021] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian.  Bottom-up temporal action localization with mutual regularization, 2021.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Zhou et al. [2023] Wei Zhou, Yinlong Qian, Zequn Jie, and Lin Ma.  Multi view action recognition for distracted driver behavior localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5374–5379, 2023.


# Flashlights: More than A Dozen High-Significance Microlensing Events of
Extremely Magnified Stars in Galaxies at Redshifts $z=0.7$–$1.5$

Patrick L. Kelly<EMAIL_ADDRESS>Minnesota Institute for Astrophysics,
University of Minnesota, 116 Church Street SE, Minneapolis, MN 55455, USA
Wenlei Chen Minnesota Institute for Astrophysics, University of Minnesota, 116
Church Street SE, Minneapolis, MN 55455, USA  Amruth Alfred  Department of
Physics, The University of Hong Kong, Pokfulam Road, Hong Kong  Thomas J.
Broadhurst Ikerbasque Foundation, University of the Basque Country, DIPC
Donostia, Spain  Jose M. Diego IFCA, Instituto de Física de Cantabria (UC-
CSIC), Av. de Los Castros s/n, 39005 Santander, Spain  Najmeh Emami  Minnesota
Institute for Astrophysics, University of Minnesota, 116 Church Street SE,
Minneapolis, MN 55455, USA  Alexei V. Filippenko Department of Astronomy,
University of California, Berkeley, CA 94720-3411, USA  Allison Keen
Minnesota Institute for Astrophysics, University of Minnesota, 116 Church
Street SE, Minneapolis, MN 55455, USA  Sung Kei Li Department of Physics, The
University of Hong Kong, Pokfulam Road, Hong Kong  Jeremy Lim  Department of
Physics, The University of Hong Kong, Pokfulam Road, Hong Kong  Ashish K.
Meena Physics Department, Ben-Gurion University of the Negev, P.O. Box 653,
Beer-Sheva 8410501, Israel  Masamune Oguri Center for Frontier Science, Chiba
University, 1-33 Yayoi-cho, Inage-ku, Chiba 263-8522, Japan; Department of
Physics, Graduate School of Science, Chiba University, 1-33 Yayoi-Cho, Inage-
Ku, Chiba 263-8522, Japan  Claudia Scarlata  Minnesota Institute for
Astrophysics, University of Minnesota, 116 Church Street SE, Minneapolis, MN
55455, USA  Tommaso Treu Department of Physics and Astronomy, University of
California, Los Angeles, CA 90095  Hayley Williams  Minnesota Institute for
Astrophysics, University of Minnesota, 116 Church Street SE, Minneapolis, MN
55455, USA  Liliya L.R. Williams Minnesota Institute for Astrophysics,
University of Minnesota, 116 Church Street SE, Minneapolis, MN 55455, USA  Rui
Zhou  Minnesota Institute for Astrophysics, University of Minnesota, 116
Church Street SE, Minneapolis, MN 55455, USA  Adi Zitrin Physics Department,
Ben-Gurion University of the Negev, P.O. Box 653, Beer-Sheva 8410501, Israel
Ryan J. Foley  Department of Astronomy and Astrophysics, UCO/Lick Observatory,
University of California, 1156 High Street, Santa Cruz, CA 95064, USA  Saurabh
W. Jha Department of Physics and Astronomy, Rutgers, The State University of
New Jersey, Piscataway, NJ 08854, USA  Nick Kaiser  Laboratoire de Physique de
l’Ecole Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université,
Université de Paris, Paris, France  Vihang Mehta  Minnesota Institute for
Astrophysics, University of Minnesota, 116 Church Street SE, Minneapolis, MN
55455, USA  Steven Rieck  Minnesota Institute for Astrophysics, University of
Minnesota, 116 Church Street SE, Minneapolis, MN 55455, USA  Laura Salo
Minnesota Institute for Astrophysics, University of Minnesota, 116 Church
Street SE, Minneapolis, MN 55455, USA  Nathan Smith Department of Astronomy,
University of Arizona, Tucson, AZ 85721, USA  Daniel R. Weisz  Department of
Astronomy, University of California, Berkeley, CA 94720-3411, USA

###### Abstract

Once only accessible in nearby galaxies, we can now study individual stars
across much of the observable universe aided by galaxy-cluster gravitational
lenses. When a star, compact object, or multiple such objects in the
foreground galaxy-cluster lens become aligned, they can magnify a background
individual star, and the timescale of a magnification peak can limit its size
to tens of AU. The number and frequency of microlensing events therefore opens
a window into the population of stars and compact objects, as well as high-
redshift stars. To assemble the first statistical sample of stars in order to
constrain the initial mass function (IMF) of massive stars at redshift
$z=0.7$–1.5, the abundance of primordial black holes in galaxy-cluster dark
matter, and the IMF of the stars making up the intracluster light, we are
carrying out a 192-orbit program with the Hubble Space Telescope called
“Flashlights,” which is now only two-thirds complete owing to scheduling
challenges. We use the ultrawide F200LP and F350LP long-pass WFC3 UVIS filters
and conduct two 16-orbit visits separated by one year. Having an identical
roll angle during both visits, while difficult to schedule, yields extremely
clean subtraction. Here we report the discovery of more than a dozen bright
microlensing events, including multiple examples in the famous “Dragon Arc”
discovered in the 1980s, as well as the “Spocks” and “Warhol” arcs that have
hosted already known supergiants. The ultradeep observer-frame ultraviolet-
through-optical imaging is sensitive to hot stars, which will complement deep
James Webb Space Telescope infrared imaging. We are also acquiring Large
Binocular Telescope LUCI and Keck-I MOSFIRE near-infrared spectra of the
highly magnified arcs to constrain their recent star-formation histories.

††facilities: HST (WFC3); Large Binocular Telescope (LUCI); Keck
(MOSFIRE)††software: Drizzlepac (Gonzaga, 2012); Source Extractor (Bertin &
Arnouts, 1996)

##  1 Introduction

Extreme gravitational lensing magnification (up to several thousand) of
individual high-redshift stars by a foreground galaxy cluster, a newly
discovered phenomenon (Kelly et al., 2018; Rodney et al., 2018; Chen et al.,
2019; Kaurov et al., 2019; Welch et al., 2022; Diego et al., 2022a; Chen et
al., 2022; Diego et al., 2022b), has the ability to address three major
outstanding questions. (a) What is dark matter? Despite decades of searches,
the constituents of dark matter remain unidentified. (b) How was the universe
reionized? The magnifying power of galaxy-cluster lenses holds great potential
for studying faint, high-redshift galaxies, provided one can construct
sufficiently accurate lens models (e.g., Bouwens et al. 2017). (c) How do
stellar populations at redshift $z=1$–2 differ from those in the nearby
universe? We have only been able to study the composite spectral energy
distributions (SEDs) of their luminous stars.

To address the major questions outlined above, we are carrying out ultradeep
Hubble Space Telescope (HST) observations of the six Hubble Frontier Field
(HFF; Lotz et al. 2017) clusters. These targets – Abell 2744,
MACSJ0416.1-2403, MACSJ0717.5+3745, MACSJ1149.5+2223, Abell S1063, and Abell
370 – are among the most powerful gravitational lensing clusters and have
multiband optical through infrared (IR) HST imaging observations as part of
the 840-orbit HFF program. The HFF survey obtained optical imaging of each
galaxy-cluster field in two-orbit visits spread across a period of
approximately a month. For a single two-orbit visit using the ACS WFC F6060W
filter, the 5$\sigma$ limiting AB (Oke & Gunn, 1983) magnitude was 28.1 within
a $0\farcs 2$-radius aperture (28.7 for an “optimal” extraction), according to
the Exposure Time Calculator
(ETC)111https://etc.stsci.edu/etc/input/acs/imaging/. Considering a single
week of HFF visits amounting to a total of six orbits, the limiting magnitude
in the F606W filter, for example, was 28.7 (or 29.3 for an optimal
measurement).

Since the expected timescale for a $\sim 10$ R⊙ star to cross a caustic can be
only hours (Miralda-Escude, 1991), we aimed to carry out a two-epoch HST
program of WFC3 UVIS observations (see Fig. 1) of galaxy clusters with a
single-visit 5$\sigma$ limiting magnitude of 30 (or optimal 31.1), $\sim 2$
mag deeper than each HFF visit, and $\sim 1.3$ mag deeper than each week of
HFF visits. While the HFF obtained imaging from the optical through the near-
infrared, our program includes observations with the ultraviolet-through-
optical F200LP filter for sensitivity to emission from hot stars. The aim of
the program is to find the first statistical samples of both microlensing
events and pairs of magnified stellar images.

Close to a so-called fold caustic, the area in the source plane with
magnification exceeding a value $\mu$ scales as $\mu^{-2}$. By carrying out a
survey with single-visit sensitivity $\sim 1.3$ mag (a factor of $\sim 3$ in
flux) greater than that of a single week of HFF imaging, we require
magnifications that are an equal factor of 3 smaller for detection.
Consequently, to first order, a factor of $\sim 9$ greater number of stars
through their microlensing events can be expected. For microlensing events
with durations of only hours, the $\sim 2$ mag difference in sensitivity
corresponds to a factor of $\sim 36$. The extremely deep imaging also enables
several additional high-impact investigations, and is needed to interpret
James Webb Space Telescope (JWST) observations of the HFFs.

In Section 2, we describe the principal science motivation for the Flashlights
survey. Four of the six target HFF galaxy-cluster fields have now been visited
twice, and Section 3 presents the high-significance microlensing events that
we have detected. We discuss conclusions and implications from the current
analysis of the data in Section 4.

Figure 1: The survey will observe each of six HFF galaxy clusters at two
epochs. During each 16-orbit visit, imaging alternates between orbits with
integrations in WFC3 F200LP and WFC3 F350LP. The full throughputs of ACS WFC
CLEAR and WFC3 F200LP as well as the SEDs of two stars at $z=1$ are plotted
here. The differences between the two filters provides constraints on the SED
(including the reddening) of each star.

##  2 Science Drivers

###  2.1 Dark Matter

Despite decades of searches, the constituents of $\sim 85$% of the matter in
the universe remain unidentified. The newly discovered extreme magnification
of high-redshift stars provides an entirely new and highly sensitive probe of
the nature of dark matter. First, the potential of a galaxy cluster acts to
exaggerate the Einstein radii of objects in its intracluster medium by factors
of up to $\sim 100$ near its critical curves, dramatically enlarging
microlensing ability (Diego et al., 2018; Venumadhav et al., 2017). Second,
light traveling through a cluster (or a galaxy) near its critical curve
traverses an extreme density of dark matter. Consequently, microlensing
fluctuations of background stars should reveal even a small fraction (1–2%) of
dark matter in the form of compact objects (Oguri et al., 2018).

Figure 2: Effect of wave dark matter on gravitational lensing magnification
adjacent to the galaxy cluster’s critical curve. Left panel: the magnification
surrounding the critical curve of a galaxy cluster with mass similar to that
of Abell 370 given a simply-parameterized smooth dark matter halo at the
cluster scale and for individual cluster members. Middle panel: the
magnification when the dark matter consists of ultralight bosons ($\psi$DM).
Right panel: projected density fluctuations arising from $\psi$DM along with
contours indicating regions of magnification of over $10^{3}$. Figure 3:
Microlensing events within $<0\farcs 1$ of the galaxy cluster critical curve
in an arc at $z=0.94$ in the MACS J0416 field. The upper panel shows a color-
composite image from HFF imaging, the middle two panels show Flashlights
F200LP imaging from 2020 and 2021, and the bottom panel indicates the
difference of the two epochs. The peak visible in the bottom panel that is
farthest to the right corresponds to one of the images of a highly magnified
blue supergiant, “Warhol” (Chen et al., 2019; Kaurov et al., 2019). Red
hatches correspond to peaks detected in the 2021 imaging, and blue hatches
correspond to those present in the 2020 imaging. Yellow dashed lines mark
approximately the line of symmetry of the arc, which corresponds closely to
the critical curve. The red solid line marks the location of the galaxy
cluster’s critical curve according to the simply-parameterized Keeton model
(v4) available from the HFF website (Keeton, 2010), although the model does
not use the pairs of knots visible in the arc as constraints.  Figure 4:
Microlensing events in the “Spocks” arc at $z=1.00$ in the MACS J0416 galaxy-
cluster field. We overplot the critical curve from the Zitrin Light-Traces-
Mass model (v1) (Zitrin et al., 2013) in red, and the Sharon (Johnson et al.,
2014) simply-parameterized model (v4cor) in blue. The locations of the two
transients reported by Rodney et al. (2018) in HFF imaging acquired in 2014
are shown with yellow circles. The positions of the two high-significance
events detected in Flashlights imaging are at distinct positions, and may
correspond to counterimages of the Rodney et al. (2018) events.  Figure 5:
Three high-significance microlensing events in a giant arc at $z=1.26$ in the
Abell S1063 galaxy cluster field. Yellow dashed lines mark approximately the
line of symmetry of the arc, which corresponds closely to the critical curve.
The plotted critical curve is that from the Keeton HFF simply-parameterized
model (v4) (Keeton, 2010).  Figure 6: Two high-significance candidate
microlensing events in a giant arc at $z=0.73$ in the Abell S1063 galaxy
cluster field. The plotted critical curve is that from the Williams free-form
Grale (Ghosh et al., 2021) HFF model (v4.1). Figure 7: High-significance
microlensing events adjacent to a model critical curve in the famous “Dragon”
giant arc at $z=0.725$ (Soucail et al., 1988). The critical curve for the
WSLAP+ model (v4.1) (Diego et al., 2005, 2007, 2016) is shown in blue, and
that for the simply-parameterized GLAFIC (Kawamata et al., 2018, 2016; Oguri,
2010) code (v4) is in red. The locations of the smaller critical curves
associated with cluster members are not yet well constrained.

Primordial black holes (PBHs) could have formed from perturbations in (for
example) an inflaton or spectator field in the early universe. Currently,
Galactic (Tisserand et al., 2007; Alcock et al., 2001) and quasar (Mediavilla
et al., 2017) microlensing could allow PBHs with $10^{-3}$
M${}_{\odot}<M<10^{2}$ M⊙ to account for up to 3–10% of dark matter. Limits on
their abundance place constraints on the primordial power spectrum associated
with inflation (e.g., Carr et al. 2017).

The Laser Interferometer Gravitational-Wave Observatory (LIGO) is now able to
detect the coalescence of neutron-star (Abbott et al., 2017) and black-hole
(Abbott et al., 2016a, b) binary systems. LIGO and surveys of X-ray binary
systems, however, are not sensitive to the populations of either single black
holes or binary systems with wide separations.

At low optical depths for microlensing, the frequency of microlensing events
should increase in approximately direct proportion to the mass density of
microlenses. Moreover, the density of foreground intracluster stars along the
line of sight toward giant arcs with $z=0.7$–1.5 that straddle the critical
curve are $\sim 1$% of the projected density of dark matter. Therefore, the
number of microlensing events will provide a straightforward test of the
abundance of PBHs comprising $\sim 1$% of dark matter across an extremely
broad mass range of at least $10^{-5}<m/{\rm M}_{\odot}<100$.

When a strongly lensed star lies close to a fold caustic in the source plane,
its two most highly magnified images will appear as a pair straddling the
critical curve. When an object in the cluster becomes temporarily aligned with
a background lensed star, one of these images brightens (or fades) temporarily
as has been observed for both Icarus (Kelly et al., 2018) and Warhol (Chen et
al., 2019; Kaurov et al., 2019). Dai et al. (2018) show that the presence of
subhalos such as those expected in the $\Lambda$CDM paradigm should imprint
detectable $0\farcs 03$–$0\farcs 1$ deflections in the positions of pairs of
lensed images of background stars, corresponding to halos of $\sim
10^{7}$–$10^{9}$ M⊙. Low-mass halos have been claimed to be identified in
optical observations of galaxy-scale gravitational lenses (Vegetti et al.,
2010, 2012) as well as at submillimeter wavelengths in the well-resolved
Einstein ring galaxy SDP.81 (Hezaveh et al., 2016). A complementary
statistical approach has found evidence for subhalos from analysis of flux
anomalies (Nierenberg et al., 2014; Gilman et al., 2020). In principle,
several magnified stars along a giant arc may be sufficient to detect the
presence of such subhalos depending on the degree of tidal stripping.

In the context of “wave” dark matter ($\psi$DM), consisting of ultralight
bosons, substructure is pervasive, caused by interference on the de Broglie
scale within galaxy and cluster halos (Schive et al., 2014; Hui et al., 2017;
Mocz et al., 2017). The de Broglie wavelength sets the scale of substructure
fluctuations, ranging from several parsecs for clusters to $\sim 1$ kpc for
dwarf galaxies for a boson mass of $\sim$10-22 eV (Pozo et al., 2021) as the
de Broglie wavelength scales inversely with momentum. The effect of this
density modulation for lensing is predicted to produce highly corrugated
critical curves (Chan et al., 2020). As shown in Fig. 2, we can approximate
the projected density field expected for a massive galaxy cluster using the
same procedure for galaxies as described by Alfred et al. (2022, submitted).
Notice in particular how small “islands” of high magnification are expected to
be significantly offset from the location of a galaxy cluster’s critical
curve, surrounding individual positive and negative density fluctuations that
are depicted as red and blue (respectively). Consequently, individual stars
may be highly magnified at projected positions that are relatively far from
the Einstein radius, providing a testable prediction of $\psi$DM.

###  2.2 Improving Galaxy Clusters as Tools

Near the critical curves of galaxy clusters, JWST should be sensitive to
sources as faint as $\sim 35$ AB mag, making it possible in principle to
measure the properties of the low-luminosity galaxies thought to drive
reionization. In areas of high magnification, it is crucial to obtain
sufficient information to constrain the lens models well enough to determine
accurate luminosities and volumes. Achieving this goal requires deep high-
resolution images to identify dozens of multiply-imaged systems, as well as
spectroscopic confirmations of large numbers of them.

In the image plane on the sky, magnification falls steeply as the inverse of
the distance from the critical curve. In regions that currently lack
sufficient constraints, however, lens models can disagree about the location
of the critical curve by of order an arcsecond, which is a principal factor
that explains the uncertain magnification of some high-redshift objects. As
Icarus and Warhol demonstrate, each of the expected pairs of highly magnified
stellar images (and those of star clusters) will pinpoint the location of the
critical curve. Furthermore, the probability of a microlensing event with a
given peak magnification should drop approximately as the inverse of the
offset from the critical curve to the fourth power (Kelly et al., 2018).
Therefore, the microlensing events will also trace critical curves, albeit not
as exactly.

###  2.3 Stellar Populations in Galaxies at $z=1$–2

Galaxies near cosmic noon (at $z=1$–2) are more intensely star forming and
exhibit more extreme nebular emission-line ratios than nearby galaxies (e.g.,
Steidel et al., 2014; Sanders et al., 2015). In nearby galaxies, we can
resolve individual massive stars, and thereby constrain their luminosity
functions and the upper end of the initial mass function (IMF). Currently,
however, we are only able to study the composite SEDs of $z=1$–2 stellar
populations, which limits our ability to constrain the upper end of the IMF.
Improved constraints are necessary, because luminous stars are responsible for
driving the evolution of galaxies through their ionizing flux, winds, and
energy input from supernovae. Moreover, the IMF is a key parameter that
affects interpretation of the observed properties of galaxy populations at
high redshift (e.g., Narayanan & Davé, 2012).

The number of detections of microlensing events should depend on the stellar
luminosity function (see Extended Data Fig. 5 of Kelly et al. 2018), and
therefore the IMF. Consequently, the microlensing events detected by
Flashlights will make it possible to measure the stellar luminosity function
and upper end of the IMF across star-forming $z=1$–2 galaxies that cross the
critical curve.

The survey includes two visits to each cluster consisting of 16 contiguous
orbits (or as close as possible), alternating between orbits with WFC3 UVIS
F350LP and WFC3 UVIS F200LP integrations. Our initial strategy was to use ACS
WFC CLEAR instead of WFC3 UVIS F350LP, but the point-spread function (PSF) was
degraded in our first observations, of Abell 370. The two visits to each HFF
cluster are separated by one year, so that we can take parallel field
observations at the same roll angle which will serve as a control experiment.

A microlensing peak should have a duration $R/v$, where $R$ is the size of the
lensed source and $v$ is the transverse velocity of the lensing system
(Miralda-Escude, 1991). Given the $\sim 1000$ km s-1 expected transverse
velocity of the cluster lens system, the several-week timescale of Icarus’ May
2016 microlensing peak (Kelly et al., 2018) implies that the source only
extends for at most several tens of AU.

###  2.4 Enabling JWST Searches for Dropout Galaxies

A key objective of JWST is to answer the question of how the universe became
reionized at $z>6$ (e.g., Castellano et al. 2016). One main observable
required to study the history of reionization is the rest-frame UV galaxy
luminosity function, which constrains the total star-formation rate (SFR) at
any given redshift. The principal technique for measuring the luminosity
functions of high-redshift galaxies is to identify Lyman-break “dropout”
galaxies. The ionizing continuum (below 912 Å) of galaxies is strongly
absorbed by neutral H I gas in the interstellar medium and in the surrounding
circumgalactic and intergalactic medium. Additionally, the continuum between
912 and 1216 Å is attenuated by the Ly$\alpha$ forest, created in discrete
systems along the line of sight (Giavalisco, 2002).

Typical Lyman-break colors (i.e., color difference between wavelengths redder
and bluer than Ly$\alpha$) used to select $z>6$ galaxies amount to $\gtrsim 1$
mag (e.g., Finkelstein 2016, for a review). The Lyman break, however, can be
mimicked by low-redshift faint galaxies showing intrinsically red colors,
including from a strong Balmer break, across the filters used to perform the
selection. These red low-redshift interlopers include $z\approx 2$ passive
galaxies, dusty star-forming objects, or strong emission-line starbursting
dwarfs (e.g., Livermore et al. 2018, Atek et al. 2011). A classic example is
the putative $z\approx 11$ object identified by Laporte et al. (2011) as a
J-band dropout. In follow-up work, Hayes et al. (2012) demonstrated
spectroscopically that the object in fact has $z\approx 2.1$, either a young
heavily reddened starburst or a maximally old system with a very pronounced
4000 Å break. Had AB $\approx 30$ optical imaging (blueward of the Lyman
break) been available, this low-redshift contaminant could have been
identified (Hayes et al., 2012).

Reaching AB magnitudes of 30, the observations provide six benchmark fields
for high-redshift galaxy searches with JWST. Specifically, the data will be
unique to identify the population of $z\approx 2$ galaxies with colors
mimicking those of $z=7$–12 candidates.

Cluster | RA | Dec | $z$ | F200LP | F350LP | $\mu$ | $\mu$ | Approx. CC
---|---|---|---|---|---|---|---|---
| (deg.) | (deg.) |  | Signif. | Signif. |  GLAFIC* |  Gravlens* | Proximity
Abell 370 | 39.9704121 | -1.5848997 | 0.73a | 19.0 | … | 1700 | 5800 |  $\lesssim$0.1′′
Abell 370 | 39.9718583 | -1.5848133 | 0.73a | 5.2 | … | 60 | 26 |  $\lesssim$0.3′′
Abell 370 | 39.9711217 | -1.5848492 | 0.73a | 3.3 | … | 30 | 28 |  $\lesssim$0.3′′
Abell 370 | 39.9702708 | -1.5848842 | 0.73a | 4.7 | … | 170 | 89 |  $\lesssim$0.1′′
Abell 370 | 39.9719888 | -1.5848139 | 0.73a | 4.5 | … | 50 | 15 |  $\lesssim$0.3′′
Abell 370 | 39.9710843 | -1.5848642 | 0.73a | 5.1 | … | 25 | 20 |  $\lesssim$0.3′′
Abell 370 | 39.9691621 | -1.5846231 | 0.73a | 5.3 | … | 15 | 10 |  $\lesssim$0.1′′
Abell 370 | 39.9672042 | -1.5849367 | 1.26b | 5.9 | … | 44 | 60 |  $\sim$0.6′′
Abell S1063 | 342.1925167 | -44.5304881 | 1.26c | 5.8 | 3.3 | 110 | 100 |  $\lesssim$0.1′′
Abell S1063 | 342.1924193 | -44.5304881 | 1.26c | 31.0 | 16.3 | 190 | 180 |  $\lesssim$0.3′′
Abell S1063 | 342.1924847 | -44.5304337 | 1.26c | 7.1 | 3.5 | 250 | 210 |  $\lesssim$0.2′′
Abell S1063 | 342.1894900 | -44.5290447 | 0.73d | 7.2 | 4.9 | 28 | 32 |  $\lesssim$1′′
Abell S1063 | 342.1890363 | -44.5286928 | 0.73d | 8.2 | 0.0 | 20 | 27 |  $\lesssim$3′′
Abell S1063 | 342.1952641 | -44.5279212 | 1.23e | 10.4 | 8.0 | 550 | 450 |  $\lesssim$0.1′′
MACS J0416 | 64.0388904 | -24.0701557 | 1.01f | 18.7 | 12.7 | 55 | 77 |  $\lesssim$0.2′′
MACS J0416 | 64.0386110 | -24.0699715 | 1.01f | 18.7 | 12.7 | 78 | 1200 |  $\lesssim$0.2′′
MACS J0416 | 64.0363046 | -24.0675050 | 0.94g | 10.2 | 8.3 | 25 | 60 |  $\lesssim$0.1′′
MACS J0416 | 64.0365524 | -24.0673339 | 0.94g | 5.9 | 6.7 | 30 | 70 |  $\lesssim$0.1′′
MACS J0416 | 64.0364386 | -24.0674422 | 0.94g | 2.7 | 4.5 | 21 | 50 |  $\lesssim$0.1′′
MACS J0416 | 64.0365133 | -24.0673656 | 0.94g | 2.6 | 2.6 | 29 | 60 |  $\lesssim$0.1′′
Table 1: Examples of high-significance transients in strongly lensing arcs.
(a) Soucail et al. (1988); (b) Lagattuta et al. (2017); (c) Caminha et al.
(2016); (d) Diego et al. (2016); (e) Karman et al. (2017); (f) Rodney et al.
(2018); (g) Caminha et al. (2017).

##  3 High-Significance Microlensing Events

In Table 1, we list high-significance detections of transients in a set of
giant arcs in the Flashlights targets Abell 370, MACS J0416, and Abell S1063
with repeat observations acquired as part of the Flashlights survey. There, we
describe whether the transients are found adjacent ($\lesssim 0\farcs 2$) to
the critical curve of the galaxy cluster, and therefore are very likely to be
microlensing events in regions of high magnification exceeding $\sim 100$. A
star adjacent to (and on the correct side of) a fold caustic in the source
plane will appear as a pair of highly magnified images. The relative time
delay between such a pair of highly magnified images is only of order days.
Given the typical timescales of outbursts of weeks to months of massive stars,
any strong asymmetry between the fluxes of a pair of images can only be
explained by stellar microlensing. Many of the microlensing events appear in
pairs across the critical curve, and simulations by Kelly et al. (2018) show
that the same star is often responsible for microlensing events on both sides
of the critical curve.

Fig. 3 shows the prominent arc at $z=0.94$ in the MACS J0416 galaxy-cluster
field where the “Warhol” (Chen et al., 2019; Kaurov et al., 2019) event was
discovered. The clear fold symmetry of the Warhol arc at $z=0.94$ makes it
possible to identify the location of the critical curve which corresponds to
the line of symmetry, and we detect three microlensing events in the arc. The
middle two panels show 2020 and 2021 F200LP Flashlights imaging, respectively,
while the bottom panel shows a difference image made by subtracting the
imaging acquired in 2021 from that taken in 2020. The red and blue cross
hatches indicate the locations of the high-significance transients. We
overplot, as an example of a simply parameterized prediction, the critical
curves for Light-Traces-Mass (v1) Zitrin et al. (2013) and the simply
parameterized Sharon (v4cor) Johnson et al. (2014) models available on the HFF
website222https://archive.stsci.edu/prepds/frontier/lensmodels/. The
microlensing events include detection of variability at location of the pair
of images of the “Warhol” blue supergiant.

In Fig. 4, we also plot our detection of two transients in the arc in the MACS
J0416 field of the “Spocks” ($z=1.01$) (Rodney et al., 2018) events. The
locations of the two new events identified by Flashlights differ from those of
the Spocks events whose positions we identify.

Fig. 5 shows high-significance microlensing events that we detect in
Flashlights imaging close to the critical curve of a giant arc in the Abell
S1063 galaxy-cluster field at $z=1.26$. In the case of this arc, the location
of the galaxy cluster’s critical curve is constrained by the mirrored
positions of knots. Finally, we detect two high-significance transients in an
adjacent arc in this field with a lower redshift of 0.73. Fig. 6 plots the
locations of these transients in F200LP imaging, and the critical curve we
show is that from the Williams free-form GRALE (Ghosh et al., 2021) HFF model
(v4.1).

The upper panel of Fig. 7 shows a color-composite image of the “Dragon” arc at
$z=0.73$ (Soucail et al., 1988) constructed from HFF optical and near-IR
imaging taken in 2014–2016. Superimposed are the critical curves of the WSLAP+
model (v4.1) (Diego et al., 2005, 2007, 2016) and the GLAFIC model (v4)
(Kawamata et al., 2018, 2016; Oguri, 2010). Three of the transients in the
Dragon arc are very likely to be microlensing events, given their close
proximity to the critical curve of the galaxy cluster. The four additional
transients, which appear in pairs, may also be microlensing events if they are
located sufficiently close to the critical curves of nearby cluster members.
Finally, in Meena et al. (2022), we present an analysis of a transient in an
additional arc at $z=1.26$ in the Abell 370 galaxy-cluster field that we infer
to likely be a microlensing event, despite a relatively large offset from the
critical curve.

##  4 Conclusions

We show that sufficiently deep exposures reveal many microlensing events in
individual galaxies at $z=0.7$–1.5. We anticipate that deep, repeat visits
with JWST will reveal populations of highly magnified stars. Given its
powerful near-IR sensitivity, JWST will detect much cooler stars, including
luminous red supergiants. At the same time, once completed, Flashlights will
provide a complementary dataset with sensitivity to the population of hot
stars near the peak of cosmic star formation.

Through detections of the first substantial sample of highly magnified stars,
Flashlights will enable a novel probe of the nature of dark matter as well as
the IMF of luminous stars at $z=0.7$–1.5. The positions of stars provide a new
constraint on galaxy-cluster models through accurate measurements of the
location of the cluster critical curve. Finally, the extremely deep UV through
optical wideband imaging provides a powerful means to identify low-redshift
interlopers in Lyman-break dropout samples.

We thank Patricia Royale at the Space Telescope Science Institute (STScI) for
her terrific effort scheduling the program, and our instrument scientist
Annalisa Calamida for her careful review of the program. This research was
supported by NASA/HST grants GO-15936 and GO-16278 from STScI, which is
operated by the Association of Universities for Research in Astronomy, Inc.,
under NASA contract NAS5-26555, and also by grant JPL-1659411 to support
ground-based follow-up observations with the Keck-I telescope. A.V.F. is
grateful for additional financial support from the Christopher R. Redlich Fund
and numerous individual donors. P.L.K. acknowledges support through NSF grant
AST-1908823, and NASA Keck JPL grant 1659411. This work was supported by JSPS
KAKENHI grants JP22H01260, JP20H05856, and JP20H00181. Work by S.K.L. and J.L.
is supported by the Collaborative Research Fund under grant C6017-20G, which
is issued by the Research Grants Council of Hong Kong S.A.R. A.K.M. and A.Z.
acknowledge support by grant 2020750 from the United States-Israel Binational
Science Foundation (BSF) and grant 2109066 from the United States National
Science Foundation (NSF), and by the Ministry of Science & Technology, Israel.

Some of the data presented herein were obtained at the W. M. Keck Observatory,
which is operated as a scientific partnership among the California Institute
of Technology, the University of California, and NASA; the observatory was
made possible by the generous financial support of the W. M. Keck Foundation.
We acknowledge the HFF Lens Model website that is hosted by the Mikulski
Archive for Space Telescopes (MAST) at STScI.

## References

  * Abbott et al. (2016a) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2016a, Physical Review Letters, 116, 061102, doi: 10.1103/PhysRevLett.116.061102
  * Abbott et al. (2016b) —. 2016b, Physical Review Letters, 116, 241103, doi: 10.1103/PhysRevLett.116.241103
  * Abbott et al. (2017) —. 2017, Physical Review Letters, 119, 161101, doi: 10.1103/PhysRevLett.119.161101
  * Alcock et al. (2001) Alcock, C., Allsman, R. A., Alves, D. R., et al. 2001, ApJ, 550, L169, doi: 10.1086/319636
  * Atek et al. (2011) Atek, H., Siana, B., Scarlata, C., et al. 2011, ApJ, 743, 121, doi: 10.1088/0004-637X/743/2/121
  * Bertin & Arnouts (1996) Bertin, E., & Arnouts, S. 1996, AJ, 117, 393
  * Bouwens et al. (2017) Bouwens, R. J., Oesch, P. A., Illingworth, G. D., Ellis, R. S., & Stefanon, M. 2017, ApJ, 843, 129, doi: 10.3847/1538-4357/aa70a4
  * Caminha et al. (2016) Caminha, G. B., Grillo, C., Rosati, P., et al. 2016, A&A, 587, A80, doi: 10.1051/0004-6361/201527670
  * Caminha et al. (2017) —. 2017, A&A, 600, A90, doi: 10.1051/0004-6361/201629297
  * Carr et al. (2017) Carr, B., Tenkanen, T., & Vaskonen, V. 2017, Phys. Rev. D, 96, 063507, doi: 10.1103/PhysRevD.96.063507
  * Castellano et al. (2016) Castellano, M., Amorín, R., Merlin, E., et al. 2016, A&A, 590, A31, doi: 10.1051/0004-6361/201527514
  * Chan et al. (2020) Chan, J. H. H., Schive, H.-Y., Wong, S.-K., Chiueh, T., & Broadhurst, T. 2020, Phys. Rev. Lett., 125, 111102, doi: 10.1103/PhysRevLett.125.111102
  * Chen et al. (2019) Chen, W., Kelly, P. L., Diego, J. M., et al. 2019, ApJ, 881, 8, doi: 10.3847/1538-4357/ab297d
  * Chen et al. (2022) Chen, W., Kelly, P. L., Treu, T., et al. 2022, arXiv e-prints, arXiv:2207.11658.  https://arxiv.org/abs/2207.11658
  * Dai et al. (2018) Dai, L., Venumadhav, T., Kaurov, A. A., & Miralda-Escudé, J. 2018, ArXiv e-prints.  https://arxiv.org/abs/1804.03149
  * Diego et al. (2016) Diego, J. M., Broadhurst, T., Wong, J., et al. 2016, MNRAS, 459, 3447, doi: 10.1093/mnras/stw865
  * Diego et al. (2022a) Diego, J. M., Pascale, M., Kavanagh, B. J., et al. 2022a, A&A, 665, A134, doi: 10.1051/0004-6361/202243605
  * Diego et al. (2005) Diego, J. M., Protopapas, P., Sandvik, H. B., & Tegmark, M. 2005, MNRAS, 360, 477, doi: 10.1111/j.1365-2966.2005.09021.x
  * Diego et al. (2007) Diego, J. M., Tegmark, M., Protopapas, P., & Sandvik, H. B. 2007, MNRAS, 375, 958, doi: 10.1111/j.1365-2966.2007.11380.x
  * Diego et al. (2018) Diego, J. M., Kaiser, N., Broadhurst, T., et al. 2018, ApJ, 857, 25, doi: 10.3847/1538-4357/aab617
  * Diego et al. (2022b) Diego, J. M., Meena, A. K., Adams, N. J., et al. 2022b, arXiv e-prints, arXiv:2210.06514.  https://arxiv.org/abs/2210.06514
  * Finkelstein (2016) Finkelstein, S. L. 2016, PASA, 33, e037, doi: 10.1017/pasa.2016.26
  * Ghosh et al. (2021) Ghosh, A., Williams, L. L. R., Liesenborgs, J., et al. 2021, MNRAS, 506, 6144, doi: 10.1093/mnras/stab1196
  * Giavalisco (2002) Giavalisco, M. 2002, ARA&A, 40, 579, doi: 10.1146/annurev.astro.40.121301.111837
  * Gilman et al. (2020) Gilman, D., Birrer, S., Nierenberg, A., et al. 2020, MNRAS, 491, 6077, doi: 10.1093/mnras/stz3480
  * Gonzaga (2012) Gonzaga, S. e. 2012, The DrizzlePac Handbook
  * Hayes et al. (2012) Hayes, M., Laporte, N., Pelló, R., Schaerer, D., & Le Borgne, J.-F. 2012, MNRAS, 425, L19, doi: 10.1111/j.1745-3933.2012.01293.x
  * Hezaveh et al. (2016) Hezaveh, Y. D., Dalal, N., Marrone, D. P., et al. 2016, ApJ, 823, 37, doi: 10.3847/0004-637X/823/1/37
  * Hui et al. (2017) Hui, L., Ostriker, J. P., Tremaine, S., & Witten, E. 2017, Phys. Rev. D, 95, 043541, doi: 10.1103/PhysRevD.95.043541
  * Johnson et al. (2014) Johnson, T. L., Sharon, K., Bayliss, M. B., et al. 2014, ApJ, 797, 48, doi: 10.1088/0004-637X/797/1/48
  * Karman et al. (2017) Karman, W., Caputi, K. I., Caminha, G. B., et al. 2017, A&A, 599, A28, doi: 10.1051/0004-6361/201629055
  * Kaurov et al. (2019) Kaurov, A. A., Dai, L., Venumadhav, T., Miralda-Escudé, J., & Frye, B. 2019, ApJ, 880, 58, doi: 10.3847/1538-4357/ab2888
  * Kawamata et al. (2018) Kawamata, R., Ishigaki, M., Shimasaku, K., et al. 2018, ApJ, 855, 4, doi: 10.3847/1538-4357/aaa6cf
  * Kawamata et al. (2016) Kawamata, R., Oguri, M., Ishigaki, M., Shimasaku, K., & Ouchi, M. 2016, ApJ, 819, 114, doi: 10.3847/0004-637X/819/2/114
  * Keeton (2010) Keeton, C. R. 2010, Gen. Rel. Grav., 42, 2151, doi: 10.1007/s10714-010-1041-1
  * Kelly et al. (2018) Kelly, P. L., Diego, J. M., Rodney, S., et al. 2018, Nature Astronomy, 2, 334, doi: 10.1038/s41550-018-0430-3
  * Lagattuta et al. (2017) Lagattuta, D. J., Richard, J., Clément, B., et al. 2017, MNRAS, 469, 3946, doi: 10.1093/mnras/stx1079
  * Laporte et al. (2011) Laporte, N., Pelló, R., Schaerer, D., et al. 2011, A&A, 531, A74, doi: 10.1051/0004-6361/201015842
  * Livermore et al. (2018) Livermore, R. C., Trenti, M., Bradley, L. D., et al. 2018, ApJ, 861, L17, doi: 10.3847/2041-8213/aacd16
  * Lotz et al. (2017) Lotz, J. M., Koekemoer, A., Coe, D., et al. 2017, ApJ, 837, 97, doi: 10.3847/1538-4357/837/1/97
  * Mediavilla et al. (2017) Mediavilla, E., Jiménez-Vicente, J., Muñoz, J. A., Vives-Arias, H., & Calderón-Infante, J. 2017, ApJ, 836, L18, doi: 10.3847/2041-8213/aa5dab
  * Meena et al. (2022) Meena, A. K., Chen, W., Zitrin, A., et al. 2022, arXiv e-prints, arXiv:2211.01402.  https://arxiv.org/abs/2211.01402
  * Miralda-Escude (1991) Miralda-Escude, J. 1991, ApJ, 379, 94, doi: 10.1086/170486
  * Mocz et al. (2017) Mocz, P., Vogelsberger, M., Robles, V. H., et al. 2017, MNRAS, 471, 4559, doi: 10.1093/mnras/stx1887
  * Narayanan & Davé (2012) Narayanan, D., & Davé, R. 2012, MNRAS, 423, 3601, doi: 10.1111/j.1365-2966.2012.21159.x
  * Nierenberg et al. (2014) Nierenberg, A. M., Treu, T., Wright, S. A., Fassnacht, C. D., & Auger, M. W. 2014, MNRAS, 442, 2434, doi: 10.1093/mnras/stu862
  * Oguri (2010) Oguri, M. 2010, PASJ, 62, 1017, doi: 10.1093/pasj/62.4.1017
  * Oguri et al. (2018) Oguri, M., Diego, J. M., Kaiser, N., Kelly, P. L., & Broadhurst, T. 2018, Phys. Rev. D, 97, 023518, doi: 10.1103/PhysRevD.97.023518
  * Oke & Gunn (1983) Oke, J. B., & Gunn, J. E. 1983, ApJ, 266, 713, doi: 10.1086/160817
  * Pozo et al. (2021) Pozo, A., Broadhurst, T., de Martino, I., et al. 2021, MNRAS, 504, 2868, doi: 10.1093/mnras/stab855
  * Rodney et al. (2018) Rodney, S. A., Balestra, I., Bradac, M., et al. 2018, Nature Astronomy, 2, 324, doi: 10.1038/s41550-018-0405-4
  * Sanders et al. (2015) Sanders, R. L., Shapley, A. E., Kriek, M., et al. 2015, ArXiv e-prints.  https://arxiv.org/abs/1509.03636
  * Schive et al. (2014) Schive, H.-Y., Chiueh, T., & Broadhurst, T. 2014, Nature Physics, 10, 496, doi: 10.1038/nphys2996
  * Soucail et al. (1988) Soucail, G., Mellier, Y., Fort, B., Mathez, G., & Cailloux, M. 1988, A&A, 191, L19
  * Steidel et al. (2014) Steidel, C. C., Rudie, G. C., Strom, A. L., et al. 2014, ApJ, 795, 165, doi: 10.1088/0004-637X/795/2/165
  * Tisserand et al. (2007) Tisserand, P., Le Guillou, L., Afonso, C., et al. 2007, A&A, 469, 387, doi: 10.1051/0004-6361:20066017
  * Vegetti et al. (2010) Vegetti, S., Koopmans, L. V. E., Bolton, A., Treu, T., & Gavazzi, R. 2010, MNRAS, 408, 1969, doi: 10.1111/j.1365-2966.2010.16865.x
  * Vegetti et al. (2012) Vegetti, S., Lagattuta, D. J., McKean, J. P., et al. 2012, Nature, 481, 341, doi: 10.1038/nature10669
  * Venumadhav et al. (2017) Venumadhav, T., Dai, L., & Miralda-Escudé, J. 2017, ApJ, 850, 49, doi: 10.3847/1538-4357/aa9575
  * Welch et al. (2022) Welch, B., Coe, D., Diego, J. M., et al. 2022, Nature, 603, 815, doi: 10.1038/s41586-022-04449-y
  * Zitrin et al. (2013) Zitrin, A., Meneghetti, M., Umetsu, K., et al. 2013, ApJ, 762, L30, doi: 10.1088/2041-8205/762/2/L30