§ INTRODUCTION

Frameworks for MW modeling, particularly for implementation in HRI and teaming, typically operate on limited biosignals and are incapable of operating online, leaving room for vast improvements in MW model development and evaluation. HR teams, in both the physical and virtual domains, are transitioning to peer-like paradigms where each team member can dynamically respond to the current state of the other [20, 22, 31]. In particular, improvements in MW for high-risk environments (e.g., driving, surgery, exploration) stand to improve safety and mission outcomes [19]. However, data collection in these environments, a known strategy for improving model outcomes [11], can be risky and quite costly [10]. VR is a key tool for improving MW models through data collection closer to the operational environment, without the costs and risks associated with real operations.

Model mismatches between HR teammates, often due to poor MW models, can risk detrimental outcomes. Human operators function optimally within individual workload boundaries [16, 45, 47]. Cognitive overload can lead to stress and operational mistakes, while cognitive underload can result in a lack of motivation or attention drift and decrease team performance. Operator interruption at inopportune times can increase cognitive demands and negatively impact mission efficacy [16].

CLT, the preeminent model of human cognitive architecture, provides a theoretical framework for characterizing and formalizing MW [34, 35, 42]. CLT suggests MW is a composite of intrinsic, extrinsic, and germane loads. Intrinsic load relates to the effort required to learn a particular concept. The section of MW related to the information presentation format defines the extrinsic load. Germane load encompasses the processing involved in understanding information relevant to the task at hand. Different component combinations indicate various levels of MW. The NASA MATB-II provides a well-studied and useful structure for manipulating MW [13, 40]. MATB-II consists of four main sections: RM, tracking, SM, and communications. The difficulty of each MATB-II section can be independently altered, allowing for fine-grained experimental control. However, the use of MATB-II has been mostly relegated to traditional on-screen testing [3, 18].

MW modeling predictions could improve via a robust multimodal model based on passive biosignals integrated into a VR system. Regardless of the chosen MW model (e.g., supervised learning, reinforcement learning), psychophysiological analysis aims to map a hidden cognitive state (e.g., MW) to observable external manifestations by teasing out concomitants (physical traits that map to many cognitive functions) [5]. Multimodal biosensor models are on the rise, but are rarely robust in design (i.e., include more than two biosignals), allowing for limited model performance [11, 45]. CLT indicates data collection and downstream model performance are likely to improve with the use of task relevant data [12]. However, most MW work does not employ a relevant task for data collection and training, introducing a major disconnect between theory and practice. A VR environment not only enables studies with higher task relevancy, which is particularly useful for MW models aimed at high-risk scenarios, but also facilitates online model use in addition to more traditional offline evaluation.

Here, we describe an online VR environment, specifically designed to modulate MW, and the real-time data the pipeline produces. A robust sweep of biosensors allows for multimodal feature extraction. The VR framework, complete with realistic motion cues, enables deployment and possible evaluation of both closed (online) and open (offline) MW models, in task relevant settings. The VR pipeline facilitates passive biosignal generation potentially closer to those that could be elicited during the actual task, thus likely improving MW model performance, particularly in high-risk scenarios.

block!tOur lunar rover VR simulation produces online time synced human performance and in-simulation objective measures for investigation of online mental workload estimation. The exploratory simulation, founded on MATB-II, allows for manipulation of each component separately. All four control components play a role in a MW model (i.e., Intrinsic, Extrinsic, Germane Load). A biosensor suite records a robust sweep of raw physiological data and extracts relevant biofeatures data during play (see <ref>). In-game objective data and extracted biofeatures are time synced through ROS. Between-run TLX survey results paired to time synced biofeatures and objective measures enable MW modeling online and post simulation.

component = [draw, text centered, rounded corners,font=, inner sep=1, outer sep=0]

[scale=1, transform shape, post/.style=->,shorten >=1pt,>=stealth',semithick, rounded corners=4pt]

(0,0) node (sim_title) [component,fill=blue!50,align=center, anchor=west,inner sep=3,text width=59ex] Lunar Rover Simulation Environment;

(sim_title.south west) +(0,-.425*) node (pupil) [component, fill=blue!30, anchor=north west,text width=18ex,text depth=10ex]

[Clip=.25 .05 .25 .05, width=.95]figures/ecg_white.png

(pupil.south west) +(0,-.15*) node (resp) [component, fill=blue!30, anchor=north west,text width=18ex,text depth=10ex]

Dermal Activity

[Clip=.25 .05 .25 .05, width=.95]figures/eda_white.png

(resp.south west) +(0,-.15*) node (eda) [component, fill=blue!30, anchor=north west,text width=18ex,text depth=10ex]

[Clip=.25 .05 .25 .05, width=.95]figures/resp_white.png

(eda.south west) +(0,-.15*) node (heart) [component, fill=blue!30, anchor=north west, text width=18ex,text depth=10ex]

[Clip=.27 .05 .25 .05, width=.95]figures/pupilDiam.png

(pupil.north east -| heart.east)+(.5*,0) node (task) [component, fill=Gray!30, anchor= north west, text width=40ex,text depth=50.5ex]

VR Exploration Task

[width=1.12,height=.58,Clip=.15 0 .01 0 ]figures/PXL_20211026_233853764.jpg

($ (resp.south west)!0.5!(eda.north west) $)+(-1.25*,0) node [draw, circle, x radius=.5em, y radius=.5em] (bind) ;

(sim_title.south west -| bind.north)+(0,) node (sim_a) ;
(task.south east)+(,-) node (sim_b) ;
[fill=blue!15,rounded corners, draw=black!50, dashed]
(sim_a) rectangle (sim_b) node (sim) ;

(sim_title.north west) + (-1.5*,-.125*) node (controls) [component,fill=Orange!30, anchor = north east]

MATB-II Style Controls

System Monitoring

Resource Management

(controls.south) + (0, -1.25*) node (processes) [component,fill=Orange!30]

Mental Workload

Intrinsic Load

Extrinsic Load

Germane Load

(processes.south |- bind.south) + (0, -2*) node (markers) [component,fill=Orange!30,align=center, text width=23ex, text depth=10ex]

2cBiosensor Suite

Empatica E4

Zephyr BH

Vive Pro Eye

(task.north east)+(3*,.25*)  node (objmeasures) [component, fill=yellow!30, anchor=north west]

(objmeasures.south)+(0,-.15*) node (subjmeasures) [component, fill=yellow!30, anchor=north]

(subjmeasures.south)+(0,-.15*) node (human_gen) [component, fill=yellow!30, anchor=north]

(objmeasures.north -| objmeasures.west)+(-,) node (output_a) ;
(human_gen.south -| objmeasures.east)+(,-) node (output_b) ;
[fill=yellow!15,rounded corners, draw=black!50, dashed]
(output_a) rectangle (output_b);

($ (output_a)!0.50!(output_b) $) + (0, -2.15*) node [draw, circle] (bind2) ;

($ (output_a)!0.50!(output_b) $) + (0, 2.15*) node (out_title) [component,fill=yellow!50,align=center,inner sep=2] Online Data Collection;

($ (output_a)!0.50!(output_b) $) + (0,-4.25*) node (ml) [component, fill=green!20, anchor=south]

Mental Workload Models
Multivariate Linear
Supervised Learning
Reinforcement Learning

[post] (controls.south)  – (processes.north);
(bind.west)+(.3*,.07) node (bind_topleft);
[post] (processes.south)  |- (bind_topleft);
(bind.west)+(.3*,-.07) node (bind_botleft);
[post] (markers.north)  |- (bind_botleft);
[post] ($ (controls.north east)!0.275!(controls.south east) $)  -| (task.north);

[post] (bind.east) |- (pupil.west);
[post] (bind.east) |- (heart.west);
[post] (bind.east) |- (eda.west);
[post] (bind.east) |- (resp.west);

[post] (pupil.east)  – (task.west |- pupil.east);
[post] (resp.east)  – (task.west |- resp.east);
[post] (eda.east)  – (task.west |- eda.east);
[post] (heart.east)  – (task.west |- heart.east);

($ (task.south)!0.5!(task.south west) $)+(0,.15*) node (task_bl2);

(task.east)+(0,.287*) node (task_rt);

($ (task.south)!0.5!(task.south east) $)+(0,.15*) node (task_br);

(ml.north)+(-1*,-.15*) node (ml_lt);

[post] (bind2.south)  – (ml.north);

(objmeasures.west) + (-0.4*,0) node (data_l);
[post] (task.east |- objmeasures.west) – (data_l);

[post,dashed,color=black] ($ (ml.south)!0.5!(ml.south east) $) – +(0,-1*) – +(2*,-1*);

($ (ml.south)!0.5!(ml.south east) $) – +(1*,-.5*) node (open_text)

[post,dashed,color=black] ($ (ml.south)!0.5!(ml.south west) $) – +(0,-1*) – +(-3.05*,-1*);

($ (ml.south)!0.5!(ml.south west) $) – +(-1.25*,-.5*) node (close_text)

§ RELEVANT BACKGROUND

We constructed our VR model by combining multiple aspects of research in both HRI and psychology into a multifaceted approach. Performance measures, both objective and subjective, enable external measurement of human achievement. Biosignals allow inference of the ANS and CNS, potentially providing deeper insights. VR is a novel test bed for MW studies promoting task relevant simulations.

§.§ Performance Measures

Performance measures display an intricate relationship to MW [14]. Optimum MWs consistently yield maximized objective performance [18, 45]. In VR, objective performance measures have demonstrated improved task performance and been linked to MW [6, 44]. A multitude of methods have been subjective surveys have been explored in relation to MW [39]. The TLX survey has demonstrated a relationship to MW in a variety of modalities [17], including proving the usefulness of VR in modulating MW [10, 37].

hud!bThe VR simulation is founded on MATB-II enabling easy manipulation of each characteristic. (A) The user must navigate the lunar terrain to an objective point. (B) Updating the radar position with respect to a fixed location requires the user concentrate on tracking. (C) Switching radio communications channels, prompted by separate voices, aims to satisfy the communications requirement. (D) The different rates of O$_2$ usage and CO$_2$ buildup force the user to consider resource management. (E) Motor temperature fluctuations and limitations require system management. Together, all pieces of the simulation act to modulate MW based on in-simulation difficulty.

component = [ text centered, rounded corners,font=, inner sep=2, outer sep=0, fill=Goldenrod, anchor=north west, text opacity=1, text=black]

[scale=1, transform shape, post/.style=->,shorten >=1pt,>=stealth',semithick, rounded corners=4pt]

(0,0) node (hud) [align=left] ;

1 = ($(hud.west)!0.125!(hud.east)$),
2 = ($(hud.north)!0.05!(hud.south)$),
3 = ($(hud.west)!0.31!(hud.east)$),
4 = ($(hud.north)!0.205!(hud.south)$),
5 = ($(hud.west)!0.055!(hud.east)$),
6 = ($(hud.north)!0.02!(hud.south)$),
7 = ($(hud.west)!0.112!(hud.east)$),
8 = ($(hud.north)!0.48!(hud.south)$),
9 = ($(hud.west)!0.43!(hud.east)$),
10 = ($(hud.north)!0.05!(hud.south)$),
11 = ($(hud.west)!0.67!(hud.east)$),
12 = ($(hud.north)!0.1!(hud.south)$),
13 = ($(hud.west)!0.755!(hud.east)$),
14 = ($(hud.north)!0.02!(hud.south)$),
15 = ($(hud.west)!0.991!(hud.east)$),
16 = ($(hud.north)!0.46!(hud.south)$),
17 = ($(hud.west)!0.695!(hud.east)$),
18 = ($(hud.north)!0.65!(hud.south)$),
19 = ($(hud.west)!0.86!(hud.east)$),
20 = ($(hud.north)!0.98!(hud.south)$),
coordinate (p1) at (1)

coordinate (p2) at (2)
coordinate (p3) at (3)
coordinate (p4) at (4)

coordinate (p5) at (5)
coordinate (p6) at (6)
coordinate (p7) at (7)
coordinate (p8) at (8)

coordinate (p9) at (9)
coordinate (p10) at (10)
coordinate (p11) at (11)
coordinate (p12) at (12)

coordinate (p13) at (13)
coordinate (p14) at (14)
coordinate (p15) at (15)
coordinate (p16) at (16)

coordinate (p17) at (17)
coordinate (p18) at (18)
coordinate (p19) at (19)
coordinate (p20) at (20);

[rounded corners, draw=white, dashed, thick] (p1 |- p2) rectangle (p3 |- p4)  node (radar) ;

[rounded corners, draw=white, dashed, thick] (p5 |- p6) rectangle (p7 |- p8)  node (sci) ;

[rounded corners, draw=white, dashed, thick] (p9 |- p10) rectangle (p11 |- p12)  node (radio) ;

[rounded corners, draw=white, dashed, thick] (p13 |- p14) rectangle (p15 |- p16)  node (levels) ;

[rounded corners, draw=white, dashed, thick] (p17 |- p18) rectangle (p19 |- p20)  node (motor) ;

($(p1 |- p4) !0.5! (p3 |- p4)$)  node (rad_mid) [component,anchor=center, fill opacity=0] ;

($(p5 |- p8) !0.5! (p7 |- p8)$)  node (obj_mid) [component,anchor=center, fill opacity=0] ;

($(p9 |- p12) !0.5! (p11 |- p12)$)  node (com_mid) [component,anchor=center, fill opacity=0] ;

($(p13 |- p14) !0.5! (p13 |- p16)$)  node (rm_lmid) [component,anchor=center, fill opacity=0] ;

($(p17 |- p18) !0.5! (p17 |- p20)$)  node (sm_lmid) [component,anchor=center, fill opacity=0] ;

(rad_mid)+(0,-.5*)  node (track) [component, anchor = north, fill opacity=.7] B.) Tracking;

(obj_mid)+(0,-.5*)  node (obj) [component, anchor = north, fill opacity=.7] A.) Objective;

(com_mid)+(0,-.5*)  node (comms) [component, anchor = north, fill opacity=.7] C.) Communications;

(rm_lmid)+(-,-)  node (resources) [component, anchor =north east, fill opacity=.7]

D.) Resource

(sm_lmid)+(-,0)  node (sys) [component, anchor = east, fill opacity=.7]

E.) System

[post,color=white] (track.north)  – (rad_mid);

[post,color=white] (obj.north)  – (obj_mid);

[post,color=white] (comms.north)  – (com_mid);

[post,color=white] (resources.north east)  – (rm_lmid);

[post,color=white] (sys.east)  – (sm_lmid);

§.§ Biosignals

Physiological signals give quantitative insight into an individual MW [11]. For our work, we selected six raw biosignals (and multiple derived features) that have demonstrated  MW correlations: ECG, EDA, PPG, Respiration, ST, and Pupillometry.

ECG, derived from cardiac electrical activity, and PPG, blood flow interpolation via light refraction, provide indirect insight into MW via cardiovascular activity measurement [11]. Temporally, ECG measures (HR and HRV) have demonstrated relationships to MW [11, 38, 45]. In the frequency domain, power bands increase or decrease as a function of task load [11]. PPG, either individually or in conjunction with ECG, may provide more useful MW estimates [46]. For example, calculating sVRI, a PPG signal derivative, requires a lower time period of analysis to estimate MW than HRV, making it a good candidate for real-time estimation [48]. Notably, physical exertion confounds cardiovascular measures, making them poorly suited for tasks with significant physical loads [15].

Dermal measures relate to MW as a response to general ANS stimulation. EDA, also known as GSR, quantifies the electrical resistance of skin. EDA waveforms consist of tonic, slow acting, and phasic, event specific, responses [9]. Tonic components have indicated moderate utility for MW estimation (e.g., tonic components loosely correlate with MW), while shorter, more frequent, phasic peaks suggest a stronger MW relationship [9]. ST can be utilized in MW estimations, but with temporal limitations of slow rise times and delayed event responses. Generally, ST decreases in response to MW [28]. However, ST responses to MW may be location dependent [1].

The Respiration Waveform gives insight into MW through breathing rate and variations [15]. In a relaxed state, a human breathes slowly and consistently. However, as MW rises, overall breathing rate increases along with an increased prevalence of irregular rhythms, quick variations, and cessations [26].

Pupillometry measures fluctuations in pupil size and reactivity, which have a direct relationship to MW through the CNS [32]. For example, variations in the rate and magnitude of microsaccades, involuntary eye movements that occur during fixation, can separate different MW levels [27]. One study specifically records pupillometry in VR and found a positive correlation between pupil diameter and subjective task load scores [41].

§.§ VR in MW modeling

VR provides an immersive environment capable of simulating complex situations in a controlled manner [10]. While VR has been employed in a variety of use cases [24, 30], the utility of VR starts to shine in high-risk scenarios. Driving simulations commonly vary route options or obstacle difficulty in order to induce different MWs [2, 21, 47]. Users of surgical simulators have subjectively reported the utility of VR in operations preparation [7] and objectively demonstrated decreased MWs after system use [49]. The use of VR for flight simulation assessment has been able to differentiate between low and high MWs [4, 23].

§ SYSTEM DESIGN

Motivated by the cognitive state insights provided by performance and biosignal measures as well as the role VR can play (see <ref>), we designed a VR exploration environment with the aim of modulating MW through simulation difficulty. While playing the simulation, the user dons three biosensors (i.e., Zephyr Bioharness, Empatica E4, HTC Vive Pro Eye). The system collects objective performance and physiological measures online, with TLX survey responses collected between runs (see <ref>). The Zephyr Bioharness communicates directly with ROS while the Empatica E4 and Unity connect to ROS through Windows via the vendor provided streaming server and rosbridge respectively. ROS time synchronizes all collected data, which can be used in a variety of MW models, enabling real-time (closed loop) use while still providing traditional offline (open loop) analysis.

The simulation begins with the user randomly placed on a 1 km$^2$ lunar texture, onboard the Apollo rover. A prompt informs the user of the mission scenario and objective. While on an exploration mission to an area of scientific interest (i.e., objective point), the rover's autonomous systems have failed. The rover's responsibilities now fall on the human as the user navigates to the objective point.

§.§ Simulation Design

The simulation tasks the user with navigating to a randomly generated objective point (represented as a green column (<ref>A)) and drop a marker while aboard the Apollo lunar rover, in the shortest amount of time. A 6-axis motion platform provides motion cues (<ref>), generated by the lunar texture the user drives across, complete with corresponding lunar physics (e.g., gravity of 0.166G). A side mounted HOTAS, in the same layout as the in simulation rover, provides in game movement and interaction (<ref>). While navigating, the user must track or respond to various rover alerts, all chosen in relationship to the MATB-II due to its ability to regulate the various components of MW. Upon objective completion, a TLX prompt appears.

§.§.§ Tracking

A radar indicator (<ref>B) details the signal strength and optimal direction of the on-board radar dish. The user must maintain optimal connection to the stationary relay station as they navigate through a button on the HOTAS stick. The user's field of view includes the actual on-board radar for added visual feedback. As signal strength worsens, the radar indicator drops bars and changes colors (i.e., green to yellow to red), while the arrow indicates the necessary directional change. If the user does not correct a drop in signal strength, the indicator will start flashing and increase flashing frequency over time.

§.§.§ Communication

While navigating, contact must be maintained with either the base station or the lunar habitat. Separate audio (i.e., male for base station and female for lunar habitat) and visual (<ref>C) prompts tell the user which frequency to broadcast to. Upon receiving the prompt, the user must switch channels via a button located on the HOTAS stick. Delays in response cause the prompt to reoccur at an increased frequency. The system may also separately ask the user to report in, prompting a separate trigger response.

§.§.§ Resource Management

Distance traveled as well as motor speed and temperature dictate the rover's limited battery, part of the RM triad (<ref>D). Overdrive capabilities, shown in the on-board display (<ref>E), can increase speed for a short burst but at the cost of significant battery life. Depletion of rover battery results in a failed run. CO$_2$ management, which requires the user to expel its contents after reaching a certain threshold, and O$_2$ consumption, the depletion of which results in a failed run, act as the other two constituents. Breathing rate, supplied by the Zephyr Bioharness, drives both CO$_2$ buildup and O$_2$ consumption, but at different scales (i.e., CO$_2$ buildup happens multiple times a run while O$_2$ depletion is a one time event).

trialA full trial consists of four separate runs, at alternating intensities, separated by three minutes of free play. Time synchronized biometrics (e.g., pupil diameter), objective performance (e.g., motor temperature), and subjective (TLX survey administration) performance measures enable online information use. At run initialization, the user must navigate to the objective point, while all metrics are recorded in real time. Periods of free play separate each run allowing the user biosignals to return to baseline.

component = [text centered, rounded corners,font=, inner sep=0, outer sep=0, fill=Goldenrod, anchor=north west, text opacity=1, text=black]

[node distance=0em, scale=1, transform shape, post/.style=->,shorten >=1pt,>=stealth',semithick]

(0,0) node (pupil) [align=left]  [scale=.680.45,Clip=   ]figures/pupilDiam.png;

(pupil.south west) + (0, ) node (obj) [align=left, anchor = north west] [scale=.680.45,Clip=   ]figures/motorTemp.png;

(obj.south west) + (0, ) node (subj) [align=left, anchor = north west] [scale=.680.45,Clip= .115  ]figures/SurveyCompletions.png;

(0,0) node (org) ;
(pupil.north west) + (0.05,-.13) node (ses1_tl) ;
(subj.south west) + (1.87*,.13) node (ses1_br) ;

(ses1_tl) + (2.3*,0) node (ses2_tl) ;
(ses1_br) + (1.95*,0) node (ses2_br) ;

(ses2_tl) + (1.95*,0) node (ses3_tl) ;
(ses2_br) + (2.25*,0) node (ses3_br) ;

(ses3_tl) + (2.15*,0) node (ses4_tl) ;
(ses3_br) + (1.6*,0) node (ses4_br) ;

[fill opacity=, fill=GreenYellow] (ses1_tl |- ses1_br) rectangle (ses1_tl -| ses1_br)  node (t1) ;
[fill opacity=, fill=GreenYellow] (ses2_tl |- ses2_br) rectangle (ses2_tl -| ses2_br)  node (t2) ;
[fill opacity=, fill=GreenYellow] (ses3_tl |- ses3_br) rectangle (ses3_tl -| ses3_br)  node (t3) ;
[fill opacity=, fill=GreenYellow] (ses4_tl |- ses4_br) rectangle (ses4_tl -| ses4_br)  node (t4) ;

($(ses1_tl |- ses1_tl) !0.5! (ses1_br |- ses1_tl)$) + (0,.13) node (rad_mid) [font = ,anchor=center] Trial I;
($(ses2_tl |- ses2_tl) !0.5! (ses2_br |- ses2_tl)$) + (0,.13) node (rad_mid) [font = ,anchor=center] Trial II;
($(ses3_tl |- ses3_tl) !0.5! (ses3_br |- ses3_tl)$) + (0,.13) node (rad_mid) [font = ,anchor=center] Trial III;
($(ses4_tl |- ses4_tl) !0.5! (ses4_br |- ses4_tl)$) + (0,.13) node (rad_mid) [font = ,anchor=center] Trial IV;

[color=white] ($(subj.south west)+(.2,0.14)$)  – ($(subj.south east)+(-.1,0.14)$);
($(subj.south west)+(.15,0.15)$) node (start) [anchor=north] 0;
($(subj.south east)+(-.25,0.15)$) node (end) [anchor=north] 60;
($(start) !0.5! (end)$) node (time)[] Time (min);

(pupil.north west) +(,) node (toppup) [anchor=north] 6;
(pupil.south west) +(,) node (botpup) [anchor=south] 3;
($(toppup) !0.5! (botpup)$) + (-.3,0) node (pupunit)[rotate=90]

(obj.north west) +(-.2,) node (topobj) [anchor=north] 140;
(obj.south west) +(-.1,) node (botobj) [anchor=south] 50;
($(topobj) !0.5! (botobj)$) + (-.2,0) node (objunit)[rotate=90]

(subj.north west) +(,) node (topsubj) [anchor=north] 1;
(subj.south west) +(,) node (botsubj) [anchor=south] 0;
($(topsubj) !0.5! (botsubj)$) + (-.25,0) node (subjunit)[rotate=90]

§.§.§ System Monitoring

Motor temperature and rover speed fulfill the role of SM (<ref>E). The on-board display reflects rover temperature variations, which are controlled by the HOTAS throttle on the motion platform. Throttle speed and terrain dictate the actual rover speed, with the user feeling all motion in real time through the motion platform. Vehicle torque drives motor temperature. If motor temperature reaches a critical temperature, the entire rover will stall while the system cools down, drastically affecting completion time. An overdrive button increases the torque over the recommended limit for a short time at the cost of an increased motor temperature.

§.§ Trial Design

We structured each run in a repeated measures fashion, designed to last for one-hour (<ref>). Each trial begins by equipping the user with biosensors and initializing the ROS data collection. Upon situating themselves in the motion platform and being briefed on the mission objective, the user explores the lunar terrain in free play mode, with the HUD and on-board display turned off, for five minutes. The five-minute period acts as an establishment of physiological baseline [18]. Afterward, the system sets the run's in-game difficulty, an objective point appears, and the information systems (i.e., the HUD and on-board display) turn on. The user then completes the task as previously described (see <ref>). After a run completion, the user enters three minutes of free play to reestablish baseline physiology. Subsequently, the system varies run difficulty (i.e., low to high or high to low) and begins the cycle again. The user completes four separate runs (two high and two low difficulty) during testing.

rawRaw biosignals, collected across three seperate devices, enable downstream feature extraction. (A-B) ECG and Respiration waveforms are collected from the Zephyr Bioharness. (C-E) EDA, PPG, and Skin Temerature are relayed through the Emaptica E4 streaming server. (F) Pupil position is reported through the HTC Vive Pro Eye. A rolling 30s window is applied to each waveform or positional data to extract and report biofeatures (<ref>).

component = [ text centered, rounded corners,font=, inner sep=2, outer sep=0, fill=Goldenrod, text opacity=1, text=black]

[ scale=1, transform shape, post/.style=->,shorten >=1pt,>=stealth',semithick, rounded corners=4pt, font =]

(0,0) node (ecg) [] [width=,Clip=   ]figures/ecg.png;

[align=left, below=.1cm of ecg] (resp)   [width=,Clip=   ]figures/resp.png;

[below= .1cm of resp] (eda)   [width=,Clip=   ]figures/eda.png;

[align=left, below=.1cm of eda] (ppg)   [width=,Clip=   ]figures/ppg.png;

[align=left, below=.1cm of ppg] (st)   [width=,Clip= 0.1  ]figures/skinTemp.png;

(pupil) [align=left, below=.2cm of st] [width=,Clip=0.125 0.105 0.1 0.1]figures/combinedpupilfig.png;

(ecg.north west) +(,) node (topecg) [anchor=north] 1.5;
(ecg.south west) +(,) node (botecg) [anchor=south] -0.5;
($(topecg) !0.5! (botecg)$) node (ecgunit)[rotate=90] Amplitude;

(eda.north west) +(,) node (topeda) [anchor=north] 0.58;
(eda.south west) +(,) node (boteda) [anchor=south] 0.55;
($(topeda) !0.5! (boteda)$) node (edaunit)[rotate=90] EDA($\mu$S);

(ppg.north west) +(,) node (topppg) [anchor=north] 100;
(ppg.south west) +(,) node (botppg) [anchor=south] -100;
($(topppg) !0.5! (botppg)$) node (ppgunit)[rotate=90] BVP;

(resp.north west) +(,) node (topresp) [anchor=north] 10;
(resp.south west) +(,) node (botresp) [anchor=south] -10;
($(topresp) !0.5! (botresp)$) node (respunit)[rotate=90] Amplitude;

(st.north west) +(,) node (topst) [anchor=north] 31.5;
(st.south west) +(,) node (botst) [anchor=south] 30.5;
($(topst) !0.5! (botst)$) node (stunit)[rotate=90] Temp.($^{\circ}$C);

(pupil.north west) +(,-.17) node (toppupil) [anchor=north] 45;
(pupil.south west) +(,) node (botpupil) [anchor=south] -45;
($(toppupil) !0.5! (botpupil)$) node (pupilunit)[rotate=90] Y Position ($^{\circ}$);

(ecg.north) node (title)[font=] Electrocardiogram;
(eda.north) node (title)[font=]  Electrodermal Activity;
(ppg.north) node (title)[font=]  Photoplethysmography;
(resp.north) node (title)[font=]  Respiration;
(st.north) node (title)[font=]  Skin Temperature;
(pupil.north)+(0,-.1) node (title)[font=]  Pupil Position;

[color=white] ($(st.south west)+(.2,0.14)$)  – ($(st.south east)+(-.1,0.14)$);
($(st.south west)+(.25,0.2)$) node (start) [anchor=north] 0;
($(st.south east)+(-.25,0.19)$) node (end) [anchor=north] 30;
($(start) !0.5! (end)$) node (time)[] Time(s);

[color=white] ($(pupil.south west)+(.2,0.14)$)  – ($(pupil.south east)+(-.1,0.14)$);
($(pupil.south west)+(.35,0.2)$) node (start) [anchor=north] -60;
($(pupil.south east)+(-.25,0.19)$) node (end) [anchor=north] 60;
($(start) !0.5! (end)$) node (xpos)[] X Position ($^{\circ}$);

(topecg.north)+(0,0.05) node (title)[font=] (A);
(topresp.north)+(0,0.05) node (title)[font=] (B);
(topeda.north)+(0,0.05) node (title)[font=] (C);
(topppg.north)+(0,0.05) node (title)[font=] (D);
(topst.north)+(0,0.05) node (title)[font=] (E);
(toppupil.north)+(0,0.05) node (title)[font=] (F);

§ SYSTEM OUTPUT

ROS time syncs all collected data across multiple sensors and operating systems (i.e., Linux and Windows). After an individual study completion, a single rosbag carries all data for a single run. The ROS architecture enables isolation of each data stream to separate topics, with each topic handling different measurement types (e.g., ECG topic, rover topic, TLX survey topic). The system collects in-game objective performance measures, between run TLX surveys, and multiple biosignal (e.g., EDA, Pupillometry) features.

§.§ Objective Measures

In-simulation measures indicate user performance. For the rover, the system reports rover pose (position, rotation), twist (velocity, angular velocity), fuel, and motor temperature at 10 Hz. Boolean flags mark radio requests and responses. Twelve total states represent antenna accuracy. We also collect current CO$_2$ and O$_2$ (at 10 Hz), the current trial leg and difficulty (see <ref>), and elapsed time of play.

§.§ Subjective Measures

A TLX survey appears after the user has reached the objective and dropped their marker. The survey asks questions aimed at subjectively evaluating user performance (e.g., How mentally demanding was the task?), in six categories: mental demand, physical demand, temporal demand, performance, effort, and frustration.

List of biosignals and the corresponding available derived features produced in the VR pipeline.

Biosignal     6cFeatures
7*ECG            2cTime Domain     2cNon-Linear    2cFrequency Domain

2cHRV (RMSSD)     2cPNN10     2cVLF Power

2cSDSD      2cPNN25     2cLF Power

2cR2R (RRMean)     2cPNN50     2cHF Power

2cRRStD     2cSD1, SD2     2cTotal Power

2cMin, Max     2cSD1/2

2cHRV Tri Index     2cSDell
3*Respiration         3cTime Domain     3cFrequency Domain

3cRespiration Rate     3cLF Power, HF Power

3c     3cPower Ratio
5*EDA            2cPhasic     2cPeak (Phasic)     2cTonic

2cMean      2cMax, Min      2cMean

2cStD       2cMean, Quantity               2cStD

2cRange     2cMean Duration              2cRange

2cAUC       2cMean Slope          2cAUC
2*ST             6cTime Domain

6cMean, Median, StD, Max, Min
3*PPG             6cTime Domain

6cDigital PA, Reflection index

6cR2R, AUC, sVRI, IPA, PRV
5*Pupillometry       6cTime Domain

6cPupil Diameter, Smooth Pursuit Frequency

6cSaccade Frequency, Mean Saccade Amplitude

6cFixation Frequency, Fixation Duration

6cPost Saccadic Oscillations (PSO) Frequency

§.§ Biofeatures

§.§.§ Raw Biosignals

Raw biosignals are collected at resolutions specified by the manufacturer. Using the vendor supplied Windows streaming server, the Empatica E4 collects and relays raw PPG at 64 Hz as well as EDA and ST at 4 Hz. The Zephyr Bioharness directly relays a raw respiration waveform at 1.008 Hz and ECG waveform at 252 Hz. Pupillometry, provided through the HTC Vive Pro Eye linked to Unity, reports 2-D eye position and pupil diameter at 120Hz.

§.§.§ Derived Biosignal Features

Using third-party packages [8, 36], our system derives biosignal features with a 30s windowing interval [18, 43]. Biofeatures are derived in both time and frequency domains, ranging from traditional features (e.g., heart rate) to newer approaches (e.g., smooth pursuits). A full list of derived biosignal features can be found in <ref>.

§ DISCUSSION

MW models are critical for effective HR teaming and immersive VR experiences, but available processes rarely include online capabilities or relevant settings. This work developed a VR pipeline capable of time synchronizing numerous objective and subjective performance data with passive biosignals, which can be utilized in MW modeling on or offline (<ref>). VR task design compels users to focus on the four pillars of MATB-II (i.e., tracking, RM, SM, and communication) while navigating to an objective (<ref>). VR enables situational data collection, by emulating high-risk extraplanetary exploration. Raw biosignal data (<ref>) empowers our pipeline to extract a plethora of relevant biofeatures (<ref>), close to real time, providing a relevant test bed for open and closed loop MW model exploration.

The immersive capability of VR, coupled with synchronized biofeatures, enable a deep exploration of MW modeling. Immersive environments ensure realistic data collection by causing more realistic mental and physical responses [12]. A multitude of biofeatures relate to MW (see <ref>), yet few studies directly compare information gain across modalities [45]. Direct comparisons could inform systems designed for resource constrained environments. ML model approaches are end goal dependent (e.g., training evaluation or HRI). Online methods, such as DL or RL models, may improve HR teaming efficacy and safety [25, 29]. Offline modeling, such as traditional supervised learning models, may be more appropriate for testing individual skill sets [33]. The flexibility of our pipeline design with ROS integration allows this approach to be ported to a plethora of VR environments, or even into physical robotic systems.

We learned many lessons and discovered implicit design limitations while executing our system design. Many technical issues stemmed from communicating and synchronizing across two different operating systems (i.e., Windows and Linux). The latest iteration of ROS, ROS2, is available for Windows. However, keeping our system in Linux enables downstream use in real-world robotic systems, the majority of which operate on Linux. The Empatica streaming server instability and hardware limitations (e.g., dropping or not discovering devices and specific Bluetooth module requirements) was an unforeseen complication, which could have been remedied with an open-source option; we are currently developing such an option for community benefit. Poor error messaging made debugging the ROS to Unity connection difficult (e.g., mistyped Unity components break the connection but do not indicate where the error occurred). Incompatibilities between Unity VR and Steam VR hindered full headset integration, prompting a switch to Unity XR. We tested many iterations in order to balance the MATB-II style features to ensure playability and goal completion while being able to modulate difficulty. Our oop approach for feature extraction allows easy third-party package changes, ensuring easy future modifications. Other biosensors could be added to the suite (e.g., EEG, EMG), but may be impractical for VR or result in large rosbag or rosmsg files that can slow down system processing. The 30s windowing time temporally limits MW predictions. Shortening the window time without significant information loss may be possible [43].

Simulation design leaves the door open for a variety of future work. An independent samples t-test would validate simulation difficulty against MW (i.e., over or under-loaded). We have begun human subject data collection to this end ($\alpha$=0.05, $\beta$=0.8 $\vert$ n$\approx$35). With a subject database, we can explore a variety of MW ML models. We intend to evaluate traditional multivariate linear and simple supervised learning approaches, followed by DL and RL methods. To utilize the MW models online, we are developing an AI agent equipped with the MW model and capable of online simulation control variation.

§ CONCLUSION

MWs could be useful in a wide variety of systems, from HRI and teaming to operator training and testing. Such systems would utilize ML-based MW models that ideally function on passive biosignals produced in realistic environments. VRs immersive capabilities facilitate reproducible simulations of complex scenarios, fulfilling this critical need. The developed simulation, based on CLT and MATB-II, aims to achieve this goal in conjunction with a robust sweep of biosignals and features. The online time synced design not only allows for offline and open loop analysis, but enables real-time MW assessment, which stands to improve a variety of MW models and form the basis for closed-loop control systems reactive to an operator's MW.

[1]
Y. Abdelrahman, E. Velloso, T. Dingler, A. Schmidt, and F. Vetere.
Cognitive heat: exploring the usage of thermal imaging to
unobtrusively estimate cognitive load.
Proceedings of the ACM on Interactive, Mobile, Wearable and
Ubiquitous Technologies, 1(3):1–20, 2017.

[2]
U. A. Abdurrahman, S.-C. Yeh, Y. Wong, and L. Wei.
Effects of neuro-cognitive load on learning transfer using a virtual
reality-based driving system.
Big Data and Cognitive Computing, 5(4):54, 2021.

[3]
I. Albuquerque, A. Tiwari, M. Parent, R. Cassani, J.-F. Gagnon, D. Lafond,
S. Tremblay, and T. H. Falk.
Wauc: a multi-modal database for mental workload assessment under
physical activity.
Frontiers in Neuroscience, 14, 2020.

[4]
P. Andrievskaia, K. Van Benthem, and C. M. Herdman.
Neural correlates of mental workload in virtual flight simulation.
In International Conference on Human-Computer Interaction, pp.
521–528. Springer, 2020.

[5]
R. W. Backs.
Application of psychophysiological models to mental workload.
In Proceedings of the Human Factors and Ergonomics Society
Annual Meeting, vol. 44, pp. 3–464. SAGE Publications Sage CA: Los Angeles,
CA, 2000.

[6]
J. Bailenson, K. Patel, A. Nielsen, R. Bajscy, S.-H. Jung, and G. Kurillo.
The effect of interactivity on learning physical actions in virtual
Media Psychology, 11(3):354–376, 2008.

[7]
J. Barré, D. Michelet, J. Truchot, E. Jolivet, T. Recanzone, S. Stiti,
A. Tesnière, and G. Pourcher.
Virtual reality single-port sleeve gastrectomy training decreases
physical and mental workload in novice surgeons: an exploratory study.
Obesity surgery, 29(4):1309–1316, 2019.

[8]
A. Bizzego, A. Battisti, G. Gabrieli, G. Esposito, and C. Furlanello.
pyphysio: A physiological signal processing library for data science
approaches in physiology.
SoftwareX, 10:100287, 2019.

[9]
J. J. Braithwaite, D. G. Watson, R. Jones, and M. Rowe.
A guide for analysing electrodermal activity (EDA) & skin
conductance responses (SCRs) for psychological experiments.
Psychophysiology, 49(1):1017–1034, 2013.

[10]
C.-J. Chao, S.-Y. Wu, Y.-J. Yau, W.-Y. Feng, and F.-Y. Tseng.
Effects of three-dimensional virtual reality and traditional training
methods on mental workload and training performance.
Human Factors and Ergonomics in Manufacturing & Service
Industries, 27(4):187–196, 2017.

[11]
R. L. Charles and J. Nixon.
Measuring mental workload using physiological measures: A systematic
Applied ergonomics, 74:221–232, 2019.

[12]
H.-H. Choi, J. J. Van Merriënboer, and F. Paas.
Effects of the physical environment on cognitive load and learning:
Towards a new model of cognitive load.
Educational Psychology Review, 26(2):225–244, 2014.

[13]
Y. Daviaux, C. Bey, L. Arsac, O. Morellec, and S. Lini.
Feedback on the use of MATB-II task for modeling of cognitive
control levels through psycho-physiological biosignals.
In 20th International Symposium on Aviation Psychology, p. 205,

[14]
E. Galy, M. Cariou, and C. Mélan.
What is the relationship between mental workload factors and
cognitive load types?
International journal of psychophysiology, 83(3):269–275,

[15]
M. Grassmann, E. Vlemincx, A. Von Leupoldt, J. M. Mittelstädt, and
O. Van den Bergh.
Respiratory changes in response to cognitive load: a systematic
Neural plasticity, 2016.

[16]
E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey.
Psycho-physiological measures for assessing cognitive load.
In Proceedings of the 12th ACM international conference on
Ubiquitous computing, pp. 301–310, 2010.

[17]
S. G. Hart.
NASA-task load index (NASA-TLX); 20 years later.
In Proceedings of the human factors and ergonomics society
annual meeting, vol. 50, pp. 904–908. Sage publications Sage CA: Los
Angeles, CA, 2006.

[18]
J. Heard and J. A. Adams.
Multi-dimensional human workload assessment for supervisory
human–machine teams.
Journal of Cognitive Engineering and Decision Making,
13(3):146–170, 2019.

[19]
J. Heard, C. E. Harriott, and J. A. Adams.
A human workload assessment algorithm for collaborative human-machine
In 2017 26th IEEE International Symposium on Robot and Human
Interactive Communication (RO-MAN), pp. 366–371. IEEE, 2017.

[20]
K. Holden, N. Ezer, and G. Vos.
Evidence report: risk of inadequate human-computer interaction.

[21]
S. Imhoff, M. Lavallière, N. Teasdale, and P. Fait.
Driving assessment and rehabilitation using a driving simulator in
individuals with traumatic brain injury: A scoping review.
NeuroRehabilitation, 39(2):239–251, 2016.

[22]
M. Johnson, J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. B. Van Riemsdijk,
and M. Sierhuis.
Coactive design: Designing support for interdependence in joint
Journal of Human-Robot Interaction, 3 (1), 2014, 2014.

[23]
I. Kakkos, G. N. Dimitrakopoulos, L. Gao, Y. Zhang, P. Qi, G. K. Matsopoulos,
N. Thakor, A. Bezerianos, and Y. Sun.
Mental workload drives different reorganizations of functional
cortical connectivity between 2D and 3D simulated flight experiments.
IEEE Transactions on Neural Systems and Rehabilitation
Engineering, 27(9):1704–1713, 2019.

[24]
M. Kaufeld and P. Nickel.
Level of robot autonomy and information aids in human-robot
interaction affect human mental workload–an investigation in virtual
In International Conference on Human-Computer Interaction, pp.
278–291. Springer, 2019.

[25]
S. K. Kim, E. A. Kirchner, A. Stefes, and F. Kirchner.
Intrinsic interactive reinforcement learning–using error-related
potentials for real world human-robot interaction.
Scientific reports, 7(1):1–16, 2017.

[26]
S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun,
A. Nijholt, and I. Patras.
DEAP: A database for emotion analysis using physiological signals.
IEEE transactions on affective computing, 3(1):18–31, 2011.

[27]
K. Krejtz, A. T. Duchowski, A. Niedzielska, C. Biele, and I. Krejtz.
Eye tracking cognitive load using pupil diameter and microsaccades
with fixed gaze.
PloS one, 13(9):e0203629, 2018.

[28]
C. Larmuseau, J. Cornelis, L. Lancieri, P. Desmet, and F. Depaepe.
Multimodal learning analytics to investigate cognitive load during
online problem solving.
British Journal of Educational Technology, 51(5):1548–1562,

[29]
T. D. Le, D. T. Huynh, and H. V. Pham.
Efficient human-robot interaction using deep learning with mask
R-CNN: detection, recognition, tracking, and segmentation.
In 2018 15th International Conference on Control, Automation,
Robotics and Vision (ICARCV), pp. 162–167. IEEE, 2018.

[30]
T. Luong, A. Lecuyer, N. Martin, and F. Argelaguet.
A survey on affective and cognitive VR.
IEEE Transactions on Visualization and Computer Graphics, 2021.

[31]
L. M. Ma, T. Fong, M. J. Micire, Y. K. Kim, and K. Feigh.
Human-robot teaming: Concepts and components for design.
In Field and service robotics, pp. 649–663. Springer, 2018.

[32]
G. Marquart, C. Cabrall, and J. de Winter.
Review of eye-related measures of drivers’ mental workload.
Procedia Manufacturing, 3:2854–2861, 2015.

[33]
K. Moustafa, S. Luz, and L. Longo.
Assessment of mental workload: a comparison of machine learning
methods and subjective assessment techniques.
In International symposium on human mental workload: Models and
applications, pp. 30–50. Springer, 2017.

[34]
F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. Van Gerven.
Cognitive load measurement as a means to advance cognitive load
Educational psychologist, 38(1):63–71, 2003.

[35]
F. G. Paas and J. J. Van Merriënboer.
Instructional control of cognitive load in the training of complex
cognitive tasks.
Educational psychology review, 6(4):351–371, 1994.

[36]
J. Pekkanen and O. Lappi.
A new and general approach to signal denoising and eye movement
classification based on segmented linear regression.
Scientific reports, 7(1):1–13, 2017.

[37]
L. Pouliquen-Lardy, I. Milleville-Pennel, F. Guillaume, and F. Mars.
Remote collaboration in virtual reality: asymmetrical effects of task
distribution on spatial processing and mental workload.
Virtual Reality, 20(4):213–220, 2016.

[38]
A. Rieger, R. Stoll, S. Kreuzfeld, K. Behrens, and M. Weippert.
Heart rate and heart rate variability as indirect markers of
surgeons’ intraoperative stress.
International archives of occupational and environmental
health, 87(2):165–174, 2014.

[39]
S. Rubio, E. Díaz, J. Martín, and J. M. Puente.
Evaluation of subjective mental workload: A comparison of SWAT,
NASA-TLX, and workload profile methods.
Applied psychology, 53(1):61–86, 2004.

[40]
Y. Santiago-Espada, R. R. Myer, K. A. Latorella, and J. R. Comstock Jr.
The Multi-Attribute Task Battery II (MATB-II) software
for human performance and workload research: A user's guide.

[41]
M. Schwalm, A. Keinath, and H. D. Zimmer.
Pupillometry as a method for measuring mental workload within a
simulated driving task.
Human Factors for assistance and automation, 1986:1–13, 2008.

[42]
J. Sweller, J. J. Van Merrienboer, and F. G. Paas.
Cognitive architecture and instructional design.
Educational psychology review, 10(3):251–296, 1998.

[43]
J. Tervonen, K. Pettersson, and J. Mäntyjärvi.
Ultra-short window length and feature importance analysis for
cognitive load detection from wearable sensors.
Electronics, 10(5):613, 2021.

[44]
J. Tichon.
Training cognitive skills in virtual reality: Measuring performance.
CyberPsychology & Behavior, 10(2):286–289, 2006.

[45]
P. Vanneste, A. Raes, J. Morton, K. Bombeke, B. B. Van Acker, C. Larmuseau,
F. Depaepe, and W. Van den Noortgate.
Towards measuring cognitive load through multimodal physiological
Cognition, Technology & Work, pp. 1–19, 2020.

[46]
C. Wang and J. Guo.
A data-driven framework for learners’ cognitive load detection
using ECG-PPG physiological feature fusion and XGBoost classification.
Procedia computer science, 147:338–348, 2019.

[47]
S. Zepf, J. Hernandez, A. Schmitt, W. Minker, and R. W. Picard.
Driver emotion recognition for intelligent vehicles: A survey.
ACM Computing Surveys (CSUR), 53(3):1–30, 2020.

[48]
X. Zhang, Y. Lyu, X. Hu, Z. Hu, Y. Shi, and H. Yin.
Evaluating photoplethysmogram as a real-time cognitive load
assessment during game playing.
International Journal of Human–Computer Interaction,
34(8):695–706, 2018.

[49]
B. Zheng, X. Jiang, G. Tien, A. Meneghetti, O. N. M. Panton, and M. S. Atkins.
Workload assessment of surgeons: correlation between NASA-TLX and
Surgical endoscopy, 26(10):2746–2750, 2012.


# FE-DeTr: Keypoint Detection and Tracking in Low-quality Image Frames with
Events

Xiangyuan Wang∗ Kuangyi Chen∗ Wen Yang Lei Yu Yannan Xing Huai Yu† ∗Equal
contribution. †Corresponding author.X. Wang, K. Chen, W. Yang, L. Yu and H. Yu
are with the EIS, Wuhan University, Wuhan, China {wangxiangyuan,chenky721,
<EMAIL_ADDRESS>Y. Xing is with the SynSense Tech. Co. Ltd.,
Chengdu, China<EMAIL_ADDRESS>
###### Abstract

Keypoint detection and tracking in traditional image frames are often
compromised by image quality issues such as motion blur and extreme lighting
conditions. Event cameras offer potential solutions to these challenges by
virtue of their high temporal resolution and high dynamic range. However, they
have limited performance in practical applications due to their inherent noise
in event data. This paper advocates fusing the complementary information from
image frames and event streams to achieve more robust keypoint detection and
tracking. Specifically, we propose a novel keypoint detection network that
fuses the textural and structural information from image frames with the high-
temporal-resolution motion information from event streams, namely FE-DeTr. The
network leverages a temporal response consistency for supervision, ensuring
stable and efficient keypoint detection. Moreover, we use a spatio-temporal
nearest-neighbor search strategy for robust keypoint tracking. Extensive
experiments are conducted on a new dataset featuring both image frames and
event data captured under extreme conditions. The experimental results confirm
the superior performance of our method over both existing frame-based and
event-based methods. Our code, pre-trained models, and dataset are available
at https://github.com/yuyangpoi/FE-DeTr.

##  I Introduction

Keypoint detection and tracking serve as critical components for a range of
applications, such as Simultaneous Localization and Mapping (SLAM) and
Structure from Motion (SfM). Traditional frame-based methods [1, 2, 3, 4, 5,
6, 7, 8, 9, 10] rely on sharp, distinct features within an image to detect and
track keypoints. However, motion blur substantially distorts these features,
making keypoints difficult to locate. Additionally, motion information present
during the exposure time can’t be captured, making accurate keypoint tracking
an insurmountable challenge. Extreme lighting conditions, such as overexposure
and low light, exacerbate these challenges by producing frames that are either
washed out or inadequately illuminated, further hampering feature
identification. Although BALF[11] introduces an MLP-based architecture for
local keypoint detection within blurred images, it still can’t effectively
handle images under extreme lighting conditions.

Event cameras can overcome these limitations because they capture
environmental changes asynchronously, providing events with extraordinarily
high temporal resolution and high dynamic range [12, 13, 14]. These
characteristics enable them to capture necessary information for keypoint
detection and tracking [15, 16, 17, 18, 19, 20, 21, 22, 23, 24] under
challenging conditions. Nonetheless, event cameras also come with their
limitations: they only capture dynamic changes in luminance, lack details on
absolute intensity or texture, and are noisy due to both operational
principles and hardware constraints. On the contrary, image frames preserve
stable structural features, and also mitigate the impact of event noise [25].
Motivated by the complementary strengths of both, we propose fusing image
frames with event data to enhance keypoint detection and tracking performance
in challenging conditions.

Figure 1: Our method (bottom) leverages the complementary characteristics of
image frames and event streams, allowing for stable keypoint detection and
tracking in extreme conditions compared to frame-based methods (top) and
event-based methods (middle).

In this paper, we propose a keypoint detection and tracking framework that
fuses image frames and event data (i.e., FE-DeTr). The detection network in
the framework is trained using a supervision strategy based on temporal
response consistency. The objective is to exploit the complementary
information from image frames and event data to identify keypoints exhibiting
temporal and spatial persistence, enabling their extended tracking using a
spatio-temporal nearest-neighbor strategy. An overall diagram of our proposed
approach is illustrated in Fig. 1. The keypoint detection network comprises a
Fusion Feature Extractor (FFE), a Motion Extractor (ME), and a Motion-Aware
Head (MAH). The FFE fuses complementary information from image frames and
events and achieves the propagation of temporal information through a
recurrent structure. The ME includes spatial and channel attention mechanisms,
designed to extract motion information from event streams. The MAH
incorporates an iterative strategy for handling relative motion, implicitly
warping keypoint responses at different time instants using deformable
convolutions to improve response repeatability. The supervision based on
temporal response consistency relies solely on the relative motion
relationships between different time instants, further ensuring the
repeatability of detected keypoints.

The main contributions are listed as follows:

  * •

We propose the first framework fusing image frames and events for robust
keypoint detection and tracking under extreme conditions.

  * •

We design a Motion-Aware Head based on an iterative strategy and introduce a
supervision strategy built on temporal response consistency, which enables the
network to produce stable and highly repeatable responses for long-term
keypoint tracking.

  * •

We contribute a new keypoint detection and tracking dataset that contains both
image frames and event data, encompassing high-speed motion and extreme
lighting scenarios. Experimental results on this dataset demonstrate our
method outperforms both frame-based and event-based methods under extreme
conditions.

##  II Related Work

In this section, we review the frame-based and event-based methods for
keypoint detection and tracking.

###  II-A Frame-Based methods

Learning-based keypoint detection and tracking on image frames have been
studied over years. Most keypoint detection methods also encode local
descriptors for matching.

Keypoint detection: Lift [26] is an early pioneer in exploring CNNs for local
feature extraction and description. Superpoint [1] presents a learning-based
local descriptor and a self-supervised keypoint detection that utilizes
pseudo-ground truth correspondences with homographic transformations.
Unsuperpoint [2] and R2D2 [3] propose to extract highly repeatable keypoints
from image geometric transformations. SiLK [27] summarizes prior work and
improves the stability of detection and matching by defining transition
probabilities on descriptor similarity. BALF[11] utilizes an MLP-based
structure to effectively detect keypoint within blurred images. However, on
the one hand, the definition of keypoints in blurred images isn’t well
clarified. On the other hand, it is only effective for handling blurred images
under favorable lighting conditions. The aforementioned methods inspire the
design of our approach, yet none of them explicitly tackles the challenges
posed by low-quality images.

Point tracking: Point tracking aims to establish point-level correspondences
across multiple images. Currently, mainstream methods encompass optical flow,
descriptor-based matching, and particle video techniques. Optical flow methods
[28, 4, 29, 5] estimate point correspondences between continuous frames based
on the intensity invariance over time, limiting their robustness for long-term
tracking. Descriptor-based matching techniques [26, 1, 2, 3, 27] overlook
motion information between consecutive frames, instead identifying point
correspondences based on descriptors across image pairs. While particle video
methods offer high accuracy [8, 9, 10], they require multiple input frames and
consequently lack real-time performance capabilities. Similarly, all these
point-tracking methods depend on high-quality images and are thus sensitive to
motion blur and extreme lighting conditions.

###  II-B Event-Based methods

In recent years, there has been a growing trend to leverage event cameras for
enhanced keypoint detection and tracking. However, varied motions elicit
unique event responses, posing challenges for early handcrafted methods [15,
16, 17, 18] in robustly extracting keypoints. As a result, learning-based
techniques have gained prominence. SILC [19] employs random forests for
keypoint localization and introduces Speed-Invariant Time-Surface to mitigate
the influence of motion speed on event representations. Inspired by the image
gradient-based corner detectors, Gradient [20] proposes to reconstruct
gradient maps from events for keypoint detection. Similarly, Long-lived[21]
adopts the same network architecture as [20] but directly predicts keypoint
heatmaps instead of image gradients, achieving superior performance. All the
above methods achieve high-frequency keypoint detection and tracking. However,
relying solely on event-based methods proves challenging for mitigating
inherent sensor noise, thereby constraining the effectiveness of keypoint
detection and tracking in event streams. In this paper, we fuse traditional
cameras with event cameras to combine the strengths of each modality, further
enhancing the robustness of keypoint detection and tracking.

##  III Method

###  III-A Event representaion

Figure 2: Overview of the proposed FE-DeTr. For each frame interval, an event
representation is generated and combined with the image frame as input to the
keypoint detection network. The network outputs a sequence of uniformly spaced
heatmaps.

To input asynchronous events to the neural network, we employ the Voxel Grid
[30] as the event representation $E$. This event representation effectively
preserves the temporal information of events. For a event
$(x_{i},y_{i},t_{i},p_{i})$ within a specific frame interval, the polarity
$p_{i}$ is allocated to the two temporally nearest bins in $E$ as following:

$\begin{split}E(x,y,t)&=\sum_{i}p_{i}\max(0,1-|t-t_{i}^{*}|),\\\
t_{i}^{*}&=\frac{(t_{i}-t_{min})}{t_{max}-t_{min}}(B-1),\end{split}$ (1)

where $x$, $y$ and $t$ are the coordinates in the x-y-time dimensions.
$t_{min}$ and $t_{max}$ are the beginning and end times of the frame interval,
which is temporally divided into $B$ bins.

###  III-B Network Architecture

As shown in Fig. 2, we first extract coarse features from image frame $I$ and
event Voxel Grid $E$, respectively. Subsequently, the image feature $F_{i}$
and event feature $F_{e}$ are fed into the Fusion Feature Extractor (FFE).
Within the FFE, a unified feature $F_{fus}$ is generated, with a focus on
emphasizing features at the end of the exposure time. This fused feature
effectively combines information from two modalities. The event features
$F_{e}$ are fed into the Motion Extractor (ME) to extract motion information,
represented as a set of offsets $\mathcal{O}$ = {$\mathit{Offset}^{N-2}$,
$\mathit{Offset}^{N-3}$, …, $\mathit{Offset}^{0}$}. This motion information is
subsequently supplied to the Motion-Aware Head (MAH), which performs implicit
warping of $F_{fus}$ to different time instants within the frame interval and
produces a set of heatmaps {$H^{0}$, $H^{1}$, …, $H^{N-1}$} corresponding to
each instant. Here, $N$ represents the number of output heatmaps, with a
larger $N$ resulting in higher temporal resolution for detection. The entire
process is designed to produce more stable detection results, which is also
the key to achieving long-term tracking.

####  III-B1 FFE

To adaptively extract complementary information from images and events, we
employ dynamic filters to enhance one modality with respect to the other. The
process is formulated as:

$\begin{split}F_{ei}&=\mathrm{Conv_{1\times
1}}(\mathcal{K}_{i}\circledast\mathcal{F}+\mathcal{F}),\\\
\mathcal{K}_{i}&=\mathrm{Conv_{3\times 3}}(\mathcal{A}(F_{i})),\\\
\mathcal{F}&=\mathrm{ReLU}(\mathrm{BN}(\mathrm{Conv_{3\times 3}}(F_{e}))),\\\
\end{split}$ (2)

where $\circledast$ represents depthwise convolution, and $\mathcal{A}$
signifies adaptive average pooling. It is worth noting that Eq. (2) is
symmetric when processing both the event and image frame.

After concatenating $F_{ei}$ and $F_{ie}$, the combined features are passed
through the CMT Block [31], which can realize global inter-modal feature
interactions and information fusion. Subsequently, a combination of
convolution, Batch Normalization (BN), and Rectified Linear Unit (ReLU) is
applied for the adjustment of the channel dimensions. The features are then
processed by a ConvLSTM layer [32], enhanced with residual connections, to
produce the final fused feature $F_{fus}$. Meanwhile, ConvLSTM handles the
propagation of temporal information, ensuring that the network produces
temporally consistent results.

####  III-B2 ME

The Motion Extractor is responsible for extracting motion information from the
high-temporal-resolution events. We utilized the MA module [33] to extract
motion features, denoted as $F_{m}$. The MA module employs spatial and channel
attention mechanisms to enhance valuable information within the event
modality, simultaneously emphasizing motion cues while suppressing noise.
Subsequently, $F_{m}$ is passed through a two-layer convolution to obtain the
required offsets for deformable convolution in MAH.

####  III-B3 MAH

The Motion-Aware Head is responsible for propagating the stable fusion feature
$F_{fus}$ to various time instants within the frame interval using motion
information $\mathcal{O}$. The module employs an iterative strategy that
incorporates deformable convolution, which can achieve implicit warping.

We first extract the feature $F^{N-1}$ aligned with the end of the exposure
time. Subsequently, we employ iterative steps through a combination of
deformable convolution, BN, and ReLU to propagate $F^{N-1}$ to other time
instants, therefore obtain features {$F^{N-2}$, $F^{N-3}$, …, $F^{0}$} for
different time instants. This process can be formulated as follows:

$\begin{split}F^{N-1}&=\mathrm{ReLU}(\mathrm{BN}(\mathrm{Conv_{3\times
3}}(F_{fus}))),\\\ F^{n-1}&=\mathrm{ReLU}(\mathrm{BN}(\mathrm{DConv_{3\times
3}}(F^{n},\mathit{Offset}^{n-1}))),\\\ \end{split}$ (3)

where $n\in\mathbb{Z},\,1\leq n\leq N-1$.

After applying channel attention [34], the features {$F^{N-1}$, $F^{N-2}$, …,
$F^{0}$} are fed into the Base Head, which is composed of several
convolutional structures with a Sigmoid function to normalize the heatmap
outputs. As a result, a set of keypoint heatmaps {$H^{N-1}$, $H^{N-2}$, …,
$H^{0}$} are generated.

###  III-C Loss Function

Generally, pseudo-labels of keypoints are generated on image frames with
Harris or SIFT features [21]. These pseudo-labels are subsequently used to
supervise event-based methods. Although this kind of supervision is
straightforward, it essentially mimics handcrafted detectors without
considering their applicability to event data. Additionally, these methods
care little about the keypoint’s repeatability.

Inspired by the self-supervised frame-based keypoint detection methods [1, 2,
3], we employ a loss function $L_{consist}$ based on temporal response
consistency to supervise the network. It relies solely on the image
transformation between different time instants. We achieve this by utilizing
temporally-sequenced heatmaps generated by our network and comparing them at
various time instances. This loss function encourages the network to identify
keypoint locations that remain stable across the entire temporal axis, while
also minimizing the variance of responses at these locations. It is expressed
as follows:

$L_{consist}=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\mathcal{C}[p],$
(4)
$\mathcal{C}[p]=\mathrm{Dist}(H^{n}[p],\mathrm{Warp}(H^{n-1},\mathcal{T}^{(n-1,n)})[p]),$
(5)

where $\mathcal{P}=\\{p\\}$ represents a collection of $M\times M$ patches
extracted from the image region. $\mathrm{Dist}$ is a distance metric such as
cosine similarity or L1 distance that reflects the difference between images.
And $\mathrm{Warp}(*,\mathcal{T}^{(a,b)})$ denotes the operation of warping
the image from time $a$ to time $b$ using the image correspondence
$\mathcal{T}^{(a,b)}$. $\mathcal{C}[p]$ signifies the magnitude of consistency
of patch $p$.

$L_{consist}$ tends to flatten the response values between each patch,
resulting in a large and smooth response on the network output. Thus,
following [3][35], we incorporate an additional loss function $L_{peaky}$ for
assistance:

$\begin{split}L_{peaky}=1-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}(\max
H^{n}[p]-\mathop{{\rm mean}}\limits H^{n}[p]).\end{split}$ (6)

$L_{peaky}$ stretches the responses between local peaks and local averages,
thereby preventing local smoothing. However, $L_{peaky}$ tends to generate
responses on each patch, which does not align with the reality of the
situation. Specifically, intensity-homogeneous areas are not informative for
keypoint detection and tracking. Therefore, building upon the $L_{peaky}$, we
propose a Consistency Peaky Loss $L_{cp}$ that includes a consistency mask to
address this issue. In intensity-homogeneous regions, the responses in
heatmaps tend to be uniform, which does not align with temporal consistency.
In other words, the responses transforming from time $a$ to time $b$ do not
align with the responses at time $b$, resulting in a large distance metric
value. Based on this observation, we suppress the responses in regions with
large distance metrics while simultaneously enhancing the peaks in other
regions:

$\begin{split}L_{cp}=1&-\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}(\max
H^{n}[p]-\mathop{{\rm mean}}\limits H^{n}[p])\mathcal{M}[p]\\\
&+\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}\mathop{{\rm mean}}\limits
H^{n}[p](1-\mathcal{M}[p]),\end{split}$ (7)
$\mathcal{M}=\frac{\mathcal{C}-\min(\mathcal{C})}{\max(\mathcal{C})-\min(\mathcal{C})},$
(8)

where $\mathcal{C}$ is the consistency magnitude defined in Eq. (5).
$L_{peaky}$ can be viewed as a special case of $L_{cp}$ with $\mathcal{M}=1$.

The overall loss function is a weighted sum of the consistency loss and the
consistency peak loss,

$L=L_{consist}+\alpha L_{cp}.$ (9)

The specific setting of $\alpha$ is detailed in the section III-D3.

###  III-D Implementation Details

####  III-D1 Spatio-Temporal Nearest-Neighbor Tracking

Benefiting from the high temporal resolution of events, our keypoint network
generates a sequence of heatmaps at evenly-spaced time intervals within the
frame interval. We identify positions on these heatmaps with values exceeding
0.95 as keypoints. To track these keypoints, we employ a spatio-temporal
nearest-neighbor strategy, which is designed to eliminate the drifts during
the tracking process.

For each identified keypoint, we search for a neighboring keypoint within a
spatial radius of 4 pixels and a temporal window of 12 milliseconds. If a
neighbor is found, we associate the new keypoint with its existing track. In
cases where multiple neighboring keypoints are found, we prioritize the
closest one for correspondence. Conversely, if no neighboring keypoint is
found, we initiate a new tracking sequence specifically from the new keypoint.

####  III-D2 Training set

Following the approach in [21], we apply a series of homography
transformations
{$\mathcal{T}^{(0,1)},\mathcal{T}^{(0,2)},...,\mathcal{T}^{(0,{M_{1})}}$} to
an initial image $I^{0}$ from COCO [36]. This yields an image sequence
{$I^{1},I^{2},...,I^{M_{1}}$} that encompasses motion:

$I^{n}=\mathrm{Warp}(I^{0},\mathcal{T}^{(0,n)}),n\in\mathbb{Z},\,1\leq n\leq
M_{1}.$ (10)

TABLE I: Performance comparison on our collected dataset. “-” indicates cases with frame intervals exceeding $\delta t$. Scene | Method | $\delta t=25ms$ | $\delta t=50ms$ | $\delta t=100ms$ | $\delta t=150ms$ | $\delta t=200ms$ | Track Time (s)$\uparrow$
---|---|---|---|---|---|---|---
RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$
Overexposure | Shi-Tomasi[37]+LK[28] | 2.47 (0.00) | 3.97 (0.0) | 17.21 (0.00) | 10.02 (0.01) | 24.02 (0.02) | 3.38
Superpoint[1] | 22.46 (0.00) | 21.76 (0.00) | 18.38 (0.00) | 36.65 (0.01) | 23.61 (0.01) | 9.25
Superpoint*[1] | 1.80 (0.00) | 2.18 (0.00) | 2.64 (0.00) | 3.02 (0.01) | 3.24 (0.01) | 8.78
Long-lived[21] |  2.36 (0.05) |  2.16 (0.19) |  2.07 (0.40) |  2.51 (0.52) |  1.82 (0.65) | 0.43
FE-DeTr (Ours) |  1.76 (0.00) |  1.90 (0.00) |  2.01 (0.01) |  2.10 (0.02) |  2.08 (0.03) | 2.15
Dark | Shi-Tomasi[37]+LK[28] | 5.63 (0.01) | 9.02 (0.05) | 9.02 (0.10) | 58.14 (0.17) | 26.20 (0.25) | 1.63
Superpoint[1] | 64.31 (0.00) | 47.43 (0.01) | 28.55 (0.06) | 21.71 (0.16) | 16.07 (0.25) | 3.16
Superpoint*[1] | 5.59 (0.00) | 8.09 (0.00) | 7.95 (0.03) | 7.70 (0.06) | 8.06 (0.09) | 3.83
Long-lived[21] |  2.28 (0.13) |  2.06 (0.27) |  1.84 (0.53) |  1.71 (0.71) |  2.11 (0.79) | 0.36
FE-DeTr (Ours) |  1.57 (0.03) |  1.86 (0.07) |  1.89 (0.16) |  2.12 (0.28) |  1.92 (0.39) | 0.91
HDR | Shi-Tomasi[37]+LK[28] | 1.75 (0.01) | 2.64 (0.02) | 4.17 (0.04) | 5.72 (0.06) | 8.20 (0.08) | 3.49
Superpoint[1] | 8.01 (0.00) | 8.61 (0.00) | 16.52 (0.01) | 10.66 (0.03) | 9.19 (0.05) | 9.01
Superpoint*[1] | 1.67 (0.00) | 1.97 (0.00) | 2.32 (0.00) | 2.51 (0.01) | 2.83 (0.01) | 8.71
Long-lived[21] |  2.20 (0.08) |  2.00 (0.16) |  2.20 (0.35) |  2.11 (0.53) |  1.74 (0.64) | 0.50
FE-DeTr (Ours) |  1.73 (0.00) |  1.86 (0.02) |  1.85 (0.06) |  1.81 (0.10) |  1.75 (0.13) | 2.49
Blur | Shi-Tomasi[37]+LK[28] | - | - | 2.67 (0.06) | - |  2.85 (0.10) | 1.49
Superpoint[1] | - | - | 20.09 (0.01) | - | 20.05 (0.08) | 3.93
Superpoint*[1] | - | - | 5.31 (0.01) | - | 11.41 (0.09) | 2.24
Long-lived[21] |  1.78 (0.01) |  1.74 (0.03) |  1.73 (0.05) |  1.71 (0.08) |  1.79 (0.16) | 0.74
FE-DeTr (Ours) |  1.60 (0.00) |  1.96 (0.01) |  2.67 (0.06) |  2.85 (0.10) | 3.46 (0.17) | 1.49

Subsequently, we simulate an event stream from this image sequence using an
event simulator [21]. We define $M_{2}<M_{1}$ as the frame interval and
uniformly divide the image sequence{$I^{1},I^{2},...,I^{M_{1}}$} into
intervals
$\\{S^{1}=\\{I^{1},I^{2},...,I^{M_{2}}\\},S^{2},...,S^{{M_{1}}/{M_{2}}}\\}$.
For each frame interval $S^{n}$, we select the last $M_{3}$ frames and
calculate their average to obtain a blurred image $I$. Here, $M_{3}$ signifies
the exposure time within the range $[0.2M_{2},0.6M_{2}]$. Additionally, we
apply random data augmentation to improve the robustness of model training,
including random event noises and random grayscale transformations on blurred
images.

####  III-D3 Training details

We configure our system with the following parameters: $B=10$ for event
representation, $N=10$ for network outputs, $M=30$ for patch size, and employ
cosine similarity as the distance metric for our loss function. We use
truncated-backpropagation-through-time of 10-time steps and employ the Adam
optimizer for the training process, which consists of two stages:

(a) We first train the network for 30 epochs with an initial learning rate
$lr$ of 0.0003 and an initial $\alpha$ of 0.25. At the 6th, 12th, and 18th
epochs, we dynamically adjust $lr$ by reducing it to $75\%$ of its previous
value while simultaneously increasing $\alpha$ by a factor of 2. This stage is
designed to enable the network to learn temporal motion relationships, thereby
enhancing the consistency of its output detection.

(b) We then train for 1 epoch with $lr=0.0003$ and $\alpha=2.0$, utilizing the
standard mask $\mathcal{M}$ representation in the Eq. (8). The primary
objective is to suppress responses from homogeneous regions to reduce false
positives.

##  IV Experiments

In this section, we first introduce the collected dataset and the evaluation
metrics, and then we compare the proposed method with the state-of-the-art
image-based and event-based methods. Finally, we give the ablation study of
FE-DeTr.

Figure 3: Tracking trajectories comparison under different conditions: Blur
(1st row), overexposure (2nd row), dark (3rd row), and HDR (4th row).

###  IV-A Datasets and Metrics

Since no publicly available keypoint dataset captures both image frames and
event data under extreme conditions, we create a new keypoint detection and
tracking dataset, named Extreme Corner. Following the setup of the HVGA ATIS
Corner Dataset [19], we employ a DAVIS346 event camera to capture planar
patterns of 4 distinct images to simplify the evaluation process. We set up
three extreme lighting scenarios: Overexposure, Dark, and High Dynamic Range
(HDR). For each scenario, we collect an event stream of approximately 10
seconds in duration, accompanied by images captured at a frame interval of 25
milliseconds. To simulate Blur condition, we extend the frame interval and
exposure time, and then record event streams and corresponding images at a 100
milliseconds frame interval, also for roughly 10 seconds. Under each extreme
condition, we record two sequences for each planar scene, culminating in a
comprehensive dataset comprising 32 sequences.

TABLE II: Ablation study on the setup of the proposed FE-DeTr. “-” indicates cases with frame intervals exceeding $\delta t$. “NaN” means failed cases. “/” indicates that the total number of tracking trajectories is less than 100. Scene | Setup | $\delta t=25ms$ | $\delta t=50ms$ | $\delta t=100ms$ | $\delta t=150ms$ | $\delta t=200ms$ | Track Time (s)$\uparrow$
---|---|---|---|---|---|---|---
RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$ | RPE (RFM)$\downarrow$
Extreme Lighting | (a). w/o $L_{cp}$ | 2.17 (0.00) | 2.55 (0.00) | 2.93 (0.01) | 3.07 (0.03) | 3.66 (0.04) | 2.27
(b). w/o MAH | 2.58 (0.01) | 4.19 (0.04) | 3.13 (0.13) | 4.98 (0.22) | 3.13 (0.31) | 0.93
(c). w/o event | 2.47 (0.42) | 8.08 (0.83) |  2.03 (0.93) | NaN (1.00) | NaN (1.00) | /
(d). w/o frame |  1.30 (0.39) |  1.59 (0.55) | 2.26 (0.75) |  2.24 (0.88) | 5.12 (0.94) | 0.29
(e). full |  1.69 (0.01) |  1.87 (0.03) |  1.92 (0.08) |  2.00 (0.13) |  1.92 (0.18) | 1.85
Blur | (a). w/o $L_{cp}$ | 1.73 (0.00) | 2.23 (0.00) | 3.34 (0.00) | 3.92 (0.02) | 4.22 (0.06) | 1.93
(b). w/o MAH | 2.37 (0.00) | 3.01 (0.01) | 4.29 (0.09) | 5.96 (0.21) | 4.76 (0.30) | 0.91
(c). w/o event | - | - | 5.07 (0.40) | - | NaN (1.00) | 0.41
(d). w/o frame |  1.34 (0.08) |  1.55 (0.13) |  1.81 (0.27) |  2.13 (0.45) |  2.60 (0.60) | 0.52
(e). full |  1.60 (0.00) |  1.96 (0.01) |  2.67 (0.06) |  2.85 (0.10) |  3.46 (0.17) | 1.49

We use two key metrics for evaluation: the $\delta t$-homography Reprojection
Error (RPE) for assessing detection accuracy, and the average Track Time for
examining detection stability. Definitions of these metrics can be found in
[21]. Alongside the RPE, we report the Ratio of Failed Matches (RFM), which
indicates the ratio of failed matches to total matches, thereby reflecting the
reliability of RPE.

###  IV-B Result Comparisons

To demonstrate the effectiveness of our method, we select several traditional
and SOTA learning-based methods using either image frames or event streams for
comparison. (1) Shi-Tomasi [37] \+ Lucas-Kanade [28] represents a traditional
frame-based keypoint detection and optical flow method. (2) Superpoint [1]
represents a learning-based keypoint detection and description method also on
frames. (3) Long-lived [21] is a state-of-the-art event-based keypoint
detection and tracking method. To ensure a fair comparison, all methods
refrain from utilizing enhancements such as image pyramids or feature
pyramids. Parameters for all methods are adapted to achieve the best
performance on the same dataset.

Superpoint generates numerous mismatches on regular chessboard sequences
because corners on the chessboard exhibit extremely high similarity.
Therefore, we also report its performance on the proposed dataset without the
chessboard sequences, denoted as Superpoint* in Table I for reference.

As shown in Table I, the proposed FE-DeTr achieves the best accuracy under
extreme lighting conditions, while also maintaining robust performance in
terms of accuracy, failure rate, and tracking time under blur conditions.
Frame-based methods face challenges due to the absence of mechanisms to filter
out mismatched points, leading to protracted tracking times and a concomitant
decrease in accuracy. The event-based method Long-lived [21] achieves good
accuracy in low-noise and high-speed motion scenes (Blur). However, its
stability is compromised under extreme lighting conditions, reflected by its
high RFM and extremely low Track Time. This performance degradation is
especially pronounced in dark (low light) scenes filled with event noise. By
integrating the complementary information of image frames and event data, the
proposed FE-DeTr achieves a balance between accuracy and stability.

Furthermore, we qualitatively plot the tracking trajectories of different
methods in Fig. 3. On the one hand, the proposed FE-DeTr exhibits smoother
trajectories compared to frame-based methods. On the other hand, our method
achieves longer tracking trajectories compared to event-based methods. These
results further demonstrate the effectiveness of our method under various
challenging conditions.

###  IV-C Ablation Study

Impact of Consistency Peaky Loss. To validate the effectiveness of the
proposed Consistency Peaky Loss in eliminating false positives, we present the
results of FE-DeTr without $L_{cp}$ in row (a) of Table II. Compared with the
full model in row (e), it can be observed that the absence of this loss
increases false responses, leading to a greater number of incorrect point
matches. While this extends tracking lifetimes, it also decreases accuracy.
Fig. 4 shows a comparison between the output heatmaps generated by a model
trained without $L_{cp}$ loss versus the full model.

Figure 4: Reference image frame (left); output heatmap after training without
$L_{cp}$ (middle); output heatmap after $L_{cp}$ supervision (right).

Impact of MAH. We verify the effectiveness of the MAH module by replacing the
deformable convolution with regular convolution and removing the iterative
steps. The results are shown in row (b) of Table II. Compared with the full
model in row (e), we can find a significant performance drop after the
modification, highlighting the strong capability of MAH in utilizing motion
information to improve the response’s temporal consistency.

Impact of Input Modalities. To validate the effectiveness of fusing images and
events, we test the FE-DeTr on a single modality with the other input to 0.
The results of rows (c) and (d) of each scene in Table II show the performance
on images and events, respectively. When using only the frame input, a
noticeable performance drop can be observed. This is because using only the
frame modality can’t generate the high-temporal-resolution responses required
for tracking. When utilizing only the event modality, tracking accuracy is
improved, indicating that the keypoint localization accuracy is somewhat
influenced by low-quality images. However, the higher RFM and extremely short
tracking time suggest poor stability when only using events, especially in
extreme lighting conditions with a high noise level. The fusion of image
frames significantly improves the stability of detection and tracking.

##  V Conclusions and Future Work

In this paper, we propose FE-DeTr, a keypoint detection and tracking method
that integrates both images and events. The keypoint detection network is
supervised based on the relative motion relationships at different time
instants, enabling the exploitation of complementary information between two
modalities. Compared to prior arts, FE-DeTr achieves the best comprehensive
performance with high localization accuracy and stable tracking duration.
Future work will focus on applying FE-DeTr to the downstream SLAM task.

##  VI Acknowledgement

This work was supported by NSFC grants under Grant 62301370 and 62271354, the
Natural Science Foundation of Hubei Province, China under Grant 2022CFB600.

## References

  * [1] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 224–236.
  * [2] P. H. Christiansen, M. F. Kragh, Y. Brodskiy, and H. Karstoft, “Unsuperpoint: End-to-end unsupervised interest point detector and descriptor,” _arXiv preprint arXiv:1907.04011_ , 2019.
  * [3] J. Revaud, P. Weinzaepfel, C. D. Souza, and M. Humenberger, “R2d2: Repeatable and reliable detector and descriptor,” in _Proceedings of the International Conference on Neural Information Processing Systems_ , 2019, p. 12414–12424.
  * [4] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2758–2766.
  * [5] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _European Conference on Computer Vision_ , 2020, pp. 402–419.
  * [6] Y. Tian, B. Fan, and F. Wu, “L2-net: Deep learning of discriminative patch descriptor in euclidean space,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 661–669.
  * [7] W. Jiang, E. Trulls, J. Hosang, A. Tagliasacchi, and K. M. Yi, “Cotr: Correspondence transformer for matching across images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2021, pp. 6207–6217.
  * [8] C. Doersch, A. Gupta, L. Markeeva, A. Recasens, L. Smaira, Y. Aytar, J. Carreira, A. Zisserman, and Y. Yang, “Tap-vid: A benchmark for tracking any point in a video,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 13 610–13 626, 2022.
  * [9] A. W. Harley, Z. Fang, and K. Fragkiadaki, “Particle video revisited: Tracking through occlusions using point trajectories,” in _European Conference on Computer Vision_ , 2022, pp. 59–75.
  * [10] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman, “Tapir: Tracking any point with per-frame initialization and temporal refinement,” _arXiv preprint arXiv:2306.08637_ , 2023.
  * [11] Z. Zhao, Y. Zhai, B. M. Chen, and P. Liu, “Balf: Simple and efficient blur aware local feature detector,” _arXiv preprint arXiv:2211.14731_ , 2022.
  * [12] G. Gallego, T. Delbrück, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, _et al._ , “Event-based vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 1, pp. 154–180, 2020.
  * [13] X. Zhang, L. Yu, W. Yang, J. Liu, and G.-S. Xia, “Generalizing event-based motion deblurring in real-world scenarios,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 10 734–10 744.
  * [14] K. Chen and L. Yu, “Motion deblur by learning residual from events,” _IEEE Transactions on Multimedia_ , 2024.
  * [15] V. Vasco, A. Glover, and C. Bartolozzi, “Fast event-based harris corner detection exploiting the advantages of event-driven cameras,” in _IEEE International Conference on Intelligent Robots and Systems_ , 2016, pp. 4144–4149.
  * [16] E. Mueggler, C. Bartolozzi, and D. Scaramuzza, “Fast event-based corner detection,” in _British Machine Vision Conference_ , 2017.
  * [17] I. Alzugaray and M. Chli, “Asynchronous corner detection and tracking for event cameras in real time,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3177–3184, 2018.
  * [18] R. Li, D. Shi, Y. Zhang, K. Li, and R. Li, “Fa-harris: A fast and asynchronous corner detector for event cameras,” in _IEEE International Conference on Intelligent Robots and Systems_ , 2019, pp. 6223–6229.
  * [19] J. Manderscheid, A. Sironi, N. Bourdis, D. Migliore, and V. Lepetit, “Speed invariant time surface for learning to detect corner points with event-based cameras,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 245–10 254.
  * [20] P. Chiberre, E. Perot, A. Sironi, and V. Lepetit, “Detecting stable keypoints from events through image gradient prediction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1387–1394.
  * [21] C. Philippe, P. Etienne, S. Amos, and L. Vincent, “Long-lived accurate keypoints in event streams,” _arXiv preprint arXiv: 2209.10385_ , 2022.
  * [22] A. Z. Zhu, N. Atanasov, and K. Daniilidis, “Event-based feature tracking with probabilistic data association,” in _IEEE International Conference on Robotics and Automation_ , 2017, pp. 4465–4470.
  * [23] D. Gehrig, H. Rebecq, G. Gallego, and D. Scaramuzza, “Asynchronous, photometric feature tracking using events and frames,” in _European Conference on Computer Vision_ , 2018, pp. 750–765.
  * [24] N. Messikommer, C. Fang, M. Gehrig, and D. Scaramuzza, “Data-driven feature tracking for event cameras,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 5642–5651.
  * [25] H. Yu, H. Li, W. Yang, L. Yu, and G.-S. Xia, “Detecting line segments in motion-blurred images with events,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [26] K. M. Yi, E. Trulls, V. Lepetit, and P. Fua, “Lift: Learned invariant feature transform,” in _European Conference on Computer Vision_ , 2016, pp. 467–483.
  * [27] P. Gleize, W. Wang, and M. Feiszli, “Silk–simple learned keypoints,” _arXiv preprint arXiv:2304.06194_ , 2023.
  * [28] B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in _International Joint Conference on Artificial Intelligence_ , vol. 2, 1981, pp. 674–679.
  * [29] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8934–8943.
  * [30] A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis, “Unsupervised event-based learning of optical flow, depth, and egomotion,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 989–997.
  * [31] J. Guo, K. Han, H. Wu, Y. Tang, X. Chen, Y. Wang, and C. Xu, “Cmt: Convolutional neural networks meet vision transformers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 175–12 185.
  * [32] X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Proceedings of the International Conference on Neural Information Processing Systems_ , 2015, pp. 802–810.
  * [33] J. Zhang, Y. Wang, W. Liu, M. Li, J. Bai, B. Yin, and X. Yang, “Frame-event alignment and fusion network for high frame rate tracking,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 9781–9790.
  * [34] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7132–7141.
  * [35] L. Zhang and S. Rusinkiewicz, “Learning to detect features in texture images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6325–6333.
  * [36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision_ , 2014, pp. 740–755.
  * [37] J. Shi and C. Tomasi, “Good features to track,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1994, pp. 593–600.