gensymbNot defining

# Distinctive pupil and microsaccade-rate signatures in self-recognition

Lisa Schwetlick1,3,6,∗ Hendrik Graupner2,4
Olaf Dimigen5 Ralf Engbert1,3

###### Abstract

Pupil dynamics and fixational eye movements are involuntary processes that
actively support visual perception during fixations. Both measures are known
to be sensitive to ongoing cognitive and affective processing. In a visual
fixation experiment ($N$=116) we demonstrate that self-recognition, familiar
faces, and unfamiliar faces elicit specific responses in pupil dynamics and
microsaccade rate. First, the pupil response comprises an immediate pupil
constriction followed by a dilation in response to stimulus onsets. We observe
attenuated constriction and greater dilation when faces are recognized
compared to unknown faces. This effect is strongest for one’s own face.
Second, microsaccade rates, which generally show inhibitory responses to
incoming stimuli, generate stronger inhibition for familiar faces compared to
unknown faces. Again, the strongest inhibition is observed in response to
one’s own face. Our results imply that eye-related physiological measures
expose hidden knowledge in face memory and could contribute to biometric
authentication and identity validation procedures.

Keywords: face perception, self-recognition, pupillometry, microsaccades

1 Department of Psychology, University of Potsdam, Germany
2 Bundesdruckerei GmbH, Berlin, Germany
3 DFG Collaborative Research Center 1294, University of Potsdam, Germany
4 Hasso Plattner Institute and University of Potsdam, Germany
5 Faculty of Behavioural and Social Sciences, University of Groningen,
Netherlands
6 Laboratory of Psychophysics, EPFL, Lausanne, Switzerland
*<EMAIL_ADDRESS>
##  1 Introduction

Eye movements are an essential component of visual perception. Easy to observe
are rapid eye movements (saccades) that generate sequences of gaze shifts and
periods of relative rest (fixations) that give rise to the eye’s trajectory
during scene exploration [62] or everyday activities. In the laboratory, we
can also investigate, microscopic eye movements that occur involuntarily
during fixations: microsaccades and changes in pupil size. Here we investigate
how both systems are influenced by face recognition.

Microsaccades are generated by the same machinery of extraocular muscles that
drive gaze shifts. These small-amplitude fixational eye movements share their
kinematic relations with larger saccades [52]. Microsaccades are distinguished
from the larger saccades primarily by their size [78, but not exclusively,
see], typically <1∘ of visual angle. During fixations, they serve vital
functions for visual processing, e.g., counteracting perceptual fading [74].
The second type of investigated eye movement, pupil dilation, driven by
intraocular muscles and primarily adapts pupil size to the surrounding
illumination (pupillary light reflex or PLR) [75]. Both microsaccades and
pupil dilation are modulated by ongoing cognitive [58] and emotional
processing [63, 99]. In the current study we set out to investigate how the
recognition of unfamiliar and familiar faces–including one’s own face–affects
microsaccades and pupil size.

Face recognition is tightly coupled to visual, cognitive and affective
processing. Evidence from the literature suggests that faces are processed in
the visual system by unique, specialized cognitive and neural mechanisms [77,
100] in the fusiform gyrus of the inferior temporal (IT) cortex [67] and in
prefrontal cortex [85]. Response latency for face-sensitive cells in IT cortex
is reported to be between 50 ms [95] and 160 ms [80]. The recognition of a
specific person, or even oneself, engages the same visual processing circuitry
as a new, unfamiliar face, but additionally causes memory retrieval, context-
dependent surprisal, and emotional responses [65]. In order to isolate visual
processing components of recognition, previous studies typically used faces
that entrained during the experiment (unfamiliar face recognition) or famous
faces (familiar face recognition). We expect that semantic and emotional
responses will be significantly stronger for faces of real-life acquaintances
and friends, as well as oneself (self-recognition).

Pupil size is a well-established physiological measure of ongoing cognition
that varies between 1.5 mm to 9 mm in diameter [91]. Modern pupillometry
distinguishes three types of pupil response: The pupilary light reflex (PLR),
pupil near response (PNR), and psychosensory pupil responses [75]. Of these,
the PLR is the strongest component, causing pupil constriction within the
first 200 ms after stimulus onset; the minimum pupil size is reached between
200 ms [75] and 1600 ms [54]. In addition to external factors, pupil size is
also determined by cognition and arousal, modulated by variables such as
interest or processing load. Physiologically, pupil size is connected to brain
regions that control sleep-wake rhythms and nervous-system activation [91].
Activity in the Locus Coeruleus, which is involved in memory retrieval and
selective attention, is highly correlated with pupil size in monkeys [70].
Early cognitive effects on pupil size (within 1 s) are related to novelty and
saliency [75, 91, 63] as well as surprisal, uncertainty, and prediction errors
[71]. The effects of (positive and negative) arousal and mental effort are
associated with longer delays [75, see], while emotional responses peak last
(>2 s after target onset) [68].

For microsaccades during cognitive tasks, the baseline rate is usually around
1 per second (1 Hz). Changes in visual input, however, can strongly modulate
the rate [60, 56, 73]. In particular, any change in the display causes a
temporary decrease in the microsaccade rate (MSR). The microsaccadic
inhibition is followed by a rise in MSR, often exceeding the baseline rate.
500-1000 ms after stimulus onset the MSR returns to its resting state.
Display-change-related microsaccade inhibition is considered a reflexive
response, generated at the level of the superior colliculus (SC) [72, 84]. The
SC, aside from generating low-level oculomotor signals, receives input from
various regions potentially conveying top-down information [92, 89]. In
summary, both previous behavioral data as well as our knowledge of the
underlying neurophysiology suggest that pupil size and MSR could vary
substantially across conditions during face recognition.

Recent studies have explored pupil dilation and MSR in the context of face
recognition, using a paradigm measuring reactions to rapid presentation of
faces. [55] showed streams of unknown faces, target faces, and a familiar face
(the participants mother) and found that pupil dilation is modulated by face
recognition, but found no effects on MSR. [86] specifically probed for
microsaccadic inhibition using a similar rapid presentation paradigm, where
they compared (1) a pre-learned target face (the “terrorist”) within a set of
8 unknown faces [86] or (2) a famous faces among unknown faces [87]. They
found evidence of stronger microsaccadic inhibition related to face
familiarity.

The goal of the present work was to establish in a large-scale eye-tracking
experiment that pupil dilation and microsaccade dynamics reveal face
recognition, and more specifically also self-recognition (as an extreme
example of a familiar face). We manipulated face familiarity by using pictures
of the participants classmates, yielding a well-controlled design with a high
statistical power. We were also interested in the influence of image
repetition, individual differences, the time-course of both measures, which
are highly relevant for potential applications. Compared to unfamiliar faces
and pre-learned faces, real-life familiar faces should be intrinsically
relevant and meaningful to the observer, even in the absence of a specific
task. We therefore hypothesized that recognition should produce reliable
oculomotor signature, akin to an oddball response. In an oddball paradigm,
participants see a sequence of frequent (expected) stimuli, interspersed with
infrequent (and therefore unexpected) but task-relevant targets (the oddball
stimulus) [96], while performing a task such as silently counting targets.
This paradigm [82, see] has revealed that a particular event-related potential
(P300, peaking around 300 ms after presentation) is strongest when the
participant is engaged in target detection [93, 96, 81] and is modulated by
the meaningfulness of stimuli [64], task difficulty [81], as well as the
internal arousal level and availability of attentional resources [69].

The oddball task has been found to affect pupil dilation [66, 94, 79], likely
via the Locus Coeruleus [79], as well as MSR [97, 98]. We expect face
recognition to comprise various reflexive, cognitive, and emotional components
and to be similar to the surprisal and increased attention modulated via
target-frequency in the oddball task. The recognition of familiar faces should
be related to a reduced MSR and increased pupil dilation, compared to
unfamiliar faces. Moreover, we expect recognition of one’s own face to evoke
the strongest response, compared to familiar and unfamiliar faces.

From the literature reviewed above, we conclude that both pupil size and MSR
are physiological measures that are functionally-driven but modulated by
ongoing cognition. The aim of the present study is to investigate whether
pupil dynamics and MSRs are specifically affected by face recognition
processes. Reliable differences in pupil dynamics and microsaccade statistics
between self-recognition and recognition of other faces are relevant to
applications in the context viewer identification or exposing viewer
knowledge.

##  2 Methods

###  2.1 Participants

We recruited participants from two graduating high school classes in Potsdam,
Germany. The advantage of this setup is that it allows a well-controlled
design where each participant personally knows a subset of the other
participants. Thus, each participant was shown faces that were either their
classmates’ faces, who they knew personally (_Peers_), or students from the
other high school (_Strangers_), or their own (_Self_). Of the initial cohort
of 127 students who’s pictures were taken, 118 students came to the eye-
tracking session. Of these, two individuals were not able to be calibrated in
the eye tracker. 116 complete data sets remain. The participants were between
the ages of 16 and 18 years old; 55 were male, 63 female and one identified as
non-binary.

The eye tracking data collection was followed by a further questionnaire,
where each participant saw the same faces they had seen during the experiment
and was asked to confirm whether they did indeed know the person in each
picture. Cases where a participant’s response to the question did not match
with the expectation, i.e., when they did know someone from the other class or
did not know someone from their own class, were removed from the final data
set. The final data set is therefore balanced in the sense that each image, in
principle, appears in each condition.

###  2.2 Photographs

In order to collect the required photographs of the participants, a
professional photographer visited the schools. Each participating student
received an anonymous code. The photographs were taken under consistent
illumination and with the heads centered, in a quasi-biometric setup, in order
to maximize consistency between the pictures. The photographs were coded with
the anonymous code, so that the mapping between participant and photograph was
possible without the necessity of saving any identifying information alongside
the image. The high-resolution photographs were further cropped and scaled, to
center each face inside a square image. For data-privacy reasons the
photographs of the faces were deleted upon completion of the study. In the
data set only the anonymous codes remain.

We computed the relative image luminance for each image and found it to be
normally distributed according to a Shapiro Wilk [90] test of normality
(p=0.24). As each image is shown in each condition (i.e., every image is seen
as Peers, Strangers, and Self), differences in image statistics would effect
all groups equally and not influence the main effects.

###  2.3 Eye Tracking Data Collection

For recording eye trajectories we used an Eyelink 1000 eye tracker (SR
Research, Osgoode, ON, Canada), which recorded both eyes at 500 Hz at an
illumination level of 75%. The screen,a ViewPIXX monitor (VPixx Technologies,
Saint-Bruno, Canada) with a resolution of 1920$\times$1080 pixels, was placed
at 70 cm distance from the participant, with the head stabilized in a chin
rest. The eyes were centered at $3/4$ of the height of the screen. We used a
5-point calibration grid and subjects were re-calibrated every 14 trials. As
the experiment required exclusively fixation in the center (and no exploration
of the outlying regions of the screen), a 5-point calibration was sufficient
to ensure data quality.

Face images were shown at a resolution of 500$\times$500 pixels centered on
the screen, meaning that each image subtended 11.4∘ of visual angle. The
experimental session proceeded as follows. Participants were informed that
they would be seeing pictures of faces and were asked to fixate the central
fixation marker for the duration of the trial. No task was given apart from
fixating the fixation marker. The face photograph appeared under the fixation
marker for a duration of 300 ms. They were asked not to blink and not to move
their eyes. Figure 1 shows the time course of a single trial.

The experiment was preceded by three training trials, in order to let
participants get used to the procedure. The three faces for the training block
were taken from the same image data set but were not used again for trials of
the same subject. After the training block, the experimental trials followed.
Each trial takes roughly 5 s, split into three phases: (a) A fixation check of
around 2 s, i.e., if the eyes are not centered on the marker a re-calibration
is initiated, (b) an interval of 300 ms of presentation of the face
photograph, and (c) a fixation cross for 3 s. After each trial participants
were encouraged to take a break and to blink. Using a key-press participants
indicated when they were ready for the next trial.

Figure 1: Time course of a trial. The 2 s fixation check ensures participants
fixate the center of the screen. The brief face presentation (0.3 s) is then
followed by a grey screen, during which the main measurement is taken.

Phases a) and c) showed a grey background which covered the same area as the
following image. The shade of grey in the background was designed to be
adjusted to the luminance of the stimulus. The luminance was proportional to,
but not equal to the stimulus luminance, ranging from a relative luminance of
3.5 to 6 % while the images varied in relative luminance from 2.5 to 6.5 %. As
the distribution of images was well controlled by the experimental design,
i.e., each image appeared in each condition, any differences between images
can be accounted for by random effect of image in a linear mixed model (LMM)
Analysis.

Each subject saw the following 100 trials in random order:

  1. a.

10 repetitions of a photograph of their own face (10 trials)

  2. b.

5 repetitions of photographs of 3 selected peers (15 trials)

  3. c.

5 repetitions of photographs of 3 selected strangers (15 trials)

  4. d.

1 photograph each of 30 strangers (30 trials)

  5. e.

1 photograph each of 30 peers (30 trials)

Categories b and c were introduced in order to control for repetition effects
that occur during the repeated viewing of one’s own face. In order to exclude
a pure oddball effect for the Peers-Strangers every participant saw is an
equal number of familiar and unfamiliar faces.

###  2.4 Pupillometry Analyses

In order to prepare the data for analysis we closely followed the recommended
pre-processing pipeline suggested by [76]. Pupil size data was down-sampled to
100 Hz, as a higher resolution is not informative for pupil responses. Missing
data were linearly interpolated (and excluded from the analyses). We converted
the pupil diameter (which is given by the Eye tracker in arbitrary units) to
mm as our base unit and then computed the pupil response of each trial
relative to the baseline. The baseline value for each trial was the average
pupil size during the 50 ms surrounding target onset (as proposed by [76]). We
ensured data quality by evaluating the pupil size during the pre-stimulus
phase. We found no indication that any trials or participants had to be
excluded on the basis of the baseline values. Trials that included a blink
after stimulus onset were excluded from the analyses.

First, as a qualitative analysis, we plotted the pupil response over time,
normalized to the stimulus onset. Second, to statistically support our
findings we use linear mixed models (LMM) with pupil size as the dependent
variable. While analyses that more efficiently take advantage of the the time
series data, such as cluster-based permutation tests, are available and
suggested for pupillometry data [76], they were not feasible for this use
case, as our design had large differences in the number of samples per group
(the Self, First Presentation condition exists only a single time for each
participant, but many times for Strangers). We therefore chose to analyze both
measures using LMMs in 5 time windows: Baseline (-50 – +50 ms), Constriction
(200 – 600 ms), Dilation (600 – 1200 ms), Late Dilation (1200 – 2000 ms), and
Stability (2000 – 3000 ms). The baseline window is included as a sanity check,
to ensure no effects are found before target onset.

Within each window we computed LMMs for the dependent variable "Pupil Size"
using the lme4 package [53] in R [83]. We define the following (custom)
contrasts [88] in our LMM, to test our hypotheses:

  1. 1.

We compare the Self condition to the Strangers and Peers condition, resulting
in a Self–Others comparison.

  2. 2.

We compare the Strangers and Peers conditions (Strangers–Peers).

  3. 3.

We compare the repetitions of individual images up to the 5th repetition,
using a sliding difference contrast, meaning that we compare presentation 1 to
presentation 2, presentation 2 to presentation 3 and so forth.

  4. 4.

We compare the interaction of Repetition and Face, which indicates whether the
difference in the face conditions is different based on Repetition.

Adding Time within the window and the ordinal trial number as covariates
yields the following fixed effects structure for the model formula:

$Pupil\\_Size\sim Face*Repetition+Time+Trial$ (1)

The selection process for the random effect structure, is described in
Appendix A.1. Following [51], we interpret all |t| > 2 as significant fixed
effects.

###  2.5 Microsaccade Rate Analyses

Microsaccades were detected with millisecond accuracy from raw data by
applying a standard microsaccade detection algorithm using a velocity
threshold [60, 58]. MSR can be estimated with the help of a response function
that applies filter kernels to the series of onset times [59, 58]. From the
series of microsaccade onset times $\\{t_{1},t_{2},t_{3},...\\}$, the
microsaccade rate $r(t)$ at time $t$ is estimated via

$r_{\mathrm{a}pprox}(t)=\int_{-\infty}^{+\infty}{\mathrm{d}}\tau
w(\tau)\rho(t-\tau)\;,$ (2)

where the microsaccadic response function $\rho(t)$ [59] is defined as

$\rho(t)=\sum_{i=1}^{N}\delta(t-t_{i})$ (3)

with Dirac’s $\delta$-function $\delta(t)$. We applied a filter kernel known
as a causal window, i.e.,

$w(\tau)=\left[\alpha^{2}\tau\exp(-\alpha\tau)\right]_{+}\;,$ (4)

where parameter $\alpha=1/30$ and $[.]_{+}$ vanishes for negative arguments.
The microsaccade rate $r(t)$ was computed by averaging over microsaccades from
all trials of a participant in a specific experimental condition. The
resulting MSR as a function of time is qualitatively similar the time-courses
of pupil size or event-related potentials, which are all stimulus-locked,
continuous response functions averaged over many experimental trials.

We identified six relevant time windows for our analyses: Baseline ($-300$ to
0 ms), Target Onset (0 to 300 ms), Target Offset (300 to 600 ms), Return (600
to 900 ms), and Stability, which we divided into two 300 ms windows for
consistency (900 to 1200 ms and 1200 to 1500 ms). The 300 ms window size is
consistent with the finding that the MSR decreases in response to a target
onset has a duration of approximately 300 ms before returning to the baseline
[60].

For the statistical analysis, we apply the knowledge that microsaccades are
Poisson-distributed [57] and conduct Poisson rate tests in each time window.
In this analysis we compare the conditions Self–Other, and Strangers–Peers and
report the ratio of the estimated Poisson rate. If the estimated Poisson rate
is the the same in both groups the ratio will be 1, indicating no difference
in MSR.

###  2.6 Relating Microsaccades and Pupil Size

As both MSR and pupil size are reactive to our face conditions, in an
explorative analysis we pose the question: are the two measures related, i.e.,
do trials with many microsaccades also show reduced dilation and vice versa? A
relationship between the two would be an indication of a shared origin. The
microsaccade effect precedes the pupil effect in time, with the drop in MSR
occurring between 50 and 1000 ms after target onset while pupil size effects
peak only after 1000 ms after target onset. In order to analyse the
relationship of microsaccades and pupil size, we begin by selecting the most
diagnostic window for identity using microsaccades (i.e., the Target Offset
Phase, 300-600 ms after target onset) and use it as a predictor for pupil
size. The distribution of microsaccade counts in this subset shows that the
majority of trials (7919 trials) have a count of 0 microsaccades. 1992 trials
had 1 or more saccades. This is a significant difference in sample size and
may influence the results. We use a binary variable which encodes the presence
of one or more microsaccades in the diagnostic window as an ad-hoc predictor
variable of pupil size in an explorative LMM using the formula

$Pupil\\_Size\sim 1+MS\\_in\\_phase*Face+(1|VP).$ (5)

The selection of the random effects structure is detailed in the appendix.

##  3 Results

In the present study we investigate the effect of seeing one’s own face
compared to familiar (Peers) and unfamiliar (Strangers) faces on involuntary
eye measures. A subset of faces, including the Self condition, was shown
repeatedly in order to explore how stimulus repetition influences the effect
and to control for novelty effects. Repetition of unfamiliar faces also
exposes a fourth recognition condition: recognition of a stranger’s face.

###  3.1 Pupil size

In Figure 2B as well as Figure 3, the average pupil reaction for each of the 3
face conditions reveals that there is a qualitative difference between all
three. In the following we report the results of the LMM in each phase (refer
also to Table 1; the coefficients in the Table may be interpreted as the
absolute difference in response in mm).

BA Figure 2: Pupil Size and Microsaccade rate over time. (A) The lines show
the change in pupil size normalized to the size at stimulus onset. The thin
ribbons represent the between-subject standard error. (B) The lines show the
rate of microsaccades after face onset (time zero), calculated for each
participant and then averaged across participants for each condition. The
shading show the between-subject standard error. The area highlighted in gray
shows the time in which the target face was present on the screen.

In the Baseline condition, as expected, the face conditions behave identically
(Figure 3 B). This is confirmed by the LMM, as presented in Table 1. The only
significant effect is a minimal dilation trend over time. The following window
represents the initial constriction, which is a reflexive response to the
stimulus presentation. In the constriction phase, too, no significant
differences for the main effects can be found, with the exception of the
comparison between the 4th and 5th presentation. In all following time
windows, i.e., Dilation, Late Dilation and Stability Phases, we observe
significant differences between the conditions. The effect size is largest in
the Late Dilation phase for the Self–Others comparison and in the Stability
phase for the Peers–Strangers comparison. We conclude that starting at 600 ms
after target onset recognition, and particularly self-recognition modulates
pupil size.

The repetition main effect is coded by a sliding differences contrast, meaning
we compare subsequent presentations. We find a large number of the comparisons
to be significant, showing that the pupil response becomes attenuated over
repetitions. Note that we investigate the difference of each presentation to
the next; a different contrast (e.g., a treatment contrast, comparing each
level to the first repetition) would have likely resulted in more consistently
significant effects. It may also be interesting to point out that while the
effect becomes more muted over the presentations, the last comparison (5th vs.
4th presentation) actually indicates a reversal, perhaps pointing to an
attenuation limit.

The interaction terms show a varying pattern of significance. The interactions
with the Self–Others comparisons is consistently negative or absent, with the
exception of the 2nd-3rd comparison. The interaction term can be interpreted
as the attenuation of the pupil-dilation effect over repetitions. A negative
term indicates that the attenuation in response to the Self condition is
stronger than in the effect in the Peers and Strangers conditions. The same
interpretation can be applied to the Peers–Strangers and Repetition
interactions. Here, a negative term means stronger attenuation in the Peers
condition. Note that in Figure 3 (e.g., in the Stability phase) it is evident
that the second presentation of stranger produces a stronger dilation than the
first (see 2nd–1st : Peers–Strangers interaction), presenting a contrast to
the repetition muting effect found in the Peers and Self conditions. This is
most likely caused by recognition of the face, when it it shown for the second
time, i.e., unfamiliar face recognition. In order to understand the full
extent of these effects please refer to Figure 3.

Table 1: Results of pupil size LMMs. The analysis was conducted using the same
model for 5 time windows: Baseline (-50 – 50 ms), Constriction (200 – 600 ms),
Dilation (600 – 1200 ms), Late Dilation (600 – 1200 ms), and Stability (2000 –
3000 ms). The t values that are marked in bold and red are considered
significant.

| Baseline | Constriction
---|---|---
| Estimate | Std. Error | t value | Estimate | Std. Error | t value
Contrast |  |  |  |  |  |
Intercept | -0.000057 | 0.000081 | -0.6984 | -0.295247 | 0.011274 | -26.1884
Self - Others (Se-O) | -0.000067 | 0.000161 | -0.4160 | 0.003681 | 0.004177 | 0.8814
Peers - Strangers (P:St) | 0.000056 | 0.000100 | 0.5603 | 0.003096 | 0.001625 | 1.9052
1st - 2nd Repetition | -0.000251 | 0.000215 | -1.1668 | -0.005188 | 0.004919 | -1.0548
2nd - 3rd Repetition | 0.000097 | 0.000237 | 0.4079 | 0.001360 | 0.005532 | 0.2459
3rd - 4th Repetition | 0.000198 | 0.000236 | 0.8377 | -0.000015 | 0.001259 | -0.0123
4th - 5th Repetition | 0.000064 | 0.000207 | 0.3106 | 0.003244 | 0.001103 | 2.9403
(Se-O) : (1st-2nd) | -0.000232 | 0.000578 | -0.4023 | -0.018211 | 0.003167 | -5.7507
(P:St) : (1st-2nd) | -0.000068 | 0.000167 | -0.4092 | -0.007136 | 0.000916 | -7.7945
(Se-O) : (2nd-3rd) | 0.000148 | 0.000590 | 0.2512 | 0.020158 | 0.003188 | 6.3232
(P:St) : (2nd-3rd) | 0.000217 | 0.000227 | 0.9581 | 0.010567 | 0.001219 | 8.6705
(Se-O) : (3rd-4th) | 0.000096 | 0.000587 | 0.1643 | -0.001853 | 0.003127 | -0.5928
(P:St) : (3rd-4th) | -0.000204 | 0.000227 | -0.9004 | 0.000771 | 0.001207 | 0.6391
(Se-O) : (4th-5th) | 0.000128 | 0.000473 | 0.2701 | -0.011617 | 0.002514 | -4.6202
(P:St) : (4th-5th) | 0.000169 | 0.000227 | 0.7460 | -0.002896 | 0.001206 | -2.4026
Time | 0.000057 | 0.000001 | 45.8196 | -0.001755 | 0.000002 | -971.5124
Trial | -0.000001 | 0.000002 | -0.5786 | 0.000062 | 0.000008 | 7.3182

| Dilation | Late Dilation | Stability
---|---|---|---
| Estimate | Std. Error | t value | Estimate | Std. Error | t value | Estimate | Std. Error | t value
Contrast |  |  |  |  |  |  |  |  |
Intercept | -0.383362 | 0.015263 | -25.1169 | -0.067456 | 0.009939 | -6.7872 | 0.032116 | 0.009002 | 3.5677
Self - Others (Se-O) | 0.052051 | 0.008528 | 6.1036 | 0.095246 | 0.011187 | 8.5140 | 0.072420 | 0.011285 | 6.4171
Peers - Strangers (P:St) | 0.008944 | 0.002811 | 3.1818 | 0.012648 | 0.003699 | 3.4190 | 0.013186 | 0.004262 | 3.0939
1st - 2nd Repetition | -0.001267 | 0.008189 | -0.1548 | -0.028325 | 0.009303 | -3.0449 | -0.044097 | 0.011878 | -3.7124
2nd - 3rd Repetition | -0.006528 | 0.009706 | -0.6725 | -0.023712 | 0.010525 | -2.2529 | 0.002909 | 0.012222 | 0.2380
3rd - 4th Repetition | -0.008203 | 0.001374 | -5.9713 | -0.014713 | 0.001434 | -10.2633 | -0.022751 | 0.001483 | -15.3451
4th - 5th Repetition | -0.001765 | 0.001204 | -1.4660 | 0.011255 | 0.001257 | 8.9562 | 0.011447 | 0.001300 | 8.8071
(Se-O) : (1st-2nd) | -0.031241 | 0.003458 | -9.0339 | -0.092357 | 0.003609 | -25.5899 | -0.140653 | 0.003733 | -37.6795
(P:St) : (1st-2nd) | -0.010495 | 0.001000 | -10.4924 | -0.027946 | 0.001044 | -26.7681 | -0.025951 | 0.001080 | -24.0306
(Se-O) : (2nd-3rd) | 0.023121 | 0.003481 | 6.6426 | 0.010414 | 0.003633 | 2.8670 | 0.038542 | 0.003757 | 10.2586
(P:St) : (2nd-3rd) | 0.002437 | 0.001331 | 1.8316 | 0.002735 | 0.001389 | 1.9698 | 0.007104 | 0.001436 | 4.9462
(Se-O) : (3rd-4th) | 0.001121 | 0.003412 | 0.3287 | 0.000110 | 0.003561 | 0.0310 | -0.012940 | 0.003683 | -3.5137
(P:St) : (3rd-4th) | 0.004935 | 0.001317 | 3.7478 | 0.008294 | 0.001374 | 6.0358 | 0.011110 | 0.001421 | 7.8176
(Se-O) : (4th-5th) | -0.030656 | 0.002744 | -11.1721 | -0.038631 | 0.002864 | -13.4906 | -0.039241 | 0.002962 | -13.2499
(P:St) : (4th-5th) | -0.003351 | 0.001316 | -2.5474 | -0.003356 | 0.001373 | -2.4441 | -0.009235 | 0.001420 | -6.5038
Time | 0.000754 | 0.000001 | 573.5500 | 0.000201 | 0.000001 | 195.0739 | 0.000068 | 0.000001 | 80.0736
Trial | 0.000028 | 0.000009 | 3.0541 | -0.000633 | 0.000010 | -65.6442 | -0.000889 | 0.000010 | -89.1710

Figure 3: Pupil size over time. The top panel shows the average measured pupil
size for each category of images. The colored background separates the
response signature into 5 time windows: Baseline, Constriction, Dilation, Late
Dilation, and Stability. A main effect of pupil size is evident from the
Dilation phase onwards, where the pupil size becomes significantly different
for Strangers, Peers, and Self. The bottom panels show the details for each
time window. Here, each line represents the nth repetition of an image, with
lighter shades representing later repetitions.

As a further analysis, we were interested in how consistently participants
were exhibiting the effect. Using the terms computed by the LMM, we compare
the effect size of the Self-Others comparison for individual subjects, as
estimated by the random effects in the LMM. The results are plotted in Figure
4. We find that the majority of subjects shows the effect in the expected
direction. Some individuals show only a very minor expression of the effect
and some even mildly reverse the trend, i.e., exhibit less dilation in
response to their own face instead of more.

Figure 4: Effect size of the Self-Others comparison, by participant. Mean
values are computed from the estimated Random Effect Terms. Zero in this plot
means that there was no effect, negative values indicate a reversal, i.e.,
reacting less strongly to Self than to Others. The great majority of
participants show the effect (to the right of the zero line). A small subset
shows a reversed effect.

####  3.1.1 Microsaccades

We calculated the MSR for each subject and condition over time (Figure 2). The
time course of microsaccades is best described as two interacting processes.
First, simple display changes cause an inhibition of microsaccades [60]. As
our paradigm involved two changes in display (image onset and image offset)
both display changes independently cause the MSR to drop. Second,
microsaccades can also be inhibited by cognitive factors such as exogeneous or
endogeneous shifts of attention [58]. In a modeling study, [56] suggested that
the combination of two processes, (i) the modulation of a threshold for
triggering of microsaccades and (ii) a transient reduction of microsaccades,
can explain most of the experimental findings for lower-level display changes
and higher-level cognitive task manipulations.

As suggested by previous research, we find that the MSR responds strongly to
display changes. An inhibition of microsaccades is clearly visible in the
rapid drop in MSR at image onset and image offset. Typically the MSR recovers
quickly, rising to the baseline level. Figure 2 shows that in the Self
condition microsaccades are inhibited for a longer period of time. In all
other conditions, the rate recovers to the baseline level before the second
inhibition occurs at target offset. In the Self condition, the second
inhibition occurs while the first is still active, leading to a further drop
in the MSR and a recovery that is correspondingly longer than in the other
conditions. Thus the average time-course of the MSR is qualitatively different
in the Self condition. Thus, our results are compatible with earlier findings
on microsaccadic inhibition in the oddball paradigm [97].

Table 2 shows the results of the Poisson rate tests. The Estimate should be
interpreted as the ratio of the Poisson rates in the compared conditions,
e.g., the estimate of $1.23$ in the Return Phase when comparing Self and
Others means that the rate in the Others condition is higher by a factor of
1.23 than in the Self Condition. We find significant differences between the
Self and Others conditions, as well as between Peers and Strangers in the
period between 300 ms and 900 ms.

Table 2: Results of the Microsaccade Poisson rate tests. The test compared the Poisson rate of two groups. P-values marked in red are considered significant. The t values marked in bold and red are considered significant. Contrast | Window | Time | Estimate | CI1 | CI2 | p-value
---|---|---|---|---|---|---
Self - Others | Baseline | -300–0 ms | 1.0187 | 0.9072 | 1.1471 | 0.7922
Self - Others | Target Onset | 0–300 ms | 1.1265 | 0.9759 | 1.3063 | 0.1094
Self - Others | Target Offset | 300–600 ms | 1.3925 | 1.2066 | 1.6145 | 0.0000
Self - Others | Return | 600–900 ms | 1.2274 | 1.0942 | 1.3808 | 0.0003
Self - Others | Stability 1 | 900–1200 ms | 0.9889 | 0.8892 | 1.1024 | 0.8279
Self - Others | Stability 2 | 1200–1500 ms | 0.9841 | 0.8860 | 1.0955 | 0.7674
Peers - Strangers | Baseline | -300–0 ms | 1.0235 | 0.9513 | 1.1012 | 0.5299
Peers - Strangers | Target Onset | 0–300 ms | 0.9457 | 0.8671 | 1.0313 | 0.2047
Peers - Strangers | Target Offset | 300–600 ms | 1.1052 | 1.0218 | 1.1956 | 0.0118
Peers - Strangers | Return | 600–900 ms | 1.1111 | 1.0395 | 1.1877 | 0.0018
Peers - Strangers | Stability 1 | 900–1200 ms | 1.0445 | 0.9759 | 1.1180 | 0.2097
Peers - Strangers | Stability 2 | 1200–1500 ms | 1.0327 | 0.9656 | 1.1046 | 0.3496

###  3.2 Relationship of Microsaccades and Pupil Size

We calculated a model using the presence of microsaccades in the most
diagnostic window, as found by the previous analysis, as a predictor for pupil
size. The result of the LMM is presented in Table 3. First, consistent with
the LMMs of pupil size alone, we find a significant effect of both Self–Others
and Peers–Strangers in the Dilation, Late Dilation and Stability time widows.
Second, a predictive effect of microsaccades is present only in the earlier
phases of the pupil response (constriction, dilation).

Table 3: Estimated terms of the LMM relating microsaccades and pupil size.
Occurrence of microsaccades in the most diagnostic window (300–600 ms) is used
as a predictor for pupil size. The predictive effect of microsaccades can only
be found in the early phases of the pupil response. The t values that are
marked in bold and red are considered significant.

Contrast | Baseline | Constriction
---|---|---
| Estimate | Std. Error | t value | Estimate | Std. Error | t value
Intercept | -0.000364 | 0.000097 | -3.7492 | -0.294699 | 0.011040 | -26.6932
Self - Others (Se-O) | -0.000143 | 0.000194 | -0.7366 | 0.007281 | 0.003744 | 1.9444
Peers - Strangers (P:St) | -0.000022 | 0.000065 | -0.3385 | 0.002272 | 0.001248 | 1.8195
MS - No MS | 0.000227 | 0.000186 | 1.2207 | -0.008003 | 0.003669 | -2.1812
(Se-O) : (MS - No MS | 0.000456 | 0.000492 | 0.9266 | -0.007382 | 0.009573 | -0.7711
(Se-O) : (MS - No MS | 0.000136 | 0.000141 | 0.9656 | 0.001129 | 0.002758 | 0.4092

Contrast | Dilation | Late Dilation | Stability
---|---|---|---
| Estimate | Std. Error | t value | Estimate | Std. Error | t value | Estimate | Std. Error | t value
Intercept | -0.378145 | 0.014605 | -25.8907 | -0.058821 | 0.009930 | -5.9236 | 0.034671 | 0.008973 | 3.8638
Self - Others (Se-O) | 0.049080 | 0.006401 | 7.6677 | 0.083602 | 0.007790 | 10.7321 | 0.066590 | 0.008908 | 7.4750
Peers - Strangers (P:St) | 0.013208 | 0.002134 | 6.1889 | 0.027017 | 0.002597 | 10.4025 | 0.025628 | 0.002970 | 8.6292
MS - No MS | -0.014450 | 0.006271 | -2.3042 | -0.010262 | 0.007622 | -1.3464 | -0.004084 | 0.008705 | -0.4691
(Se-O) : (MS - No MS | -0.009118 | 0.016365 | -0.5571 | -0.002875 | 0.019914 | -0.1444 | 0.019757 | 0.022770 | 0.8677
(Se-O) : (MS - No MS | -0.004482 | 0.004715 | -0.9507 | -0.006422 | 0.005737 | -1.1194 | -0.006034 | 0.006559 | -0.9199

##  4 Discussion

In a large-scale eye-tracking experiment, we investigated the behavioral
correlates of face recognition in involuntary eye movements. We recorded pupil
size and fixational eye movements in two groups of participants. Each
participant saw their own face (Self), the familiar faces of members of their
own peer group (Peers), or unfamiliar faces (Strangers). Selected faces,
including the Self condition, were repeated to investigate familiarization
effects. This study design provides a well-controlled data set, where, across
participants, each individual face appeared in each condition (Peers,
Strangers, Self). We observe that face recognition, particularly self-
recognition, modulates the response signatures of both pupil dilation and MSR.

###  4.1 Recognition of Familiar Faces

First, we compared the recognition of familiar faces to completely unfamiliar
faces, i.e. Peers vs. Strangers. Recognizing a familiar faces is related to a
stronger and longer-lasting inhibition of microsaccades, starting around 200
ms after target onset. Recognition effects on MSR were first documented by
[86, 87], however a study by [55] did not reproduce this result. We also
observed increased pupil dilation in response to familiar faces starting
around 600 ms after target onset. Our findings regarding pupil dilation are in
agreement with previous work [55]. In accordance with the literature, we
propose that early effects (microsaccade response, early stages of pupil
response), are related purely to recognition and saliency [75, 71]. Early
recognition responses in the IT cortex occur 50-100 ms after target onset [95,
80] and the data show effects on microsaccade inhibition less than 100 ms
later, suggesting that feedback from the fusiform face area (FFA) may have a
direct influence on processing in the superior colliculus (SC). The mid and
late stages of the pupil response are likely related more to arousal and
emotional responses [75].

###  4.2 Self-Recognition

Similar effects as in the Peers–Strangers comparison are evident in the
Self–Others comparison, albeit significantly larger in scale. We hypothesized
that the faces of strangers capture little attention and are grouped together,
while one’s own face is qualitatively different, even to familiar faces. We
find a strong individual variability in the effect, which, we speculate, may
be related to differences in the emotional response related to self-image. It
is important to note that the Self–Others comparison is not perfectly balanced
regarding stimulus frequency. While many different Peers and Strangers were
shown, there is only ever one Self face, which is repeated. We controlled for
repetition by also repeating certain Peers’ and Strangers’ faces. However, the
relative frequency of Self images was still lower than of the Others images.

We find the largest effects for the first presentation of Self, and an
attenuation over repetitions. Thus, it is likely that genuine self-recognition
effects are further enhanced by a stimulus-probability effect, similar to the
Oddball paradigm.

###  4.3 Repetition Effects and Recognition of Unfamiliar Faces

Repetitions of the same image led to reduced pupil dilation. We observed a
strong attenuation of the effect over repetitions, such that the 10th Self-
presentation is comparable to the Strangers response (see Figure 3). The
strongest repetition effects are visible in the Late Dilation and Stability
Phases. Unlike the neurophysiological novelty response which occurs 300 ms
after target onset (compare: P300 component), this novelty effect on pupil
constriction begins much later, around 1,000 ms.

On its own, the attenuation over repetitions seems discouraging for real-world
applications, where pictures may be presented repeatedly. However, this is not
true of the early response: in the dilation phase, the faster and stronger
dilation in the Self condition is still evidently different from the average
Strangers condition. Whether this effect is short- or long-term, i.e., whether
it would persist over multiple sessions remains to be investigated.

The first response to seeing one’s own face is comparatively much larger than
in any other condition, and is much reduced in subsequent repetitions (see
Figure 3, or "(Se-O) : (1st-2nd)" in Stability phase in Table 1). Moreover,
when a stranger’s face is repeated, i.e., during the first recognition of a
now-familiarized stranger, the dilation is stronger than during the very first
viewing (see Figure 3, or "(F:St) : (1st-2nd)" in Late Dilation phase in Table
1). This outlines the effect of the pure recognition of a face, in the absence
of any emotional reaction. Further recognitions of the stranger reduce the
effect in accordance with the repetition effect.

###  4.4 The Relationship of Pupil Size and Microsaccade Rate

We find that microsaccadic inhibition and the pupil dilation are only weakly
correlated. Trials that have particularly few microsaccades do not serve well
as a predictor for strong dilation. This observation is surprising, as we
expected both measures to share a common cause. Our results suggest that MSR
and pupil dilation are controlled independently and via separate pathways,
although they commonly react to cognitive and emotional factors. The lack of
correlation could confer a positive effect on models using a conjunction of
both features for its prediction.

###  4.5 Potential applications

The question of facial familiarity is crucial in security applications.
Concealed information testing (CIT) serves to identify individuals with
knowledge of a crime or perpetrator without relying on verbal statements. The
robust effects presented here serve as a good basis for applications in
continuous authentication procedures or identity validation. Initial
explorations show the feasibility of using these measures as traits for
biometric classification [61]. Possible real-world scenarios include the
detection of face presentation attacks (e.g., video deepfakes) in online video
conferences. As we found an attentuation of the effect over image repetition,
a remaining important consideration is whether the observed differences can be
discerned at the level of individual trials and sustained over repetitions.
More application-oriented studies will be required to validate the
effectiveness of involuntary eye movements for practical use cases.

## Acknowledgements

The authors acknowledge support from Bundesdruckerei GmbH, Berlin, Germany.

## References

  * [1] R.. Baayen, D.. Davidson and D.. Bates “Mixed-effects Modeling with Crossed Random Effects for Subjects and Items” In _Journal of Memory and Language_ 59.4 Elsevier BV, 2008, pp. 390–412 DOI: 10.1016/j.jml.2007.12.005
  * [2] A. Bahill, Michael R. Clark and Lawrence Stark “The Main Sequence, a Tool for Studying Human Eye Movements” In _Mathematical Biosciences_ 24.3 Elsevier, 1975, pp. 191–204 DOI: 10.1016/0025-5564(75)90075-9
  * [3] Douglas Bates, Martin Mächler, Ben Bolker and Steve Walker “Fitting Linear Mixed-effects Models Using lme4” In _Journal of Statistical Software_ 67.1, 2015, pp. 1–48 DOI: 10.18637/jss.v067.i01
  * [4] Margaret M. Bradley, Laura Miccoli, Miguel A. Escrig and Peter J. Lang “The Pupil As a Measure of Emotional Arousal and Autonomic Activation” In _Psychophysiology_ 45.4 Wiley, 2008, pp. 602–607 DOI: 10.1111/j.1469-8986.2008.00654.x
  * [5] Ivory Yining Chen et al. “Concealed information detection in rapid serial visual presentation with oculomotor measures” Center for Open Science, 2023 DOI: 10.31234/osf.io/rtyu6
  * [6] R. Engbert “Computational Modeling of Collicular Integration of Perceptual Responses and Attention in Microsaccades” In _Journal of Neuroscience_ 32.23 Society for Neuroscience, 2012, pp. 8035–8039 DOI: 10.1523/jneurosci.0808-12.2012
  * [7] R. Engbert and K. Mergenthaler “Microsaccades Are Triggered by Low Retinal Image Slip” In _Proceedings of the National Academy of Sciences_ 103.18 Proceedings of the National Academy of Sciences, 2006, pp. 7192–7197 DOI: 10.1073/pnas.0509557103
  * [8] Ralf Engbert “Microsaccades: A Microcosm for Research on Oculomotor Control, Attention, and Visual Perception” In _Visual Perception - Fundamentals of Vision: Low and Mid-level Processes in Perception_ 154 Elsevier, 2006, pp. 177–192 DOI: 10.1016/s0079-6123(06)54009-9
  * [9] Ralf Engbert “Dynamical Models in Neurocognitive Psychology” Springer Nature Publishing, 2021 DOI: 10.1007/978-3-030-67299-7
  * [10] Ralf Engbert and Reinhold Kliegl “Microsaccades Uncover the Orientation of Covert Attention” In _Vision Research_ 43.9 Elsevier BV, 2003, pp. 1035–1045 DOI: 10.1016/s0042-6989(03)00084-1
  * [11] Hendrik Graupner, Lisa Schwetlick, Ralf Engbert and Christoph Meinel “Unconventional Biometrics: Exploring the Feasibility of a Cognitive Trait based on Visual Self-Recognition” In _IEEE International Joint Conference on Biometrics 2023_ , 2023, pp. 326–335
  * [12] John M. Henderson “Human Gaze Control during Real-world Scene Perception” In _Trends in Cognitive Sciences_ 7.11 Elsevier, 2003, pp. 498–504 DOI: 10.1016/j.tics.2003.09.006
  * [13] Eckhard H. Hess and James M. Polt “Pupil Size As Related to Interest Value of Visual Stimuli” In _Science_ 132.3423 American Association for the Advancement of Science (AAAS), 1960, pp. 349–350 DOI: 10.1126/science.132.3423.349
  * [14] Mary Margaret Johnson and J.Peter Rosenfeld “Oddball-evoked P300-based method of deception detection in the laboratory II: Utilization of non-selective activation of relevant knowledge” In _International Journal of Psychophysiology_ 12.3 Elsevier BV, 1992, pp. 289–306 DOI: 10.1016/0167-8760(92)90067-l
  * [15] Robert A. Johnston and Andrew J. Edmonds “Familiar and unfamiliar face recognition: A review” In _Memory_ 17.5 Informa UK Limited, 2009, pp. 577–596 DOI: 10.1080/09658210902976969
  * [16] Siri-Maria Kamp and Emanuel Donchin “Erp and Pupil Responses to Deviance in an Oddball Paradigm” In _Psychophysiology_ 52.4, 2015, pp. 460–471 DOI: https://doi.org/10.1111/psyp.12378
  * [17] Nancy Kanwisher, Josh McDermott and Marvin M. Chun “The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception” In _The Journal of Neuroscience_ 17.11 Society for Neuroscience, 1997, pp. 4302–4311 DOI: 10.1523/jneurosci.17-11-04302.1997
  * [18] Valerie L. Kinner et al. “What Our Eyes Tell Us about Feelings: Tracking Pupillary Responses during Emotion Regulation Processes” In _Psychophysiology_ 54.4 Wiley, 2017, pp. 508–518 DOI: 10.1111/psyp.12816
  * [19] Albert Kok “On the Utility of P3 Amplitude As a Measure of Processing Capacity” In _Psychophysiology_ 38.3 Wiley, 2001, pp. 557–577 DOI: 10.1017/s0048577201990559
  * [20] Bruno Laeng, Sylvain Sirois and Gustaf Gredebäck “Pupillometry” In _Perspectives on Psychological Science_ 7.1 SAGE Publications, 2012, pp. 18–27 DOI: 10.1177/1745691611427305
  * [21] Rylan S. Larsen and Jack Waters “Neuromodulatory Correlates of Pupil Dilation” In _Frontiers in Neural Circuits_ 12 Frontiers Media SA, 2018 DOI: 10.3389/fncir.2018.00021
  * [22] Jochen Laubrock, Anke Cajar and Ralf Engbert “Control of Fixation Duration during Scene Viewing by Interaction of Foveal and Peripheral Processing” In _Journal of Vision_ 13(12):11 The Association for Research in VisionOphthalmology, 2013 DOI: 10.1167/13.12.11
  * [23] Jacob G. Martin, Charles E. Davis, Maximilian Riesenhuber and Simon J. Thorpe “Microsaccades during high speed continuous visual search” In _Journal of Eye Movement Research_ 13.5 University of Bern, 2020 DOI: 10.16910/jemr.13.5.4
  * [24] Susana Martinez-Conde, Stephen L. Macknik and David H. Hubel “The Role of Fixational Eye Movements in Visual Perception” In _Nature Reviews Neuroscience_ 5.3 Springer ScienceBusiness Media LLC, 2004, pp. 229–240 DOI: 10.1038/nrn1348
  * [25] Sebastiaan Mathôt “Pupillometry: Psychology, Physiology, and Function” In _Journal of Cognition_ 1.1 Ubiquity Press, Ltd., 2018, pp. 16 DOI: 10.5334/joc.18
  * [26] Sebastiaan Mathôt and Ana Vilotijević “Methods in Cognitive Pupillometry: Design, Preprocessing, and Statistical Analysis” In _Behavior Research Methods_ Springer ScienceBusiness Media LLC, 2022 DOI: 10.3758/s13428-022-01957-7
  * [27] Elinor McKone and Nancy Kanwisher “From Monkey Brain to Human Brain” The MIT Press, 2005, pp. 339–356 DOI: 10.7551/mitpress/3136.003.0024
  * [28] Konstantin Mergenthaler and Ralf Engbert “Microsaccades Are Different from Saccades in Scene Perception” In _Experimental Brain Research_ 203.4 Springer ScienceBusiness Media LLC, 2010, pp. 753–757 DOI: 10.1007/s00221-010-2272-9
  * [29] Peter R. Murphy et al. “Pupil Diameter Covaries with BOLD Activity in Human Locus Coeruleus” In _Human Brain Mapping_ 35.8 Wiley, 2014, pp. 4140–4154 DOI: 10.1002/hbm.22466
  * [30] D.I. Perrett, E.T. Rolls and W. Caan “Visual neurones responsive to faces in the monkey temporal cortex” In _Experimental Brain Research_ 47.3 Springer, 1982, pp. 329–342 DOI: 10.1007/bf00239352
  * [31] Terence W. Picton “The P300 Wave of the Human Event-related Potential” In _Journal of Clinical Neurophysiology_ 9.4 LWW, 1992, pp. 456–479 DOI: 10.1097/00004691-199210000-00002
  * [32] John Polich “Updating P300: An Integrative Theory of P3a and P3b” In _Clinical Neurophysiology_ 118.10 Elsevier BV, 2007, pp. 2128–2148 DOI: 10.1016/j.clinph.2007.04.019
  * [33] R Core Team “R: A Language and Environment for Statistical Computing”, 2019 R Foundation for Statistical Computing URL: https://www.R-project.org/
  * [34] Martin Rolfs “Microsaccades: Small Steps on a Long Way” In _Vision Research_ 49.20 Elsevier BV, 2009, pp. 2415–2441 DOI: 10.1016/j.visres.2009.08.010
  * [35] Edmund T. Rolls, Hugo D. Critchley, Andrew S. Browning and Kazuo Inoue “Face-selective and auditory neurons in the primate orbitofrontal cortex” In _Experimental Brain Research_ 170.1 Springer ScienceBusiness Media LLC, 2005, pp. 74–87 DOI: 10.1007/s00221-005-0191-y
  * [36] Gal Rosenzweig and Yoram S. Bonneh “Familiarity revealed by involuntary eye movements on the fringe of awareness” In _Scientific Reports_ 9.1 Springer ScienceBusiness Media LLC, 2019 DOI: 10.1038/s41598-019-39889-6
  * [37] Gal Rosenzweig and Yoram S. Bonneh “Concealed information revealed by involuntary eye movements on the fringe of awareness in a mock terror experiment” In _Scientific Reports_ 10.1 Springer ScienceBusiness Media LLC, 2020 DOI: 10.1038/s41598-020-71487-9
  * [38] Daniel J. Schad, Shravan Vasishth, Sven Hohenstein and Reinhold Kliegl “How to Capitalize on a Priori Contrasts in Linear (mixed) Models: A Tutorial” In _Journal of Memory and Language_ 110 Elsevier BV, 2020, pp. 104038 DOI: 10.1016/j.jml.2019.104038
  * [39] Jeffrey D. Schall “Neural Basis of Saccade Target Selection” In _Reviews in the Neurosciences_ 6.1 Walter de Gruyter GmbH, 1995 DOI: 10.1515/revneuro.1995.6.1.63
  * [40] S.. Shapiro and M.. Wilk “An analysis of variance test for normality (complete samples)” In _Biometrika_ 52.3–4 Oxford University Press (OUP), 1965, pp. 591–611 DOI: 10.1093/biomet/52.3-4.591
  * [41] Sylvain Sirois and Julie Brisson “Pupillometry” In _WIREs Cognitive Science_ 5.6 Wiley, 2014, pp. 679–692 DOI: 10.1002/wcs.1323
  * [42] David L. Sparks “The Brainstem Control of Saccadic Eye Movements” In _Nature Reviews Neuroscience_ 3.12 Nature Publishing Group UK London, 2002, pp. 952–964 DOI: https://doi.org/10.1038/nrn986
  * [43] Nancy K. Squires, Kenneth C. Squires and Steven A. Hillyard “Two Varieties of Long-latency Positive Waves Evoked by Unpredictable Auditory Stimuli in Man” In _Electroencephalography and Clinical Neurophysiology_ 38.4 Elsevier BV, 1975, pp. 387–401 DOI: 10.1016/0013-4694(75)90263-1
  * [44] Christoph Strauch, Ina Koniakowsky and Anke Huckauf “Decision Making and Oddball Effects on Pupil Size: Evidence for a Sequential Process” In _Journal of Cognition_ 3.1 Ubiquity Press, Ltd., 2020, pp. 7 DOI: 10.5334/joc.96
  * [45] Yasuko Sugase, Shigeru Yamane, Shoogo Ueno and Kenji Kawano “Global and fine information coded by single neurons in the temporal visual cortex” In _Nature_ 400.6747 Springer ScienceBusiness Media LLC, 1999, pp. 869–873 DOI: 10.1038/23703
  * [46] Samuel Sutton, Margery Braren, Joseph Zubin and E.. John “Evoked-potential Correlates of Stimulus Uncertainty” In _Science_ 150.3700 American Association for the Advancement of Science (AAAS), 1965, pp. 1187–1188 DOI: 10.1126/science.150.3700.1187
  * [47] Matteo Valsecchi, Elena Betta and Massimo Turatto “Visual Oddballs Induce Prolonged Microsaccadic Inhibition” In _Experimental Brain Research_ 177.2 Springer ScienceBusiness Media LLC, 2006, pp. 196–208 DOI: 10.1007/s00221-006-0665-6
  * [48] Matteo Valsecchi et al. “Microsaccadic inhibition and P300 enhancement in a visual oddball task” In _Psychophysiology_ 46.3 Wiley, 2009, pp. 635–644 DOI: 10.1111/j.1469-8986.2009.00791.x
  * [49] Barbara J. Winterson and Han Collewijn “Microsaccades during Finely Guided Visuomotor Tasks” In _Vision Research_ 16.12 Elsevier BV, 1976, pp. 1387–1390 DOI: 10.1016/0042-6989(76)90156-5
  * [50] Robert K. Yin “Looking at upside-down faces.” In _Journal of Experimental Psychology_ 81.1 American Psychological Association (APA), 1969, pp. 141–145 DOI: 10.1037/h0027474

## References

  * [51] R.. Baayen, D.. Davidson and D.. Bates “Mixed-effects Modeling with Crossed Random Effects for Subjects and Items” In _Journal of Memory and Language_ 59.4 Elsevier BV, 2008, pp. 390–412 DOI: 10.1016/j.jml.2007.12.005
  * [52] A. Bahill, Michael R. Clark and Lawrence Stark “The Main Sequence, a Tool for Studying Human Eye Movements” In _Mathematical Biosciences_ 24.3 Elsevier, 1975, pp. 191–204 DOI: 10.1016/0025-5564(75)90075-9
  * [53] Douglas Bates, Martin Mächler, Ben Bolker and Steve Walker “Fitting Linear Mixed-effects Models Using lme4” In _Journal of Statistical Software_ 67.1, 2015, pp. 1–48 DOI: 10.18637/jss.v067.i01
  * [54] Margaret M. Bradley, Laura Miccoli, Miguel A. Escrig and Peter J. Lang “The Pupil As a Measure of Emotional Arousal and Autonomic Activation” In _Psychophysiology_ 45.4 Wiley, 2008, pp. 602–607 DOI: 10.1111/j.1469-8986.2008.00654.x
  * [55] Ivory Yining Chen et al. “Concealed information detection in rapid serial visual presentation with oculomotor measures” Center for Open Science, 2023 DOI: 10.31234/osf.io/rtyu6
  * [56] R. Engbert “Computational Modeling of Collicular Integration of Perceptual Responses and Attention in Microsaccades” In _Journal of Neuroscience_ 32.23 Society for Neuroscience, 2012, pp. 8035–8039 DOI: 10.1523/jneurosci.0808-12.2012
  * [57] R. Engbert and K. Mergenthaler “Microsaccades Are Triggered by Low Retinal Image Slip” In _Proceedings of the National Academy of Sciences_ 103.18 Proceedings of the National Academy of Sciences, 2006, pp. 7192–7197 DOI: 10.1073/pnas.0509557103
  * [58] Ralf Engbert “Microsaccades: A Microcosm for Research on Oculomotor Control, Attention, and Visual Perception” In _Visual Perception - Fundamentals of Vision: Low and Mid-level Processes in Perception_ 154 Elsevier, 2006, pp. 177–192 DOI: 10.1016/s0079-6123(06)54009-9
  * [59] Ralf Engbert “Dynamical Models in Neurocognitive Psychology” Springer Nature Publishing, 2021 DOI: 10.1007/978-3-030-67299-7
  * [60] Ralf Engbert and Reinhold Kliegl “Microsaccades Uncover the Orientation of Covert Attention” In _Vision Research_ 43.9 Elsevier BV, 2003, pp. 1035–1045 DOI: 10.1016/s0042-6989(03)00084-1
  * [61] Hendrik Graupner, Lisa Schwetlick, Ralf Engbert and Christoph Meinel “Unconventional Biometrics: Exploring the Feasibility of a Cognitive Trait based on Visual Self-Recognition” In _IEEE International Joint Conference on Biometrics 2023_ , 2023, pp. 326–335
  * [62] John M. Henderson “Human Gaze Control during Real-world Scene Perception” In _Trends in Cognitive Sciences_ 7.11 Elsevier, 2003, pp. 498–504 DOI: 10.1016/j.tics.2003.09.006
  * [63] Eckhard H. Hess and James M. Polt “Pupil Size As Related to Interest Value of Visual Stimuli” In _Science_ 132.3423 American Association for the Advancement of Science (AAAS), 1960, pp. 349–350 DOI: 10.1126/science.132.3423.349
  * [64] Mary Margaret Johnson and J.Peter Rosenfeld “Oddball-evoked P300-based method of deception detection in the laboratory II: Utilization of non-selective activation of relevant knowledge” In _International Journal of Psychophysiology_ 12.3 Elsevier BV, 1992, pp. 289–306 DOI: 10.1016/0167-8760(92)90067-l
  * [65] Robert A. Johnston and Andrew J. Edmonds “Familiar and unfamiliar face recognition: A review” In _Memory_ 17.5 Informa UK Limited, 2009, pp. 577–596 DOI: 10.1080/09658210902976969
  * [66] Siri-Maria Kamp and Emanuel Donchin “Erp and Pupil Responses to Deviance in an Oddball Paradigm” In _Psychophysiology_ 52.4, 2015, pp. 460–471 DOI: https://doi.org/10.1111/psyp.12378
  * [67] Nancy Kanwisher, Josh McDermott and Marvin M. Chun “The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception” In _The Journal of Neuroscience_ 17.11 Society for Neuroscience, 1997, pp. 4302–4311 DOI: 10.1523/jneurosci.17-11-04302.1997
  * [68] Valerie L. Kinner et al. “What Our Eyes Tell Us about Feelings: Tracking Pupillary Responses during Emotion Regulation Processes” In _Psychophysiology_ 54.4 Wiley, 2017, pp. 508–518 DOI: 10.1111/psyp.12816
  * [69] Albert Kok “On the Utility of P3 Amplitude As a Measure of Processing Capacity” In _Psychophysiology_ 38.3 Wiley, 2001, pp. 557–577 DOI: 10.1017/s0048577201990559
  * [70] Bruno Laeng, Sylvain Sirois and Gustaf Gredebäck “Pupillometry” In _Perspectives on Psychological Science_ 7.1 SAGE Publications, 2012, pp. 18–27 DOI: 10.1177/1745691611427305
  * [71] Rylan S. Larsen and Jack Waters “Neuromodulatory Correlates of Pupil Dilation” In _Frontiers in Neural Circuits_ 12 Frontiers Media SA, 2018 DOI: 10.3389/fncir.2018.00021
  * [72] Jochen Laubrock, Anke Cajar and Ralf Engbert “Control of Fixation Duration during Scene Viewing by Interaction of Foveal and Peripheral Processing” In _Journal of Vision_ 13(12):11 The Association for Research in VisionOphthalmology, 2013 DOI: 10.1167/13.12.11
  * [73] Jacob G. Martin, Charles E. Davis, Maximilian Riesenhuber and Simon J. Thorpe “Microsaccades during high speed continuous visual search” In _Journal of Eye Movement Research_ 13.5 University of Bern, 2020 DOI: 10.16910/jemr.13.5.4
  * [74] Susana Martinez-Conde, Stephen L. Macknik and David H. Hubel “The Role of Fixational Eye Movements in Visual Perception” In _Nature Reviews Neuroscience_ 5.3 Springer ScienceBusiness Media LLC, 2004, pp. 229–240 DOI: 10.1038/nrn1348
  * [75] Sebastiaan Mathôt “Pupillometry: Psychology, Physiology, and Function” In _Journal of Cognition_ 1.1 Ubiquity Press, Ltd., 2018, pp. 16 DOI: 10.5334/joc.18
  * [76] Sebastiaan Mathôt and Ana Vilotijević “Methods in Cognitive Pupillometry: Design, Preprocessing, and Statistical Analysis” In _Behavior Research Methods_ Springer ScienceBusiness Media LLC, 2022 DOI: 10.3758/s13428-022-01957-7
  * [77] Elinor McKone and Nancy Kanwisher “From Monkey Brain to Human Brain” The MIT Press, 2005, pp. 339–356 DOI: 10.7551/mitpress/3136.003.0024
  * [78] Konstantin Mergenthaler and Ralf Engbert “Microsaccades Are Different from Saccades in Scene Perception” In _Experimental Brain Research_ 203.4 Springer ScienceBusiness Media LLC, 2010, pp. 753–757 DOI: 10.1007/s00221-010-2272-9
  * [79] Peter R. Murphy et al. “Pupil Diameter Covaries with BOLD Activity in Human Locus Coeruleus” In _Human Brain Mapping_ 35.8 Wiley, 2014, pp. 4140–4154 DOI: 10.1002/hbm.22466
  * [80] D.I. Perrett, E.T. Rolls and W. Caan “Visual neurones responsive to faces in the monkey temporal cortex” In _Experimental Brain Research_ 47.3 Springer, 1982, pp. 329–342 DOI: 10.1007/bf00239352
  * [81] Terence W. Picton “The P300 Wave of the Human Event-related Potential” In _Journal of Clinical Neurophysiology_ 9.4 LWW, 1992, pp. 456–479 DOI: 10.1097/00004691-199210000-00002
  * [82] John Polich “Updating P300: An Integrative Theory of P3a and P3b” In _Clinical Neurophysiology_ 118.10 Elsevier BV, 2007, pp. 2128–2148 DOI: 10.1016/j.clinph.2007.04.019
  * [83] R Core Team “R: A Language and Environment for Statistical Computing”, 2019 R Foundation for Statistical Computing URL: https://www.R-project.org/
  * [84] Martin Rolfs “Microsaccades: Small Steps on a Long Way” In _Vision Research_ 49.20 Elsevier BV, 2009, pp. 2415–2441 DOI: 10.1016/j.visres.2009.08.010
  * [85] Edmund T. Rolls, Hugo D. Critchley, Andrew S. Browning and Kazuo Inoue “Face-selective and auditory neurons in the primate orbitofrontal cortex” In _Experimental Brain Research_ 170.1 Springer ScienceBusiness Media LLC, 2005, pp. 74–87 DOI: 10.1007/s00221-005-0191-y
  * [86] Gal Rosenzweig and Yoram S. Bonneh “Familiarity revealed by involuntary eye movements on the fringe of awareness” In _Scientific Reports_ 9.1 Springer ScienceBusiness Media LLC, 2019 DOI: 10.1038/s41598-019-39889-6
  * [87] Gal Rosenzweig and Yoram S. Bonneh “Concealed information revealed by involuntary eye movements on the fringe of awareness in a mock terror experiment” In _Scientific Reports_ 10.1 Springer ScienceBusiness Media LLC, 2020 DOI: 10.1038/s41598-020-71487-9
  * [88] Daniel J. Schad, Shravan Vasishth, Sven Hohenstein and Reinhold Kliegl “How to Capitalize on a Priori Contrasts in Linear (mixed) Models: A Tutorial” In _Journal of Memory and Language_ 110 Elsevier BV, 2020, pp. 104038 DOI: 10.1016/j.jml.2019.104038
  * [89] Jeffrey D. Schall “Neural Basis of Saccade Target Selection” In _Reviews in the Neurosciences_ 6.1 Walter de Gruyter GmbH, 1995 DOI: 10.1515/revneuro.1995.6.1.63
  * [90] S.. Shapiro and M.. Wilk “An analysis of variance test for normality (complete samples)” In _Biometrika_ 52.3–4 Oxford University Press (OUP), 1965, pp. 591–611 DOI: 10.1093/biomet/52.3-4.591
  * [91] Sylvain Sirois and Julie Brisson “Pupillometry” In _WIREs Cognitive Science_ 5.6 Wiley, 2014, pp. 679–692 DOI: 10.1002/wcs.1323
  * [92] David L. Sparks “The Brainstem Control of Saccadic Eye Movements” In _Nature Reviews Neuroscience_ 3.12 Nature Publishing Group UK London, 2002, pp. 952–964 DOI: https://doi.org/10.1038/nrn986
  * [93] Nancy K. Squires, Kenneth C. Squires and Steven A. Hillyard “Two Varieties of Long-latency Positive Waves Evoked by Unpredictable Auditory Stimuli in Man” In _Electroencephalography and Clinical Neurophysiology_ 38.4 Elsevier BV, 1975, pp. 387–401 DOI: 10.1016/0013-4694(75)90263-1
  * [94] Christoph Strauch, Ina Koniakowsky and Anke Huckauf “Decision Making and Oddball Effects on Pupil Size: Evidence for a Sequential Process” In _Journal of Cognition_ 3.1 Ubiquity Press, Ltd., 2020, pp. 7 DOI: 10.5334/joc.96
  * [95] Yasuko Sugase, Shigeru Yamane, Shoogo Ueno and Kenji Kawano “Global and fine information coded by single neurons in the temporal visual cortex” In _Nature_ 400.6747 Springer ScienceBusiness Media LLC, 1999, pp. 869–873 DOI: 10.1038/23703
  * [96] Samuel Sutton, Margery Braren, Joseph Zubin and E.. John “Evoked-potential Correlates of Stimulus Uncertainty” In _Science_ 150.3700 American Association for the Advancement of Science (AAAS), 1965, pp. 1187–1188 DOI: 10.1126/science.150.3700.1187
  * [97] Matteo Valsecchi, Elena Betta and Massimo Turatto “Visual Oddballs Induce Prolonged Microsaccadic Inhibition” In _Experimental Brain Research_ 177.2 Springer ScienceBusiness Media LLC, 2006, pp. 196–208 DOI: 10.1007/s00221-006-0665-6
  * [98] Matteo Valsecchi et al. “Microsaccadic inhibition and P300 enhancement in a visual oddball task” In _Psychophysiology_ 46.3 Wiley, 2009, pp. 635–644 DOI: 10.1111/j.1469-8986.2009.00791.x
  * [99] Barbara J. Winterson and Han Collewijn “Microsaccades during Finely Guided Visuomotor Tasks” In _Vision Research_ 16.12 Elsevier BV, 1976, pp. 1387–1390 DOI: 10.1016/0042-6989(76)90156-5
  * [100] Robert K. Yin “Looking at upside-down faces.” In _Journal of Experimental Psychology_ 81.1 American Psychological Association (APA), 1969, pp. 141–145 DOI: 10.1037/h0027474

##  Appendix A Supplemental Online Materials

###  A.1 LMM Random Effects Structure

We used linear mixed models (LMMs) to analyze the pupillometry data in 5 time
windows. In all time windows we employed the same fixed and random effect
structure. Random effects were estimated for both participants and images, but
mainly with the aim of correcting for their influence. We initially formulated
a hypothesis-driven random effect structure, including random intercepts for
both participants and images, and random slopes for each the Faces
comparisons. We included the Repetition comparisons as random slopes for each
subject (but not each image, as not all images were necessarily seen in each
repetition). We also excluded random interaction slopes. Model reduction was
then performed iteratively until convergence was achieved without issues.
Specifically, correlation terms were first removed, followed by the least
varying random effect terms. In cases where two converging models with
different random effect terms were obtained, we used Bayesian-Information-
Criterion (BIC) to select the best model. To ensure that none of the models
were degenerate, a principal component analysis was performed on the random
effect terms [53]. The final model structure is as follows,

$+(1+Face+Rep2+Rep3|VP)+\\\ (1+Face(Se-O)|Img).$ (6)

We performed the same procedure for the random effects structure of the
explorative LMM that explores the relationship between microsaccade occurrence
and pupil size. Here we included random intercepts by subject. As the
predictor variable was ad-hoc, group sizes were uneven and a random effect of
image was excluded. After the model reduction procedure we arrived at a model
which defines only a random intercept for each subject.

$+(1|VP).$ (7)

###  A.2 Controlling for Knowing

After the experiment, participants were shown all faces one more time. They
were asked to indicate by pressing one of 3 buttons on a _ViewPixx_ button box
whether they know the person in the picture, did not know them, or know them
very well. With this information we can exclude trials with subjects who
coincidentally did know each other across the schools. Figure A1 shows a
visualization of the social network between the participants. This data also
allows potential exploratory analyses using the distinction of close peers
versus acquaintances.

Figure A1: Social Network of participants. The two clusters represent the two
school classes. The right panel shows the social participants that knew each
other. The left panel shows the connections after removing trials that
violated the expectation that students only know people from their own class.

###  A.3 Controlling for Self Image

For the purposes of an exploratory analysis, we asked three questions that
were to be answered on a 5 point scale, i.e.,

  1. 1.

How did you like the photographs in general? _original: Wie fanden Sie die
Fotos allgemein?_

  2. 2.

How did you like the photograph of yourself? _original: Wie fanden Sie das
Foto von sich selbst?_

  3. 3.

How happy are you with your own appearance in general? _original: Wie
glücklich sind Sie mit Ihrem eigenen Erscheinungsbild allgemein?_

Questions 1 and 2 were to be answered with

  1. 1.

Liked very much _original: Sehr gelungen_

  2. 2.

Liked _original: Gelungen_

  3. 3.

Neutral _original: Durchschnittlich_

  4. 4.

Did not like much _original: Eher nicht gelungen_

  5. 5.

Did not like _original: Nicht gelungen_

and question 3 with

  1. 1.

Very happy _original: Sehr glücklich_

  2. 2.

Happy _original: Glücklich_

  3. 3.

Neutral _original: neutral_

  4. 4.

Rather unhappy _original: Eher unglücklich_

  5. 5.

Unhappy _original: unglücklich_