# Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object
Discovery with Transformers

Zhiwei Lin∗ Zengyu Yang∗† Yongtao Wang${}^{\textrm{\Letter}}$ Wangxuan
Institute of Computer Technology, Peking University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Unsupervised object discovery (UOD) has recently shown encouraging progress
with the adoption of pre-trained Transformer features. However, current
methods based on Transformers mainly focus on designing the localization head
(_e.g_., seed selection-expansion and normalized cut) and overlook the
importance of improving Transformer features. In this work, we handle UOD task
from the perspective of feature enhancement and propose FOReground guidance
and MUlti-LAyer feature fusion for unsupervised object discovery, dubbed
FORMULA. Firstly, we present a foreground guidance strategy with an off-the-
shelf UOD detector to highlight the foreground regions on the feature maps and
then refine object locations in an iterative fashion. Moreover, to solve the
scale variation issues in object detection, we design a multi-layer feature
fusion module that aggregates features responding to objects at different
scales. The experiments on VOC07, VOC12, and COCO_20k show that the proposed
FORMULA achieves new state-of-the-art results on unsupervised object
discovery. The code will be released at https://github.com/VDIGPKU/FORMULA.

(a) LOST

(b) TokenCut

(c) FORMULA-L (ours)

(d) FORMULA-TC (ours)

Figure 1: Example results of UOD on VOC12. In (a) and (b), we show results
obtained by LOST [40] and TokenCut [52] respectively, which are two state-of-
the-art UOD methods. The results of our method are presented in (c) and (d).
Predictions are in red, and ground-truth boxes are in yellow. We can find that
the proposed FORMULA localizes object more accurately. Best viewed in color.

##  1 Introduction

††footnotetext: ∗Equal contribution. †As an intern at PKU.
${}^{\textrm{\Letter}}$Corresponding author.

Object detection is one of the fundamental problems in computer vision, which
serves a wide range of applications such as face recognition [42], pose
estimation [57], and autonomous driving [44]. In recent years, significant
success has been achieved in the field [36, 35] thanks to the increasing
amount of annotated training data. However, the labeling of large-scale
datasets [31, 38] is rather costly. Although multiple techniques, including
semi-supervised learning [2], weakly-supervised learning [17], and self-
supervised learning [19] have been proposed to alleviate this issue, manual
labeling is still required.

Here we concentrate on a fully unsupervised task for object detection, named
unsupervised object discovery. Previous CNN-based methods [13, 47, 48, 54]
leverage region proposal and localize objects by comparing the proposed
bounding boxes of each image across a whole image collection. However, these
approaches are difficult to scale to large datasets due to the quadratic
complexity brought by the comparison process [49]. Recently, DINO [6] has
found that the attention maps of a Vision Transformer (ViT) [14] pre-trained
with self-supervised learning reveal salient foreground regions. Motivated by
DINO, LOST [40] and TokenCut [52] are proposed to discover objects by
leveraging the high-quality ViT features. Both methods first construct an
undirected graph using the similarity of patch-wise features from the last
layer of the ViT. Then, a two-step seed-selection-expansion strategy and
Normalized Cut [39] are adopted respectively to segment the foreground
objects. While both approaches have achieved superior results over previous
state-of-the-art [48, 49], we have found that they mainly focus on the
construction of the localization head and overlook the potential of improving
the ViT features.

In this paper, we propose a simple but effective feature enhancement method
for existing ViT-based UOD frameworks, named FORMULA. Our method consists of
two parts, i.e., the foreground guidance module and the multi-layer feature
fusion module. For the foreground guidance module, we utilize the object mask
predicted by an off-the-shelf UOD detector to highlight the foreground object
region and then refine the object location through an iterative process.
Specifically, we first generate an object mask from the original ViT feature
map using an existing UOD detector (_e.g_., LOST or TokenCut). Then, we
construct a probability map with 2D Gaussian distribution from the mask, which
roughly localizes the foreground objects. After that, we highlight the
foreground area by applying the probability map to the original ViT feature
map. Finally, the updated feature map is used for the UOD detector to obtain a
refined object mask. The whole process can be iterated. In this way, we
enhance the ViT feature map by introducing the foreground object information
and suppressing background interference. Our method can localize objects much
more accurately with only a few iterations.

Besides, we note that LOST and TokenCut only use the feature map from the last
layer of ViT. However, the scale of objects in non-object-centric images, like
those in COCO [31], can vary greatly. The feature from the last layer of a
pre-trained ViT mainly captures the key areas for classification, which is
usually at a larger scale. Thus, the performance on smaller objects is hurt.
To address this issue, we propose the multi-layer feature fusion module. In
detail, we simply merge the features from the last several layers through a
weighted summation to aggregate multi-scale information for unsupervised
object discovery.

Our main contributions can be summarized as follows:

  * •

We introduce foreground guidance from the object mask predicted by an existing
UOD detector to the original ViT feature map and propose an iterative process
to refine the object location.

  * •

We further design a multi-layer feature fusion module to address the scale
variation issues in object detection, releasing the potential of ViT feature
representation for object discovery.

  * •

The proposed method can be easily incorporated into any existing UOD methods
based on ViT and achieves new state-of-the-art results on the popular PASCAL
VOC and COCO benchmarks.

##  2 Related Work

Self-supervised learning. Learning powerful feature representations in a self-
supervised manner that dispenses with human annotation has made great progress
recently. This is performed by defining a pretext task that provides surrogate
supervision for feature learning [34, 59, 19]. Despite no labels, models
trained with self-supervision have outperformed their supervised counterparts
on several downstream tasks, such as image classification [19, 55, 8, 23, 9,
5, 20, 10, 6, 50, 11, 22] and object detection [53, 7]. While [55, 8, 23, 5,
20, 10] have adopted CNN as pre-training backbones, recent works [6, 50, 11,
22, 29, 28] have explored Transformers [46] for self-supervised visual
learning, demonstrating their superiority over traditional CNN. Our work
utilizes the strong localization capability of self-supervised ViT for
unsupervised object discovery.

Unsupervised object discovery. The goal of unsupervised object discovery is to
localize objects without any supervision. Early works generally rely on image
features encoded by a CNN [24, 48, 54, 49, 12]. These methods need to compare
the extracted features of each image across those of every other image in the
dataset, leading to quadratic computational overhead [49]. Besides, the
dependence on the inter-image similarity results in these methods being unable
to run on a single image. Recently, LOST [40] and TokenCut [52] have been
proposed to address these issues by leveraging the high-quality feature
representation generated from a self-supervised ViT. Their motivation is that
the attention map extracted from the last layer of a pre-trained ViT contains
explicit information about the foreground objects. Specifically, both methods
first propose the construction of an intra-image similarity graph using
features extracted from the ViT backbone [6, 14]. A heuristic seed-selection-
expansion strategy and Normalized Cut [39] are then adopted respectively by
the two methods to segment a single foreground object in an image.

Although achieving excellent performance, these methods mainly concentrate on
localization design and fail to further improve ViT features for unsupervised
object discovery. Instead, our work starts from the perspective of feature
enhancement. Concretely, we introduce the foreground guidance to highlight the
foreground regions on ViT features and propose a multi-layer feature fusion
module to aggregate multi-scale features.

Figure 2: The pipeline of FORMULA. To enhance ViT features for unsupervised
object discovery, we extract multi-layer features to aggregate information
from different scales and introduce foreground guidance from the predicted
segmentation to the input of the UOD detector.

Multi-layer Feature Representations. One of the main challenges in object
detection is to effectively represent features on different scales. Extensive
works have been proposed over the years to deal with the multi-scale problem
with multi-layer features. These methods leverage the pyramidal features of
CNN to compute a multi-scale feature representation. [33, 21, 26, 1, 37, 18,
30] combine low-resolution and high-resolution features with upsampling or
lateral connections to aggregate semantic information from all levels. [32, 4,
30] make predictions at different scales from different layers and use post-
processing to filter the final predictions. In addition to CNN, several works
have recently exploited the multi-layer representation for Transformer
networks. [51] aggregates the class tokens from each Transformer layer to
gather the local, low-level, and middle-level information that is crucial for
fine-grained visual categorization. In [56], the authors divide the multi-
layer representations of the Transformer’s encoder and decoder into different
groups and then fuse these group features to fully leverage the low-level and
high-level features for neural machine translation.

These works inspire us to explore the multi-layer features of ViT for better
object localization. Instead of designing complicated fusion modules, we
propose a simple and efficient fusion method that sums the feature from each
layer of the Transformer with different weights.

##  3 Approach

In this section, we introduce our method for unsupervised object discovery,
i.e., FORMULA. The overall pipeline of FORMULA is presented in Fig. 2.
Firstly, we briefly review Vision Transformers and their previous applications
in UOD as preliminary knowledge. Then, we describe the two modules of FORMULA,
namely foreground guidance and multi-layer feature fusion.

###  3.1 Preliminary

Vision Transformers [14] receive a sequence of image patches and use stacked
multi-head self-attention blocks to extract feature maps from images. It
divides an input image of $H\times W$ into a sequence of $N=HW/P^{2}$ patches
of fixed resolution $P\times P$. Patch embeddings are then formed by mapping
the flattened patches to a $D$-dimensional latent space with a trainable
linear projection. An extra learnable `[CLS]` token is attached to the patch
embeddings and position embeddings are added to form the standard transformer
input in $\mathbb{R}^{(N+1)\times D}$.

DINO [6] has shown that the attention map extracted from the last layer of a
self-supervised ViT indicates prominent foreground areas. Following this
observation, LOST and TokenCut propose to localize objects using the key
features $k\in\mathbb{R}^{N\times D}$ from the last layer in two steps. First,
an intermediate feature map $F_{int}\in\mathbb{R}^{h\times w}$ is constructed
from the inter-patch similarity graph, where $h=H/P$ and $w=W/P$.
Specifically, for LOST, it is the map of inverse degrees; for TokenCut, it is
the second smallest eigenvector of the graph. Second, a object mask
$m\in\\{0,1\\}^{h\times w}$ is generated from $F_{int}$ to segment the
foreground object.

Figure 3: Illustration of the iteration process. The initial prediction made
by LOST [40] (framed in yellow) and the final prediction after four iterations
(right). The red dot in each iteration is the object center calculated from
the previous segmentation.

###  3.2 Foreground Guidance with Self-iteration

In the foreground guidance module, the predicted object mask is treated as the
foreground guidance to highlight the foreground region and guide the
segmentation process. Specifically, given an existing unsupervised object
detector $\mathcal{D}$ and an intermediate feature map $F_{int}$ extracted
from the pre-trained ViT, the binary foreground object mask
$m\in\\{0,1\\}^{h\times w}$ of the object can be generated as follows:

$m=\mathcal{D}(F_{int}).$ (1)

Here, $\mathcal{D}$ could be any ViT-based object discovery methods, e.g.,
LOST and TokenCut. Moreover, the value of $m(\bm{x}_{i})$ equals to 1 if the
corresponding patch $i$ with coordinates $\bm{x}_{i}$ is predicted to belong
to the foreground object. With the foreground mask $m$, the approximate
coordinates of the object center $\mathcal{O}$ can be calculated by

$\texttt{}\bm{x}_{\mathcal{O}}=\frac{1}{\sum_{i=1}^{h\times
w}m(\bm{x}_{i})}\sum_{i=1}^{h\times w}m(\bm{x}_{i})\bm{x}_{i}.$ (2)

Then, we construct a probability map $P\in\mathbb{R}^{h\times w}$ using the 2D
Gaussian distribution function $g$:

$P(i)=g(i|\bm{x}_{\mathcal{O}},\sigma^{2})=\frac{1}{2\pi\sigma^{2}}e^{-\frac{\|\bm{x}_{i}-\bm{x}_{\mathcal{O}}\|^{2}}{2\sigma^{2}}},$
(3)

where $\sigma$ is a hyper-parameter. Intuitively, the value of $P$ indicates
the regions in an image that are likely to belong to an object. The
probability map $P$ can be viewed as the foreground guidance for object
localization, guiding the detector $\mathcal{D}$ to refine the final
prediction in each iteration step. Specifically, we achieve this by applying
the Hadamard product to $P$ and $F_{int}$:

$\widetilde{F_{int}}=F_{int}\circ P.$ (4)

The new feature map $\widetilde{F_{int}}$ can be interpreted as a re-weighting
of $F_{int}$ and the foreground part of $F_{int}$ is emphasized. Thus, during
unsupervised object discovery, the detector will focus more on the foreground
part instead of the whole image.

Finally, we can generate a new object mask through Eq. 1 from
$\widetilde{F_{int}}$ and iterate the whole procedure until convergence, i.e.,
the distance between the centers in two consecutive iterations is smaller than
$\tau$. The overall process is presented in Algorithm 1. An example of the
iteration process is illustrated in Fig. 3. It is worth noting that the
efficiency bottleneck of ViT-based UOD methods lies in the process of feature
extraction rather than localization. Thus, our iteration module only brings
marginal extra computational overhead. More details can be found in 4.3.

Algorithm 1 Foreground Guidance with Self-iteration

Input: Unsupervised Object Detector $\mathcal{D}$, Intermediate Feature Map
$F_{int}$, Standard Deviation $\sigma$ of Gaussian Distribution $g$.

Initialize: $m=\mathcal{D}(F_{int})$, $\bm{x}_{\mathcal{O}}=0$,
$\widehat{\bm{x}_{\mathcal{O}}}=\infty$.

while
$\|\widehat{\bm{x}_{\mathcal{O}}}-\bm{x}_{\mathcal{O}}\|^{2}\geqslant\tau$ do

$\widehat{\bm{x}_{\mathcal{O}}}=\bm{x}_{\mathcal{O}}$

$\bm{x}_{\mathcal{O}}=Eq.\ \ref{eq:eq2}(m)$

$P=g(\bm{x}_{\mathcal{O}},\sigma^{2})$

$\widetilde{F_{int}}=F_{int}\circ P$

$m=\mathcal{D}(\widetilde{F_{int}})$

end while

Output: $m_{ref}=m$.

###  3.3 Multi-layer Feature Fusion

Different layers of Transformers encode features at different scales. Deeper
layers tend to gather global and semantic information and focus on the
discriminative parts of objects. Consequently, the activated areas in their
feature maps are smaller (Fig. 7(b), 7(c)). In contrast, shallower layers
focus on local information and thus the activated areas in their feature maps
are broader (Fig. 7(d), 7(e)). However, the scale of the objects in non-
object-centric images can vary greatly. Only using the feature from one layer
is insufficient to deal with the scale variation problem. To address this
issue, we propose the multi-layer feature fusion module to aggregate
information from various scales.

In detail, from each multi-head attention layer $l$ of a ViT, we can extract
the key feature $k_{l}\in\mathbb{R}^{(N+1)\times D}$. Then, we drop the
`[CLS]` token to be consistent with LOST and TokenCut. The aggregated feature
for unsupervised object discovery is obtained by a weighted summation of the
key features from all layers:

$f=\sum_{l=1}^{L}\alpha_{l}k_{l},$ (5)

where $\alpha_{l}$ is the weight of layer $l$. The features from different
layers of ViT contain information of objects at various scales. The aggregated
feature incorporates more comprehensive object information at different scales
to better localize objects.

##  4 Experiments

In this section, we conduct extensive experiments on various datasets to
demonstrate the effectiveness of our method. FORMULA achieves new state-of-
the-art results in unsupervised object discovery tasks. In addition, we
conduct ablation studies to discuss the effect of foreground guidance and
multi-layer feature fusion modules.

Method | Backbone | VOC07$(\uparrow)$ | VOC12$(\uparrow)$ | COCO_20k$(\uparrow)$
---|---|---|---|---
Selective Search [45] | - | 18.8 | 20.9 | 16.0
EdgeBoxes [60] | - | 31.1 | 31.6 | 28.8
Kim _et al_. [25] | - | 43.9 | 46.4 | 35.1
Zhange _et al_. [58] | VGG16 [41] | 46.2 | 50.5 | 34.8
DDT+ [54] | VGG19 [41] | 50.2 | 53.1 | 38.2
rOSD [48] | VGG16&19 [41] | 54.5 | 55.3 | 48.5
LOD [49] | VGG16 [41] | 53.6 | 55.1 | 48.5
DINO-seg [6, 40] | ViT-S/16 [6, 14] | 45.8 | 46.2 | 42.1
LOST [40] | ViT-S/16 [6, 14] | 61.9 | 64.0 | 50.7
TokenCut [52] | ViT-B/16 [6, 14] | 68.8 | 72.4 | 59.0
FORMULA-L | ViT-S/16 [6, 14] | 64.4 | 67.7 | 54.0
FORMULA-TC | ViT-B/16 [6, 14] | 69.4 | 73.2 | 59.7
LOD + CAD [40] | VGG16 [41] | 56.3 | 61.6 | 52.7
rOSD + CAD [40] | VGG16&19 [41] | 58.3 | 62.3 | 53.0
LOST + CAD [40] | ViT-S/16 [6, 14] | 65.7 | 70.4 | 57.5
TokenCut + CAD∗ [40] | ViT-B/16 [6, 14] | 71.4 | 75.5 | 62.7
FORMULA-L + CAD [40] | ViT-S/16 [6, 14] | 66.8 | 71.5 | 59.0
FORMULA-TC + CAD [40] | ViT-B/16 [6, 14] | 72.2 | 76.2 | 63.7
Table 1: Unsupervised single-object discovery. We compare FORMULA with current
state-of-the-art unsupervised object discovery methods. We use DINO pre-
trained ViT [6] as our backbone. ∗ is our implementation. ‘-L/TC’ means to
adopt LOST/TokenCut as the unsupervised object detector $\mathcal{D}$. ‘+CAD’
means to train a second-stage object detector using pseudo-labels generated
through each method.

Method | Supervis. | aero | bike | bird | boat | bottle | bus | car | cat | chair | cow | table | dog | horse | mbike | person | plant | sheep | sofa | train | tv |  mean$(\uparrow)$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
WSDDN [3] | weak | 39.4 | 50.1 | 31.5 | 16.3 | 12.6 | 64.5 | 42.8 | 42.6 | 10.1 | 35.7 | 24.9 | 38.2 | 34.4 | 55.6 | 9.4 | 14.7 | 30.2 | 40.7 | 54.7 | 46.9 | 34.8
PCL [43] | weak | 54.4 | 9.0 | 39.3 | 19.2 | 15.7 | 62.9 | 64.4 | 30.0 | 25.1 | 52.5 | 44.4 | 19.6 | 39.3 | 67.7 | 17.8 | 22.9 | 46.6 | 57.5 | 58.6 | 63.0 | 43.5
rOSD [48] \+ OD [40] | - | 38.8 | 44.7 | 25.2 | 15.8 | 0.0 | 52.9 | 45.4 | 38.9 | 0.0 | 16.6 | 24.4 | 43.3 | 57.2 | 51.6 | 8.2 | 0.7 | 0.0 | 9.1 | 65.8 | 9.4 | 27.4
LOST [40] \+ OD [40] | - | 57.4 | 0.0 | 40.0 | 19.3 | 0.0 | 53.4 | 41.2 | 72.2 | 0.2 | 24.0 | 28.1 | 55.0 | 57.2 | 25.0 | 8.3 | 1.1 | 0.9 | 21.0 | 61.4 | 5.6 | 28.6
FORMULA-L + OD [40] | - | 62.8 | 3.2 | 45.6 | 23.9 | 0.0 | 53.3 | 41.2 | 74.3 | 0.1 | 18.9 | 32.7 | 60.7 | 59.8 | 27.2 | 11.4 | 0.0 | 0.0 | 38.5 | 39.8 | 2.3 | 29.8
FORMULA-TC + OD [40] | - | 61.4 | 19.2 | 49.4 | 26.1 | 0.0 | 60.5 | 46.6 | 78.7 | 0.4 | 21.2 | 31.8 | 73.7 | 55.2 | 15.8 | 12.2 | 0.0 | 0.0 | 41.5 | 51.5 | 8.9 | 32.7

Table 2: Unsupervised object detection (OD). We evaluate FORMULA on VOC07 test
using<EMAIL_ADDRESS>All the ‘+OD’ methods are trained on VOC07 trainval.

###  4.1 Experimental Settings

Datasets. Following LOST and TokenCut, we evaluate the proposed approach on
three widely-adopted benchmarks for object discovery: VOC07 [15], VOC12 [16],
and COCO [31]. Specifically, for VOC07 and VOC12, we use `trainval` set to
evaluate our method. For COCO, we only use a subset of the COCO2014 `trainval`
dataset, named COCO_20k [48].

Evaluation metric. Same to [48, 54, 49, 40, 52], we use the Correct
Localization (CorLoc) metric for evaluation. In this metric, a predicted
bounding box is considered correct if its intersection over union (IoU) score
with one of the ground truth boxes in an image is greater than 0.5.

Implementation details. We use the weights from the publicly available DINO
pre-trained ViT model [6]. We adopt LOST/TokenCut as our basic unsupervised
object detector $\mathcal{D}$, named FORMULA-L/FORMULA-TC. For backbone
architecture, we adopt ViT-S/16 and ViT-B/16 [14]. We set $\sigma$=0.1 for
FORMULA-L and $\sigma$=1 for FORMULA-TC. $\tau$ is set to $\sqrt{2}$ in our
experiments. For multi-layer feature fusion, we only fuse the features from
the last four layers.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |
(a) LOST |  (b) TokenCut |  (c) FORMULA-L (ours) |  (d) FORMULA-TC (ours)
Figure 4: Example results of unsupervised single-object discovery on VOC07 and
VOC12. In (a) and (b), we show the results obtained by LOST [40] and TokenCut
[52]. Our results are illustrated in (c) and (d). Red and yellow bounding
boxes indicate the predicted bounding boxes and the ground-truth respectively.
From top to bottom, the scale of the objects becomes smaller.

###  4.2 Main Results

####  4.2.1 Unsupervised Single-object Discovery

In Table 1, we present unsupervised single-object discovery performance of
FORMULA on three popular datasets. As shown in the table, our method
outperforms the previous state-of-the-art, LOST and TokenCut, on all three
datasets. Specifically, on VOC07, our method surpasses LOST and TokenCut by
2.5% and 0.6%, respectively. On VOC12, FORMULA obtains 67.7% with ViT-S and
73.2% with ViT-B, improving the performance of LOST and TokenCut by 3.7% and
0.8%, respectively. On the COCO_20k dataset, FORMULA achieves 54.0% and 59.7%,
which significantly surpasses LOST and TokenCut by 3.3% and 0.7%,
respectively. In Fig. 4, we provide some visual results obtained by our method
and the two baselines, i.e., LOST and TokenCut. It can be seen that our method
boosts detection performance on various scales.

Additionally, following [40], we also train a second-stage class-agnostic
object detector (CAD) for unsupervised single-object discovery. Concretely, we
train a Faster R-CNN [36] using the bounding boxes generated by FORMULA as
pseudo-labels. It can be seen from Table 1 that our method outperforms the
state-of-the-art by an average of 0.8% on the three datasets. The consistent
gain over baseline methods shows the effectiveness of FORMULA for unsupervised
object discovery.

####  4.2.2 Unsupervised Object Detection

Following LOST [40], we also evaluate our method on unsupervised object
detection. Similar to training a CAD, a Faster R-CNN detector is trained using
pseudo-labels generated by our method. Concretely, to generate pseudo-labels
for the class-aware detectors, we cluster the boxes produced by FORMULA via
K-means algorithm and then map them to ground-truth classes via Hungarian
matching [27] for evaluation. It is worth noting that no human supervision is
involved during the training process. We use the Average Precision metric with
the threshold of IoU being 0.5, a common setting in PASCAL VOC detection. The
results are shown in Table 2. As we can see, we improve the state-of-the-art
results of unsupervised object detection on VOC07 `test` by a significant
4.1%.

###  4.3 Analysis and Discussion

In this section, we investigate the effectiveness of foreground guidance and
multi-layer feature fusion modules. We conduct the following experiments with
FORMULA-L and use ViT-S/16 as the backbone.

Figure 5: Study of the iteration process. (a) result variations with the number of iterations; (b) the running time of our method with different iteration numbers. Both experiments are conducted on VOC07 trainval. Foreground guidance | Multi-layer | CorLoc$(\uparrow)$
---|---|---
|  | 61.9
✓ |  | 63.7
| ✓ | 63.3
✓ | ✓ | 64.4
Table 3: Ablation experiments. Results on VOC07 trainval. “Foreground guidance” and “Multi-layer” represent foreground guidance module and multi-layer feature fusion module, respectively. Method | Backbone | VOC07$(\uparrow)$ | VOC12$(\uparrow)$ | COCO_20k$(\uparrow)$
---|---|---|---|---
LOST [40] | ViT-S/8 [6, 14] | 55.5 | 57.0 | 49.5
FORMULA-L | ViT-S/8 [6, 14] | 57.9 | 61.3 | 49.6
LOST [40] | ViT-S/16 [6, 14] | 61.9 | 64.0 | 50.7
FORMULA-L | ViT-S/16 [6, 14] | 64.4 | 67.7 | 54.0
LOST [40] | ViT-B/16 [6, 14] | 60.1 | 63.3 | 50.0
FORMULA-L | ViT-B/16 [6, 14] | 62.8 | 66.5 | 53.4
Table 4: Analysis of the backbone. CorLoc score comparison across three
different backbones.

####  4.3.1 Ablation Study

We perform a set of ablation experiments to show the effectiveness of each
component of our FORMULA. Results of the ablations are shown in Table 3. We
can see that the foreground guidance module can bring an improvement of 1.8%
and the multi-layer feature fusion module improves model performance by 1.4%.
When both modules are applied, we have a total 2.5% improvement. The results
demonstrate the effectiveness of the two modules.

####  4.3.2 Main Properties

We discussed some of the properties of FORMULA here.

Foreground guidance. To better understand the effect of foreground guidance,
we manually change the $\bm{x}_{\mathcal{O}}$ of probability map $P$ in Eq. 3
to different locations in the image. The effects are shown in Fig. 6. When
$\bm{x}_{\mathcal{O}}$ falls into the background, the model would miss the
most area of the object. By contrast, if it is located inside the foreground
object and closer to the center of the object, the model could better capture
the semantic layout of the object and improve its performance. The result
suggests that our guidance mechanism can help the model focus more on the
region of interest.

Backbone architecture. In Table 4, we show the results of FORMULAwith
different Transformer backbones. We compare the results between ViT [14] small
(ViT-S) and base (ViT-B) with a patch size of 8 or 16. It can be seen that
FORMULA consistently outperforms LOST across different backbone architectures,
demonstrating the effectiveness and architecture generalizability of FORMULA.

Figure 6: Study of foreground guidance. We manually place the center $\bm{x}_{\mathcal{O}}$ (red points in the first row) in Eq. 3 at different locations. The center $\bm{x}_{\mathcal{O}}$ is placed outside the foreground object (left), at a corner part of the object (middle), and near the center of the object (right). The first and second rows are the resulting object masks and bounding boxes. $\sigma$ | VOC07$(\uparrow)$ | VOC12$(\uparrow)$ | COCO_20k$(\uparrow)$
---|---|---|---
0.01 | 63.7 | 66.6 | 52.8
0.05 | 63.7 | 66.6 | 52.8
0.1 | 63.7 | 66.6 | 52.8
0.3 | 63.1 | 66.3 | 52.6
0.5 | 62.4 | 66.0 | 52.3
Table 5: Ablation of $\sigma$. CorLoc scores for different $\sigma$ on three
datasets. The results are obtained using only the foreground guidance module.

Iteration number. Fig. 5 shows the influence of the number of iterations from
1 to 8. We find surprisingly that just one iteration can improve performance
significantly. An optimal value for our method is 4, which achieves the best
performance of 63.7%. More iterations may slightly reduce performance, which
we assign to the random perturbations after convergence.

Running Time. We present the running time of our method for different numbers
of iterations in Fig. 5. Results of LOST and TokenCut are shown as well. We
measure the inference time on all images of VOC07 `trainval` with a single GTX
TITAN X GPU. It can be seen that our method only brings marginal extra
computational overhead even with 8 iterations.

Datasets | FORMULA-L | FORMULA-TC
---|---|---
VOC07 | [2,1,1,6] | [3,5,1,1]
VOC12 | [1,1,2,6] | [1,6,1,2]
COCO_20k | [0,2,3,5] | [2,7,0,1]
Table 6: Layer weights. The layer weights are the relative weight of each layer, with the sum being one. The weight of the last layer is in front and the others are in the following order. $layer^{-1}$ | $layer^{-2}$ | $layer^{-3}$ | $layer^{-4}$ | CorLoc$(\uparrow)$
---|---|---|---|---
✓ |  |  |  | 61.9
| ✓ |  |  | 61.5
|  | ✓ |  | 62.9
|  |  | ✓ | 63.3
Table 7: Ablation of the contribution of each layer. We use features of each
of the last four layers as input. ‘$layer^{-1}$’ represents the last layer.
The results are acquired on VOC07 trainval.

(a) Input

(b) $layer^{-1}$

(c) $layer^{-2}$

(d) $layer^{-3}$

(e) $layer^{-4}$

Figure 7: The activated areas of attention maps from the last four layers of
ViT. (a) Image sampled from VOC07 [15]; (b)-(e) are the visualized attentions
$F_{int}$. The deeper layers gather global information and focus on the
discriminative parts of the object.

Analysis of $\sigma$. We report the results of using different value of
$\sigma$ in Table 5. It can be observed that the performance slightly
decreases as $\sigma$ increases above 0.1, which we attribute to the fact
that, with a higher value of $\sigma$, the broader 2D Gaussian distribution
would make the model focus on larger areas and thus hurt detection on small
objects. Additionally, our method performs consistently with a $\sigma$
between 0.01 and 0.1. Thus, we simply adopt a typical value of 0.1 in our
experiments.

Multi-layer Fusion weights. The weights of multi-layer fusion weights for our
results in Table 1 are presented in Table 6. FORMULA with LOST is better at
capturing local information of the foreground, thus requiring more global
knowledge from low layers to perform better. Different from FORMULA-L, the
information from the last two layers plays a more important role for FORMULA-
TC.

Besides, to better understand how each layer contributes to the overall
performance, we conduct experiments using each of the four layers. The results
are presented in Table 7. We can see that low layers, such as the third and
fourth ones, play a more important role in gathering features at different
scales, which aligns with the results in Table 6. We also visualize the
attention maps of the last four layers in Fig. 7. The activated areas of the
attention map vary at different scales for different layers. These results
together indicate the ViT features from different layers can help detect
objects at various scales.

##  5 Conclusion

In this work, we propose FORMULA, a simple and effective feature enhancement
method for unsupervised object discovery. We utilize the foreground guidance
acquired by an existing UOD detector to highlight the foreground regions on
the feature maps and iteratively refine the segmentation predictions. In
addition, by fusing the multi-layer features from a self-supervised ViT, we
further aggregate multi-scale information for the feature representation. Our
approach can work with any existing ViT-based unsupervised object discovery
methods. Moreover, FORMULA achieves new state-of-the-art results on three
datasets for unsupervised object discovery task. We hope our work could
inspire more future research on enhancing ViT features for unsupervised visual
learning.

## References

  * [1] Sean Bell, C. Lawrence Zitnick, Kavita Bala, and Ross B. Girshick.  Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [2] David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In Neural Information Processing Systems (NeurIPS), 2019.
  * [3] Hakan Bilen and Andrea Vedaldi.  Weakly supervised deep detection networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [4] Zhaowei Cai, Quanfu Fan, Rogério Schmidt Feris, and Nuno Vasconcelos.  A unified multi-scale deep convolutional neural network for fast object detection.  In European Conference on Computer Vision (ECCV), 2016.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [7] Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, and Dit-Yan Yeung.  Multisiam: Self-supervised multi-instance siamese representation learning for autonomous driving.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In International Conference on Machine Learning (ICML), 2020.
  * [9] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [10] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [11] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [12] Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang, and Jia-Bin Huang.  Show, match and segment: Joint learning of semantic matching and object co-segmentation.  IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2020.
  * [13] Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce.  Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results, 2007.
  * [16] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results, 2012.
  * [17] Xiaoxu Feng, Xiwen Yao, Gong Cheng, and Junwei Han.  Weakly supervised rotation-invariant aerial object detection network.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [18] Golnaz Ghiasi and Charless C. Fowlkes.  Laplacian pyramid reconstruction and refinement for semantic segmentation.  In European Conference on Computer Vision (ECCV), 2016.
  * [19] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In International Conference on Learning Representations (ICLR), 2018\.
  * [20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent - A new approach to self-supervised learning.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [21] Bharath Hariharan, Pablo Andrés Arbeláez, Ross B. Girshick, and Jitendra Malik.  Hypercolumns for object segmentation and fine-grained localization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  arXiv preprint arXiv:2111.06377, 2021.
  * [23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.  Momentum contrast for unsupervised visual representation learning.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [24] Kuang-Jui Hsu, Yen-Yu Lin, and Yung-Yu Chuang.  Co-attention cnns for unsupervised object co-segmentation.  In International Joint Conference on Artificial Intelligence (IJCAI), 2018.
  * [25] Gunhee Kim and Antonio Torralba.  Unsupervised detection of regions of interest using iterative link analysis.  In Neural Information Processing Systems (NeurIPS), 2009.
  * [26] Tao Kong, Anbang Yao, Yurong Chen, and Fuchun Sun.  Hypernet: Towards accurate region proposal generation and joint object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [27] H. W. Kuhn.  The hungarian method for the assignment problem.  Naval Research Logistics Quarterly, 1955.
  * [28] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao.  Efficient self-supervised vision transformers for representation learning.  International Conference on Learning Representations (ICLR), 2022\.
  * [29] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, and Jinqiao Wang.  MST: masked self-supervised transformer for visual representation.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [30] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [31] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In European Conference on Computer Vision (ECCV), 2014.
  * [32] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg.  SSD: single shot multibox detector.  In European Conference on Computer Vision (ECCV), 2016.
  * [33] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [34] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision (ECCV), 2016.
  * [35] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [36] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  In Neural Information Processing Systems (NeurIPS), 2015.
  * [37] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
  * [38] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal on Computer Vision (IJCV), 2015.
  * [39] Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2000.
  * [40] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce.  Localizing objects with self-supervised transformers and no labels.  In British Machine Vision Conference (BMVC), 2021.
  * [41] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations (ICLR), 2015\.
  * [42] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [43] Peng Tang, Xinggang Wang, Song Bai, Wei Shen, Xiang Bai, Wenyu Liu, and Alan L. Yuille.  PCL: proposal cluster learning for weakly supervised object detection.  IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2020.
  * [44] Marvin Teichmann, Michael Weber, J. Marius Zöllner, Roberto Cipolla, and Raquel Urtasun.  Multinet: Real-time joint semantic reasoning for autonomous driving.  In IEEE Intelligent Vehicles Symposium (IV), 2018.
  * [45] Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arnold W. M. Smeulders.  Selective search for object recognition.  International Journal on Computer Vision (IJCV), 2013.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [47] Huy V. Vo, Francis R. Bach, Minsu Cho, Kai Han, Yann LeCun, Patrick Pérez, and Jean Ponce.  Unsupervised image matching and object discovery as optimization.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [48] Huy V. Vo, Patrick Pérez, and Jean Ponce.  Toward unsupervised, multi-object discovery in large-scale image collections.  In European Conference on Computer Vision (ECCV), 2020.
  * [49] Huy V. Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce.  Large-scale unsupervised object discovery.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [50] Feng Wang, Tao Kong, Rufeng Zhang, Huaping Liu, and Hang Li.  Self-supervised learning by estimating twin class distributions.  arXiv preprint arXiv:2110.07402, 2021.
  * [51] Jun Wang, Xiaohan Yu, and Yongsheng Gao.  Feature fusion vision transformer for fine-grained visual categorization.  In British Machine Vision Conference (BMVC), 2021.
  * [52] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz.  Self-supervised transformers for unsupervised object discovery using normalized cut.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [53] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin.  Aligning pretraining for detection via object-level contrastive learning.  In Neural Information Processing Systems (NeurIPS), 2021.
  * [54] Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou.  Unsupervised object discovery and co-localization by deep descriptor transformation.  Pattern Recognition (PR), 2019.
  * [55] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [56] Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Furu Wei, and Zhoujun Li.  Gtrans: Grouping and fusing transformer layers for neural machine translation.  arXiv preprint arXiv:2207.14467, 2022.
  * [57] Sen Yang, Zhibin Quan, Mu Nie, and Wankou Yang.  Transpose: Keypoint localization via transformer.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [58] Runsheng Zhang, Yaping Huang, Mengyang Pu, Qingji Guan, Jian Zhang, and Qi Zou.  Mining objects: Fully unsupervised object discovery and localization from a single image.  arXiv preprint arXiv:1902.09968, 2019.
  * [59] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization.  In European Conference on Computer Vision (ECCV), 2016.
  * [60] C. Lawrence Zitnick and Piotr Dollár.  Edge boxes: Locating object proposals from edges.  In European Conference on Computer Vision (ECCV), 2014.


# Reply to Sergiu I. Vacaru’s “Critical remarks on Finsler modifications of
gravity and cosmology by Zhe Chang and Xin Li”

Xin Li1,3<EMAIL_ADDRESS>Zhe Chang2,3<EMAIL_ADDRESS>1Institute of
Theoretical Physics, Chinese Academy of Sciences, 100190 Beijing, China
2Institute of High Energy Physics, Chinese Academy of Sciences, 100049
Beijing, China
3Theoretical Physics Center for Science Facilities, Chinese Academy of
Sciences

###### Abstract

This is our reply to ”Critical remarks on Finslerian modifications of gravity
and cosmology by Zhe Chang and Xin Li”, Sergiu I. Vacaru, Phys. Lett. B 690
(2010) 224. It is pointed out that the Finslerian modifications of gravity and
cosmology (Zhe Chang and Xin Li, Phys. Lett. B 676 (2009) 173; ibid 668 (2008)
453) is a suggestion on the generalization of Einstein’s gravity and
cosmology, but not a proof for theorems in geometry. False or true of the
theory should be tested by experiments or observations. We show that the
arguments of Sergiu I. Vacaru were based a wrong logic. A personal claim can
not be used to prove any other theory be wrong. To get the claim: “we may
construct more “standard” physical Finsler classical/quantum gravity theories
for metric compatible connections like the Cartan d-connection” , Sergiu I.
Vacaru should complete a consistent presentation at least. We suggest Sergiu
I. Vacaru to make some predictions on gravity and cosmoligy using his
“standard” physical Finsler classical/quantum gravity theories as we did, and
compare them with astronomical observations. By the way, we should say that it
is still really far from a theory of quantum gravity.

Recently, Sergiu I. Vacaru published critical remarksVacaru  on our work
Finslerian modifications of gravity and cosmology Finsler DE ; Finsler DM .
First of all, we thank Vacaru for paying attention to our researches. We are
happy to read any kind of criticisms and comments on the papers. The
Finslerian modifications of gravity and cosmology are really not complete and
in the course of development. However, we found that the comments of Sergiu I.
Vacaru was based a wrong logic. The Finslerian modifications of gravity and
cosmology is a suggestion on the generalization of Einstein’s gravity and
cosmology, but not a proof for theorems in geometry. False or true of the
theory should be tested by experiments or observations. A personal claim can
not be used to prove any other researches to be wrong. To get the claim: “we
may construct more “standard” physical Finsler classical/quantum gravity
theories for metric compatible connections like the Cartan d-connection” ,
Sergiu I. Vacaru should complete a consistent presentation at least. We
suggest Sergiu I. Vacaru to make some predictions on gravity and cosmoligy
using his “standard” physical Finsler classical/quantum gravity theories as we
did, and compare them with astronomical observations.

In the following, we will reply to some key points in the comments of Sergiu
I. Vacaru.

Question1: “The Chern connection is not metric compatible and not the unique
connection in Finsler geometry”.

Reply: It is correct. The Chern connection is not metric compatible. The
statement we gave in Finsler DM  should be replaced by a more clear
presentation. In fact, in the second paper of the seriesFinsler DE , just
before the formula (2), we have already altered the statement as: “In Finsler
manifold, there exists a unique linear connection-the Chern connection. It is
torsion freeness and almost metric compatibility”. Here the word “unique” just
means that the Chern connection is determined by the conditions(or structural
equation) of torsion freeness and almost metric compatibility. It should not
be read as that the Chern connection is the unique connection in Finsler
geometry.

Question2: “The metric incompatibility make more difficult the definition of
spinors and conservation laws in Finsler gravity and does not allow “simple”
(super) string and noncommutative generalizations like we proposed.”

Reply: Our paper just presented a classical modification of gravity and
cosmology and did not concern any aspect of quantum theory of gravity.
Therefore, we do not think it is a comment on our papers. It is strange that a
claim on quantum gravity can be used to criticize a classical theory of
gravity. We should say that here the logic of Sergiu I. Vacaru is wrong. Even
though, we still would like to point out that the comment made a strong
conclusion without any proof. It is a pity that a proof of the no go theorem
can not be found in the commentsVacaru .

Question3: “The Ricci tensor introduced by H. Akbar-Zadeh Akbar  is not
correct for all Finsler geometry/gravity models. There were considered various
types of Ricci type tensors in Finsler geometries.”

Reply: We do not know any mathematician has presented the theorem. Sergiu I.
Vacaru did not give any sound proof about his assertion either. In fact, these
various types of Ricci tensors in Finsler geometry and the so-called
“gravitational field equations” constructed by them depend on the chosen
connection. It implies that different gravitational field equations could be
obtained while one uses different connections to calculate it, even all the
connections are metric compatible. This brings up the problem that which
gravitational field equation is the physical one. On the contrary, the Ricci
tensor that introduced by Akbar-Zadeh Akbar  does not face such a problem. It
is given as

$Ric_{\mu\nu}=\frac{\partial^{2}\left(\frac{1}{2}F^{2}Ric\right)}{\partial
y^{\mu}\partial y^{\nu}},$ (1)

where the Finsler metric is defined as
$g_{\mu\nu}\equiv\frac{\partial^{2}\left(\frac{1}{2}F^{2}\right)}{\partial
y^{\mu}\partial y^{\nu}}$, and the Ricci scalar “$Ric$” is the trace of the
predecessor of the flag curvature. The flag curvature Book by Bao  in Finsler
geometry is the counterpart of the sectional curvature in Riemannian geometry.
It is a geometrical invariant. Furthermore, the same flag curvature is
obtained for any connection that chosen in Finsler space. Thus, the same Ricci
tensor is obtained for any connection that chosen in Finsler space Therefore,
the Ricci tensor introduced by Akbar-Zadeh Akbar  is a reasonable and well-
defined one.

Question4: “Sergiu I. Vacaru claimed that he may construct more “standard”
physical Finsler classical/quantum gravity theories for metric compatible
connections like the Cartan d-connection”.

Reply: To become a theory, a claim should be complete and consistent at least.
The most important thing for a physical theory is that it makes predictions
that can be tested through experiments and observations. To our point of view,
Sergiu I. Vacaru’s Finsler gravity theories are still premature. Newton’s
theory of gravity can not be used to kill Einstein’s general relativity.
Einstein’s equations of gravitational field can not be used to kill the
hypothesis of dark energy and dark matter. Another reason for the
prematureness is that the Einstein’s tensor $E(\hat{D})$ in Sergiu I. Vacaru’s
Finsler gravity theories is not a conserved quantity Vacaru1  (in the sense of
covariant differentiation). This is also pointed out by Sergiu I. Vacaru
himself in his comments (see the footnote 3 and the formula (3) in Vacaru ).
It is well-known that in general relativity the Einstein tensor is a conserved
quantity. The Einstein’s gravitational field equation constructed in such a
form due to the requirement that the energy-momentum tensor must conserve. And
this conservation law of energy-momentum tensor is of vital importance and has
been extensively used and embedded in different branches of modern physics.
Any theory that does not subject to this rule can not be recognized as a
physical one. Neither can Sergiu I. Vacaru’s Finsler gravity theories. At
least, Sergiu I. Vacaru should give a conserved quantity which is the
counterpart of energy-momentum tensor in the framework of Finsler geometry.
Sergiu I. Vacaru has claimed that the quantum gravity theory is “almost sure”
of generalized Finsler type. We wish him publish his claim in an isolated
paper. To discuss details on this topic is out of range of our reply.

Acknowledgements

We would like to thank Prof. Z. Shen and Dr. M. Li for useful discussions. The
work was supported by the NSF of China under Grant No. 10525522 and 10875129.

## References

  * (1) S. Vacaru, Phys. Lett. B 690 (2010) 224, arXiv:1003.0044v2 [gr-qc].
  * (2) Z. Chang and X. Li, Phys. Lett. B 676 (2009) 173.
  * (3) Z. Chang and X. Li, Phys. Lett. B 668 (2008) 453.
  * (4) H. Akbar-Zadeh, Acad. Roy. Belg. Bull. Cl. Sci. (5) 74 (1988) 281.
  * (5) D. Bao, S. S. Chern and Z. Shen, An Introduction to Riemann–Finsler Geometry, Graduate Texts in Mathematics 200, Springer, New York, 2000.
  * (6) S. Vacaru, P. Stavrinos, E. Gaburov, D. Gonta, arXiv:gr-qc/ 0508023\.


# Discrete Hawkes process with flexible residual distribution and filtered
historical simulation

###### Abstract

We introduce a new model which can be considered as a extended version of the
Hawkes process in a discrete sense. This model enables the integration of
various residual distributions while preserving the fundamental properties of
the original Hawkes process. The rich nature of this model enables a filtered
historical simulation which incorporate the properties of original time series
more accurately. The process naturally extends to multi-variate models with
easy implementations of estimation and simulation. We investigate the effect
of flexible residual distribution on estimation of high frequency financial
data compared with the Hawkes process.

##  1 Introduction

In this study, we introduce a discrete version of the Hawkes process. The
Hawkes process introduced by Hawkes (1971) is defined by a stochastic
intensity that is excited by an event and diminishes over time. This process
has been studied for a wide variety of applications, and it has been studied
extensively in the financial field Bacry et al. (2015); Hawkes (2018); Law and
Viens (2015), especially for high-frequency financial data Chen et al. (2022);
Morariu-Patrichi and Pakkanen (2022); Muni Toke and Yoshida (2022); Nyström
and Zhang (2022); Zhang et al. (2023).

The residual extracted through the Hawkes model follows the standard
exponential distribution. However, this property may not be guaranteed in
practice, for example in high-frequency financial markets which are prone to
considerable noise and irregular behavior Lee and Seo (2023).

We propose a discrete version of the Hawkes model, that is considered as an
extended version of the Hawkes process, with a more flexible residual
distribution. In contrast to the previous emphasis on the extension of the
Hawkes model in terms of decay function Chen et al. (2021); Lesage et al.
(2022), quadratic type feedback Aubrun et al. (2023), non-linear model Gao and
Zhu (2018), multi-kernel and multi-variate versionsLu and Abergel (2018), our
model places emphasis on the residual process..

Our enhanced model is capable of describing a diverse range of residuals and
can be effectively utilized in applications such as filtered historical
simulation. Following parameter estimation from real data, the residual
distribution can be inferred under the discrete Hawkes model. Subsequently,
the extracted residuals can be employed in a filtered historical simulation,
which is a form of bootstrap method. The paths thus generated are expected to
have similar probabilistic properties to the original time series compared to
the Hawkes model which uses only the unit exponential as residuals. This
distinct feature underscores the significance of the discrete Hawkes model
relative to its original counterpart.

The remainder of this paper is organized as follows. Section 2 introduces the
discrete Hawkes process, its estimation, and simulation procedure. The Hawkes
process has many uses in finance, and the discrete Hawkes model has many
possibilities for use in the finance field. Accordingly, an example of
financial volatility measurement is introduced in Section 3. Section 4
concludes this paper.

##  2 Model

The Hawkes process is one of point processes. The point process is represented
by a random countable set of points $\\{t_{1},t_{2},\cdots\\}$ on a measurable
space $\mathcal{X}$ with a probability space
$(\Omega,\mathcal{F},\mathbb{P})$. The point process is also interpreted using
a counting measure $N(A)$ which counts the number of points in
$A\subseteq\mathcal{X}$. In particualr, when $\mathcal{X}$ denotes the
timeline represented by $\mathbb{R}$, $N$ can also be considered as a counting
process such that

$N(t)=N((0,t])=\text{the number of }t_{i}\text{ in }(0,t].$

For more precise definitions, see Daley and Vere-Jones (2007); we recall the
definition of the Hawkes process.

######  Definition 1 (Hawkes process).

The Hawkes process $N$ is a point process on $\mathbb{R}$, the stochastic
intensity process of which is given by

$\lambda(t)=\mu+\int_{-\infty}^{t}\alpha\mathrm{e}^{-\beta(t-s)}\mathrm{d}N(s)$

where $\mu,\alpha,\beta>0$ and $\alpha<\beta$.

The autoregressive form with exponential decay represents excitation by events
and diminishing intensities over time. The intensity process is left
continuous with right limits. The log-likelihood function of the Hawkes
process is given by

$\sum_{t_{n}\leq t}\log\lambda(t_{n})-\int_{0}^{t}\lambda(s)\mathrm{d}s.$

Residual of the Hawkes process

$\int_{t_{n}}^{t_{n+1}}\lambda(s)\mathrm{d}s$

follows the standard exponential distribution.

We introduce a discrete version of the Hawkes process where the distribution
of residuals can be defined more flexibly while maintaining excitation and
decreasing intensity.

######  Definition 2.

We assume that a sequence of random variables,
$\\{\varepsilon_{n}\\}_{n=1}^{\infty}$, is independent and identically
distributed on a positive support with $\mathbb{E}[\varepsilon]=1$. Let
$\bm{\theta}^{h}=\\{\mu,\alpha,\beta\\}$ be a parameter set with
$\mu,\alpha,\beta>0$, and $\alpha<\beta$. Then define stochastic processes
$\\{\tau_{n},\lambda_{n}\\}$ such that

$\displaystyle\tau_{n}=\phi^{-1}(\varepsilon_{n};\lambda_{n-1},\bm{\theta}^{h}),$
(1) $\displaystyle\lambda_{n}=\psi(\tau_{n};\lambda_{n-1},\bm{\theta}^{h})$
(2)

where $\lambda_{0}>\mu$ and

$\displaystyle\psi(t;\lambda,\bm{\theta}^{h})$
$\displaystyle=\mu+(\lambda-\mu+\alpha)\mathrm{e}^{-\beta t}$ (3)
$\displaystyle\phi(t;\lambda,\bm{\theta}^{h})$
$\displaystyle=\int_{0}^{t}\psi(s;\lambda,\bm{\theta}^{h})\mathrm{d}s=\mu
t+(\lambda-\mu+\alpha)\frac{1-\mathrm{e}^{-\beta t}}{\beta}.$ (4)

######  Remark 1.

The definition above can be further extended as follows:

$\psi(t,\lambda,\bm{\theta}^{h})=\mu+(\lambda-\mu+\alpha)\zeta(t)$

where $\zeta(t)$ is positive and monotonically decreases with
$\zeta(t)\rightarrow 0$ as $s\rightarrow\infty$ and

$\int_{0}^{\infty}\alpha\zeta(s)\mathrm{d}s<1.$

For simplicity, we sometimes use the following notation:

$\psi_{n}(t)=\psi(t;\lambda_{n-1},\bm{\theta}^{h}),\quad\phi_{n}(t)=\phi(t;\lambda_{n-1},\bm{\theta}^{h}).$

According to Definition 2, $\tau_{n}$ represents the inter-arrival times
between events; that is, $\tau_{n}=t_{n}-t_{n-1}$, and $\lambda_{n}$ is a
discrete version of continuous intensity process; that is,
$\lambda_{n}=\lambda(t_{n})$ where $\lambda$ without subscript denotes the
intensity of the Hawkes process. Since $\lambda_{0}>\mu$, by the positiveness
of the parameters, $\lambda_{n}>\mu$ for all $n>0$ and

$\frac{\partial\phi(t)}{\partial t}=\psi(t)>0.$

Hence $\phi(t)$ is strictly monotonically increasing, and the inverse of
$\phi$ exists. Note that

$\mathbb{E}[\phi_{n}(\tau_{n})|\mathcal{F}_{n-1}]=1,$

where $\mathcal{F}_{n-1}$ denotes the $\sigma$-algebra that consists of
information up to time $t_{n-1}$. If $\varepsilon$ follows a unit exponential
distribution, Definition 2 is equivalent to the definition of the Hawkes
process.

Let $f_{\varepsilon}$ be a probability density function (p.d.f.) of
$\varepsilon$ with parameter set $\bm{\theta}^{\varepsilon}$. As by Eq. (1),

$\tau_{n}=\phi_{n}^{-1}(\varepsilon_{n}).$

the p.d.f. of $\tau_{n}$ is represented by

$f_{\tau_{n}}(t)=f_{\varepsilon}(\phi_{n}(t))\phi^{\prime}_{n}(t)=f_{\varepsilon}(\phi_{n}(t))\psi_{n}(t).$

Thus, the log-likelihood function with the given observations of inter-
arrivals $\\{\tau_{n}\\}_{n=1}^{N},$ is

$\log\mathcal{L}_{N}(\bm{\theta})=\sum_{n=1}^{N}\ell_{n}(\bm{\theta})$

where $\ell_{n}$, the contribution to the log-likelihood of the $n$-th step
with parameter set $\bm{\theta}=\bm{\theta}^{h}\cup\bm{\theta}^{\varepsilon}$,
is

$\ell_{n}(\bm{\theta})=\log
f_{\varepsilon}(\phi_{n}(\tau_{n}))+\log\psi_{n}(\tau_{n})=\log
f_{\varepsilon}(\phi_{n}(\tau_{n}))+\log\lambda_{n}.$ (5)

The maximum likelihood estimator is as follows:

$\hat{\bm{\theta}}=\operatorname*{arg\,max}_{\bm{\theta}}\log\mathcal{L}_{N}(\bm{\theta})=\sum_{n=1}^{N}\log
f_{\varepsilon}(\phi_{n}(\tau_{n}))+\log\psi_{n}(\tau_{n}).$ (6)

Note that if $\varepsilon$ follows the standard exponential distribution, that
is, $f_{\varepsilon}(t)=\mathrm{e}^{-t}$, then

$\displaystyle\ell_{n}(\bm{\theta})$
$\displaystyle=\log\lambda_{n}-\phi_{n}(\tau_{n})=\log\lambda_{n}-\int_{0}^{\tau_{n}}\psi(s;\lambda_{n-1},\bm{\theta})\mathrm{d}s$

which is the same as the log-likelihood function in the Hawkes process.

We assume that $\varepsilon$ does not follow the standard exponential
distribution and is unknown. For example, when fitted to the Hawkes model
using inter-arrivals between events observed in high-frequency financial data,
the residual often differs from the exponential distribution. If we estimate
$\bm{\theta}$ using Eq. (2), which is based on the Hawkes process, without
correctly specifying the distribution of $\varepsilon$, the estimation is a
quasi-maximum likelihood estimation (QMLE). This practice is common because
the natural choice of the quasi-log-likelihood function for the distribution
on semi-infinite interval is exponential. Unlike in the autoregressive
conditional duration model (Engle and Russell, 1998), the QMLE of the discrete
Hawkes process may be biased and inconsistent.

Consider a QMLE under the standard exponential distribution of the residual,
even if it follows a different distribution. A sufficient condition for
consistency is the martingale difference property of the score function
(Engle, 2000). With true parameter $\bm{\theta}_{0}$, to satisfy

$\mathbb{E}[\bm{\mathrm{s}}_{n}(\bm{\theta}_{0})|\mathcal{F}_{n-1}]=\mathbb{E}\left[\left.\frac{\partial\ell_{n}(\bm{\theta}_{0})}{\partial\bm{\theta}}\right|\mathcal{F}_{n-1}\right]=\bm{0},$

that is, the score evaluated at the true parameter is a vector martingale
difference sequence, we should have

$\frac{\partial}{\partial\bm{\theta}}\mathbb{E}[\log\lambda_{n}-\phi_{n}(\tau_{n};\bm{\theta}_{0})|\mathcal{F}_{n-1}]=\mathbb{E}\left[\left.\frac{1}{\psi_{n}(\tau_{n},\bm{\theta}_{0})}\frac{\partial\psi_{n}(\tau_{n},\bm{\theta}_{0})}{\partial\bm{\theta}}-\int_{0}^{\tau_{n}}\frac{\partial\psi_{n}(s,\bm{\theta}_{0})}{\partial\bm{\theta}}\mathrm{d}s\right|\mathcal{F}_{n-1}\right]=\bm{0}.$

More precisely,

$\displaystyle\mathbb{E}\left[\left.\frac{1}{\psi_{n}(\tau_{n})}\mathrm{e}^{-\beta\tau_{n}}-\frac{1-\mathrm{e}^{-\beta\tau_{n}}}{\beta}\right|\mathcal{F}_{n-1}\right]=0$
$\displaystyle\mathbb{E}\left[\left.\frac{1}{\psi_{n}(\tau_{n})}-\tau_{n}\right|\mathcal{F}_{n-1}\right]=0$
$\displaystyle\mathbb{E}\left[\left.\frac{(1-\beta\tau_{n})\mathrm{e}^{-\beta\tau_{n}}}{\psi_{n}(\tau_{n})}-\tau_{n}\mathrm{e}^{-\beta\tau_{n}}\right|\mathcal{F}_{n-1}\right]=0$

which are conditions that are too strong for the flexible distributions of
$\tau$. Therefore, to use the discrete time Hawkes process, it is recommended
to specify the distribution of $\varepsilon$ precisely.

What happens if we use QMLE without specifying the distribution of
$\varepsilon$ exactly? Its consequences will depend on how close the
distribution of the residual is to the unit exponential and on the practice.
An example of the financial data is presented in the next section.

To this end, we propose an algorithm for a filtered historical simulation
(FHS), introduced for the evaluation of risk under volatility models (Barone-
Adesi et al., 1998; Boudoukh et al., 1998), based on the bootstrap method. The
availability of FHS is an advantage of the discrete Hawkes model compared to
the Hawkes model. Based on the discussion thus far, the FHS of the discrete
Hawkes process is established as follows.

  1. 1.

Using the maximum likelihood estimation with Eq. (6), we estimate
$\bm{\theta}^{h}$ based on the observed inter-arrivals
$\\{\tau_{n}\\}_{n=1}^{N}$.

  2. 2.

With estimates $\hat{\bm{\theta}}^{h}$, we compute $\hat{\lambda}_{n}$ based
on observed $\tau_{n}$ and Eq. (2), for $n=1,\cdots,N$, with some
$\hat{\lambda}_{0}$.

  3. 3.

By recalling the definition of $\tau$ in Eq. (1), we infer the residual
process by

$\hat{\varepsilon}_{n}=\phi(\tau_{n};\hat{\lambda}_{n-1},\hat{\bm{\theta}}^{h})$

for $n=1,\cdots,N$.

  4. 4.

We then simulate inter-arrival times and intensities by

$\displaystyle\tilde{\tau}_{n}=\phi^{-1}(\tilde{\varepsilon}_{n};\hat{\lambda}_{n-1},\hat{\bm{\theta}}^{h})$
$\displaystyle\tilde{\lambda}_{n}=\psi(\tilde{\tau}_{n};\hat{\lambda}_{n-1},\hat{\bm{\theta}}^{h})$

where $\tilde{\varepsilon}_{n}$ is randomly selected from
$\\{\hat{\varepsilon}_{n}\\}_{n=1}^{N}$ with replacement, for $n=1,\cdots,N$.

This model can be extended to multivariate versions. Recall that the
definition of the multivariate Hawkes process which is described by the
counting processes $N_{i}$ for each type $i=1,\cdots,m$. More precisely,

$N_{i}(t)=N_{i}((0,t])=\\#\text{ of }t_{i,n}\in(0,t],$

where $\\{t_{i,n}\\}$ is a strictly increasing sequence of random variables
with index $n$ for type $i$.

######  Definition 3 (Multivariate Hawkes process).

With vector representations, the multivariate Hawkes process and intensity
processes are defined by

$\bm{N}_{t}=\begin{bmatrix}N_{1}(t)\\\ \vdots\\\
N_{m}(t)\end{bmatrix},\quad\bm{\lambda}_{t}=\begin{bmatrix}\lambda_{1}(t)\\\
\vdots\\\ \lambda_{m}(t)\end{bmatrix},$

respectively, where

$\bm{\lambda}_{t}=\bm{\mu}+\int_{-\infty}^{t}\bm{h}(t-u)\mathrm{d}\bm{N}_{u}$

with constant $m\times 1$ vector
$\bm{\mu}=\begin{bmatrix}\mu_{1}\cdots\mu_{m}\end{bmatrix}^{\top}$; $\bm{h}$
is defined as follows:

$\bm{h}(t)=\bm{\alpha}\circ\begin{bmatrix}\mathrm{e}^{-\beta_{1}t}&\cdots&\mathrm{e}^{-\beta_{1}t}\\\
\vdots&\ddots&\\\
\mathrm{e}^{-\beta_{m}t}&\cdots&\mathrm{e}^{-\beta_{m}t}\end{bmatrix}$

where $\bm{\alpha}$ is a constant $m\times m$ matrix and $\circ$ denotes the
Hadamard product.

For convenience, we define the parameter sets as follows:

$\bm{\theta}_{ij}^{h}=\\{\mu_{i},\alpha_{ij},\beta_{i}\\},\quad\bm{\theta}_{i}^{h}=\\{\mu_{i},\alpha_{ij},\beta_{i}:1\leq
j\leq m\\},\quad\bm{\theta}^{h}=\bigcup_{i=1}^{m}\bm{\theta}_{i}^{h}$

The discrete version of the multivariate Hawkes process is similarly definede.

######  Definition 4.

We assume that the sequences of random variables,
$\\{\varepsilon_{i,n}\\}_{n=1}^{\infty}$, are independent and identically
distributed on positive support with $\mathbb{E}[\varepsilon_{i}]=1$ for
$i=1,\cdots,m$. We define stochastic processes
$\\{\tau_{n},z_{n},\lambda_{i,n}\\}$ such that

$\displaystyle\tau_{i,n}$
$\displaystyle=\phi^{-1}(\varepsilon_{i,n};\lambda_{i,n-1},\bm{\theta}_{ij}^{h})$
(7) $\displaystyle\tau_{n}$ $\displaystyle=\min_{i}\tau_{i,n},$ (8)
$\displaystyle z_{n}$ $\displaystyle=\operatorname*{arg\,min}_{i}\tau_{i,n},$
(9) $\displaystyle\varepsilon_{n}$ $\displaystyle=\varepsilon_{z_{n},n},$ (10)
$\displaystyle\lambda_{i,n}$
$\displaystyle=\psi(\tau_{n};\lambda_{i,n-1},\bm{\theta}_{ij}^{h})$ (11)

with $j=z_{n-1}$, that is, the type of event in the previous step, and
$\lambda_{i,0}>\mu_{i}$. In addition,

$\displaystyle\psi(t;\lambda,\bm{\theta}_{ij}^{h})$
$\displaystyle=\mu_{i}+(\lambda-\mu_{i}+\alpha_{ij})\mathrm{e}^{-\beta_{i}t}$
(12) $\displaystyle\phi(t;\lambda,\bm{\theta}_{ij}^{h})$
$\displaystyle=\int_{0}^{t}\psi(s;\lambda,\bm{\theta}_{ij}^{h})\mathrm{d}s=\mu_{i}t+(\lambda-\mu_{i}+\alpha_{ij})\frac{1-\mathrm{e}^{-\beta_{i}t}}{\beta_{i}}.$
(13)

For simplicity, let

$\psi_{i,n}(t)=\psi(t;\lambda_{i,n-1},\bm{\theta}_{iz_{n-1}}^{h}),\quad\phi_{i,n}(t)=\phi(t;\lambda_{i,n-1},\bm{\theta}_{iz_{n-1}}^{h}).$

For the multivariate version, the $n$-th contribution of the log-likelihood
function $\ell_{n}$ is

$\ell_{n}(\bm{\theta})=\log\psi_{z_{n},n}(\tau_{n})+\log
f_{\varepsilon}(\phi_{z_{n},n}(\tau_{n}))+\sum_{i\neq
z_{n}}\log(1-F_{\varepsilon}(\phi_{i,n}(\tau_{n})))$ (14)

where $F_{\varepsilon}$ denotes the cumulative distribution function of
$\varepsilon$ and the MLE is

$\hat{\bm{\theta}}=\operatorname*{arg\,max}_{\bm{\theta}}\sum_{n=1}^{N}\ell_{n}(\bm{\theta}).$
(15)

Let $n_{i}\subseteq\\{1,\cdots,N\\}$ be the ordered sequence of indices
associated with type $i$. Additionally, let $n_{i}(k)$ denote the $k$-th
element of $n_{i}$. Subsequently, the residual process associated with $i$,
$\varepsilon_{i,k}$, is defined by

$\varepsilon_{i,k}=\sum_{n=n_{i}(k)+1}^{n_{i}(k+1)}\phi_{i}(\tau_{n};\lambda_{i,n-1},\bm{\theta}_{iz_{n-1}}^{h}).$
(16)

The filtered historical simulation of the multivariate discrete Hawkes process
is established as follows:

  1. 1.

With given observations, $\\{\tau_{n},z_{n}\\}_{n=1}^{N}$, and using the
maximum likelihood estimation with Eq. (15), we estimate $\bm{\theta}$.

  2. 2.

With $\\{\tau_{n},z_{n}\\}_{n=1}^{N}$ and estimates $\hat{\bm{\theta}}^{h}$
obtained in step 1, we compute $\hat{\lambda}_{i,n}$ by Eq. (11) with some
$\hat{\lambda}_{i,0}$ for each $i$.

  3. 3.

We then infer residual processes $\\{\hat{\varepsilon}_{i}\\}_{n=1}^{N}$ by
Eq. (16) for each type $i$ by substituting $\hat{\lambda}_{i,n-1}$ in the
position of $\lambda_{i,n-1}$ and $\hat{\bm{\theta}}^{h}$ in the position of
$\bm{\theta}^{h}$.

  4. 4.

Subsequently, we simulate inter-arrival time, type, and intensity processes as
follows:

$\displaystyle\tilde{\tau}_{i,n}=\phi^{-1}(\tilde{\varepsilon}_{i,n};\tilde{\lambda}_{i,n-1},\hat{\bm{\theta}}^{h})$
$\displaystyle\tilde{\tau}_{n}=\min_{i}\tilde{\tau}_{i,n},$
$\displaystyle\tilde{z}_{n}=\operatorname*{arg\,min}_{i}\tilde{\tau}_{i,n},$
$\displaystyle\tilde{\lambda}_{i,n}=\psi(\tilde{\tau}_{n};\tilde{\lambda}_{i,n-1},\hat{\bm{\theta}}^{h})$

for some $\tilde{\lambda}_{i,0}$, where $\tilde{\varepsilon}_{i,n}$ is
randomly selected from $\\{\hat{\varepsilon}_{i}\\}_{n=1}^{N}$ with
replacement.

##  3 Empirical example

In this section, we apply the filtered historical simulation of the discrete
Hawkes model with high-frequency financial data. The high-frequency raw
financial data on asset price dynamics published by major stock exchanges are
recorded in nanoseconds. Owing to microstructure noise in raw data, sparsely
modified data are often used for financial quantitative analysis instead of
raw data. An example is the Hawkes volatility derived from the parameters in
$\bm{\theta}^{h}$ based on sparsely observed financial high-frequency price
data that shows good performance (Lee, 2022).

There could several ways to generate sparse data, but in this study, the
following method was used. Let $P(t)$ denote the mid-price, the average of the
best bid and ask prices, at time $t$ and $\Delta t$ be a fixed interval to be
used for sparse modification. We start with, for example, time 0 and $P(0)$.
We then observe $P(\Delta t)$ and found the last time $t<\Delta t$, when the
price changed to $P(\Delta t)$. This time and price $P(\Delta t)$ are recorded
as sparsely modified data. The procedure are repeated for each $n\Delta t$ and
$P(n\Delta t)$. During the procedure, if $P(n\Delta t)$ is the same as the
previously recorded price, that is, there was no price change, we move to the
next step without record.

The interval length $\Delta t$ is determined subjectively because the optimal
interval is not known. We examine the effect of the interval length using the
discrete Hawkes model and filtered historical simulations.

First, we fit the sparsely modified high-frequency price dynamics to the
bivariate Hawkes model of Definition 3 with parameter constraints
$\alpha_{11}=\alpha_{22}$ and $\alpha_{12}=\alpha_{21}$, for parsimony, where
$N_{1}$ and $N_{2}$ denote the number of up and down movements of price
dynamics, respectively. More precisely, the price dynamics follows a marked
Hawkes model (Lee and Seo, 2017), but for simplicity, we assume that the mark
sizes (price jump) are independent without future impact on intensities. Using
the Hawkes model estimates, we compute the standard deviation of the daily
price change using the formula in Eq. (17).

Second, the same data are fitted to the bivariate discrete Hawkes model of
Definition 4 with a more flexible p.d.f. of $\varepsilon$ than the standard
exponential, such that

$f_{\varepsilon}(x)=\begin{cases}\frac{p\ell-c}{a}x+c,&\text{if }0<x<a\\\
p\ell\mathrm{e}^{-\ell(x-a)},&\text{if }a\leq x\end{cases}$

where $c=\dfrac{2-2p-pa\ell}{a}$. For $0<x<a$, the p.d.f. is trapezoidal with
probability $1-p$, and for $a\leq x$, it is exponential distribution of the
rate $\ell$ with probability $p$. Given that the expected value of
$\varepsilon$ is one,

$p=\frac{6\ell-2a\ell}{a^{2}\ell^{2}+4a\ell+6}.$

Note that, with $p=1$ and $a=0$, the distribution is the standard exponential.
Using the estimated parameters, $M=10,000$ paths of price dynamics are
generated using filtered historical simulation with the same time horizon as
the original data. During the simulation, the jump size in price change is
also bootstrapped from the set of realized jumps in the sparsely modified
data. Therefore, we generate $M$ simulated counting processes for up and down
movements, along with marks, and the sample standard deviation of the
difference between the up and down price processes is computed for comparison
with the Hawkes volatility.

Figure 1 shows an example of a residual histogram with the stock price of AAPL
on Dec 29, 2015, with an interval length of 0.5 seconds. Stock, date and
interval length are arbitrarily selected. The left side of the figure shows a
histogram of the residual estimated using the Hawkes model. As shown in the
figure, the residual does not follow the standard exponential distribution.
The histogram of residuals obtained using the flexible p.d.f. and discrete
Hawkes model to fit the shape of this residual is shown on the right and shows
a better fitting result.

Figure 1: The histograms of residuals under Haweks process (left) and discrete
Hawkes process (right), AAPL, Dec 29, 2015

From the estimation results in Figure 2, the estimates of the Hawkes model
vary more than thoes of the discrete Hawkes model according to the interval,
especially for $\hat{\alpha}$ and $\hat{\beta}$. However, in the discrete
Hawkes model, $\hat{\alpha}$ and $\hat{\beta}$ are relatively constant over
the interval. The estimates associated with $\mu$ exhibit a similar trend.
Interestingly, both the Hawkes and discrete Hawkes volatilities show stable
estimates over intervals.

(a) $\hat{\mu}$

(b) $\hat{\alpha}$

(c) $\hat{\beta}$

(d) Daily volatility

Figure 2: Various estimates and estimated daily price volatility

##  4 Conclusion

In this study, a discrete Hawkes process with a flexible residual distribution
was introduced. This flexible nature enables filtered historical simulations
and preserved the original properties of the stochastic processes. The
usefulness of the discrete Hawkes model was examined through examples using
high-frequency financial data. Although only a fraction of the applications of
the discrete Hawkes process were investigated in this paper, more diverse
research possibilities could be expected in the future using the discrete
Hawkes model.

##  Appendix A Hawkes volatility formula

We assume that jump size in price dynamics is independent of the previous
information. Subsequently, the Hawkes volatility under the model in Definition
3 is computed by

$\mathrm{Hvol}_{t}=\sqrt{\bm{\mathrm{u}}^{\top}\left[\overline{\bm{\mathrm{Z}}}\circ\bm{\mathrm{B}}+(\overline{\bm{\mathrm{Z}}}\circ\bm{\mathrm{B}})^{\top}+\overline{\bm{\mathrm{Z}}}^{(2)}\circ\mathrm{Dg}(\mathbb{E}[\bm{\lambda}_{t}])\right]\bm{\mathrm{u}}t}$
(17)

where $\bm{\mathrm{u}}=\begin{bmatrix}1&-1\end{bmatrix}^{\top}$ and

$\overline{\bm{\mathrm{Z}}}=\mathbb{E}\begin{bmatrix}z_{1}&z_{2}\\\
z_{1}&z_{2}\end{bmatrix},\quad\overline{\bm{\mathrm{Z}}}^{(2)}=\mathbb{E}\begin{bmatrix}z_{1}^{2}&z_{2}^{2}\\\
z_{1}^{2}&z_{2}^{2}\end{bmatrix}$

with $z_{i}$ representing jump size in price dynamics with type $i$ and
$\mathrm{Dg}$ denoting a diagonal matrix. In addition, $\bm{\mathrm{B}}$ is a
$2\times 2$ matrix that satisfies

$\bm{\mathrm{B}}(\bm{\alpha}-\bm{\beta})^{\top}+\overline{\bm{\mathrm{Z}}}^{\top}\circ\mathbb{E}[\bm{\lambda}_{t}\bm{\lambda}_{t}^{\top}]+\mathrm{Dg}(\mathbb{E}[\bm{\lambda}_{t}])\left(\bm{\alpha}\circ\overline{\bm{\mathrm{Z}}}\right)^{\top}-\mathrm{Dg}(\overline{\bm{\mathrm{Z}}})\mathbb{E}[\bm{\lambda}_{t}]\mathbb{E}[\bm{\lambda}_{t}]^{\top}=\bm{0}.$
(18)

where $\mathbb{E}[\bm{\lambda}_{t}\bm{\lambda}_{t}^{\top}]$ is a $2\times 2$
matrix that satisfies

$(\bm{\alpha}-\bm{\beta})\mathbb{E}[\bm{\lambda}_{t}\bm{\lambda}_{t}^{\top}]+\mathbb{E}[\bm{\lambda}_{t}\bm{\lambda}_{t}^{\top}](\bm{\alpha}-\bm{\beta})^{\top}+\bm{\alpha}\mathrm{Dg}(\mathbb{E}[\bm{\lambda}_{t}])\bm{\alpha}^{\top}=\bm{0}$
(19)

and $\mathbb{E}[\bm{\lambda}_{t}]$ is a $2\times 1$ matrix that satisfies

$\mathbb{E}[\bm{\lambda}_{t}]=(\bm{\beta}-\bm{\alpha})^{-1}\bm{\beta}\bm{\mu}$
(20)

with

$\bm{\mu}=\begin{bmatrix}\mu_{1}\\\
\mu_{2}\end{bmatrix},\quad\bm{\alpha}=\begin{bmatrix}\alpha_{11}&\alpha_{12}\\\
\alpha_{12}&\alpha_{11}\end{bmatrix},\quad\bm{\beta}=\begin{bmatrix}\beta_{1}&0\\\
0&\beta_{2}\end{bmatrix}.$

For more detailed information, please refer to Lee (2022).

## References

  * Aubrun et al. (2023) Aubrun, C., M. Benzaquen, and J.-P. Bouchaud (2023).  Multivariate quadratic Hawkes processes—part I: theoretical analysis.  Quant. Finance 23(5), 741–758.
  * Bacry et al. (2015) Bacry, E., I. Mastromatteo, and J.-F. Muzy (2015).  Hawkes processes in finance.  Market Microstructure and Liquidity 1, 1550005.
  * Barone-Adesi et al. (1998) Barone-Adesi, G., F. Bourgoin, and K. Giannopoulos (1998).  Don’t look back.  Risk 11, 100–103.
  * Boudoukh et al. (1998) Boudoukh, J., M. Richardson, and R. Whitelaw (1998).  The best of both worlds.  Risk 11, 64–67.
  * Chen et al. (2021) Chen, J., A. G. Hawkes, and E. Scalas ([2021] ©2021).  A fractional Hawkes process.  In Nonlocal and fractional operators, Volume 26 of SEMA SIMAI Springer Ser., pp. 121–131. Springer, Cham.
  * Chen et al. (2022) Chen, J., N. Taylor, S. Yang, and Q. Han (2022).  Hawkes processes in finance: market structure and impact.  The European Journal of Finance 28, 621–626.
  * Daley and Vere-Jones (2007) Daley, D. J. and D. Vere-Jones (2007).  An introduction to the theory of point processes: volume II: general theory and structure.  Springer Science & Business Media.
  * Engle (2000) Engle, R. F. (2000).  The econometrics of ultra-high-frequency data.  Econometrica 68, 1–22.
  * Engle and Russell (1998) Engle, R. F. and J. R. Russell (1998).  Autoregressive conditional duration: A new model for irregularly spaced transaction data.  Econometrica 66, 1127–1162.
  * Gao and Zhu (2018) Gao, F. and L. Zhu (2018).  Some asymptotic results for nonlinear Hawkes processes.  Stochastic Process. Appl. 128(12), 4051–4077.
  * Hawkes (1971) Hawkes, A. G. (1971).  Point spectra of some mutually exciting point processes.  Journal of the Royal Statistical Society. Series B (Methodological) 33, 438–443.
  * Hawkes (2018) Hawkes, A. G. (2018).  Hawkes processes and their applications to finance: a review.  Quantitative Finance 18, 193–198.
  * Law and Viens (2015) Law, B. and F. Viens (2015).  Hawkes processes and their applications to high-frequency data modeling.  In I. Florescu, M. C. Mariani, H. E. Stanley, and F. G. Viens (Eds.), Handbook of High-Frequency Trading and Modeling in Finance, Chapter 6, pp. 183–219. John Wiley & Sons.
  * Lee (2022) Lee, K. (2022).  Application of Hawkes volatility in the observation of filtered high-frequency price process in tick structures.  Working paper, arXiv:2207.05939.
  * Lee and Seo (2017) Lee, K. and B. K. Seo (2017).  Marked Hawkes process modeling of price dynamics and volatility estimation.  Journal of Empirical Finance 40, 174–200.
  * Lee and Seo (2023) Lee, K. and B. K. Seo (2023).  Modeling bid and ask price dynamics with an extended Hawkes process and its empirical applications for high-frequency stock market data.  Journal of Financial Econometrics 21.
  * Lesage et al. (2022) Lesage, L., M. Deaconu, A. Lejay, J. A. Meira, G. Nichil, and R. State (2022).  Hawkes processes framework with a Gamma density as excitation function: application to natural disasters for insurance.  Methodol. Comput. Appl. Probab. 24(4), 2509–2537.
  * Lu and Abergel (2018) Lu, X. and F. Abergel (2018).  High-dimensional Hawkes processes for limit order books: modelling, empirical analysis and numerical calibration.  Quant. Finance 18(2), 249–264.
  * Morariu-Patrichi and Pakkanen (2022) Morariu-Patrichi, M. and M. S. Pakkanen (2022).  State-dependent Hawkes processes and their application to limit order book modelling.  Quant. Finance 22(3), 563–583.
  * Muni Toke and Yoshida (2022) Muni Toke, I. and N. Yoshida (2022).  Marked point processes and intensity ratios for limit order book modeling.  Jpn. J. Stat. Data Sci. 5(1), 1–39.
  * Nyström and Zhang (2022) Nyström, K. and C. Zhang (2022).  Hawkes-based models for high frequency financial data.  Journal of the Operational Research Society 73, 2168–2185.
  * Zhang et al. (2023) Zhang, J., J. Wen, and J. Chen (2023).  Modeling market fluctuations under investor sentiment with a hawkes-contact process.  The European Journal of Finance 29, 17–32.

## Acknowledgements

This work has supported by the National Research Foundation of Korea(NRF)
grant funded by the Korea government(MSIT)(No. NRF-2021R1C1C1007692).